YaBB Community and Support Forum
YaBB Home About YaBB Download YaBB YaBB Support Customize Your Forum Development Contribute to the Project
  Welcome, Guest. Please Login or Register


 
Page Index Toggle Pages: 1
Topic Tools
 
Googlebot problem (Read 1,698 times)
 Jun 6th, 2010 at 10:39pm
There are no actions to perform.  

RonS2 
Full Member
Beta Testers
***
Offline
Posts: 447


YaBB 2.5
Googlebot problem
I don't know if this is a good thing or a bad thing

Googlebot has been really hammering my forum. I guess it's good because I have great placement for different types of searches. (Always on the first page, mostly the top 2 and many times #1)

One minor concern is google is about 25% of the daily bandwidth that is used.  Sad

What I'm really concerned with is I get on the average of  20-30 error message PER HOUR from googlebot (ID'ed from the IP addy)

Errors like this for example:
Code Select All
Personal Messaging is only for members

http://www.mysite.com/cgi-bin/yabb2/YaBB.pl?board=&action=imshow 



or

Code Select All
Sorry, this service is for registered members only!

http://www.mysite.com/cgi-bin/yabb2/YaBB.pl?board=&action=viewprofile 




I tried different things with sitemaps and robot.txt and the problem seems that the bots don't understand the highlighted part
Code Select All
Without giving a reason for applying for membership we cannot process your application.

http://www.mysite.com/cgi-bin/yabb2/YaBB.pl?board=&action=register2 



I tried
Code Select All
<a href="http://www.mysite.com/cgi-bin/yabb2/YaBB.pl" rel="nofollow">login2</a> 


in /default.html which of course shows up as links on the forum.

I'm stumped, any ideas that would help?

Thanks
« Last Edit: Jun 6th, 2010 at 10:41pm by RonS2 »  
 
IP Logged  
 Reply #1 - Jun 6th, 2010 at 10:42pm
There are no actions to perform.  

L_ith 
Full Member
***
Offline
Posts: 246


None
Re: Googlebot problem
so... what your saying is..


you checked the error log and you see these messages? to me it just looks like the bot is trying to access sections on your "Forum"  right?


on my forum i have that "Yandax" bot all the time, and if your host is "unlimited bandwidth" why are you worried? lol


unless your not unlimitied.

 
 
IP Logged  
 Reply #2 - Jun 6th, 2010 at 11:22pm
There are no actions to perform.  

RonS2 
Full Member
Beta Testers
***
Offline
Posts: 447


YaBB 2.5
Re: Googlebot problem
I guess I just hate seeing error messages  Wink

If I log 200 error messages, 180 of them would be from googlebot, which makes it difficult to view if there are any problems I may need to deal with.

I can't see any upside of googlebot trying to sign in as a member a few dozen times in a single morning.

I average about 450,000 page views a month. I'm not unlimited, but I barely use 15% of my allotted bandwidth per month.
« Last Edit: Jun 6th, 2010 at 11:29pm by RonS2 »  
 
IP Logged  
 Reply #3 - Jun 7th, 2010 at 12:29am
There are no actions to perform.  
Captain John 
Ex Member


None
Re: Googlebot problem
Googlebot should never cause these or this amount of errors.  Block that IP, if it is a legitimate bot, google has others.
 
 
IP Logged  
 Reply #4 - Jun 7th, 2010 at 1:08am
There are no actions to perform.  

RonS2 
Full Member
Beta Testers
***
Offline
Posts: 447


YaBB 2.5
Re: Googlebot problem
Thanks Captain, I'll give it a try and see what happens.
 
 
IP Logged  
 Reply #5 - Jun 7th, 2010 at 2:55am
There are no actions to perform.  
Captain John 
Ex Member


None
Re: Googlebot problem
Welcome ...
 
 
IP Logged  
 Reply #6 - Jun 15th, 2010 at 5:45pm
There are no actions to perform.  

RonS2 
Full Member
Beta Testers
***
Offline
Posts: 447


YaBB 2.5
Re: Googlebot problem
Blocking the googlebot dropped me from the top 2 position down to #8  Sad

I hate to unblock the IP but I dislike not having a high ranking more

Any other tips?
 
 
IP Logged  
 Reply #7 - Jun 18th, 2010 at 5:04pm
There are no actions to perform.  

RonS2 
Full Member
Beta Testers
***
Offline
Posts: 447


YaBB 2.5
Re: Googlebot problem
Here's how I solved the problem

Google webmaster tools > Site Configuration > Settings > Parameter handling
 
 
IP Logged  
 Reply #8 - Jun 22nd, 2010 at 5:46pm
There are no actions to perform.  

Homer J. S. 
God Member
*****
Offline
Posts: 1,949
Germany


YaBB 2.5
Re: Googlebot problem
Googlebot visits me very seldom although we are also #1 if you search for "silent hill forum". If a googlebot is on, then it's never more than one at the same time. Could be because of my robots.txt, but I don't know if it's really working. But somehow I manage to prevent my forum from being flooded by bots  Grin
 
My Homer is not a communist. He may be a liar, a pig, an idiot, a communist, but he is not a porn star.

www.silenthill-forum.de (YaBB 1.3.1 with 150+ mods)
www.retrogamerwelt.de (YaBB 2.5.2)
WWW 87313348  
IP Logged  
Page Index Toggle Pages: 1
Topic Tools
 

Get Yet another Bulletin Board at SourceForge.net. Fast, secure and Free Open Source software downloads Support This Project BoardMod - YaBB features and templates YaBB Codex - support on installation and usage YaBB Toolbar for your browser

YaBB Facebook Group Page

Vulnerability Scanner

Valid RSS Valid XHTML Valid CSS Powered by Perl
YaBB Chat and Support Community » Powered by YaBB 3.0 Beta!
YaBB Forum Software © 2000-2011. All Rights Reserved.