Page Index Toggle Pages: 1
Topic Tools
Normal Topic Googlebot problem (Read 1,930 times)
------
Senior Member
Beta Testers
****
Offline



Posts: 528
Googlebot problem
Jun 6th, 2010 at 10:39pm
Post Tools
I don't know if this is a good thing or a bad thing

Googlebot has been really hammering my forum. I guess it's good because I have great placement for different types of searches. (Always on the first page, mostly the top 2 and many times #1)

One minor concern is google is about 25% of the daily bandwidth that is used.  Sad

What I'm really concerned with is I get on the average of  20-30 error message PER HOUR from googlebot (ID'ed from the IP addy)

Errors like this for example:
Code
Select All
Personal Messaging is only for members

http://www.mysite.com/cgi-bin/yabb2/YaBB.pl?board=&action=imshow 



or

Code
Select All
Sorry, this service is for registered members only!

http://www.mysite.com/cgi-bin/yabb2/YaBB.pl?board=&action=viewprofile 




I tried different things with sitemaps and robot.txt and the problem seems that the bots don't understand the highlighted part
Code
Select All
Without giving a reason for applying for membership we cannot process your application.

http://www.mysite.com/cgi-bin/yabb2/YaBB.pl?board=&action=register2 



I tried
Code
Select All
<a href="http://www.mysite.com/cgi-bin/yabb2/YaBB.pl" rel="nofollow">login2</a> 


in /default.html which of course shows up as links on the forum.

I'm stumped, any ideas that would help?

Thanks
« Last Edit: Jun 6th, 2010 at 10:41pm by ------ »  
Back to top
 
IP Logged
 
L_ith
Full Member
***
Offline



Posts: 246
Re: Googlebot problem
Reply #1 - Jun 6th, 2010 at 10:42pm
Post Tools
so... what your saying is..


you checked the error log and you see these messages? to me it just looks like the bot is trying to access sections on your "Forum"  right?


on my forum i have that "Yandax" bot all the time, and if your host is "unlimited bandwidth" why are you worried? lol


unless your not unlimitied.

  
Back to top
 
IP Logged
 
------
Senior Member
Beta Testers
****
Offline



Posts: 528
Re: Googlebot problem
Reply #2 - Jun 6th, 2010 at 11:22pm
Post Tools
I guess I just hate seeing error messages  Wink

If I log 200 error messages, 180 of them would be from googlebot, which makes it difficult to view if there are any problems I may need to deal with.

I can't see any upside of googlebot trying to sign in as a member a few dozen times in a single morning.

I average about 450,000 page views a month. I'm not unlimited, but I barely use 15% of my allotted bandwidth per month.
« Last Edit: Jun 6th, 2010 at 11:29pm by ------ »  
Back to top
 
IP Logged
 
Captain John
Ex Member


Re: Googlebot problem
Reply #3 - Jun 7th, 2010 at 12:29am
Post Tools
Googlebot should never cause these or this amount of errors.  Block that IP, if it is a legitimate bot, google has others.
  
Back to top
 
IP Logged
 
------
Senior Member
Beta Testers
****
Offline



Posts: 528
Re: Googlebot problem
Reply #4 - Jun 7th, 2010 at 1:08am
Post Tools
Thanks Captain, I'll give it a try and see what happens.
  
Back to top
 
IP Logged
 
Captain John
Ex Member


Re: Googlebot problem
Reply #5 - Jun 7th, 2010 at 2:55am
Post Tools
Welcome ...
  
Back to top
 
IP Logged
 
------
Senior Member
Beta Testers
****
Offline



Posts: 528
Re: Googlebot problem
Reply #6 - Jun 15th, 2010 at 5:45pm
Post Tools
Blocking the googlebot dropped me from the top 2 position down to #8  Sad

I hate to unblock the IP but I dislike not having a high ranking more

Any other tips?
  
Back to top
 
IP Logged
 
------
Senior Member
Beta Testers
****
Offline



Posts: 528
Re: Googlebot problem
Reply #7 - Jun 18th, 2010 at 5:04pm
Post Tools
Here's how I solved the problem

Google webmaster tools > Site Configuration > Settings > Parameter handling
  
Back to top
 
IP Logged
 
Homer J. S.
God Member
*****
Offline



Posts: 1,949
Location: Germany

None
Re: Googlebot problem
Reply #8 - Jun 22nd, 2010 at 5:46pm
Post Tools
Googlebot visits me very seldom although we are also #1 if you search for "silent hill forum". If a googlebot is on, then it's never more than one at the same time. Could be because of my robots.txt, but I don't know if it's really working. But somehow I manage to prevent my forum from being flooded by bots  Grin
  

My Homer is not a communist. He may be a liar, a pig, an idiot, a communist, but he is not a porn star.
www.silenthill-forum.de (YaBB 1.3.1 with 150+ mods)
www.retrogamerwelt.de (YaBB 2.5.2)
Back to top
WWWICQ  
IP Logged
 
Page Index Toggle Pages: 1
Topic Tools