Page Index Toggle Pages: 1
Topic Tools
Normal Topic What Search Engines? (Read 4,986 times)
JonB
YaBB Administrator
YaBB Next Team
Operations Team
Beta Testers
Support Team
*****
Offline



Posts: 3,814
Location: Land of the Blazing Sun!

YaBB 2.6.0
Re: What Search Engines?
Reply #9 - Sep 8th, 2012 at 3:18am
Post Tools
This Topic was moved here from YaBB-Friendly Website Hosts and Feedback [move by] JonB.
  

I find your lack of faith disturbing.
Back to top
IP Logged
 
JonB
YaBB Administrator
YaBB Next Team
Operations Team
Beta Testers
Support Team
*****
Offline



Posts: 3,814
Location: Land of the Blazing Sun!

YaBB 2.6.0
Re: What Search Engines?
Reply #8 - Jun 8th, 2012 at 4:32pm
Post Tools
This Topic was moved here from YaBB-Friendly Website Hosts and Other Tips [move by] JonB.
  

I find your lack of faith disturbing.
Back to top
IP Logged
 
BloodyRue
Junior Member
**
Offline



Posts: 83

None
Re: What Search Engines?
Reply #7 - Jan 19th, 2012 at 12:07pm
Post Tools
I am working on a "Bot Hunter" mod.

It finds new bots based on potential search items you list in a file.

it also keeps a running stat count of  any known bots hitting your board.

I am currently rewriting the reporting code but the collection code so far has turned up 3 new bots on my board not previously listed in the search engine section.

it can also hunt for anything else you want to find hitting the board if asked.
  

   
Back to top
IP Logged
 
Corey Chapman
YaBB Administrator
*****
Offline



Posts: 10,015
Location: Rock Hill, South Carolina

None
Re: What Search Engines?
Reply #6 - Jan 19th, 2012 at 4:35am
Post Tools
street wrote on Jan 17th, 2012 at 7:06pm:
Can someone please explain what we can do with this search engine list?

In the YaBB admin center, there is a link on the left for "Search Engines".  This simply allows you to put in a list of engine names that YaBB parses (based on the search string you give it) from the environment variables as an engine visits your site.  All of the search engines (and total number of unique visits of each) that are currently traveling your forum will display at the bottom of the board index where it also shows what users are logged in and how many guests are visiting.  It's merely for fun and informational/statistical purposes.

However, these statistics will point out if you are being abused by a particular engine.  Then you can add their information to your site's robots.txt to set the rules that the search engine's spider must follow for your entire site (which can include not allowing it to travel or index at all).
  

Back to top
IP Logged
 
Bill Myers
God Member
Beta Testers
*****
Offline



Posts: 1,602
Location: Los Angeles

YaBB 2.4
Re: What Search Engines?
Reply #5 - Jan 17th, 2012 at 7:17pm
Post Tools
street wrote on Jan 17th, 2012 at 7:06pm:
...
I am getting hit by tons of spammers and it is a private site ...

Like you, I use YaBB 2.4 and I had a lot of spam-bots trying to register in our forum before I implemented a relatively simple 2-step process to stop them. On average, I was deleting or rejecting about 250 spam-bot registrations a day.

So check it out with a big thanks going to ggn and George as you'll be able to see. Click here to this post in the forum to effectively stop those spammers.

Edited:
Smiley

I should add that since implementing these changes, our forum hasn't had one single spam-bot registration get through; only legitimate registrations have been processed.

Smiley


« Last Edit: Jan 17th, 2012 at 7:27pm by Bill Myers »  

Morning, noon, or night, have a great one!

Note: This forum doesn't allow for us to freely edit our posts or topics to make corrections as needed, so please remember to look for subsequent posts if you see any mistakes or outdated information. Sorry for the inconvenience.
Back to top
IP Logged
 
street
YaBB Newcomer
*
Offline



Posts: 4

YaBB 2.6.0
Re: What Search Engines?
Reply #4 - Jan 17th, 2012 at 7:06pm
Post Tools
Can someone please explain what we can do with this search engine list?

Also, I searched for noindex and search engines on this forum and didn't come up with anything related. Where to I add a robots noindex to my forum? I am getting hit by tons of spammers and it is a private site. I figure if they can't find me, it will cut it down. It isn't clear where the code is for the home page. I am on 2.4 and this would be a great feature in a admin checkbox to turn on or off public indexing if you haven't thought of it before. Thanks a ton.
  
Back to top
 
IP Logged
 
Corey Chapman
YaBB Administrator
*****
Offline



Posts: 10,015
Location: Rock Hill, South Carolina

None
Re: What Search Engines?
Reply #3 - Jan 4th, 2012 at 2:27pm
Post Tools
I have updated the list in the first post.
  

Back to top
IP Logged
 
Corey Chapman
YaBB Administrator
*****
Offline



Posts: 10,015
Location: Rock Hill, South Carolina

None
Re: What Search Engines?
Reply #2 - Dec 21st, 2011 at 4:26am
Post Tools
I just added these:

ahrefsbot|Ahrefs Backlinks

seznambot|Seznam CZ

seznam|Seznam CZ

spbot|SEO Profiler
  

Back to top
IP Logged
 
The Boy
Full Member
***
Offline



Posts: 339
Location: UK
Re: What Search Engines?
Reply #1 - Dec 19th, 2011 at 7:07pm
Post Tools
Seeing roughly similar.

Had to block Baidu at robots.txt level, as it was hammering mine (50+ concurrent sessions).
  
Back to top
WWW  
IP Logged
 
Corey Chapman
YaBB Administrator
*****
Offline



Posts: 10,015
Location: Rock Hill, South Carolina

None
What Search Engines?
Dec 18th, 2011 at 4:01pm
Post Tools
What search engines are crawling your website?  I turned on the forum log here for a while to see what new search engines are visiting yabbforum.com.  Surprisingly, a lot of the new crawlers are not for search engines, but rather for SEO optimization sites and marketing sites following particular brands.  Here's the updated list here.  Note, that I could easily add lists that other websites report, but they are huge and contain a lot of small or obsolete crawlers.  Everything you see below has visited here in the last 11 years.

Code
Select All
ahrefsbot|Ahrefs Backlinks
alexa|Alexa
ia_archiver|Alexa
altavista|AltaVista
scooter|AltaVista
sv.av|AltaVista
tarantula|AltaVista
alta-vista|Altavista
av.com|Altavista
aport|Aport
archiver-web|Archive.org
ask.com|Ask Jeeves
askjeeves|Ask Jeeves
directhit.com|Ask Jeeves
teoma|Ask Jeeves
atomz|Atomz
baiduspider|Baidu
baidu|Baidu
net263|Baidu
magpie|Brandwatch
deepindex|DeepIndex
dotbot|Dotbot Research
dotnetdotcom.org|Dotbot Research
dotnetdotcom|Dotbot Research
ezooms|EZooms
muscat|EuroFerret
arachnoidea|EuroSeek
euroseek|EuroSeek
exabot|Exava
architext|Excite
atext|Excite
excite|Excite
fastsearch|Fast Crawler
feedfetcher|Feed Fetcher
feedhub.com|Feed Fetcher
gigabot|Gigablast
backrub|Google
googlebot|Google
google|Google
inktomi|HotBot
inktomisearch|Hotbot
igde|Igde
infoseeksidewinder|InfoSeek
infoseek|InfoSeek
ultraseek|InfoSeek
jike|Jike Search
linklint-checkonly|LinkLint.org
linklint|LinkLint.org
linkdex|Linkdex
magma.com|LookBot
fuzine.mt.cs.cmu.edu|Lycos
lycos|Lycos
bingbot|MS Bing
msnbot-media|MSN Search
msnbot|MSN Search
search.msn|MSN Search
msnbot-udiscovery|MSN Search
renlifangbot|MSN Search
mj12bot|Majestic-12
search.live|Microsoft Live Search
msr-isrccrawler|Microsoft Research
nationaldirectory|National Directory
netinfo.bg|Net Info
netinfobot|Net Info
northernlight|Northern Light Gulliver
omgilibot|Omgili
openfind|Openfind Shark
picosearch|PicoSearch
plonebot|Plone Spambot
reklamza|Reklamza!
teal.rutgers.edu|Rutgers Ru-Robot
spbot|SEO Profiler
scoutjet|ScoutJet
search4free|Search 4 Free
charlotte|SearchMe Visual Search
searchme|SearchMe Visual Search
seznambot|Seznam CZ
seznam|Seznam ZCZ
similarpages|Similar Pages
sistrix|Sistrix
sogou|Sogou
sphere.com|Sphere Scout
rambler|StackRambler
cuill|Twiceler
twiceler|Twiceler
orange-ftpgroup|Voila Spambot
voilabot|Voila Spambot
voila|Voila Spambot
webalta|WebAlta Crawler
webcrawler|WebCrawler
WebmasterCoffee|Webmaster Coffee
ext-gw.trd.fast.no|Wget
wisewire|WiseWire
address vamos.inria.fr|Xyro
yahoo-mmcrawler|Yahoo!
yahoo|Yahoo!
yandex|Yandex
yadirectbot|Yandex
yanga|Yanga WorldSearch
naver|Yeti
nhn|Yeti
yetiBot|Yeti
yeti|Yeti
youdaobot|Youdao
youdao|Youdao
whatuseek|whatUseek
winona|whatUseek 


« Last Edit: Jan 4th, 2012 at 2:26pm by Corey Chapman »  

Back to top
IP Logged
 
Page Index Toggle Pages: 1
Topic Tools
 
  « Board Index ‹ Board  ^Top