Page Index Toggle Pages: 1
Topic Tools
Normal Topic My forum isn't showing up in search engines unless I put in URL (Read 1,086 times)
Lizbit
YaBB Newcomer
*
Offline



Posts: 43
My forum isn't showing up in search engines unless I put in URL
Dec 31st, 2013 at 10:37pm
Post Tools
Not sure what's going on. Moved my forum from one host to another recently. Discovered I'm not coming up in search engines. I'm pretty sure I used to. Someone said I needed a robots.txt file so I created one. Not knowing what folders were safe to allow/disallow, I looked at a couple other YaBB forums and used theirs as a guide. I'm not sure if that's why, but I am not coming up in Google search engines using my most popular key words - UNLESS I put the URL (my site name) in the google search box as well as my keywords.  I have an html page as my "enter the forum" welcome page, then when they click the link they go into the forum.  What's going on? I need to get better rankings. The site has been on line for five or more years already.  I'm pretty sure it used to come up all the time.  Please help - this is the time of year the site is used the most.  Thank you.

  
Back to top
 
IP Logged
 
Bill Myers
God Member
Beta Testers
*****
Offline



Posts: 1,676
Location: Los Angeles

YaBB 2.5
Re: My forum isn't showing up in search engines unless I put in URL
Reply #1 - Jan 1st, 2014 at 6:32pm
Post Tools
Lizbit wrote on Dec 31st, 2013 at 10:37pm:
Someone said I needed a robots.txt file so I created one. Not knowing what folders were safe to allow/disallow, I looked at a couple other YaBB forums and used theirs as a guide. I'm not sure if that's why, but I am not coming up in Google search engines using my most popular key words - UNLESS I put the URL (my site name) in the google search box as well as my keywords

Unfortunately you got bad advice; very bad advice. There's really no valid need to have a robots.txt file in terms of being listed in search engines.

Quote:
Google's JohnMu confirmed in a Google Webmaster Help thread and even recommended to one webmaster that he/she should remove their robots.txt file "completely."

Note: Google Webmaster Help is a good place for you to search for discussions about using a robots.txt file. Or rather, not using it.

What I can tell you from personal experience is that our most popular site has consistently ranked in the top 10 of both Google's search engine, and Yahoo's. This particular site of ours has pretty much always been on the 1st page of Yahoo since 1997 when our site was introduced online for the 1st time, and it's consistently been on the 1st page of Google's search engine since that company began (founded in 1998).

As of this post, in both Google and Yahoo we're in the number 1 position with our chosen key words. We've never used a robots.txt file.

Bottom line: Because both Google and Yahoo do such a great job at giving users what they want in any given search, you do not need to do anything at all to optimize your site or your forum. Or perhaps more accurately, do not take the advice of so-called search engine experts as they might like to call themselves.

All you really need to do is to be true to what your site and your forum are about, and the search engines will follow. Also, you do not need to be listed on the first few pages of any given search engine to build up a following. That will come by word of mouth, which ironically will help in your search engine standings.
Have fun with your forum, and happy new year! Smiley

« Last Edit: Jan 1st, 2014 at 6:36pm by Bill Myers »  

Morning, noon, or night, have a great one!

Note: This forum doesn't allow for us to freely edit our posts or topics to make corrections as needed, so please remember to look for subsequent posts if you see any mistakes or outdated information. Sorry for the inconvenience.
Back to top
IP Logged
 
Lizbit
YaBB Newcomer
*
Offline



Posts: 43
Re: My forum isn't showing up in search engines unless I put in URL
Reply #2 - Jan 2nd, 2014 at 1:26am
Post Tools
Bill - thanks for the detailed explanation.  What's odd is that my forum DID show up on first or second page of search engines before I moved it to a different server (without a robots.txt file) .  After moving it, I began getting error messages from the Google Webmaster Tools saying the site didn't have a robots.txt file. My friend suggested creating one, saying I *should* have one, so I did.  And since then, my site isn't coming up on the search engines.

I took your advice and deleted the robots.txt file and will see if my site comes up using the keywords I defined.  I'm really not pushing for it to show up in the number one spot - it's not that urgent of a site -- but I'd at least like to see it show up somewhere on page 1 or 2 like it did before. 

Hopefully the removal of the robots.txt file will resolve the search engine issue.  If you think removing the file will *not* have a bearing on how the search engines find me, then I will need more advice.... because at one time it did show up on page 1 or 2. Now, nothing, unless I put the URL in the search box.

Happy New Year to all of you at Yabb, users and admins alike!  Smiley
  
Back to top
 
IP Logged
 
Bill Myers
God Member
Beta Testers
*****
Offline



Posts: 1,676
Location: Los Angeles

YaBB 2.5
Re: My forum isn't showing up in search engines unless I put in URL
Reply #3 - Jan 2nd, 2014 at 6:07am
Post Tools
Lizbit wrote on Jan 2nd, 2014 at 1:26am:
I took your advice and deleted the robots.txt file and will see if my site comes up using the keywords I defined.

Good luck!

If it's any consolation, it's not at all uncommon for sites to use a robots.txt file. It's very common. I think the key in using one, however, is that it be written correctly. As I understand it, the cleanest way to make sure search engines will index your site, which will include your forum, is as follows:

User-agent: *
Disallow:


Coincidentally, the robot.txt file above is what's used for a very popular, and active YaBB forum. As a test to check it out, I queried a post in that particular forum that was made only yesterday (as of this post), and it's already coming up in a Google search.

Still, my preference is not to use a robots.txt file. After all (referenced here), the following is how Google explains it:

Quote:
You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file (not even an empty one).


************************************

Interestingly, this very forum incorrectly uses its own robots.txt file as follows:

Quote:
User-agent: *
Crawl-delay: 30
Disallow: /cgi-bin/

User-agent: Googlebot-Image
Disallow: /

Hint: "Disallow: /" actually tells a search engine to "disallow everything" that begins with a slash. Thankfully for this forum, and generally speaking, Google routinely ignores that kind of a robots.txt command, and indexes sites to better serve their users.

  

Morning, noon, or night, have a great one!

Note: This forum doesn't allow for us to freely edit our posts or topics to make corrections as needed, so please remember to look for subsequent posts if you see any mistakes or outdated information. Sorry for the inconvenience.
Back to top
IP Logged
 
Lizbit
YaBB Newcomer
*
Offline



Posts: 43
Re: My forum isn't showing up in search engines unless I put in URL
Reply #4 - Jan 5th, 2014 at 7:04pm
Post Tools
Bill,

Now that my robots.txt file has been deleted, the emails from Googlebot / webmaster tools are coming back in on a daily basis:  "Googlebot can't access your site."  Encountered two errors while attempting to access my robots.txt files. Overall error rate is 100%. 

What can I do to a) appease Webmaster Tools, and b) stop these emails from coming in daily?  Should I put in a robots.txt file with the correct (altho not recommended) useage as in your example, above?


User-agent: *
Disallow:

I'm going to say my robots.txt file was not prepared correctly, either; now that I've deleted it altogether - per your advice - my site is back to page 2 on the google search engines, and should be back on page one soon.  My forum is pretty much relevant to a small geographical area, so when someone is googling for info, my site is one of very few that is relevant - so it's likely to be back on page one shortly.

  
Back to top
 
IP Logged
 
Bill Myers
God Member
Beta Testers
*****
Offline



Posts: 1,676
Location: Los Angeles

YaBB 2.5
Re: My forum isn't showing up in search engines unless I put in URL
Reply #5 - Jan 5th, 2014 at 7:55pm
Post Tools
Lizbit wrote on Jan 5th, 2014 at 7:04pm:
I'm going to say my robots.txt file was not prepared correctly, either; now that I've deleted it altogether - per your advice - my site is back to page 2 on the google search engines, and should be back on page one soon.

I believe you are correct. Because you have a relationshiop/account with Google in terms of being registered with them for their webmaster tools, the wisest thing to do is to follow their instructions/prompts despite what I've advised.

At the same time, Google is clearly indexing your site even without a robots.txt file (as that is what they routinely do anyway). In any case, I think you'll be fine with the following robots.txt file in the root of your site as follows:

User-agent: *
Disallow:


Happily, as you were able to see for yourself, removing the robots.txt file brought you back into Google's good graces, so congratulations on that. As I understand it, however, the robots.txt file is or will be a thing of the past as search engines evidently search per their own parameters anyway, and I'm guessing they'll most likely rely on properly placed .htaccess files to skip past folders that a webmaster doesn't want indexed.

*************************************

On a related note, between <HEAD> and </HEAD> in your site's index page, years ago the following was recommended:

<META NAME="robots" CONTENT="FOLLOW,INDEX">

And with sites that had new content every day, the following was also recommended as an addition to the text above (underneath the line in bold above):

<meta name="revisit-after" CONTENT="1 days">

However, as I understand it, that too is old hat, and no longer a recommendation or a concern. We still have those two lines on our most updated site, and a variation of the content portion for revisiting on the other sites that aren't updated so regularly.

But again, I think that's old school, and no longer needed. Smiley

  

Morning, noon, or night, have a great one!

Note: This forum doesn't allow for us to freely edit our posts or topics to make corrections as needed, so please remember to look for subsequent posts if you see any mistakes or outdated information. Sorry for the inconvenience.
Back to top
IP Logged
 
Page Index Toggle Pages: 1
Topic Tools
 
  « Board Index ‹ Board  ^Top