Page Index Toggle Pages: 1
Topic Tools
Hot Topic (More than 10 Replies) Web Crawler accessing Private Section (Read 3,620 times)
Dandello
YaBB Administrator
YaBB Next Team
Operations Team
Beta Testers
Support Team
*****
Offline



Posts: 2,262
Location: Earth

YaBB 2.6.0
Re: Web Crawler accessing Private Section
Reply #13 - Feb 13th, 2012 at 3:43pm
Post Tools
Corey - should this be considered a bug or a feature with 'interesting' consequences?

Ah, I see you responded here: http://www.dev.yabbworld.com/cgi-bin/forum/YaBB.pl?num=1329070698

« Last Edit: Feb 13th, 2012 at 4:10pm by Dandello »  

If you only have one solution to a problem you're not trying hard enough!
Back to top
WWW  
IP Logged
 
greydane
Junior Member
**
Offline



Posts: 61
Location: Windsor, Nova Scotia

YaBB 2.6.0
Re: Web Crawler accessing Private Section
Reply #12 - Feb 13th, 2012 at 2:55am
Post Tools
Thanks JonB.  I'll try your .HTACCESS code in the attachments directory.  I did check the forum and the Membership List attachment is only in the private section.  Nothing in the Public or linked to anything that I can find.  Thanks Bruce
  
Back to top
WWW  
IP Logged
 
Corey Chapman
YaBB Administrator
*****
Offline



Posts: 10,015
Location: Rock Hill, South Carolina

None
Re: Web Crawler accessing Private Section
Reply #11 - Feb 13th, 2012 at 12:03am
Post Tools
What probably allowed this to be spidered is what I discovered below.

I don't recall this attachments log feature.  The bad part is that it seems to be accessible to any member even if it is from a board that is private.  I have to manually create that link on any YaBB forum to see it, but it works.  You just add "action=viewdownloads" to the URL right after YaBB.pl? and change "num" to "thread" for a URL of a topic display.

Try it here:  http://www.dev.yabbworld.com/cgi-bin/forum/YaBB.pl?num=1329061489/3#3

You will see what I mean.  This would become:
http://www.dev.yabbworld.com/cgi-bin/forum/YaBB.pl?action=viewdownloads;thread=1...
  

Back to top
IP Logged
 
Dandello
YaBB Administrator
YaBB Next Team
Operations Team
Beta Testers
Support Team
*****
Offline



Posts: 2,262
Location: Earth

YaBB 2.6.0
Re: Web Crawler accessing Private Section
Reply #10 - Feb 12th, 2012 at 8:55pm
Post Tools
Some hosting services also have utilities that will guide you through setting up hotlink protection. (Which is what Bill is describing.) And it's not just big sites that need it - little sites with lots of pictures should use it too.
« Last Edit: Feb 12th, 2012 at 8:57pm by Dandello »  

If you only have one solution to a problem you're not trying hard enough!
Back to top
WWW  
IP Logged
 
Bill Myers
God Member
Beta Testers
*****
Offline



Posts: 1,605
Location: Los Angeles

YaBB 2.4
Re: Web Crawler accessing Private Section
Reply #9 - Feb 12th, 2012 at 7:06pm
Post Tools

Like Jon mentioned, keep in mind that files can be retrieved from a number of places, even those that should otherwise be private. General exceptions are files in your cgi-bin or in a password protected part of your site.

In the case with one of our domains in particular, because we host photos from contributors, including video files (from 1997-2011), we had to curtail the "theft" of our bandwidth. Otherwise, our profits would have been substantially lowered due to unnecessary bandwidth costs.

Thankfully, placing an .htaccess file on our server in a parent folder leading to the folders and files we wanted to protect solved our "bandwidth theft" issues.

In fact, not only did our bandwidth usage drop substantially, but we had a very healthy uptick in our revenues because of the error_page.shtml file/page that showed up whenever an outside link tried to download any of our files.

Important: If you create and upload an .htaccess file as I've coded it below, do yourself a favor, and create/upload an error_page.shtml file to send unwanted traffic to a page of your choice; in our case; advertisements that sustain, and earn money for our site.

Note: You can name the error_page.shtml file whatever you want. Just Remember to include it in the .htaccess file as I've coded it below.

Code
Select All
AuthUserFile /dev/null
AuthGroupFile /dev/null

RewriteEngine On
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://www.PutYourDomainNameHere.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://PutYourDomainNameHere.com/.*$ [NC]
RewriteRule .*\.*$ http://www.PutYourDomainNameHere.com/error_page.shtml [R,L]  



If it matters, the code that I've provided above is standard code that's been passed around to webmasters who have sites with a large amount of traffic. It works.  Smiley

  

Morning, noon, or night, have a great one!

Note: This forum doesn't allow for us to freely edit our posts or topics to make corrections as needed, so please remember to look for subsequent posts if you see any mistakes or outdated information. Sorry for the inconvenience.
Back to top
IP Logged
 
Corey Chapman
YaBB Administrator
*****
Offline



Posts: 10,015
Location: Rock Hill, South Carolina

None
Re: Web Crawler accessing Private Section
Reply #8 - Feb 12th, 2012 at 6:37pm
Post Tools
There's more to it as I've discovered.  I posted it at the development forum, but basically there is a way to manipulate the link to create a YaBB-generated attachments log page that appears to be accessible to all even if it is a private board.
  

Back to top
IP Logged
 
JonB
YaBB Administrator
YaBB Next Team
Operations Team
Beta Testers
Support Team
*****
Offline



Posts: 3,816
Location: Land of the Blazing Sun!

YaBB 2.6.0
Re: Web Crawler accessing Private Section
Reply #7 - Feb 12th, 2012 at 5:43pm
Post Tools
I had somewhat concluded the same (a link in a post) as the cause. But, it does not have to be that -- it could also be the document is referenced in a link elsewhere, such as on a web page (wouldn't even have to be on the same site).

I pointed out this particular security flaw in Attachments previously. (that's why I knew the cure  Wink )  At minimum, an index.html placeholder should be placed in the folder on Apache servers to prevent browsing.

I'm unsure how to construct a completely generalized version of the referring domain, I guess I have some reading to do.

Smiley
  

I find your lack of faith disturbing.
Back to top
IP Logged
 
Corey Chapman
YaBB Administrator
*****
Offline



Posts: 10,015
Location: Rock Hill, South Carolina

None
Re: Web Crawler accessing Private Section
Reply #6 - Feb 12th, 2012 at 4:58pm
Post Tools
I think that is more likely.  Someone has posted the link.  Although if it's the attachment link through YaBB, it should still not be served up, but rather produce a page with an error about no access.  The only way they should be able to get the file is if it is a direct link to it in the attachments folder.  Then a .htaccess is the only way to protect it (from outside domain access), which is something that needs to be in the default package.
  

Back to top
IP Logged
 
Dandello
YaBB Administrator
YaBB Next Team
Operations Team
Beta Testers
Support Team
*****
Offline



Posts: 2,262
Location: Earth

YaBB 2.6.0
Re: Web Crawler accessing Private Section
Reply #5 - Feb 12th, 2012 at 4:18pm
Post Tools
That was my thought as well, Corey.

I haven't had any spiders get into private sections. (Heck, I can't even get my site checker into private sections and I know it ignores robots.txt and other robot related tags.)  I'm wondering if a member has the direct link posted somewhere in a public section or somewhere else entirely.
  

If you only have one solution to a problem you're not trying hard enough!
Back to top
WWW  
IP Logged
 
Corey Chapman
YaBB Administrator
*****
Offline



Posts: 10,015
Location: Rock Hill, South Carolina

None
Re: Web Crawler accessing Private Section
Reply #4 - Feb 12th, 2012 at 4:11pm
Post Tools
Is it just showing the attachment PDF or is it showing  links to the protected board and posts?

If it's in a private board, a spider should not be able to navigate to the board or post that is protected by YaBB.  A spider is essentially a guest user on YaBB, so it should not be presented by YaBB with pages that only certain member groups can access.  While a .htaccess can protect access to it, my question is how did the spider find it in the first place?
« Last Edit: Feb 12th, 2012 at 4:12pm by Corey Chapman »  

Back to top
IP Logged
 
JonB
YaBB Administrator
YaBB Next Team
Operations Team
Beta Testers
Support Team
*****
Offline



Posts: 3,816
Location: Land of the Blazing Sun!

YaBB 2.6.0
Re: Web Crawler accessing Private Section
Reply #3 - Feb 12th, 2012 at 7:35am
Post Tools
THe .htaccess file goes in the yabbfiles/Attachments folder.

The way to test it is to try to open a file that is in the Attachments folder by using its full URL without using the forum.

I guess I should say this will only work on servers that support .htaccess  Wink

Good Luck
Cool

  

I find your lack of faith disturbing.
Back to top
IP Logged
 
greydane
Junior Member
**
Offline



Posts: 61
Location: Windsor, Nova Scotia

YaBB 2.6.0
Re: Web Crawler accessing Private Section
Reply #2 - Feb 12th, 2012 at 1:44am
Post Tools
Thanks JonB!!  Do I create an .htaccess file with the code inside for the /httpdocs/yabbfiles/attachments dir.  Ie inside the Attachments directory itself.

  Or do I add the code to the existing .htaccess file in the yabb2 directory.

Thanks Again
Bruce
  
Back to top
WWW  
IP Logged
 
JonB
YaBB Administrator
YaBB Next Team
Operations Team
Beta Testers
Support Team
*****
Offline



Posts: 3,816
Location: Land of the Blazing Sun!

YaBB 2.6.0
Re: Web Crawler accessing Private Section
Reply #1 - Feb 11th, 2012 at 3:10am
Post Tools
Yes, I think I have solved that before with a .htaccess file for referring domain. I have to go look for the syntax.

It would mean that ONLY requests coming from your domain would be honored (i.e. the forum).

If someone else chimes in in the interim, I will not be offended, LOL. I did that a LONG time ago.

Edited:
I has found it!


Code (HTML)
Select All
SetEnvIf Referer www\.avonflyers\.ns\.ca intra_site_referral
Order Deny,Allow
Deny from all
Allow from env=intra_site_referral

 


Cheesy
« Last Edit: Feb 11th, 2012 at 3:16am by JonB »  

I find your lack of faith disturbing.
Back to top
IP Logged
 
greydane
Junior Member
**
Offline



Posts: 61
Location: Windsor, Nova Scotia

YaBB 2.6.0
Web Crawler accessing Private Section
Feb 10th, 2012 at 4:01am
Post Tools
We have a Yabb 2.4 forum for Radio Controlled Aircraft.  There are 11 Boards with 9 Boards setup for guests and Forum Members and 2 Private Boards set up for Club Members only.  The Club Members are set as a Post Independant User group called "club member.  It works very well, in that, Club Members can see all boards and guests and Forum members can only see the first 9 Boards.  Hope this makes sense....

 Anyway, a google web crawler, and probably others, have found and presented on google our Membership list.  The list is a PDF attachment to a post in the private section.  the bot obviously tunnelled in using the following...www.avonflyers.ns.ca/cgi-bin/ Forum/ yabb2/YaBB.pl?action=downloadfile; file=Avon_RC_Flyers _Membership.pdf.

   Finally the question!! Smiley  Is there any way to prevent this???

Thanks Bruce
  
Back to top
WWW  
IP Logged
 
Page Index Toggle Pages: 1
Topic Tools
 
  « Board Index ‹ Board  ^Top