Page Index Toggle Pages: 1
Topic Tools
Hot Topic (More than 10 Replies) Re: URGENT: Possible Solution to Gateway Timeout (Read 6,030 times)
Corey Chapman
YaBB Administrator
*****
Offline



Posts: 10,015
Location: Rock Hill, South Carolina

None
Re: URGENT: Possible Solution to Gateway Timeout
Reply #13 - Aug 29th, 2010 at 6:58pm
Post Tools
FYI to get around it, I just moved the entire forum folder which only contains the source files at this point anyway to the persistent folder and created a symlink.  That works fine too, so maybe this option is needed.  We just have to make sure people are very careful not to set everything in that folder as writeable, which they shouldn't be doing anyway.
  

Back to top
IP Logged
 
Corey Chapman
YaBB Administrator
*****
Offline



Posts: 10,015
Location: Rock Hill, South Carolina

None
Re: URGENT: Possible Solution to Gateway Timeout
Reply #12 - Aug 29th, 2010 at 6:12pm
Post Tools
I have it implemented here, but some files using symlinks still cannot be written to even though they are actually in the persistent writeable folder.

Can we add a feature to allow the NSFlock files to go in a folder of choice as Michael indicates?  It could be a simple checkbox and a folder path option to go with it.  I think this would be useful to others in the same situation as we are here.
  

Back to top
IP Logged
 
Jet Li
Legacy Dev Team
Development Team
****
Offline



Posts: 6,588
Location: Hong Kong
Re: URGENT: Possible Solution to Gateway Timeout
Reply #11 - Aug 21st, 2010 at 7:10am
Post Tools
@ Michael Prager
I have tested on my Dev Board too. Still works fine. Smiley

New
Setup.pl

Admin/NewSettings.pl
Admin/Settings_Advanced.pl

Languages/English/Admin.lng
Languages/English/Error.lng

Modules/File/NFSLock.pm

Sources/Subs.pl

in CVS.
  

PM me for YaBB Installation Service
Back to top
WWWGTalkFacebook  
IP Logged
 
Michael Prager
Boardmod Team
Development Team
*****
Offline



Posts: 975
Location: Germany
Re: URGENT: Possible Solution to Gateway Timeout
Reply #10 - Aug 20th, 2010 at 12:47pm
Post Tools
Corey, here is my solution, ready to install. Devs, please review these changes, obviously a bug could have big impact. I've tested this on a test installation on boardmod.org.

To apply the changes, place nfslock.patch in the same dir as YaBB.pl and execute the following in that dir:
Code
Select All
patch -p1 < nfslock.patch 



The patch will add a new file locking option to the admin center.

Please note that since every file reading needs a lock, and every lock needs write permissions in that directory, you will need to chmod the directories Templates/default, Templates/yabb21, Languages/English, yabbfiles/Templates/Forum and yabbfiles/Templates/Admin to allow writing. Otherwise you will get an file permission denied error for every file read from those dirs. Note that this behaviour could be changed if desired. For example, the NFSLock module could be altered to use a single directory to do all the file locking.

This patch also fixes three other issues with the code:

1. Multiple calls to print_output_header() are prevented. Otherwise, invalid http header might be generated

2. template() no longer calls die() when failing to read from template file. Calling die() would create an invalid http response resulting in your browser beeing unable to display the error.

3. Admin/Settings_Advanced.pl did have the wrong validation method set for the $use_flock setting. It was set to boolean instead of number.



Impact on performance:
Enabling this locking method will slow down the board a bit - no surprise here. With locking completely disabled, a board index page load takes between 0.2 and 0.8 secs. With traditional locking enabled, this will go up to 0.3 to 1.0 secs. With NFSLock file locking enabled, the required time goes up to 1.1 to 1.6 secs.
« Last Edit: Aug 20th, 2010 at 1:02pm by Michael Prager »  

nfslock_patch.txt ( 30 KB | 226 Downloads )

Nail here for a new monitor! --> [x]
Back to top
WWWICQ  
IP Logged
 
Michael Prager
Boardmod Team
Development Team
*****
Offline



Posts: 975
Location: Germany
Re: URGENT: Possible Solution to Gateway Timeout
Reply #9 - Aug 19th, 2010 at 11:04am
Post Tools
Wow... fopen code looks really clumsy Shocked. And to make things worse: NFSLock needs to lock the file before it is open()ed. For flock() it's the other way around.  Tongue
  

Nail here for a new monitor! --> [x]
Back to top
WWWICQ  
IP Logged
 
Corey Chapman
YaBB Administrator
*****
Offline



Posts: 10,015
Location: Rock Hill, South Carolina

None
Re: URGENT: Possible Solution to Gateway Timeout
Reply #8 - Aug 19th, 2010 at 1:00am
Post Tools
Devs, we need to code this option into YaBB 2.6 and implement it here ASAP.  This is probably an issue for others in rare cases too. 

Our only other option, although I think we should do both since this is a known issue, is to use MySQL for data here.

This is the update today from SourceForge:
Quote:
Our version of NFS does not, and likely will not in the forseeable future
support locking. Our general recommendation is to use a database for
storage when you need multiple writers. Otherwise feel free to test the
alternate options out and determine what is best for your specific case.

As for other hosts supporting locking, that's typically because they give
you one webhead. If that webhead dies, your website is offline until its
restored onto another host or migrated over. As we are doing mass vhosting
here, there certainly are some limitations. While we do have plans around
getting rid of some of the limitations, flock is not one of them.
  

Back to top
IP Logged
 
Michael Prager
Boardmod Team
Development Team
*****
Offline



Posts: 975
Location: Germany
Re: URGENT: Possible Solution to Gateway Timeout
Reply #7 - Aug 19th, 2010 at 12:56am
Post Tools
Okay some more results:

NFSLock seems seems to do the job!

At least it prevents corruption for my testcase. There are a couple of things to keep in mind though:

1. The package will create temp files and hard links to manage locking. This means that the directory containing the file to lock needs to be writeable (which is the case for all yabb data file I think).

2. The package is not installed on sourceforge. It is however possible to put the .pm file into a local directory and use its functionality.

3. I did not notice any slow down using this method Cool
« Last Edit: Aug 19th, 2010 at 12:58am by Michael Prager »  

Nail here for a new monitor! --> [x]
Back to top
WWWICQ  
IP Logged
 
Corey Chapman
YaBB Administrator
*****
Offline



Posts: 10,015
Location: Rock Hill, South Carolina

None
Re: URGENT: Possible Solution to Gateway Timeout
Reply #6 - Aug 19th, 2010 at 12:52am
Post Tools
Right now we are still running without file locking and have been lucky I suppose.
  

Back to top
IP Logged
 
Michael Prager
Boardmod Team
Development Team
*****
Offline



Posts: 975
Location: Germany
Re: URGENT: Possible Solution to Gateway Timeout
Reply #5 - Aug 18th, 2010 at 4:38pm
Post Tools
I finally found some time to dig a bit into the problem. I've attached a sample script to trigger file corruption. Just open it with your browser to run the test. You can pass a parameter to try yabb's different locking methods:

lockingtest.pl?use_flock=0 # no locking
lockingtest.pl?use_flock=1 # unix
lockingtest.pl?use_flock=2 # windows
lockingtest.pl?use_flock=3 # unix + "use Fcntl ':flock';" call

So here are the results for sourceforge.net:
  • Unix file locking IS working, but will cause exactly 30 secounds deplay. Note that the deplay is only 100% reproducable if the file gets modified by another process (e.g. by modifying via ssh).
  • Window file locking does not work (what a surprise)
  • Disabling locking causes file corruption (no surprise either)
  • adding
    Code
    Select All
    use Fcntl ':flock'; 
    
    
    at the beginning of the script does not resolve the 30 secounds delay, but locking keeps working.

Will have to investigate other alternative file locking mechanisms.
  

lockingtest_pl.txt ( 8 KB | 210 Downloads )

Nail here for a new monitor! --> [x]
Back to top
WWWICQ  
IP Logged
 
Corey Chapman
YaBB Administrator
*****
Offline



Posts: 10,015
Location: Rock Hill, South Carolina

None
Re: URGENT: Possible Solution to Gateway Timeout
Reply #4 - Aug 17th, 2010 at 1:19pm
Post Tools
http://en.wikipedia.org/wiki/File_locking

Quote:
Both flock and fcntl have quirks that occasionally puzzle programmers more familiar with other operating systems.

Mandatory locks have no effect on the unlink function. As a result, certain programs may, effectively, circumvent mandatory locking. The authors of Advanced Programming in the UNIX Environment (Second Edition) observed that the ed editor did so (page 456).

Whether flock locks work on network filesystems, such as NFS, is implementation-dependent. On BSD systems flock calls are successful no-ops. On Linux prior to 2.6.12 flock calls on NFS files would only act locally. Kernel 2.6.12 and above implement flock calls on NFS files using POSIX byte range locks. These locks will be visible to other NFS clients that implement fcntl()/POSIX locks.[1]

Lock upgrades and downgrades release the old lock before applying the new lock. If an application downgrades an exclusive lock to a shared lock while another application is blocked waiting for an exclusive lock, the latter application will get the exclusive lock and lock the first application out.

All fcntl locks associated with a file for a given process are removed when any file descriptor for that file is closed by that process, even if a lock was never requested for that file descriptor. Also, fcntl locks are not inherited by a child process. The fcntl close semantics are particularly troublesome for applications that call subroutine libraries that may access files.
  

Back to top
IP Logged
 
Spikecity
Requiescat In Pace
*
Offline



Posts: 7,981
Location: Third rock from the sun !
Re: URGENT: Possible Solution to Gateway Timeout
Reply #3 - Aug 17th, 2010 at 10:53am
Post Tools
Michael Prager wrote on Aug 17th, 2010 at 12:43am:
I think the first step would be to write a small test environment which lets you reproduce data loss reliable without locking. That way we can test different methods and see if it resolves the problem.


That would be a neat trick to write, as it requires two or more separate users opening/editing a datafile simultaneously in order to get data loss or corruption.
It still sounds strange that NFS does not use flock as this is a Perl hardware IO call, and I presume these drives are mounted using something like Samba, which is fully controlled by the Linux core, also for file locking.
  

Back to top
 
IP Logged
 
Michael Prager
Boardmod Team
Development Team
*****
Offline



Posts: 975
Location: Germany
Re: URGENT: Possible Solution to Gateway Timeout
Reply #2 - Aug 17th, 2010 at 12:43am
Post Tools
I think the first step would be to write a small test environment which lets you reproduce data loss reliable without locking. That way we can test different methods and see if it resolves the problem.
  

Nail here for a new monitor! --> [x]
Back to top
WWWICQ  
IP Logged
 
Corey Chapman
YaBB Administrator
*****
Offline



Posts: 10,015
Location: Rock Hill, South Carolina

None
Re: URGENT: Possible Solution to Gateway Timeout
Reply #1 - Aug 13th, 2010 at 4:01am
Post Tools
I'm a little afraid having it off, but if it's true that the locking method YaBB supports is not supported on the server, we're accomplishing the same thing (no locking) anyway as you stated.

Therefore, I have disabled file locking here for now.  Cross your fingers that the data is OK and that the forum is faster.

We need to find an alternate solution for servers such as this.
  

Back to top
IP Logged
 
Michael Prager
Boardmod Team
Development Team
*****
Offline



Posts: 975
Location: Germany
Re: URGENT: Possible Solution to Gateway Timeout
Aug 13th, 2010 at 12:46am
Post Tools
What do you guys think about turing file locking off until a solution has been found? If file locking isn't working anyway right now, we can aswell turn it off, so it no longer slows down the forum and people stop complaining (for now). Wink

I will not be able to look into the issue the next couple of days as I have to finish my master thesis till monday...
« Last Edit: Aug 13th, 2010 at 12:48am by Michael Prager »  

Nail here for a new monitor! --> [x]
Back to top
WWWICQ  
IP Logged
 
Page Index Toggle Pages: 1
Topic Tools
 
  « Board Index ‹ Board  ^Top