Problem with WURFL lock functionality

coretrek
Posts: 4
Joined: Wed Aug 17, 2011 5:49 am

Problem with WURFL lock functionality

Postby coretrek » Wed Jan 18, 2012 5:49 am

Hi,

The class WURFL_DeviceRepositoryBuilder tries to obtain an exclusive file lock to prevent multiple instances of the updater from updating the WURFL at the same time.

The lock is by default (none-solaris) taken by running flock() on a file -inside- the WURFL codebase.

This seems risky, as there is no guarantee that the current user (webserver) has write access to the files in the WURFL code base.
Some of our customers have the application files (including wurfl) mounted over NFS, as this enables them to maintain one set of application files on one server or SAN, and then mount this server/location on to several frontend web servers.
When running WURFL on such a file system, flock() will in most cases not work. On the specific server where we encountered this, flock() actually hangs forever, resulting in the entire server going down as all Apache processes were waiting for flock() to return.

We suggest the lock file is
a) Written inside one of the configured directories (persistant or cache), which should already be writable for the web server
and
b) be optional configurable, so that for those having NFS file systems etc, one can configure WURFL to create it's lock file at a safe, local file system.

Best regards,
Arve Skjørestad
Product Manager, CoreTrek AS

kamermans
Posts: 393
Joined: Mon Jun 06, 2011 9:50 am

Re: Problem with WURFL lock functionality

Postby kamermans » Wed Jan 18, 2012 11:12 am

Hi Arve,

I see your problem and indeed, we will address this. Here are a few more options that I can think of:
  • Use sys_get_temp_dir() or something similar to find the system temp dir, then touch() a file called "wurfl-update.lock" and flock() it. If the temp dir is unavailable, we can use the preconfigured folder.
  • flock() the WURFL file itself
  • Create a temp file and unlink() it when finished with the update, but this runs the risk of the update crashing and never releasing it's lock
Can you tell me which OS and PHP version you're using out of curiosity?
Thanks,

Steve Kamerman
ScientiaMobile

Make sure you check out our WURFL Cloud, WURFL InSight and WURFL InFuze products!

coretrek
Posts: 4
Joined: Wed Aug 17, 2011 5:49 am

Re: Problem with WURFL lock functionality

Postby coretrek » Mon Feb 06, 2012 7:13 am

Hi Steve,

Sorry for my late reply, busy times at CoreTrek for the time beeing ;)

The OS version is CentOS.
PHP version is: PHP 5.3.3-7+squeeze3 with Suhosin-Patch
Zend Engine v2.3.0,

I would appreciate that the temp dir was configurable, as most applications embedding WURFL (at least ours) already has a way of setting/getting the temp-directory for the application itself, and then WURFL as well should use the same. If no explicit configuration is set, syst_get_temp_dir() seems fine to use.
The way we do it when creating such locks, is that we use mkdir() to avoid race conditions (see examples of why here: http://docstore.mik.ua/orelly/webprog/pcook/ch18_25.htm). So I would vote for creating a directory inside the temp-directory as the lock mechanism.

best regards,
Arve Skjørestad

kamermans
Posts: 393
Joined: Mon Jun 06, 2011 9:50 am

Re: Problem with WURFL lock functionality

Postby kamermans » Mon Feb 06, 2012 10:30 am

Hi Arve,

Thanks for the input - the temp dir will be configurable, and since we've already got problems with flock() on Solaris and NFS (perhaps other filesystems as well), I think I'll take your advice and use a directory entry as a exclusive lock indicator, with a destructor that removes the directory in case an exception is thrown while loading.
Thanks,

Steve Kamerman
ScientiaMobile

Make sure you check out our WURFL Cloud, WURFL InSight and WURFL InFuze products!

coretrek
Posts: 4
Joined: Wed Aug 17, 2011 5:49 am

Re: Problem with WURFL lock functionality

Postby coretrek » Tue Apr 22, 2014 6:16 am

Hi again,

We experienced the same problem again in a newer version of the WURFL API (version 1.5).

Do you have any ETA for when this will be fixed?

regards,
Arve Skjørestad
CoreTrek AS

kamermans
Posts: 393
Joined: Mon Jun 06, 2011 9:50 am

Re: Problem with WURFL lock functionality

Postby kamermans » Tue Apr 22, 2014 9:42 am

Hi Arve,

Are you referring to the locking exclusivity? There is code in place to handle this which creates a temp file and attempts to flock() it. If this is not working, perhaps your temp dir is on a filesystem that does not support locking. I'll need to be able to reproduce this in order to fix it, so any information you can provide would be helpful.
Thanks,

Steve Kamerman
ScientiaMobile

Make sure you check out our WURFL Cloud, WURFL InSight and WURFL InFuze products!

coretrek
Posts: 4
Joined: Wed Aug 17, 2011 5:49 am

Re: Problem with WURFL lock functionality

Postby coretrek » Wed Apr 23, 2014 5:23 am

Hi again,

Just read the entire thread here, the problem is still the same as described in the first posts.

The code from DeviceRepositoryBuilder.php (from wurfl-php-1.5.1.0.tar.gz downloaded today) is still:

Code: Select all

public function build($wurflFile, $wurflPatches = array(), $capabilityFilter = array()) {
		// TODO: Create a better locking solution
		if (!$this->isRepositoryBuilt()) {
			// Determine Lockfile location
			if (strpos(PHP_OS, 'SunOS') === false) {
				$this->lockFile = dirname(__FILE__)."/DeviceRepositoryBuilder.php";
			} else {
				// Solaris can't handle exclusive file locks on files unless they are opened for RW
				$this->lockStyle = 'w+';
				$this->lockFile = WURFL_FileUtils::getTempDir().'/wurfl.lock';
			}
	
So as long as we are not using sunOS, the WURFL code still tries to flock() a file inside the code base and NOT in the temp directory. The code base is in our case located on a NFS share, where flock() has no effect. It would probably be better to just always use the solaris solution (or, as I suggested above, use mkdir() instead).

kamermans
Posts: 393
Joined: Mon Jun 06, 2011 9:50 am

Re: Problem with WURFL lock functionality

Postby kamermans » Wed Apr 23, 2014 10:13 am

Hi Arve,

Sorry about the confusion, I see what you are referring to. We will work on this today and have something ready by tomorrow. I do like the mkdir approach since it's atomic, but I need to figure out how to handle stray lockfile conditions, where the API crashes during a load and the lockfile is orphaned. I'll probably do some timestamp checking on the lockfile, and if it's older than, say, 1 day, touch it and reload.
Thanks,

Steve Kamerman
ScientiaMobile

Make sure you check out our WURFL Cloud, WURFL InSight and WURFL InFuze products!

kamermans
Posts: 393
Joined: Mon Jun 06, 2011 9:50 am

Re: Problem with WURFL lock functionality

Postby kamermans » Wed Apr 23, 2014 3:39 pm

Hi Arve,

I've implemented a locking system that uses mkdir and it tests OK. I've uploaded it to your customer vault (go to http://www.scientiamobile.com/myaccount and look for File Manager, then in the Downloads folder). Please give it a spin to confirm it solves your problem, then I will release it as v1.5.1.1.

Here is the new logic:

Code: Select all

	/**
	 * Builds DeviceRepository in PersistenceProvider from $wurflFile and $wurflPatches using $capabilityFilter 
	 * @param string $wurflFile Filename of wurfl.xml or other complete WURFL file
	 * @param array $wurflPatches Array of WURFL patch files
	 * @param array $capabilityFilter Array of capabilities to be included in the DeviceRepository
	 * @return WURFL_CustomDeviceRepository
	 */
	public function build($wurflFile, $wurflPatches = array(), $capabilityFilter = array()) {
		if (!$this->isRepositoryBuilt()) {
			// If acquireLock() is false, the WURFL is being reloaded in another thread
			if ($this->acquireLock()) {
				set_time_limit(600);
				$infoIterator = new WURFL_Xml_VersionIterator($wurflFile);
				$deviceIterator = new WURFL_Xml_DeviceIterator($wurflFile, $capabilityFilter);
				$patchIterators = $this->toPatchIterators($wurflPatches , $capabilityFilter);
				
				$this->buildRepository($infoIterator, $deviceIterator, $patchIterators);
				$this->setRepositoryBuilt();
				$this->releaseLock();
			}
			
		}
		
		$deviceClassificationNames = $this->deviceClassificationNames();
		return new WURFL_CustomDeviceRepository($this->persistenceProvider, $deviceClassificationNames);
	}
	
	public function __destruct() {
		$this->releaseLock();
	}
	
	/**
	 * Acquires a lock so only this thread reloads the WURFL data, returns false if it cannot be acquired
	 * @return boolean
	 */
	private function acquireLock() {
		 
		if (file_exists($this->lockFile)) {
			$stale_after = filemtime($this->lockFile) + $this->maxLockAge;
			if (time() > $stale_after) {
				// The lockfile is stale, delete it and reacquire a lock
				@rmdir($this->lockFile);
			} else {
				// The lockfile is valid, WURFL is probably be reloaded in another thread
				return false;
			}
		}
		
		// Using mkdir instead of touch since mkdir is atomic
		$this->isLocked = @mkdir($this->lockFile, 0775);
		return $this->isLocked;
	}
	
	/**
	 * Releases the lock if one was acquired
	 */
	private function releaseLock() {
		if (!$this->isLocked) {
			return;
		}
		
		@rmdir($this->lockFile);
		$this->isLocked = false;
	}
Thanks,

Steve Kamerman
ScientiaMobile

Make sure you check out our WURFL Cloud, WURFL InSight and WURFL InFuze products!


Who is online

Users browsing this forum: No registered users and 2 guests