Automatically updating commercial WURFL file
Posted: Mon Feb 13, 2012 11:21 am
Edit: For the most up-to-date method of automatically updating the wurfl.xml, please read the support documentation here. The information below may be outdated and is left intact to serve as an archive.
All ScientiaMobile API customers are given access to a direct download URL for obtaining the latest WURFL snapshot. This is either the weekly snapshot (released on Sunday night), or an Out-Of-Band snapshot that we release to our customers between weekly snapshots in case of a significant improvement to the data (for example, if a high-profile device is released mid-week and it is not yet in WURFL). The snapshots are licensed to your organization with your commercial license included.
To create a direct download URL, login to http://www.scientiamobile.com/ and go to My Account in the top-right of the screen. Click on "direct download link" to create a URL for your access. This URL is unique to you and must be kept private.
Checking for WURFL updates automatically
The direct download URL supports the following methods to check for updates:
Wget
Warning: Wget does not properly implement any of the above methods, and only simulates them by requesting the entire file and breaking the connection after the HTTP response headers arrive!
To get the latest WURFL Snapshot with Wget in a UNIX-like environment, use this command, replacing /path/for/wurfl and the download URL with your information. Note that /path/for/wurfl is a directory name without the trailing slash; the WURFL file (wurfl.zip) will be located inside that directory.
curl
The curl program supports most methods above, but there seems to be a bug with the way it generates the If-Modified-Since timestamp, causing it to download the file even though it has not changed.
This is the preferred method to use curl in a UNIX-like environment for downloading the WURFL Snapshot:
Windows
I'm trying to find a good method for retrieving a file only if it has been updated for Windows, but there's not a simple one-liner that I can find. The closest built-in partial solutions that I can find are by using Bitsadmin with /setcustomheaders or by writing a short PowerShell script. Feel free to post a better solution.
aria2
The aria2 project is a very robust file downloader that can be used to update the WURFL file:
You can download aria2 from http://aria2.sourceforge.net/ .
.NET Framework
Here's a fully-function C# Console Application for downloading the latest WURFL.
All ScientiaMobile API customers are given access to a direct download URL for obtaining the latest WURFL snapshot. This is either the weekly snapshot (released on Sunday night), or an Out-Of-Band snapshot that we release to our customers between weekly snapshots in case of a significant improvement to the data (for example, if a high-profile device is released mid-week and it is not yet in WURFL). The snapshots are licensed to your organization with your commercial license included.
To create a direct download URL, login to http://www.scientiamobile.com/ and go to My Account in the top-right of the screen. Click on "direct download link" to create a URL for your access. This URL is unique to you and must be kept private.
Checking for WURFL updates automatically
The direct download URL supports the following methods to check for updates:
- HTTP HEAD Request: You can send an HTTP HEAD request to your direct download URL and it will return the size of the current snapshot and the date last modified. Note that the snapshot is not created until the first time you visit this link after the snapshot is released, for example, we release a snapshot on Sunday night and you check it on Tuesday, the last modified date will be Tuesday, not Sunday.
- HTTP If-Modified-Since: If you include the If-Modified-Since HTTP header, the server will only return the snapshot if it is newer than the specified date.
- HTTP If-None-Match / ETag: Our download system sends an HTTP ETag header in along with the snapshot. If this ETag is recorded, you can send it back on subsequent requests in the HTTP If-None-Match header, and the snapshot will only be returned if it is different than yours.
Wget
Warning: Wget does not properly implement any of the above methods, and only simulates them by requesting the entire file and breaking the connection after the HTTP response headers arrive!
To get the latest WURFL Snapshot with Wget in a UNIX-like environment, use this command, replacing /path/for/wurfl and the download URL with your information. Note that /path/for/wurfl is a directory name without the trailing slash; the WURFL file (wurfl.zip) will be located inside that directory.
Code: Select all
WURFL_DIR=/path/for/wurfl; wget -N -P "$WURFL_DIR" --header="If-Modified-Since: $(date -r $WURFL_DIR/wurfl.zip --utc --rfc-2822 2>/dev/null || date --utc --rfc-2822 --date='1 week ago')" http://www.scientiamobile.com/wurfl/xxxxx/wurfl.zipThe curl program supports most methods above, but there seems to be a bug with the way it generates the If-Modified-Since timestamp, causing it to download the file even though it has not changed.
This is the preferred method to use curl in a UNIX-like environment for downloading the WURFL Snapshot:
Code: Select all
curl -Rz "$(date -r /path/to/wurfl.zip --utc --rfc-2822 2>/dev/null || date --utc --rfc-2822 --date='1 week ago')" -o /path/to/wurfl.zip http://www.scientiamobile.com/wurfl/xxxxx/wurfl.zipI'm trying to find a good method for retrieving a file only if it has been updated for Windows, but there's not a simple one-liner that I can find. The closest built-in partial solutions that I can find are by using Bitsadmin with /setcustomheaders or by writing a short PowerShell script. Feel free to post a better solution.
aria2
The aria2 project is a very robust file downloader that can be used to update the WURFL file:
Code: Select all
aria2c.exe --remote-time=true --allow-overwrite=true --conditional-get=true --dir=C:\path\to\wurfl https://data.scientiamobile.com/xxxxx/wurfl.zip.NET Framework
Here's a fully-function C# Console Application for downloading the latest WURFL.
Code: Select all
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Net;
namespace WURFLDownloader
{
class Program
{
static void Main(string[] args)
{
string remoteUrl = "http://www.scientiamobile.com/wurfl/xxxxxx/wurfl.zip";
string localFileName = @"C:\temp\wurfl.zip";
DateTime lastModified = File.Exists(localFileName) ? File.GetLastWriteTime(localFileName) : DateTime.Now.AddDays(-7.0);
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(remoteUrl);
request.Timeout = 10000;
request.AllowWriteStreamBuffering = false;
request.UserAgent = "WURFL Downloader/dotNet";
request.IfModifiedSince = lastModified;
Console.WriteLine("Downloading WURFL file...");
try
{
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream streamResponse = response.GetResponseStream();
FileStream fileStream = new FileStream(localFileName, FileMode.Create);
byte[] read = new byte[4096];
int count = streamResponse.Read(read, 0, read.Length);
while (count > 0)
{
fileStream.Write(read, 0, count);
count = streamResponse.Read(read, 0, read.Length);
}
fileStream.Close();
File.SetLastWriteTime(localFileName, response.LastModified);
streamResponse.Close();
response.Close();
Console.WriteLine("Successfully downloaded WURFL to: " + localFileName);
}
catch (WebException e)
{
if (e.Response != null)
{
if (((HttpWebResponse)e.Response).StatusCode == HttpStatusCode.NotModified)
Console.WriteLine("The WURFL File is up to date.");
else
Console.WriteLine("Unexpected status code = " + ((HttpWebResponse)e.Response).StatusCode);
}
else
Console.WriteLine("Unexpected Web Exception " + e.Message);
}
}
}
}