ScientiaMobile

Hi,

We have two concerns regarding your service and they are:

1) We are having exceptions when our site connects to your API. Most probably, all the API calls are working well with all the (human) clients but when it come to bots we have been receiving lot of errors. However we are not sure if this a bot specific error or a general error.

Following are the details of each exceptions we have been collecting so far. (User agent vs the exception)

30th Jan
HTTP_USER_AGENT: Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+) Exception: HTTP Request timed out
31st Jan
HTTP_USER_AGENT: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Exception: HTTP Request timed out
HTTP_USER_AGENT: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Exception: Unable to contact server: fsock Error: Connection timed out

FYI, generally, we have been receiving 2-3 exceptions per day.
Can you please check and let us know the cause of the above issues.

FYI, the following is the API functions we are using
---------------------------------------
require_once path .'library/WURFL/data/WurflCloudClient/Client/Client.php';
// Create a WURFL Cloud Config
$config = new WurflCloud_Client_Config();
// Set your API Key here
$config->api_key = 'API key';
// Create a WURFL Cloud Client
$client = new WurflCloud_Client_Client($config, new WurflCloud_Cache_Null());
// Detect the visitor's device
$client->detectDevice();
$client->getDeviceCapability('is_tablet')
$client->getDeviceCapability('can_assign_phone_number')
----------------------------------------

2) Additionally, can you please confirm the IP addresses from which you serve our API requests, because we are going to implement a restriction on outbound traffic on our firewall, so we need to whitelist all (IP, port) combinations:

I believe they are:

a) WURFL - Scientia Mobile - 50.57.185.227 | port 80
b) WURFL secondary IP - 31.222.190.95 | port 80

Thanks,
Ashan

Ashan,

I am checking with a few of our engineers to see what might be causing this issue for you and will let you know as soon as I know more.

Thank you,

Elliot

First, you should enable caching in production:
That will probably make the errors disappear. The underlying problem is probably due to random latency on your connection. A few failed HTTP requests per day is not totally unusual considering the nature of the public internet. We make sure that all requests received are answered quickly, usually in 2 milliseconds or less. If a request takes over 1 second, it is forcefully closed to prevent pending connections from piling up on your web servers.

With regards to the IP addresses, you should whitelist the following hosts, understanding that their IPs could change in the future:
All traffic is served on port 80.

Note also that the client will do DNS lookups for api.wurflcloud.com.

HI Steve/Elliot,

Thanks for your reply.

We have now enabled the caching.

However Re. HTTP requests timeout issue: Something common that we noted is all these requests are from search engine bots. Please note the below "timeout" list within 2nd of Feb.

9:00a.m - HTTP_USER_AGENT: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Exception: HTTP Request timed out
6:39p.m. - HTTP_USER_AGENT: Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+) Exception: Unable to contact server: fsock Error: Connection timed out
6:39p.m. - HTTP_USER_AGENT: adidxbot/2.0 (+http://search.msn.com/msnbot.htm) Exception: Unable to contact server: fsock Error: Connection timed out
7:35p.m. - HTTP_USER_AGENT: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Exception: HTTP Request timed out

Please confirm if the above is normal for a day and again these are with the bots. So we can consider this be solved.

Thanks,
Ashan

Ashan,

Are you able to let me know if these errors were occurring before or after caching was enabled?

Thank you,

Elliot

I would not consider a few timeouts per day to be out of the ordinary. Note that you can use a try...catch block to catch exceptions from the cloud client and handle them appropriately (like by serving the desktop site).

Hi Elliot/Steve

Thanks for your reply.

Elliot - Re. your question
I can confirm that we still receive this time out issue - Today morning we received the following.
HTTP_USER_AGENT: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Exception: HTTP Request timed out

Steve - Yes, we have already done this and we serve our desktop site in this scenario.

Elliot, if you agree with Steve's comment, we can consider this to be solved.

Thanks,
Ashan

Hi,

In addition to my previous reply:

With regard to the reply from Steve on Mon Jun 06 above, Steve clarified the hosts used for WURFL. Our firewall needs to be upgraded to whitelist by host, so actually at the moment we would like to whitelist by IP. Therefore can you confirm:

a) If the IPs change, can we be notified of such changes so we can place the necessary arrangements to our firewall outbound policy in advance? If so, how can we be notified?

b) Can you confirm they definitely mean these IP's:

lb01-us.wurflcloud.com = 50.57.185.227 (api.wurflcloud.com as well)
lb02-us.wurflcloud.com = 50.57.182.149
lb01-uk.wurflcloud.com = 31.222.190.95
lb02-uk.wurflcloud.com = 31.222.191.152
lb01-za.wurflcloud.com = 41.76.214.41

Thanks,
Ashan

Ashan,

I certainly do agree with Steve in regards to a few failed HTTP requests each day and I can also confirm the IP's listed as well. If you do not already receive ScientiaMobile announcements feel free to register you email address here and you will be notified of any IP changes.

Thank you,

Elliot

Hi Elliot,

Thanks for your confirmation and we have subscribed ourselves to the announcements.

Thanks,
Ashan