Exception - HTTP Request timed out

Questions about the WURFL Cloud service.
duoboots3000
Posts: 5
Joined: Fri Jan 31, 2014 7:35 am

Exception - HTTP Request timed out

Postby duoboots3000 » Fri Jan 31, 2014 7:42 am

Hi,

We have two concerns regarding your service and they are:

1) We are having exceptions when our site connects to your API. Most probably, all the API calls are working well with all the (human) clients but when it come to bots we have been receiving lot of errors. However we are not sure if this a bot specific error or a general error.

Following are the details of each exceptions we have been collecting so far. (User agent vs the exception)

30th Jan
HTTP_USER_AGENT: Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+) Exception: HTTP Request timed out
31st Jan
HTTP_USER_AGENT: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Exception: HTTP Request timed out
HTTP_USER_AGENT: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Exception: Unable to contact server: fsock Error: Connection timed out

FYI, generally, we have been receiving 2-3 exceptions per day.
Can you please check and let us know the cause of the above issues.

FYI, the following is the API functions we are using
---------------------------------------
require_once path .'library/WURFL/data/WurflCloudClient/Client/Client.php';
// Create a WURFL Cloud Config
$config = new WurflCloud_Client_Config();
// Set your API Key here
$config->api_key = 'API key';
// Create a WURFL Cloud Client
$client = new WurflCloud_Client_Client($config, new WurflCloud_Cache_Null());
// Detect the visitor's device
$client->detectDevice();
$client->getDeviceCapability('is_tablet')
$client->getDeviceCapability('can_assign_phone_number')
----------------------------------------

2) Additionally, can you please confirm the IP addresses from which you serve our API requests, because we are going to implement a restriction on outbound traffic on our firewall, so we need to whitelist all (IP, port) combinations:

I believe they are:

a) WURFL - Scientia Mobile - 50.57.185.227 | port 80
b) WURFL secondary IP - 31.222.190.95 | port 80

Thanks,
Ashan

Elliotfehr

Re: Exception - HTTP Request timed out

Postby Elliotfehr » Fri Jan 31, 2014 9:56 am

Ashan,

I am checking with a few of our engineers to see what might be causing this issue for you and will let you know as soon as I know more.

Thank you,

Elliot

kamermans
Posts: 393
Joined: Mon Jun 06, 2011 9:50 am

Re: Exception - HTTP Request timed out

Postby kamermans » Fri Jan 31, 2014 10:13 am

First, you should enable caching in production:

Code: Select all

// Change this:
$client = new WurflCloud_Client_Client($config, new WurflCloud_Cache_Null()); 

// To this:
$client = new WurflCloud_Client_Client($config); 
That will probably make the errors disappear. The underlying problem is probably due to random latency on your connection. A few failed HTTP requests per day is not totally unusual considering the nature of the public internet. We make sure that all requests received are answered quickly, usually in 2 milliseconds or less. If a request takes over 1 second, it is forcefully closed to prevent pending connections from piling up on your web servers.

With regards to the IP addresses, you should whitelist the following hosts, understanding that their IPs could change in the future:

Code: Select all

lb01-us.wurflcloud.com
lb02-us.wurflcloud.com
lb01-uk.wurflcloud.com
lb02-uk.wurflcloud.com
lb01-za.wurflcloud.com
All traffic is served on port 80.

Note also that the client will do DNS lookups for api.wurflcloud.com.
Thanks,

Steve Kamerman
ScientiaMobile

Make sure you check out our WURFL Cloud, WURFL InSight and WURFL InFuze products!

duoboots3000
Posts: 5
Joined: Fri Jan 31, 2014 7:35 am

Re: Exception - HTTP Request timed out

Postby duoboots3000 » Mon Feb 03, 2014 1:29 am

HI Steve/Elliot,

Thanks for your reply.

We have now enabled the caching.

However Re. HTTP requests timeout issue: Something common that we noted is all these requests are from search engine bots. Please note the below "timeout" list within 2nd of Feb.

9:00a.m - HTTP_USER_AGENT: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Exception: HTTP Request timed out
6:39p.m. - HTTP_USER_AGENT: Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+) Exception: Unable to contact server: fsock Error: Connection timed out
6:39p.m. - HTTP_USER_AGENT: adidxbot/2.0 (+http://search.msn.com/msnbot.htm) Exception: Unable to contact server: fsock Error: Connection timed out
7:35p.m. - HTTP_USER_AGENT: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Exception: HTTP Request timed out

Please confirm if the above is normal for a day and again these are with the bots. So we can consider this be solved.

Thanks,
Ashan

Elliotfehr

Re: Exception - HTTP Request timed out

Postby Elliotfehr » Mon Feb 03, 2014 10:46 am

Ashan,

Are you able to let me know if these errors were occurring before or after caching was enabled?

Thank you,

Elliot

kamermans
Posts: 393
Joined: Mon Jun 06, 2011 9:50 am

Re: Exception - HTTP Request timed out

Postby kamermans » Mon Feb 03, 2014 4:38 pm

I would not consider a few timeouts per day to be out of the ordinary. Note that you can use a try...catch block to catch exceptions from the cloud client and handle them appropriately (like by serving the desktop site).
Thanks,

Steve Kamerman
ScientiaMobile

Make sure you check out our WURFL Cloud, WURFL InSight and WURFL InFuze products!

duoboots3000
Posts: 5
Joined: Fri Jan 31, 2014 7:35 am

Re: Exception - HTTP Request timed out

Postby duoboots3000 » Wed Feb 05, 2014 12:29 am

Hi Elliot/Steve

Thanks for your reply.

Elliot - Re. your question
I can confirm that we still receive this time out issue - Today morning we received the following.
HTTP_USER_AGENT: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Exception: HTTP Request timed out

Steve - Yes, we have already done this and we serve our desktop site in this scenario.

Elliot, if you agree with Steve's comment, we can consider this to be solved.

Thanks,
Ashan

duoboots3000
Posts: 5
Joined: Fri Jan 31, 2014 7:35 am

Re: Exception - HTTP Request timed out

Postby duoboots3000 » Wed Feb 05, 2014 12:52 am

Hi,

In addition to my previous reply:

With regard to the reply from Steve on Mon Jun 06 above, Steve clarified the hosts used for WURFL. Our firewall needs to be upgraded to whitelist by host, so actually at the moment we would like to whitelist by IP. Therefore can you confirm:

a) If the IPs change, can we be notified of such changes so we can place the necessary arrangements to our firewall outbound policy in advance? If so, how can we be notified?

b) Can you confirm they definitely mean these IP's:

lb01-us.wurflcloud.com = 50.57.185.227 (api.wurflcloud.com as well)
lb02-us.wurflcloud.com = 50.57.182.149
lb01-uk.wurflcloud.com = 31.222.190.95
lb02-uk.wurflcloud.com = 31.222.191.152
lb01-za.wurflcloud.com = 41.76.214.41

Thanks,
Ashan

Elliotfehr

Re: Exception - HTTP Request timed out

Postby Elliotfehr » Wed Feb 05, 2014 11:41 am

Ashan,

I certainly do agree with Steve in regards to a few failed HTTP requests each day and I can also confirm the IP's listed as well. If you do not already receive ScientiaMobile announcements feel free to register you email address here and you will be notified of any IP changes.

Thank you,

Elliot

duoboots3000
Posts: 5
Joined: Fri Jan 31, 2014 7:35 am

Re: Exception - HTTP Request timed out

Postby duoboots3000 » Thu Feb 06, 2014 1:27 am

Hi Elliot,

Thanks for your confirmation and we have subscribed ourselves to the announcements.

Thanks,
Ashan


Who is online

Users browsing this forum: No registered users and 14 guests