GoogleBot detection (Updated) - Cloud API Results incorrect!

[datapharm]

Hi,

I am getting conflicting results when call the CloudClientManager.GetDeviceInfo() method when specifying GoogleBot UserAgents. The code (.Net) we are using is as follows:

var config = new DefaultCloudClientConfig { ApiKey = ApiKey };
var mgr = new CloudClientManager(config);
var deviceInfo = mgr.GetDeviceInfo(new HttpContextWrapper(HttpContext.Current));

This code is contained within a simple ASPX page from which we get HttpContext.Current.

I am using FireFox with the UserAgentSwitcher plugin to test different agents. I have disabled cookies as per the instructions on your FAQ.

I am testing against the GoogleBot user agents below, taken from http://support.google.com/webmasters/bi ... er=1061943

• Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
• Googlebot/2.1 (+http://www.google.com/bot.html)

In both cases the code above returns the following information (from our capabilities):

• Id: lg_un200_ver1_subuanovarravision
• Capability (mobile_browser): Novarra nWeb
• Capability (is_tablet): False
• Capability (device_os):
• Capability (is_wireless_device): True
• Capability (model_name): UN200

It seems to be that the wrong useragent is being detected.

I have tested the same user agents using your tools.scientiamobile.com page and they do return the correct information.

Please can you advise what the problem is as soon as possible as this is causing us serious issues as our site is no longer being indexed by Google, as we automatically direct mobile or wireless devices to a non-indexed mobile version of our site.

Updated
Looking at the DefaultCloudClientConfig settings that appear to be hard coded(?) the API is pointing to staging.wurflcloud.com which might not be helping at all.This is based on the assumption that staging refers to a non-live environment; if this is not the case then what else can this be? There is NO information in any documentation I have seen that indicates that this is the case, not how to correct this.

Thanks,

Simon Masterton
smasterton@medicines.org.uk

On behalf of Alan Henderson

[datapharm]

I have confirmed that there are issues with your api. The user agent for GoogleBot,

• Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Is incorrectly identified by:

• The ScientiaMobile Cloud API
• The dbapi site @ http://dbapi.scientiamobile.com/explore ... User+Agent
• The tera-wurfl site @ http://www.tera-wurfl.com/explore/index ... User+Agent

In all cases, the response seems to think that the user agent is

WURFL ID: stupid _novarra_proxy
WURFL User Agent: (compare to your user agent)
Mozilla/5.0 (X11; U; Linux i686; xx-xx; xx-xx:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 MG (Novarra-Vision/6.1)

Something is clearly wrong with your database. This is causing us major problems as google is unable to index our site due to GoogleBot being misidentified. We need a quick resolution to this or we will have to explore alternatives to your service.

Please contact me as soon as you can as the lack of feedback is not helpful.

Simon Masterton
smasterton@medicines.org.uk

[kamermans]

Sorry for the inconvenience. We are testing an update to the cloud service in staging now, and will be rolling it out over the next hour to all servers.

With regards to "staging.wurflcloud.com", this is indeed a bug in the .NET API and will be fixed ASAP, however, it should not cause problems since it is currently an alias to the production WURFL Cloud environment.

[kamermans]

Hi Simon,

All the WURFL Cloud Servers have been updated with the latest WURFL data, which correctly identifies the GoogleBot UAs. If you are still having issues, please post back and/or PM me or email me and I will work with you more directly (via Skype/phone/chat/etc) to make sure this issue is resolved as quickly as possible.

[datapharm]

I would hardly class this as an inconvenience considering we have lost 40% of our traffic over the last two weeks due to this issue. We cannot afford to manage our traffic using a service that does not appear to have been tested properly.

I am very concerned that the API was returning the wrong detection information for important user agents such as the Google and Yahoo search crawlers, which I would expect to be correctly identified. It is not as if the issue was with the detection of some obscure mobile handset here after all.

[kamermans]

Simon,

I can understand your frustration, and I apologize for the inaccuracy of the detection results and the impact it has had in your case. I can also tell you that we have a very strict testing protocol that we use to verify that cases like this (where the Googlebot was coincidentally matching a mobile device) do not regress back into production. We have not had issues with Googlebot in the past, so we have now added it and all of its permutations to our system, so, if a device is added in the future that would cause Googlebot to be incorrectly identified, our system will not allow the WURFL to be released, or the Cloud Service to be automatically updated until the situation is resolved. You are always welcome to contact me directly and I will discuss this with you personally.