Java API performance issue in standalone configuration

wurflaoltest
Posts: 5
Joined: Tue Dec 10, 2013 6:07 pm

Java API performance issue in standalone configuration

Postby wurflaoltest » Mon Dec 30, 2013 9:11 am

Hi,

We have implemented a standalone setup that processes approximately 50 million user agents in about 4 hours on a certain Unix host with Java 1.6.0.2 . The same code takes ages(literally) to run on all other hosts using same Java version as well as the latest Java version.Our aim right now is to use a Unix VM instead of an actual host but we are not sure the performance degradation is due to VM being used or due to a Java version specific dependency. We have already tried compiling the code using respective JREs installed on each host and that did not solve our problem.

Please let me know whether any specific Java version is recommended for 1.4.2 version of your API . Also , we are planning to upgrade to 1.5.0.1 and there also we are getting same issue. On existing Unix host performance is far greater than all other hosts . The CPU utilization on maxes out at 12-15% on all other hosts except the existing host on which we get about 90-95% utilization and great performance.Please give your insights so as to understand what might be causing this.

sm_support2
Posts: 294
Joined: Mon Jun 20, 2011 5:04 pm

Re: Java API performance issue in standalone configuration

Postby sm_support2 » Mon Dec 30, 2013 9:30 am

Hi there, I will pass this to our Engineering team and see what they have to say, albeit I am not sure that we can do something about the limitations that have been introduced for Java applications by whoever manages your virtual hosts.

BTW, in case we need it, can you provide access to one of the virtual hosts in questions so we can debug there?

Also, while probably premature in the context of this discussion, if performance is your requirement, you may want to consider our InFuze C++ based products (including API).

Thanks

fulvio.crivellaro
Posts: 38
Joined: Tue Jan 31, 2012 10:27 am

Re: Java API performance issue in standalone configuration

Postby fulvio.crivellaro » Mon Dec 30, 2013 10:15 am

Hi.
Actually, there should be no such a performance difference depending on the Java version.
Same can be said about WURFL library version.

Can I ask you some details about your hosts?
As I understood, the well-performing Unix host is a real machine, and you are planning to move the WURFL installation to a Unix VM.
What is not clear to me is whether the hosts issueing the performance problems (those with 15% CPU load) are virtual machines or not.
Moreover, are you using a custom cache configuration?

At a first glance, anyway, I suggest you to check RAM I/O operations, together CPU load.
WURFL makes a heavy use of caching, hence such a low CPU load may be caused by the fact that RAM operations (maybe paging) are being performed.
Of course, if you can exclude that the performance is actually limited by the VM configuration.

Thanks,
Fulvio

wurflaoltest
Posts: 5
Joined: Tue Dec 10, 2013 6:07 pm

Re: Java API performance issue in standalone configuration

Postby wurflaoltest » Mon Dec 30, 2013 11:04 am

Hi Fulvio,

The VM migration is the final motive. The under performing hosts are actual machines as well (that max out on 15% CPU usage) . It would be worth mentioning that we run other Java applications on the hosts in picture as well and those don't stop utilizing CPU(just talking in terms of numbers here) once 12-15 % CPU usage starts reflecting .
I am not clear about custom cache configuration and I think we are not using it.
We have tried running the API on ~Zero load as well . We have tried giving near real time priority to the WURFL API process (using renice commands in Unix).

To add , I'll let you know the OS ,Java version and system summary of two hosts . The WURFL API works well on first one . On the second one , not so much.

1)
Linux <hostname> 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:13:42 EST 2007 x86_64 x86_64 x86_64 GNU/Linux

java version "1.6.0_02"
Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode)

top - 10:59:58 up 153 days, 11:56, 1 user, load average: 0.00, 0.02, 0.11
Tasks: 128 total, 1 running, 127 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.1% us, 0.3% sy, 0.0% ni, 98.9% id, 0.7% wa, 0.0% hi, 0.0% si
Mem: 8158348k total, 6420948k used, 1737400k free, 117600k buffers
Swap: 4192880k total, 1076k used, 4191804k free, 4021588k cached


2)
Linux <hostname> 2.6.9-55.ELsmp #1 SMP Fri Apr 20 17:03:35 EDT 2007 i686 i686 i386 GNU/Linux

java version "1.6.0_35"
Java(TM) SE Runtime Environment (build 1.6.0_35-b10)
Java HotSpot(TM) Server VM (build 20.10-b01, mixed mode)

top - 11:00:17 up 1504 days, 3:58, 2 users, load average: 0.00, 0.00, 0.00
Tasks: 103 total, 1 running, 102 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 16633348k total, 11487328k used, 5146020k free, 153564k buffers
Swap: 4192880k total, 0k used, 4192880k free, 10926336k cached

So , resource wise the second host seems to be more powerful but when the code is triggered we see a different picture.

Thanks.

fulvio.crivellaro
Posts: 38
Joined: Tue Jan 31, 2012 10:27 am

Re: Java API performance issue in standalone configuration

Postby fulvio.crivellaro » Tue Dec 31, 2013 11:05 am

Hi.

We have run a test on the following machines (both virtual and real):
- Real Ubuntu 13.04 32-bit
- Virtual Debian 64-bit
- Virtual Fedora 17 64-bit
- Virtual CentOS 6.4 32-bit
We run a performance test over tenths of thousands UAs, in a single-threaded environment, running with JDK-6u45 (the latest 1.6 release), and using the latest WURFL core library 1.5, which I remember is compiled with JDK 1.5.
We have not faced any sensible performance difference: outcome was always between 3500 and 4000 UAs per second.

Can you please give us more details about the hardware/software you are running on?
For example, I can see that the well performing machine has a 64-bit OS, while the other has 32-bit one.
You have always talked about Unix, while looks like machines are running on Linux: which distribution are them, so we can support you running even mode specific tests?

Any further detail is welcome.
Looking for you answer.

Thanks,
Fulvio

support-all
Posts: 32
Joined: Mon Oct 08, 2012 2:47 am

Re: Java API performance issue in standalone configuration

Postby support-all » Thu Jan 02, 2014 6:11 am

Hello,

looking at the linux kernel version on your actual hosts I'd say that you are using RHEL 4.4 64 bit (linux kernel version 2.6.9-42) on the first one and RHEL 4.5 32 bit (linux kernel version 2.6.9-55) on the under performing one.

After some research I've discovered that 2.6.9-55 kernel version have many performance issues experienced by the users which were it using back in 2007, for example poor disk access performances, kernel panics and various hangs on bootstrap. Take a look at the following links:

https://sourceware.org/bugzilla/show_bug.cgi?id=4553
https://bugzilla.redhat.com/show_bug.cgi?id=239484

Moreover in the first link the bug has never been fixed..
I've also found an interesting discussion on the VMWave forums which is related to a poor disk access performance when using the 2.6.9-55 linux kernel:

https://communities.vmware.com/message/671657

As some users pointed out in those links I would recommend a kernel downgrade from 2.6.9-55 to 2.6.9-42 (previous stable kernel release).

We would also like to know if there are any other actual hosts in which you are running WURFL Java besides the two RHEL distributions.

Looking forward to your comments,
Andrea


Who is online

Users browsing this forum: No registered users and 16 guests