Does Jmeter give inconsistent results?

I am always interested in understanding the limitations of Jmeter so that I can make sure I avoid them and don’t use Jmeter inappropriately. I was there for interested in the following blog post by Michael Czeiszperger (http://www.webperformance.com/load-testing/blog/2013/05/measuring-web-page-load-times-using-jmeter/). Admittedly Michael is the driving force behind WebPerformance a company that has its own performance test tool. I thought Michael tried to provide a reasonable balanced.

What did concern me was the results from his test that showed that Jmeter provided inconsistent results for what sounded like a simple test.

It is always difficult to try to replicate results from a high-level blog. However, it seemed that the problem occurred at 50 threads running a 3 page test with a 1 second delay between each thread. I ran the test within the data centre from a VM against the target application. I only tested 3 static pages and did not download the embedded static content (image, javascript).

The test run from a 2 vCPU VM with 4G of memory, the test ramped up to 50 threads. I cleared the statistics (ctrl-E) and then ran the test for 10 mins before noting the results. This was repeated three times and the results are shown below:

jmeter consistant results

As you can see a more consistent set of Jmeter results. Admittedly the application is not as complex as the application tested by Michael but in away the tool should be independent of the application.

In a follow up message from Michael he said that the problem was noticable on the page that downloaded embedded static content. I enabled for one of the pages to “Retreive All Embedded Resources from HTML File” and rerun the test. Here are the results.

jmeter consistant results2

Advertisements

One thought on “Does Jmeter give inconsistent results?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s