I am always interested in understanding the limitations of Jmeter so that I can make sure I avoid them and don’t use Jmeter inappropriately. I was there for interested in the following blog post by Michael Czeiszperger (http://www.webperformance.com/load-testing/blog/2013/05/measuring-web-page-load-times-using-jmeter/). Admittedly Michael is the driving force behind WebPerformance a company that has its own performance test tool. I thought Michael tried to provide a reasonable balanced.
What did concern me was the results from his test that showed that Jmeter provided inconsistent results for what sounded like a simple test.
The test run from a 2 vCPU VM with 4G of memory, the test ramped up to 50 threads. I cleared the statistics (ctrl-E) and then ran the test for 10 mins before noting the results. This was repeated three times and the results are shown below:
As you can see a more consistent set of Jmeter results. Admittedly the application is not as complex as the application tested by Michael but in away the tool should be independent of the application.
In a follow up message from Michael he said that the problem was noticable on the page that downloaded embedded static content. I enabled for one of the pages to “Retreive All Embedded Resources from HTML File” and rerun the test. Here are the results.