Is this real performance testing?
Especially some load testers raise this question. Their framework of reference, limited to only mainstream load & stress testing, does not leave them enough room to imagine any other technique than pure Load & Stress Testing. They find it hard to understand that there are other ways such as Model Based Performance Testing and that pure Load & Stress Testing is not that suitable in agile settings.
Here is some explanation:
1. What is testing?
In essence testing is a process like this:
There is a product
The product has several quality aspects including performance (time behavior).
You want to assess one or more of such quality aspects.
The quality aspect has metrics, e.g. response time.
The consumers of the product have set requirements such as a norm or standard for response time.
Testing is observing the metric of the product while it is used, comparing it against the norm and deciding if the metric meets the norm or not.
What does mBrace? It takes very accurate measurements for each transaction of the application under test and creates a profile for each and every one of them. The transaction profiles give a basic idea of how the transaction performs in the test environment. However we are not interested in the performance of the transactions in the test environment but in the production environment. With help of the mBrace model we recalculate response times to make predictive projections of the performance in the production environment.
This way we calculate response times for all relevant circumstances such as changing infrastructure, load and capacities.
The response times are compared against the norm to enable the OK / Not OK decision.
So pure testing comes down to deciding OK or Not OK for the performance of the transactions under test.
Handing over a list of transactions that fail the test and wish them good luck is not what our customers are waiting for. We have to help them to solve the problems in the transactions that failed the test. mBrace provides extensive intelligence about the application under test that helps customers to solve these problems.
2. Performance testing is a means to an end, not a goal.
What is that goal? The goal is to prevent damage, which can be considerable, caused by insufficient performance. The mBrace Methodology is a three stage approach that both supplements and replaces load & stress testing in whole or part.
The first two stages are entirely model based and sufficient in many cases. The third stage is based on load and stress testing. As part of the mBrace Methodology Stage 3 is not performance evaluation but functional testing. Why do we do it? Exclusively to find and eliminate defects and not performance evaluation because we do that in the first two stages. We do it because it is a necessary step to prevent performance problems caused by functional defects and thus necessary to provide enough risk reduction. The insights gained from the first two model based stages enable us to restrict the work to be done in Stage 3. Governed by Product Risk Assessments this provides ample choice between quick + very low cost and extensive testing at high cost. Based on a judgment of the risks it is possible to skip performance testing in the first place, only conduct stage 1 or stage 1 + 2, or stage 1 + 2 +3.
It should be up to the customer to pick his choice based on well communicated insights. The mBrace Methodology provides a performance testing capability giving the customer more choice than any other approach.
The three stage approach has proven in numerous projects at Fortune 500 companies that it is completely water tight for mitigating the risk of poor performance.
Voila