In software testing, it is most important to measure the quality, cost and effectiveness of the project and the processes. Without measuring these, project can’t be completed successfully. The goal of testing is to determine if the requirements are met. During the course of testing, we find defects, or instances where the software does not meet requirements. Hence, in the area of software testing metrics, there has been abundant work in analyzing defects via different Defect metrics.
With respect to defects, there are many flow charts detailing the defect life-cycle. There are also numerous defect tracking and management systems that help us track defects at different phases of SDLC. Additionally there are many defect metrics listed out with definitions and calculations that help us provide greater insight into the effectiveness of the testing effort. Let’s have a look at some of them…
Total Defect Metrics
What if the Test team found 0 defects? Yeah! Everyone will be skeptical about the build quality. Agree or not nobody trusts developers when it comes to “Zero defects” 🙂 Therefore the first & foremost defect metrics to look at is – “Total defects”. The relative assessment of Total defects vs. Module size & complexity can tell you a lot of things about the effectiveness of testing efforts.
Note: For Module size & complexity, we hope everybody in the team is already aware of (or at least have a rough idea about) it before the start of Test execution.
Defects Distribution by Priority & Severity
Confused about Priority & Severity? We got it already covered – “Priority & Severity”!
Now that we know the total number of defects, what if all the defects that are identified are low priority & severity? And there is no functional defect that could bother the build quality? For example – low GUI defects. Yeah! Manager’s first thought would be –
“What! No functional defect? Only GUI defects? Either the test team don’t know the functionality OR they didn’t test the system properly. Because there is no such system (close to impossible) as Zero defect and everything is perfect. Need to discuss this with the test team, I guess they are relaxing and not bothered about the quality”.
Though total defects gives a good idea about the test efforts, but it doesn’t reflect on the ‘quality’ of testing or the build. A more appropriate metrics in this case would be “Defects Distribution by Priority & Severity”. Priority and Severity distribution serves as an input to both Business as well as Project team, which helps them gauge the actual “Quality” of the Test team’s efforts. Additionally, critical & high defects can be presented as a percentage,
(# Critical or High Defects / Total Defects) * 100
Defect Density Metric | Test Design Efficiency
Most of the testing terminologies can be understood by their straight-forward English meaning. Density as in Thickness or Concentration. You must have studied ‘concentration’ as part of the science syllabus during your school, i.e. the relative amounts of solute and solvent in a solution – Percent concentration is the volume of solute divided by the total volume of the solution, multiplied by 100. Remember?
On a similar line we can also identify the percent concentration of defects within a software build. How?
Defect Density = (No. of Defects identified / size) * 100
Here “Size” can be considered as the number of requirements or test cases. Hence the Defect Density is calculated as number of defects identified per requirement or test case. Example:
- Total test cases: 1200
- Passed: 900
- Failed: 300 (or Total defects = 300)
- Defect Density = (300 / 1200) * 100 = 25%
Now what does this mean? Simple, 25% of your test cases failed during execution OR 25% of your test design was able to catch relevant defects. What if you wrote 1000 test cases only to find 10 defects? Yeah! Your defect density will drop below the industry-wide benchmark to just 1%, i.e. your test design efforts were not worth it!
Defect Removal Efficiency (DRE) | Positive outlook
Why do we do Testing? Yeah! To identify defects and improve the Software quality by removing (fixing) them. One of the most important defect metrics, Defect removal efficiency is a measure of Test team’s competence to remove (identify) maximum defects before a software is moved to the subsequent stage. Here defects that matter are the ones caught by either the test team or by other users in the next phase.
Defect Removal Efficiency = #Defects found during testing / (#Defects found during testing + #Defects found during next phase) * 100
DRE is the percentage of defects that have been removed during an activity. Say the Test team identified 100 defects during system testing. After fixing, retest & regression the build is promoted for User acceptance tests. The UAT team identifies 25 more defects in the build which could have been identified during system testing. What does it mean? The system testing team could not find these 25 defects and they were NOT removed from the system. Hence,
Defect Removal Efficiency = [100 / (100 + 25)] * 100 = 80%
Means that the System testing team was able to remove (identify) only 80% of the defects, rest 20% were identified during the subsequent phase. As you might have guessed, DRE can be (or rather should be) computed for each SDLC phase and plotted on a bar graph to show the relative defect removal efficiencies for each phase.
Defect Leakage metrics | Negative outlook
Defect Leakage, as in what percentage of defects actually leaked from the current testing phase to the subsequent phase. As you might have guessed, Defect leakage (defects passed to next phase) is just the reverse of Defect removal (defects identified in current phase) efficiency. Defect leakage is another defect metrics but a deadly one 😉 i.e. defect leakage percentage should be minimal in order to prove test team’s worth!
Defect Leakage = #Defects found during next phase / (#Defects found during testing + #Defects found during next phase) * 100
Say the Test team identified 100 defects during system testing. After fixing, retest & regression the build is promoted for User acceptance tests. The UAT team identifies 25 more defects in the build which could have been identified during system testing. What does it mean? The system testing team could not find these 25 defects and they were LEAKED to the next phase. Hence,
Defect Leakage = [25 / (100 + 25)] * 100 = 20%
Means that 20% of the defects were leaked by the Testing team to the next phase, or in other words test team wasn’t able to identify these 20% of the defects. Similar to DRE, Defect Leakage can be (or rather should be) computed for each SDLC phase and plotted on a bar graph to show the relative defect leakage for each phase.
Cost of finding a Defect
Cost is an important factor for any project. Testing is no different. Client won’t be happy in case even after paying a huge amount, the testing team didn’t find defects. Or after the release bugs are encountered by end-users. How to measure test effectiveness with respect to the money spent? Yeah! The cost of finding a defect!
Cost of finding a defect = Total effort (money) spent on testing / total defects found
Note: Total effort can be calculated by considering total resources, the duration and the billing rate.
Like everything in this universe, defects too have a life-cycle – from birth (new) till death (closed). Defect age is a measure of its oldness, i.e. how much time has elapsed since it was identified and NOT closed.
Defect Age (in Time) = Current Date (or Closed date) – Defect detection date
Note: Defect age can be calculated in hours or days. It is a measure to gauge the responsiveness of the development/testing team. Lesser the age better the responsiveness.
Weighted Defect Density
We already discussed the Defects Distribution by Priority & Severity. Now how to find the defect density with respect to severity or priority? i.e. what was the average defect severity per test case (or requirement) created.
Weighted Defect Density = [(5*High + 3*Medium + 1*Low Defects) / Size] * 100
Here “Size” can be considered as the number of requirements or test cases. Weights 5, 3 and 1 are assigned based on the defect severity of High, Medium and Low.
We hope some of the defect metrics are now clear to you. Please let us know which all defect metric do you use (or have used) in your project, in the comments below…