Defect Leakage is natural. It’s common. If you build a software, bugs will be there. Similarly even if you Test a software, you cannot prove their absence. But we can do our best to eliminate maximum of them. Typically in most of the projects even if you complete functional testing (system testing) without any time & cost constraints, still UAT team (Business) manages to catch some more bugs. Any idea why? There has to be something missing in the system test coverage. What is it?
“Defect Leakage are the defects that could not be identified in the current phase and subsequently escaped (were identified) to next phase.”
This was one of the interview questions I was asked when searching for a new job opportunity. Yeah! Organizations now-a-days are adopting a more practical questionnaire apart from some theoretical concepts. It’s to gauge your practical experience in the Software Testing technology & process. Coming to our topic, defect leakage don’t have a single reason. Everything we do in Software Testing is to minimize defects leakage to production environment. Client expects a flawless product for the money paid. Since there is no one reason for defects leakage, me alone cannot come up with all the shortcomings. So what did I do? I ran a social campaign to get people feedback on the very same query – Why are some defects leaked to UAT? Whoa! Got a superb response. Let’s get started…
For those who don’t know what UAT is, please refer to our earlier post “Software Testing Levels | 5 Stages to Deliver a Quality Software”
User Acceptance Testing (UAT)
To summarize: Whenever we purchase an electronic or electrical item, we always ask the seller to first give a demo so as to check if it’s working fine. In case of Software – it is not used by the client, instead by the end-users, i.e. client’s business reputation is at stake if it doesn’t work. In this case how do you think client would accept your tested system without a walk-through? As the name suggests – Business Users walk-through the system during the UAT phase to evaluate the system’s compliance with the business requirements and assess whether it is acceptable for delivery / release (mainly from user’s point of view)
Note: The goal of User Acceptance Testing is not destructive but to establish confidence in the system that it meets the end-user requirements and is fit for use. Ideally it should be done by a bunch of actual users or stakeholders from the client side but recently we have seen an increase in UAT outsourcing as well.
The Obvious | Business Usage
The most common social response was the expected one –
“System Testing, though should cover all the possible “user-scenarios” but still UAT team is closer to business hence more aware about the actual business use cases.”
UAT is usually driven by SMEs or business experts who will have in-depth knowledge of how the system works or should work. But that’s a negative. What you are telling is that ‘Our System Test team doesn’t have Business knowledge at par with the UAT team’. Despite the argument, it is obvious that not every System Test team is as good as UAT when it comes to understanding the Business logic. And what can we do for it? Yeah! Business sessions, Functional overviews, Knowledge transfers, Production scanning, Business Analyst collaboration, etc.
The Noticeable | Test Environment
How is your System Test environment configured? Does it replicate production? No?
Ideally, any Test environment should be a replica of actual production environment to catch all the environmental issues before going live. But that’s the ‘Ideal’ scenario. Let’s talk about the reality. Why do you think we have separate performance environment? In real, environments too are in hierarchy with respect to configurations,
The Development / Test environment >> The UAT environment >> Go-live, actual Production
UAT environment is closer to the actual production environment in configuration. System Test environment is more sort of a scaled-down version. Some of the defects leaked to UAT are not actually ‘leaked’ in this case, these are environmental defects which are way too tough to catch in the lower environments. Environmental issues identified in UAT could be configurations issues, browser-specific issues, Operating system or Device specific. And what can be done about it? Mirror the production environment in case your application shows a large no. of environmental defects. Or use a Cloud Test environment to keep budget in check.
The Hidden | Test Data
The Test Data. It is the most neglected yet important concept in Software Testing. Not many people like to talk about the Test data (or it is simply ignored).
In simplest terms: How do you test a text-box? Yeah! By entering & submitting different text. The permutations & combinations of this ‘text’ used for testing purpose is known as ‘Test Data’, i.e. data used for testing purpose. For testing a text-box we will first prepare the P&C of test data – numeric, alphabets, alphanumeric, max length, special characters, etc. It may be any kind of input to the application-under-test (User credentials, Environmental data, Setup data, Input / Output data, Uploaded files, Database tables, Transitional data, etc.)
Carefully prepared Test data is an important concept for successful Software Testing. Absent, obsolete or wrong test data might result in unreliable test results – a missed or an invalid defect. Here lies the hidden line – UAT team has access to masked production data, i.e. more realistic set of Test Data OR they are more business-aware to prepare realistic Test data. UAT team is more inclined towards testing the boundary conditions where they know systems break. And what can be done about it? System Test team can also request for a subset of masked production data in order to test the application with realistic test data sets. Utilize the different Test Design techniques like Equivalence partitioning and Boundary value analysis to increase the Test data coverage!
The Lame | It’s Natural
“Testing can never be complete. Every phase will have some or the other defect waiting to be caught.”
That’s a lame response. It’s like saying ‘I cannot help it. It’s natural’. Some Testers are lame, irresponsible, lazy, or just ignorant. I agree with the first part though ‘Testing can never be complete’ but it is no excuse for defect leakage. Instead it should be a trigger for the Test team to work hard & identify as many loopholes as possible – delivering the best quality product. No more discussions on this one!
The Complicated | Constraints
There is a limit to human thinking. Everyone thinks different and that’s what makes a successful team. A single tester will come up with say 50 test scenarios, other might come up with 25 different scenarios, and so on…As we start testing the initially developed build, it will obviously have more defects than what is delivered to UAT. And within the time frame, we try to catch maximum bugs – say improving the quality from 60% to 97%. That’s where the test team exhausts. The thinking stops. Whatever ‘thought’ possible has been tested. The build is deployed in UAT environment. UAT team (different humans) then starts from a relatively stable build (97%) & focus on identifying the remaining bugs with some different thought process. Does it make sense? Don’t know…
The Straight-forward | Requirements Glitch
Requirements, the foundation to any Software project. There could be nothing if you don’t know the requirements. And only knowing is not enough – getting a job is very different than getting a Manual Testing job with 4 Years work experience, right? Requirements have to be clear with no room for ambiguity. Many a times, same requirement is interpreted differently by different teams – leading to defects being leaked. Any Test team need to first understand the requirements clearly – talk with the Business analysts, clear your doubts with the stakeholders, demo a prototype, have a feedback loop, etc. Removal of requirement related defects is essential to eliminating the risk of exponential cost overruns and schedule delays. At the end of the day – both System & UAT teams should have the same understanding of each & every requirement!
The Altered | Real situations
UAT users have the time to actually ‘explore’ the system. They mimic some out-of-the-box real world situations that may arise in real-time. Functional specification is important, but think of it as a complete system and how it will be used in real-time. That way UAT is able to identify some areas that are not even documented in the requirements but are necessary for software to function properly. It may be a change request, but that’s a value add they offer.
The Side-effect | Regression impact
“But we tested it in System Testing. It was working fine that time. How come it popped up again in the UAT?”
Common remarks, right? During System testing, defects are identified and fixed until no more defects could be found. At the end, ‘some’ time is allocated for regression testing to verify the build as a whole. Most of the times not much importance is given to regression tests which leads to known-defects being leaked to UAT. Why? Cause that was the side-effect of other defects’ fixing. That’s why Regression testing is one of the important phases of any Software Testing project and it need its due share of importance. Regress the application seriously, include business scenarios, use masked-real-data, etc.
The Method | The Auto-magic
How are you utilizing automation in your current project? According to me, automation should not be used to replace Manual tests but to free up Manual testers of the monotonous tasks so that they can focus more on exploratory tests. Some projects don’t utilize the full potential of automation, leading to manual team being overloaded or else bored. Automation should complement Manual tests so that Testers have enough time to actually explore the application. Start exploring the automation tools like Selenium Webdriver or HPE UFT.
The Unknown | Communication Gaps
Team coordination & communication is the key to successful project delivery. Everyone know it well but still there are communication gaps between business, development & testing team. What we don’t understand is that it is not a “Test escape” but a “Team escape”. Quality is not only a department, it is also a discipline – it is everybody’s business. Sometimes small last-minute application changes are done without informing the Test team, or some requirement is not communicated, bugs are not reported diligently, etc. Unless your communication mechanism is defect-free, your application can’t be!
The Logical | No exploration
In general we have seen that very few Test teams keep a buffer for exploratory tests. Without a doubt, it is one of the most effective technique to identify bugs. End-user won’t have a Test script to read & execute, he/she will just explore and use the system. Move from checking to Testing. There should also be non-scripted exploratory testing which will help testers to think outside of the box and capture defects that wouldn’t traditionally be picked up by a standard scripted test phase. Exploration helps in catching the non-scripted bugs!
The Others | System Tests vs. UAT
There are some other factors as well which might have an impact on your defects metrics,
- The overall team’s domain knowledge & understanding
- Deployment issues in the software build
- Lack of proper walk-through & review process
Quality is everybody’s Business
Whatever you follow, defect leakage is possible. This is why Acceptance testing is becoming more and more important as a business function. Though the severity reduces with each test cycle. There shouldn’t be any critical/high bug identified in the UAT phase at least. We should have an acceptable benchmark defined for the leakage to every next phase. The same should be agreed upon by all the stakeholders. It can be anything between 0% to 5% but not more than 5% (it’s too large a percent for defects).
Don’t forget we are the quality gate-keepers for any software product, it’s our responsibility to deliver better than what client expected. The cost of fixing a defect increases proportionately with every next phase. Playing blame-games won’t help. Rather have a feedback loop in place wherein people brainstorm on the missed defects & the opportunities to improve the next time. Work as a Team, Think like a Customer! I hope this article will at least trigger someone’s thinking process and help to nurture ideas. Any other perspective or rationale you can add to this? Please comment…
Social discussions that ensued
Ramnath Calidas: Some of the issues found during UAT could be process oriented.
Yatin desai: Environment, configuration issue, Browser, O. S or Device specific issue.
Nensi Garasiya: Heck in a specifications, Misunderstand, Device/OS version updates, Diff. configuration, exactly can’t think like user, Device resolutions.
Jayesh Kapadi: Test team may missing the test cases which cover the corner cases.
Raja Yerra: They do more business oriented testing rather functional testing.
Sameer Pota: Agreed. They are masters in their businesses. So no one can do better UAT or review system as end-user/regular users can do.
Bhavin Shah: As a tester his primary duty is to verify test cases which are written against BRD/FS or other relevant documents. But as business user he thinks from real situations and those may arises in rare time.
Sadhvi Singh: After system testing whose main focus is to do functionality testing and not business oriented testing, we should target for regression testing after each big release. This regression testing should have business scenarios on which regression tests are based on. Doing so leads to removal of large number of bugs encountered during UAT. We in our team have tend to do so where business cases were developed and incorporated in regression scenarios.
Yashshvi Mittal: Possible due to multiple business agreement gaps say design gaps or due to improper System Test, browser compatibility, Regression- due to only manual efforts and no use of automation tools. No one is perfect, so we expect a bug leakage limited to 2-3% of overall system. This- (bug leakage) must need to be defined for every system initially and the same must be communicate to client.
Vivek Sharma: Corner cases not taken care of, Manual tester sometimes not execute all the test cases written, sometimes proper regression testing not done after bug fix. That is the reason automation testing is more important which executes all the test cases written.
Arun Jutur: Some of the Excuses are –
- Excuse 1: Bug is missed because this scenario is not in the test case. These test cases are reviewed and approved by Business Analyst/Product Manager/XXXX person.
- Excuse 2: Testing team already reported a similar bug which is not fixed yet. That’s why we got this new bug in UAT/Production. [Most common excuse].
- Excuse 3: The bug is occurring because of the last minute changes in the application by development team. Project management team should come up with a strategy so that we can avoid last minute changes.
- Excuse 4: Bug is missed because this scenario/rule is not mentioned in the requirement document [most common excuse].
- Excuse 5: Testing is done in hurry because enough time was not given to testing team. Project management team should be proactive and should make an effective plan.
- Excuse 6: Bug is missed because we (testers) did not test this functionality in later testing cycles. This functionality is not included in the testing scope.
Karuvarasan (Karu) Sundaravelu: Hope you are referring to waterfall and V model testing lifecycle. Even bugs are found in Live production test environment as well. As far as we have enough timeframe the bugs can be uprooted in all stages… but we have to agree that the severity of the bugs decreases as the build moves from SIT, UAT to OAT…. unless any code been regressed…
Amalan Xavier Arokiasamy: Try using W Model instead of V Model test approach
Arvind Pandey: It has more to do with mindsets of dev and QA engineers. Majority tend to deliver ONLY what they are assigned to complete within given constraints. Missing link here is they don’t get into customer/user thinking to realize how UAT found bugs can be eliminated at the beginning stage by pro-active thinking and delivering the best. Have had an opportunity to work with some exceptional dev and QA engineers who had a record of not having any UAT bugs for their deliverables with open challenge to find a bug with dev or QA escapes. They were no different than others but were developing/testing solutions as a users’ mindset enhanced with their experience.
Nicole Heisler: Arvind for the win! It’s not easy to keep the “user” mindset when testing because we keep seeing the same thing and begin to anticipate what we expect to see… For me it’s staying mindful and present in the process that makes the difference…
Arun Mishra: In general I have often seen Ad-hoc testing/exploratory testing altogether missing from the to-do list of testing team. This should never be the case. Ad-hoc testing really helps to catch a few bugs which could have been left out in the usual SIT.
Many a time testers aren’t aware about the real time business scenarios and therefore tend to ignore certain test conditions with multiple data combinations. This usually gets caught during UAT which is being done by seasoned business users dealing with customers on a daily basis.
Laurentiu Vrabie: There are checks to do:
- Test coverage of the requirement have to be 100%
- Environment of UAT if it’s different(mirror of production/live) you can have a lot of surprises
- As a business tester in UAT you can do workarounds like they are used to do it but these are not covered because simply they were not documented
Gaurav Anand: Hi Kanchan, Test environment is also a crucial factor here. Probably UAT test environment may be different from that was used during system testing. Here need to cover all possible combinations. Apart from this lack of Gap analysis and flow of information from top to bottom may also be a cause here.
Josemaria Ramirez: This is why the test team (as all the teams involved in the process) has to be involved in the Gathering requirement analysis, for them to fully understand all the possible scenarios.
Vignesh Sathyamoorthy: Lack of Domain knowledge is the most important factor for the defect leakage. As a tester, I must know what all the possible user actions to identify defects are. Certain defects identified during SIT may not be critical to the business and not a valid scenario or they can live with that. Working closely with Product owner and BA is critical and agile helps us to do it.
Anuj Malaiya: A tester should have a thought process similar to that of a user. He/she must be clear with how user is going to test the application thus covering all possible cases. Of course, testing is a never ending process , and factors like flow of information and difference in environment should be considered.
Balaji Thiyagarajan: I accept your point the more volume of test data will not be done in system testing ,whereas UAT will have huge lump of real test data in order to reach the threshold of the product and scalability.(Real Standard of product should be tested in beta site which huge lump of Realistic Production dummy data )
Mohammad Ajmal Fazli: Very correct Arvind, Adding to what you have written… What I have observed working with different programmers at various levels during my career…
Most of the programmers follow a very straight forward approach while testing interface, which they have designed, they tend to put in similar kind of data as it is required to input in respective fields on interface, they don’t experiment with data or rather effects of interface accepting no data of wrong data, for multiple reasons.
They also have a habit of pretending that they have designed the interfaces using best practices, so no bug/ error will appear.
Some programmers have a habit of not testing interfaces by discounting their responsibility and leaving for QA team to do all testing…
Alok Garg: Because UAT tests the business processes rather the functionality and the approach of testing is also completely different. In case UAT unveils any functionality defects then it is a matter of concern for SIT else not.
Sanjeev Singh: UAT defects depend on the usage of end product. Rocket launcher, airplanes directly go in production…so no compromise. It also depends on the idea of quality a company has. At apple the same ratio of defects are not detected… in same company defects ratios at multiple phases differ for different client. If you are delivering something for GE or CITIBANK then high chance that the company will never get its next project.
James Brodie: This is why Acceptance testing is becoming more and more important as a business function. Functional testing is a script-driven activity that given the usual time constraints can often be a happy-path at best. UAT testing is where the fundamental question is asked, does this piece of software meet the needs of the business? UAT is usually driven by SME or business experts who will have in-depth knowledge of how the system works or should work. This is usually different from function testing, whereas a script is followed without necessarily understanding the system itself under test. This is why there should also be non-scripted exploratory testing which will help testers to think outside of the box and capture defects that wouldn’t traditionally be picked up by a standard scripted test phase. If you understand what the system is supposed to do, it follows you will understand better if there is an issue and hence that is why you get additional defects. However, if you are finding more defects in UAT this is systematic of an issue with the coverage of testing that has been done so far. In reality, you shouldn’t be uncovering critical defects as part of the UAT phase.
Brijesh Deb: A very interesting discussion here… I read through all the comments (well almost TBH) and I find that most people speak in a more waterfall-ish context. That is when you talk about different phases with one following the other and so on. One might argue that this is what we find in V model and is the way to go however, try shaking things up a bit and do the phases differently. But how? Well let’s start thinking Agile… Self-organizing cross functional teams testing alongside development and keeping business and the customer involved at all times. Possible? Yeah, of course. Defects are caught early, systems get tested faster and user scenarios get covered when customers and business are involved. Do we see defects slipping through? Well yes but only a few and at all times the customer is aware.
Hey, but what about test environment… A replica of production environment at all times though the test data may not be actual end-user data but something similar. So you minimize the risks too!
These were my two cents. Would love to hear what others think though.
Ramesh Babu Veerabadran: The reason being
- Validate the functional test coverage % against your requirement / user story.
- Tighten quality gates on every stage of test phases starting from Unit testing, ST, SIT [Entry and Exit Criteria].
- Try to use the PROD mock up data.
- Think outside of box to cover another ad-hoc scenarios.
- Plan for Couple of Regression testing Iterations.
- If any gaps during your test prep / requirement phase, try to close out to avoid more bugs.