The QA team starts testing a software/product and there are “way too many” defects. Every other scenario is failing, new flows are explored & clarifications sought. What would be the strategy now?
It’s a complex application with tight schedule. Too many defects add cherry on the cake. The blame game starts. But at the end, development & QA are expected to work as a team & deliver the product/software. What are the options?
- Reject the build. But that would mean delay in delivery.
- Get into a war room, daily. It helps to triage the defects & increase the seriousness.
- Work as a team & ensure priority flows are working as soon as possible.
- Identify super-devs & super-QAs to take up some additional tasks & ensure minimum turn-around.
- Involve the higher management (risk aware) just in case something goes wrong. It shouldn’t come as a surprise.
- Since timelines are fixed, make sure turnaround time is as minimum as possible – be it defect fix, retesting, business clarifications, anything. Everybody has to be on their toes.
Note: Retrospective w.r.t. estimations, build quality, management screw-up, etc. is altogether a different discussion.
Defects Management Strategy
Brian Lutz | Experienced mobile/web QA engineer
Ideally, long before the product reaches this point there should be a well-defined set of exit criteria agreed upon by the product owners, the developers and QA that define the bar that the product is expected to meet, and if the build is as bad as this scenario makes it out to be then there’s no way it’s meeting the bar. Unfortunately, there’s only so much the QA team can do (especially in a black box scenario) but they’re just the messengers in this scenario. If all of this is in writing beforehand it shifts a lot of the pressure away from QA.
Of course, QA needs to be doing their job too and making sure they prioritize correctly. Ideally the test suites should do this already, but the focus needs to be on meeting the exit criteria, and spending a bunch of time filling low severity bugs doesn’t help any here.
Kishore Sharma | ISTQB CSPO CSM SAFe Agilist
- Send an email to higher ups to let them aware of the situation with facts & figure.
- Do risk based testing and let the team collaborate without any fear of being pointed out for flaws. Let them share/ bear the responsibilities. If you really want to deliver the project else restructure now to avoid failure at later stage.
- Nothing much can be done for such situation unless people interact and collaborate.
Talib Mushtaq Rizvi | Senior Software Tester | Jmeter | Manual Testing | Postman | Agile/Scrum | Automation
First of all try to make test cases covering each and every joints that would help the application getting failed at the start up only and then on every release run those cases and fail the build if anything gets failed. This will save your work load again and again and make sure developers are properly guided with the UNIT TESTING. Which they did was not up to the mark and if required you can guide them.
Paul Yates | Quality Management, Testing and Process Professional – ISTQB CTFL
There are obviously multiple problems here but let’s address the pressing issue of the release. No QA professional ever said “lets release this software which doesn’t meet the client requirements and hope they don’t notice” If the major use cases aren’t working then NO ONE will be happy, regardless of whether you meet the delivery deadline. It’s far better to prioritize the bug fixing based on the clients priorities and get them something that meets their basic needs even if it means missing your deadline. Communication is also key, be honest about backlog but have a plan to remedy this in the future with maintenance releases. AFTER the initial release there are obviously issues that need addressing in the way QA interacts with development and the point in the development cycle at which bugs are identified and fixed.
Viji Menon | Delivery Excellence, Quality Professional, Lean Sigma, Agile Evangelist
In a complex application like this, why first of all testing coming too late in the cycle? Was there any convergence between testing and development team. This is where agility proves. Stop delivery. Fix the issues and streamline things. Get customer into confidence first.
Ashish Yadav, PMP® | IT Quality Consultant at UnitedHealth Group
There are different responses possible based on your role- for a PM:
- Maintain motivation of the team
- Secure critical path
- Communicate risk
- Prioritize functionalities
- Control scope
Subbarayudu Gopanapalli | PO| BA| |MS CRM, BPM, Pharma, Finance, Wealth Management| Available Immediately| KMP 1 | CSPO | SAFe Agilist |
I am sure in this kind of environment there might be a big backlog of defects, so team will be firefighting between defect backlog and product backlog. It means that the system is not stable. First Identify the list of all defects and take the list to client, here transparency is important discuss with them on stabilizing the system rather than focusing on delivery. Doing this we need not separately discuss on extending the timelines, they will be aligned with your plan. Without fixing the defects and focusing on delivery will make the situation worse.
If the situation is not as stated above. Defects raised are because of missed scenarios and requirements are not clear. Change the approach in requirement gathering .To start with, gather requirements collectively along with BA, QA and Dev (one or more from each team), document the gathered requirements and get them validated from client. Post validation inform client that requirements are freeze for the sprint and there will not be any changes in between. We need not be stubborn or strict to do this with client, if we follow a best practice transparently, clients understands and respects the discipline.
Prioritize. Work as a team & ensure priority flows are working as soon as possible.
Finger pointing is a frequent occurrence on failed projects. So it’s good to involve the higher management (risk aware) just in case something goes wrong.
Mustkim Sayyad | Senior Software Developer at TCS
As per agile methodology convert application functionalities into smaller business deliverable epics or features and get it prioritized from client. Get client into confidence and deliver product as per priority. Going into war room and pressurizing developers and testers may solve your problem for now but it may create a poor maintainable product for future.
Moinur Rahman | Software Quality Assurance Analyst at CompliaHealth (Formerly Procura)
First of all is there any signed and locked requirement document? If yes then first stop development and preparer test case based on the requirement document. Verify the test cases with the client. After that start testing case by case. If require divide the one functionality in multiple cases. Then based on the test cases pass or fail management can decide what else development is require or not. Also must use any testing bug tracking software.
Rajagopalan VenkataKrishnan | Test Manager (PMP, CSTE)
This is where we can apply Pareto. 80% of problems / defects / blockages in test is due to 20% of causes. Start by doing a practical review of all your observations and group/sort to find your top most issues. 2nd- address defects with high impact & retest in quick turnaround. 3rd As problems are starting to get addressed create a practical and reliable plan. 4th most important key point is to work collaboratively through all this instead of blame and shrugging responsibility / ownership. Collectively all together create the mess and all clean it together.
Anand Bhaskar, PMP®, MBA | Senior Project Manager at BMO Financial Group
Oh Yes, it brings the memories back. However, I do wonder if the first option is chosen (“Reject the build. But that would mean delay in delivery”) would there be any true delays. And by true delays, I mean the delays over and above the delays that you are going to face if you are in this situation. However if you do have to fix and go in QA (not recommended), data is your best friend. You will need a super team to go identify, prioritize, and resolve issue.
John Wilson | Senior Tester at PCMS
So, proposed solutions are;
- Reject the build. But that would mean delay in delivery. – Instead you can try to inspect quality into a known low quality build? – Please lord no!
- Get into a war room, daily. It helps to triage the defects & increase the seriousness. – Instead of discovering the root cause and fixing. Waste time reviewing and re-reviewing defects. Fixing the stable door after the horse has bolted.
- Work as a team & ensure priority flows are working as soon as possible. – Neglect the root cause and look busy doing something that gives an illusion of control.
- Identify super-devs & super-QAs to take up some additional tasks & ensure minimum turn-around. – Ensure your ‘stars’ are demotivated and want to leave.
- Involve the higher management (risk aware) just in case something goes wrong. It shouldn’t come as a surprise. – CYA
- Since timelines are fixed, make sure turnaround time is as minimum as possible – be it defect fix, retesting, business clarifications, anything. Everybody has to be on their toes. – Persecute the innocent.
I’m sure that the shrewd can determine the difference between the points of my comments and a leader that can’t listen and build on ideas, when a donkey is dead and needs to be dropped, and determining the actual problem to solve sounds so easy, but in reality is the trickiest bit.
- You can’t make a crappy release good, only less crappy, potentially.
- You can’t solve a problem unless you know what the actual problem is.
- You can waste time, a lot of time, patching symptoms of the problem.
- Everyone is responsible for quality, and should know they are.
- This is what you get when you test at the end.
So the only proposal I have is stop the release and do it again, but right this time. When done right it will be a lot quicker than doing it wrong and then trying to make it right.
It’s the stakeholder’s money and therefore their choice – waste money trying to make a silk purse out of a sow’s ear or spend money trying to build what is wanted. Seems obvious to me!
Adam Bardell | Test Consultant with nFocus Testing
Stop delivering new functionality and stabilize the product. Track bugs and regression test well. New functionality has to be prioritized in terms of impact once it does start being checked in again.
Priyanka Hasija | QA Engineer at GeoOp
I liked this approach. I also had this problem with one of my products I worked for and I followed the same strategy. Just test that whole sections and provide correct scenarios with acceptance criteria and it really worked.
Karen Lewis | Senior Technical Project Manager at BBC
Pair up testers and developers. You are all engineers working to achieve the same result.
Tanya Pandey | Test Analyst at AgilityWorks
I think you should log all the defects starting from critical one. Don’t give your testing sign off. Say them clearly we need all issues fixed. So it will not come to QA head at end. Go by process. If sign off process is not there list out issues and send to all leads /managers to just show your participation and concern that build is not stable.
Juliet Blake | Acting Delivery Manager at Open University
Identify the true cause of the issues so they can be addressed otherwise the problems will continue. What about Unit Testing? Is Unit Testing being undertaken and passing successfully? If yes, then you need to look at whether the tests are robust enough, If no, this might demonstrates the importance of ensuring all activities of the SDLC are executed correctly. On this occasion my suggestion would be to pair the developers and testers up, before code delivery (even of fixes), to identify if the correct scenarios have been considered by the developers and requirements have been understood correctly. In similar situation I introduced developer peer review of test scripts. The result was increased quality out of development by improved understanding of the requirements and unit testing. The added benefit was stronger working relationships with improved moral. It’s not a blame game it’s about coming together as ONE team.
Nishi kant gautam | Sr Team Lead – QA
This is the story of every fixed time project. The thing higher management have to create a mitigation plan. Now coming to the fact that this way testing defect rate will increase why because in hurry developer miss many thing like unit testing, code quality , business flow implementation gap. Sit with developer round the table along with product manager and start doing thing then and there.
- Try to cover and plan out testing activity within QA team and share it with Dev team to get timelines for each module
- Work with developer to raise, get fix and verify it to fasten the dev and bug fix cycle.
- Try to share one bug status report in morning and one in the evening including management people to showcase what is the fix rate and how much effort QA are putting to get these done with developers.
- Always ready with proper QA plan in-case of any kind of delay in delivery or any major gap.
- Always make sure that in these type of timelines low and medium bug are acceptable but no critical and blocker.
- Work as a team even if someone is not supporting QA because ultimate culprit is QA for any kind of quality issue.
Sanu Menon | Lead Engineering at Nggawe Nirman Technologies Pvt. Ltd
- Work as a team & ensure priority flows are working as soon as possible.
- Involve the higher management (risk aware) just in case something goes wrong. It shouldn’t come as a surprise.
Deepti Sinha | QA Lead at Compunnel Digital
I have tried all the points mentioned by you but the only thing that worked was — Prioritize. Work as a team & ensure priority flows are working as soon as possible. Rest everything didn’t work in favor. I believe the last thing to do in the project is blame game which unfortunately happens the most.
Bathin Vattivella | Senior Manager at RealPage, Inc.
It’s a genuine current phenomena in all the companies. We need to deliver the product as cohesive Engineering efforts. Try to convince and get confidence from Dev team to adopt left shift testing and follow bottom up approach in building test suites for Unit testing (70%), API testing (20%) and Functionalities UI testing (10%) which gives us better test coverage with low maintenance.
Rishi Mathur | Senior Quality Analyst at Punchh, Inc.
First step is to report this situation to management and then start prioritizing the higher risks areas and works as a team. By doing this management got aware early and cannot blame that they came to know when it is hard to control.
Bhavesh Nakum | Sr. QA Lead at Synoverge Technologies Pvt. Ltd.
I would suggest to take preventive actions along with corrective actions for fixing issues. Preventive action could be get proper Unit testing practices in place and let developers also own quality for their own part. QA should not end up doing unit testing (meaning just verifying basic functions and CRUD operations). QA must get good Quality builds so their bandwidth is utilized in effective testing and verifying real cases. All the Best!
Arun Motoori | Founder at QAFox.com
Good suggestions. For the QA to be on safe side, preparing a quality report having list of reproducing defects and also the QA recommendations for release is vital. This report can be shared with all the required stakeholders a day or two before release.
Sahil Suri | Senior QA Specialist
Focus should be in fixing crucial bugs first and asking the test team to again run Build verification tests and Adhoc tests. Once it is stable then focus should be on regression tests, etc.
Brijesh Deb | Agile Testing Evangelist – Helping Teams Test Better!
My questions are,
- What was the strategy for identifying the test cases especially Regression?
- Where any of the tests prioritized up front? If yes how?
- What was done to identify the risks involved in project and the product? What were the parties involved in risk identification? Was only the “risk aware higher management” involved or were others like product owners, developers and most importantly testers involved too?
Here are my 2 cents…
Instead of thinking about a strategy after being in this situation, Why not think about a strategy up front? Do a thorough risk analysis and use Risk Based Testing as the primary test strategy and build on that.
Anmol Bagga | Lead QE/ SDET at QA InfoTech
Based on my experience in Expedia and now in Walmart, there are small teams and QE (note I am not using QA here) can fix or help in fixing issues. The emphasis is not logging defects but reporting it effectively and getting it resolved as soon as possible based on priority.
Meeta Mukund Dayama | QA Tester at Secret Location| Freelance Software testing coach
I am a peace lover but frankly when such a chaos arise, before getting into a solution my mind battles for below questions Surely probing these questions will not handle the situation then and there but I can’t help it but think.
- Aren’t the developer testing before delivering the code to testers?
- Aren’t the testers doing smoke testing to before jumping right into the testing?
- What is the process and approach they have – “Big bang model”?
- Wasn’t there a strategy/plan defined?
- And was this not a foresighted risk? What did the Analyst then?
- And what did they learn in their previous retrospective?
We all know that 80% of problems in software arise due to 20% of bugs, I think same applies to the process and people. Honestly all of us when face a similar situation would definitely prioritize our tasks to control the scene of crime, but for the long run one needs to think on above questions as well and RETROSPECT them. I beg pardon if I am harsh and correct me I am wrong.
Jonty jain | Quality Analyst at Saitech IT Solutions
On this situation firstly we need to team work then ensure priority flow, and the most important thing if you want deliver the project to your client, Make sure he is not trouble with basic thing of your software, because your software must walk at least normal and good condition.
Atulya Krishna Mishra | Educator, ICAgile Authorized Instructor
Before answering this question, my question would be,
- Where were these folks who’re testing application now; when ‘someone’ was writing such a crampy code?
- Why these two ‘departments’ working in silos?
- Who ceased these departments not to collaborate right from the ‘very beginning’?
The current situation is more like a ‘Firefighting’ scene which is not a Normal Way of Life and it would never be. At the edge of ‘Chaos’ whatever seems good do it because in ‘chaotic’ domain the first thing you should do is to act. Things got crazy here, stuffs are off the rails. Your first ‘firefighting’ steps should be correct problem and contain the issue. Your initial steps may not be best but it’ll help you stop bleeding and somewhat be soothing. But don’t stop here find out the ‘Real Solution’ of problem.
Prabhu Mishra | Software Test engineer at Ericsson
What we do – There are there environments for filtering.
- ST- Development environment
- SIT- Test environment
- UAT environment
First point is timelines should be fixed generously for the User Stories.
- Before the deployment we need to identify the already present defects in Test environment, which will reduce the confusion whether it’s a regression or newly added defects and raise them.
- Ask the dev team to prioritize the regression defects and fix them before the new deployment
- Start testing in the dev environment (System Testing) after the dev team completes their own unit testing (2 days timeline depending on the complexity of the user story)- Try to find as many bugs as possible to filter out more
- After these defects are fixed deployment is given and 2 weeks of testing will be done in SIT environment along with regression- More filtering will be done
- UAT testing will be done and make sure no major defects may come out during this phase which is our responsibility
Rohit Handa | Sr Software Test Engineer at GlobalLogic
First focus on creating the defect report and send it to development leads and then basis the priority and severity first those bugs should be fixed which are directly impacting the business and prepare the build cycle for rest of the bugs for next release.