Recently I got a query on LinkedIn asking for some advice on how to test and find bugs in an application. A single line query but it encompass many different aspects of software testing. There is no one tip, trick or technique. There is no one process or method. There is no one tester or a bug hunter. There is no one approach or strategy. Software Testing aimed at finding bugs in an application is a vast discipline in itself. It take years of experience, just to learn new aspects and get better at it every year. This query got me thinking as to how & what should I reply to this gentlemen tester. Here is what I replied,
Thanks for reaching out. Happy to learn about your curiosity related to Software Testing & Defect detection. However, ‘How to Test & find bugs’ is a vast topic of study in itself. But will try to write a blog post for some bullet points. Will share the link as soon as it is completed. Hope that will help.
It took me a while to collate all my experience & do some research – and here I am with this blog post. No, this is not a complete list. And it can never be. As I said earlier, there is no silver bullet and it take years of experience, just to learn new aspects and get better at it every year. So you might have different perspectives, techniques, approach and strategy. Feel free to provide your valuable inputs in the comments section and I will try to incorporate every aspect by updating this particular blog post. Let’s brainstorm on this very important goal of Software Testing “How to Test and find Bugs in any Application”. Here is to your Bug Hunting success!
Find Bugs | A very good & important aspect
Software Testing started as a debugging activity and has now evolved into a multi-aspect discipline. Call it anything – Software testing, Verification and Validation, Quality Assurance, Development in Test – one of the primary goal (not the only goal though) still remains the same – ‘Find Bugs’. If you are a Software Tester or a QA Engineer, then you must be thinking every minute to find bugs in an application. And you should be. Finding a tricky bug is challenging and equally rewarding. It gives that satisfaction of your hard work.
Everything starts from the Requirements
“Necessity is the mother of all invention”
Everything in the software world starts from a ‘requirement’. Requirement to build a super-blog, or a career website, a CRM portal, banking application, healthcare monitoring, insurance tracking, almost anything. If you don’t understand the requirements clearly, you cannot test. As simple as that.
In order to test effectively, a tester need to clearly understand the requirements. What is it that end-user is expecting out of an application? What are the different functional flows? How different functional flows integrate to solve a purpose? As a tester, you need to understand the written requirements first. Mind it – not all requirements can be written. Why I say that? Because as a tester we need to understand the implicit requirements as well. Use your brains. Use some logical thinking. E.g. Requirement is to build a case assignment rule so that cases are assigned to users automatically. But to which users? What is the user profile or role hierarchy? Which type of cases need auto-assignment. When will a case be assigned? When it is submitted? Or when it is marked complete? How will the user be notified of the assignment? There are multiple questions/clarifications that can be sought based on each & every single requirement. Always think of the bigger picture and frame your clarifications accordingly. It is only when you understand the written & unwritten (implicit) requirements that you can proceed with the next steps.
Then comes the System Design
“Systems design is the process of defining the architecture, modules, interfaces, and data for a system to satisfy specified requirements.”
System design is as important to Testers as it is for an architect and a developer. What all modules constitute your system-in-test? How are these modules integrated? How do they communicate with each other? What is the data flow between modules? Which technology is being used for development? Any specific architecture model being followed? Like Service Oriented Architecture, or a Multi-tenant architecture? A client-server model or just a web-based application? What is the database schema?
Technical details about the system design helps testers to think ‘technically’. By default testers are domain-driven, i.e. business perspective. But it is equally important to understand how those business requirements are being implemented in order to test effectively. Unless you know the data model & data exchange, you cannot test the integration. Domain knowledge is important, but thinking technically gives you as a tester an additional perspective to test.
Verify the Happy flow first
User should not get an error when logging to the application, right? 🙂 Too embarrassing. Or when initiating a payment. Or raising a request. Approving some workflow. Every application has certain core functionalities or happy flows that should not break at any cost. First things first – as soon as the build is deployed, test for the happy functional flows. Yeah! That’s called smoke tests or sanity tests or extended sanity tests. Quickly verify the core application functionalities for which the build-under-test is developed. Though developers must have paid attention to the happy flows, but why take chances. Re-iterating the fact – Happy flows should always work flawless. And validating the same builds confidence to progress further and get your hands dirty.
Test Data is most important
“You follow the money to find the thief | I follow the data to find the vulnerabilities.”
Attack the system, looking to send it into a state of panic by filling in the wrong thing. If a field is required, leave it blank. If the user interface implies a workflow, try to take a different route. If the input field is clearly supposed to be a number, try typing a word, or try typing a number too large for the system to handle. If you must use numbers, figure out whether the system expects a whole number (an integer), and use a decimal-point number instead. The skill is relatively easy to learn, and once you’ve attained some mastery your quick-attack session will probably produce a few bugs.
Try Equivalence partitioning or Boundary Conditions to generate some effective test data. Equivalence partitioning helps in sampling of test data whereas Boundary testing is designed to catch the off-by-one errors, where a programmer uses “greater than” to compare two numbers, when he should have used “greater than or equal to.
Knack for Details
When hunting for bugs in an application, pay attention to each & every detail. The forms, UI, functional flows, integrations, back-end data, errors, everything. ‘Knack for Details’ is a tester’s much-wanted skill. Many-a-times people miss out on certain bugs which are out in open just waiting to be explored.
Alternate flows still have some defects | including Negative tests
Happy flows keep everyone happy, including developers. Written requirements are generally the happy flows, which form the basis of development. During development, much emphasis is on the happy flows. So they are bound to pass 90% of the time (unless your development team is reckless even in basic unit tests). So how do you find bugs?
‘Alternate flows’ are left for testers to explore. End-user is not a robot to follow a test case or a pre-defined written workflow. He/she can follow any flow – cancel the request in between, save & come later for completion, click on help, edit and fill additional info, wrong password by mistake, forgot password, payment failure, anything. Always try to cover as many alternate flows as possible.
Negative Testing too is an important aspect when trying to find bugs in an application. Negative testing ensures that your application can gracefully handle invalid input or unexpected user behavior. How would you feel if the page is not getting submitted & no error is displayed? Or a payment fails in the back-end without any error being displayed? Frustrated right? Disappointed. Exactly. Developers are not very good at handling negative flows where application should display a proper error message to the user. Test data preparation comes in handy here – blank value, boundary limits, invalid input, special characters, etc. As a tester always check for graceful error messages when inputting invalid data. Or when performing an invalid operation. The best would be to extract a list of expected error messages & then validate one by one >> and then moving on to some random (valid & invalid) inputs & verifying the results.
Old is Gold
Platforms often have the same bug (kind-of) coming up again and again. Mine your bug-tracking software to figure out what happens a lot, and then test for it. Many times failure has a common problem. Say for a mobile app, experimenting with losing coverage, or having too many applications open at the same time with a low-memory device. Over time programmers learn and then do not make the same mistakes, thereby improving the code quality before it gets to hands-on exploration. The heart of this method is to figure out what failures are common for the platform, the project, or the team; then try that test again on this build.
Exploration always helps
Explore! Do not always stick to the pre-written test cases. You are missing the real QA fun. Test cases are important and may be good at catching bugs. But the major problem, in my opinion, with test cases is that it is as difficult to attain near-100% test coverage using test cases, as it is impossible to attain a so called 100% bug free product. So along with the test case, also try to explore the functionality under test. I have found it an effective method to catch more bugs that lurk around the functionality but are not covered somehow by the test cases.
So today’s build works, and you deploy. That’s great! But tomorrow we’ll have a new build, with a different set of risks. We need to prove out the new functionality and also make sure that nothing broke—at least, to the best of our knowledge. We want to make sure that the software didn’t regress, with something failing today that worked yesterday. Best approach is to make a list of ‘impacted’ modules/functionality and re-verify (regress) the same in order to find bugs.
The intent to find bugs
Why do you test a software? To confirm it meets the requirements? To validate it is working as expected? To verify its functional behavior? Or the intent is to find bugs? Yes! The intent to find bugs is different to validating the application against some written requirements. The mindset is different. My job is not done if the application functional flows are working as per requirement. As a Bug hunter, my job is well done if I am able to extrapolate the requirements or uncover the hidden requirements and am able to find bugs in the area where developers are reckless. That’s what makes a difference. Else anyone can do testing by reading the requirements & validating the application. As a tester, bug hunting has to be one of the major driving force behind your skill set. The intent is to find bugs – to think beyond developers, as if you are the end user who is free to explore anything in the application – to input any data – to play around.
Testing is a skill. It can be learned, and it improves with practice.
Last but not the least, don’t be disappointed if you are not able to find bugs the first time. Or the second time. Software Testing is a skill which improves with practice. Experience plays an important role in identifying the loose ends of an application, focusing on error-prone areas, unconsciously discovering the alternate flows in search of bugs, or just re-verifying the already-working functionality.
Find Bugs | Building a better Application
Industry is sure moving to Automated Test execution. But as I say – Automation is confirmatory, Manual tests are more exploratory. Only Automation testing is not enough, so testers need to be good at finding bugs. “Finding Bugs” is one characteristic that differentiates a good tester from a mediocre tester. The basic principle is to combine things that programmers didn’t expect with common failure modes of your platform. Always remember,
Testers don’t break the software. It is already broken. You just need to find those broken pieces and help make the software better.
Hope this article helped to point out some small yet powerful methods that can help you in getting an edge over others as far as bug hunting is concerned. The focus was on generating test ideas. Go dive into the great game of testing. Have some fun. This set is by no means exhaustive. You might have some different techniques to find bugs. Do let me know your ideas and/or views through a comment.