Curiosity is an inherent trait of humans and so all of us, from the greatest to the simplest minds, explore. We explore from Windows to Mac, from beaches to mountains, from indoor games to outdoor games, from new mobile phones to new software, from Black box to white box testing; we also explore code to test its functionality. Exploratory Testing is a powerful and fun approach to testing. In some situations, it can be orders of magnitude more productive than scripted testing. There isn’t a tester who didn’t, at least unconsciously, perform exploratory testing at one time or another. Yet few of us study this approach, and it doesn’t get much respect in our field. It’s high time we stop the denial, and publicly recognize the exploratory approach for what it is: scientific thinking in real-time.
What is Exploratory Testing?
Cem Kaner, who coined the term in 1984, defines exploratory testing as,
A style of Software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.
As its name implies, Exploratory Testing is about exploring, finding out about the software, what it does, what it doesn’t do, what works and what doesn’t work. The plainest definition of exploratory testing is test design and test execution at the same time. This is the opposite of scripted testing (predefined test procedures, whether manual or automated). Exploratory tests, unlike scripted tests, are not defined in advance and carried out precisely according to plan. It’s a testing approach that allows you to apply your ability and skill as a tester in a powerful way.
Exploratory Tests depends heavily on a Tester’s understanding of the requirements and logical exploration of application functionality against that understanding.
Exploratory testing, is all about discovery, investigation and learning. It’s like Machine learning, in concept. It empathizes on learning and adaptability. While the software is being tested, the tester learns things that together with experience and creativity generates new good tests to run. It emphasizes on personal freedom and responsibility of the individual tester. Exploratory testing is done without any specific plans and schedules. Test cases are not created in advance but testers check system on the fly. They may note down ideas about what to test before test execution. The focus of exploratory testing is more on testing as a “thinking” activity.
Exploratory Testing is NOT Ad-hoc Testing
Hearing of the ‘exploratory’ mode of testing, the first thing that comes to mind is ‘ad-hoc’ testing. But exploratory testing is a lot more than wandering aimlessly through an application looking for bugs; it is a ‘thoughtful approach’ which involves simultaneous learning, test design, and test execution. Ad hoc testing normally refers to a process of improvised, impromptu bug searching. By definition, anyone can do ad hoc testing. Whereas exploratory testing refers to a sophisticated, thoughtful approach to ad hoc testing. Even a free-form exploratory test session will involve tacit constraints or mandates about what parts of the product to test, or what strategies to use. A good exploratory tester will write down test ideas and use them in later test cycles. Such notes sometimes look a lot like test scripts, even if they aren’t.
According to Cem Kaner & James Marcus Bach, exploratory testing is more a mindset or “…a way of thinking about testing” than a methodology. They also say that it crosses a continuum from slightly exploratory (slightly ambiguous or vaguely scripted testing) to highly exploratory (freestyle exploratory testing).
The planning involves the creation of a Test charter, a short declaration of the scope of a short (1 to 2 hour) time-boxed test effort, the objectives and possible approaches to be used. The Test design and Test execution activities are performed in parallel typically without formally documenting the test conditions, test cases or test scripts.
Exploratory Testing vs. Scripted Testing
To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory testing. When performing exploratory testing, the tester configures, operates, observes, and evaluates the product and its behavior, critically investigating the result, and reporting information that seems likely to be a bug (which threatens the value of the product to some person) or an issue (which threatens the quality of the testing effort). It’s not exactly a testing ‘free-for-all’ but it allows the tester to go where an end user might go rather than where a script tells them to go. If we are running scripted tests, and new information comes to light that suggests a better test strategy, we may switch to an exploratory mode.
In scripted testing, test cases are designed in advance. The script includes test cases and test steps that are documented. There can be no deviation from the path laid out in the script. The tester’s job in a scripted environment is to color within the lines only, follow each instruction to the nth degree and report on the findings. We take a more scripted approach when there is little uncertainty about how we want to test, new tests are relatively unimportant, the need for efficiency and reliability in executing those tests is worth the effort of scripting, and when we are prepared to pay the cost of documenting and maintaining tests.
Why Do Exploratory Testing?
Imagine for a moment that you are a police crime scene investigator who has been assigned a murder to investigate. But for this investigation you are specifically restricted to a predefined set of scripted actions you can take to get to the bottom of the crime. You cannot deviate from this script and thus are severely hamstrung in solving the murder unless the outcomes of the scripted actions and questions lineup.
Scripted testing can simply lead a tester to a desired result. It’s like a classroom teacher who teaches to the test. They are not so much concerned about what is learned by the student as much as they are concerned that the student produces a desired result. This type of approach can severely limit learning in a testing environment. Many potential problems that real-world users of a program might encounter may never fall into the scope of what the tester is told to test. The results of this testing may be a success in the areas that the tester was directed to explore but could leave many stones unturned. The result can be a buggy release that makes the end user scratch their head in disappointment.
Exploratory testing, on the other hand, gives software designers and developers the chance to see how their product may be used ‘in the wild’. While not without its restraints, exploratory testing can touch upon areas never considered by those who birthed the program. Like anything else in life, if you are too close to something you often miss even the most obvious issues and concerns. Being too close or familiar with a project lends to missing the details. Exploratory testing can help project owners see their work through a fresh set of lenses and give them insights that their own scripted tests would have missed completely.
Exploratory testing is especially useful in complex testing situations, when little is known about the product, or as part of preparing a set of scripted tests.
The scripted approach to testing attempts to mechanize the test process by taking test ideas out of a test designer’s head and putting them on paper. There’s a lot of value in that way of testing. But exploratory testers take the view that writing down test scripts and following them tends to disrupt the intellectual processes that make testers able to find important problems quickly. The more we can make testing intellectually rich and fluid, the more likely we will hit upon the right tests at the right time. That’s where the power of exploratory testing comes in: the richness of this process is only limited by the breadth and depth of our imagination and our emerging insights into the nature of the product under test. Free from the encumbrance of pre-documentation, testers immediately incorporate new ideas into their tests.
We do have Tools & Techniques for Exploratory Testing
As opposed to common belief, exploratory testing is a disciplined investigative process with a well-defined role and guidelines that lead to an efficient and effective testing approach. In a nutshell, it is a simultaneous combination of learning, designing tests and executing those tests. The emphasis is on quick and useful results versus the development of formal test cases, test scripts and so on.
Test charters/outlines are more useful than formal test cases when performing exploratory tests. They provide more degrees of freedom and less overhead. These may be guided by functional areas and developed from heuristics. Exploratory testers often use tools, including screen capture or video tools as a record of the exploratory session, or tools to quickly help generate situations of interest, e.g. James Bach’s Perlclip. Another Exploratory testing tool is “Session Tester” that can be used as for managing and recording Session-Based Testing. Some other tools include Test Studio, QTest Explorer and Microsoft Test Manager.
Benefits of Exploratory Testing
We will not beat around the bush. One of the major benefit of exploratory testing is ‘freedom’. Software Testing is NOT just limited to scripted test cases – Testers have all the freedom to ‘explore’ the application-under-test and report any defects or enhancements using their acquired skills.
- Exploratory testing can sometimes catch really good defects which go unseen during scripted testing.
- Replicated experiment has shown that while scripted and exploratory testing result in similar defect detection effectiveness (the total number of defects found) exploratory results in higher efficiency (the number of defects per time unit) as no effort is spent on pre-designing the test cases.
- For Sprints to work well, it is critical that developers receive instant feedback on their work. Exploratory testing can provide a rapid feedback about the application-under-test since it doesn’t require testers to spend time on Test planning and design. This allows testers to quickly dive deep into a feature and assess its quality, providing the team with rapid feedback on how to move the feature forward.
- It results is an evolving picture of the software that better mirrors a user’s experience versus that of those who created it.
- It can serve to complement other more formal testing, helping to establish greater confidence in the software.
- Exploratory testing tends to be more intellectually stimulating than execution of scripted tests. Generation of new ideas during test execution which encourages creativity and intuition.
- In addition to being used to discover specific bugs, exploratory testing is also understood as a way to learn about the application and design functional and regression test cases to be executed in the future.
Challenges in Exploratory Testing
While exploratory testing is an excellent way of uncovering previously unknown bugs and issues, there are some drawbacks to relying on exploratory testing too heavily.
If it can’t be measured, it can’t be managed.
The major drawback that we see is ‘How do you measure the results of exploratory testing?’ Yeah! The defects count is the only measure of effectiveness of exploratory testing. Else a tester might go for a long coffee-break and at the end say that the application is working perfectly fine? Just to encounter a failure when live.
Other drawbacks include,
- Tests invented and performed on the fly can’t be reviewed in advance (and by that prevent errors in code and test cases), and that it can be difficult to show exactly which tests have been run.
- How do you report the progress on exploratory tests? To the stakeholders.
- Exploratory testing is not suitable for long execution time since it is difficult to track.
- It’s tough to track & managing test coverage.
Note: Some people say that it is challenging to replicate and assess bugs found in exploratory tests. We have NOT included this as a drawback since tools such as screen capture and video recording can be useful in documenting exploratory testing.
Exploratory testing has always been performed by skilled testers. In the early 1990s, ad hoc was too often synonymous with sloppy and careless work. As a result, a group of test methodologists began using the term “exploratory” seeking to emphasize the dominant thought process involved in unscripted testing, and to begin to develop the practice into a teachable discipline. This new terminology was first published by Cem Kaner in his book “Testing Computer Software” and expanded upon in “Lessons Learned in Software Testing”. Exploratory testing can be as disciplined as any other intellectual activity.
Exploratory Testing in an agile team
Agile teams tend to shift and redraw the role boundaries between “developer” and “tester,” mostly as a result of the heavy use by developers of automated unit and functional tests – in one sense an extreme form of scripted testing. These activities are not sufficient to ensure quality, and Agile teams can find valuable assets in team members who have well-developed testing skills. These skills will be deployed more effectively in the exploratory style on an agile team, as this style is more consistent with an agile approach than the “scripted testing” style.
In the world of software testing, exploratory testing offers a place where structure and process give way to intuition and gut feel, at least to some degree. It is the “natural” way to test, we as software testers naturally do this to an extent already, even if we are following scripts. It requires critical thinking. Testers should be able to reproduce, review and express their ideas in a logical way. They must be able to think like the application’s users and anticipate how they will behave. The use of knowledge about the domain, the system under test, and customers is an important factor explaining the effectiveness of exploratory testing. Exploratory testing triggers the thought process and helps to just explore and test the application without following any rules.
According to Cem Kaner,
Most testers spend 25% of their time doing exploratory testing, with 25% creating and documenting new test cases and 50% doing regression testing (i.e. running old tests).
Exploratory testing is a critical step in determining the overall quality of an application because it serves to surface unexpected or unknown bugs. The quality of the testing is dependent on the tester’s skill of inventing test cases and finding defects. As the tester starts exploring the system, new test design ideas are thought of on the fly and executed against the software under test. The more the tester knows about the product and different test methods, the better the testing will be.
Exploratory testing is one of many effective techniques in the pursuit of software quality. It is sometimes used in critical domains and places high demands on the person performing the testing. It is frequently misunderstood as an unguided, random process, which it is not. The proper application of exploratory testing significantly enhances the effectiveness of the entire testing effort. The key to an effective use of exploratory testing is to bear in mind that it is an adaptive process rather than a predictive one. A balance must be struck between the freedom to explore and formal tracking methods aimed at pinning down metrics such as defect rates and code coverage. We would see a mixture of both scripted and exploratory testing applied to a piece of software. This would give the best of both worlds and allow the design team to see as many angles as possible, thus making the release as bulletproof as possible.
- What is Exploratory Testing? – James Bach – Satisfice, Inc.
- A Tutorial in Exploratory Testing – Cem Kaner, JD, Ph.D.
- Exploratory Testing « Developsense Blog
What are your thoughts on exploratory testing and scripted testing? Do you prefer one to the other? Let us know in a comment.