Test Automation framework plays a vital role in the success or failure of any software automation project. For testers – it forms the foundation to a successful career. Software Testing involves multiple components like application-under-test, test cases, test data, test environment, etc. and a good automation framework needs to take care of all of these. In our pursuit to explore the topic of automation frameworks, let’s get started with collating different aspects of software testing that are to be taken care when designing a Test Automation framework.
Application under Test
The first and foremost. The reason for everything. The application-under-test. It can be a web-based application OR a desktop application OR requires extensive API testing OR database migration OR a Mobile app. The list can go on. When you are dealing with functional testing, ideally the automation framework should be independent of the application-under-test. I.e. it should be scalable with increase in functional flows or the Test data.
A test case is a set of conditions or variables under which a tester will determine whether a system under test satisfies requirements or works correctly.
During the Test Design phase, we write Test cases keeping in mind the requirements. Usually these test cases act as inputs to the automation team. Though I am against this practice. This handicaps the automation team to better understand application functionality – it’s like spoon feeding. Ideally automation testers should come up with their own test cases which are then automated into test scripts. Coming back to automation framework, we need to decide where these automated test scripts will be stored and how to organize test script writing. Also we need to think about how to organize test scripts into test suites for modularity & logical grouping.
What’s the basic process of writing automation scripts? Yeah! Identify the application elements (or objects) to be acted upon. Just like common functions to be re-used in different scripts, there can be multiple application elements (like username text-box, submit button, etc.) which are to be acted upon in different scripts. So what do you do? Build an Object repository, i.e. a warehouse of all the application elements.
We have iterated it multiple times – Carefully prepared Test data is an important concept for successful Software Testing. Absent, obsolete or wrong test data might result in unreliable test results – a missed or an invalid defect. How do you use Test data in your automation scripts? Taking a step forward, how do you maintain Test data in your automation framework? Excel files? XML files? Or a database?
Test Environment is nothing but a replica of actual production environment (being used by end-users) with close-enough hardware and software configurations, where the testing would happen for the developed application.
Test environment is where all the action happens – experiments, defect identification, fixing, retesting, regression and final sign-off. Once the testing is completed in this environment >> Test team will sign-off the application for actual production deployment. There can be multiple test environments like system test, performance test, UAT environment, etc. How do you manage Test environment configurations in automation framework? Each having its own URL and login credentials.
The most important aspect of any software testing project. A group of software testers will login to application and start executing the test cases. But it is different with automation. There is no manual intervention. Then how do you organize test scripts with respect to different functional modules? Where are the test scripts stored? And how do you plan script execution?
What if a Test script fails? What to do then? How are the errors handled? Will your framework automatically log defects in the tracking system? Or will you continue the scripts execution? Or what if there are technical errors in your script, what happens then? Framework will abort the execution until error is fixed or the execution will continue? It is important to handle the errors properly, otherwise what’s the use of designing a framework 😛
How do you track Test execution? Using a test management system? What if a script fails? How will you capture the screenshots? Where will the execution logs be saved? How to configure multiple runs? It’s like keeping record of your automated test execution in your automation diary!
‘Defects’ – a central concept to software testing. Anything to everything in software testing revolves around defects. So how can an automation framework leave this important aspect un-handled? 😛 What if a test script fails? How are defects logged in the management system? How are screenshots captured? Though confirmatory but still automation testing might uncover any regression defects and it is important to design test automation framework to handle such situations.
How will the Test manager, higher management and the client know about your Test automation efforts? Yeah! Reports. Will it be HTML reports? Or an excel report? Some frameworks offer HTML reports that are automatically e-mailed to all the stakeholders once test execution is completed. Others help prepare excel formats. Or it can also be simple HTML reporting. Or it can just list failed test scripts. How you design test reports is an important aspect of building a test automation framework.
Continuous Integration is a development practice that requires automation testers to integrate code into a shared repository at regular intervals. This concept was meant to remove the problem of finding later occurrence of issues in the build life-cycle. Generally a tool like Jenkins is utilized for continuous integration – an open source automation server written in Java that allows continuous integration. It is a server-based system that runs in servlet containers such as Apache Tomcat. Builds can be triggered by various means – commit in a version control system, scheduling via a cron-like mechanism or requesting a specific build URL.
Cross-browser (or parallel) execution
The real benefit of Test automation is realized when execution is done in parallel on multiple machines, browsers or operating systems. But how do you achieve it? Running multiple instances of same test scripts. How do you configure automation framework to handle parallel test execution?
Does your test automation framework fit in the methodology followed by the project? It can be waterfall, V-model or the now-popular Agile. It can be short sprints of test automation or you might get plenty of time for Test design & then execution. Agile offers short sprints for test design & execution so your automation framework must be compatible with the timelines. I.e. quick & easy for test design & execution.
Different tools are required for management of Software testing life-cycle. Some are required for test design, others for defects management. Some use exclusive agile management tools as well. Then we have automation tools like Selenium and HPE UFT. Build automation tools like Maven and Ant. Continuous Integration tools like Jenkins. An effective Test automation framework is designed keeping in mind the usage of all different Test tools & leveraging benefits of each as & when possible.
Test Automation Framework Design
Test automation has become an integral component of Software testing life cycle. The creation and maintenance of a test automation framework are key to the success of any test automation project within an organization. A framework is a real or conceptual structure created to provide support or guidance to an entity that could expand in future. This article attempted to list some of the key parameters that a software tester needs to keep in mind, while developing a test automation framework. By keeping above pointers in check, software testers and software testing companies can immensely benefit by designing an ‘overall’ Test Automation frameworks. Think something is missing? Do mention it in the comments section…