A client tries to download an App “MyBlog” from the Google Play Store. Ooops! He gets an error: MyBlog could not be downloaded due to an error (403). Now let’s try downloading the App on WIFI without using the Carrier provided Network. Whoa! It works and the App is downloaded in a flash. But it would not work on the Carrier provided network. There it is, “A BUG”.
Why was this not tested? What’s going on? The QA team has a special focus on Products, how could such a thing happen? Aaah! QAs were accustomed to testing Apps only on a selection of devices, but carriers required more money for procurement. OMG! Mobile Testing is completely different compared to Desktop Testing. If we understand the distinction and challenges of testing Mobile Apps, it will be a bit easier to tackle.
Device & OS – Permutations & Combinations
Supported platforms and devices mean you have more combinations to test. Desktop apps were usually targeted for specific platforms. Web based applications made it a bit more challenging by adding: browsers. Mobile applications take complexity of supported platforms to the next level by adding devices.
Ensuring that mobile apps are working on all type of devices (smartphone, tablets, and phablets) supplied by major brands (various models from Samsung, Sony, Nokia, HTC, Apple, etc.) and on all the platforms (iOS, Android, Windows, BlackBerry, etc.) is challenging. On top of that, new devices are hitting market so often that it becomes impossible to cover all the major devices.
Design & Usability
How Apps adapt themselves for various screen sizes, layout, and configuration is a challenging task. Apart from adaptability to different screen sizes, mobile applications have to deal with the limited screen size. Limited screen size means that Usability poses different challenges and as a tester we need to think beyond what is developed and always think of who will use it and in what circumstances.
Complex User interaction
User interaction in desktop and browser based applications was pretty much limited to mouse and keyboard. Mobile applications on the other hand are trying to make user interaction as fluid as possible. We had touch screen and with new phones from Samsung, you can just wave your hand to give commands.
Siri is becoming more and more advanced and gives us a glimpse of future that voice commands may become part of every application in future. Devices are smart enough to understand complex gestures, eye movement, direction, tilt, movement, acceleration, GPS coordinates, surroundings, sound, and so on.
Application types – Web, Native or Hybrid
In the desktop and browser world, applications were straightforward. They were either desktop or web applications. However, with the adoption and support of HTML5, it is not difficult to find HTML5 applications, native applications and hybrid applications in a single mobile. Testing for hybrid applications would be different from testing native applications.
Emulators and Simulators
For the desktop and browsers, developers always had access to the platform or browsers they were targeting with their applications. Mobile devices on the other hand relies on emulator and simulators. However, they are still not true representation of the devices. As a tester, we have to be aware of the capabilities and limitations of these emulators and simulators and figure out what can be tested on them and what cannot.
Security and Privacy
Though most mobile applications live in their own sandbox but many platform features are accessible to them. For example, applications such as phone book, pictures, and videos are accessible to many other applications. These are all personal user data, and any defect around the (unintentional) misuse of this data can jeopardize trust of the application. In mobile world, it is important to ensure that applications are secure from the intruders, and it is equally important to ensure that applications are not intruding or accessing data unintentionally.
Networks and carriers
In desktop and web world, most users were either on LAN or Wireless. Many connected mobile applications rely on the network – how application responds to 3G, 4G, weak signals, no signals, powerful signals, switching from cellular to wireless and vice-versa or when user is moving at different speeds, etc. – can affect how the application will behave. It is often not possible to come up with or simulate real life situations for mobile applications. Apart from the variation in signal strength and type, mobile apps can respond differently to different carriers.
Mobile apps are installed, removed, or updated more frequently than desktop applications. Also, underlying OS and platform is updated more frequently as well. As an app developer and tester in the mobile world, you have to be on top of what changes are coming in the next revision of OS / Platform and how it might affect the application. Things like backward compatibility, simultaneous support for multiple versions, data preservation, restoring state and data, the ability to install / upgrade multiple times are all part of important checks for mobile application testing.
Session management and interruptions
Handling interruptions are the way of life for mobile applications. Apps and users are constantly interrupted by calls, SMS, push notifications, and so on. How applications handle these interruptions and how they maintain their state are important.
Mobile-specific Non-functional testing
Old school performance of the application is the obvious one, but there are many other factors as well which should be considered. How much data your application is consuming? How much it would cost the user (data usage) to use this application? How much battery is consumed by applications? Does it behave differently in high battery and low battery conditions? How much space is it occupying? How much trail is it leaving? How is it clearing the trails / logs, etc. are important non-functional factors which should be considered as part of mobile testing strategy.
These differences in testing Mobile apps and testing Desktop & Web applications points out the complexities and nuances that make mobile testing a unique skill for testers. Mobile App Testing when approached with a different perspective, results in efficacy, efficiency, and cost effectiveness.
Feel free to add anything that we missed, in the comments below…Thanks!