Considering the wide variety of mobile devices from Apple to Samsung to android and to iOS and so on, it is very hard for the testing team to arrange all sorts of mobile devices while working under considerable amount of budget and timeline related constraints. Solution – Mobile emulators are powerful tools for testing mobile applications and are widely used for both manual and automated mobile application testing. But can they deliver 100% coverage? Are these reliable? Can we neglect Real-device Testing? Are there any limitations to Emulators? When to use Real devices for Testing Mobile applications?
If you don’t know what an Emulator is and what is the need for Mobile Emulator in testing mobile applications, please visit our previous posts “Emulator, Simulator & Real-device Testing“ and “The need for Mobile Emulator / Simulator (s)“ to get the context.
Since mobile apps are used on real handsets and not emulators, testing on real devices is required to ensure the highest level of application quality. Though testing mobile applications using emulators is better than not testing at all, but Emulators too have limitations. Let’s have a look…
Testing Mobile Applications for the Real World
Hardware and Software related issues
Hardware testing is one of the biggest challenges in testing mobile applications. There are hundreds of makes, models, screen sizes and versions of mobile devices, and new ones are coming out weekly. Chip sets in particular can also provide vastly different user experiences between high and low end devices, yet can completely fool an emulator, because the processor of the PC they run on is ten times more powerful than that of a given mobile device. One best practice would be to develop a prioritized mobile hardware device make, model and version list that represents the devices that the majority of your customers are using. Once you have physically tested the priority devices, emulators may be used to provide some of the other screen resolutions and display sizes that may be popular but that you do not have on hand.
The emulator is typically a “plain vanilla” version of the OS and often does not reflect the specific hardware and software features of each supported device. Different OEMs need to make an impact on the market, therefore each vendor will try to offer more appealing hardware & software features (e.g., faster browser, White Paper better launcher, more efficient memory management, etc.) Whether those enhancement work well with your application is another story.
The mobile handset is a compact platform with limited CPU power and memory. An application running on a PC does not accurately reflect the user experience on your handset, which, in extreme cases, it may not even work on at all. A Mobile application never works alone and always interacts with other software, including multiple web browsers, and simply coexisting with other software installed on the device. All this software is competing for resources and needs to be properly configured.
The exact color display, different resolutions, how bright or dim the app appears, simulating light and dark situations, whether the color scheme works in outdoor light – these may differ with respect to the emulator. An emulator may not give the information regarding brightness, saturation, and many other factors.
While testing mobile applications, one of the physical phone attribute that cannot be tested by an emulator is the battery. There is simply no way to test the efficiency and power consumption of your application via an emulator. An app that kills your phone in an hour will be discarded very quickly.
Usability, Look and Feel
Can you operate the device easily while walking around? Emulators can be used to navigate through an application and look at the screen layout and presentation, but the true user experience, look, and feel can only be properly tested as a customer would: on a physical device. It is a totally different experience using a full-sized keyboard and mouse than physically navigating the mobile application using touch and gestures.
Testing Mobile Applications on real devices is the only way to truly understand the user experience, taking into account the CPU, memory, screen size, etc. for a given device. When it comes to the quality of your services, there is no substitute for using a real handset operating in one or more live carrier networks.
Performance & Memory Issues
The emulator/simulator is not always the best type of solution for validating the performance of the application for a longer period of time. The performance of the emulator tends to be slower / faster than the original devices at times. It’s easier to expose performance defects with real handsets.
Memory issues are rampant in testing mobile applications, and cannot be tested adequately by emulators. An emulator, with dedicated resources just for running the app, will quite often tell you that everything is fine, simply because it does not account for the other processes and functions that might be in play on a low-end systems. The performance of your app on an emulator may be much better than on a real device. This could cause a misconception of how quick your app responds. Testing Mobile Applications on multiple real devices will alert you to any such issues.
The emulator might not be able to simulate the incoming interrupts like SMS, incoming calls and push notification interruptions. An emulator cannot adequately show how such network related events will react with your application. Using Real device tests, one can find out how the application would react to unexpected events. For example, how a mobile banking application would respond in the event of incoming call, message or email during an ongoing transaction? Another is the difference between mouse and keyboard interactions vs. actual touchscreen.
Platform and Carrier Network Considerations
Typically, your emulator will be running a vanilla version of the OS. This means that fundamental differences between vendors and carriers may not adequately be covered. Carriers and vendors alike add interface layers, skins and other middle-ware over the top of the stock OS. In some cases, both the carrier and the device manufacturer add layers. This can add another layer of fragmentation that an emulator simply cannot keep up with.
The uniqueness of mobile devices is that they use wireless and/or cellular networks. In many cases, the low level network stack itself is completely different. The network environment is different and will result in your application behaving in another way. Testing mobile applications for network and connectivity includes Interrupt testing, signal connectivity, network radio switching, OEM and carrier, and network inter-operability. None of these types of testing can be completed correctly using an emulator, as the emulating software uses much, if not all, of the desktop computer’s connectivity.
Application Support or Application-specific Testing
Every application is different. If your application has responsive Web design (RWD) or is a native application, each has their own different challenges. Not all the emulators support the complete gamut of mobile applications. You might need to invest in software patches in order to support additional applications. Native applications are far more unpredictable and require much more physical device testing, as the emulators do not exactly mirror the operating systems your customers are using.
Many sensor issues simply cannot be tested via emulators. While Android and iOS emulators can test some sensor functions (with limitations), others like push notifications, Geo-location, orientation, and other functionality are impossible to test adequately without a real device. NFC and other functions also require a real device. Please refer our previous post “Smartphones | Sensor List” for complete list of Smartphone Sensors.
A major concern for mobile users is security. People are more concern to their data like bank account details/passwords etc. So, testing mobile applications for these types of security concerns in a simulated environment is not a good use of time because here we need to test the actual behavior of handset. And we need to test this on each device individually as not all handsets have the same security designs.
Even if the testing goes perfectly, you cannot be 100% sure that your data can actually apply to a real device. This raises the question of which tests need to be double-checked on a real device and which can be assumed reliable on the emulator. Also, in the event of a test that fails on the emulator, testers need to decide whether to perform the test on a real device or simply assume that the function does not work and requires correction. Testing mobile applications on real handsets always gives you accurate results. There is no need to worry about “false negatives” or, more importantly, the ever – dangerous “false positives.”
You cannot emulate real life usage with software. How does the mobile app look when you are outside in the sun? Can you use the app while walking? Does the interface translate well with swipes and finger usage? When testing mobile applications all these situations are impossible to replicate with emulators.
While an emulator is perfectly adequate for initial testing during development, especially for iterative tests of small portions of code, they are not adequate for final testing of a product because they simply cannot test everything that needs to be tested. Testing mobile applications on real handsets always gives accurate results for the device in question, without false positives or negatives. Real device testing is an absolute requirement for a successful app – it is the only way to be a hundred percent sure that a given feature works on a given device.
These limitations of Mobile Emulators must be carefully considered when designing strategy for testing mobile applications. The optimal Mobile Testing solution for enterprises is neither putting all the eggs into the basket of the real devices nor putting them into the emulator but rather what we need is an optimum combination of both to maintain strong standards and quality assurance.