Mobile app testing – how to start

Mobile applications have taken over our everyday life for good. The latest mobirank report shows that 66,6% of the population uses a smartphone, of which 92,6% use it to connect to the Internet. Moreover, mobile devices users constitute as much as 91,5% of all Internet users and these numbers are constantly growing. No wonder the ability to get the necessary information with one click, anywhere, anytime is a very convenient solution. The world has become mobile, and such a state requires a change in the current approach. How to meet the growing requirements of users, increase the comfort of using our software, and make its operation simple and pleasant for the user? The key to success is a well-planned test of the mobile application.

There is no need to convince anyone about the importance of testing. The purpose of the tests is to quickly detect errors and check whether they have been fixed in order to reduce the risk and amount of potential financial losses. That’s why tests should be implemented at the earliest stages of the software development process, starting with testing specifications and business requirements.

We must remember that releasing software with bugs results not only in financial consequences but above all a bad user experience and loss of customer trust. Did you know that even small things such as typos or a misplaced button can effectively and permanently discourage users from using your application?

In 2012, Apple decided to create its own maps application. Unfortunately, something went wrong and, instead of a full-fledged product, users received an application full of bugs and problems. Object names and locations were mismatched or out of date, and navigation often led to non-existent places. In subsequent versions of the maps, Apple eliminated bugs and introduced corrections step by step, but they never managed to rebuild the users’ trust and catch up with the popularity of the Google Maps.

Testing is crucial in providing the customer with the highest quality product. With this in mind, it’s worth considering what are the differences in testing mobile, web, and desktop applications, what problems we may encounter, and what types of tests will work best when it comes to mobile applications testing.

There are 3 types of mobile applications: native, hybrid, and mobile internet applications. They differ from each other in terms of the development process and technology, but the principles of designing and testing mobile applications are the same for everyone. The main problem of testing mobile applications is fragmentation of the devices, standards, and software.

There are many devices in many sizes. These differences are influenced by: 

When planning the test process, we must remember that mobile applications have a different ecosystem than web pages, i.e. features and behavior. Therefore, they require a special approach and skillful combination of technical aspects and user experience.

When testing mobile applications, we must also take into account: 

There are many pitfalls in testing mobile applications. Even if we handle the error related to the lack of access to the network, we need to check how the application will behave when someone calls us while using it, whether using the application doesn’t load the battery too much, and whether the user interface won’t crash when we turn the phone on the side. It’s worth checking each application on devices with different screen sizes, because it may turn out that not all elements fit on smaller displays or look unnatural on a large screen.

Before planning tests of a mobile application, we need to know the end-users and the market for which the application is developed. In addition to standard data such as region, time zone, language, we should also get familiar with technical details, e.g. the operating system and its version, the mobile device, etc.

There are many types of mobile devices and it’s very difficult to test the application on all of them. The Android system in various versions can run on thousands of devices. There are a lot fewer devices for iOS, but there are also a lot of configurations available here. Of course, we can use emulators and simulators during the software development process, but at the end of the day, tests should always be performed on a physical device. Solutions such as device farms or beta testing come to the rescue.

Device farms are definitely a cheaper solution than purchasing mobile devices for our own. Tests are carried out remotely, a large number of tests can be performed usually very fast, and we’re charged only for the duration of the use of a specific device. On the device farm, we can carry out most types of tests, f.e. UI tests, functional tests, manual tests, memory management test, and many more. After the tests are completed, we receive a detailed report.

Beta testing is another way to test a mobile application on a large group of different devices and systems in different versions. We provide users with a new mobile application or its new module before its official release. This way, end-users can share their opinions with us, report bugs, and propose improvements. In addition, thanks to automation, we can receive reports on the causes of crashes or other problems with the operation of the application. Beta tests are often used by the gamedev industry, such as Blizzard. After all, who better to test the game and see if it meets the expectations of players than themselves?

Native and hybrid applications have the advantage over mobile web applications that they can also work offline. However, this is often not enough. As we mentioned earlier, mobile applications users often use them in various places, where they can’t always count on a stable Internet connection. While we don’t need a very fast connection to browse Instagram, in the case of making online payments our requirements are increasing.

To ensure a high level of user experience, we should conduct tests of the application’s behavior in conditions of unstable Internet connection. In an IT company located in the center of a big city, usually, the connection doesn’t generate problems, which is why connection simulators come to the rescue. For example, for iOS and macOS, we can use the Network Link Conditioner. If we want to test other systems as well, a good solution is CDNS, which we install directly on the router and that is compatible with all operating systems.

There are so many external factors that affect the operation of the application that we can expect that even the best-tested mobile application won’t work perfectly with every single use. Therefore, it’s worth preparing for errors, testing how the application behaves in case of problems, and taking care of handling potential irregularities. A much better solution is to display an error message, such as “The server is overloaded. Try again in a moment.”, rather than no action, or the worst option – app crash.

Each of our actions should be taken with the end-user of the mobile application in mind and the environment in which they will use it. The user of the web application usually sits comfortably at a desk, having at their disposal a computer with a large screen, additional input devices (keyboard, mouse, or touchpad), and a stable Internet connection. A mobile application user has to deal with it in completely different conditions: on a small device, often on the move, in a hurry, in crowded rooms, or in places where the network connection is unreliable, and in addition they must handle the application with their own fingers.

Let’s imagine a situation where the user has to quickly make a bank transfer while standing in a queue in a little village shop. An application with an unreadable interface, which is difficult to log in to and which won’t be able to handle the transfer with only 3G coverage, will certainly not meet the user’s expectations. A completely different level of customer experience will be achieved by providing the end-user with an intuitive interface, an easy and secure login module. Research shows that as many as 88% of users wouldn’t return to the e-store or would give up using the application as a result of poor user experience.

We should take care of a positive user experience already at the early stage of designing the application. Both Appleand the Open Mindset Alliance provide special guidelines for creating mobile applications in a specific operating system. It’s worth using them and remembering that we don’t always have to stand out – the user will be more willing to use a mobile application whose interface is similar to other well-known applications than one that is very original, but it’s difficult to navigate.

The types of tests that are most useful for testing mobile applications are functional tests, acceptance tests, and system tests. What are the differences, what benefits do they bring, and when should they be used?

Testing new functionalities is the most basic form of testing. Errors can be detected by both the tester and the developer, so the cooperation of the development team and QA team is very important. Testing is carried out during software development, based on a list of dedicated functionalities (sprint tasks), and errors are reported internally through the project board. Often we’re also dealing here with exploratory tests, which don’t need to be documented.

Acceptance testing is no longer just the work of a tester and a programmer. At this stage, the customer is involved in the test process, so we must provide them with a fully working product or module. Verification is primarily carried out on the implementation of business requirements and user experience, and not individual errors in functionalities. The tests are holistic, so we can support ourselves with exploratory tests or simple test cases.

Reporting takes place not only within the team project but also in the project board to which the client has access. It’s also a good time to perform regression tests, which are designed to check that the changes made didn’t cause another defect or error in the previously tested area.

Acceptance tests also include the aforementioned beta tests. Thanks to them, we get the most complete picture of the implementation of the user’s expectations and collect information that will allow us to improve the application and fix any errors.

Quality Control system tests are a completely different caliber of tests. They include any defects collected outside the planned test process or that aren’t functional test errors, e.g. customer and production errors. Bugs detected in system tests are usually reported and handled in a separate project board and marked with the appropriate priority.

We all know that testing on production is a bad idea, but in this case, it’s justified. Thanks to this, we gain information about errors directly from users, which is valuable, because end-users provide us with testing our application on different devices, for different operating systems and their versions, and in conditions of variable stability of the Internet connection.

Let’s imagine the following situation: we need to test the application login module. On the surface, this is a simple task, what can go wrong? When we play the role of a user who, in addition, isn’t very familiar with technology, it turns out that there can be many problems. The tester must be prepared for such scenarios as entering the wrong login or password, trying to log in without providing a password, providing the wrong e-mail address, trying to log in as a user who doesn’t yet have an account, etc. Does this mean that for each potential problem we have to manually recreate the long sequence of actions?

This is where automation comes to the rescue. Entering test data manually doesn’t have to be a sad necessity, as most repetitive and time-consuming test processes can be automated. Although in some cases (e.g. performance tests) automation is irreplaceable, this doesn’t mean that we can completely abandon manual tests.

When considering the implementation of test automation, we must answer the following questions: 

The main purpose of automation is to speed up the test process. Automation isn’t just about e2e testing. In addition to performance tests, automated tests should be used in API testing, post-implementation tests, smoke tests, and regression tests. They are also helpful in acceptance testing.

An example of a mobile app test automation is the following sequence, which aims to check the “add an existing email account in Gmail” feature.

Sample test sequence: 

Appium is an open-source tool dedicated to many platforms, which is used to perform automated tests of all types of mobile applications. Appium allows us to create tests for mobile applications for both iOS and Android using the same API.

Thanks to the client-server architecture, we gain the ability to send HTTP requests to the server, regardless of the language in which the application was written. This way, automated tests can be used for different programming languages.

Given the prevalence of Selenium Webdriver, Appium has expanded its API with specific methods for testing mobile applications. In Appium, a web server using REST API receives calls to the client, listens for commands, and executes them on a given mobile device. The results of executing a given command are visible thanks to the response to HTTP responses.

The percentage of mobile Internet users is constantly growing. Mobile applications should be designed for the end-user who may encounter various adverse external factors while using the application. The purpose of mobile application testing is to detect errors and irregularities in the functioning of the application early to minimize financial losses and ensure a high user experience.

When testing mobile applications, we should take into account not only the software itself, but also its integration with other systems, the stability of the Internet connection, the type of device, and many others. Test automation can significantly improve the process of testing mobile applications, provided that the implementation of automation isn’t a coincidence, but an element of a well-thought-out strategy.

Still hungry for knowledge? If you want to learn more about mobile applications, check our website or read more on our blog: