This article is focused on the test approach chosen for initial performance testing of the ERNI Info application. It describes the goal of the test as well as the technologies used.
Short introduction to the ERNI Info application
The App should help ERNI employees gain access to critical information about the company and work environment on the move, while mirroring internal permission levels. To reach all ERNIans it has to work on all devices. The app should also provide employees a simple and intuitive overview of all of ERNI’s mobile services and Apps already available.
Figure 1 - ERNI Today main screen
Goal of the performance test
It is critical for the application’s usability to be highly responsive in all cases. This means that even when the webserver is under a heavy load, user experience with the application must not be affected. Parallely, neither server response time, nor the application’s behavior should slow down.
Structure of the performance test
The application’s <<ERNI Today>> –part of our SharePoint platform that hosts: internal news and networking - main menu’s responsiveness on real mobile devices is checked throughout the test, with the webserver under a load of hundreds of simulated users.
The main test case follows:
- User clicks on the article link on the main screen
- Time 1 is recorded
- User waits until the article page is fully loaded
- Time 2 is recorded
- The difference between Time 2 and Time 1 is the article page load time
Test automation approach, tools used
Appium is an open-source, cross-platform tool for automating native, mobile web, and hybrid applications on iOS and Android platforms, using the same API. This enables code reuse between iOS and Android test suites.
Appium was designed to meet mobile automation needs according to a philosophy outlined by the following four tenets:
- You shouldn't have to recompile your app or modify it in any way in order to automate it.
- You shouldn't be locked into a specific language or framework to write and run your tests.
- A mobile automation framework shouldn't reinvent the wheel when it comes to automation APIs.
- A mobile automation framework should be open source, in spirit and practice as well as in name.
Appium is a webserver that exposes a REST API. It receives connections from a client, listens for commands, executes those commands on a mobile device, and responds with an HTTP response representing the result of the command execution. The fact that there is a client / server architecture opens up a lot of possibilities: tests can be written in any language that has a http client API, but it is easier to use one of the Appium client libraries.
Automation is always performed in the context of a session. Clients initiate a session with a server in ways specific to each library, but they all end up sending a POST /session request to the server, with a JSON object called the 'desired capabilities' object. At this point the server will start up the automation session and respond with a session ID which is used to send further commands.
Supported client languages
Figure 2 - Appium functional principle diagram
The Apache JMeter™ application is open source software, a 100% pure Java application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.
The test realization
The main idea of the test was to simulate the activity of a large number of ERNI employees by capturing the request / response traffic of a single user browsing the app’s main site, simulate all these requests coming from the mobile device with a Jmeter tool and to set up hundreds of virtual users that will be sending these recorded requests to the webserver in parallel. This activity will generate a load on the webserver comparable to the actual load that hundreds of ERNI employees would cause if using they had been using the application at the same time.
//var et_pagename = ""; //var et_areas = ""; //var et_url = ""; //var et_target = ""; //var et_ilevel = 0; //var et_tval = ""; //var et_cust = 0; //var et_tonr = ""; //var et_tsale = 0; //var et_basket = ""; //var et_lpage = ""; //var et_trig = ""; //var et_sub = ""; //var et_se = ""; //var et_tag = "";
Main measure to be taken:
How long does it take to load the page with “ERNI Today” articles on the mobile device while the webserver is under a heavy load?
The test is performed via an automated test case, a simple client written in Java, using the AndroidDriver library offered by an Appium server solution.
First, the Android Driver object has to be initiated, then the Desired Capabilities - minimum required attributes must be set. These define the target device’s preferred web browser and ID of the device.
Finally the URL and port of the Appium server have to be defined. (In this case it runs on the local machine on the default port, but can be also placed on the remote host)
After this initialization, the URL of the webpage that is the subject of the test is provided to the driver’s object and opened.
Figure 3 - Simple code used for setting up a connection to the Appium server and defining the target mobile device
The main test activity on the site is to open an article on the ERNI Today website, measure the page load duration and navigate back to the main page. This simple process is repeated for all of the articles and average load time is calculated afterwards
Figure 4 - The main testing method
The aim of this test was to make sure, that the ERNI Info application is both stable enough and ready to handle the activity of large number of users at the same time.
The test has proven that the application is still comfortably usable, even with 1000 parallel simulated users connected and actively browsing articles. All tools used are Open source; delivered java code as well as whole test execution was documented and made public both on –the ERNI- Wiki and SharePoint sites for the ERNI Info App team for the next test executions and test enhancements.
Possible enhancement for the future testing activities
Determine the number of active parallel simulated users that will significantly slow down the application’s behavior (by ramping up the concurrent user count during the test and repeatedly measure the page’s responsiveness on the mobile device).