Android Performance Testing

1. Intro

I'm building a distributed testing infrastructure on top of KATS for a DARPA project, and we had a need for performance monitoring of CPU, memory, and process profiling information (including context switches) throughout a test run. MADARA already has a library called MAML which allows for quick Python script development to instrument a phone via the Android Monkeyrunner tool, but Monkeyrunner doesn't really provide for performance profiling. So, how do you quickly and easily retrieve a summary of CPU and memory utilization on your Android phone?

2. Solution

The information is available through several utilities in the Android Debug Bridge including top and the varied information stored inside of the /proc directory. What I've done is make these more accessible through additions to the open-source maml.py library and the new maal.py (Madara Android ADB Library) which does not require Monkeyrunner at all.

The MAAL provides much of the same functionality that MAML does, but is much slower with keyevents (I will fix this by reusing the same shell session, but it isn't a priority right now). MAAL and MAML also have a new library function called print_device_stats which allows for printing both a long form and a one line summary for CPU and memory usage.

Three scripts have also been added to utilise these libraries and provide general-purpose testing information for Android smartphone programmers. For example, the following is a detailed view of the current memory and CPU usage on a Motorola Droid in our lab:

---

2.1. maal_monitor.py

The command line arguments for maal_monitor.py are available by passing -h or --help to the script. The following script execution monitors performance for 1 iteration (-n 1) and prints the top 10 cpu-intensive processes running on the phone.

./maal_monitor.py -p 10 -n 1
Memory: 5892 kB free of 230852 kB
User 4%, System 6%, IOW 0%, IRQ 0%
User 15 + Nice 0 + Sys 21 + Idle 273 + IOW 0 + IRQ 0 + SIRQ 0 = 309

  PID CPU% S  #THR     VSS     RSS PCY UID      Name
 1021   4% S    57 215056K  56096K  fg system   system_server
30994   3% S    20 140436K  24940K  bg app_24   edu.vu.isis.ammo.spotreport
23827   2% R     1    876K    392K  fg shell    top
  177   0% S     1      0K      0K  fg root     omap2_mcspi
    5   0% S     1      0K      0K  fg root     events/0
  995   0% S     2   1272K    128K  fg compass  /system/bin/akmd2
 1053   0% S     1      0K      0K  fg root     tiwlan_wq
  160   0% S     1      0K      0K  fg root     cqueue
  180   0% S     1      0K      0K  fg root     cpcap_irq/0
  238   0% S     1      0K      0K  fg root     ksuspend_usbd

The summarized view looks like this for maal_monitor.py:

./maal_monitor.py -p 10 -n 1 -s
Memory: 5952 kB free of 230852 kB. CPU: Total 6% (User: 2% Sys: 4%)

Maal_monitor.py is great for taking periodic measurements, but this may be too coarse-grained and may be too inaccurate with regards to CPU utilisation for your testing needs (with maal_monitor.py, we're essentially polling every five seconds for current utilization, which is approximated). If you need hard numbers for cpu usage, context switches, number of processes launched, etc., I provide a separate set of scripts.

---

2.2. maal_proc_stats.py and maal_stats_cmp.py

These scripts are generally used in the following way: 1) call maal_proc_stats.py with an outfile location to your storage drive, 2) run your test, 3) call maal_proc_stats.py with an outfile location that is different from #1, and 4) call maal_stats_cmp.py on the two files created in #1 and #3 to get the performance difference for your test.

The output for maal_proc_stats.py will look like the following:

./maal_proc_stats.py -s
Clockticks: User 5451112. System 5357590. IO 566. 
Processes: Num 56061. Switches 812051296.

The first line shows the clock ticks spent in user processes, system processes and dispatching IO (it does not count idle ticks). The second line shows the number of processes that have been launched since phone boot, and the number of context switches since boot.

Now, after creating two files with this script according to the process above noted in #1 and #3, you can process the performance information that changed during the test by running the following:

./maal_stats_cmp.py --infile1 first.stats --infile2 second.stats
Clockticks: User 629. System 393. IO 0. 
Processes: Num 23. Switches 35566.

Not only can you use maal_stats_cmp.py on the output from maal_proc_stats.py, but you can also process the difference between copies of the /proc/stats file (you just need to adb pull these files to your computer or something, if that's what you would like to do).

With these numbers in place, you should be able to configure systems like BuildBot or your favorite scoreboarding system to display test results via threshold values based on known good test run resource usages. If changes have caused huge CPU or memory spikes, they should show up in one or both of these MAAL performance logging methodologies.

3. Downloads

MAAL and associated scripts
MAML and any open-source scripts
MADARA KATS for synchronizing and coordinating testing processes