Mobile Automation Device Lab

Mobile Automation Device Lab

As the world moves mobile so must the effort for testing and automating the quality checks for our apps. In today’s fragmented world of Android and IOS with different versions and different handset vendors, there is much to verify. In Rakuten, we need to test on real devices and to achieve this level of automation, we have hacked together our very own device lab.

What is a Device Lab?

A device lab is a highly functional way to provide handsets, tablets, and other Internet-capable devices to those interested in testing their work across a variety of screens and hardware. Designers, developers, content creators, and others can use a device lab to gain a better understanding of what users experience when visiting your site from these devices. By creating a device lab, you can empower your teammates to test their own work on real physical devices in an easy and intuitive way.

Why do we use device lab as opposed to something like cloud?
There are two main reasons we chose to go in for a device lab as opposed to something like cloud:

  • Cloud services are useful, but tend to be rather expensive to use.
    For example, one of the cloud services asks for $4800 per month for 16 devices on an annual contract. There are risks involved related to cost inflation over the years as well.
  • Cloud service providers usually do not feature most of the Japanese Handsets, that we need to test most of our products on. Testing on international models does not necessarily guarantee compliance with Japanese handsets. There is also a possibility of missing Japanese fonts on these international models.

What kind of automation frameworks do we utilize?
We utilize mainly:
Calabash + Ruby
STF for Android Remote Control

What is the architecture like?

Linux Machines => Android Devices (via hub)
5 Mac Machines => 8 IOS Devices

Limitations

Cannot execute parallel on IOS
No automation with bad internet
No siri automation (voice search)