Automated Control of an Android Device with Python

2010-06-17T20:30:59Z

Dustingram:

==Intro==
As mentioned in my previous article, [[Device Screenshots with the Android Debug Bridge]], I'm currently working on an application which required the ability to get a live screenshot of my physical Android device. The second component to that problem, however, involved processing that image, and then automating a UI/Application response exactly like an actual user is (read: automatically touch the screen).

Again, like the ability to grab the device's current display, I figured this would be simple to do through the Android Debug Bridge. However, as it turned out, this was much simpler than I expected, and unlike the screenshot tool, I could avoid using Java for this.

===References===
The first hint that I could send various UI events to the device came from the documentation for the [http://developer.android.com/guide/developing/tools/monkey.html Android UI/Application Exerciser Monkey], although being able to send completely random events proved pretty useless. However, it led me to the [http://android.git.kernel.org/?p=platform/development.git;a=tree;f=cmds/monkey Git Repository] for the tool, which in turn had a [http://android.git.kernel.org/?p=platform/development.git;a=blob;f=cmds/monkey/README.NETWORK.txt README file] which described the Simple Protocol for Automated Network Control.

==Using the Network Control Protocol==
From the README:
The Simple Protocol for Automated Network Control was designed to be a
low-level way to programmability inject KeyEvents and MotionEvents
into the input system. The idea is that a process will run on a host
computer that will support higher-level operations (like conditionals,
etc.) and will talk (via TCP over ADB) to the device in Simple
Protocol for Automated Network Control. For security reasons, the
Monkey only binds to localhost, so you will need to use adb to setup
port forwarding to actually talk to the device.

What this means, in a nutshell, is that a developer can use any language to command an ADB device (on loopback) using a simple TCP message.

The first step is to make the ADB forward TCP traffic on the localhost to the device, in this case, port <tt>1080</tt>:

$ ./adb forward tcp:1080 tcp:1080

Simple! At this point, the README attempts to discuss all the possible commands. I was purely interested in a <tt>touch</tt> command (which is, in fact, two commands), so if you're looking for something else, look at the [http://android.git.kernel.org/?p=platform/development.git;a=blob;f=cmds/monkey/README.NETWORK.txt README file].

If touch is what you want, though, here's what you need to know:

Note that sending a full button press requires sending both the down
and the up event for that key

This means we need a <tt>touch down</tt> command followed by a <tt>touch up</tt> command, separated by a newline, as follows (which touches point <tt>(x=30,y=60)</tt>:

"touch down 30 60\ntouch up 30 60"

'''Important:''' In practice, though, this won't work every single time. It seems that the device may get confused by a series of strung-together commands in quick succession, especially a pair of <tt>touch</tt> events. In most practical applications, it would make sense send commands individually, and sleep in between -- after all, real users don't move that fast!

==A Sample Python Script==
Using our new knowledge of Android's Simple Protocol for Automated Network Control, here's a very simple Python script to send a single pair of <tt>touch</tt> events, using the <tt>socket</tt> module:

<syntax lang="python">
#!/usr/bin/env python

import socket

TCP_IP = '127.0.0.1'
TCP_PORT = 1080
BUFFER_SIZE = 1024

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
s.send("touch down 30 60\ntouch up 30 60")
data = s.recv(BUFFER_SIZE)
s.close()

print "Received: ", data
</syntax>

And that's it!

[[Category:Code]]
[[summary::Using Android's Simple Protocol for Automated Network Control to send touch events to a physical device using Python and a TCP socket. | ]]
[[timestamp::{{BORNONDATE}}|]]

Device Screenshots with the Android Debug Bridge

2010-06-16T20:28:05Z

Dustingram: /* Starting the ADB */

==Intro==
Certain recent endeavors have lead me to desire a way to automatically get pixel-perfect screenshots of my physical Android device (Motorola Droid). I knew it was possible to use the "adb", or Android Debug Bridge tool; however, the actual tool has a number of features I didn't need, plus I wanted to automate the entire process.

===References===
A number of mailing list posters and/or bloggers, some dating back a couple years, have attempted a solution, but almost every one either depended on an out-of-date library, or simply didn't work. The majority of my solution is taken from
Jon Larimer's code [http://blog.codetastrophe.com/2008/12/using-androiddebugbridge-api-to-get.html here], but I had to make a number of modifications to render the resulting image properly.

==Starting the ADB==
I had a minor issue getting the ADB started. What happens is, if the ADB server isn't started with the proper permissions (root), there will be no device access. The tricky thing is, if you've got Eclipse running with the Android plugins, it'll have started up the ADB server already, with basic permissions. The following is an example of what you'll get from a running server without enough permissions:

$ ./adb devices
List of devices attached
???????????? no permissions

The solution is:

$ ./adb kill-server
$ sudo ./adb start-server
* daemon not running. starting it now *
* daemon started successfully *
$ ./adb devices
List of devices attached
04XXXXXXXXXXXX13 device

==Connecting to the ADB==
Create the DebugBridge as follows:

<syntax lang="java">
AndroidDebugBridge.init(false);
AndroidDebugBridge adb = AndroidDebugBridge.createBridge();
</syntax>
We're going to add an instance of our class as a device change listener, so we'll know when changes occur, like the connection to a device. This also means the class will implement <tt>IDeviceChangeListener</tt>.

<syntax lang="java">
Screenshot ss = new Screenshot();
AndroidDebugBridge.addDeviceChangeListener(ss);
</syntax>
Use this to collect all attached devices:

<syntax lang="java">
IDevice[] devices = adb.getDevices();
</syntax>

==Getting the Raw Screenshot==
Get the RawImage from the device:

<syntax lang="java">
RawImage raw = device.getScreenshot();
</syntax>
Set up the BufferedImage with the appropriate dimensions:

<syntax lang="java">
Boolean landscape = false;
BufferedImage image = null;
Dimension size = new Dimension();
int width2 = landscape ? raw.height : raw.width;
int height2 = landscape ? raw.width : raw.height;

if (image == null) {
image = new BufferedImage(width2, height2, BufferedImage.TYPE_INT_RGB);
size.setSize(image.getWidth(), image.getHeight());
} else {
if (image.getHeight() != height2 || image.getWidth() != width2) {
image = new BufferedImage(width2, height2, BufferedImage.TYPE_INT_RGB);
size.setSize(image.getWidth(), image.getHeight());
}
}
</syntax>
Finally, make the appropriate adjustments to the RawImage's RGB values:

<syntax lang="java">
int index = 0;
int indexInc = raw.bpp >> 3;
for (int y = 0; y < raw.height; y++) {
for (int x = 0; x < raw.width; x++, index += indexInc) {
int value = raw.getARGB(index);
if (landscape)
image.setRGB(y, raw.width - x - 1, value);
else
image.setRGB(x, y, value);
}
}
</syntax>
There are also a number of overrides for our class to fill the requirements of the implementation, for the various changes

==All together==
You can download the entire working class [[Media:Screenshot.java|here]].

[[Category:Code]]
[[summary::Getting a screen shot of a live or emulated Android Device automatically using the Android Debug Bridge. | ]]
[[timestamp::{{BORNONDATE}}|]]

Dustin Ingram - New pages [en]

Automated Control of an Android Device with Python

Device Screenshots with the Android Debug Bridge