Android UI Automation via adb + uiautomator

Purpose

Drive an Android UI without human fingers: read what is on screen, compute where to tap, act, and verify the result. Works over plain adb (SSH tunnel, USB, TCP). No Appium/instrumentation needed — only adb, Python 3 (stdlib), and bash.

Core mental model: the DUMP-TAP-VERIFY loop

Never tap blind. Every interaction follows this loop:

1. STATE   confirm foreground app/window (dumpsys window)
2. DUMP    uiautomator dump /sdcard/x.xml → adb pull
3. PARSE   python scripts/parse_ui.py x.xml
           (or find_node.py for a specific label)
4. LOCATE  bounds="[x1,y1][x2,y2]" → center = ((x1+x2)/2, (y1+y2)/2)
5. ACT     adb shell input tap X Y  (or swipe/keyevent/text)
6. VERIFY  sleep 2-5 s → fresh dump → assert new state

Golden rules

1. Coordinates are disposable; the loop is not. Never reuse coordinates from a previous dump — layouts shift. Always re-dump before every tap.
2. Match text/content-desc first, coordinates last.
3. Verify after every action with a fresh dump or mCurrentFocus.
4. One device per command: always -s SERIAL.

Setup / prerequisites

# 1. Tunnel (if device lives behind SSH) — run as background job:
ssh -i <key> -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 \
    -N -L 127.0.0.1:5557:127.0.0.1:5557 user@host

# 2. Connect + sanity check
adb connect 127.0.0.1:5557 && adb devices       # expect: ...device
adb -s 127.0.0.1:5557 shell getprop sys.boot_completed   # must print "1"

Command reference (quick)

Task	Command
Dump screen	adb -s S shell uiautomator dump /sdcard/d.xml then adb -s S pull /sdcard/d.xml
List visible labels	python skills/ui-automator/scripts/parse_ui.py d.xml
Find one node	python skills/ui-automator/scripts/find_node.py d.xml "Play Protect"
Icon in header	python skills/ui-automator/scripts/list_top_nodes.py d.xml 170
Verdict icons (Pass/Fail)	python skills/ui-automator/scripts/check_verdict.py d.xml
Tap	adb -s S shell input tap X Y
Scroll	adb -s S shell input swipe X1 Y1 X2 Y2 MS
Back/Home/Enter	adb -s S shell input keyevent KEYCODE_BACK (BACK=4)
Type	adb -s S shell input text 'hello%sworld' (%s = space)
Launch app	adb -s S shell am start -W -n PKG/.MainActivity
Foreground window	adb -s S shell dumpsys window | grep mCurrentFocus

Full details: references/adb-uiautomator-cheatsheet.md

Scripts (in scripts/)

Script	What it does
ui_dump.sh [serial] [out]	Dump current screen + pull locally (one step)
parse_ui.py dump.xml	Print every visible TEXT/DESC node with bounds
find_node.py dump.xml <regex> [--desc]	Find node(s); prints BOUNDS + CENTER + CLICKABLE; exit 2 = no match
list_top_nodes.py dump.xml [ymax]	Header-bar nodes lacking labels (y < ymax)
check_verdict.py dump.xml	Count Pass/Fail content-desc icons (integrity checker)
tap_text.sh [-s S] [-d] "<regex>"	One-shot: dump → find → tap center
screen_state.sh [serial]	Focus + top activity + screen size snapshot

Worked examples (real sequences from this repo)

A. Open Play Store account sheet → Play Protect (scrollable bottom sheet)

SER=127.0.0.1:5557
adb -s $SER shell input tap 648 104                    # avatar (found via list_top_nodes.py)
sleep 3
adb -s $SER shell input swipe 360 1000 360 450 400     # scroll; rows ~116 px apart
bash ui_dump.sh $SER sheet.xml
python find_node.py sheet.xml "Play Protect"           # → CENTER=(360, 898)
adb -s $SER shell input tap 360 898                    # tap row center, not text edge

B. Capture a Play Integrity verdict

adb -s $SER shell am force-stop gr.nikolasspyr.integritycheck
adb -s $SER shell am start -W -n gr.nikolasspyr.integritycheck/.MainActivity
adb -s $SER shell input tap 360 877        # CHECK button ([272,829][448,925])
sleep 20                                   # token round-trip takes 10-18 s
bash ui_dump.sh $SER v.xml && python check_verdict.py v.xml

Note: the checker's raw-JSON dialog body is NOT visible in uiautomator dumps and clipboard reads are blocked on Android 14 — read verdicts from the Pass/Fail icons.

More recipes: references/proven-workflows.md

Decision tree

Need to interact with Android UI
├─ Device connected?          no → tunnel up → adb connect → boot_completed=1
├─ Target app foreground?     no → am start -W ... (verify Status: ok)
├─ Element has text?          yes → dump → find_node.py "<text>" → tap CENTER
│    └─ below fold?                → input swipe up → re-dump → retry
├─ Element is an icon?
│   ├─ content-desc present?      → find_node.py --desc "<desc>"
│   └─ unlabeled header item?     → list_top_nodes.py, identify by class+bounds
├─ Need action result?        sleep → fresh dump → parse/assert state
└─ Something misbehaved?      references/troubleshooting.md

Safety notes

• pm clear PKG wipes ALL app data (GSF ids, sign-ins, scan history) — only use with explicit intent.
• Do not spam taps against rate-limited flows (integrity checks, logins).
• On shared hosts: container restarts usually fine; full VPS reboots may be forbidden — check project docs first.

Host pitfalls digest (Windows/PowerShell)

1. No < stdin redirect → pipe files: Get-Content -Raw s.sh | ssh h "tr -d '\r' | bash -s"
2. & starts PS background jobs; chain commands with ;.
3. $C:/path inside double quotes breaks PS variable parsing — avoid $var: patterns.
4. Nested-quote python one-liners are fragile — use script files from scripts/.
5. Strip CRLF (tr -d '\r') before piping scripts to remote bash.

Full symptom/fix table: references/troubleshooting.md