mobctl

An LLM-native CLI to drive a physical iPhone. Give an AI agent a small set of verbs — observe, tap, type, swipe, scrollto, launch, deeplink, assert — and let it operate a real device on your desk: test a flow, reproduce a bug, walk an onboarding, take a proof screenshot. No paid cloud, no simulator. One ~350-line file, zero npm dependencies.

$ mobctl observe
Other #login-screen @207,448
TextField #email-input "Email" @207,320
SecureTextField #password-input "Password" @207,390
Button #login-submit "Sign in" @207,470
Button #forgot-password "Forgot password?" [disabled] @207,540

$ mobctl tap --id login-submit
tap #login-submit

Why it exists

Most "let an AI use your phone" tools are hosted SaaS with per-device pricing, and most open automation (Maestro, XCUITest test bundles) only targets the simulator. I wanted an agent to drive my real iPhone — because some things only exist there: the App Store, third-party apps, real push, real biometrics, real camera. So mobctl is:

• Physical-device first. It talks to a real iPhone over USB via Appium + WebDriverAgent.
• Designed for an LLM, not a human. observe returns a compact text UI tree — one line per element with its accessibility id (testID), label, value and tap center — instead of a heavy screenshot. That's cheap to feed to a model, and the agent targets elements by id (--id login-submit) rather than guessing pixels. Screenshots are opt-in, for when you actually need to look (layout, colors).
• Self-healing. WebDriverAgent dies after ~90 s of inactivity; every command transparently recreates the session, so an agent never has to think about session lifecycle.
• Dependency-free. Node 20+ only (global fetch). Session state persists between invocations in a small file, so each command is a fresh, cheap process — ideal for an agent that shells out one verb at a time.

How it works

  AI agent / you
        │  one verb per call
        ▼
   mobctl.mjs ──HTTP (W3C WebDriver)──► Appium ──► WebDriverAgent (XCUITest) ──► iPhone
        │                                                                          ▲
        └── parses XCUITest XML → compact UI tree ─────────────────────────────────┘

observe fetches the XCUITest source (XML), keeps only visible interactive/text elements, and prints them as Type #testID "label" =value @cx,cy. Element finders use accessibility-id and iOS predicate-string lookups, so --id and --text targeting map straight onto native queries. Gestures go through Appium mobile: commands (tap, dragFromToForDuration, launchApp, deepLink).

Requirements

• macOS with Xcode + command-line tools
• Appium 2 with the XCUITest driver: appium driver install xcuitest
• Node.js ≥ 20.19
• An iPhone in developer mode, connected over USB, with a signing identity (a free Apple developer account works for personal-device automation)

Setup

git clone <this-repo> mobctl && cd mobctl
cp mobctl.config.example.json mobctl.config.json

Edit mobctl.config.json:

• appium:udid — your device UDID. Find it with xcrun xctrace list devices.
• appium:xcodeOrgId — your 10-char Apple Team ID (from your developer account).
• appium:updatedWDABundleId — a bundle id you control for the WDA runner, e.g. com.yourname.WebDriverAgentRunner.
• app.bundleId / app.scheme — the app you want to drive and its deep-link scheme.

Any field can also be set via env (MOBCTL_UDID, MOBCTL_TEAM_ID, MOBCTL_BUNDLE_ID, MOBCTL_SCHEME, MOBCTL_APPIUM_URL, MOBCTL_SHOT_DIR, MOBCTL_CONFIG) — handy for CI or keeping secrets out of the repo.

Then:

./scripts/up.sh          # start Appium, check the iOS tunnel
./bin/mobctl session     # first run builds + signs WebDriverAgent onto the device

Usage

./bin/mobctl launch --fresh              # cold-launch the app (do this before observe)
./bin/mobctl observe                     # compact UI tree
./bin/mobctl observe --shot              # + screenshot (into the shot dir)
./bin/mobctl tap --id login-submit       # tap by testID (most reliable)
./bin/mobctl tap --text "Sign in"        # tap by text (substring, case-insensitive)
./bin/mobctl tap --xy 207,470            # tap by coordinates (from observe's @cx,cy)
./bin/mobctl type "leo@example.com" --id email-input --enter
./bin/mobctl swipe up --dist medium      # direction = finger movement
./bin/mobctl scrollto --id checkout-cta  # scroll until the element is visible
./bin/mobctl deeplink myapp://home       # direct navigation
./bin/mobctl assert --id home-screen     # assert presence (--absent for the inverse)
./bin/mobctl raw "mobile: terminateApp" '{"bundleId":"com.example.myapp"}'  # escape hatch

The core loop for an agent: launch → observe → act on a #testID seen in observe → observe again to verify. Prefer observe (light) over screenshots; screenshot only when a human/agent needs to visually judge layout or colors.

Tip: put a testID (accessibility id) on every interactive element in your app. That's what makes --id targeting robust across layout changes.

Using it with an AI coding agent

mobctl was built to close a loop: code a UI change → drive the real phone → judge the result → iterate → post a proof screenshot. examples/ai-agent-skill.md is the exact "skill" I hand to my coding agent (Claude Code) so it drives the device autonomously and refuses to claim "done" without a screenshot. It's a good template for wiring mobctl into any agent harness.

Hard-won gotchas (physical device)

These cost me real time; documented so they don't cost you any:

• Code=41 "Not authorized for performing UI testing actions" → enable Settings → Developer → Enable UI Automation, then reboot the iPhone. The entitlement is only read at boot; without the reboot it stays stuck.
• /source "Application …pid.0 is not running" → nothing is foregrounded. Run launch first.
• WDA dies after ~90 s idle → mobctl auto-recreates the session on the next command; it's just slower (~30–60 s) on that first call. Act promptly after a session is created.
• iOS 18.x: automation often works over plain USB/usbmux with an empty RemoteXPC tunnel — try without the sudo tunnel.sh first, only start it if session creation fails with RemoteXPC/connection errors.
• Appium started before the tunnel → repeated ~13 s RemoteXPC timeouts. up.sh handles ordering; otherwise restart Appium after the tunnel.
• The iPhone must stay unlocked during a session (iOS suspends WDA when locked).
• Actions needing real hardware gestures — Face ID / Touch ID, unlocking, side-button double-click, plugging a cable — can't be automated. The agent pattern in the example brings the UI to the blocking point, asks for the one physical gesture, then polls and resumes on its own.

License

MIT — see LICENSE.