e2e framework benchmark — ios 2026-10-25

Fairness rules

• Same .app for every framework (sha256 recorded above)
• Zero LLM calls
• Same assertions per canonical test id
• Same retry helpers (dropped first tap, dropped deep link, system alerts)
• Fresh app launch per test
• retries=0
• Frameworks run sequentially
• caffeinate -dims held for the run
• Warm medians exclude rep 1 (cold)
• Machine-readable reporters only
• Host otherwise idle; load average recorded
• Android sim counts the app and every framework's on-device helpers (uiautomator, instrumentation runners, agent-device, Argent, warden-uiad), matched case-insensitively
• Host processes are tracked by pid and start time, so a reused pid is never counted
• Android animation scales are reset to the device's original values before every framework and restored at the end

Summary

Warm suite wall time

• tester-army

Bar = warm median. Whisker = p95. Dot = cold rep 1. Lower is faster; ratios are against the fastest framework.

Per-test medians (ms)

Median per test

• tester-army

Spread across warm reps

Whisker = min to max. Shaded box = min to p95. Tick = median. A distribution carries no quartiles, so the box is not an interquartile range.

Per-step latency

• tester-army

Whisker = min to max. Shaded box = min to p95. Tick = median. Only frameworks that report steps appear.

Where step time goes

Step scopes differ per tool: warden's one fill step spans the focus tap, select-all, typing and read-back confirm, argent reports a separate idle step and into …, agent-device reports fill and type separately, and tester.army reports locator steps. Compare a verb's share within a tool, not its median across tools.

tester-army

Resources

• tester-army

CPU

Memory

CPU-s / rep is the host CPU time of the timed run (warm median) and the headline. CPU avg and peak are ps %cpu, a decaying average on macOS: they move with suite length, do not judge on them.

Where the CPU goes

One table per framework, from the warm-median rep's timed run (the rep whose host CPU-seconds is the lower median). Rows are processes grouped by label (top 12 per scope, the rest in other); host is the framework's own process tree, sim the simulator's launchd_sim tree (iOS only). Share is of the scope's CPU-seconds. RSS is the label's summed RSS per 500 ms tick (0 where it had no process). Spawns counts distinct pids a tick saw: a process that lives between two ticks is only seen through the (exit charge) rows (CPU of a child the bench ran, booked at its exit), so many short-lived xcrun simctl calls read low in Spawns and in CPU.

tester-army

Residue

• tester-army: driver apps: com.callstack.agentdevice.runner.uitests.xctrunner; paths: /Users/expo/.agent-device-bench/tester-army (4.9 MB)

Unsupported / failed

None.