Same .app for every framework (sha256 recorded above)
Zero LLM calls
Same assertions per canonical test id
Same retry helpers (dropped first tap, dropped deep link, system alerts)
Fresh app launch per test
retries=0
Frameworks run sequentially
caffeinate -dims held for the run
Warm medians exclude rep 1 (cold)
Machine-readable reporters only
Host otherwise idle; load average recorded
Android sim counts the app and every framework's on-device helpers (uiautomator, instrumentation runners, agent-device, Argent, warden-uiad), matched case-insensitively
Host processes are tracked by pid and start time, so a reused pid is never counted
Android animation scales are reset to the device's original values before every framework and restored at the end
Summary
Framework
Status
Version
Suite median (warm)
p95
Ratio
Cold rep 1
Cold-start cost
Prepare
Teardown
Overhead
Pass rate
Flake
Residue
Footprint (LOC / deps)
warden
✓ ok
bun 1.3.14 + warden-e2e
86.30 s
86.71 s
1.00×
71.98 s
none (rep 1 not slower, -13.72 s)
596 ms
72 ms
67 ms
100%
0%
0 proc / 0 app / 0 path
379 LOC / 0 B
argent
✓ ok
0.30.1
134.32 s
134.65 s
1.56×
133.61 s
1.01 s
1.72 s
293 ms
163 ms
100%
0%
0 proc / 0 app / 1 path
161 LOC / 0 B
agent-device
✓ ok
agent-device 0.21.31
91.61 s ✗
91.70 s
1.06×
87.46 s
12.23 s
16.37 s
315 ms
213 ms
97.5%
2.5%
0 proc / 1 app / 1 path
158 LOC / 0 B
tester-army
✓ ok
e2e 0.20.0
132.52 s
132.89 s
1.54×
150.63 s
18.11 s
0 ms
495 ms
2.24 s
100%
0%
0 proc / 1 app / 1 path
307 LOC / 0 B
Warm suite wall time
warden
argent
agent-device
tester-army
Bar = warm median. Whisker = p95. Dot = cold rep 1. Lower is faster; ratios are against the fastest framework.
Per-test medians (ms)
Test
warden
argent
agent-device
tester-army
onboarding steps through to sign-in
2929 (1.00×)
8301 (2.83×)
3830 (1.31×)
5964 (2.04×)
invalid email and short password show field errors
16054 (1.00×)
18221 (1.13×)
16579 ✗ (1.03×)
21119 (1.32×)
a locked account shows the form alert
10096 (1.04×)
14442 (1.49×)
9673 (1.00×)
12914 (1.34×)
valid credentials land on todos
9788 (1.00×)
15416 (1.58×)
9784 (1.00×)
14850 (1.52×)
add, complete and filter todos
14494 (1.00×)
21110 (1.46×)
15470 (1.07×)
21230 (1.46×)
open a todo, mark it done, then delete it
11089 (1.00×)
19105 (1.72×)
12422 (1.12×)
18121 (1.63×)
deep link opens a todo by id
10656 (1.02×)
17811 (1.70×)
10483 (1.00×)
18094 (1.73×)
settings shows the account, toggles switches and signs out
10945 (1.00×)
19616 (1.79×)
13136 (1.20×)
18075 (1.65×)
Median per test
warden
argent
agent-device
tester-army
Spread across warm reps
Whisker = min to max. Shaded box = min to p95. Tick = median. A distribution carries no quartiles, so the box is not an interquartile range.
Per-step latency
warden
argent
agent-device
tester-army
Whisker = min to max. Shaded box = min to p95. Tick = median. Only frameworks that report steps appear.
Framework
Steps
Median
p95
Max
warden
504
85 ms
3.63 s
4.98 s
argent
1192
321 ms
1.82 s
2.82 s
agent-device
632
471 ms
1.62 s
1.89 s
tester-army
592
481 ms
3.21 s
4.01 s
Where step time goes
Step scopes differ per tool: warden's one fill step spans the focus tap, select-all, typing and read-back confirm, argent reports a separate idle step and into …, agent-device reports fill and type separately, and tester.army reports locator steps. Compare a verb's share within a tool, not its median across tools.
warden
Verb
n
Share
Median
p95
Max
fill
68
80.4%
3.38 s
4.20 s
4.98 s
tap
180
10.6%
75 ms
333 ms
620 ms
expect
256
9.1%
21 ms
558 ms
578 ms
argent
Verb
n
Share
Median
p95
Max
into
68
27.9%
2.15 s
2.81 s
2.82 s
idle
232
27.5%
925 ms
1.04 s
1.24 s
visible
320
18.3%
13 ms
1.03 s
1.04 s
id=
380
12.8%
322 ms
352 ms
372 ms
launch
32
10.9%
1.80 s
1.82 s
1.83 s
hidden
32
2.4%
16 ms
1.05 s
1.05 s
open-url
4
0.1%
116 ms
124 ms
124 ms
next-test.yaml
28
0%
4 ms
6 ms
6 ms
other
96
0%
1 ms
5 ms
6 ms
agent-device
Verb
n
Share
Median
p95
Max
press
244
40.6%
520 ms
1.02 s
1.09 s
type
64
23.9%
1.26 s
1.86 s
1.89 s
wait
276
20.9%
55 ms
969 ms
1.08 s
open
36
9%
1.02 s
1.07 s
1.10 s
fill
12
5.5%
1.69 s
1.75 s
1.75 s
tester-army
Verb
n
Share
Median
p95
Max
getbytestid
496
69.5%
267 ms
3.23 s
4.01 s
dismiss
52
22.3%
2.26 s
2.33 s
2.33 s
(unlabelled)
32
6.3%
1.03 s
1.05 s
1.06 s
accept
4
1.6%
2.02 s
2.03 s
2.03 s
warden-example
4
0.2%
290 ms
295 ms
295 ms
getbytext
4
0.1%
65 ms
66 ms
66 ms
Resources
warden
argent
agent-device
tester-army
CPU
Memory
Framework
Scope
CPU-s / rep
CPU avg
CPU peak
RSS avg
RSS peak
warden
host
2.97 s
3.0%
38.8%
145.9 MB
158.6 MB
warden
sim
—
74.3%
282.8%
10.6 GB
10.7 GB
argent
host
4.13 s
2.7%
41.8%
181.4 MB
236.7 MB
argent
sim
—
44.5%
209.9%
10.5 GB
10.6 GB
agent-device
host
11.18 s
13.6%
611.5%
711.9 MB
2.4 GB
agent-device
sim
—
86.0%
248.7%
10.8 GB
11.0 GB
tester-army
host
17.74 s
13.8%
611.2%
598.2 MB
2.5 GB
tester-army
sim
—
84.1%
248.1%
11.1 GB
11.4 GB
CPU-s / rep is the host CPU time of the timed run (warm median) and the headline. CPU avg and peak are ps %cpu, a decaying average on macOS: they move with suite length, do not judge on them.
Where the CPU goes
One table per framework, from the warm-median rep's timed run (the rep whose host CPU-seconds is the lower median). Rows are processes grouped by label (top 12 per scope, the rest in other); host is the framework's own process tree, sim the simulator's launchd_sim tree (iOS only). Share is of the scope's CPU-seconds. RSS is the label's summed RSS per 500 ms tick (0 where it had no process). Spawns counts distinct pids a tick saw: a process that lives between two ticks is only seen through the (exit charge) rows (CPU of a child the bench ran, booked at its exit), so many short-lived xcrun simctl calls read low in Spawns and in CPU.