warden

Framework comparison

The same 8-test suite on one build and one device in warden, Argent, agent-device and tester.army. Suite time, per-test medians, cold start, resources and footprint.

Every framework runs the same 8 parity tests against the same .app, with no LLM involved. Every run happens on EAS Workflows, one macos-large worker and one fresh simulator per framework (see iOS run 2026-10-25 on EAS for how, and for every artifact). See Benchmark e2e frameworks to run it yourself.

On the 2026-10-25 run (warden 0.3.1) warden finished the warm suite in 86.30 s: 1.56× faster than Argent, 1.06× faster than agent-device (which dropped one test) and 1.54× faster than tester.army. Per step its median was 85 ms against 321 to 481 ms (3.8× to 5.7×), on 2.97 host CPU-seconds per rep against 4.13 to 17.74 (1.4× to 6.0×).

iOS run of 2026-10-25 · full report

Fastest, all tests passing

86.30 s

warden, warm suite median

Ratio spread

1× to 1.56×

warden to Argent

Passed every test

3 of 4

agent-device 97.5%

Suite

8 tests

5 runs each, run 1 cold

  • warden
  • Argent
  • agent-device
  • tester.army

Suite time

Warm suite wall time, measured by the harness around the whole suite process. Warm numbers leave out the first, cold run.

Warm suite wall time per frameworkWarm suite median: warden 86.30 s; Argent 134.32 s; agent-device 91.61 s; tester.army 132.52 s050 s100 s150 s200 swardenwarden: 86.30 s · 1.00× (p95 86.71 s, cold rep 1 71.98 s)86.30 s · 1.00×Argentargent: 134.32 s · 1.56× (p95 134.65 s, cold rep 1 133.61 s)134.32 s · 1.56×agent-device ✗ failing testsagent-device: 91.61 s · 1.06× ✗ (p95 91.70 s, cold rep 1 87.47 s)91.61 s · 1.06× ✗tester.armytester-army: 132.52 s · 1.54× (p95 132.89 s, cold rep 1 150.63 s)132.52 s · 1.54×
Bar: warm median. Whisker: p95. Dot: cold first run. The ratio is the median over the fastest framework that passed every test. A ✗ marks a framework with failing tests, whose time is not comparable.

Per test

Warm median per canonical test. Compare these before the step numbers.

Warm median per test and framework
TestwardenArgentagent-devicetester.army
onboarding steps through to sign-in2.93 s1.00×8.30 s2.83×3.83 s1.31×5.96 s2.04×
invalid email and short password show field errors16.05 s1.00×18.22 s1.13×✗ 16.58 s1.03×21.12 s1.32×
a locked account shows the form alert10.10 s1.04×14.44 s1.49×9.67 s1.00×12.91 s1.34×
valid credentials land on todos9.79 s1.00×15.42 s1.58×9.78 s1.00×14.85 s1.52×
add, complete and filter todos14.49 s1.00×21.11 s1.46×15.47 s1.07×21.23 s1.46×
open a todo, mark it done, then delete it11.09 s1.00×19.11 s1.72×12.42 s1.12×18.12 s1.63×
deep link opens a todo by id10.66 s1.02×17.81 s1.70×10.48 s1.00×18.09 s1.73×
settings shows the account, toggles switches and signs out10.95 s1.00×19.62 s1.79×13.14 s1.20×18.07 s1.65×

Bold: fastest in the row. ✗: the test failed in at least one run, so the time is not comparable.

Step latency

Steps are not the same size in every tool. warden's fill is one step covering the focus tap, select-all, typing and the read-back confirm, so its long tail is the typing cost of a text field (the per-test totals above are the like-for-like number). argent reports a separate idle step and into …, agent-device reports fill and type separately, and tester.army reports locator steps. The table under the chart shows which verbs the step time goes to.

Per-step latency per frameworkMedian step latency: warden 85 ms; Argent 321 ms; agent-device 471 ms; tester.army 481 ms02 s4 s6 sStep latencywarden: warden · median 85 ms (p95 3.63 s, max 4.98 s, 504 steps)warden · median 85 msargent: Argent · median 321 ms (p95 1.82 s, max 2.82 s, 1192 steps)Argent · median 321 msagent-device: agent-device · median 471 ms (p95 1.62 s, max 1.89 s, 632 steps)agent-device · median 471 mstester-army: tester.army · median 481 ms (p95 3.21 s, max 4.01 s, 592 steps)tester.army · median 481 ms
Whisker: min to max. Box: min to p95. Tick: median. Step timings are not defined the same way in every tool, so compare per-test totals first.
Share of step time by verb, per framework
VerbnShareMedianp95Max
warden
fill6880.4%3.38 s4.20 s4.98 s
tap18010.6%75 ms333 ms620 ms
expect2569.1%21 ms558 ms578 ms
Argent
into6827.9%2.15 s2.81 s2.82 s
idle23227.5%925 ms1.04 s1.24 s
visible32018.3%13 ms1.03 s1.04 s
id=38012.8%322 ms352 ms372 ms
launch3210.9%1.80 s1.82 s1.83 s
hidden322.4%16 ms1.05 s1.05 s
open-url40.1%116 ms124 ms124 ms
next-test.yaml280%4 ms6 ms6 ms
other960%1 ms5 ms6 ms
agent-device
press24440.6%520 ms1.02 s1.09 s
type6423.9%1.26 s1.86 s1.89 s
wait27620.9%55 ms969 ms1.08 s
open369%1.02 s1.07 s1.10 s
fill125.5%1.69 s1.75 s1.75 s
tester.army
getbytestid49669.5%267 ms3.23 s4.01 s
dismiss5222.3%2.26 s2.33 s2.33 s
(unlabelled)326.3%1.03 s1.05 s1.06 s
accept41.6%2.02 s2.03 s2.03 s
warden-example40.2%290 ms295 ms295 ms
getbytext40.1%65 ms66 ms66 ms

Share is of the framework's total step time, from the warm runs. What one step covers differs per tool, so compare a verb's share within a tool, not its median across tools.

Cold start, prepare, teardown

Cold-start cost is the first run minus the warm median, plus the time spent in prepare: what installing a runner app or starting a daemon costs, not steady-state speed. Overhead is suite wall time minus the sum of test durations.

A negative cold-start cost is not a duration. It means the cold first run finished faster than the warm median (run-to-run noise on a framework with little startup cost), so the table shows "none (rep 1 not slower, ...)" and leaves it out of the fastest-cell highlight.

Suite time, cold start, prepare, teardown and reliability
MetricwardenArgentagent-devicetester.army
Warm suite median86.30 s134.32 s91.61 s132.52 s
Warm suite p9586.71 s134.65 s91.70 s132.89 s
Cold first run71.98 s133.61 s87.47 s150.63 s
Cold-start costnone (rep 1 not slower, -13.72 s)1.01 s12.23 s18.11 s
Prepare596 ms1.72 s16.37 s0 ms
Teardown72 ms293 ms315 ms495 ms
Overhead outside tests67 ms163 ms213 ms2.24 s
Run-to-run spread (cv)0.5%0.2%0.2%0.2%
Pass rate100%100%97.5%100%
Flake rate0%0%2.5%0%

Resources

CPU is judged in CPU-seconds per rep: the CPU time the framework's own processes used during one timed run (warm median), from per-process deltas of ps time. The sampled %cpu is shown last as a secondary number. Memory is the framework's own processes, sampled every 500 ms.

Host CPU-seconds per repHost CPU-seconds per rep: warden 3.0 s; Argent 4.1 s; agent-device 11.2 s; tester.army 17.7 s0 s5 s10 s15 s20 sHost CPU-seconds per repwarden: warden · 3.0 swarden · 3.0 sargent: Argent · 4.1 sArgent · 4.1 sagent-device: agent-device · 11.2 sagent-device · 11.2 stester-army: tester.army · 17.7 stester.army · 17.7 s
CPU time the framework's own processes used during one timed run (warm median), from per-process deltas of ps time. Host means the framework's own processes.
Host memory, peakHost memory, peak: warden 158.6 MB; Argent 236.7 MB; agent-device 2.4 GB; tester.army 2.5 GB0953.7 MB1.9 GB2.8 GBHost memory, peakwarden: warden · 158.6 MBwarden · 158.6 MBargent: Argent · 236.7 MBArgent · 236.7 MBagent-device: agent-device · 2.4 GBagent-device · 2.4 GBtester-army: tester.army · 2.5 GBtester.army · 2.5 GB
Sampled every 500 ms while the suite runs. Host means the framework's own processes, summed per process, so shared pages can be counted twice.
Host CPU-seconds, memory and %cpu
MetricwardenArgentagent-devicetester.army
Host CPU-s / rep3.0 s4.1 s11.2 s17.7 s
Host memory avg145.9 MB181.4 MB711.9 MB598.2 MB
Host memory peak158.6 MB236.7 MB2.4 GB2.5 GB
Host %cpu avg3.0%2.7%13.6%13.8%
Host %cpu peak39%42%612%611%
Simulator %cpu avg74.3%44.5%86.0%84.1%
Simulator %cpu peak283%210%249%248%

Host CPU-s / rep: warm median of the timed run. %cpu is a decaying average, inflated by a short suite; compare CPU-seconds.

Where the CPU goes

The same CPU-seconds split by process name, from the warm-median rep. A name keeps its colour in every bar, the six biggest names are shown and the rest are folded into other. The sampler ticks every 500 ms, so a process that lives between two ticks only shows up through the exit charge booked when its parent reaps it.

Host processes (the framework's own tree)

  • xcodebuild
  • simulator-server
  • bun cli.ts
  • warden-simd
  • snapshot-bridge
  • node entry.js
  • other
  • warden2.9 s
  • Argent4.1 s
  • agent-device11.1 s
  • tester.army17.7 s
Table
warden
ProcessCPU-sShareRSS avg
warden-simd1.8 s60.0%56.5 MB
bun test1.2 s39.5%90.1 MB
bun test (exit charge)0.0 s0.5%0 B
bun cli.ts0.0 s0.0%0 B
simctl0.0 s0.0%0 B
Argent
ProcessCPU-sShareRSS avg
simulator-server3.9 s95.6%96.0 MB
node argent0.2 s4.3%85.7 MB
node argent (exit charge)0.0 s0.1%0 B
(simctl)0.0 s0.0%0 B
node tool-server.cjs0.0 s0.0%0 B
simctl0.0 s0.0%0 B
system_profiler0.0 s0.0%0 B
agent-device
ProcessCPU-sShareRSS avg
xcodebuild8.0 s71.9%219.6 MB
bun cli.ts1.9 s17.4%239.8 MB
snapshot-bridge0.8 s7.0%116.6 MB
ps0.2 s2.0%1.8 MB
node agent-device0.1 s1.3%61.1 MB
simctl0.0 s0.4%20.3 MB
node agent-device (exit charge)0.0 s0.1%0 B
DTServiceHub0.0 s0.0%16.4 MB
caffeinate0.0 s0.0%7.1 MB
(clang)0.0 s0.0%0 B
(ps)0.0 s0.0%0 B
(simctl)0.0 s0.0%0 B
other0.0 s0.0%0 B
tester.army
ProcessCPU-sShareRSS avg
xcodebuild14.7 s83.0%221.6 MB
node entry.js1.2 s6.7%85.2 MB
node e2e (exit charge)0.8 s4.3%0 B
snapshot-bridge0.6 s3.5%116.4 MB
node e2e0.4 s2.4%104.8 MB
other (exit charge)0.0 s0.2%0 B
simctl0.0 s0.0%20.5 MB
DTServiceHub0.0 s0.0%16.8 MB
(ps)0.0 s0.0%5.0 KB
(clang)0.0 s0.0%0 B
(pgrep)0.0 s0.0%0 B
(plutil)0.0 s0.0%0 B
other0.0 s0.0%0 B

Simulator processes

  • wardenexample
  • iOS
  • AgentDeviceRunnerUITests-Runner
  • bun cli.ts
  • SimulatorFrameworkBridge-iOS serve
  • CoreSimulatorBridge
  • other
  • warden65.8 s
  • Argent63.7 s
  • agent-device85.2 s
  • tester.army122.6 s
Table
warden
ProcessCPU-sShareRSS avg
wardenexample32.1 s48.8%373.1 MB
iOS25.4 s38.6%9.7 GB
SimulatorFrameworkBridge-iOS serve3.5 s5.4%185.6 MB
bun cli.ts2.3 s3.5%228.3 MB
CoreSimulatorBridge1.3 s1.9%84.9 MB
ps0.7 s1.0%1.8 MB
launchd_sim0.3 s0.5%36.8 MB
xpcproxy_sim0.2 s0.3%411.4 KB
pkd0.0 s0.0%58.9 MB
aslmanager0.0 s0.0%21.2 MB
caffeinate0.0 s0.0%7.3 MB
(launchctl)0.0 s0.0%0 B
other0.0 s0.0%0 B
Argent
ProcessCPU-sShareRSS avg
wardenexample33.7 s52.9%374.7 MB
iOS24.3 s38.2%9.7 GB
bun cli.ts3.3 s5.2%225.2 MB
ps1.1 s1.7%1.8 MB
CoreSimulatorBridge0.5 s0.8%117.8 MB
simctl0.4 s0.6%351.9 KB
launchd_sim0.2 s0.3%36.9 MB
xpcproxy_sim0.1 s0.2%155.7 KB
(wardenexample)0.0 s0.1%1.5 MB
pkd0.0 s0.0%59.0 MB
aslmanager0.0 s0.0%21.1 MB
caffeinate0.0 s0.0%7.2 MB
other0.0 s0.0%0 B
agent-device
ProcessCPU-sShareRSS avg
iOS37.6 s44.2%10.1 GB
wardenexample36.0 s42.3%375.2 MB
AgentDeviceRunnerUITests-Runner10.0 s11.7%269.7 MB
simctl0.6 s0.6%749.3 KB
CoreSimulatorBridge0.5 s0.6%81.6 MB
launchd_sim0.2 s0.2%37.0 MB
xpcproxy_sim0.2 s0.2%257.5 KB
launchctl0.1 s0.1%24.8 KB
pkd0.0 s0.0%59.1 MB
aslmanager0.0 s0.0%21.2 MB
(launchctl)0.0 s0.0%0 B
(managedappdistri)0.0 s0.0%0 B
other0.0 s0.0%0 B
tester.army
ProcessCPU-sShareRSS avg
wardenexample50.8 s41.4%381.6 MB
iOS50.7 s41.4%10.1 GB
AgentDeviceRunnerUITests-Runner16.4 s13.4%271.0 MB
bun cli.ts2.5 s2.0%264.5 MB
CoreSimulatorBridge0.6 s0.5%81.5 MB
simctl0.6 s0.5%563.5 KB
launchd_sim0.4 s0.3%37.7 MB
(launchctl)0.1 s0.1%62.1 KB
ps0.1 s0.1%1.8 MB
(managedappdistri)0.1 s0.1%68.6 KB
xpcproxy_sim0.1 s0.1%119.0 KB
launchctl0.1 s0.1%10.4 KB
other0.0 s0.0%87.2 MB

Residue and footprint

Residue is what is left on the machine after teardown. Footprint is how much suite source you maintain and how large the runner binaries are.

Footprint and residue
MetricwardenArgentagent-devicetester.army
Suite source (lines)379161158307
Suite source (size)17.5 KB7.1 KB5.4 KB13.8 KB
Runner binaries on disk7.5 MB77.8 MB6.9 MB—
Leftover processes0000
Leftover driver apps0011
Leftover directories0111

Versions and run details

Framework versions and status
FrameworkVersionStatus
wardenbun 1.3.14 + warden-e2eran, all tests passed
Argent0.30.1ran, all tests passed
agent-deviceagent-device 0.21.31ran, 97.5% of tests passed
tester.armye2e 0.20.0ran, all tests passed
Host
VirtualMac2,1, Apple M4 Pro (Virtual), 10 cores, 40.0 GB, macOS 26.5.2
Device
warden-iphone-17-1 iOS 26.5
App
nz.co.delacour.warden.example, 96.4 MB, build import:c283e, sha256 c283eae41c7a
Tools
bun 1.3.14, warden 0.3.1
Run
5 runs per framework, 0 retries, load average 2.87 at start
Report
docs/bench/ios-2026-10-25.md

Fairness rules

  • Same .app for every framework, with its sha256 recorded.
  • Zero LLM calls.
  • Same assertions per canonical test id.
  • Same retry helpers: dropped first tap, dropped deep link, system alerts.
  • The iOS "Open in …?" sheet is answered once per boot by the harness before every framework's prepare (iOS asks it for the first custom-scheme link after a boot, not again).
  • Fresh app launch per test, retries=0.
  • Frameworks run sequentially with daemons stopped between them, or, on EAS, each on its own worker and simulator.
  • caffeinate -dims held for the run, host otherwise idle, load average recorded.
  • Warm medians exclude rep 1.
  • Machine-readable reporters only.
  • A framework with any failing test never sets the pace, so it is never the fastest.

Caveats per framework

FrameworkGap
wardenCopied from the example app, retries written with warden's own options: tap({ until }), a deep-link retry, systemAlerts: "dismiss". The hand-rolled helpers it replaced ran the published headline numbers; the idiomatic run is reported beside them.
ArgentRetries are nested when: guards capped at two retaps. Flow selectors read only the app's view hierarchy, never SpringBoard, so the flow cannot answer the "Open in …?" sheet itself. A checkbox's checked state is not readable, so the remaining-count assertion covers the effect.
agent-device.ad scripts have no retry or conditional, only wait and press, so the deep-link script cannot answer a sheet that may or may not show. Needs a runner prepare step and a daemon. On the 2026-10-25 run its fill read back an empty field once (run 1, sign-in/field-errors), so it is not clean and never sets the pace.
tester.armyHas its own copy of the hand-rolled helpers warden used before. Runs with a private agent-device state dir and Node 24 on PATH.

Each run is a snapshot on one machine class with few warm repetitions: read the ratios as a guide, not a verdict. The full Markdown report for each run is in docs/bench/ in the repo; the EAS runs also publish every worker's artifact on their run page.

App builds

Install the native build matching your Expo fingerprint — from the device, cache, EAS or a local build.

Benchmark e2e frameworks

Run the same 8-test suite through warden, Argent, agent-device and tester.army on one device and compare the numbers.

On this page

Suite timePer testStep latencyCold start, prepare, teardownResourcesWhere the CPU goesResidue and footprintVersions and run detailsFairness rulesCaveats per framework