e2e framework benchmark — ios 2026-10-25
Run eas-ios-argent, started 2026-10-25T23:39:57.034Z
Fairness rules
- Same
.appfor every framework (sha256 recorded above) - Zero LLM calls
- Same assertions per canonical test id
- Same retry helpers (dropped first tap, dropped deep link, system alerts)
- Fresh app launch per test
retries=0- Frameworks run sequentially
caffeinate -dimsheld for the run- Warm medians exclude rep 1 (cold)
- Machine-readable reporters only
- Host otherwise idle; load average recorded
- Android
simcounts the app and every framework's on-device helpers (uiautomator, instrumentation runners, agent-device, Argent, warden-uiad), matched case-insensitively - Host processes are tracked by pid and start time, so a reused pid is never counted
- Android animation scales are reset to the device's original values before every framework and restored at the end
Summary
| Framework | Status | Version | Suite median (warm) | p95 | Ratio | Cold rep 1 | Cold-start cost | Prepare | Teardown | Overhead | Pass rate | Flake | Residue | Footprint (LOC / deps) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| argent | ✓ ok | 0.30.1 | 134.32 s | 134.65 s | 1.00× | 133.61 s | 1.01 s | 1.72 s | 293 ms | 163 ms | 100% | 0% | 0 proc / 0 app / 1 path | 161 LOC / 0 B |
Warm suite wall time
- argent
Bar = warm median. Whisker = p95. Dot = cold rep 1. Lower is faster; ratios are against the fastest framework.
Per-test medians (ms)
| Test | argent |
|---|---|
| onboarding steps through to sign-in | 8301 (1.00×) |
| invalid email and short password show field errors | 18221 (1.00×) |
| a locked account shows the form alert | 14442 (1.00×) |
| valid credentials land on todos | 15416 (1.00×) |
| add, complete and filter todos | 21110 (1.00×) |
| open a todo, mark it done, then delete it | 19105 (1.00×) |
| deep link opens a todo by id | 17811 (1.00×) |
| settings shows the account, toggles switches and signs out | 19616 (1.00×) |
Median per test
- argent
Spread across warm reps
Whisker = min to max. Shaded box = min to p95. Tick = median. A distribution carries no quartiles, so the box is not an interquartile range.
Per-step latency
- argent
Whisker = min to max. Shaded box = min to p95. Tick = median. Only frameworks that report steps appear.
| Framework | Steps | Median | p95 | Max |
|---|---|---|---|---|
| argent | 1192 | 321 ms | 1.82 s | 2.82 s |
Where step time goes
Step scopes differ per tool: warden's one fill step spans the focus tap, select-all, typing and read-back confirm, argent reports a separate idle step and into …, agent-device reports fill and type separately, and tester.army reports locator steps. Compare a verb's share within a tool, not its median across tools.
argent
| Verb | n | Share | Median | p95 | Max |
|---|---|---|---|---|---|
| into | 68 | 27.9% | 2.15 s | 2.81 s | 2.82 s |
| idle | 232 | 27.5% | 925 ms | 1.04 s | 1.24 s |
| visible | 320 | 18.3% | 13 ms | 1.03 s | 1.04 s |
| id= | 380 | 12.8% | 322 ms | 352 ms | 372 ms |
| launch | 32 | 10.9% | 1.80 s | 1.82 s | 1.83 s |
| hidden | 32 | 2.4% | 16 ms | 1.05 s | 1.05 s |
| open-url | 4 | 0.1% | 116 ms | 124 ms | 124 ms |
| next-test.yaml | 28 | 0% | 4 ms | 6 ms | 6 ms |
| other | 96 | 0% | 1 ms | 5 ms | 6 ms |
Resources
- argent
CPU
Memory
| Framework | Scope | CPU-s / rep | CPU avg | CPU peak | RSS avg | RSS peak |
|---|---|---|---|---|---|---|
| argent | host | 4.13 s | 2.7% | 41.8% | 181.4 MB | 236.7 MB |
| argent | sim | — | 44.5% | 209.9% | 10.5 GB | 10.6 GB |
CPU-s / rep is the host CPU time of the timed run (warm median) and the headline. CPU avg and peak are ps %cpu, a decaying average on macOS: they move with suite length, do not judge on them.
Where the CPU goes
One table per framework, from the warm-median rep's timed run (the rep whose host CPU-seconds is the lower median). Rows are processes grouped by label (top 12 per scope, the rest in other); host is the framework's own process tree, sim the simulator's launchd_sim tree (iOS only). Share is of the scope's CPU-seconds. RSS is the label's summed RSS per 500 ms tick (0 where it had no process). Spawns counts distinct pids a tick saw: a process that lives between two ticks is only seen through the (exit charge) rows (CPU of a child the bench ran, booked at its exit), so many short-lived xcrun simctl calls read low in Spawns and in CPU.
argent
| Process | Scope | CPU-s | Share | RSS avg | RSS peak | Spawns |
|---|---|---|---|---|---|---|
| simulator-server | host | 3.92 s | 95.6% | 96.0 MB | 96.0 MB | 1 |
| node argent | host | 0.18 s | 4.3% | 85.7 MB | 88.1 MB | 1 |
| node argent (exit charge) | host | 0.00 s | 0.1% | 0 B | 0 B | 0 |
| (simctl) | host | 0.00 s | 0.0% | 0 B | 0 B | 0 |
| node tool-server.cjs | host | 0.00 s | 0.0% | 0 B | 0 B | 0 |
| simctl | host | 0.00 s | 0.0% | 0 B | 0 B | 0 |
| system_profiler | host | 0.00 s | 0.0% | 0 B | 0 B | 0 |
| wardenexample | sim | 33.67 s | 52.9% | 374.7 MB | 434.8 MB | 8 |
| iOS | sim | 24.31 s | 38.2% | 9.7 GB | 9.7 GB | 106 |
| bun cli.ts | sim | 3.34 s | 5.2% | 225.2 MB | 229.3 MB | 1 |
| ps | sim | 1.07 s | 1.7% | 1.8 MB | 1.8 MB | 267 |
| CoreSimulatorBridge | sim | 0.48 s | 0.8% | 117.8 MB | 118.2 MB | 1 |
| simctl | sim | 0.41 s | 0.6% | 351.9 KB | 20.5 MB | 5 |
| launchd_sim | sim | 0.21 s | 0.3% | 36.9 MB | 36.9 MB | 1 |
| xpcproxy_sim | sim | 0.15 s | 0.2% | 155.7 KB | 12.1 MB | 4 |
| (wardenexample) | sim | 0.04 s | 0.1% | 1.5 MB | 395.3 MB | 1 |
| pkd | sim | 0.00 s | 0.0% | 59.0 MB | 59.0 MB | 1 |
| aslmanager | sim | 0.00 s | 0.0% | 21.1 MB | 21.1 MB | 1 |
| caffeinate | sim | 0.00 s | 0.0% | 7.2 MB | 7.2 MB | 1 |
| other | sim | 0.00 s | 0.0% | 0 B | 0 B | 0 |
Residue
- argent: paths: /Users/expo/.argent (328.0 KB)
Unsupported / failed
None.