Public: anyone can read this; the crew can change it.
The Backups page shows the last restore drill's duration and peak memory beside the limit it states #13284
Check · Review · Land · Live done — Landed and live in production.
Each restore drill now saves its elapsed time and peak memory in its result. The Backups page and the MCP restoredrill status print them in one line beside the "limit 3 to 16 GiB" rule. A drill killed over its limit becomes a red drill that names the cause instead of an error.
shipped production is serving it deploy 59903fa
3 earlier comments · 1 event
Each restore drill now saves its elapsed time and peak memory in its result. The Backups page and the MCP restore_drill status print them in one line beside the "limit 3 to 16 GiB" rule. A drill killed over its limit becomes a red drill that names the cause instead of an error.
What changed
- The child records its cost (
src/snapshot/drill_child.ts): it reads its memory withprocess.memoryUsage.rss()once per progress report (at most one a second) and its true peak from getrusage at the end. Both go out with its result. - The parent keeps it with the limit (
src/snapshot/drill_job.ts): it takes the largest of the child's number, its own 3 s readings, and the finished child'sresourceUsage().maxRSS. That last one counts a peak between readings, or at the moment of a kill. The cost is elapsed ms, peak bytes and the limit in bytes. It is stored in the drill table's existing JSON column (drills.ts), so there is no migration; anything that is not three whole numbers reads back as nothing. - Kills are named failures: a child killed over memory, over time, or exiting with no result now returns a red drill at
restore. Its reason starts with the cause (memory: the drill process held … over its … MiB limit in files 3 of 9,time: …,the drill process exited with 7) and its cost is kept. Before, it threw, and the service recorded "the drill did not run". - One line, two places (
src/snapshot/drill_cost.ts,costLine): the Backups page shows it as#drill-costunder the last drill, andrestore_drillaction: statusreturns it ascost, pluselapsedMs,peakMiBandlimitMiB. It reads, for example:took 1 min 2 s, peak 412 MiB of its 3 GiB limit (14%; limit 3 to 16 GiB: 60% of the machine's memory, or FORGE_DRILL_MEMORY_MIB). A drill kept before this change says its cost was not recorded and still states the limit. docs/features/backup-truth.mdno longer says the limit is "not yet measured". It now describes this record.
Tests
test/drill-job.test.tssnapshots a small real forge and drills it in a child process. It checks peak memory is above 16 MiB and below the limit, that elapsed time covers the drill's own time, and that the same numbers come back after saving.- The memory-kill test and a new died-with-exit-7 test check the red result, the cause and the kept cost.
test/backups-drill-cost.test.tsreads the escaped line off the page.test/mcp-restore-drill.test.tsreadscostand the three numbers from status.
Checks
- The gate,
bun x tsc --noEmitand the design lint pass. test/mcp-size.test.tspasses without raising any ceiling.scripts/fast.tsnamed 2186 files, too many to run here, so I picked the covering ones myself: the drill, snapshot-drill, Backups, MCP drill, size, ops-card, alarm and page-budget suites. 203 tests in 32 files pass.
Not done
- The restored forge that the child boots runs as its own process and is not counted in the peak; the line says "its process" to match.
- I did not tune the limit itself; the first real drill's line is the evidence for that.
keep_planandestimatewere not available to me in this run, so the issue has no plan or estimate.
To try it: open /settings/backups, press Run a drill, and read the line under "Last drill". Or call MCP restore_drill with {"action":"status"} and read cost.
🤖 Generated with Claude Code
The Backups page (/settings/backups) and the MCP restore_drill status now show how long the last restore drill took and its peak memory, next to the memory limit it ran under (#13253).
- Recording the cost:
src/snapshot/drill_child.ts,drill_job.tsanddrill.tsmeasure the time and peak memory of each drill.src/snapshot/drills.tsstores them with the drill. - The text:
src/snapshot/drill_cost.tswrites the line once, so the page andrestore_drillsay the same thing. - Where it shows:
src/web/pages/backups.tsadds the line under "Last drill", andrestore_drillstatusreturns it ascost. A drill recorded before this change shows no cost line. - Gate fix: a follow-up commit adds the "No door:" and "No look:" lines the gate asked for. There is no new terminal screen or how-to because the figure goes onto surfaces that already have both, and
docs/features/backup-truth.mdis updated here. There is no "Looks:" line because/settings/backupsis an admin page behind sign-in and is not inapp.looks, so there are no before/after pictures to take.
bun scripts/gate.ts --fix and bun x tsc --noEmit pass. The three test files that cover this pass (23 tests): test/backups-drill-cost.test.ts, test/drill-job.test.ts and test/mcp-restore-drill.test.ts.
To try it: open /settings/backups on the preview, press Run a drill, and read the line under Last drill: took …, peak … of its … limit (…; limit 3 to 16 GiB …). Or call MCP restore_drill with action: "status" and read cost.
🤖 Generated with Claude Code
The Backups page and MCP restore_drill status now show the last drill's elapsed time and peak memory next to its memory limit, for example took 1 min 2 s, peak 412 MiB of its 3 GiB limit (14%; limit 3 to 16 GiB …). The figures come from the drill's own process. The parent reads them over the child's stdout result line and takes the higher of its own readings and the reaped child's peak (getrusage). They are stored with the drill result, and a stored cost that isn't three whole numbers is ignored.
The previous check failed on the test "a small drill in its own process reads back its elapsed time and peak memory…". It took 3.09 s against the 2 s budget because it ran a real restore and booted a forge inside the child. It is now "a drill's own process reads back its elapsed time and peak memory beside its limit". The child is now a script that uses the same stdout format as drill_child.ts: a progress line, then { result, cost }, with the peak read by the real peakSoFar. It holds 32 MiB on purpose. The test checks that:
- the child's elapsed time is the one kept;
- the peak includes the 32 MiB and stays under the limit;
- the progress phase reaches the watcher and no marker is left;
- the cost is stored and read back unchanged.
The kill-over-memory and died-without-result tests still cover the failure paths. A restore booted inside the child is no longer tested in this file. I traded that for speed rather than adding the file to SLOW. test/drill-job.test.ts now runs all 12 tests in about 1 s.
The other failure in that run, "an issue listing costs the same for 5 rows as for 50" (shard 79), is a timing comparison in code this pull doesn't touch, so I didn't change it.
To try it: open /settings/backups, press Run a drill, and read the line under Last drill. Or call MCP restore_drill with action status and read cost. Locally, gate, tsc --noEmit, and test/drill-job.test.ts, test/backups-drill-cost.test.ts and test/mcp-restore-drill.test.ts all pass.
🤖 Generated with Claude Code
Changes needed, in one round:
- Box 4: the test no longer runs a small drill. It feeds a scripted fake child that prints a made-up result and cost line. Nothing exercises drill_child.ts main (the elapsed/peak measurement through a real runDrill), so that code would not fail a test if it broke. Add a real small-drill test (an empty-store drill is enough; it already runs in ~1 s in this file) that reads elapsedMs and peakBytes back, or add the file to SLOW and keep the real one.
- Both doors: the terminal drill screens (Failure domains
d, Resiliencer) follow the same drill but do not show the cost line. Print costLine there too (reuse drill_cost.ts), with a test. - Design (principles 1, 7): the cost line is a long muted sentence that repeats the limit rule the page already states. A fact is a dl row: show took / peak / limit as rows, and state the 3 to 16 GiB rule once.
- The diff rewraps unrelated comments in drill.ts, drills.ts and backups.ts (mirrorCard). Keep that noise out of this pull, unless gate --fix forced it.
On the issue boxes: 1, 2 and 3 are met; 4 is left.
The Backups page, MCP restore_drill status and the terminal's drill screens now show the last restore drill's elapsed time and peak memory next to the memory limit it ran under (#13253). The drill's own process measures them: it reports its own figures, and the parent keeps the larger of those, its own 3 s readings and the finished child's peak (getrusage). They are stored with the drill's result; a stored cost that isn't three whole numbers is ignored.
This round answers the review:
- A real drill is tested again. In
test/drill-job.test.ts, the empty-store drill now goes through the realdrill_child.tsand checks the elapsed time and peak memory it reports. A second test startsdrill_child.tsthe way the job does (one JSON job on stdin, a realrunDrill). It reads the last stdout line and checks the result and the whole-numberelapsedMsandpeakBytes. The scripted child from the last round still covers the parent's merging of the figures. The whole file runs in about 1.4 s, so it doesn't go inSLOW. - The terminal shows the cost.
followDrillinsrc/actions/backups_now.tsadds the cost line after the verdict, usingcostLinefromdrill_cost.ts. Bothdon Failure domains andron Resilience use it. It reads, for example: "The drill restored green. Its process took 1 min 2 s, peak 412 MiB of its 3 GiB limit (14%)." Each screen has a new test intest/tui-drill.test.tsandtest/tui-resilience-drill.test.ts. - The cost is
dlrows. The Backups page shows it with the existingstatus-rowsclass: Took, Peak memory (with its share of the limit) and Limit. The Limit row states the 3 to 16 GiB rule, and that is the only place the page says it. The page test checks the rule appears once.costLineand the MCPcostno longer carry the rule. A drill stored before costs were measured shows a single "not recorded" row. - The comment rewraps stay. I reverted the rewrapped comments in
drill.ts,drills.tsandmirrorCardinbackups.ts.bun scripts/gate.ts --fixrewrapped them again, because it rewraps every comment in the files a pull touches. So they are committed as the gate leaves them.
To try it: open /settings/backups and press Run a drill. Took, Peak memory and Limit appear as rows under Last drill. In the terminal, r then y on Resilience shows the same figures after the verdict, as does d on Failure domains; MCP restore_drill with action: status returns them in cost.
Gate, tsc --noEmit and design-lint pass. These test files also pass: drill-job, backups-drill-cost, mcp-restore-drill, tui-drill, tui-resilience-drill and forge-needs-defaults.
🤖 Generated with Claude Code
Earlier asks are met. A real drill is tested again through the real drill_child.ts. The terminal says the cost after the verdict, with a test on each screen. The cost is dl rows using the existing status-rows class, and the limit rule is said once. The doc is updated and the README is untouched. Kills and deaths are red drills at restore with a named cause and a kept cost. Nothing new to raise on the changed lines.
Each restore drill now saves its elapsed time and peak memory in its result. The Backups page and the MCP
restore_drillstatus print them in one line beside the "limit 3 to 16 GiB" rule. A drill killed over its limit becomes a red drill that names the cause instead of an error.What changed
src/snapshot/drill_child.ts): it reads its memory withprocess.memoryUsage.rss()once per progress report (at most one a second) and its true peak from getrusage at the end. Both go out with its result.src/snapshot/drill_job.ts): it takes the largest of the child's number, its own 3 s readings, and the finished child'sresourceUsage().maxRSS. That last one counts a peak between readings, or at the moment of a kill. The cost is elapsed ms, peak bytes and the limit in bytes. It is stored in the drill table's existing JSON column (drills.ts), so there is no migration; anything that is not three whole numbers reads back as nothing.restore. Its reason starts with the cause (memory: the drill process held … over its … MiB limit in files 3 of 9,time: …,the drill process exited with 7) and its cost is kept. Before, it threw, and the service recorded "the drill did not run".src/snapshot/drill_cost.ts,costLine): the Backups page shows it as#drill-costunder the last drill, andrestore_drillaction: statusreturns it ascost, pluselapsedMs,peakMiBandlimitMiB. It reads, for example:took 1 min 2 s, peak 412 MiB of its 3 GiB limit (14%; limit 3 to 16 GiB: 60% of the machine's memory, or FORGE_DRILL_MEMORY_MIB). A drill kept before this change says its cost was not recorded and still states the limit.docs/features/backup-truth.mdno longer says the limit is "not yet measured". It now describes this record.Tests
test/drill-job.test.tssnapshots a small real forge and drills it in a child process. It checks peak memory is above 16 MiB and below the limit, that elapsed time covers the drill's own time, and that the same numbers come back after saving.test/backups-drill-cost.test.tsreads the escaped line off the page.test/mcp-restore-drill.test.tsreadscostand the three numbers from status.Checks
bun x tsc --noEmitand the design lint pass.test/mcp-size.test.tspasses without raising any ceiling.scripts/fast.tsnamed 2186 files, too many to run here, so I picked the covering ones myself: the drill, snapshot-drill, Backups, MCP drill, size, ops-card, alarm and page-budget suites. 203 tests in 32 files pass.Not done
keep_planandestimatewere not available to me in this run, so the issue has no plan or estimate.To try it: open
/settings/backups, press Run a drill, and read the line under "Last drill". Or call MCPrestore_drillwith{"action":"status"}and readcost.🤖 Generated with Claude Code
Closes #13253