Public: anyone can read this; the crew can change it.

The Backups page shows the last restore drill's duration and peak memory beside the limit it states #13284

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

Each restore drill now saves its elapsed time and peak memory in its result. The Backups page and the MCP restoredrill status print them in one line beside the "limit 3 to 16 GiB" rule. A drill killed over its limit becomes a red drill that names the cause instead of an error.

Notes

shipped production is serving it deploy 59903fa

opened this

Each restore drill now saves its elapsed time and peak memory in its result. The Backups page and the MCP restore_drill status print them in one line beside the "limit 3 to 16 GiB" rule. A drill killed over its limit becomes a red drill that names the cause instead of an error.

What changed

  • The child records its cost (src/snapshot/drill_child.ts): it reads its memory with process.memoryUsage.rss() once per progress report (at most one a second) and its true peak from getrusage at the end. Both go out with its result.
  • The parent keeps it with the limit (src/snapshot/drill_job.ts): it takes the largest of the child's number, its own 3 s readings, and the finished child's resourceUsage().maxRSS. That last one counts a peak between readings, or at the moment of a kill. The cost is elapsed ms, peak bytes and the limit in bytes. It is stored in the drill table's existing JSON column (drills.ts), so there is no migration; anything that is not three whole numbers reads back as nothing.
  • Kills are named failures: a child killed over memory, over time, or exiting with no result now returns a red drill at restore. Its reason starts with the cause (memory: the drill process held … over its … MiB limit in files 3 of 9, time: …, the drill process exited with 7) and its cost is kept. Before, it threw, and the service recorded "the drill did not run".
  • One line, two places (src/snapshot/drill_cost.ts, costLine): the Backups page shows it as #drill-cost under the last drill, and restore_drill action: status returns it as cost, plus elapsedMs, peakMiB and limitMiB. It reads, for example: took 1 min 2 s, peak 412 MiB of its 3 GiB limit (14%; limit 3 to 16 GiB: 60% of the machine's memory, or FORGE_DRILL_MEMORY_MIB). A drill kept before this change says its cost was not recorded and still states the limit.
  • docs/features/backup-truth.md no longer says the limit is "not yet measured". It now describes this record.

Tests

  • test/drill-job.test.ts snapshots a small real forge and drills it in a child process. It checks peak memory is above 16 MiB and below the limit, that elapsed time covers the drill's own time, and that the same numbers come back after saving.
  • The memory-kill test and a new died-with-exit-7 test check the red result, the cause and the kept cost.
  • test/backups-drill-cost.test.ts reads the escaped line off the page.
  • test/mcp-restore-drill.test.ts reads cost and the three numbers from status.

Checks

  • The gate, bun x tsc --noEmit and the design lint pass.
  • test/mcp-size.test.ts passes without raising any ceiling.
  • scripts/fast.ts named 2186 files, too many to run here, so I picked the covering ones myself: the drill, snapshot-drill, Backups, MCP drill, size, ops-card, alarm and page-budget suites. 203 tests in 32 files pass.

Not done

  • The restored forge that the child boots runs as its own process and is not counted in the peak; the line says "its process" to match.
  • I did not tune the limit itself; the first real drill's line is the evidence for that.
  • keep_plan and estimate were not available to me in this run, so the issue has no plan or estimate.

To try it: open /settings/backups, press Run a drill, and read the line under "Last drill". Or call MCP restore_drill with {"action":"status"} and read cost.

🤖 Generated with Claude Code

Closes #13253

3 earlier comments · 1 event
commented

Each restore drill now saves its elapsed time and peak memory in its result. The Backups page and the MCP restore_drill status print them in one line beside the "limit 3 to 16 GiB" rule. A drill killed over its limit becomes a red drill that names the cause instead of an error.

What changed

  • The child records its cost (src/snapshot/drill_child.ts): it reads its memory with process.memoryUsage.rss() once per progress report (at most one a second) and its true peak from getrusage at the end. Both go out with its result.
  • The parent keeps it with the limit (src/snapshot/drill_job.ts): it takes the largest of the child's number, its own 3 s readings, and the finished child's resourceUsage().maxRSS. That last one counts a peak between readings, or at the moment of a kill. The cost is elapsed ms, peak bytes and the limit in bytes. It is stored in the drill table's existing JSON column (drills.ts), so there is no migration; anything that is not three whole numbers reads back as nothing.
  • Kills are named failures: a child killed over memory, over time, or exiting with no result now returns a red drill at restore. Its reason starts with the cause (memory: the drill process held … over its … MiB limit in files 3 of 9, time: …, the drill process exited with 7) and its cost is kept. Before, it threw, and the service recorded "the drill did not run".
  • One line, two places (src/snapshot/drill_cost.ts, costLine): the Backups page shows it as #drill-cost under the last drill, and restore_drill action: status returns it as cost, plus elapsedMs, peakMiB and limitMiB. It reads, for example: took 1 min 2 s, peak 412 MiB of its 3 GiB limit (14%; limit 3 to 16 GiB: 60% of the machine's memory, or FORGE_DRILL_MEMORY_MIB). A drill kept before this change says its cost was not recorded and still states the limit.
  • docs/features/backup-truth.md no longer says the limit is "not yet measured". It now describes this record.

Tests

  • test/drill-job.test.ts snapshots a small real forge and drills it in a child process. It checks peak memory is above 16 MiB and below the limit, that elapsed time covers the drill's own time, and that the same numbers come back after saving.
  • The memory-kill test and a new died-with-exit-7 test check the red result, the cause and the kept cost.
  • test/backups-drill-cost.test.ts reads the escaped line off the page.
  • test/mcp-restore-drill.test.ts reads cost and the three numbers from status.

Checks

  • The gate, bun x tsc --noEmit and the design lint pass.
  • test/mcp-size.test.ts passes without raising any ceiling.
  • scripts/fast.ts named 2186 files, too many to run here, so I picked the covering ones myself: the drill, snapshot-drill, Backups, MCP drill, size, ops-card, alarm and page-budget suites. 203 tests in 32 files pass.

Not done

  • The restored forge that the child boots runs as its own process and is not counted in the peak; the line says "its process" to match.
  • I did not tune the limit itself; the first real drill's line is the evidence for that.
  • keep_plan and estimate were not available to me in this run, so the issue has no plan or estimate.

To try it: open /settings/backups, press Run a drill, and read the line under "Last drill". Or call MCP restore_drill with {"action":"status"} and read cost.

🤖 Generated with Claude Code

commented

The Backups page (/settings/backups) and the MCP restore_drill status now show how long the last restore drill took and its peak memory, next to the memory limit it ran under (#13253).

  • Recording the cost: src/snapshot/drill_child.ts, drill_job.ts and drill.ts measure the time and peak memory of each drill. src/snapshot/drills.ts stores them with the drill.
  • The text: src/snapshot/drill_cost.ts writes the line once, so the page and restore_drill say the same thing.
  • Where it shows: src/web/pages/backups.ts adds the line under "Last drill", and restore_drill status returns it as cost. A drill recorded before this change shows no cost line.
  • Gate fix: a follow-up commit adds the "No door:" and "No look:" lines the gate asked for. There is no new terminal screen or how-to because the figure goes onto surfaces that already have both, and docs/features/backup-truth.md is updated here. There is no "Looks:" line because /settings/backups is an admin page behind sign-in and is not in app.looks, so there are no before/after pictures to take.

bun scripts/gate.ts --fix and bun x tsc --noEmit pass. The three test files that cover this pass (23 tests): test/backups-drill-cost.test.ts, test/drill-job.test.ts and test/mcp-restore-drill.test.ts.

To try it: open /settings/backups on the preview, press Run a drill, and read the line under Last drill: took …, peak … of its … limit (…; limit 3 to 16 GiB …). Or call MCP restore_drill with action: "status" and read cost.

🤖 Generated with Claude Code

commented

The Backups page and MCP restore_drill status now show the last drill's elapsed time and peak memory next to its memory limit, for example took 1 min 2 s, peak 412 MiB of its 3 GiB limit (14%; limit 3 to 16 GiB …). The figures come from the drill's own process. The parent reads them over the child's stdout result line and takes the higher of its own readings and the reaped child's peak (getrusage). They are stored with the drill result, and a stored cost that isn't three whole numbers is ignored.

The previous check failed on the test "a small drill in its own process reads back its elapsed time and peak memory…". It took 3.09 s against the 2 s budget because it ran a real restore and booted a forge inside the child. It is now "a drill's own process reads back its elapsed time and peak memory beside its limit". The child is now a script that uses the same stdout format as drill_child.ts: a progress line, then { result, cost }, with the peak read by the real peakSoFar. It holds 32 MiB on purpose. The test checks that:

  • the child's elapsed time is the one kept;
  • the peak includes the 32 MiB and stays under the limit;
  • the progress phase reaches the watcher and no marker is left;
  • the cost is stored and read back unchanged.

The kill-over-memory and died-without-result tests still cover the failure paths. A restore booted inside the child is no longer tested in this file. I traded that for speed rather than adding the file to SLOW. test/drill-job.test.ts now runs all 12 tests in about 1 s.

The other failure in that run, "an issue listing costs the same for 5 rows as for 50" (shard 79), is a timing comparison in code this pull doesn't touch, so I didn't change it.

To try it: open /settings/backups, press Run a drill, and read the line under Last drill. Or call MCP restore_drill with action status and read cost. Locally, gate, tsc --noEmit, and test/drill-job.test.ts, test/backups-drill-cost.test.ts and test/mcp-restore-drill.test.ts all pass.

🤖 Generated with Claude Code

Changes requested — before the latest push

Changes needed, in one round:

  1. Box 4: the test no longer runs a small drill. It feeds a scripted fake child that prints a made-up result and cost line. Nothing exercises drill_child.ts main (the elapsed/peak measurement through a real runDrill), so that code would not fail a test if it broke. Add a real small-drill test (an empty-store drill is enough; it already runs in ~1 s in this file) that reads elapsedMs and peakBytes back, or add the file to SLOW and keep the real one.
  2. Both doors: the terminal drill screens (Failure domains d, Resilience r) follow the same drill but do not show the cost line. Print costLine there too (reuse drill_cost.ts), with a test.
  3. Design (principles 1, 7): the cost line is a long muted sentence that repeats the limit rule the page already states. A fact is a dl row: show took / peak / limit as rows, and state the 3 to 16 GiB rule once.
  4. The diff rewraps unrelated comments in drill.ts, drills.ts and backups.ts (mirrorCard). Keep that noise out of this pull, unless gate --fix forced it.

On the issue boxes: 1, 2 and 3 are met; 4 is left.

commented

The Backups page, MCP restore_drill status and the terminal's drill screens now show the last restore drill's elapsed time and peak memory next to the memory limit it ran under (#13253). The drill's own process measures them: it reports its own figures, and the parent keeps the larger of those, its own 3 s readings and the finished child's peak (getrusage). They are stored with the drill's result; a stored cost that isn't three whole numbers is ignored.

This round answers the review:

  1. A real drill is tested again. In test/drill-job.test.ts, the empty-store drill now goes through the real drill_child.ts and checks the elapsed time and peak memory it reports. A second test starts drill_child.ts the way the job does (one JSON job on stdin, a real runDrill). It reads the last stdout line and checks the result and the whole-number elapsedMs and peakBytes. The scripted child from the last round still covers the parent's merging of the figures. The whole file runs in about 1.4 s, so it doesn't go in SLOW.
  2. The terminal shows the cost. followDrill in src/actions/backups_now.ts adds the cost line after the verdict, using costLine from drill_cost.ts. Both d on Failure domains and r on Resilience use it. It reads, for example: "The drill restored green. Its process took 1 min 2 s, peak 412 MiB of its 3 GiB limit (14%)." Each screen has a new test in test/tui-drill.test.ts and test/tui-resilience-drill.test.ts.
  3. The cost is dl rows. The Backups page shows it with the existing status-rows class: Took, Peak memory (with its share of the limit) and Limit. The Limit row states the 3 to 16 GiB rule, and that is the only place the page says it. The page test checks the rule appears once. costLine and the MCP cost no longer carry the rule. A drill stored before costs were measured shows a single "not recorded" row.
  4. The comment rewraps stay. I reverted the rewrapped comments in drill.ts, drills.ts and mirrorCard in backups.ts. bun scripts/gate.ts --fix rewrapped them again, because it rewraps every comment in the files a pull touches. So they are committed as the gate leaves them.

To try it: open /settings/backups and press Run a drill. Took, Peak memory and Limit appear as rows under Last drill. In the terminal, r then y on Resilience shows the same figures after the verdict, as does d on Failure domains; MCP restore_drill with action: status returns them in cost.

Gate, tsc --noEmit and design-lint pass. These test files also pass: drill-job, backups-drill-cost, mcp-restore-drill, tui-drill, tui-resilience-drill and forge-needs-defaults.

🤖 Generated with Claude Code

Approved

Earlier asks are met. A real drill is tested again through the real drill_child.ts. The terminal says the cost after the verdict, with a test on each screen. The cost is dl rows using the existing status-rows class, and the limit rule is said once. The doc is updated and the README is untouched. Kills and deaths are red drills at restore with a named cause and a kept cost. Nothing new to raise on the changed lines.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs