Public: anyone can read this; the crew can change it.

The restore drill: cut by the edge, buffered in the serving process, lost on a restart, and silent while it runs #12600

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

The restore drill is now a job that runs in its own process, so it no longer holds the snapshot in the serving process or ends in a 524. The gate, bun x tsc --noEmit and the drill, MCP-size and Backups-page tests pass here. Nothing has run against a real store, and the forge's check is still to come on this push.

Notes

shipped production is serving it deploy 5bb6da3

opened this

The restore drill is now a job that runs in its own process, so it no longer holds the snapshot in the serving process or ends in a 524. The gate, bun x tsc --noEmit and the drill, MCP-size and Backups-page tests pass here. Nothing has run against a real store, and the forge's check is still to come on this push. The sandbox skips the unix-socket suites.

What changed

  • Job and child process. MCP restore_drill takes action: start (the default) or status. start answers at once with an id, the phase, the bytes and the memory limit. A second ask while one runs gets started: false and the running drill's id. status answers the running drill's phase and bytes, or the verdict once it has ended. No tool was added, so mcp-size is unchanged.
  • Memory bound. The drill runs in src/snapshot/drill_child.ts, started by drill_job.ts with --smol, with its own new session. The job reads the child's resident memory every 3 s and kills it, the drill red, above 3 GiB. The limit is shown on the Backups page. This limits the child, not the restore itself: I did not change how restore.ts buffers, so a large snapshot may well reach the 3 GiB cap.
  • Progress. The Backups page shows phase, done of total, bytes and memory while a drill runs. The log gets a restore drill working line every 15 s and lines at start and end.
  • Restart. A marker file, drill/running.json, is left while a drill runs. At boot, a marker left behind is recorded as a red drill "cut by a restart", and a drill is started again after the 5-minute catch-up delay. Every drill/restore-* directory is swept at boot, and again when any drill ends, passed or not.
  • Status. It now reads "restore drill 2026-11-08 passed (6 d ago)". It goes amber when the newest snapshot is more than 3 days newer than the drill. A snapshot whose format differs from the last drilled one starts a drill on its own, once the last drill has recorded a format (older rows carry none).
  • Backups button. The page button now starts the job and answers at once, so it no longer waits out the edge.
  • Tests and docs. New test/drill-job.test.ts covers the sweep, the child, the memory kill, the status age and a restart mid-drill. I changed mcp-restore-drill and snapshot-drill-page because they assumed the call waits for the verdict. The restore_drill row in docs/MCP.md and docs/features/backup-truth.md describe the job.

Not done

  • The resilience screen, the TUI and the needs-you action still call drill() and wait for the verdict. Only the MCP tool and the web button are non-blocking.
  • No MCP end-to-end run of the done-when through the public address, and no before/after memory reading on production. The commit carries No look: since the Backups page changes only while a drill runs.

Try it on the preview: call restore_drill with action: start, then action: status, and open /settings/backups to watch it.

🤖 Generated with Claude Code

Closes #12592

Show 2 earlier items

7 earlier comments · 1 event
commented

This push fixes the seven test files that were red because of this pull. bun x tsc --noEmit is clean. The nine test files that cover the change pass locally, and bun scripts/gate.ts printed nothing. I did not run the whole suite; the forge's check does that on this push.

Four tests were red because the repository's rule lists didn't yet know about the new src/snapshot/drill_job.ts:

  • Spawn lists: I added drill_job.ts to test/spawn-pipes.test.ts (it pipes stdout and stderr) and to test/spawn-sites.test.ts (its argv is only the bun path and a fixed script).
  • Timers: the watcher setInterval now runs inside during("drill-watch", …), and the 5-minute-plus-budget kill timer is on the KILL allowlist in test/timers-named.test.ts.
  • Unused exports: markerLeft is only used by tests, so I added it to scripts/unused-exports-allow.json rather than removing it.
  • MCP table: I regenerated the restore_drill row in docs/MCP.md with bun scripts/mcp-table.ts.

The other three failures were tests that assumed the old synchronous drill:

  • Poll loops: the two in test/mcp-restore-drill.test.ts now go through one settled() helper that throws if the drill is still running after about 2 s. The flake rule flagged the old loops because they carried on after timing out.
  • Card test: restore_drill with no arguments now only starts the job, so the card test in test/mcp-ops-tool-cards.test.ts starts a drill and polls status until the card appears.

The five suites the check log marked as red on main too (release-handover-keepalive, apps-ws, apps-open-host, ask-to-reach, check-install) are not touched by this change. I did not run them.

commented

The one failure from this pull is fixed and pushed. test/job-slices.test.ts requires every during job to be either timed or listed in UNTIMED with a reason. The new drill-watch job was in neither. I added it to UNTIMED, with the reason that it is a memory read and a state write that run only while a restore drill is going.

test/job-slices.test.ts and test/timers-named.test.ts pass locally (11 tests) and bun x tsc --noEmit is clean. I did not run the whole suite.

The forge's log marks five other test files as red on main too: apps-ws, apps-open-host, ask-to-reach, check-install and release-handover-keepalive. I did not touch them or run them. The log is cut off, so I can't see the rest of the 11 failures. The only one it blames on this pull is job-slices.

commented

The restore drill now runs as a job in its own process, started at once and one at a time. A cut drill is recorded and swept, progress is shown, and the age of the last proof is said. This push only merges main into the pull, so the drill code is unchanged from before.

  • Merge: the merge conflicted in test/job-slices.test.ts, where both drill-watch (this pull) and desk (main) were added to the list of untimed jobs. I kept both entries.
  • Failures not from this pull: test/module-state.test.ts and test/apps-open-host.test.ts fail locally as they do on main. The first fails on src/crew/ceiling_stagger.ts:memo, a file this pull doesn't touch. The second fails on a "proxy reads such a Host" case that throws Invalid URL.
  • Not reproduced: the check said test/migrate.test.ts is green on main and red on this pull, but it passes here, before and after the merge. I'm treating that as flaky unless the new check shows it red again.
  • Checks run: bun scripts/gate.ts and bun x tsc --noEmit printed nothing. The tests for the drill, MCP drill tools, drill page, job-slices, timers-named, spawn-sites and spawn-pipes pass, apart from the apps-open-host failure above.
  • Not seen: the check log was cut off, so I saw only 2 of the 15 failures named. The other 13 I didn't see or re-run.
commented

I can't find anything in this pull that causes the failures, and I couldn't reproduce the one failure the check blames on it. The push only merges the latest main (which brings in a change to test/data-query-limits.test.ts); no drill code changed.

  • Failures red on main too: the install cache, release keep-alive, /apps/open host, module-state and ask-to-reach tests fail on main as well, so the check doesn't count them against this pull. I didn't run the install cache and keep-alive tests locally. The ask-to-reach failures reproduce here. They try to reach maria--forge.test and otto--forge.test, which this sandbox blocks and the repo's egress list doesn't allow.
  • The two migrate failures in shard 73: "a name that resolves to a private address is refused" and "a vetted host is pinned" are the ones the check says are red only on this pull. test/migrate.test.ts passes here, including when run in the same process as test/ask-to-reach.test.ts, so I can't reproduce them. The likely cause is the shard-73 ask-to-reach failures disturbing the migrate tests. I didn't change them or any egress code.
  • Slow tests: the 50 s crew-proposal case and the other over-budget ones look like a loaded shard. test/crew-proposal.test.ts passes locally with the drill tests.
  • Checks run: bun scripts/gate.ts and bun x tsc --noEmit printed nothing. drill-job, job-slices, mcp-restore-drill and crew-proposal pass.

If the migrate pair is red again after this push, it needs a look at what ask-to-reach leaves behind in a shared shard.

commented

I merged main into the branch and pushed it. The only conflict was in src/snapshot/service.ts, at the boot catch-up of the restore drill.

main had added an app argument to the during(...) call there, and this pull had changed the same line so the drill can start with drill() when the previous one was cut. I kept both, so the call still starts the drill directly after a cut one and now passes undefined, app as well.

bun x tsc --noEmit and bun scripts/gate.ts print nothing. The drill, MCP restore-drill, drill page, job-slices and all test/snapshot*.test.ts tests pass (181 pass, 0 fail). The full check will run on the push.

commented

I found no fault in this pull behind the 11 failures and changed no code. The push only merges the newest main into the branch. The check blamed this pull for two tests, and both pass here.

  • Blamed on this pull: test/boundary-tokens-follow.test.ts passes (10 tests) on the merged tree. The check says it is green on main, but this pull doesn't touch tokens or boundaries. test/quotas-running.test.ts, which holds "a fresh account's second run waits queued", also passes here (7 tests). I think both failed because the shard was loaded: the log shows several tests running over 2 s.
  • Red on main too: /apps/open host, release keep-alive, install cache and ask-to-reach fail on main as well. The check doesn't count them against this pull.
  • Not checked: I didn't re-run the other failures behind the "11 of 17734" count.
  • Checks run before the push: bun scripts/gate.ts and bun x tsc --noEmit printed nothing. The drill, MCP restore-drill and job-slices tests pass (17).

If the next check shows those two red again, send me the log lines for them.

Changes requested — before the latest push

Changes needed, first one first.

  1. (high) The drill will go red on the real store. The 3 GiB kill limit sits under the 9 to 13 GiB that restore.ts buffers, and the pull leaves restore.ts unchanged. Moving the buffer to a child changes where it lives, not how big it is. Either make the restore stream to the scratch directory, or set a limit the restore fits in and show that on a real-size snapshot. As it is, a drill that kills itself at 3 GiB is a new way to fail, and nothing in the pull has run against a real store.
  1. (medium) Both doors. The terminal screen and the resilience screen still call drill() and block, while the page button and MCP start a job. Give the TUI the same start and status behaviour, entered where AGENTS.md says, and write the how-to in docs/features/backup-truth.md (already there, but name the terminal).
  1. (medium) Tests that can't fail. 'a drill over its memory limit is killed' only asserts if an error happened (if (err !== "")), so it passes when nothing is killed. Make it deterministic, for example a child that allocates, or a limit seam that always trips. snapshot-drill-page's post() waits only if (gate === undefined). Also in test/drill-job.test.ts: remove the dead void dirStore;, move the mid-file imports to the top, and replace the ls spawn with readdirSync (it adds a spawn site to the test).
  1. (low) Design: the running line on the Backups page packs phase, counts, bytes, process memory and limit into one sentence (principles 1 and 8). Say 'A drill is running: <phase>, X of Y.' and put memory in a muted small span. A fact like this could be a dl row (principle 7).
  1. (low) In drill_job.ts the child's stderr tail is put into the thrown error, which is recorded and shown. Cap and scrub it, since it may carry paths from the store spec.
commented

Points 2 to 5 are done. Point 1 is only partly fixed: the limit no longer kills a drill at 3 GiB, but I have not run it on a real-size store.

1. Memory limit (partly done). The limit in drill_job.ts is now 60% of the machine's memory, between 3 and 16 GiB. FORGE_DRILL_MEMORY_MIB (at least 512) overrides it. The Backups page still states it.

  • What I could not do: I have no access to the real store from the sandbox, so I did not run a real-size drill. restore.ts is unchanged.
  • Why not change restore.ts: the chunked path already streams to disk, with at most 24 reads in flight. I think the 9 to 13 GiB in the serving process was read buffers the collector had not yet given back, but that is a guess.
  • What changed: the child now collects after each progress report. The restore drill working log lines record what the child holds, so the first real drill gives the number to tune the limit from. docs/features/backup-truth.md says this plainly.

2. Both doors. d on Failure domains and r on Resilience now start the same job and follow it. They show "A drill is running: <phase>, X of Y." (polled every 250 ms) and then the verdict. While a drill runs, d is not offered. The how-to in docs/features/backup-truth.md now names the terminal.

3. Tests. All in test/drill-job.test.ts unless named.

  • Over-limit kill: a child that idles, a limit of one byte and a 50 ms reading. It asserts the kill message and fails if nothing is killed.
  • New tests: the limit's bounds and override, the scrubbed error, and a TUI test with a held drill.
  • Cleanup: I removed void dirStore;, moved the imports to the top and replaced the ls spawn with readdirSync.
  • Page test: post() in test/snapshot-drill-page.test.ts now waits unless told not to. The two-at-once test checks that the page says a drill is running.

4. Design. The Backups line is now "A drill is running: <phase>, X of Y." with bytes read, memory and limit in a muted small span. I did not use a dl row, as it is one fact beside a button.

5. Child error. The child's stderr tail is scrubbed in drill_job.ts: tokens containing a / become <path>, whitespace is collapsed, and it is capped at 200 characters.

bun scripts/gate.ts, bun scripts/design-lint.ts and bun x tsc --noEmit are clean. The drill, TUI, page, MCP, timers and job-slices tests pass. fast.ts named 2,111 files, too broad to run, so I picked the covering tests myself. The full check runs on push.

commented

The gate failure is fixed and pushed; the forge's check has not run on the new push yet, so it is not confirmed green.

  • Real-timer rule: the two flagged waits are now marked // real-timer:. One is the 100 ms poll in test/tui-drill.test.ts. The other is the idle child script in test/drill-job.test.ts. Both are there because a real child process is what the tests exercise.
  • Comment width: I rewrapped the comments over 100 columns in src/actions/backups_now.ts, src/tui/screens/resilience.ts and test/tui-drill.test.ts. No code or behaviour changed.

bun scripts/gate.ts and bun x tsc --noEmit both pass locally. test/tui-drill.test.ts and test/drill-job.test.ts also pass, 14 tests in total.

Approved

Earlier asks are met. (1) The memory limit is now 60% of the machine, 3 to 16 GiB, with a FORGE_DRILL_MEMORY_MIB override, and the Backups page states it. (2) Both doors: d on Failure domains and r on Resilience start and follow the same job, and the how-to is in docs/features/backup-truth.md with README untouched. (3) The new behaviours have tests, including the over-limit kill, the limit bounds, the scrubbed error and the TUI. (4) The Backups line is one sentence plus a muted span, with no card. (5) The child's stderr is scrubbed to no paths and 200 characters. Security holds: admin is checked on the page, MCP and terminal; the page POST goes through readForm admin; the job goes to the child on stdin; and the child env is minimal. I could not diff since 6a8fc01dc711 because the commit is not in the clone, so I read the whole change. Non-blocking: the memory limit is untuned on a real-size store, so read the first real drill's restore drill working lines and adjust it.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs