Public: anyone can read this; the crew can change it.
The restore drill: cut by the edge, buffered in the serving process, lost on a restart, and silent while it runs #12600
Check · Review · Land · Live done — Landed and live in production.
The restore drill is now a job that runs in its own process, so it no longer holds the snapshot in the serving process or ends in a 524. The gate, bun x tsc --noEmit and the drill, MCP-size and Backups-page tests pass here. Nothing has run against a real store, and the forge's check is still to come on this push.
shipped production is serving it deploy 5bb6da3
7 earlier comments · 1 event
This push fixes the seven test files that were red because of this pull. bun x tsc --noEmit is clean. The nine test files that cover the change pass locally, and bun scripts/gate.ts printed nothing. I did not run the whole suite; the forge's check does that on this push.
Four tests were red because the repository's rule lists didn't yet know about the new src/snapshot/drill_job.ts:
- Spawn lists: I added
drill_job.tstotest/spawn-pipes.test.ts(it pipes stdout and stderr) and totest/spawn-sites.test.ts(its argv is only the bun path and a fixed script). - Timers: the watcher
setIntervalnow runs insideduring("drill-watch", …), and the 5-minute-plus-budget kill timer is on the KILL allowlist intest/timers-named.test.ts. - Unused exports:
markerLeftis only used by tests, so I added it toscripts/unused-exports-allow.jsonrather than removing it. - MCP table: I regenerated the
restore_drillrow indocs/MCP.mdwithbun scripts/mcp-table.ts.
The other three failures were tests that assumed the old synchronous drill:
- Poll loops: the two in
test/mcp-restore-drill.test.tsnow go through onesettled()helper that throws if the drill is still running after about 2 s. The flake rule flagged the old loops because they carried on after timing out. - Card test:
restore_drillwith no arguments now only starts the job, so the card test intest/mcp-ops-tool-cards.test.tsstarts a drill and pollsstatusuntil the card appears.
The five suites the check log marked as red on main too (release-handover-keepalive, apps-ws, apps-open-host, ask-to-reach, check-install) are not touched by this change. I did not run them.
The one failure from this pull is fixed and pushed. test/job-slices.test.ts requires every during job to be either timed or listed in UNTIMED with a reason. The new drill-watch job was in neither. I added it to UNTIMED, with the reason that it is a memory read and a state write that run only while a restore drill is going.
test/job-slices.test.ts and test/timers-named.test.ts pass locally (11 tests) and bun x tsc --noEmit is clean. I did not run the whole suite.
The forge's log marks five other test files as red on main too: apps-ws, apps-open-host, ask-to-reach, check-install and release-handover-keepalive. I did not touch them or run them. The log is cut off, so I can't see the rest of the 11 failures. The only one it blames on this pull is job-slices.
The restore drill now runs as a job in its own process, started at once and one at a time. A cut drill is recorded and swept, progress is shown, and the age of the last proof is said. This push only merges main into the pull, so the drill code is unchanged from before.
- Merge: the merge conflicted in
test/job-slices.test.ts, where bothdrill-watch(this pull) anddesk(main) were added to the list of untimed jobs. I kept both entries. - Failures not from this pull:
test/module-state.test.tsandtest/apps-open-host.test.tsfail locally as they do onmain. The first fails onsrc/crew/ceiling_stagger.ts:memo, a file this pull doesn't touch. The second fails on a "proxy reads such a Host" case that throwsInvalid URL. - Not reproduced: the check said
test/migrate.test.tsis green onmainand red on this pull, but it passes here, before and after the merge. I'm treating that as flaky unless the new check shows it red again. - Checks run:
bun scripts/gate.tsandbun x tsc --noEmitprinted nothing. The tests for the drill, MCP drill tools, drill page,job-slices,timers-named,spawn-sitesandspawn-pipespass, apart from theapps-open-hostfailure above. - Not seen: the check log was cut off, so I saw only 2 of the 15 failures named. The other 13 I didn't see or re-run.
I can't find anything in this pull that causes the failures, and I couldn't reproduce the one failure the check blames on it. The push only merges the latest main (which brings in a change to test/data-query-limits.test.ts); no drill code changed.
- Failures red on
maintoo: the install cache, release keep-alive,/apps/openhost, module-state and ask-to-reach tests fail onmainas well, so the check doesn't count them against this pull. I didn't run the install cache and keep-alive tests locally. The ask-to-reach failures reproduce here. They try to reachmaria--forge.testandotto--forge.test, which this sandbox blocks and the repo's egress list doesn't allow. - The two
migratefailures in shard 73: "a name that resolves to a private address is refused" and "a vetted host is pinned" are the ones the check says are red only on this pull.test/migrate.test.tspasses here, including when run in the same process astest/ask-to-reach.test.ts, so I can't reproduce them. The likely cause is the shard-73 ask-to-reach failures disturbing themigratetests. I didn't change them or any egress code. - Slow tests: the 50 s
crew-proposalcase and the other over-budget ones look like a loaded shard.test/crew-proposal.test.tspasses locally with the drill tests. - Checks run:
bun scripts/gate.tsandbun x tsc --noEmitprinted nothing.drill-job,job-slices,mcp-restore-drillandcrew-proposalpass.
If the migrate pair is red again after this push, it needs a look at what ask-to-reach leaves behind in a shared shard.
I merged main into the branch and pushed it. The only conflict was in src/snapshot/service.ts, at the boot catch-up of the restore drill.
main had added an app argument to the during(...) call there, and this pull had changed the same line so the drill can start with drill() when the previous one was cut. I kept both, so the call still starts the drill directly after a cut one and now passes undefined, app as well.
bun x tsc --noEmit and bun scripts/gate.ts print nothing. The drill, MCP restore-drill, drill page, job-slices and all test/snapshot*.test.ts tests pass (181 pass, 0 fail). The full check will run on the push.
I found no fault in this pull behind the 11 failures and changed no code. The push only merges the newest main into the branch. The check blamed this pull for two tests, and both pass here.
- Blamed on this pull:
test/boundary-tokens-follow.test.tspasses (10 tests) on the merged tree. The check says it is green onmain, but this pull doesn't touch tokens or boundaries.test/quotas-running.test.ts, which holds "a fresh account's second run waits queued", also passes here (7 tests). I think both failed because the shard was loaded: the log shows several tests running over 2 s. - Red on
maintoo:/apps/openhost, release keep-alive, install cache and ask-to-reach fail onmainas well. The check doesn't count them against this pull. - Not checked: I didn't re-run the other failures behind the "11 of 17734" count.
- Checks run before the push:
bun scripts/gate.tsandbun x tsc --noEmitprinted nothing. The drill, MCP restore-drill andjob-slicestests pass (17).
If the next check shows those two red again, send me the log lines for them.
Changes needed, first one first.
- (high) The drill will go red on the real store. The 3 GiB kill limit sits under the 9 to 13 GiB that restore.ts buffers, and the pull leaves restore.ts unchanged. Moving the buffer to a child changes where it lives, not how big it is. Either make the restore stream to the scratch directory, or set a limit the restore fits in and show that on a real-size snapshot. As it is, a drill that kills itself at 3 GiB is a new way to fail, and nothing in the pull has run against a real store.
- (medium) Both doors. The terminal screen and the resilience screen still call drill() and block, while the page button and MCP start a job. Give the TUI the same start and status behaviour, entered where AGENTS.md says, and write the how-to in docs/features/backup-truth.md (already there, but name the terminal).
- (medium) Tests that can't fail. 'a drill over its memory limit is killed' only asserts if an error happened (
if (err !== "")), so it passes when nothing is killed. Make it deterministic, for example a child that allocates, or a limit seam that always trips. snapshot-drill-page's post() waits onlyif (gate === undefined). Also in test/drill-job.test.ts: remove the deadvoid dirStore;, move the mid-file imports to the top, and replace thelsspawn with readdirSync (it adds a spawn site to the test).
- (low) Design: the running line on the Backups page packs phase, counts, bytes, process memory and limit into one sentence (principles 1 and 8). Say 'A drill is running: <phase>, X of Y.' and put memory in a muted small span. A fact like this could be a dl row (principle 7).
- (low) In drill_job.ts the child's stderr tail is put into the thrown error, which is recorded and shown. Cap and scrub it, since it may carry paths from the store spec.
Points 2 to 5 are done. Point 1 is only partly fixed: the limit no longer kills a drill at 3 GiB, but I have not run it on a real-size store.
1. Memory limit (partly done). The limit in drill_job.ts is now 60% of the machine's memory, between 3 and 16 GiB. FORGE_DRILL_MEMORY_MIB (at least 512) overrides it. The Backups page still states it.
- What I could not do: I have no access to the real store from the sandbox, so I did not run a real-size drill.
restore.tsis unchanged. - Why not change
restore.ts: the chunked path already streams to disk, with at most 24 reads in flight. I think the 9 to 13 GiB in the serving process was read buffers the collector had not yet given back, but that is a guess. - What changed: the child now collects after each progress report. The
restore drill workinglog lines record what the child holds, so the first real drill gives the number to tune the limit from.docs/features/backup-truth.mdsays this plainly.
2. Both doors. d on Failure domains and r on Resilience now start the same job and follow it. They show "A drill is running: <phase>, X of Y." (polled every 250 ms) and then the verdict. While a drill runs, d is not offered. The how-to in docs/features/backup-truth.md now names the terminal.
3. Tests. All in test/drill-job.test.ts unless named.
- Over-limit kill: a child that idles, a limit of one byte and a 50 ms reading. It asserts the kill message and fails if nothing is killed.
- New tests: the limit's bounds and override, the scrubbed error, and a TUI test with a held drill.
- Cleanup: I removed
void dirStore;, moved the imports to the top and replaced thelsspawn withreaddirSync. - Page test:
post()intest/snapshot-drill-page.test.tsnow waits unless told not to. The two-at-once test checks that the page says a drill is running.
4. Design. The Backups line is now "A drill is running: <phase>, X of Y." with bytes read, memory and limit in a muted small span. I did not use a dl row, as it is one fact beside a button.
5. Child error. The child's stderr tail is scrubbed in drill_job.ts: tokens containing a / become <path>, whitespace is collapsed, and it is capped at 200 characters.
bun scripts/gate.ts, bun scripts/design-lint.ts and bun x tsc --noEmit are clean. The drill, TUI, page, MCP, timers and job-slices tests pass. fast.ts named 2,111 files, too broad to run, so I picked the covering tests myself. The full check runs on push.
The gate failure is fixed and pushed; the forge's check has not run on the new push yet, so it is not confirmed green.
- Real-timer rule: the two flagged waits are now marked
// real-timer:. One is the 100 ms poll intest/tui-drill.test.ts. The other is the idle child script intest/drill-job.test.ts. Both are there because a real child process is what the tests exercise. - Comment width: I rewrapped the comments over 100 columns in
src/actions/backups_now.ts,src/tui/screens/resilience.tsandtest/tui-drill.test.ts. No code or behaviour changed.
bun scripts/gate.ts and bun x tsc --noEmit both pass locally. test/tui-drill.test.ts and test/drill-job.test.ts also pass, 14 tests in total.
Earlier asks are met. (1) The memory limit is now 60% of the machine, 3 to 16 GiB, with a FORGE_DRILL_MEMORY_MIB override, and the Backups page states it. (2) Both doors: d on Failure domains and r on Resilience start and follow the same job, and the how-to is in docs/features/backup-truth.md with README untouched. (3) The new behaviours have tests, including the over-limit kill, the limit bounds, the scrubbed error and the TUI. (4) The Backups line is one sentence plus a muted span, with no card. (5) The child's stderr is scrubbed to no paths and 200 characters. Security holds: admin is checked on the page, MCP and terminal; the page POST goes through readForm admin; the job goes to the child on stdin; and the child env is minimal. I could not diff since 6a8fc01dc711 because the commit is not in the clone, so I read the whole change. Non-blocking: the memory limit is untuned on a real-size store, so read the first real drill's restore drill working lines and adjust it.
The restore drill is now a job that runs in its own process, so it no longer holds the snapshot in the serving process or ends in a 524. The gate,
bun x tsc --noEmitand the drill, MCP-size and Backups-page tests pass here. Nothing has run against a real store, and the forge's check is still to come on this push. The sandbox skips the unix-socket suites.What changed
restore_drilltakesaction: start(the default) orstatus.startanswers at once with an id, the phase, the bytes and the memory limit. A second ask while one runs getsstarted: falseand the running drill's id.statusanswers the running drill's phase and bytes, or the verdict once it has ended. No tool was added, somcp-sizeis unchanged.src/snapshot/drill_child.ts, started bydrill_job.tswith--smol, with its own new session. The job reads the child's resident memory every 3 s and kills it, the drill red, above 3 GiB. The limit is shown on the Backups page. This limits the child, not the restore itself: I did not change howrestore.tsbuffers, so a large snapshot may well reach the 3 GiB cap.restore drill workingline every 15 s and lines at start and end.drill/running.json, is left while a drill runs. At boot, a marker left behind is recorded as a red drill "cut by a restart", and a drill is started again after the 5-minute catch-up delay. Everydrill/restore-*directory is swept at boot, and again when any drill ends, passed or not.test/drill-job.test.tscovers the sweep, the child, the memory kill, the status age and a restart mid-drill. I changedmcp-restore-drillandsnapshot-drill-pagebecause they assumed the call waits for the verdict. Therestore_drillrow indocs/MCP.mdanddocs/features/backup-truth.mddescribe the job.Not done
drill()and wait for the verdict. Only the MCP tool and the web button are non-blocking.No look:since the Backups page changes only while a drill runs.Try it on the preview: call
restore_drillwithaction: start, thenaction: status, and open/settings/backupsto watch it.🤖 Generated with Claude Code
Closes #12592