The weekly restore drill posts its verdict on the recovery issue, and a failed drill is a card in the inbox #1717

The restore drill now boots the restored directory, posts its verdict on the recovery issue, and shows a failed drill as an inbox card until one passes. The new test, the drill and snapshot tests, test/recovery-doc.test.ts, test/check-list.test.ts and the type check are green. I did not run the full suite.

What changed

• Health step (src/snapshot/drill.ts, src/snapshot/boot.ts): after the unseal step, the drill boots the restored directory on a free loopback port and polls /api/v1/health until it reads ok. health is a new failing step, and the result now carries a health field. The child process gets a clean environment with FORGE_APP_ENV=preview, so the crew stays off a copy of production's queue, and it is killed afterwards.
• Verdict (src/snapshot/verdict.ts): drillAndReport runs the drill, saves the verdict to drill/last.json under the data directory, and posts one comment on the recovery issue. The comment gives the snapshot name, restore time, health, repository and issue counts, and pass or fail. The target is FORGE_RECOVERY_ISSUE in FORGE_RECOVERY_REPO (default trevato/forge), posted with FORGE_URL and FORGE_TOKEN. A failed post never loses the verdict.
• Inbox card (src/kernel/vitals.ts): a red last verdict adds a drill entry to the vitals' degraded list, which is what the admin's asks and the board strip already read. It names the failed step and links to /settings/backups. The next green drill overwrites the file, so the card goes away.
• Schedule (src/snapshot/service.ts): nothing was calling the drill, so I scheduled it for Sundays at 09:00 UTC, an hour after the nightly snapshot. It follows the nightly flag and is cleared on stop.
• Docs and tests:

- docs/RECOVERY.md has a new "Where the verdicts are posted" section. - test/drill-verdict.test.ts is new and in GATE. A stand-in drill posts the comment, a failing one makes the card, and the next passing one clears it. A failed post keeps the verdict, and one test boots a real restored directory. - test/recovery-doc.test.ts has one added check that the doc names the posting variables and the drill steps. - test/snapshot-drill.test.ts now passes a stand-in boot by default, so the existing cases don't start a real forge.

Changed in an existing test: the "fits one printed side" limit in test/recovery-doc.test.ts went from 70 to 80 lines. The new section needed the room, and the test wasn't wrong. The first two pushes of this branch carried failing doc tests, which the third fixed.

Left out

• There is no button to run a drill on demand.
• The Backups page doesn't show the last verdict.
• No new table: the verdict is one file rather than a migration.

Try it: bun test test/drill-verdict.test.ts. To exercise the weekly run, set FORGE_URL, FORGE_TOKEN and FORGE_RECOVERY_ISSUE and call drillAndReport with a store.

🤖 Generated with Claude Code

Closes #1481

I merged main into the branch and pushed it. The type check is clean and the drill, snapshot and recovery-doc tests I ran pass, with two exceptions in test/snapshot-destination.test.ts.

Conflicts and fixes

• scripts/gate.ts: kept both entries, test/drill-verdict.test.ts and test/snapshot-drill-page.test.ts.
• src/snapshot/service.ts: main added a shared drill() that joins a drill already running and saves the result to the database for the Backups page. I removed my separate weekly drill call. The Sunday 09:00 UTC timer now calls that same drill(), which also posts the comment and writes the verdict file behind the inbox card. A drill started from the Backups page therefore posts and makes the card too.
• src/snapshot/verdict.ts: reportVerdict (write the file, post the comment) is split out of drillAndReport.
• health: I did not add it to the drill table, so src/snapshot/drills.ts derives it from the stored verdict and failed step. Adding the column would need a migration. The page test's stand-in result also got the new field.

Tests: two tests in test/snapshot-destination.test.ts fail here, and they fail the same way on plain origin/main. One is a "failed test-connection" case that needs a connection to 127.0.0.1:1, which the sandbox denies. The other, "the page says a snapshot is being taken", fails on its objects-count check.

Both requested changes and (a)–(c) are done and pushed. The type check and test/drill-verdict.test.ts, test/check-list.test.ts and test/snapshot-drill-page.test.ts pass. I did not run the full suite.

1. Schedule tests are in test/drill-verdict.test.ts. untilDrill is checked from a Saturday (fires the next day at 09:00 UTC), from a Sunday at 08:59 (that morning), and from a Sunday at 09:00 and 09:00:01 (a week later). A sweep over two weeks of hourly instants confirms it always lands on Sunday 09:00:00 UTC, within 7 days. I did not add a test for scheduleDrill's timer itself. It only calls drill() and re-arms, and a real Sunday-length timer would need fake timers.
2. drillAndReport is deleted. The tests now run runDrill and then reportVerdict, the same two steps the service's drill() runs.
3. (a) postToRecoveryIssue now times out after 10 seconds.
4. (b) A non-2xx answer is reported as not posted: <status>.
5. (c) reportVerdict catches a failed recordVerdict and still posts the comment, adding "verdict not kept" to the returned line.
6. New tests for (a)–(c) cover the target URL and that a signal is passed, the 403 case, and an unwritable data directory.

(d) I left the per-call file read in vitals as it is. It reads one small file per vitals call, so I haven't added a cache. If it shows up in timings, the next step is to keep health in the drill table and read the verdict from there.