Public: anyone can read this; the crew can change it.

The backup alarm is a dead man's switch: a missed check is itself red #8980

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

Most of the dead man's switch was already on main. This pull adds the two missing pieces: a record of each check run, and the time of the last run in the red message.

shipped production is serving it deploy 2e641a3

opened this

Most of the dead man's switch was already on main. This pull adds the two missing pieces: a record of each check run, and the time of the last run in the red message.

Already on main (#8753 and #8831):

  • watchBucket reads the bucket in the forge's own process on its own timer.
  • judge turns red once no check has run for CHECK_STALE_MS (20 min).
  • That verdict feeds home's backups tile, the vitals "backup" card and /api/v1/health for admins.

Changed:

  • Run record: every check run appends one JSON line to backup-checks.jsonl under the data directory: when it ran and each lane's ok/newest. The file is trimmed to the last 200 runs. checkRuns() reads it back. A write failure never stops the check.
  • Red message: it now includes the time of the last run: "The backup check has not run for 25 min (last ran 2026-11-02T08:00:00Z); it is the alarm, and it is down." The vitals card and health show the same sentence.
  • Test: in test/backup-truth.test.ts, the check runs twice and each run is recorded. Then nothing reads the bucket while the clock moves past the interval. The verdict and the vitals card go red with the last-run time, and vitals.ok is false.

I did not add a separate test of the tile's rendering; the test covers the verdict, the vitals card and ok. The tile and /api/v1/health read those same values.

Checks: bun test on backup-truth and vitals passes, tsc --noEmit is clean, and scripts/gate.ts printed nothing. I didn't run the rest of the suite.

Not done: I couldn't update the plan on the issue because the keep_plan tool wasn't available.

Try it: bun test test/backup-truth.test.ts.

🤖 Generated with Claude Code

Closes #8959

commented

Most of the dead man's switch was already on main. This pull adds the two missing pieces: a record of each check run, and the time of the last run in the red message.

Already on main (#8753 and #8831):

  • watchBucket reads the bucket in the forge's own process on its own timer.
  • judge turns red once no check has run for CHECK_STALE_MS (20 min).
  • That verdict feeds home's backups tile, the vitals "backup" card and /api/v1/health for admins.

Changed:

  • Run record: every check run appends one JSON line to backup-checks.jsonl under the data directory: when it ran and each lane's ok/newest. The file is trimmed to the last 200 runs. checkRuns() reads it back. A write failure never stops the check.
  • Red message: it now includes the time of the last run: "The backup check has not run for 25 min (last ran 2026-11-02T08:00:00Z); it is the alarm, and it is down." The vitals card and health show the same sentence.
  • Test: in test/backup-truth.test.ts, the check runs twice and each run is recorded. Then nothing reads the bucket while the clock moves past the interval. The verdict and the vitals card go red with the last-run time, and vitals.ok is false.

I did not add a separate test of the tile's rendering; the test covers the verdict, the vitals card and ok. The tile and /api/v1/health read those same values.

Checks: bun test on backup-truth and vitals passes, tsc --noEmit is clean, and scripts/gate.ts printed nothing. I didn't run the rest of the suite.

Not done: I couldn't update the plan on the issue because the keep_plan tool wasn't available.

Try it: bun test test/backup-truth.test.ts.

🤖 Generated with Claude Code

Changes requested — before the latest push

Most of the switch was already on main (#8753/#8831); this adds the run record and the last-run time. Three things to fix in one round:

  1. checkRuns is exported but nothing but the test calls it, and backup-checks.jsonl is written and never read by any door. Done means reachable: show the recent runs (or at least the last run time and count) on the backups tile / the vitals backup card, and in the terminal's backups screen, so a person can see the check is alive. Otherwise drop the log.
  2. Docs: docs/features/backup-truth.md does not mention the run record, the file name, its 200-run trim, or the new red wording. Update it (not README).
  3. Tests: the new test covers verdict, vitals card and ok, but not the home tile or /api/v1/health that the issue's box 2 names, and not the trim at 2x200 or the never-throws path. Add a test rendering home and /api/v1/health (admin) at the late clock and asserting the red and 'last ran' text; add a trim test. Also the test asserts on dataDir shared with other tests (before+2 count) - use a fresh temp dir.

Smaller: recordRun reads and splits the whole file on every run; trim by size or count with a counter instead. Show the last-run time in a human form beside the ISO if the tile prints it.

Preview /api/v1/health answers ok, but the dead-man state cannot be seen there without waiting 20 minutes, so the tests must carry that proof.

commented

All three review points are fixed and pushed to the same branch. The tests I ran pass, and so do tsc --noEmit, scripts/gate.ts and scripts/design-lint.ts. I ran six files (backup-truth, home-kpi-row, vitals, tui-backups, kpi-wrap, backups-line) and did not run the full suite; the forge's check does that.

What changed

  • The run record is now readable. BackupVerdict carries runs and checkLine, for example "The backup check last ran 2026-11-02T08:00:00Z (3 min ago); 42 runs on record." This is the human form beside the ISO time that you asked for.

- The line appears in the Resilience dialog (verdictLines). - It also appears in the home tile's title, and in the backup vitals card. - The card skips it when the stale reason already says "last ran". - checkRuns is gone, since the verdict is the door.

  • Cheaper trimming. The run count lives in memory and is seeded from the file once, when the watch starts. A run no longer reads the file. Only when the log passes 400 runs is it rewritten to the last 200. recordRun is exported so the trim and never-throws tests can call it.
  • Docs. docs/features/backup-truth.md now covers the new red wording, the file name, the 400-to-200 trim and checkLine.
  • Tests, using fresh temp dirs rather than the shared one:

- The stopped-check test (test/backup-truth.test.ts) runs the check twice, then moves the clock past the interval. It asserts red and "last ran" in the verdict, the home tile (kpiRowFor), the vitals card, the admin /api/v1/health data (readHealth) and the terminal lines. It mocks Date.now, because readHealth reads the real clock. - A separate test covers the trim and the never-throws path.

Not changed

  • The health test calls readHealth rather than fetching the HTTP route. The route is a thin wrapper around it and takes no clock. I did not add an HTTP-level test.
  • I did not put a "last ran" line on /settings/backups. The tile links there and the Resilience dialog carries the line.

🤖 Generated with Claude Code

commented

The failed check was the gate rule "say how to try it on the preview": none of the pull's commit messages had a Try: line. I pushed an empty commit that adds one. No code changed, and the tests were not re-run.

I ran scripts/gate.ts locally after the commit and it printed nothing. I did not re-run the forge's full check.

The Try: line says to open the home page's backups tile title, or the terminal's Resilience dialog. Either shows when the backup check last ran and how many runs are on record.

🤖 Generated with Claude Code

Approved

Earlier asks (readable run record via checkLine, cheaper trimming with an in-memory count, docs, fresh-tempdir tests incl. tile/health/terminal) are met in the head. The preview answers 200. Two things I would do next, both non-blocking: (1) src/kernel/vitals.ts: the card decides whether to append checkLine by searching reason text for "last ran"; let judge expose a flag or put checkLine in one place so wording changes cannot double or drop it. (2) Add an HTTP-level /api/v1/health test, and a "last ran" line on /settings/backups, so the page the tile links to also shows the check is alive.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs