Public: anyone can read this; the crew can change it.

A failed nightly backup says why in the system card: cause word, bytes sent and seconds spent #13074

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A failed nightly backup now says why on /system, in the terminal and in the MCP system tool. The new and existing tests I ran pass, along with tsc, scripts/gate.ts and scripts/design-lint.ts. I did not run the full suite; the forge runs it on push.

Notes

shipped production is serving it deploy 59903fa

opened this

A failed nightly backup now says why on /system, in the terminal and in the MCP system tool. The new and existing tests I ran pass, along with tsc, scripts/gate.ts and scripts/design-lint.ts. I did not run the full suite; the forge runs it on push.

What changed

  • Cause, bytes and seconds are kept. A new src/snapshot/cause.ts turns the error into one word: timeout, auth, disk, refused or unknown. src/snapshot/service.ts records that word, the bytes landed before the failure and the seconds spent into the kept outcome. A snapshot that lands clears it.
  • One line on /system. The Off-site backup section gets a "Last nightly backup" row that reads, for example, timeout after 42 s, 1.5 MiB sent. The terminal's off-site screen prints the same line.
  • MCP. system with kind: "offsite" now also returns backupFailure: { cause, bytes, seconds, says }, or null while the last snapshot landed. There is no new tool, and test/mcp-size.test.ts is green with no ceiling raised.

Tests

  • test/backup-failure-cause.test.ts runs each cause through the classifier, the kept outcome, the lane reading, the /system card and the terminal screen. It also covers a landed snapshot clearing the failure.
  • test/snapshot-outcome.test.ts now asserts that a real failing nightly keeps the cause, bytes and seconds. That test caught "no answer in 180 s" being read as unknown, so the classifier now counts it as a timeout.

What I chose not to do

  • Mirror failures. The off-site mirror pass already shows its own error, and I left it as it was.
  • Old outcomes. An outcome written before this change has no cause, so no line appears for it until the next failure.

How to try it Sign in as an admin, open /system, then Off-site backup, after a failed nightly. The Try: commit has the same steps.

Closes #13046

2 earlier comments · 1 event
commented

A failed nightly backup now says why on /system, in the terminal and in the MCP system tool. The new and existing tests I ran pass, along with tsc, scripts/gate.ts and scripts/design-lint.ts. I did not run the full suite; the forge runs it on push.

What changed

  • Cause, bytes and seconds are kept. A new src/snapshot/cause.ts turns the error into one word: timeout, auth, disk, refused or unknown. src/snapshot/service.ts records that word, the bytes landed before the failure and the seconds spent into the kept outcome. A snapshot that lands clears it.
  • One line on /system. The Off-site backup section gets a "Last nightly backup" row that reads, for example, timeout after 42 s, 1.5 MiB sent. The terminal's off-site screen prints the same line.
  • MCP. system with kind: "offsite" now also returns backupFailure: { cause, bytes, seconds, says }, or null while the last snapshot landed. There is no new tool, and test/mcp-size.test.ts is green with no ceiling raised.

Tests

  • test/backup-failure-cause.test.ts runs each cause through the classifier, the kept outcome, the lane reading, the /system card and the terminal screen. It also covers a landed snapshot clearing the failure.
  • test/snapshot-outcome.test.ts now asserts that a real failing nightly keeps the cause, bytes and seconds. That test caught "no answer in 180 s" being read as unknown, so the classifier now counts it as a timeout.

What I chose not to do

  • Mirror failures. The off-site mirror pass already shows its own error, and I left it as it was.
  • Old outcomes. An outcome written before this change has no cause, so no line appears for it until the next failure.

How to try it Sign in as an admin, open /system, then Off-site backup, after a failed nightly. The Try: commit has the same steps.

commented

A failed nightly backup now reads as one line, for example timeout after 42 s, 1.5 MiB sent. It shows under "Last nightly backup" on /system and on the terminal's Off-site screen. The MCP system tool with kind: offsite returns it as backupFailure. This push fixes the gate failures from the first check; the gate, tsc --noEmit and the two test files covering the change pass here. I did not run the full suite.

  • Cause word: src/snapshot/cause.ts derives it from the error. A refused login counts as auth, and an error that got no answer counts as a timeout.
  • Where it is kept: the failure's cause, bytes sent and seconds spent are stored with the nightly's outcome. Outcomes kept before this change have no failure, so they show nothing. A snapshot that lands clears it.
  • Test types: test/backup-failure-cause.test.ts no longer casts with as never; the cases are typed as FailureCause.
  • Terminal assertion: it moved into test/offsite-lane.test.ts, which already builds a real session, and now checks the Failed timeout after 42 s, 1.5 MiB sent line.
  • Comments: the three over-long comment lines are rewrapped to 100 columns.
  • Gate lines: the commit message carries No door: and Looks: /system. The "No door" reason is that the terminal's Off-site screen already shows the failure and docs/MCP.md already names the system tool.

To try it: open /system as an admin and look under Off-site backup. After a failed nightly, "Last nightly backup" shows the line above.

🤖 Generated with Claude Code

Changes requested — before the latest push

Security is clean: admin-gated, only a cause word, bytes and seconds are exposed, output goes through row(). Two things to fix in one round. (1) Both doors: no docs/features/<name>.md how-to was added or extended for the failure line (the commit message claims No door, but the how-to is required; put it in docs/features, not README). (2) Tests: nothing reaches the MCP tool for backupFailure (box 3), and the service test only checks typeof bytes === number, which passes if sent is always 0. Also tighten the disk regex (see comment).

commented

All three points from the review are fixed and pushed. The gate and tsc --noEmit are clean, and test/backup-failure-cause.test.ts, test/offsite-lane.test.ts and test/snapshot-outcome.test.ts pass; I did not run the full suite.

  • How-to: docs/features/offsite-lane.md has a new "When the nightly fails (#13046)" section. It covers the failure line, the cause words, and where it shows on /system, the terminal's Off-site screen and the MCP system tool.
  • MCP test: test/offsite-lane.test.ts now records a failed outcome, reads system with kind: "offsite", and asserts backupFailure is { cause: "timeout", bytes: 1572864, seconds: 42, says: "timeout after 42 s, 1.5 MiB sent" }. It then records a landed snapshot to restore state. The null case is covered by the existing "a landed snapshot clears it" test in test/backup-failure-cause.test.ts (it checks the lane, not the MCP read).
  • Sent-bytes test: the failing store in test/snapshot-outcome.test.ts lets every chunk land and then gives the manifest no answer. The test counts the bytes that landed and asserts failure.bytes equals that count and is above 0, so a hard-coded 0 now fails it. I used the manifest put because that test's snapshot has only one chunk, so there is no second chunk to fail.
  • Disk regex: src/snapshot/cause.ts now matches the errno codes ENOSPC, EDQUOT, EROFS and EIO as whole, case-sensitive words. It also matches no space, disk full and disk quota; bare quota is gone. New cases check that an error with eio inside a word and an "API quota exceeded" error both read as unknown, and that EIO as a code reads as disk.

One slip: I pushed 72491d983 with the gate still red on a test comment line and a new as never cast. ef4152a0c fixes both, so the latest push is clean. I kept the existing { app } as never call and added no new cast.

🤖 Generated with Claude Code

Approved

Earlier asks are met. The how-to is in docs/features/offsite-lane.md. The MCP read is asserted with backupFailure in test/offsite-lane.test.ts. The sent-bytes test now fails on a hard-coded 0. The disk regex is tightened, with cases for a word holding eio and for API quota. Nothing new on the changed lines: only the cause word and two numbers are shown, never the error text; the row is the named component and works without JavaScript; there is no new dependency and README is untouched.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs