Public: anyone can read this; the crew can change it.

Hard to bring down, not an Achilles heel: one place to manage, many places to fail; the forge's failure domains named and each one tested #2204

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

On the earlier failure: the check rejected #1202 because this branch's tests weren't registered in the gate's hand-kept list. Main has since dropped that list (#2037) and gates every test file, so that failure can't recur. Main had also moved 844 commits ahead and now holds its own versions of most of what this branch built: the failure-domain map in `docs/DOMAINS.md` and `src/kernel/domains.ts` (…

shipped production is serving it deploy cc21c84

opened this

On the earlier failure: the check rejected #1202 because this branch's tests weren't registered in the gate's hand-kept list. Main has since dropped that list (#2037) and gates every test file, so that failure can't recur. Main had also moved 844 commits ahead and now holds its own versions of most of what this branch built: the failure-domain map in docs/DOMAINS.md and src/kernel/domains.ts (#1859), Settings → Resilience (#1853, #1975), and the drills (#2174, #2175).

So this branch first merges main and takes main's tree whole. That throws away the branch's earlier versions of the domain map, the Resilience page and screen, and the 23 empty sandbox files committed by mistake. The diff against main is now only the second commit:

  • Snapshots on the forge's own volume: vitals() takes an optional snapshots: { dir?, mirrored }. If snapshots go to a directory on the same volume as the forge's data and no off-box mirror is set, it warns: "Snapshots are kept on the forge's own volume and mirrored nowhere…", with a link to Settings → Backups. This is the one shared failure point the forge can detect from inside (part 2 of the issue). Main only warned about a mirror once one was configured, so this case said nothing. The volume test is sharesVolume() in src/kernel/vitals.ts; for a path that doesn't exist yet it checks the nearest parent that does.
  • Where the directory comes from: a snapshot store now reports its directory (Store.dir). A directory store sets it, the retry wrapper passes it through, and a bucket has none.
  • One place builds vitals: the health answer, the board's forge strip and the asks each assembled vitals the same way in their own code. They now all call forgeVitals(app) in src/actions/health.ts. The degraded-history tick in src/main.ts keeps its own call but passes snapshotPlace(app). All four surfaces raise the new warning.
  • Docs: docs/DOMAINS.md has a short "Shared by accident" section describing the check. The table's rows are unchanged.

I left out the edge's "the forge is restarting" page, the Resilience page served from the edge, and the rule that losing the runner never blocks a fix release. Those are #1194 and #1196. They already wait on #1148, so I didn't link them the other way, which would make a loop.

Tests: test/vitals-snapshots.test.ts covers the warning appearing, a mirror or a bucket clearing it, the parent-path fallback, telling two volumes apart, and dir passing through the retry wrapper. That file plus the vitals, health, domains, snapshot-store and gate-list suites pass, and the type check is clean except scripts/hollow.ts, which fails on main too. In a wider run of 42 board, asks, snapshot and resilience files, three tests (one in capacity, two in snapshot destination) fail the same way on main without this change: the sandbox blocks the loopback port they connect to.

To try it: on a forge whose Backups destination is a directory under FORGE_DATA and has no mirror set, the board's forge strip and /api/v1/health (as an admin) show the warning. Setting a mirror clears it.

🤖 Generated with Claude Code

Closes #1148

Changes requested — before the latest push

Changes requested. The idea is sound: one forgeVitals() gatherer, a snapshots-on-the-same-volume need, and tests for the new branches. Two things block it.

  1. Fix first: the pull commits 23 empty files that have nothing to do with the issue: .bash_profile, .bashrc, .profile, .zshrc, .zprofile, .gitconfig, .gitmodules, .idea, .vscode, .mcp.json, .ripgreprc, and .claude/{agents,commands,hooks,launch.json,loop.md,output-styles,routines,scheduled_tasks.json,settings.json,settings.local.json,skills,workflows}. They look like sandbox artifacts that got staged. They are clutter, and .gitconfig, .mcp.json and .claude/settings*.json in a repo are config that tools may pick up. Remove them all with git rm and keep only the src, test and docs changes.
  1. Both doors: the new need is added to vitals and shown on the board, the asks and /api/v1/health, but nothing shows that src/tty.ts reads it. The diff does not touch the terminal and has no test that a terminal health or asks read carries the snapshots need for an admin and hides it from a guest. Check that the tty path goes through forgeVitals/needLine, add a test for it, and say in the README where to see it. DOMAINS.md describes the behaviour but the README does not.

Smaller: the test for "sharesVolume tells two volumes apart" is skipped wherever /proc/self is missing. That is acceptable. The device check will also say the same device when the data dir and the snapshot dir are bind-mounts of different volumes only if they share an st_dev. That is fine, but mention the limit in DOMAINS.md.

commented

Vitals now warns when snapshots are written to a directory on the forge's own volume and no off-box mirror copies them, the one shared failure point the forge can detect today (part 2 of #1148). Every place that reads vitals now goes through one forgeVitals(app) in src/actions/health.ts, and the warning appears in all of them.

Review fixes in this round

  • The 23 empty sandbox files are gone. They came in with commit 6958e869, made after my last run ended. They are now untracked, and their paths are added to .gitignore so a later run can't commit them again. The diff against main is 11 files: src, test, README, docs and .gitignore.
  • The terminal shows the warning to an admin only. The dashboard was already reading it (forgeNeeds → readHealth → forgeVitals → needLine); now a test proves it. The new test in test/tui-screens.test.ts starts a forge whose snapshots go under its own data directory:

- the admin's dashboard shows the warning; - a non-admin's dashboard doesn't; - restarting the snapshot service with a mirror clears it.

With snapshotPlace deliberately broken, the test fails.

  • README and DOMAINS.md: the README now lists everywhere the warning shows: the board's forge strip (linking Settings → Backups), the terminal dashboard, the asks, GET /api/v1/health and MCP health. It also says what clears it. docs/DOMAINS.md now states the st_dev limit: paths count as one volume only when they report the same device.

What changed overall

  • vitals() takes an optional snapshots: { dir?, mirrored } and checks the volume with sharesVolume() in src/kernel/vitals.ts. For a path that doesn't exist yet, it checks the nearest parent that does.
  • A snapshot store reports its directory (Store.dir). A directory store sets it, the retry wrapper passes it through, and a bucket has none.
  • The health answer, the board's forge strip and the asks each built vitals the same way on their own; they now call forgeVitals. The degraded-history tick in src/main.ts passes snapshotPlace(app).

I left out the edge's "the forge is restarting" page, the Resilience page served from the edge, and the rule that losing the runner never blocks a fix release. Those are #1194 and #1196, which already wait on #1148.

Tests: test/vitals-snapshots.test.ts, the two terminal-health tests in test/tui-screens.test.ts, and the domains and gate-list suites pass. The type check is clean except for four errors in scripts/hollow.ts, which main has too. The "two volumes apart" test is skipped where /proc/self doesn't exist.

To try it: on a forge whose Backups destination is a directory under FORGE_DATA and has no mirror set, an admin sees the warning on the board, the terminal dashboard and /api/v1/health. Setting a mirror clears it.

🤖 Generated with Claude Code

Approved

Approved from the diff and tests. I could not reach the preview from my sandbox, so I did not check it in a browser or read the screenshots.

The change does what #1148 part 2 asks. It adds one detector for snapshots that share the data volume with no mirror. Every vitals reader now goes through forgeVitals, so the board, asks, health API, MCP and terminal agree. The terminal test shows the warning to an admin only and clears it when a mirror is set. The warning is one line, shown to the admin only, with a link to Settings → Backups.

Two things I would do next:

  1. Move the .gitignore stub entries into their own change. They are not part of this pull. /.claude/ and /.vscode also hide files a person may legitimately want to commit.
  2. Give the board strip a test that asserts the Backups link. Only the dashboard text and the vitals unit are covered today. Also let tickAsks in src/main.ts use forgeVitals, which would drop the one remaining separate vitals call.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs