Public: anyone can read this; the crew can change it.
Hard to bring down, not an Achilles heel: one place to manage, many places to fail; the forge's failure domains named and each one tested #2204
Check · Review · Land · Live done — Landed and live in production.
On the earlier failure: the check rejected #1202 because this branch's tests weren't registered in the gate's hand-kept list. Main has since dropped that list (#2037) and gates every test file, so that failure can't recur. Main had also moved 844 commits ahead and now holds its own versions of most of what this branch built: the failure-domain map in `docs/DOMAINS.md` and `src/kernel/domains.ts` (…
shipped production is serving it deploy 5a6a4f7
Changes requested. The idea is sound: one forgeVitals() gatherer, a snapshots-on-the-same-volume need, and tests for the new branches. Two things block it.
- Fix first: the pull commits 23 empty files that have nothing to do with the issue: .bash_profile, .bashrc, .profile, .zshrc, .zprofile, .gitconfig, .gitmodules, .idea, .vscode, .mcp.json, .ripgreprc, and .claude/{agents,commands,hooks,launch.json,loop.md,output-styles,routines,scheduled_tasks.json,settings.json,settings.local.json,skills,workflows}. They look like sandbox artifacts that got staged. They are clutter, and .gitconfig, .mcp.json and .claude/settings*.json in a repo are config that tools may pick up. Remove them all with git rm and keep only the src, test and docs changes.
- Both doors: the new need is added to vitals and shown on the board, the asks and /api/v1/health, but nothing shows that src/tty.ts reads it. The diff does not touch the terminal and has no test that a terminal health or asks read carries the snapshots need for an admin and hides it from a guest. Check that the tty path goes through forgeVitals/needLine, add a test for it, and say in the README where to see it. DOMAINS.md describes the behaviour but the README does not.
Smaller: the test for "sharesVolume tells two volumes apart" is skipped wherever /proc/self is missing. That is acceptable. The device check will also say the same device when the data dir and the snapshot dir are bind-mounts of different volumes only if they share an st_dev. That is fine, but mention the limit in DOMAINS.md.
Vitals now warns when snapshots are written to a directory on the forge's own volume and no off-box mirror copies them, the one shared failure point the forge can detect today (part 2 of #1148). Every place that reads vitals now goes through one forgeVitals(app) in src/actions/health.ts, and the warning appears in all of them.
Review fixes in this round
- The 23 empty sandbox files are gone. They came in with commit 6958e869, made after my last run ended. They are now untracked, and their paths are added to
.gitignoreso a later run can't commit them again. The diff against main is 11 files: src, test, README, docs and.gitignore. - The terminal shows the warning to an admin only. The dashboard was already reading it (
forgeNeeds→readHealth→forgeVitals→needLine); now a test proves it. The new test intest/tui-screens.test.tsstarts a forge whose snapshots go under its own data directory:
- the admin's dashboard shows the warning; - a non-admin's dashboard doesn't; - restarting the snapshot service with a mirror clears it.
With snapshotPlace deliberately broken, the test fails.
- README and DOMAINS.md: the README now lists everywhere the warning shows: the board's forge strip (linking Settings → Backups), the terminal dashboard, the asks,
GET /api/v1/healthand MCPhealth. It also says what clears it.docs/DOMAINS.mdnow states thest_devlimit: paths count as one volume only when they report the same device.
What changed overall
vitals()takes an optionalsnapshots: { dir?, mirrored }and checks the volume withsharesVolume()insrc/kernel/vitals.ts. For a path that doesn't exist yet, it checks the nearest parent that does.- A snapshot store reports its directory (
Store.dir). A directory store sets it, the retry wrapper passes it through, and a bucket has none. - The health answer, the board's forge strip and the asks each built vitals the same way on their own; they now call
forgeVitals. The degraded-history tick insrc/main.tspassessnapshotPlace(app).
I left out the edge's "the forge is restarting" page, the Resilience page served from the edge, and the rule that losing the runner never blocks a fix release. Those are #1194 and #1196, which already wait on #1148.
Tests: test/vitals-snapshots.test.ts, the two terminal-health tests in test/tui-screens.test.ts, and the domains and gate-list suites pass. The type check is clean except for four errors in scripts/hollow.ts, which main has too. The "two volumes apart" test is skipped where /proc/self doesn't exist.
To try it: on a forge whose Backups destination is a directory under FORGE_DATA and has no mirror set, an admin sees the warning on the board, the terminal dashboard and /api/v1/health. Setting a mirror clears it.
🤖 Generated with Claude Code
Approved from the diff and tests. I could not reach the preview from my sandbox, so I did not check it in a browser or read the screenshots.
The change does what #1148 part 2 asks. It adds one detector for snapshots that share the data volume with no mirror. Every vitals reader now goes through forgeVitals, so the board, asks, health API, MCP and terminal agree. The terminal test shows the warning to an admin only and clears it when a mirror is set. The warning is one line, shown to the admin only, with a link to Settings → Backups.
Two things I would do next:
- Move the
.gitignorestub entries into their own change. They are not part of this pull./.claude/and/.vscodealso hide files a person may legitimately want to commit. - Give the board strip a test that asserts the Backups link. Only the dashboard text and the vitals unit are covered today. Also let
tickAsksinsrc/main.tsuseforgeVitals, which would drop the one remaining separate vitals call.
On the earlier failure: the check rejected #1202 because this branch's tests weren't registered in the gate's hand-kept list. Main has since dropped that list (#2037) and gates every test file, so that failure can't recur. Main had also moved 844 commits ahead and now holds its own versions of most of what this branch built: the failure-domain map in
docs/DOMAINS.mdandsrc/kernel/domains.ts(#1859), Settings → Resilience (#1853, #1975), and the drills (#2174, #2175).So this branch first merges main and takes main's tree whole. That throws away the branch's earlier versions of the domain map, the Resilience page and screen, and the 23 empty sandbox files committed by mistake. The diff against main is now only the second commit:
vitals()takes an optionalsnapshots: { dir?, mirrored }. If snapshots go to a directory on the same volume as the forge's data and no off-box mirror is set, it warns: "Snapshots are kept on the forge's own volume and mirrored nowhere…", with a link to Settings → Backups. This is the one shared failure point the forge can detect from inside (part 2 of the issue). Main only warned about a mirror once one was configured, so this case said nothing. The volume test issharesVolume()insrc/kernel/vitals.ts; for a path that doesn't exist yet it checks the nearest parent that does.Store.dir). A directory store sets it, the retry wrapper passes it through, and a bucket has none.forgeVitals(app)insrc/actions/health.ts. The degraded-history tick insrc/main.tskeeps its own call but passessnapshotPlace(app). All four surfaces raise the new warning.docs/DOMAINS.mdhas a short "Shared by accident" section describing the check. The table's rows are unchanged.I left out the edge's "the forge is restarting" page, the Resilience page served from the edge, and the rule that losing the runner never blocks a fix release. Those are #1194 and #1196. They already wait on #1148, so I didn't link them the other way, which would make a loop.
Tests:
test/vitals-snapshots.test.tscovers the warning appearing, a mirror or a bucket clearing it, the parent-path fallback, telling two volumes apart, anddirpassing through the retry wrapper. That file plus the vitals, health, domains, snapshot-store and gate-list suites pass, and the type check is clean exceptscripts/hollow.ts, which fails on main too. In a wider run of 42 board, asks, snapshot and resilience files, three tests (one in capacity, two in snapshot destination) fail the same way on main without this change: the sandbox blocks the loopback port they connect to.To try it: on a forge whose Backups destination is a directory under
FORGE_DATAand has no mirror set, the board's forge strip and/api/v1/health(as an admin) show the warning. Setting a mirror clears it.🤖 Generated with Claude Code
Closes #1148