trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns13g rEventsFlows

Hard to bring down, not an Achilles heel: one place to manage, many places to fail; the forge's failure domains named and each one tested #2204

Merged masoncrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

mason/forge:mason/1148-hard-to-bring-down-not-an-achilles-heel → main 14 commits fork 2 threads · 2 unresolved

  • Merged. mason/1148-hard-to-bring-down-not-an-achilles-heel was merged into main as 9b68d0d.

Review

Sign in to review this change.
11 files changed +251 −44
.gitignore +15
@@ -22,3 +22,18 @@ typesafe-api-key
2222
2323 # what a check keeps for the next one (src/checks/cache.ts)
2424 /.forge-cache/
25
26 # empty stubs an agent sandbox mounts over home-directory config; never the
junipercrew

Unrelated to the issue. Ignoring /.claude/, /.vscode and /.idea repo-wide can hide config a person wants to commit. Split this into its own change, or scope it to the sandbox stub files only.
27 # forge's own files, and never committed (#1148)
28 /.bash_profile
29 /.bashrc
30 /.profile
31 /.zshrc
32 /.zprofile
33 /.gitconfig
34 /.gitmodules
35 /.idea
36 /.vscode
37 /.mcp.json
38 /.ripgreprc
39 /.claude/
README.md +9
@@ -699,6 +699,15 @@ wakes every 10 minutes for it. The time the forge first owed a drill is kept
699699 at boot in `drill/first-seen`; if the disk will not take it, a warning is
700700 logged and a forge with no drill ever is not flagged.
701701
702 Snapshots written to a directory on the forge's own volume, with no off-box
703 mirror set, are lost with that volume. That is one line for the forge's admin,
704 "Snapshots are kept on the forge's own volume and mirrored nowhere: losing the
705 volume loses them too.", on the board's forge strip (linking Settings →
706 Backups), in the terminal's dashboard (same `forgeNeeds`), in the asks, in
707 `GET /api/v1/health` and MCP `health`. Anyone else sees nothing. It clears when
708 the destination moves to another volume or a bucket, or a mirror is set. Every
709 one of those reads the vitals through `forgeVitals` in `src/actions/health.ts`.
710
702711 `docs/EXTRACT.md` is the forge on a machine of its own, and the platform it
703712 grew up in switched off: what a host owes it, the moves in order, and what is
704713 given up.
docs/DOMAINS.md +13
@@ -19,6 +19,19 @@ keeps a drill's verdict for any domain here (see the README).
1919 | The model credential | Agents cannot start work; everything else runs. | nothing yet | 5 | Set a fresh credential in the settings. | expire-token |
2020 | The runner | Checks and runs queue behind a lane that is not moving. | "wait behind one lane; the oldest since"; "is stuck: no heartbeat for" | 10 | Cancel the stuck run; restart the runner if the lane stays empty. | lose-runner |
2121
22 ## Shared by accident
23
24 Two domains that fail together are one domain. The forge checks the one
25 pairing it can see from inside: snapshots written to a directory on the
26 volume they copy, with no off-box mirror, are lost with that volume. Vitals
27 then says "Snapshots are kept on the forge's own volume and mirrored
28 nowhere" on the board, the asks and `/api/v1/health`, and links Settings →
29 Backups, until the destination moves or a mirror is set. "One volume" means the
30 two paths report the same device (`st_dev`): a snapshot directory on a
31 different filesystem bind-mounted into the data directory is told apart, but
32 two directories on one disk under different mounts that share a device, or
33 two devices backed by the same physical disk, are not.
34
2235 ## Running a drill
2336
2437 `scripts/drills/run.ts` walks you through one. It refuses a domain whose drill
src/actions/health.ts +37 −12
@@ -13,7 +13,42 @@ import { appsRuntime } from "../apps/runtime.ts";
1313 import { listApps } from "../core/apps.ts";
1414 import type { User } from "../core/users.ts";
1515 import { forgeHealth, rolloutLine, type ForgeHealth } from "../kernel/guard.ts";
16 import { laneLine, sinceWords, vitals, type Degraded, type Vitals } from "../kernel/vitals.ts";
16 import {
17 laneLine,
18 sinceWords,
19 vitals,
20 type Degraded,
21 type Vitals,
22 type VitalsInput,
23 } from "../kernel/vitals.ts";
24 import { snapshotsOf } from "../snapshot/service.ts";
25
26 /** Where this forge's snapshots go, as vitals reads it; undefined when it takes none. */
27 export function snapshotPlace(app: App): VitalsInput["snapshots"] {
28 const handle = snapshotsOf(app);
29 if (handle === undefined) return undefined;
30 const dir = handle.store?.dir;
31 return { ...(dir === undefined ? {} : { dir }), mirrored: handle.mirror !== undefined };
32 }
33
34 /**
35 * This forge's vitals, as every surface reads them: the health answer, the
36 * board's forge strip and the asks. One place gathers what they measure.
37 */
38 export function forgeVitals(app: App): Vitals {
39 const runtime = appsRuntime(app);
40 const envs = runtime
41 ? listApps(app.db).flatMap((record) => runtime.status(record))
42 : [];
43 return vitals({
44 db: app.db,
45 dataDir: app.config.dataDir,
46 envs,
47 signals: app.signals,
48 forgeUrl: app.config.url ?? `http://localhost:${app.config.port}`,
49 snapshots: snapshotPlace(app),
50 });
51 }
1752
1853 export function readHealth(
1954 app: App,
@@ -21,17 +56,7 @@ export function readHealth(
2156 ): ForgeHealth | (Vitals & ForgeHealth) {
2257 const forge = forgeHealth();
2358 if (!actor?.isAdmin) return forge;
24 const runtime = appsRuntime(app);
25 const envs = runtime
26 ? listApps(app.db).flatMap((record) => runtime.status(record))
27 : [];
28 const v = vitals({
29 db: app.db,
30 dataDir: app.config.dataDir,
31 envs,
32 signals: app.signals,
33 forgeUrl: app.config.url ?? `http://localhost:${app.config.port}`,
34 });
59 const v = forgeVitals(app);
3560 return { ...v, ...forge, ok: forge.ok && v.ok };
3661 }
3762
src/kernel/vitals.ts +33 −1
@@ -9,7 +9,8 @@
99 */
1010
1111 import type { Database } from "bun:sqlite";
12 import { statfsSync } from "node:fs";
12 import { statfsSync, statSync } from "node:fs";
13 import { dirname } from "node:path";
1314 import { freemem, totalmem } from "node:os";
1415 import { refusedInternal, type RefusedLedger } from "./internal_refused.ts";
1516 import { recentErrors, type ErrorLedger, type ErrorLine } from "./log.ts";
@@ -36,6 +37,9 @@ export const ERROR_BURST = 20;
3637 export const NO_PROGRESS_MS = 6 * 3_600_000;
3738 /** A run that has waited this long to start is a queue that is not moving. */
3839 export const QUEUE_WAIT_MAX_MS = 30 * 60_000;
40 /** Said when snapshots share the volume they are a copy of and nothing copies them off it. */
41 export const SNAPSHOTS_ON_STATE_VOLUME =
42 "Snapshots are kept on the forge's own volume and mirrored nowhere: losing the volume loses them too.";
3943
4044 export interface EnvCount {
4145 readonly env: string;
@@ -104,6 +108,25 @@ export interface VitalsInput {
104108 readonly diskFreeMin?: number;
105109 /** One forge's own signals (`App.signals`). */
106110 readonly signals: { errors: ErrorLedger; refused: RefusedLedger };
111 /** Where snapshots go: the directory, when the store is one, and whether an off-box mirror copies them. */
112 readonly snapshots?: { readonly dir?: string; readonly mirrored: boolean } | undefined;
113 }
114
115 /** The device a path lives on, read from it or its nearest parent that exists. */
116 function deviceOf(path: string): number | undefined {
117 for (let at = path; ; at = dirname(at)) {
118 try {
119 return statSync(at).dev;
120 } catch {
121 if (dirname(at) === at) return undefined;
122 }
123 }
124 }
125
126 /** Whether two paths live on one volume: what loses one loses the other. */
127 export function sharesVolume(a: string, b: string): boolean {
128 const da = deviceOf(a);
129 return da !== undefined && da === deviceOf(b);
107130 }
108131
109132 /**
@@ -250,6 +273,15 @@ export function vitals(input: VitalsInput): Vitals {
250273 "/settings/capacity",
251274 );
252275
276 // Snapshots on the volume they copy, and no mirror, share its one point of failure.
277 const snaps = input.snapshots;
278 if (
279 snaps?.dir !== undefined &&
280 !snaps.mirrored &&
281 sharesVolume(input.dataDir, snaps.dir)
282 )
283 need("snapshots", SNAPSHOTS_ON_STATE_VOLUME, undefined, "/settings/backups");
284
253285 const running = listRunning(input.db);
254286 const lane = readLane(input.db, running, now);
255287 const queue: Record<string, number> = {};
src/main.ts +2
@@ -53,6 +53,7 @@ import { configProblem, dataDirOf, preflight } from "./kernel/preflight.ts";
5353 import { running } from "./kernel/version.ts";
5454 import { record as recordDegraded } from "./kernel/degraded_history.ts";
5555 import { vitals } from "./kernel/vitals.ts";
56 import { snapshotPlace } from "./actions/health.ts";
5657
5758 // `forge hook <name>` re-enters here from a git hook stub. It relays to the
5859 // running forge over its socket and exits — it must never boot a server.
@@ -447,6 +448,7 @@ const tickAsks = async (): Promise<void> => {
447448 dataDir: app.config.dataDir,
448449 signals: app.signals,
449450 forgeUrl,
451 snapshots: snapshotPlace(app),
junipercrew

This is the one reader that still builds vitals by hand. Use forgeVitals(app).degraded here so a new input cannot be forgotten again.
450452 }).degraded,
451453 Date.now(),
452454 app.log,
src/snapshot/store.ts +4
@@ -12,6 +12,8 @@ import { dirname, join, relative } from "node:path";
1212 export interface Store {
1313 /** What this store is, for a page and a log; never a secret. */
1414 readonly where: string;
15 /** The directory it writes under, when it is one; a bucket has none. */
16 readonly dir?: string;
1517 put(key: string, body: Uint8Array | string): Promise<void>;
1618 get(key: string): Promise<Uint8Array | undefined>;
1719 /** Every key under `prefix`, in no particular order. */
@@ -90,6 +92,7 @@ export function withRetries(
9092 }
9193 return {
9294 where: inner.where,
95 ...(inner.dir === undefined ? {} : { dir: inner.dir }),
9396 put: (key, body) => tried(`put ${key}`, () => inner.put(key, body)),
9497 get: (key) => tried(`get ${key}`, () => inner.get(key)),
9598 list: (prefix) => tried(`list ${prefix}`, () => inner.list(prefix)),
@@ -192,6 +195,7 @@ export function dirStore(root: string): Store {
192195 };
193196 return {
194197 where: root,
198 dir: root,
195199 async put(key, body) {
196200 const path = at(key);
197201 mkdirSync(dirname(path), { recursive: true });
src/web/pages/board.ts +4 −16
@@ -19,7 +19,7 @@ import { appsForRepos, type AppRecord } from "../../core/apps.ts";
1919 import { askMeasure } from "../../core/ask_ledger.ts";
2020 import { reviewersOf, waitsForPerson } from "../../crew/review.ts";
2121 import { tokenReturns } from "../../core/returns.ts";
22 import { needLine } from "../../actions/health.ts";
22 import { forgeVitals, needLine } from "../../actions/health.ts";
2323 import { returnsRepos } from "../../actions/returns.ts";
2424 import {
2525 deciderCounts,
@@ -31,7 +31,6 @@ import {
3131 type Decision,
3232 type DecisionKind,
3333 } from "../../core/decisions.ts";
34 import { appsRuntime } from "../../apps/runtime.ts";
3534 import { efficiencyForRepos } from "../../core/efficiency.ts";
3635 import { reviewStats, type ReviewStats } from "../../core/review-stats.ts";
3736 import { standings, WINDOW_MS } from "../../core/standing.ts";
@@ -44,9 +43,8 @@ import {
4443 import { readSpend } from "../../actions/spend.ts";
4544 import { spendView } from "./spend.ts";
4645 import type { HandContext } from "../../crew/handover.ts";
47 import { listApps } from "../../core/apps.ts";
4846 import { forgeHealth, rolloutLine } from "../../kernel/guard.ts";
49 import { laneLine, vitals, type Vitals } from "../../kernel/vitals.ts";
47 import { laneLine, type Vitals } from "../../kernel/vitals.ts";
5048 import { getIssue, getIssuesByIds, type Issue } from "../../core/issues.ts";
5149 import { looksForApps } from "../../core/looks.ts";
5250 import {
@@ -328,17 +326,7 @@ function sentinelSpan(app: Ctx["app"]): Html {
328326 */
329327 function forgeStrip(ctx: Ctx): Html | null {
330328 if (!ctx.auth.user?.isAdmin) return null;
331 const app = ctx.app;
332 const runtime = appsRuntime(app);
333 const v = vitals({
334 db: app.db,
335 dataDir: app.config.dataDir,
336 signals: app.signals,
337 forgeUrl: app.config.url ?? `http://localhost:${app.config.port}`,
338 envs: runtime
339 ? listApps(app.db).flatMap((record) => runtime.status(record))
340 : [],
341 });
329 const v = forgeVitals(ctx.app);
342330 const fh = forgeHealth();
343331 const rollout = rolloutLine(fh);
344332 const others = fh.incidents.filter(
@@ -354,7 +342,7 @@ function forgeStrip(ctx: Ctx): Html | null {
354342 data-rollout="${fh.rollout.state}"
355343 role="status"
356344 >
357 ${sentinelSpan(app)}
345 ${sentinelSpan(ctx.app)}
358346 ${rollout === null ? "" : html`<span class="board-forge-need">${rollout}</span>`}
359347 ${others.map((i) => html`<span class="board-forge-need">${i.key}: ${i.message}</span>`)}
360348 ${lane === null ? "" : html`<span class="board-forge-lane">Lane: ${lane}</span>`}
src/web/studio/asks.ts +3 −15
@@ -8,14 +8,13 @@
88 */
99
1010 import type { Database } from "bun:sqlite";
11 import { needLine } from "../../actions/health.ts";
11 import { forgeVitals, needLine } from "../../actions/health.ts";
1212 import type { Degraded } from "../../kernel/vitals.ts";
1313 import { previewEnv } from "../../apps/data.ts";
1414 import { waitingFor, type WaitingFor } from "../../core/waiting_for.ts";
1515 import { crewForRepos } from "../../core/agents.ts";
1616 import { capacityDefault, isCapacityKey, storedCapacity } from "../../core/capacity.ts";
17 import { appsRuntime } from "../../apps/runtime.ts";
18 import { appsForRepos, getAppByRepo, listApps } from "../../core/apps.ts";
17 import { appsForRepos, getAppByRepo } from "../../core/apps.ts";
1918 import { latestDecision, stopStands } from "../../core/decisions.ts";
2019 import { getHold, holdsFor } from "../../core/holds.ts";
2120 import { issueTitles, proposalsFor, type Proposal } from "../../core/proposals.ts";
@@ -62,7 +61,6 @@ import { handoverLines, handoverOf, type Handover } from "../../crew/guard.ts";
6261 import { conflictsOf, isStalePull, mergesAgainSince } from "../../crew/stale_asks.ts";
6362 import { reviewAttempts, waitsForPerson, waitsForReviewer, type ReviewWait } from "../../crew/review.ts";
6463 import type { Ctx } from "../../http/context.ts";
65 import { vitals } from "../../kernel/vitals.ts";
6664 import { repoBase } from "../chrome.ts";
6765 import { html, type Html } from "../html.ts";
6866 import { btn, card, icon, row } from "../ui.ts";
@@ -1338,17 +1336,7 @@ export function decisionButtons(ctx: Ctx, d: Decision): { primary: Html; others:
13381336 */
13391337 export function forgeNeedsFor(ctx: Ctx, user: User | undefined): Degraded[] {
13401338 if (!user?.isAdmin) return [];
1341 const runtime = appsRuntime(ctx.app);
1342 const envs = runtime
1343 ? listApps(ctx.app.db).flatMap((record) => runtime.status(record))
1344 : [];
1345 return vitals({
1346 db: ctx.app.db,
1347 dataDir: ctx.app.config.dataDir,
1348 signals: ctx.app.signals,
1349 forgeUrl: ctx.app.config.url ?? `http://localhost:${ctx.app.config.port}`,
1350 envs,
1351 }).degraded;
1339 return forgeVitals(ctx.app).degraded;
13521340 }
13531341
13541342 /**
test/tui-screens.test.ts +61
@@ -25,6 +25,8 @@ import {
2525 } from "../src/actions/repos.ts";
2626 import { addSshKey, signUp } from "../src/actions/users.ts";
2727 import { createApp, type App as Forge } from "../src/app.ts";
28 import { startSnapshots } from "../src/snapshot/service.ts";
29 import { dirStore } from "../src/snapshot/store.ts";
2830 import { listEvents } from "../src/core/events.ts";
2931 import { getIssue, listComments } from "../src/core/issues.ts";
3032 import { getPull } from "../src/core/pulls.ts";
@@ -789,4 +791,63 @@ describe("the forge's health in the terminal", () => {
789791 await rm(guard, { recursive: true, force: true });
790792 }
791793 });
794
795 test("snapshots on the forge's own volume, mirrored nowhere: the admin's dashboard says so, nobody else's does", async () => {
796 const dataDir = await mkdtemp(join(tmpdir(), "forge-tui-snapvol-"));
797 const guard = await mkdtemp(join(tmpdir(), "forge-tui-guard-"));
798 process.env["FORGE_GUARD_DIR"] = guard;
799 const forge = createApp({ dataDir, port: 0 });
800 const store = dirStore(join(dataDir, "snapshots"));
801 let svc = startSnapshots(forge, store, { version: "t", nightly: false });
802 try {
803 const a = await signUp(forge, {
804 username: "chief",
805 email: "[email protected]",
806 password: "a-long-passphrase",
807 });
808 const b = await signUp(forge, {
809 username: "bob",
810 email: "[email protected]",
811 password: "another-long-passphrase",
812 });
813 if (!a.ok || !b.ok) throw new Error("signup failed");
814 const as = (user: User): TuiIdentity => ({
815 sshUsername: user.username,
816 remoteAddress: "127.0.0.1",
817 fingerprint: "SHA256:test",
818 user,
819 });
820 const said = "Snapshots are kept on the forge's own volume";
821
822 const admin = boot(forge, as(a.value));
823 await admin.settle();
824 expect(admin.text()).toContain(said);
825 expect(admin.text()).not.toContain("All fine");
826 admin.stop();
827
828 const other = boot(forge, as(b.value));
829 await other.settle();
830 expect(other.text()).not.toContain(said);
831 other.stop();
832
833 // A mirror copying them off the box clears it.
834 await svc.stop();
835 svc = startSnapshots(forge, store, {
836 version: "t",
837 nightly: false,
838 mirror: dirStore(join(dataDir, "mirror")),
839 mirrorEveryMs: 3_600_000,
840 });
841 const mirrored = boot(forge, as(a.value));
842 await mirrored.settle();
843 expect(mirrored.text()).not.toContain(said);
844 mirrored.stop();
845 } finally {
846 await svc.stop();
847 delete process.env["FORGE_GUARD_DIR"];
848 forge.close();
849 await rm(dataDir, { recursive: true, force: true });
850 await rm(guard, { recursive: true, force: true });
851 }
852 });
792853 });
test/vitals-snapshots.test.ts added +70
@@ -0,0 +1,70 @@
1 /**
2 * Snapshots kept on the volume they copy, with no off-box mirror, share its
3 * one point of failure: vitals says so, and says nothing once they live
4 * elsewhere or a mirror copies them off (#1148).
5 */
6
7 import { afterAll, beforeAll, expect, test } from "bun:test";
8 import { existsSync } from "node:fs";
9 import { mkdtemp, rm } from "node:fs/promises";
10 import { tmpdir } from "node:os";
11 import { join } from "node:path";
12 import { createApp, type App } from "../src/app.ts";
13 import { forgeVitals, snapshotPlace } from "../src/actions/health.ts";
14 import { sharesVolume, SNAPSHOTS_ON_STATE_VOLUME, vitals } from "../src/kernel/vitals.ts";
15 import { dirStore, withRetries } from "../src/snapshot/store.ts";
16
17 let dataDir: string;
18 let app: App;
19
20 beforeAll(async () => {
21 dataDir = await mkdtemp(join(tmpdir(), "forge-vitals-snapshots-"));
22 app = createApp({ dataDir, port: 0 });
23 });
24
25 afterAll(async () => {
26 app.db.close();
27 await rm(dataDir, { recursive: true, force: true });
28 });
29
30 const read = (snapshots?: { dir?: string; mirrored: boolean }) =>
31 vitals({ db: app.db, signals: app.signals, dataDir, snapshots });
32 const flagged = (v: ReturnType<typeof read>) =>
33 v.degraded.find((d) => d.cause === "snapshots");
34
35 test("a snapshot directory on the data volume, mirrored nowhere, is a need pointing at Backups", () => {
36 const v = read({ dir: join(dataDir, "snapshots"), mirrored: false });
37 expect(v.needs).toContain(SNAPSHOTS_ON_STATE_VOLUME);
38 expect(flagged(v)?.href).toBe("/settings/backups");
39 });
40
41 test("a mirror copying them off the box clears it", () => {
42 expect(flagged(read({ dir: join(dataDir, "snapshots"), mirrored: true }))).toBeUndefined();
43 });
44
45 test("a bucket store, which has no directory, is never flagged", () => {
46 expect(flagged(read({ mirrored: false }))).toBeUndefined();
47 });
48
49 test("a forge told nothing about snapshots is never flagged", () => {
50 expect(flagged(read())).toBeUndefined();
51 });
52
53 test("sharesVolume reads a path that does not exist yet by its nearest parent", () => {
54 expect(sharesVolume(dataDir, join(dataDir, "not", "made", "yet"))).toBe(true);
55 });
56
57 test.skipIf(!existsSync("/proc/self"))("sharesVolume tells two volumes apart", () => {
58 expect(sharesVolume(dataDir, "/proc/self")).toBe(false);
59 });
60
61 test("a directory store names its directory, through the retry wrapper too", () => {
62 const root = join(dataDir, "store");
63 expect(dirStore(root).dir).toBe(root);
64 expect(withRetries(dirStore(root)).dir).toBe(root);
65 });
66
67 test("a forge running no snapshot service has no place, and forgeVitals carries no such need", () => {
68 expect(snapshotPlace(app)).toBeUndefined();
69 expect(forgeVitals(app).degraded.some((d) => d.cause === "snapshots")).toBe(false);
70 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs