trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns13g rEventsFlows

A forge rollout never restarts an app that did not change: apps keep running (or hand over) through the forge's own deploys #4510

Merged torquecrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/4496-a-forge-rollout-never-restarts-an-app-th → main 3 commits fork 3 threads · 3 unresolved

  • Merged. torque/4496-a-forge-rollout-never-restarts-an-app-th was merged into main as b4281f2.

Review

Sign in to review this change.
14 files changed +881 −19
docs/APPS.md +26
@@ -772,6 +772,32 @@ a forge coming up fails only runs whose lease has lapsed, so the old
772772 process's live work is never mistaken for a crash's leftovers, and a real
773773 crash is still cleaned up within a minute or two.
774774
775 An app goes through a release without a refused connection (#4496). A
776 forge with apps on beats `serving/<id>.json` on the data volume every 5 s
777 (`apps/presence.ts`). A new forge that finds another still beating — fresh
778 within 15 s, and on the same host a pid that runs — starts every app's
779 production and answers `/readyz` 503 (`starting N apps`) until each serves
780 or fails, for up to two minutes (`STARTING_BUDGET_MS`, `http/health.ts`);
781 the platform keeps routing to the old forge, whose copies still serve,
782 until then. A cold start — a crash, an eviction, the first boot — finds
783 nobody serving in its place and is ready at once, as is a forge whose
784 predecessor stops beating during the wait. The old forge stops
785 its apps only after its linger, once routers have moved: each proxied
786 websocket is closed with 1012 Service Restart (`apps/handoff.ts`) — a
787 client reconnects at once, to the new forge — and each request still with
788 the app is given up to 30 s to be answered before the app is stopped. An
789 app's own deploy hands over the same way: the new version serves first,
790 then the old one's sockets get 1012 and its requests their 30 s. The
791 **App restarts** board counts every production start over one that
792 served, per day and per app, with its cause: its own deploy, a forge
793 roll, a crash, or the memory cap (`oom`).
794
795 The app's process still restarts on a forge release: it lives in the
796 forge's pod, and the old pod's processes go with it. A forge roll is
797 counted on that board so the rate is visible; keeping the process itself
798 running across a roll needs the cells supervised outside the forge
799 process, which this does not do.
800
775801 ## Where state lives
776802
777803 Git is truth for code. The app's serving configuration (slug, access, the
src/apps/handoff.ts added +100
@@ -0,0 +1,100 @@
1 /**
2 * What the proxy has in flight to one deployment, by the loopback port it
3 * listens on — so a deployment that is being replaced can be let go of
4 * without dropping what it was doing (#4496).
5 *
6 * A request passed to a port is held until its answer begins; a proxied
7 * websocket is held until it closes. `drainPort` is what happens to the
8 * old deployment once the new one serves (its own deploy) or once the
9 * forge that proxies it is leaving (a forge rollout): every socket on it
10 * is closed with 1012 Service Restart — RFC 6455 §7.4.1 via the IANA
11 * registry: "the server is restarting, a client may reconnect" — so a
12 * browser reconnects at once, to whatever serves now; and the requests
13 * still waiting on it are given up to the budget to be answered before
14 * the process is stopped.
15 *
16 * Plain process memory, keyed by port: a port is unique among the live
17 * deployments of one forge (`apps/process.ts` `freePort`).
18 */
19
20 /** RFC 6455 close code 1012: the service is restarting; reconnect. */
21 export const SERVICE_RESTART = 1012;
22 export const SERVICE_RESTART_REASON = "service restart";
23
24 /** How long a replaced deployment keeps answering what it already took (#4496). */
25 export const DRAIN_BUDGET_MS = 30_000;
26 const POLL_MS = 25;
27
28 interface Held {
29 requests: number;
30 readonly sockets: Set<{ close(code: number, reason: string): void }>;
31 }
32
33 const ports = new Map<number, Held>();
34
35 function heldAt(port: number): Held {
36 let held = ports.get(port);
37 if (held === undefined) {
38 held = { requests: 0, sockets: new Set() };
39 ports.set(port, held);
40 }
41 return held;
42 }
43
44 function forget(port: number, held: Held): void {
45 if (held.requests === 0 && held.sockets.size === 0 && ports.get(port) === held)
46 ports.delete(port);
47 }
48
49 /** A request is on its way to `port`; the returned release is called once its answer begins. */
50 export function holdPortRequest(port: number): () => void {
51 const held = heldAt(port);
52 held.requests++;
53 let open = true;
54 return () => {
55 if (!open) return;
56 open = false;
57 held.requests--;
58 forget(port, held);
59 };
60 }
61
62 /** A websocket through to `port` is open; the returned release is called when it closes. */
63 export function holdPortSocket(
64 port: number,
65 socket: { close(code: number, reason: string): void },
66 ): () => void {
67 const held = heldAt(port);
68 held.sockets.add(socket);
69 return () => {
70 held.sockets.delete(socket);
71 forget(port, held);
72 };
73 }
74
75 /** What is in flight to `port` this instant. */
76 export function inFlight(port: number): { requests: number; sockets: number } {
77 const held = ports.get(port);
78 return { requests: held?.requests ?? 0, sockets: held?.sockets.size ?? 0 };
79 }
80
81 /**
82 * Let go of the deployment on `port`: close its sockets with 1012, then wait
83 * — at most `budgetMs` — for the requests it is still answering. Resolves
84 * with how many requests were still waiting when it gave up (0 when all
85 * were answered).
86 */
87 export async function drainPort(port: number, budgetMs = DRAIN_BUDGET_MS): Promise<number> {
88 const held = ports.get(port);
89 if (held === undefined) return 0;
90 for (const socket of [...held.sockets]) {
91 try {
92 socket.close(SERVICE_RESTART, SERVICE_RESTART_REASON);
93 } catch {
94 // Already closing: its own close handler releases it.
95 }
96 }
97 const deadline = Date.now() + budgetMs;
98 while (held.requests > 0 && Date.now() < deadline) await Bun.sleep(POLL_MS);
99 return held.requests;
100 }
src/apps/presence.ts added +138
@@ -0,0 +1,138 @@
1 /**
2 * Which forges are serving apps from this data directory right now (#4496).
3 *
4 * A forge with apps on writes `serving/<id>.json` — its host, pid and the
5 * time — every `BEAT_MS`, and removes it once its apps are stopped. A new
6 * forge reads the others: one that beat within `STALE_MS` (and, on this
7 * host, whose pid still runs) is a predecessor still serving its copies of
8 * the apps, the surge pod of a rollout. Only then is it worth being
9 * unready while this forge's own copies start (`warmBehind`); a cold start
10 * — a crash, an eviction, the first boot — has nobody serving in its
11 * place, and is ready at once.
12 *
13 * Plain JSON written by rename, one file per process, so two forges never
14 * write the same file.
15 */
16
17 import { randomUUID } from "node:crypto";
18 import { mkdirSync, readdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
19 import { hostname } from "node:os";
20 import { join } from "node:path";
21 import { startWarming } from "../http/health.ts";
22 import type { Log } from "../kernel/log.ts";
23
24 export const BEAT_MS = 5_000;
25 /** Three beats missed: the forge that wrote it is gone. */
26 export const STALE_MS = 3 * BEAT_MS;
27
28 export interface Beat {
29 readonly id: string;
30 readonly host: string;
31 readonly pid: number;
32 /** ms since the epoch. */
33 readonly at: number;
34 }
35
36 export interface Presence {
37 /** Whether another forge is serving this data's apps this instant. */
38 others(now?: number): boolean;
39 /** Stop beating and remove this forge's file. Idempotent. */
40 stop(): void;
41 }
42
43 const dirOf = (dataDir: string): string => join(dataDir, "serving");
44 const ID = /^[0-9a-f-]{36}$/;
45
46 function parse(raw: string): Beat | undefined {
47 try {
48 const b = JSON.parse(raw) as Partial<Beat>;
49 if (
50 typeof b.id === "string" && ID.test(b.id) &&
51 typeof b.host === "string" && typeof b.pid === "number" && Number.isInteger(b.pid) &&
52 typeof b.at === "number" && Number.isFinite(b.at)
53 )
54 return { id: b.id, host: b.host, pid: b.pid, at: b.at };
55 } catch {
56 // A torn or foreign file says nobody is there.
57 }
58 return undefined;
59 }
60
61 function running(pid: number): boolean {
62 try {
63 process.kill(pid, 0);
64 return true;
65 } catch (error) {
66 // EPERM: it runs, as someone else.
67 return (error as NodeJS.ErrnoException).code === "EPERM";
68 }
69 }
70
71 /** Whether `beat` is a forge still serving, seen from `host` at `now`. */
72 export function alive(beat: Beat, now: number, host = hostname()): boolean {
73 if (now - beat.at >= STALE_MS) return false;
74 // On this host the pid settles it at once: a crash restarted in the same
75 // pod is not waited for while its last beat ages out.
76 return beat.host !== host || running(beat.pid);
sentinelcrew

Low: inside a container, a forge that crashes and restarts in place often gets the same pid back (pid 1, or a fixed pid under an init). running(beat.pid) then returns true because it is checking this very process. The fresh beat the crash left behind reads as a live predecessor, so /readyz stays 503 for up to 15 s while nobody else is serving. Fix: on this host, also treat beat.pid === process.pid as not alive, e.g. return beat.host !== host || (beat.pid !== process.pid && running(beat.pid));, and add a test that a same-host beat carrying this process's pid is not a predecessor.
77 }
78
79 /** Every other forge's beat on this data. */
80 export function beatsOf(dataDir: string, self?: string): Beat[] {
81 let names: string[];
82 try {
83 names = readdirSync(dirOf(dataDir));
84 } catch {
85 return [];
86 }
87 const beats: Beat[] = [];
88 for (const name of names) {
89 if (!name.endsWith(".json") || name === `${self}.json`) continue;
90 try {
91 const beat = parse(readFileSync(join(dirOf(dataDir), name), "utf8"));
92 if (beat !== undefined) beats.push(beat);
93 } catch {
94 // Removed between the listing and the read: gone.
95 }
96 }
97 return beats;
98 }
99
100 /** Start beating for this forge; files long dead are removed on the way. */
101 export function takePresence(dataDir: string, beatMs = BEAT_MS): Presence {
102 const self: Beat = { id: randomUUID(), host: hostname(), pid: process.pid, at: Date.now() };
103 const dir = dirOf(dataDir);
104 mkdirSync(dir, { recursive: true });
105 const now = Date.now();
106 for (const beat of beatsOf(dataDir, self.id))
107 if (now - beat.at >= 10 * STALE_MS) rmSync(join(dir, `${beat.id}.json`), { force: true });
108 const file = join(dir, `${self.id}.json`);
109 const write = (): void => {
110 try {
111 writeFileSync(`${file}.tmp`, JSON.stringify({ ...self, at: Date.now() }));
112 renameSync(`${file}.tmp`, file);
113 } catch {
114 // A full disk misses a beat; the next one tries again.
115 }
116 };
117 write();
118 let timer: ReturnType<typeof setInterval> | undefined = setInterval(write, beatMs);
119 timer.unref?.();
120 return {
121 others: (at = Date.now()) => beatsOf(dataDir, self.id).some((b) => alive(b, at, self.host)),
122 stop: () => {
123 if (timer === undefined) return;
124 clearInterval(timer);
125 timer = undefined;
126 rmSync(file, { force: true });
127 },
128 };
129 }
130
131 /**
132 * Unready while this boot's apps start — only behind a forge still serving
133 * them, and only while it still is: one that dies meanwhile ends the wait.
134 */
135 export function warmBehind(log: Log, presence: Presence, starting: () => number): void {
136 if (!presence.others()) return;
137 startWarming(log, () => (presence.others() ? starting() : 0));
138 }
src/apps/process.ts +6
@@ -37,6 +37,8 @@ export interface ProcessHandle {
3737 readonly exited: Promise<number>;
3838 /** SIGTERM, a grace period, then SIGKILL. Resolves once it is gone. */
3939 stop(): Promise<void>;
40 /** The cell limit it broke, once it has ended over one (`Cell.breach`). */
41 breach?: () => string | undefined;
4042 }
4143
4244 export interface StartInput {
@@ -459,7 +461,10 @@ export async function startProcess(
459461 // The cell's sockets and watch end with the process, however it ends.
460462 // Without a cell, so does the rest of its group: a child that outlived
461463 // the app, stopped or crashed, is not left running on its data.
464 // Read before the cell closes: its cgroup's OOM count goes with it.
465 let breach: string | undefined;
462466 void proc.exited.then(() => {
467 breach = cell?.breach();
463468 cell?.close();
464469 if (cell === undefined) signalGroup(proc.pid, "SIGKILL");
465470 });
@@ -488,6 +493,7 @@ export async function startProcess(
488493 },
489494 port,
490495 exited: proc.exited,
496 breach: () => breach,
491497 stop: async () => {
492498 if (proc.exitCode !== null || proc.signalCode !== null) return;
493499 // In a cell the pid is bubblewrap's, which passes no signal on: the
src/apps/proxy.ts +19 −3
@@ -39,6 +39,7 @@ import { crewNow, tellLink } from "./crew.ts";
3939 import { doorPage } from "./door.ts";
4040 import { notFound } from "../http/respond.ts";
4141 import { appMeasure } from "./measure.ts";
42 import { holdPortRequest, holdPortSocket } from "./handoff.ts";
4243 import { htmlAnswer, readPage, rewritable, withToolbar, type PillWhere } from "./toolbar.ts";
4344
4445 export interface AppTarget {
@@ -705,7 +706,14 @@ export async function proxyApp(
705706 }
706707
707708 // Held from here until the answer begins: a slow answer is not idle.
708 const answered = runtime.holdRequest(target.app, target.env);
709 // And held against this deployment's port, so one being replaced
710 // answers it before it is stopped (`apps/handoff.ts`).
711 const envAnswered = runtime.holdRequest(target.app, target.env);
712 const portAnswered = holdPortRequest(port);
713 const answered = (): void => {
714 envAnswered();
715 portAnswered();
716 };
709717 let response: Response;
710718 try {
711719 response = await fetch(upstream, {
@@ -859,6 +867,8 @@ export interface AppSocket {
859867 readonly userId: string | undefined;
860868 /** The forge the socket is held open on. */
861869 readonly db: Database;
870 /** The deployment's loopback port, which a handoff drains (`apps/handoff.ts`). */
871 readonly port: number;
862872 }
863873
864874 /**
@@ -868,6 +878,9 @@ export interface AppSocket {
868878 */
869879 const heldSockets = new WeakMap<Bun.ServerWebSocket<AppSocket>, () => void>();
870880
881 /** Each proxied socket's place among its deployment's, released when it closes. */
882 const portSockets = new WeakMap<Bun.ServerWebSocket<AppSocket>, () => void>();
883
871884 /**
872885 * The most one frame may carry, in either direction. The browser's side is
873886 * enforced by Bun before a frame reaches us; the app's side is checked
@@ -936,7 +949,7 @@ async function proxyWebSocket(
936949 runtime.holdStream(target.app, target.env),
937950 );
938951 const upgraded = server.upgrade(request, {
939 data: { upstream, userId: who?.id, db: runtime.app.db },
952 data: { upstream, userId: who?.id, db: runtime.app.db, port: Number(url.port) },
940953 // The app's choice of subprotocol is what the browser hears; without
941954 // one Bun echoes the first the browser offered, which is what the app
942955 // itself would have done on Bun.
@@ -1018,7 +1031,8 @@ export const appWebSocket: Bun.WebSocketHandler<AppSocket> = {
10181031 idleTimeout: WS_IDLE_SECONDS,
10191032
10201033 open(ws) {
1021 const { upstream, userId, db } = ws.data;
1034 const { upstream, userId, db, port } = ws.data;
1035 portSockets.set(ws, holdPortSocket(port, ws));
10221036 if (userId !== undefined)
10231037 heldSockets.set(
10241038 ws,
@@ -1078,6 +1092,8 @@ export const appWebSocket: Bun.WebSocketHandler<AppSocket> = {
10781092 const { upstream } = ws.data;
10791093 heldSockets.get(ws)?.();
10801094 heldSockets.delete(ws);
1095 portSockets.get(ws)?.();
1096 portSockets.delete(ws);
10811097 if (upstream.readyState === WebSocket.CLOSED) return;
10821098 // 1006 here is the browser gone without a close frame.
10831099 if (code === 1006) upstream.close(1001, "the browser went away");
src/apps/runtime.ts +39 −2
@@ -38,6 +38,7 @@ import { runningLimitSentence } from "../settings/quotas.ts";
3838 import type { User } from "../core/users.ts";
3939 import {
4040 cancelRun,
41 getRun,
4142 listRuns,
4243 claimRun,
4344 createRun,
@@ -71,6 +72,7 @@ import { dirBytes } from "./bytes.ts";
7172 import { BUSY_WORDS, busyReason, type BusyReason } from "./busy.ts";
7273 import { exportTree, readFileAt } from "./checkout.ts";
7374 import { seconds, Stopwatch } from "./timing.ts";
75 import { DRAIN_BUDGET_MS, drainPort } from "./handoff.ts";
7476 import type { DomainResolver } from "./domains.ts";
7577 import {
7678 appsRoot,
@@ -609,6 +611,26 @@ export class AppsRuntime {
609611
610612 // ---- lifecycle ----
611613
614 /** Each production environment the boot asked to deploy, and the run that does it. */
615 private readonly bootRuns = new Map<Environment, string>();
616
617 /**
618 * How many of the productions this boot asked for are neither serving
619 * nor settled yet. Until it reads 0 the forge is not ready (`/readyz`),
620 * so in a rollout the forge it replaces keeps proxying to its own copy
621 * of each app and no request finds an app that is still starting (#4496).
622 */
623 startingCount(): number {
624 let count = 0;
625 for (const [state, runId] of this.bootRuns) {
626 const run = state.live?.runId === runId ? undefined : getRun(this.app.db, runId);
627 const pending = run?.state === "queued" || run?.state === "running";
628 if (pending) count++;
629 else this.bootRuns.delete(state);
630 }
631 return count;
632 }
633
612634 /** Rebuild every environment from git and the open pulls, then listen. */
613635 async start(): Promise<void> {
614636 this.cells = await cellsAvailable();
@@ -670,7 +692,11 @@ export class AppsRuntime {
670692 cancelRun(this.app.db, state.queued.id, STOPPED_SUMMARY);
671693 if (state.live) {
672694 revokeAppToken(this.app.db, state.live.tokenId);
673 stops.push(state.live.handle.stop());
695 // A forge leaving (a rollout) lets its sockets reconnect to the
696 // forge that replaced it, and answers what it took, before it
697 // stops the app (#4496).
698 const handle = state.live.handle;
699 stops.push(drainPort(handle.port).then(() => handle.stop()));
674700 }
675701 // Abort only cancels the step in flight; the `execute()` continuation
676702 // chained onto `state.chain` still runs afterwards to record the
@@ -956,6 +982,7 @@ export class AppsRuntime {
956982 state.deploying === undefined &&
957983 state.queued === undefined &&
958984 reason !== "restart" &&
985 reason !== "oom" &&
959986 reason !== "redeploy"
960987 ) {
961988 return undefined;
@@ -983,6 +1010,7 @@ export class AppsRuntime {
9831010 );
9841011 }
9851012 state.queued = run;
1013 if (reason === "boot" && env === "production") this.bootRuns.set(state, run.id);
9861014 state.chain = state.chain.then(async () => {
9871015 if (state.queued?.id !== run.id) return; // superseded
9881016 state.queued = undefined;
@@ -1162,6 +1190,11 @@ export class AppsRuntime {
11621190 `stopping the previous deployment (${previous.commitOid.slice(0, 7)})`,
11631191 );
11641192 revokeAppToken(this.app.db, previous.tokenId);
1193 // Traffic already goes to the new one: the old one's sockets are told
1194 // to reconnect (1012), and what it was answering is answered (#4496).
1195 const unanswered = await drainPort(previous.handle.port);
1196 if (unanswered > 0)
1197 write(`${unanswered} request(s) to the previous deployment were still waiting after ${seconds(DRAIN_BUDGET_MS)}`);
11651198 await previous.handle.stop();
11661199 if (previous.dir !== live.dir)
11671200 rmSync(previous.dir, { recursive: true, force: true });
@@ -1477,6 +1510,10 @@ export class AppsRuntime {
14771510 state.live = undefined;
14781511 this.resting(state);
14791512 revokeAppToken(this.app.db, live.tokenId);
1513 // The cell's memory cap — its own watch or the kernel's OOM killer —
1514 // is a cause of its own on the restarts board (`boards/sources.ts`).
1515 const breach = live.handle.breach?.();
1516 const oom = breach !== undefined && /memory/.test(breach);
14801517 state.lastError = `exited with code ${code}`;
14811518 appendRunLog(
14821519 this.app.config.dataDir,
@@ -1503,7 +1540,7 @@ export class AppsRuntime {
15031540 fresh,
15041541 state.env,
15051542 live.commitOid,
1506 "restart",
1543 oom ? "oom" : "restart",
15071544 );
15081545 }, delay);
15091546 })
src/boards/sources.ts +22
@@ -323,6 +323,28 @@ export const SOURCES: Readonly<Record<string, Source>> = {
323323 value: "milliseconds; the budget is 180000 (three minutes)",
324324 },
325325 },
326 // Every time an app's production process was started again over one that
327 // served (#4496), from its deploy runs: one row per deploy that came to
328 // serve, timed by that instant, with why — the reason a deploy carries,
329 // mapped to four fixed words. A wake from sleep is not a restart: nothing
330 // was serving to be interrupted.
331 restarts: {
332 table: "restart",
333 owner: "repo_id",
334 select:
335 "SELECT repo_id, serving_at AS at, CASE reason WHEN 'boot' THEN 'forge roll' WHEN 'restart' THEN 'crash' " +
336 "WHEN 'oom' THEN 'oom' ELSE 'own deploy' END AS cause FROM run " +
337 "WHERE kind = 'deploy' AND env = 'production' AND serving_at IS NOT NULL AND reason NOT IN ('wake', 'enabled')",
338 time: "at",
339 columns: { repo_id: "text", cause: "text", at: "time" },
340 docs: {
341 cause:
342 "own deploy (a push, a merge, a redeploy), forge roll (the forge itself restarted: a release, " +
343 "and also a crash or eviction of the forge, which its deploys cannot tell apart), " +
344 "crash (the process exited and was started again) or oom (over its memory cap, by its cell's watch or the kernel)",
345 at: "when the new process began serving, UTC",
346 },
347 },
326348 // Items entering each stage from issue to live, one row per entry, read
327349 // from issue, run and pull as `flow/throughput.ts` reads them: an issue
328350 // opens, a builder first starts on it (a retry is not a second item), its
src/boards/starting.ts +12 −1
@@ -1,5 +1,5 @@
11 /**
2 * The boards the forge offers a new install: Usage, Serving, Crew, Releases, Apps, Security, Throughput, Forges and Reach. They are plain
2 * The boards the forge offers a new install: Usage, Serving, Crew, Releases, Apps, App restarts, Security, Throughput, Forges and Reach. They are plain
33 * boards in `forge/boards` once written, so a person edits or deletes them
44 * like any other; `.offered` (a JSON list of slugs, committed with the
55 * boards) records what was offered so a deletion sticks and an edit is
@@ -96,6 +96,17 @@ export const STARTING_BOARDS: Readonly<Record<string, Board>> = {
9696 panel("Requests per hour by surface", "bar", { source: "usage", aggregate: "sum", column: "n", groupBy: "hour", series: "surface" }, 2),
9797 ],
9898 },
99 // Every app restart and its cause (#4496): a forge roll should read zero.
100 restarts: {
101 title: "App restarts",
102 range: "7d",
103 refresh: "1m",
104 panels: [
105 panel("App restarts per day, by cause", "bar", { source: "restarts", aggregate: "count", groupBy: DAY, series: "cause" }, 2),
106 panel("Restarts per app (by repo id), by cause", "bar", { source: "restarts", aggregate: "count", groupBy: "repo_id", series: "cause" }, 2),
107 panel("Restarts by cause", "bar", { source: "restarts", aggregate: "count", groupBy: "cause" }),
108 ],
109 },
99110 crew: {
100111 title: "Crew",
101112 range: "7d",
src/http/health.ts +62 −8
@@ -7,20 +7,68 @@
77 * opens, however slow the moment; the only 503 is a database that will
88 * not answer at all. The liveness probe asks this, and a restart is its
99 * only remedy, so it says no only when a restart could help.
10 * - `/readyz` — ready. 503 once the forge is draining — a release's old
11 * process, finishing what it holds while the new one takes everything
12 * new. The readiness probe asks this; it routes, it never restarts.
10 * - `/readyz` — ready. 503 while a new forge's apps are still starting
11 * (the old one serves them meanwhile, #4496), and 503 once the forge is
12 * draining — a release's old process, finishing what it holds while the
13 * new one takes everything new. The readiness probe asks this; it
14 * routes, it never restarts.
1315 *
1416 * Every deliberate 503 is said in the log, once, at the moment it starts.
1517 */
1618
1719 import type { Log } from "../kernel/log.ts";
1820
19 export const health: { draining: boolean; drainingSince: number | null } = {
21 export const health: {
22 draining: boolean;
23 drainingSince: number | null;
24 /** How many apps this boot is still bringing up, once the apps runtime is on. */
25 starting?: (() => number) | undefined;
26 /** When the wait for them began; past `STARTING_BUDGET_MS` it is ready regardless. */
27 startingSince?: number | undefined;
28 } = {
2029 draining: false,
2130 drainingSince: null,
2231 };
2332
33 /**
34 * How long a new forge's readiness waits for the apps its boot is starting
35 * (#4496): the forge it replaces keeps serving them meanwhile. Bounded, so
36 * one app that never comes up cannot hold a rollout; past it the rest come
37 * up behind a ready forge, as before.
38 */
39 export const STARTING_BUDGET_MS = 120_000;
40
sentinelcrew

STARTING_BUDGET_MS bounds a rollout well, but with no old pod it is two minutes of outage. Readiness should know whether anyone else is serving before it says 503.
41 /**
42 * Not ready until `count` apps are up. Said in the log once, when the wait
43 * starts and when it ends. Nothing to wait for says nothing.
44 */
45 export function startWarming(log: Log, count: () => number, now = Date.now()): void {
46 const waiting = count();
47 if (waiting === 0) return;
48 health.starting = count;
49 health.startingSince = now;
50 log.info("forge starting its apps: /readyz answers 503 until they serve", {
51 apps: waiting,
52 budgetS: STARTING_BUDGET_MS / 1000,
53 });
54 startedLog = log;
55 }
56
57 let startedLog: Log | undefined;
58
59 /** Whether readiness still waits on the apps, and how many. Ends the wait for good once it is over. */
60 function stillStarting(now: number): number {
61 if (health.starting === undefined) return 0;
62 const waiting = health.starting();
63 const over = now - (health.startingSince ?? now) >= STARTING_BUDGET_MS;
64 if (waiting > 0 && !over) return waiting;
65 startedLog?.info("forge ready: its apps serve", over && waiting > 0 ? { stillStarting: waiting } : {});
66 health.starting = undefined;
67 health.startingSince = undefined;
68 startedLog = undefined;
69 return 0;
70 }
71
2472 /** Whether the last liveness answer was a 503, so the log says so once. */
2573 let databaseDown = false;
2674
@@ -81,9 +129,15 @@ export function drainLingerMs(raw: string | undefined): number | null {
81129 return seconds <= DRAIN_LINGER_MAX_S ? seconds * 1000 : null;
82130 }
83131
84 /** Readiness: 200 while the forge takes new work, 503 once it drains. */
85 export function readyResponse(): Response {
86 return health.draining
87 ? new Response("draining", { status: 503 })
132 /**
133 * Readiness: 503 while the apps this boot started do not serve yet (within
134 * `STARTING_BUDGET_MS`), 200 while the forge takes new work, 503 once it
135 * drains.
136 */
137 export function readyResponse(now = Date.now()): Response {
138 if (health.draining) return new Response("draining", { status: 503 });
139 const waiting = stillStarting(now);
140 return waiting > 0
141 ? new Response(`starting ${waiting} app${waiting === 1 ? "" : "s"}`, { status: 503 })
88142 : new Response("ok", { status: 200 });
89143 }
src/main.ts +11 −1
@@ -43,6 +43,7 @@ import { resolveMirror } from "./snapshot/mirror.ts";
4343 import { releaseOrphans } from "./core/run_secret.ts";
4444 import { failAbandonedRuns } from "./core/runs.ts";
4545 import { DRAIN_LINGER_MAX_S, drainLingerMs, startDraining } from "./http/health.ts";
46 import { takePresence, warmBehind } from "./apps/presence.ts";
4647 import { STALE_AFTER_MS } from "./runs/lease.ts";
4748 import { startCrew } from "./crew/dispatch.ts";
4849 import { CREW_OFF_IN_PREVIEW, crewOffInPreview } from "./crew/preview.ts";
@@ -334,7 +335,14 @@ const apps = appDomain?.ok
334335 idlePullMs: nonNegative(process.env.FORGE_PREVIEW_IDLE_DAYS, 3) * 86_400_000,
335336 })
336337 : undefined;
338 // Which other forge serves this data's apps, read before this one starts its own.
339 const presence = apps ? takePresence(app.config.dataDir) : undefined;
337340 await apps?.start();
341 // In a rollout this forge takes no traffic until the apps it just started
sentinelcrew outdated

High: this gates readiness on every boot. With one replica and no old pod (crash, eviction, first start), nothing routes to the forge for up to 120 s: web, API and git all go dark, which works against #3319. Gate this only when a predecessor is still live, for example a fresh run lease or heartbeat on the shared data, and test the cold-start case.
342 // serve: the one it replaces keeps proxying to its own copies until then,
343 // so no request meets an app still starting (#4496). A cold start has no
344 // one serving in its place, and is ready at once.
345 if (apps && presence) warmBehind(app.log, presence, () => apps.startingCount());
338346 // What deploys before the record carried, so ask-to-live is not blank for them.
339347 void backfillCarried(app.db, app.reposRoot).catch((error) =>
340348 app.log.error("could not backfill what deploys carried", { error: String(error) }),
@@ -653,7 +661,9 @@ function drain(signal: NodeJS.Signals): Promise<unknown> {
653661 prompts.stop(),
654662 notifier.stop(),
655663 crew?.stop({ drain: true }),
656 apps?.stop(),
664 // Routers still send this process requests for the linger's length;
665 // its apps answer them, and stop only after (#4496).
666 (linger ?? Promise.resolve()).then(() => apps?.stop()).finally(() => presence?.stop()),
657667 // A step leads a process group of its own, so a terminal's Ctrl-C no
658668 // longer ends it: a person's stop cuts checks off (queued again) at once.
659669 checks?.stop({ drain: signal === "SIGTERM" }),
test/apps-handoff.test.ts added +81
@@ -0,0 +1,81 @@
1 /**
2 * Letting go of a deployment (#4496): `apps/handoff.ts` counts what the
3 * proxy has in flight per port, closes its sockets with 1012 and waits —
4 * bounded — for its requests; and `/readyz` waits for the apps a boot
5 * started, bounded too.
6 */
7
8 import { afterEach, expect, test } from "bun:test";
9 import { drainPort, holdPortRequest, holdPortSocket, inFlight, SERVICE_RESTART } from "../src/apps/handoff.ts";
10 import { health, readyResponse, STARTING_BUDGET_MS, startWarming } from "../src/http/health.ts";
11 import { createLog } from "../src/kernel/log.ts";
12
13 afterEach(() => {
14 health.starting = undefined;
15 health.startingSince = undefined;
16 health.draining = false;
17 });
18
19 test("a port with nothing in flight drains at once", async () => {
20 expect(await drainPort(40001, 5_000)).toBe(0);
21 });
22
23 test("draining closes every socket with 1012 and waits for the request still being answered", async () => {
24 const closes: [number, string][] = [];
25 const socket = { close: (code: number, reason: string) => void closes.push([code, reason]) };
26 const releaseSocket = holdPortSocket(40002, socket);
27 const answered = holdPortRequest(40002);
28 expect(inFlight(40002)).toEqual({ requests: 1, sockets: 1 });
29
30 let drained = false;
31 const draining = drainPort(40002, 10_000).then((left) => {
32 drained = true;
33 return left;
34 });
35 await Bun.sleep(100);
36 expect(closes).toEqual([[SERVICE_RESTART, "service restart"]]);
37 expect(drained).toBe(false);
38 answered();
39 answered(); // a second release does nothing
40 expect(await draining).toBe(0);
41 releaseSocket();
42 expect(inFlight(40002)).toEqual({ requests: 0, sockets: 0 });
43 });
44
45 test("a request that never answers holds the drain only for its budget, and is counted", async () => {
46 const hung = holdPortRequest(40003);
47 expect(await drainPort(40003, 300)).toBe(1);
48 hung();
49 });
50
51 test("readiness waits for the apps a boot is starting, and says how many", () => {
52 let starting = 2;
53 const now = Date.now();
54 startWarming(createLog("test"), () => starting, now);
55 const waiting = readyResponse(now + 1_000);
56 expect(waiting.status).toBe(503);
57 starting = 1;
58 expect(readyResponse(now + 2_000).status).toBe(503);
59 starting = 0;
60 expect(readyResponse(now + 3_000).status).toBe(200);
61 // Over for good: an app redeploying later never makes the forge unready.
62 starting = 1;
63 expect(readyResponse(now + 4_000).status).toBe(200);
64 });
65
66 test("readiness waits no longer than its budget for an app that never comes up", async () => {
67 const now = Date.now();
68 startWarming(createLog("test"), () => 1, now);
69 const before = readyResponse(now + STARTING_BUDGET_MS - 1);
70 expect(before.status).toBe(503);
71 expect(await before.text()).toBe("starting 1 app");
72 expect(readyResponse(now + STARTING_BUDGET_MS).status).toBe(200);
73 });
74
75 test("nothing to start is ready at once, and draining is never ready", () => {
76 startWarming(createLog("test"), () => 0);
77 expect(health.starting).toBeUndefined();
78 expect(readyResponse().status).toBe(200);
79 health.draining = true;
80 expect(readyResponse().status).toBe(503);
81 });
test/apps-presence.test.ts added +75
@@ -0,0 +1,75 @@
1 /**
2 * Who else serves this data's apps (#4496, `apps/presence.ts`): a forge
3 * still beating is there, one that stopped or went silent is not, and only
4 * behind one that is there does readiness wait.
5 */
6
7 import { afterEach, expect, test } from "bun:test";
8 import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
9 import { hostname, tmpdir } from "node:os";
10 import { join } from "node:path";
11 import { alive, STALE_MS, takePresence, warmBehind } from "../src/apps/presence.ts";
12 import { health, readyResponse } from "../src/http/health.ts";
13 import { createLog } from "../src/kernel/log.ts";
14
15 const dirs: string[] = [];
16 const fresh = (): string => {
17 const dir = mkdtempSync(join(tmpdir(), "forge-presence-"));
18 dirs.push(dir);
19 return dir;
20 };
21
22 afterEach(() => {
23 health.starting = undefined;
24 health.startingSince = undefined;
25 for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true });
26 });
27
28 test("the first forge on its data sees nobody, and is ready while its apps start", () => {
29 const first = takePresence(fresh());
30 expect(first.others()).toBe(false);
31 warmBehind(createLog("test"), first, () => 3);
32 expect(readyResponse().status).toBe(200);
33 first.stop();
34 });
35
36 test("a forge beside one still serving waits for its apps; once the old one stops, it waits no more", () => {
37 const dataDir = fresh();
38 const old = takePresence(dataDir);
39 const next = takePresence(dataDir);
40 expect(next.others()).toBe(true);
41 warmBehind(createLog("test"), next, () => 1);
42 expect(readyResponse().status).toBe(503);
43 old.stop();
44 expect(next.others()).toBe(false);
45 expect(readyResponse().status).toBe(200);
46 next.stop();
47 });
48
49 test("a beat older than three missed beats is a forge that died", () => {
50 const beat = { id: "00000000-0000-4000-8000-000000000000", host: "elsewhere", pid: 1, at: 1_000_000 };
51 expect(alive(beat, 1_000_000 + STALE_MS - 1, "here")).toBe(true);
52 expect(alive(beat, 1_000_000 + STALE_MS, "here")).toBe(false);
53 });
54
55 test("on this host a fresh beat from a pid that no longer runs is a crash, not a predecessor", () => {
56 const dataDir = fresh();
57 const dead = Bun.spawnSync(["true"]).pid;
58 const beat = { id: "11111111-1111-4111-8111-111111111111", host: hostname(), pid: dead, at: Date.now() };
59 expect(alive(beat, Date.now())).toBe(false);
60 const next = takePresence(dataDir);
61 writeFileSync(join(dataDir, "serving", `${beat.id}.json`), JSON.stringify(beat));
62 expect(next.others()).toBe(false);
63 writeFileSync(join(dataDir, "serving", `${beat.id}.json`), JSON.stringify({ ...beat, pid: process.pid }));
64 expect(next.others()).toBe(true);
65 next.stop();
66 });
67
68 test("a torn or foreign file in the directory says nobody is there", () => {
69 const dataDir = fresh();
70 const next = takePresence(dataDir);
71 writeFileSync(join(dataDir, "serving", "junk.json"), "{not json");
72 writeFileSync(join(dataDir, "serving", "other.json"), JSON.stringify({ id: "../x", host: "h", pid: 1, at: Date.now() }));
73 expect(next.others()).toBe(false);
74 next.stop();
75 });
test/apps-rollout.test.ts added +286
@@ -0,0 +1,286 @@
1 /**
2 * An app through a rollout (#4496): its own redeploy and the forge's.
3 *
4 * - Its own deploy: the new version serves before the old one is let go
5 * of; a socket held to the old one closes with 1012 Service Restart and
6 * reconnects at once, a request it was still answering is answered, and
7 * a request loop across it sees no 502.
8 * - The forge's: a new forge on the same data is not ready (`/readyz`)
9 * until the app it started serves; the old forge, leaving, closes its
10 * sockets with 1012 so they reconnect to the new one in under a second,
11 * and a request loop that follows readiness from the old forge to the
12 * new sees no 502.
13 */
14
15 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
16 import { mkdtemp, rm } from "node:fs/promises";
17 import { tmpdir } from "node:os";
18 import { join } from "node:path";
19 import { createApp, type App } from "../src/app.ts";
20 import { AppsRuntime } from "../src/apps/runtime.ts";
21 import { newApp, redeployEnv } from "../src/actions/apps.ts";
22 import { commitToBranch } from "../src/actions/repos.ts";
23 import type { Repo } from "../src/core/repos.ts";
24 import { listRuns, type Run } from "../src/core/runs.ts";
25 import { createUser, type User } from "../src/core/users.ts";
26 import { serve, type Servers } from "../src/http/server.ts";
27 import { health } from "../src/http/health.ts";
28 import { takePresence, warmBehind, type Presence } from "../src/apps/presence.ts";
29 import { runQuery } from "../src/boards/query.ts";
30
31 /** `/` answers, `/slow` answers after a second and a half, `/ws` echoes. */
32 const SERVER = `export const server = Bun.serve({
33 port: Number(process.env.PORT ?? "0"),
34 async fetch(request, server) {
35 const url = new URL(request.url);
36 if (url.pathname === "/ws")
37 return server.upgrade(request) ? undefined : new Response("no", { status: 400 });
38 if (url.pathname === "/slow") {
39 await Bun.sleep(1500);
40 return new Response("slow and answered");
41 }
42 return new Response("up " + process.pid);
43 },
44 websocket: { message(ws, message) { ws.send(message); } },
45 });
46 `;
47
48 const SERVER_TEST = `import { expect, test } from "bun:test";
49 process.env.PORT = "0";
50 const { server } = await import("./server.ts");
51 test("serves the page", async () => {
52 expect((await fetch(\`http://\${server.hostname}:\${server.port}/\`)).status).toBe(200);
53 });
54 `;
55
56 const DOMAIN = "apps.test";
57 const HOST = `rollme.${DOMAIN}`;
58
59 interface Forge {
60 app: App;
61 servers: Servers;
62 runtime: AppsRuntime;
63 presence: Presence;
64 }
65
66 async function bootForge(dataDir: string): Promise<Forge> {
67 const app = createApp({ dataDir, port: 0 });
68 const servers = await serve(app);
69 const runtime = new AppsRuntime(app, {
70 domain: DOMAIN,
71 scheme: "http",
72 publicPort: servers.port,
73 forgeUrl: servers.url,
74 concurrency: 2,
75 });
76 // As `main.ts` does: who else serves this data is read before the apps start.
77 const presence = takePresence(dataDir);
78 await runtime.start();
79 warmBehind(app.log, presence, () => runtime.startingCount());
80 return { app, servers, runtime, presence };
81 }
82
83 const get = (forge: Forge, path: string) =>
84 fetch(`${forge.servers.url}${path}`, { headers: { host: HOST }, redirect: "manual" });
85
86 /** A socket through `forge`, and when and how it closed. */
87 function connect(forge: Forge) {
88 const ws = new WebSocket(`ws://localhost:${forge.servers.port}/ws`, { headers: { host: HOST } });
89 ws.addEventListener("error", () => {});
90 const opened = new Promise<boolean>((resolve) => {
91 ws.addEventListener("open", () => resolve(true));
92 ws.addEventListener("close", () => resolve(false));
93 });
94 const closed = new Promise<{ code: number; reason: string; at: number }>((resolve) =>
95 ws.addEventListener("close", (e) => resolve({ code: e.code, reason: e.reason, at: Date.now() })),
96 );
97 const echo = (text: string) =>
98 new Promise<string>((resolve) => {
99 ws.addEventListener("message", (e) => resolve(String(e.data)), { once: true });
100 ws.send(text);
101 });
102 return { ws, opened, closed, echo };
103 }
104
105 /** Ask `/` until `stop` is set, from whichever forge `target` names; every status seen. */
106 function loop(target: () => Forge, stop: { done: boolean }): Promise<number[]> {
107 return (async () => {
108 const seen: number[] = [];
109 while (!stop.done) {
110 const res = await get(target(), "/");
111 await res.arrayBuffer();
112 seen.push(res.status);
113 await Bun.sleep(20);
114 }
115 return seen;
116 })();
117 }
118
119 describe("apps: through a rollout", () => {
120 let dataDir: string;
121 let a: Forge;
122 let b: Forge | undefined;
123 let c: Forge | undefined;
124 let alice: User;
125 let repo: Repo;
126
127 async function settled(forge: Forge, where: (run: Run) => boolean, timeoutMs = 90_000): Promise<Run> {
128 const deadline = Date.now() + timeoutMs;
129 while (Date.now() < deadline) {
130 const run = listRuns(forge.app.db, repo.id, { kind: "deploy", limit: 50 }).find(where);
131 if (run && (run.state === "ok" || run.state === "failed" || run.state === "cancelled")) return run;
132 await Bun.sleep(100);
133 }
134 throw new Error("run did not settle in time");
135 }
136
137 beforeAll(async () => {
138 dataDir = await mkdtemp(join(tmpdir(), "forge-apps-rollout-"));
139 a = await bootForge(dataDir);
140 const created = await createUser(a.app.db, {
141 origin: "seed",
142 trusted: true,
143 username: "alice",
144 email: "[email protected]",
145 password: "a-good-passphrase",
146 adminIfFirst: true,
147 });
148 if (!created.ok) throw new Error(created.error);
149 alice = created.value;
150 const made = await newApp(a.app, alice, { name: "rollme", description: "Rolls", visibility: "public" });
151 if (!made.ok) throw new Error(made.error);
152 repo = made.value.repo;
153 await settled(a, (run) => run.env === "production");
154 const committed = await commitToBranch(a.app, alice, repo, {
155 message: "Echo, slowly",
156 files: [
157 { path: "src/server.ts", content: SERVER },
158 { path: "src/server.test.ts", content: SERVER_TEST },
159 { path: "src/audit.test.ts", content: null },
160 { path: "src/openapi.test.ts", content: null },
161 { path: "src/security.test.ts", content: null },
162 { path: "src/site.test.ts", content: null },
163 ],
164 });
165 if (!committed.ok) throw new Error(String(committed.error));
166 const run = await settled(a, (r) => r.env === "production" && r.commitOid === committed.value.commitOid);
167 if (run.state !== "ok") throw new Error(`deploy failed: ${run.summary}`);
168 }, 180_000);
169
170 afterAll(async () => {
171 for (const forge of [c, b, a]) {
172 if (forge === undefined) continue;
173 await forge.runtime.stop();
174 forge.presence.stop();
175 forge.servers.stop();
176 forge.app.close();
177 }
178 health.starting = undefined;
179 health.startingSince = undefined;
180 await rm(dataDir, { recursive: true, force: true });
181 });
182
183 test("its own redeploy: the old version's socket is told to reconnect, its slow answer arrives, and no request sees a 502", async () => {
184 const socket = connect(a);
185 expect(await socket.opened).toBe(true);
186 expect(await socket.echo("before")).toBe("before");
187 const before = await (await get(a, "/")).text();
188
189 const stop = { done: false };
190 const statuses = loop(() => a, stop);
191 const slow = get(a, "/slow");
192 // The slow request is with the old version before the new one is asked for.
193 await Bun.sleep(200);
194 const asked = await redeployEnv(a.app, alice, repo, "production");
195 expect(asked.ok).toBe(true);
196 const runId = asked.ok ? asked.value?.id : undefined;
197 expect(runId).toBeDefined();
198
199 const closed = await socket.closed;
200 expect(closed.code).toBe(1012);
201 expect(closed.reason).toBe("service restart");
202 const again = connect(a);
203 expect(await again.opened).toBe(true);
204 expect(Date.now() - closed.at).toBeLessThan(1000);
205 expect(await again.echo("after")).toBe("after");
206 again.ws.close();
207
208 const answered = await slow;
209 expect(answered.status).toBe(200);
210 expect(await answered.text()).toBe("slow and answered");
211
212 await settled(a, (r) => r.id === runId);
213 stop.done = true;
214 const seen = await statuses;
215 expect(seen.length).toBeGreaterThan(0);
216 expect(seen.filter((s) => s !== 200)).toEqual([]);
217 expect(await (await get(a, "/")).text()).not.toBe(before);
218 }, 120_000);
219
220 test("the forge's rollout: the new forge is ready only once the app serves, and a socket reconnects to it in under a second with no 502", async () => {
221 const socket = connect(a);
222 expect(await socket.opened).toBe(true);
223 expect(await socket.echo("held")).toBe("held");
224
225 const stop = { done: false };
226 let current: Forge = a;
227 const statuses = loop(() => current, stop);
228
229 // The new forge comes up on the same data, as the surge pod does.
230 b = await bootForge(dataDir);
231 const next = b;
232 expect(next.runtime.startingCount()).toBe(1);
233 const notYet = await fetch(`${next.servers.url}/readyz`);
234 expect(notYet.status).toBe(503);
235 expect(await notYet.text()).toBe("starting 1 app");
236
237 const deadline = Date.now() + 90_000;
238 while ((await fetch(`${next.servers.url}/readyz`)).status !== 200 && Date.now() < deadline)
239 await Bun.sleep(100);
240 expect(next.runtime.startingCount()).toBe(0);
241 expect((await fetch(`${next.servers.url}/readyz`)).status).toBe(200);
242 // Ready means its own copy already answers.
243 expect((await get(next, "/")).status).toBe(200);
244
245 // The platform routes to the new forge; the old one stops its apps.
246 current = next;
247 await a.runtime.stop();
248 a.presence.stop();
249 const closed = await socket.closed;
250 expect(closed.code).toBe(1012);
251 expect(closed.reason).toBe("service restart");
252 const again = connect(next);
253 expect(await again.opened).toBe(true);
254 expect(Date.now() - closed.at).toBeLessThan(1000);
255 expect(await again.echo("reconnected")).toBe("reconnected");
256 again.ws.close();
257
258 await Bun.sleep(200);
259 stop.done = true;
260 const seen = await statuses;
261 expect(seen.filter((s) => s !== 200)).toEqual([]);
262
263 // And the restarts board names both causes.
264 const ran = runQuery(next.app.db, { source: "restarts", aggregate: "count", groupBy: "cause" }, { repoIds: "all" });
265 expect(ran.ok).toBe(true);
266 const causes = ran.ok ? ran.rows.map((r) => String(r.group)) : [];
267 expect(causes).toContain("forge roll");
268 expect(causes).toContain("own deploy");
269 }, 180_000);
270
271 test("a cold start, with no forge serving in its place, is ready at once while its app still starts", async () => {
272 // The forge before it is gone and said so: nobody serves these apps now.
273 const before = b!;
274 await before.runtime.stop();
275 before.presence.stop();
276 before.servers.stop();
277 before.app.close();
278 b = undefined;
279
280 c = await bootForge(dataDir);
281 expect(c.runtime.startingCount()).toBe(1);
282 const ready = await fetch(`${c.servers.url}/readyz`);
283 expect(ready.status).toBe(200);
284 expect(await ready.text()).toBe("ok");
285 }, 60_000);
286 });
test/boards-starting.test.ts +4 −4
@@ -1,5 +1,5 @@
11 /**
2 * The forge offers the nine starting boards on boot: every panel runs, an
2 * The forge offers the ten starting boards on boot: every panel runs, an
33 * edit survives the next boot, and a deletion is not undone.
44 */
55 import { afterAll, beforeAll, expect, test } from "bun:test";
@@ -36,10 +36,10 @@ test("a forge with no person to found the repo writes nothing", async () => {
3636 expect(await offerStartingBoards(bare)).toEqual([]);
3737 });
3838
39 test("a fresh boot writes the nine starting boards, and every panel runs", async () => {
40 expect((await offerStartingBoards(app)).sort()).toEqual(["apps", "crew", "forges", "reach", "releases", "security", "serving", "throughput", "usage"]);
39 test("a fresh boot writes the ten starting boards, and every panel runs", async () => {
40 expect((await offerStartingBoards(app)).sort()).toEqual(["apps", "crew", "forges", "reach", "releases", "restarts", "security", "serving", "throughput", "usage"]);
4141 const boards = await listBoards(app.reposRoot);
42 expect(boards.map((b) => b.slug)).toEqual(["apps", "crew", "forges", "reach", "releases", "security", "serving", "throughput", "usage"]);
42 expect(boards.map((b) => b.slug)).toEqual(["apps", "crew", "forges", "reach", "releases", "restarts", "security", "serving", "throughput", "usage"]);
4343 let panels = 0;
4444 for (const { board } of boards) {
4545 for (const p of board.panels) {

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs