Public: anyone can read this; the crew can change it.

At the awake ceiling a visitor is told "the forge is full": nothing idle is put to sleep to make room, and boot and spawn do not count #13831

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/13812-at-the-awake-ceiling-a-visitor-is-told-t → main 8 commits fork

  • Merged. scout/13812-at-the-awake-ceiling-a-visitor-is-told-t was merged into main as f1c6342.

Review

Sign in to review this change.
9 files changed +373 −14
AGENTS.md +1 −1
@@ -49,7 +49,7 @@ The check queue hands work to a runner through `src/checks/runner.ts` (#13652, R
4949 `src/spawn/child_reach.ts`: `said`, when each hosted forge's refusal of each name was last written to the log (one `[egress] <child> was refused <host>:<port>` line an hour; pruned past 2,000 keys), and `calls`, each child's signed calls in the last minute; both gone on restart, and neither is state (asks and grants are rows: `child_reach_ask`, `child_reach_grant`). `src/actions/ask_host.ts` on a hosted forge: `looking` (the one refresh in flight), `lookedAt` (when the host was last asked, so a page is heard at most every 3 s) and `watching` (the one timer that looks every 10 s while an ask is under an hour old); a restart loses nothing, since every ask is a row (`reach_ask`) and the heartbeat looks again. The terminal door is `src/tui/screens/hostreach.ts` (palette entry for admins; `a` allows for that forge, `e` for every hosted forge, `n` not now, `x` takes a grant back; `allowReach`, `putAsideReach` and `revokeReach` in `actions/child_reach.ts` give the pages and the terminal the same gate and the `forge.reach_*` events).
5050
5151 ## Awake ceiling and the boot
52 `FORGE_CHILD_AWAKE_MAX` counts forges in use. Hosted children the boot opens (so their apps come back) are exempt from `awakeCount` until a credentialed request reaches them or they sleep (`bootOpened` in `src/spawn/host.ts`). The exempt set holds only body ids of existing finished children and is emptied when the parent closes, so it is bounded by the number of children. Cell children are always counted. The boot starts only the forges that were awake and in use when the parent stopped (`awake-at-stop.json` under the data directory, written by `closeHostedChildren`, read and removed once by `bootHostedChildren`, `src/spawn/awake_at_stop.ts`), up to the ceiling; a crash leaves no file, so nothing starts and forges wake on their next request. A sleeping cell forge with a run due has its wake armed. The idle sweeper starts at boot.
52 The capacity setting `children.awake` (boot default `FORGE_CHILD_AWAKE_MAX`) counts forges in use; a wake or a spawn at the ceiling puts the longest-idle forge to sleep first (`roomFor` in `src/spawn/host.ts`), and the boot never opens a child past it. Hosted children the boot opens (so their apps come back) are exempt from `awakeCount` until a credentialed request reaches them or they sleep (`bootOpened` in `src/spawn/host.ts`). The exempt set holds only body ids of existing finished children and is emptied when the parent closes, so it is bounded by the number of children. Cell children are always counted. The boot starts only the forges that were awake and in use when the parent stopped (`awake-at-stop.json` under the data directory, written by `closeHostedChildren`, read and removed once by `bootHostedChildren`, `src/spawn/awake_at_stop.ts`), up to the ceiling; a crash leaves no file, so nothing starts and forges wake on their next request. A sleeping cell forge with a run due has its wake armed. The idle sweeper starts at boot.
5353
5454 An app token's requests go behind people's and repeat reads are 304s (`src/http/app_turn.ts`, `repeat_reads.ts`, `perf/asking.ts`): see `docs/features/app-asking.md`.
5555 `src/core/reports.ts`: `closedSet`, the ids of repositories whose owner closed reports from outside (#12321), read once per database and kept by `setReportsClosed`, the only writer, so every repository page's head link costs no statement; at most one id per closed repository, gone on restart (the rows are `report_closed`). `src/core/contribute_token.ts`: `openSet`, the ids of repositories that allow outside agents, read once per database and kept by `setContributeOpen`, the only writer, at most one id per allowing repository, gone on restart; a raw SQL write to `contribute_open` bypasses it. The terminal door for reports is `r` on the outside agents screen (`setOutsideReports`, admins).
docs/features/forge-sleep.md +1 −1
@@ -3,7 +3,7 @@
33 A person (or their assistant, over the MCP `spawn` tool with their consent) makes a forge at `/forges/new`, the address `<name>--<this host>`. Free within the limits below; no payment path yet.
44
55 - **Sleep.** A forge with no request for `FORGE_CHILD_IDLE_S` (600) sleeps: a hosted child's App and apps close, a cell child's process stops. Data stays under `children/<bodyId>`. The next page load, `git push` or API call wakes it. A page for a sleeping cell forge answers a plain "waking up" page at once (git and API calls wait up to 4 s) (503, `retry-after`, reloads itself); a hosted one opens in-process.
6 - **Limits, each refused in words.** The admin limits `forges.per_person` (3) and `forges.storage_mb` (1000 MB), set at `/admin/limits` (the same list is on the terminal's limits screen), each refused in a sentence naming the limit and the person's use (site admins exempt), `FORGE_CHILD_AWAKE_MAX` (8 awake at once; sleeping forges do not count), `FORGE_CHILD_WAKES_PER_HOUR` (20 per forge), the admin limit `forges.wakes_per_person_hour` (30 wakes an hour, counted against the forge's owner once someone has claimed it and against the forge itself until then, never against whoever sent it; a site admin's own forge is counted like anyone's; refused in a sentence that says whose wakes are spent, in how long the forge can be woken again, and the limit's name and where an admin raises it), plus the existing size/quota limits (`FORGE_CHILD_*_MIB`), and each cell's own disk cap.
6 - **Limits, each refused in words.** The admin limits `forges.per_person` (3) and `forges.storage_mb` (1000 MB), set at `/admin/limits` (the same list is on the terminal's limits screen), each refused in a sentence naming the limit and the person's use (site admins exempt), the capacity setting `children.awake` (default from memory, at least 8, one per 720 MiB above 2 GiB; boot default `FORGE_CHILD_AWAKE_MAX`; sleeping forges do not count, new ones do; at the ceiling the forge idle longest, quiet for 10 s, is put to sleep to make room, and a wake is refused only when every awake forge is in use), `FORGE_CHILD_WAKES_PER_HOUR` (20 per forge), the admin limit `forges.wakes_per_person_hour` (30 wakes an hour, counted against the forge's owner once someone has claimed it and against the forge itself until then, never against whoever sent it; a site admin's own forge is counted like anyone's; refused in a sentence that says whose wakes are spent, in how long the forge can be woken again, and the limit's name and where an admin raises it), plus the existing size/quota limits (`FORGE_CHILD_*_MIB`), and each cell's own disk cap.
77 - **Owner.** `/forges` shows awake/asleep, size, last activity, with Stop (sleep now) and Remove (ledger and data).
88 - **Secrets.** A child starts from `childEnv` and never sees this forge's model credential; the person brings their own or its crew stays off.
99
src/core/capacity.ts +23
@@ -26,6 +26,7 @@ export type CapacityKey =
2626 | "queue.batch"
2727 | "week.use"
2828 | "children.max"
29 | "children.awake"
2930 | "cells.egress"
3031 | "children.modelEgress"
3132 | "children.registryEgress"
@@ -104,6 +105,13 @@ export const CAPACITY: Record<CapacityKey, Definition> = {
104105 fallback: "",
105106 help: "How many hosted child forges this one holds at once. Empty: one per 512 MiB of memory above this forge's own 2 GiB, from the pod's limit. Raising it lets the New forge door open again at once.",
106107 },
108 "children.awake": {
109 label: "Forges awake at once",
110 env: "FORGE_CHILD_AWAKE_MAX",
111 kind: { type: "count", min: 1, max: 1000 },
112 fallback: "8",
113 help: "How many hosted child forges run at once; a sleeping one does not count, and a new one counts from its birth. At the ceiling, the forge idle longest (no request under way, no crew at work) is put to sleep to make room, and a visitor is told the forge is full only when every awake forge is in use. Empty: one per 720 MiB of memory above this forge's own 2 GiB, from the pod's limit (an awake idle small forge costs about 490 MiB resident), at least 8. Read on every wake.",
114 },
107115 "cells.egress": {
108116 label: "Egress from cells",
109117 // No environment default: report is a decision an admin makes and the
@@ -202,6 +210,7 @@ export function capacityDefault(key: CapacityKey, cpus?: number): string {
202210 const sized = CAPACITY[key].sized;
203211 if (sized !== undefined) return String(sized(cpus ?? cpuLimit()));
204212 if (key === "children.max") return String(childrenForMemory(memoryLimitBytes()));
213 if (key === "children.awake") return String(awakeForMemory(memoryLimitBytes()));
205214 if (key === "crew.concurrency") return String(crewForMemory(memoryLimitBytes()));
206215 return CAPACITY[key].fallback;
207216 }
@@ -359,6 +368,20 @@ export function childrenForMemory(bytes: number): number {
359368 }
360369 const MIN_CHILDREN = 5;
361370
371 /**
372 * Forges awake at once by default: one per 720 MiB above the forge's own 2 GiB (an awake idle small
373 * forge measured about 490 MiB resident; 180 GiB gives 250), never fewer than 8.
374 */
375 export function awakeForMemory(bytes: number): number {
376 const n = Math.floor((bytes - 2 * GIB) / (720 * MIB));
377 return Math.min(1000, Math.max(8, Number.isFinite(n) ? n : 8));
378 }
379
380 /** How many hosted children may be awake now; read on every use. */
381 export function maxAwake(db: Database): number {
382 return Number(capacityValue(db, "children.awake"));
383 }
384
362385 /**
363386 * One crew run per 3 GiB of room above the forge's own 2 GiB (32 GiB gives
364387 * 10, 64 GiB the ceiling of 16), never fewer than two: a big machine runs
src/spawn/cell.ts +19
@@ -262,6 +262,12 @@ export interface CellChildren {
262262 * disk).
263263 */
264264 sleepIdle(idleMs: number, now?: number, onAsleep?: (name: string, bodyId: string) => void): Promise<string[]>;
265 /**
266 * Stop the running child idle longest to make room: not mid-request, no crew at work. A new forge
267 * nobody has asked for since its birth may be the one; one that was asked for keeps its ten
268 * minutes. Its name, or undefined when every child is in use.
269 */
270 sleepLongestIdle(now?: number, minIdleMs?: number): Promise<string | undefined>;
265271 }
266272
267273 /**
@@ -552,6 +558,19 @@ export function makeRegistry(parent: App, launcher?: Launcher): CellChildren {
552558 await Promise.all(Array.from({ length: Math.min(SLEEP_AT_ONCE, queue.length) }, worker));
553559 return asleep;
554560 },
561 async sleepLongestIdle(now = Date.now(), minIdleMs = 0) {
562 const order = [...held.values()]
563 .filter((c) => (busy.get(c.bodyId) ?? 0) === 0 && now - (seen.get(c.bodyId) ?? now) >= minIdleMs)
564 .filter((c) => !isNewForge(startedAt.get(c.bodyId), now) || (seen.get(c.bodyId) ?? 0) <= (startedAt.get(c.bodyId) ?? 0))
565 .sort((a, b) => (seen.get(a.bodyId) ?? now) - (seen.get(b.bodyId) ?? now));
566 for (const child of order) {
567 if (await childCrewAtWork(childDirOf(parent, child.bodyId), now)) continue;
568 seen.delete(child.bodyId);
569 await this.stop(child.bodyId);
570 return child.name;
571 }
572 return undefined;
573 },
555574 };
556575 }
557576
src/spawn/host.ts +71 −7
@@ -244,6 +244,58 @@ export async function sleepIdleChildren(parent: App, idleMs = idleAfterMs(), now
244244 return asleep;
245245 }
246246
247 /**
248 * Put the forge idle longest to sleep to make room under the awake ceiling: not mid-request, no
249 * crew at work; a hosted forge in this process first, then a cell's. The name, or undefined when
250 * every awake forge is in use.
251 */
252 export async function sleepLongestIdle(parent: App, now = Date.now()): Promise<string | undefined> {
253 const cache = caches.get(parent);
254 const unasked = bootOpened.get(parent);
255 let oldest: { bodyId: string; child: App; name: string; at: number } | undefined;
256 for (const [bodyId, child] of cache ?? []) {
257 const mark = touched.get(parent)?.get(bodyId);
258 if (mark === undefined || unasked?.has(bodyId) || now - mark.at < EVICT_GRACE_MS || working(parent, child)) continue;
259 if (oldest === undefined || mark.at < oldest.at) oldest = { bodyId, child, name: mark.name, at: mark.at };
260 }
261 if (oldest !== undefined && cache !== undefined) {
262 await closeOne(parent, cache, oldest.bodyId, oldest.child, oldest.name);
263 sleepHooks.get(parent)?.();
264 parent.log.info("child forge asleep to make room", { name: oldest.name });
265 return oldest.name;
266 }
267 const name = await cellChildren(parent).sleepLongestIdle(now, EVICT_GRACE_MS);
268 if (name === undefined) return undefined;
269 const ledger = readLedgerSync(parent, name);
270 if (ledger !== undefined) {
271 touched.get(parent)?.delete(ledger.bodyId);
272 waking.get(parent)?.delete(ledger.bodyId);
273 const due = await childNextRunDue(childDirOf(parent, ledger.bodyId));
274 if (due !== undefined) wakeWhenDue(parent, ledger, due);
275 }
276 parent.log.info("child forge asleep to make room", { name });
277 return name;
278 }
279
280 /** The room-making in progress for each parent: one at a time, so two wakes never sleep two forges. */
281 const roomLocks = new WeakMap<App, Promise<unknown>>();
282
283 /**
284 * Why one more forge may not start now, or undefined: at the ceiling the forge idle longest makes
285 * room first. One at a time per parent, and the count is read again inside the turn: a wake that
286 * waited finds the room the one before it made, and puts no second forge to sleep for it.
287 */
288 export function roomFor(parent: App, now = Date.now()): Promise<string | undefined> {
289 const turn = (roomLocks.get(parent) ?? Promise.resolve()).then(async () => {
290 const why = awakeRefusal(awakeCount(parent), process.env, parent.db);
291 if (why === undefined) return undefined;
292 if ((await sleepLongestIdle(parent, now)) === undefined) return why;
293 return awakeRefusal(awakeCount(parent), process.env, parent.db);
294 });
295 roomLocks.set(parent, turn.catch(() => undefined));
296 return turn;
297 }
298
247299 /**
248300 * What to do when a forge that ran in this process goes to sleep; set by whoever moves such forges
249301 * into cells.
@@ -264,6 +316,11 @@ const dueWakes = new WeakMap<App, Map<string, ReturnType<typeof setTimeout>>>();
264316 * A due wake the awake ceiling or the disk floor turned away is tried again this much later, ms.
265317 */
266318 const DUE_RETRY_MS = 60_000;
319 /**
320 * A forge visited within this long is in use: making room never sleeps it, so a wake cannot thrash
321 * one just served, ms.
322 */
323 const EVICT_GRACE_MS = 10_000;
267324 /** A forge the host stopped may take the room of one quiet this long, ms. */
268325 const STOPPED_EVICT_IDLE_MS = 60_000;
269326 /** The longest a timer waits in one step (setTimeout's own limit); a longer wait is armed again. */
@@ -306,9 +363,9 @@ async function wakeDue(parent: App, name: string): Promise<void> {
306363 if (isAwakeBody(parent, ledger.bodyId) || lapsedNone(parent, name) !== undefined) return;
307364 // A forge the host stopped for a limit does not wait for an idle one's ten minutes: forges quiet
308365 // for a minute and with no crew at work are put to sleep to make its room.
309 if (lastStop(parent, ledger.bodyId) !== undefined && awakeRefusal(awakeCount(parent)) !== undefined)
366 if (lastStop(parent, ledger.bodyId) !== undefined && awakeRefusal(awakeCount(parent), process.env, parent.db) !== undefined)
310367 await sleepIdleChildren(parent, STOPPED_EVICT_IDLE_MS);
311 if (floorRefusal(parent.config.dataDir) !== undefined || awakeRefusal(awakeCount(parent)) !== undefined) {
368 if (floorRefusal(parent.config.dataDir) !== undefined || awakeRefusal(awakeCount(parent), process.env, parent.db) !== undefined) {
312369 wakeWhenDue(parent, ledger, Date.now() + DUE_RETRY_MS);
313370 tryAgainAt(parent, ledger.bodyId, Date.now() + DUE_RETRY_MS);
314371 return;
@@ -581,7 +638,7 @@ export async function wakeGate(parent: App, request: Request): Promise<Response
581638 const wait = people.waitS(budget.key, max, at);
582639 if (wait > 0)
583640 return wakingResponse(parent, request, name, budgetRefusal({ name, budget, max, waitS: wait, limitsUrl: `${parentOrigin(parent.config)}/admin/limits` }), wait);
584 const why = awakeRefusal(awakeCount(parent)) ?? (known ? undefined : rates.refusal(ledger.bodyId, at));
641 const why = (await roomFor(parent)) ?? (known ? undefined : rates.refusal(ledger.bodyId, at));
585642 if (why !== undefined) {
586643 const stop = lastStop(parent, ledger.bodyId);
587644 return wakingResponse(parent, request, name, stop === undefined ? why : stopSentence(stop));
@@ -850,10 +907,17 @@ function openOnce(parent: App, ledger: ChildLedger): Promise<App> | App {
850907 * Open one finished child now, which starts its apps: after a spawn, so
851908 * the first visit after a claim finds them awake. Never throws.
852909 */
853 export async function bootHostedChild(parent: App, name: string): Promise<void> {
910 export async function bootHostedChild(parent: App, name: string, evict = true): Promise<void> {
854911 let ledger = readLedgerSync(parent, name);
855912 if (ledger === undefined || !ledger.done || ledger.moving) return;
856 if (!isAwakeBody(parent, ledger.bodyId) && startRefused(parent, ledger) !== undefined) return;
913 if (!isAwakeBody(parent, ledger.bodyId)) {
914 if (startRefused(parent, ledger) !== undefined) return;
915 // A new forge counts toward the awake ceiling. A person's spawn makes room by putting the
916 // longest idle to sleep; the boot never does, so a restart cannot sleep what was in use. A
917 // forge left asleep wakes at its first request.
918 const full = evict ? await roomFor(parent) : awakeRefusal(awakeCount(parent), process.env, parent.db);
919 if (full !== undefined) return;
920 }
857921 if (ledger.placement === "cell") {
858922 if ((await settleCell(parent, ledger)) !== undefined) return;
859923 // Still a cell child unless this host holds no cell at all: it is opened here only then.
@@ -900,9 +964,9 @@ export async function bootHostedChildren(parent: App, wasAwake: ReadonlySet<stri
900964 continue;
901965 }
902966 // Boot-opened forges are exempt from `awakeCount`, so the boot counts its own.
903 if (awakeRefusal(awakeCount(parent) + (bootOpened.get(parent)?.size ?? 0)) !== undefined) continue;
967 if (awakeRefusal(awakeCount(parent) + (bootOpened.get(parent)?.size ?? 0), process.env, parent.db) !== undefined) continue;
904968 const was = caches.get(parent)?.has(ledger.bodyId) === true;
905 await bootHostedChild(parent, name);
969 await bootHostedChild(parent, name, false);
906970 if (!was && caches.get(parent)?.has(ledger.bodyId) === true) {
907971 let set = bootOpened.get(parent);
908972 if (!set) bootOpened.set(parent, (set = new Set()));
src/spawn/move_cell.ts +1 −1
@@ -358,7 +358,7 @@ async function move(
358358 `${name} is awake in this forge's process; it is moved when it next sleeps`,
359359 );
360360 // Its cell needs a place among the forges awake; closed here, it gives its own back first.
361 if (awakeRefusal(awakeCount(app) - (open ? 1 : 0)) !== undefined)
361 if (awakeRefusal(awakeCount(app) - (open ? 1 : 0), process.env, app.db) !== undefined)
362362 return no(
363363 "busy",
364364 `${name} is not moved yet: this host has as many forges awake as it holds, and it is moved when one sleeps`,
src/spawn/sleep.ts +5 −4
@@ -14,6 +14,7 @@
1414 import type { Database } from "bun:sqlite";
1515 import { readdirSync, readFileSync, statSync } from "node:fs";
1616 import { join } from "node:path";
17 import { maxAwake } from "../core/capacity.ts";
1718 import { LIMITS, limitValue } from "../settings/limits.ts";
1819
1920 type Env = Record<string, string | undefined>;
@@ -28,8 +29,8 @@ function whole(env: Env, name: string, fallback: number): number {
2829 /** A forge quiet for this long sleeps: FORGE_CHILD_IDLE_S, ten minutes by default. */
2930 export const idleAfterMs = (env: Env = process.env): number => whole(env, "FORGE_CHILD_IDLE_S", 600) * 1000;
3031 /**
31 * Most hosted forges awake at once on this host: FORGE_CHILD_AWAKE_MAX. A sleeping forge does not
32 * count.
32 * Most hosted forges awake at once from the environment alone (the capacity setting
33 * `children.awake` is read where a database is at hand). A sleeping forge does not count.
3334 */
3435 export const awakeCeiling = (env: Env = process.env): number => whole(env, "FORGE_CHILD_AWAKE_MAX", 8);
3536 /** Most wakes of one forge in an hour: FORGE_CHILD_WAKES_PER_HOUR. */
@@ -104,8 +105,8 @@ export function budgetRefusal(input: { readonly name: string; readonly budget: W
104105 }
105106
106107 /** Why this host will not wake one more forge while `awake` are running, or undefined. */
107 export function awakeRefusal(awake: number, env: Env = process.env): string | undefined {
108 const max = awakeCeiling(env);
108 export function awakeRefusal(awake: number, env: Env = process.env, db?: Database): string | undefined {
109 const max = db === undefined ? awakeCeiling(env) : maxAwake(db);
109110 return awake >= max
110111 ? "the forge is full, try again shortly"
111112 : undefined;
test/awake-room-hosted.test.ts added +124
@@ -0,0 +1,124 @@
1 /**
2 * Making room under the awake ceiling for forges hosted in this process (#13831): the oldest
3 * visited goes first, none in use or inside its grace does, a wake that made room goes ahead, and a
4 * boot never sleeps what was in use while a spawn may.
5 */
6 import { afterAll, beforeAll, beforeEach, expect, test } from "bun:test";
7 import { mkdtemp, rm } from "node:fs/promises";
8 import { tmpdir } from "node:os";
9 import { join } from "node:path";
10 import { spawnForge } from "../src/actions/spawn.ts";
11 import { createApp, type App } from "../src/app.ts";
12 import { createUser } from "../src/core/users.ts";
13 import { bootHostedChild, bootHostedChildren, childAwake, closeHostedChildren, holdChild, resolveHostedChild, sleepLongestIdle, wakeGate } from "../src/spawn/host.ts";
14 import { hostedPlacement } from "../src/spawn/placement.ts";
15 import { quietLogs } from "./helpers/globals.ts";
16
17 quietLogs();
18
19 const NAMES = ["aa", "bb", "cc"];
20 let dir = "";
21 let parent: App;
22 const prior = process.env["FORGE_CHILD_AWAKE_MAX"];
23
24 const visit = (name: string) => resolveHostedChild(parent, new Request("https://forge.test/healthz", { headers: { host: `${name}--forge.test` } }));
25 const awake = () => NAMES.filter((n) => childAwake(parent, n));
26 const later = () => Date.now() + 60_000;
27
28 beforeAll(async () => {
29 dir = await mkdtemp(join(tmpdir(), "awake-hosted-"));
30 parent = createApp({ dataDir: dir, url: "https://forge.test", port: 0 });
31 const made = await createUser(parent.db, { origin: "seed", trusted: true, username: "alice", email: "[email protected]", isAdmin: true });
32 if (!made.ok) throw new Error("user");
33 for (const name of NAMES) {
34 const spawned = await spawnForge(
35 parent,
36 made.value,
37 { name, org: name, apps: [], carryBriefs: false, carrySubscriptions: false },
38 { catalog: [], briefs: [], subscriptions: [] },
39 {
40 placement: hostedPlacement({ host: "forge.test" }),
41 request: new Request("https://forge.test/spawn", { method: "POST", headers: { origin: "https://forge.test" } }),
42 url: new URL("https://forge.test/spawn"),
43 },
44 );
45 if (!spawned.ok) throw new Error(JSON.stringify(spawned.error));
46 }
47 });
48
49 beforeEach(() => {
50 closeHostedChildren(parent);
51 process.env["FORGE_CHILD_AWAKE_MAX"] = "2";
52 });
53
54 afterAll(async () => {
55 if (prior === undefined) delete process.env["FORGE_CHILD_AWAKE_MAX"];
56 else process.env["FORGE_CHILD_AWAKE_MAX"] = prior;
57 parent.close();
58 await rm(dir, { recursive: true, force: true });
59 });
60
61 test("the forge visited longest ago goes first, and none inside its grace goes at all", async () => {
62 await visit("bb");
63 await Bun.sleep(5); // real-timer: lets the clock tick so the two timestamps differ
64 await visit("aa");
65 expect(await sleepLongestIdle(parent, Date.now())).toBeUndefined(); // both just served
66 expect(awake().sort()).toEqual(["aa", "bb"]);
67 expect(await sleepLongestIdle(parent, later())).toBe("bb");
68 expect(awake()).toEqual(["aa"]);
69 });
70
71 test("a forge in a request is never the one put to sleep", async () => {
72 const bb = (await visit("bb")) as App;
73 await Bun.sleep(5); // real-timer: lets the clock tick so the two timestamps differ
74 await visit("aa");
75 const end = holdChild(parent, bb);
76 expect(await sleepLongestIdle(parent, later())).toBe("aa");
77 expect(awake()).toEqual(["bb"]);
78 end();
79 });
80
81 test("a forge the boot opened and nobody has asked for is not put to sleep to make room", async () => {
82 await bootHostedChildren(parent, new Set(["aa", "bb"]));
83 expect(awake().sort()).toEqual(["aa", "bb"]);
84 expect(await sleepLongestIdle(parent, later())).toBeUndefined();
85 expect(awake().sort()).toEqual(["aa", "bb"]);
86 });
87
88 test("at the ceiling a wake puts the idle one to sleep and goes ahead; with none idle it is told the forge is full", async () => {
89 const wake = (name: string) => wakeGate(parent, new Request("https://forge.test/", { headers: { host: `${name}--forge.test`, "x-forwarded-for": "9.9.9.9" } }));
90 await visit("aa");
91 await Bun.sleep(5); // real-timer: lets the clock tick so the two timestamps differ
92 await visit("bb");
93 // Both were just served: in use, so the third is refused in words.
94 const full = await wake("cc");
95 expect(full?.status).toBe(429);
96 expect(await full?.text()).toContain("the forge is full");
97 expect(awake().sort()).toEqual(["aa", "bb"]);
98 // Quiet past the grace: the one idle longest makes room.
99 const real = Date.now;
100 Date.now = () => real() + 60_000;
101 try {
102 expect(await wake("cc")).toBeUndefined();
103 } finally {
104 Date.now = real;
105 }
106 expect(childAwake(parent, "aa")).toBe(false);
107 expect(childAwake(parent, "bb")).toBe(true);
108 });
109
110 test("the boot never makes room; a spawn does", async () => {
111 await visit("aa");
112 await Bun.sleep(5); // real-timer: lets the clock tick so the two timestamps differ
113 await visit("bb");
114 const real = Date.now;
115 Date.now = () => real() + 60_000;
116 try {
117 await bootHostedChild(parent, "cc", false);
118 expect(awake().sort()).toEqual(["aa", "bb"]);
119 await bootHostedChild(parent, "cc", true);
120 } finally {
121 Date.now = real;
122 }
123 expect(awake().sort()).toEqual(["bb", "cc"]);
124 });
test/awake-room.test.ts added +128
@@ -0,0 +1,128 @@
1 /**
2 * At the awake ceiling the forge idle longest is put to sleep to make room, and a visitor is told
3 * the forge is full only when every awake forge is in use (#13831). The ceiling is a capacity
4 * setting.
5 */
6 import { afterAll, expect, test } from "bun:test";
7 import { mkdtempSync, rmSync } from "node:fs";
8 import { tmpdir } from "node:os";
9 import { join } from "node:path";
10 import { createApp } from "../src/app.ts";
11 import { awakeForMemory, capacityValue, CAPACITY_KEYS } from "../src/core/capacity.ts";
12 import { makeRegistry, cellChildren, type CellChild } from "../src/spawn/cell.ts";
13 import { roomFor } from "../src/spawn/host.ts";
14 import { quietLogs } from "./helpers/globals.ts";
15
16 quietLogs();
17 const tmp = mkdtempSync(join(tmpdir(), "awake-room-"));
18 afterAll(() => rmSync(tmp, { recursive: true, force: true }));
19 const GIB = 1024 ** 3;
20
21 test("the ceiling is a capacity setting, sized from memory: 180 GiB holds about 250", () => {
22 expect(CAPACITY_KEYS).toContain("children.awake");
23 expect(awakeForMemory(180 * GIB)).toBe(253);
24 expect(awakeForMemory(4 * GIB)).toBe(8);
25 });
26
27 test("the longest idle forge is the one put to sleep; one in a request is never", async () => {
28 const parent = createApp({ dataDir: join(tmp, "a"), url: "https://f.test" });
29 const stopped: string[] = [];
30 const procs: ReturnType<typeof Bun.spawn>[] = [];
31 const cells = makeRegistry(parent, async (input) => {
32 const proc = Bun.spawn(["sleep", "30"], { stdin: "ignore", stdout: "ignore", stderr: "ignore" });
33 procs.push(proc);
34 return { ...input, port: 1, limits: {} as CellChild["limits"], pid: proc.pid, exited: proc.exited };
35 });
36 const real = cells.stop.bind(cells);
37 cells.stop = async (id) => void (stopped.push(id), await real(id));
38 try {
39 for (const id of ["a", "b", "c"]) await cells.start({ bodyId: id, name: id, size: "small" });
40 const t = Date.now();
41 cells.touch("a", t + 3000);
42 cells.touch("b", t + 1000); // the longest idle
43 cells.touch("c", t + 2000);
44 const end = cells.hold("b");
45 // All three are new and were asked for: none may go.
46 expect(await cells.sleepLongestIdle(t + 5000, 0)).toBeUndefined();
47 // Past their first ten minutes, b is in a request, so c is the longest idle that is free.
48 const later = t + 11 * 60_000;
49 expect(await cells.sleepLongestIdle(later)).toBe("c");
50 end();
51 expect(await cells.sleepLongestIdle(later)).toBe("b");
52 expect(stopped).toEqual(["c", "b"]);
53 } finally {
54 for (const p of procs) p.kill("SIGKILL");
55 parent.close();
56 }
57 });
58
59 test("a new forge nobody has asked for since its birth may be put to sleep", async () => {
60 const parent = createApp({ dataDir: join(tmp, "n"), url: "https://f.test" });
61 const cells = makeRegistry(parent, async (input) => ({ ...input, port: 1, limits: {} as CellChild["limits"], pid: 0, exited: new Promise<number>(() => {}) }));
62 cells.stop = async () => {};
63 try {
64 await cells.start({ bodyId: "x", name: "x", size: "small" });
65 // Not one just born and served within the grace: that is in use.
66 expect(await cells.sleepLongestIdle(Date.now(), 10_000)).toBeUndefined();
67 expect(await cells.sleepLongestIdle(Date.now() + 11_000, 10_000)).toBe("x");
68 } finally {
69 parent.close();
70 }
71 });
72
73 test("at the ceiling a wake makes room and goes ahead; with every forge in use it is refused", async () => {
74 const prior = process.env["FORGE_CHILD_AWAKE_MAX"];
75 process.env["FORGE_CHILD_AWAKE_MAX"] = "2";
76 const parent = createApp({ dataDir: join(tmp, "r"), url: "https://f.test" });
77 const awake = new Set(["p", "q"]);
78 const fake = (id: string) => ({ bodyId: id, name: id, port: 1 }) as unknown as CellChild;
79 const cells = cellChildren(parent);
80 cells.list = () => [...awake].map(fake);
81 cells.get = (id) => (awake.has(id) ? fake(id) : undefined);
82 let inUse = false;
83 cells.sleepLongestIdle = async () => {
84 if (inUse) return undefined;
85 awake.delete("p");
86 return "p";
87 };
88 try {
89 expect(capacityValue(parent.db, "children.awake")).toBe("2");
90 expect(await roomFor(parent)).toBeUndefined();
91 expect(awake.has("p")).toBe(false);
92 awake.add("p");
93 inUse = true;
94 expect(await roomFor(parent)).toBe("the forge is full, try again shortly");
95 } finally {
96 parent.close();
97 if (prior === undefined) delete process.env["FORGE_CHILD_AWAKE_MAX"];
98 else process.env["FORGE_CHILD_AWAKE_MAX"] = prior;
99 }
100 });
101
102 test("two wakes at the ceiling at once put one forge to sleep, not two", async () => {
103 const prior = process.env["FORGE_CHILD_AWAKE_MAX"];
104 process.env["FORGE_CHILD_AWAKE_MAX"] = "2";
105 const parent = createApp({ dataDir: join(tmp, "c"), url: "https://f.test" });
106 const awake = new Set(["p", "q"]);
107 const fake = (id: string) => ({ bodyId: id, name: id, port: 1 }) as unknown as CellChild;
108 const cells = cellChildren(parent);
109 cells.list = () => [...awake].map(fake);
110 cells.get = (id) => (awake.has(id) ? fake(id) : undefined);
111 let slept = 0;
112 cells.sleepLongestIdle = async () => {
113 slept++;
114 await Bun.sleep(10); // real-timer: lets the clock tick so the two timestamps differ
115 awake.delete([...awake][0] as string);
116 return "x";
117 };
118 try {
119 const both = await Promise.all([roomFor(parent), roomFor(parent)]);
120 expect(both).toEqual([undefined, undefined]);
121 expect(slept).toBe(1);
122 expect(awake.size).toBe(1);
123 } finally {
124 parent.close();
125 if (prior === undefined) delete process.env["FORGE_CHILD_AWAKE_MAX"];
126 else process.env["FORGE_CHILD_AWAKE_MAX"] = prior;
127 }
128 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs