Public: anyone can read this; the crew can change it.
At the awake ceiling a visitor is told "the forge is full": nothing idle is put to sleep to make room, and boot and spawn do not count #13831
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/13812-at-the-awake-ceiling-a-visitor-is-told-t → main 8 commits fork
9 files
changed +373
−14
AGENTS.md +1 −1
| @@ -49,7 +49,7 @@ The check queue hands work to a runner through `src/checks/runner.ts` (#13652, R | ||
| 49 | 49 | `src/spawn/child_reach.ts`: `said`, when each hosted forge's refusal of each name was last written to the log (one `[egress] <child> was refused <host>:<port>` line an hour; pruned past 2,000 keys), and `calls`, each child's signed calls in the last minute; both gone on restart, and neither is state (asks and grants are rows: `child_reach_ask`, `child_reach_grant`). `src/actions/ask_host.ts` on a hosted forge: `looking` (the one refresh in flight), `lookedAt` (when the host was last asked, so a page is heard at most every 3 s) and `watching` (the one timer that looks every 10 s while an ask is under an hour old); a restart loses nothing, since every ask is a row (`reach_ask`) and the heartbeat looks again. The terminal door is `src/tui/screens/hostreach.ts` (palette entry for admins; `a` allows for that forge, `e` for every hosted forge, `n` not now, `x` takes a grant back; `allowReach`, `putAsideReach` and `revokeReach` in `actions/child_reach.ts` give the pages and the terminal the same gate and the `forge.reach_*` events). |
| 50 | 50 | |
| 51 | 51 | ## Awake ceiling and the boot |
| 52 | `FORGE_CHILD_AWAKE_MAX` counts forges in use. Hosted children the boot opens (so their apps come back) are exempt from `awakeCount` until a credentialed request reaches them or they sleep (`bootOpened` in `src/spawn/host.ts`). The exempt set holds only body ids of existing finished children and is emptied when the parent closes, so it is bounded by the number of children. Cell children are always counted. The boot starts only the forges that were awake and in use when the parent stopped (`awake-at-stop.json` under the data directory, written by `closeHostedChildren`, read and removed once by `bootHostedChildren`, `src/spawn/awake_at_stop.ts`), up to the ceiling; a crash leaves no file, so nothing starts and forges wake on their next request. A sleeping cell forge with a run due has its wake armed. The idle sweeper starts at boot. | |
| 52 | The capacity setting `children.awake` (boot default `FORGE_CHILD_AWAKE_MAX`) counts forges in use; a wake or a spawn at the ceiling puts the longest-idle forge to sleep first (`roomFor` in `src/spawn/host.ts`), and the boot never opens a child past it. Hosted children the boot opens (so their apps come back) are exempt from `awakeCount` until a credentialed request reaches them or they sleep (`bootOpened` in `src/spawn/host.ts`). The exempt set holds only body ids of existing finished children and is emptied when the parent closes, so it is bounded by the number of children. Cell children are always counted. The boot starts only the forges that were awake and in use when the parent stopped (`awake-at-stop.json` under the data directory, written by `closeHostedChildren`, read and removed once by `bootHostedChildren`, `src/spawn/awake_at_stop.ts`), up to the ceiling; a crash leaves no file, so nothing starts and forges wake on their next request. A sleeping cell forge with a run due has its wake armed. The idle sweeper starts at boot. | |
| 53 | 53 | |
| 54 | 54 | An app token's requests go behind people's and repeat reads are 304s (`src/http/app_turn.ts`, `repeat_reads.ts`, `perf/asking.ts`): see `docs/features/app-asking.md`. |
| 55 | 55 | `src/core/reports.ts`: `closedSet`, the ids of repositories whose owner closed reports from outside (#12321), read once per database and kept by `setReportsClosed`, the only writer, so every repository page's head link costs no statement; at most one id per closed repository, gone on restart (the rows are `report_closed`). `src/core/contribute_token.ts`: `openSet`, the ids of repositories that allow outside agents, read once per database and kept by `setContributeOpen`, the only writer, at most one id per allowing repository, gone on restart; a raw SQL write to `contribute_open` bypasses it. The terminal door for reports is `r` on the outside agents screen (`setOutsideReports`, admins). |
docs/features/forge-sleep.md +1 −1
| @@ -3,7 +3,7 @@ | ||
| 3 | 3 | A person (or their assistant, over the MCP `spawn` tool with their consent) makes a forge at `/forges/new`, the address `<name>--<this host>`. Free within the limits below; no payment path yet. |
| 4 | 4 | |
| 5 | 5 | - **Sleep.** A forge with no request for `FORGE_CHILD_IDLE_S` (600) sleeps: a hosted child's App and apps close, a cell child's process stops. Data stays under `children/<bodyId>`. The next page load, `git push` or API call wakes it. A page for a sleeping cell forge answers a plain "waking up" page at once (git and API calls wait up to 4 s) (503, `retry-after`, reloads itself); a hosted one opens in-process. |
| 6 | - **Limits, each refused in words.** The admin limits `forges.per_person` (3) and `forges.storage_mb` (1000 MB), set at `/admin/limits` (the same list is on the terminal's limits screen), each refused in a sentence naming the limit and the person's use (site admins exempt), `FORGE_CHILD_AWAKE_MAX` (8 awake at once; sleeping forges do not count), `FORGE_CHILD_WAKES_PER_HOUR` (20 per forge), the admin limit `forges.wakes_per_person_hour` (30 wakes an hour, counted against the forge's owner once someone has claimed it and against the forge itself until then, never against whoever sent it; a site admin's own forge is counted like anyone's; refused in a sentence that says whose wakes are spent, in how long the forge can be woken again, and the limit's name and where an admin raises it), plus the existing size/quota limits (`FORGE_CHILD_*_MIB`), and each cell's own disk cap. | |
| 6 | - **Limits, each refused in words.** The admin limits `forges.per_person` (3) and `forges.storage_mb` (1000 MB), set at `/admin/limits` (the same list is on the terminal's limits screen), each refused in a sentence naming the limit and the person's use (site admins exempt), the capacity setting `children.awake` (default from memory, at least 8, one per 720 MiB above 2 GiB; boot default `FORGE_CHILD_AWAKE_MAX`; sleeping forges do not count, new ones do; at the ceiling the forge idle longest, quiet for 10 s, is put to sleep to make room, and a wake is refused only when every awake forge is in use), `FORGE_CHILD_WAKES_PER_HOUR` (20 per forge), the admin limit `forges.wakes_per_person_hour` (30 wakes an hour, counted against the forge's owner once someone has claimed it and against the forge itself until then, never against whoever sent it; a site admin's own forge is counted like anyone's; refused in a sentence that says whose wakes are spent, in how long the forge can be woken again, and the limit's name and where an admin raises it), plus the existing size/quota limits (`FORGE_CHILD_*_MIB`), and each cell's own disk cap. | |
| 7 | 7 | - **Owner.** `/forges` shows awake/asleep, size, last activity, with Stop (sleep now) and Remove (ledger and data). |
| 8 | 8 | - **Secrets.** A child starts from `childEnv` and never sees this forge's model credential; the person brings their own or its crew stays off. |
| 9 | 9 |
src/core/capacity.ts +23
| @@ -26,6 +26,7 @@ export type CapacityKey = | ||
| 26 | 26 | | "queue.batch" |
| 27 | 27 | | "week.use" |
| 28 | 28 | | "children.max" |
| 29 | | "children.awake" | |
| 29 | 30 | | "cells.egress" |
| 30 | 31 | | "children.modelEgress" |
| 31 | 32 | | "children.registryEgress" |
| @@ -104,6 +105,13 @@ export const CAPACITY: Record<CapacityKey, Definition> = { | ||
| 104 | 105 | fallback: "", |
| 105 | 106 | help: "How many hosted child forges this one holds at once. Empty: one per 512 MiB of memory above this forge's own 2 GiB, from the pod's limit. Raising it lets the New forge door open again at once.", |
| 106 | 107 | }, |
| 108 | "children.awake": { | |
| 109 | label: "Forges awake at once", | |
| 110 | env: "FORGE_CHILD_AWAKE_MAX", | |
| 111 | kind: { type: "count", min: 1, max: 1000 }, | |
| 112 | fallback: "8", | |
| 113 | help: "How many hosted child forges run at once; a sleeping one does not count, and a new one counts from its birth. At the ceiling, the forge idle longest (no request under way, no crew at work) is put to sleep to make room, and a visitor is told the forge is full only when every awake forge is in use. Empty: one per 720 MiB of memory above this forge's own 2 GiB, from the pod's limit (an awake idle small forge costs about 490 MiB resident), at least 8. Read on every wake.", | |
| 114 | }, | |
| 107 | 115 | "cells.egress": { |
| 108 | 116 | label: "Egress from cells", |
| 109 | 117 | // No environment default: report is a decision an admin makes and the |
| @@ -202,6 +210,7 @@ export function capacityDefault(key: CapacityKey, cpus?: number): string { | ||
| 202 | 210 | const sized = CAPACITY[key].sized; |
| 203 | 211 | if (sized !== undefined) return String(sized(cpus ?? cpuLimit())); |
| 204 | 212 | if (key === "children.max") return String(childrenForMemory(memoryLimitBytes())); |
| 213 | if (key === "children.awake") return String(awakeForMemory(memoryLimitBytes())); | |
| 205 | 214 | if (key === "crew.concurrency") return String(crewForMemory(memoryLimitBytes())); |
| 206 | 215 | return CAPACITY[key].fallback; |
| 207 | 216 | } |
| @@ -359,6 +368,20 @@ export function childrenForMemory(bytes: number): number { | ||
| 359 | 368 | } |
| 360 | 369 | const MIN_CHILDREN = 5; |
| 361 | 370 | |
| 371 | /** | |
| 372 | * Forges awake at once by default: one per 720 MiB above the forge's own 2 GiB (an awake idle small | |
| 373 | * forge measured about 490 MiB resident; 180 GiB gives 250), never fewer than 8. | |
| 374 | */ | |
| 375 | export function awakeForMemory(bytes: number): number { | |
| 376 | const n = Math.floor((bytes - 2 * GIB) / (720 * MIB)); | |
| 377 | return Math.min(1000, Math.max(8, Number.isFinite(n) ? n : 8)); | |
| 378 | } | |
| 379 | ||
| 380 | /** How many hosted children may be awake now; read on every use. */ | |
| 381 | export function maxAwake(db: Database): number { | |
| 382 | return Number(capacityValue(db, "children.awake")); | |
| 383 | } | |
| 384 | ||
| 362 | 385 | /** |
| 363 | 386 | * One crew run per 3 GiB of room above the forge's own 2 GiB (32 GiB gives |
| 364 | 387 | * 10, 64 GiB the ceiling of 16), never fewer than two: a big machine runs |
src/spawn/cell.ts +19
| @@ -262,6 +262,12 @@ export interface CellChildren { | ||
| 262 | 262 | * disk). |
| 263 | 263 | */ |
| 264 | 264 | sleepIdle(idleMs: number, now?: number, onAsleep?: (name: string, bodyId: string) => void): Promise<string[]>; |
| 265 | /** | |
| 266 | * Stop the running child idle longest to make room: not mid-request, no crew at work. A new forge | |
| 267 | * nobody has asked for since its birth may be the one; one that was asked for keeps its ten | |
| 268 | * minutes. Its name, or undefined when every child is in use. | |
| 269 | */ | |
| 270 | sleepLongestIdle(now?: number, minIdleMs?: number): Promise<string | undefined>; | |
| 265 | 271 | } |
| 266 | 272 | |
| 267 | 273 | /** |
| @@ -552,6 +558,19 @@ export function makeRegistry(parent: App, launcher?: Launcher): CellChildren { | ||
| 552 | 558 | await Promise.all(Array.from({ length: Math.min(SLEEP_AT_ONCE, queue.length) }, worker)); |
| 553 | 559 | return asleep; |
| 554 | 560 | }, |
| 561 | async sleepLongestIdle(now = Date.now(), minIdleMs = 0) { | |
| 562 | const order = [...held.values()] | |
| 563 | .filter((c) => (busy.get(c.bodyId) ?? 0) === 0 && now - (seen.get(c.bodyId) ?? now) >= minIdleMs) | |
| 564 | .filter((c) => !isNewForge(startedAt.get(c.bodyId), now) || (seen.get(c.bodyId) ?? 0) <= (startedAt.get(c.bodyId) ?? 0)) | |
| 565 | .sort((a, b) => (seen.get(a.bodyId) ?? now) - (seen.get(b.bodyId) ?? now)); | |
| 566 | for (const child of order) { | |
| 567 | if (await childCrewAtWork(childDirOf(parent, child.bodyId), now)) continue; | |
| 568 | seen.delete(child.bodyId); | |
| 569 | await this.stop(child.bodyId); | |
| 570 | return child.name; | |
| 571 | } | |
| 572 | return undefined; | |
| 573 | }, | |
| 555 | 574 | }; |
| 556 | 575 | } |
| 557 | 576 |
src/spawn/host.ts +71 −7
| @@ -244,6 +244,58 @@ export async function sleepIdleChildren(parent: App, idleMs = idleAfterMs(), now | ||
| 244 | 244 | return asleep; |
| 245 | 245 | } |
| 246 | 246 | |
| 247 | /** | |
| 248 | * Put the forge idle longest to sleep to make room under the awake ceiling: not mid-request, no | |
| 249 | * crew at work; a hosted forge in this process first, then a cell's. The name, or undefined when | |
| 250 | * every awake forge is in use. | |
| 251 | */ | |
| 252 | export async function sleepLongestIdle(parent: App, now = Date.now()): Promise<string | undefined> { | |
| 253 | const cache = caches.get(parent); | |
| 254 | const unasked = bootOpened.get(parent); | |
| 255 | let oldest: { bodyId: string; child: App; name: string; at: number } | undefined; | |
| 256 | for (const [bodyId, child] of cache ?? []) { | |
| 257 | const mark = touched.get(parent)?.get(bodyId); | |
| 258 | if (mark === undefined || unasked?.has(bodyId) || now - mark.at < EVICT_GRACE_MS || working(parent, child)) continue; | |
| 259 | if (oldest === undefined || mark.at < oldest.at) oldest = { bodyId, child, name: mark.name, at: mark.at }; | |
| 260 | } | |
| 261 | if (oldest !== undefined && cache !== undefined) { | |
| 262 | await closeOne(parent, cache, oldest.bodyId, oldest.child, oldest.name); | |
| 263 | sleepHooks.get(parent)?.(); | |
| 264 | parent.log.info("child forge asleep to make room", { name: oldest.name }); | |
| 265 | return oldest.name; | |
| 266 | } | |
| 267 | const name = await cellChildren(parent).sleepLongestIdle(now, EVICT_GRACE_MS); | |
| 268 | if (name === undefined) return undefined; | |
| 269 | const ledger = readLedgerSync(parent, name); | |
| 270 | if (ledger !== undefined) { | |
| 271 | touched.get(parent)?.delete(ledger.bodyId); | |
| 272 | waking.get(parent)?.delete(ledger.bodyId); | |
| 273 | const due = await childNextRunDue(childDirOf(parent, ledger.bodyId)); | |
| 274 | if (due !== undefined) wakeWhenDue(parent, ledger, due); | |
| 275 | } | |
| 276 | parent.log.info("child forge asleep to make room", { name }); | |
| 277 | return name; | |
| 278 | } | |
| 279 | ||
| 280 | /** The room-making in progress for each parent: one at a time, so two wakes never sleep two forges. */ | |
| 281 | const roomLocks = new WeakMap<App, Promise<unknown>>(); | |
| 282 | ||
| 283 | /** | |
| 284 | * Why one more forge may not start now, or undefined: at the ceiling the forge idle longest makes | |
| 285 | * room first. One at a time per parent, and the count is read again inside the turn: a wake that | |
| 286 | * waited finds the room the one before it made, and puts no second forge to sleep for it. | |
| 287 | */ | |
| 288 | export function roomFor(parent: App, now = Date.now()): Promise<string | undefined> { | |
| 289 | const turn = (roomLocks.get(parent) ?? Promise.resolve()).then(async () => { | |
| 290 | const why = awakeRefusal(awakeCount(parent), process.env, parent.db); | |
| 291 | if (why === undefined) return undefined; | |
| 292 | if ((await sleepLongestIdle(parent, now)) === undefined) return why; | |
| 293 | return awakeRefusal(awakeCount(parent), process.env, parent.db); | |
| 294 | }); | |
| 295 | roomLocks.set(parent, turn.catch(() => undefined)); | |
| 296 | return turn; | |
| 297 | } | |
| 298 | ||
| 247 | 299 | /** |
| 248 | 300 | * What to do when a forge that ran in this process goes to sleep; set by whoever moves such forges |
| 249 | 301 | * into cells. |
| @@ -264,6 +316,11 @@ const dueWakes = new WeakMap<App, Map<string, ReturnType<typeof setTimeout>>>(); | ||
| 264 | 316 | * A due wake the awake ceiling or the disk floor turned away is tried again this much later, ms. |
| 265 | 317 | */ |
| 266 | 318 | const DUE_RETRY_MS = 60_000; |
| 319 | /** | |
| 320 | * A forge visited within this long is in use: making room never sleeps it, so a wake cannot thrash | |
| 321 | * one just served, ms. | |
| 322 | */ | |
| 323 | const EVICT_GRACE_MS = 10_000; | |
| 267 | 324 | /** A forge the host stopped may take the room of one quiet this long, ms. */ |
| 268 | 325 | const STOPPED_EVICT_IDLE_MS = 60_000; |
| 269 | 326 | /** The longest a timer waits in one step (setTimeout's own limit); a longer wait is armed again. */ |
| @@ -306,9 +363,9 @@ async function wakeDue(parent: App, name: string): Promise<void> { | ||
| 306 | 363 | if (isAwakeBody(parent, ledger.bodyId) || lapsedNone(parent, name) !== undefined) return; |
| 307 | 364 | // A forge the host stopped for a limit does not wait for an idle one's ten minutes: forges quiet |
| 308 | 365 | // for a minute and with no crew at work are put to sleep to make its room. |
| 309 | if (lastStop(parent, ledger.bodyId) !== undefined && awakeRefusal(awakeCount(parent)) !== undefined) | |
| 366 | if (lastStop(parent, ledger.bodyId) !== undefined && awakeRefusal(awakeCount(parent), process.env, parent.db) !== undefined) | |
| 310 | 367 | await sleepIdleChildren(parent, STOPPED_EVICT_IDLE_MS); |
| 311 | if (floorRefusal(parent.config.dataDir) !== undefined || awakeRefusal(awakeCount(parent)) !== undefined) { | |
| 368 | if (floorRefusal(parent.config.dataDir) !== undefined || awakeRefusal(awakeCount(parent), process.env, parent.db) !== undefined) { | |
| 312 | 369 | wakeWhenDue(parent, ledger, Date.now() + DUE_RETRY_MS); |
| 313 | 370 | tryAgainAt(parent, ledger.bodyId, Date.now() + DUE_RETRY_MS); |
| 314 | 371 | return; |
| @@ -581,7 +638,7 @@ export async function wakeGate(parent: App, request: Request): Promise<Response | ||
| 581 | 638 | const wait = people.waitS(budget.key, max, at); |
| 582 | 639 | if (wait > 0) |
| 583 | 640 | return wakingResponse(parent, request, name, budgetRefusal({ name, budget, max, waitS: wait, limitsUrl: `${parentOrigin(parent.config)}/admin/limits` }), wait); |
| 584 | const why = awakeRefusal(awakeCount(parent)) ?? (known ? undefined : rates.refusal(ledger.bodyId, at)); | |
| 641 | const why = (await roomFor(parent)) ?? (known ? undefined : rates.refusal(ledger.bodyId, at)); | |
| 585 | 642 | if (why !== undefined) { |
| 586 | 643 | const stop = lastStop(parent, ledger.bodyId); |
| 587 | 644 | return wakingResponse(parent, request, name, stop === undefined ? why : stopSentence(stop)); |
| @@ -850,10 +907,17 @@ function openOnce(parent: App, ledger: ChildLedger): Promise<App> | App { | ||
| 850 | 907 | * Open one finished child now, which starts its apps: after a spawn, so |
| 851 | 908 | * the first visit after a claim finds them awake. Never throws. |
| 852 | 909 | */ |
| 853 | export async function bootHostedChild(parent: App, name: string): Promise<void> { | |
| 910 | export async function bootHostedChild(parent: App, name: string, evict = true): Promise<void> { | |
| 854 | 911 | let ledger = readLedgerSync(parent, name); |
| 855 | 912 | if (ledger === undefined || !ledger.done || ledger.moving) return; |
| 856 | if (!isAwakeBody(parent, ledger.bodyId) && startRefused(parent, ledger) !== undefined) return; | |
| 913 | if (!isAwakeBody(parent, ledger.bodyId)) { | |
| 914 | if (startRefused(parent, ledger) !== undefined) return; | |
| 915 | // A new forge counts toward the awake ceiling. A person's spawn makes room by putting the | |
| 916 | // longest idle to sleep; the boot never does, so a restart cannot sleep what was in use. A | |
| 917 | // forge left asleep wakes at its first request. | |
| 918 | const full = evict ? await roomFor(parent) : awakeRefusal(awakeCount(parent), process.env, parent.db); | |
| 919 | if (full !== undefined) return; | |
| 920 | } | |
| 857 | 921 | if (ledger.placement === "cell") { |
| 858 | 922 | if ((await settleCell(parent, ledger)) !== undefined) return; |
| 859 | 923 | // Still a cell child unless this host holds no cell at all: it is opened here only then. |
| @@ -900,9 +964,9 @@ export async function bootHostedChildren(parent: App, wasAwake: ReadonlySet<stri | ||
| 900 | 964 | continue; |
| 901 | 965 | } |
| 902 | 966 | // Boot-opened forges are exempt from `awakeCount`, so the boot counts its own. |
| 903 | if (awakeRefusal(awakeCount(parent) + (bootOpened.get(parent)?.size ?? 0)) !== undefined) continue; | |
| 967 | if (awakeRefusal(awakeCount(parent) + (bootOpened.get(parent)?.size ?? 0), process.env, parent.db) !== undefined) continue; | |
| 904 | 968 | const was = caches.get(parent)?.has(ledger.bodyId) === true; |
| 905 | await bootHostedChild(parent, name); | |
| 969 | await bootHostedChild(parent, name, false); | |
| 906 | 970 | if (!was && caches.get(parent)?.has(ledger.bodyId) === true) { |
| 907 | 971 | let set = bootOpened.get(parent); |
| 908 | 972 | if (!set) bootOpened.set(parent, (set = new Set())); |
src/spawn/move_cell.ts +1 −1
| @@ -358,7 +358,7 @@ async function move( | ||
| 358 | 358 | `${name} is awake in this forge's process; it is moved when it next sleeps`, |
| 359 | 359 | ); |
| 360 | 360 | // Its cell needs a place among the forges awake; closed here, it gives its own back first. |
| 361 | if (awakeRefusal(awakeCount(app) - (open ? 1 : 0)) !== undefined) | |
| 361 | if (awakeRefusal(awakeCount(app) - (open ? 1 : 0), process.env, app.db) !== undefined) | |
| 362 | 362 | return no( |
| 363 | 363 | "busy", |
| 364 | 364 | `${name} is not moved yet: this host has as many forges awake as it holds, and it is moved when one sleeps`, |
src/spawn/sleep.ts +5 −4
| @@ -14,6 +14,7 @@ | ||
| 14 | 14 | import type { Database } from "bun:sqlite"; |
| 15 | 15 | import { readdirSync, readFileSync, statSync } from "node:fs"; |
| 16 | 16 | import { join } from "node:path"; |
| 17 | import { maxAwake } from "../core/capacity.ts"; | |
| 17 | 18 | import { LIMITS, limitValue } from "../settings/limits.ts"; |
| 18 | 19 | |
| 19 | 20 | type Env = Record<string, string | undefined>; |
| @@ -28,8 +29,8 @@ function whole(env: Env, name: string, fallback: number): number { | ||
| 28 | 29 | /** A forge quiet for this long sleeps: FORGE_CHILD_IDLE_S, ten minutes by default. */ |
| 29 | 30 | export const idleAfterMs = (env: Env = process.env): number => whole(env, "FORGE_CHILD_IDLE_S", 600) * 1000; |
| 30 | 31 | /** |
| 31 | * Most hosted forges awake at once on this host: FORGE_CHILD_AWAKE_MAX. A sleeping forge does not | |
| 32 | * count. | |
| 32 | * Most hosted forges awake at once from the environment alone (the capacity setting | |
| 33 | * `children.awake` is read where a database is at hand). A sleeping forge does not count. | |
| 33 | 34 | */ |
| 34 | 35 | export const awakeCeiling = (env: Env = process.env): number => whole(env, "FORGE_CHILD_AWAKE_MAX", 8); |
| 35 | 36 | /** Most wakes of one forge in an hour: FORGE_CHILD_WAKES_PER_HOUR. */ |
| @@ -104,8 +105,8 @@ export function budgetRefusal(input: { readonly name: string; readonly budget: W | ||
| 104 | 105 | } |
| 105 | 106 | |
| 106 | 107 | /** Why this host will not wake one more forge while `awake` are running, or undefined. */ |
| 107 | export function awakeRefusal(awake: number, env: Env = process.env): string | undefined { | |
| 108 | const max = awakeCeiling(env); | |
| 108 | export function awakeRefusal(awake: number, env: Env = process.env, db?: Database): string | undefined { | |
| 109 | const max = db === undefined ? awakeCeiling(env) : maxAwake(db); | |
| 109 | 110 | return awake >= max |
| 110 | 111 | ? "the forge is full, try again shortly" |
| 111 | 112 | : undefined; |
test/awake-room-hosted.test.ts added +124
| @@ -0,0 +1,124 @@ | ||
| 1 | /** | |
| 2 | * Making room under the awake ceiling for forges hosted in this process (#13831): the oldest | |
| 3 | * visited goes first, none in use or inside its grace does, a wake that made room goes ahead, and a | |
| 4 | * boot never sleeps what was in use while a spawn may. | |
| 5 | */ | |
| 6 | import { afterAll, beforeAll, beforeEach, expect, test } from "bun:test"; | |
| 7 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 8 | import { tmpdir } from "node:os"; | |
| 9 | import { join } from "node:path"; | |
| 10 | import { spawnForge } from "../src/actions/spawn.ts"; | |
| 11 | import { createApp, type App } from "../src/app.ts"; | |
| 12 | import { createUser } from "../src/core/users.ts"; | |
| 13 | import { bootHostedChild, bootHostedChildren, childAwake, closeHostedChildren, holdChild, resolveHostedChild, sleepLongestIdle, wakeGate } from "../src/spawn/host.ts"; | |
| 14 | import { hostedPlacement } from "../src/spawn/placement.ts"; | |
| 15 | import { quietLogs } from "./helpers/globals.ts"; | |
| 16 | ||
| 17 | quietLogs(); | |
| 18 | ||
| 19 | const NAMES = ["aa", "bb", "cc"]; | |
| 20 | let dir = ""; | |
| 21 | let parent: App; | |
| 22 | const prior = process.env["FORGE_CHILD_AWAKE_MAX"]; | |
| 23 | ||
| 24 | const visit = (name: string) => resolveHostedChild(parent, new Request("https://forge.test/healthz", { headers: { host: `${name}--forge.test` } })); | |
| 25 | const awake = () => NAMES.filter((n) => childAwake(parent, n)); | |
| 26 | const later = () => Date.now() + 60_000; | |
| 27 | ||
| 28 | beforeAll(async () => { | |
| 29 | dir = await mkdtemp(join(tmpdir(), "awake-hosted-")); | |
| 30 | parent = createApp({ dataDir: dir, url: "https://forge.test", port: 0 }); | |
| 31 | const made = await createUser(parent.db, { origin: "seed", trusted: true, username: "alice", email: "[email protected]", isAdmin: true }); | |
| 32 | if (!made.ok) throw new Error("user"); | |
| 33 | for (const name of NAMES) { | |
| 34 | const spawned = await spawnForge( | |
| 35 | parent, | |
| 36 | made.value, | |
| 37 | { name, org: name, apps: [], carryBriefs: false, carrySubscriptions: false }, | |
| 38 | { catalog: [], briefs: [], subscriptions: [] }, | |
| 39 | { | |
| 40 | placement: hostedPlacement({ host: "forge.test" }), | |
| 41 | request: new Request("https://forge.test/spawn", { method: "POST", headers: { origin: "https://forge.test" } }), | |
| 42 | url: new URL("https://forge.test/spawn"), | |
| 43 | }, | |
| 44 | ); | |
| 45 | if (!spawned.ok) throw new Error(JSON.stringify(spawned.error)); | |
| 46 | } | |
| 47 | }); | |
| 48 | ||
| 49 | beforeEach(() => { | |
| 50 | closeHostedChildren(parent); | |
| 51 | process.env["FORGE_CHILD_AWAKE_MAX"] = "2"; | |
| 52 | }); | |
| 53 | ||
| 54 | afterAll(async () => { | |
| 55 | if (prior === undefined) delete process.env["FORGE_CHILD_AWAKE_MAX"]; | |
| 56 | else process.env["FORGE_CHILD_AWAKE_MAX"] = prior; | |
| 57 | parent.close(); | |
| 58 | await rm(dir, { recursive: true, force: true }); | |
| 59 | }); | |
| 60 | ||
| 61 | test("the forge visited longest ago goes first, and none inside its grace goes at all", async () => { | |
| 62 | await visit("bb"); | |
| 63 | await Bun.sleep(5); // real-timer: lets the clock tick so the two timestamps differ | |
| 64 | await visit("aa"); | |
| 65 | expect(await sleepLongestIdle(parent, Date.now())).toBeUndefined(); // both just served | |
| 66 | expect(awake().sort()).toEqual(["aa", "bb"]); | |
| 67 | expect(await sleepLongestIdle(parent, later())).toBe("bb"); | |
| 68 | expect(awake()).toEqual(["aa"]); | |
| 69 | }); | |
| 70 | ||
| 71 | test("a forge in a request is never the one put to sleep", async () => { | |
| 72 | const bb = (await visit("bb")) as App; | |
| 73 | await Bun.sleep(5); // real-timer: lets the clock tick so the two timestamps differ | |
| 74 | await visit("aa"); | |
| 75 | const end = holdChild(parent, bb); | |
| 76 | expect(await sleepLongestIdle(parent, later())).toBe("aa"); | |
| 77 | expect(awake()).toEqual(["bb"]); | |
| 78 | end(); | |
| 79 | }); | |
| 80 | ||
| 81 | test("a forge the boot opened and nobody has asked for is not put to sleep to make room", async () => { | |
| 82 | await bootHostedChildren(parent, new Set(["aa", "bb"])); | |
| 83 | expect(awake().sort()).toEqual(["aa", "bb"]); | |
| 84 | expect(await sleepLongestIdle(parent, later())).toBeUndefined(); | |
| 85 | expect(awake().sort()).toEqual(["aa", "bb"]); | |
| 86 | }); | |
| 87 | ||
| 88 | test("at the ceiling a wake puts the idle one to sleep and goes ahead; with none idle it is told the forge is full", async () => { | |
| 89 | const wake = (name: string) => wakeGate(parent, new Request("https://forge.test/", { headers: { host: `${name}--forge.test`, "x-forwarded-for": "9.9.9.9" } })); | |
| 90 | await visit("aa"); | |
| 91 | await Bun.sleep(5); // real-timer: lets the clock tick so the two timestamps differ | |
| 92 | await visit("bb"); | |
| 93 | // Both were just served: in use, so the third is refused in words. | |
| 94 | const full = await wake("cc"); | |
| 95 | expect(full?.status).toBe(429); | |
| 96 | expect(await full?.text()).toContain("the forge is full"); | |
| 97 | expect(awake().sort()).toEqual(["aa", "bb"]); | |
| 98 | // Quiet past the grace: the one idle longest makes room. | |
| 99 | const real = Date.now; | |
| 100 | Date.now = () => real() + 60_000; | |
| 101 | try { | |
| 102 | expect(await wake("cc")).toBeUndefined(); | |
| 103 | } finally { | |
| 104 | Date.now = real; | |
| 105 | } | |
| 106 | expect(childAwake(parent, "aa")).toBe(false); | |
| 107 | expect(childAwake(parent, "bb")).toBe(true); | |
| 108 | }); | |
| 109 | ||
| 110 | test("the boot never makes room; a spawn does", async () => { | |
| 111 | await visit("aa"); | |
| 112 | await Bun.sleep(5); // real-timer: lets the clock tick so the two timestamps differ | |
| 113 | await visit("bb"); | |
| 114 | const real = Date.now; | |
| 115 | Date.now = () => real() + 60_000; | |
| 116 | try { | |
| 117 | await bootHostedChild(parent, "cc", false); | |
| 118 | expect(awake().sort()).toEqual(["aa", "bb"]); | |
| 119 | await bootHostedChild(parent, "cc", true); | |
| 120 | } finally { | |
| 121 | Date.now = real; | |
| 122 | } | |
| 123 | expect(awake().sort()).toEqual(["bb", "cc"]); | |
| 124 | }); |
test/awake-room.test.ts added +128
| @@ -0,0 +1,128 @@ | ||
| 1 | /** | |
| 2 | * At the awake ceiling the forge idle longest is put to sleep to make room, and a visitor is told | |
| 3 | * the forge is full only when every awake forge is in use (#13831). The ceiling is a capacity | |
| 4 | * setting. | |
| 5 | */ | |
| 6 | import { afterAll, expect, test } from "bun:test"; | |
| 7 | import { mkdtempSync, rmSync } from "node:fs"; | |
| 8 | import { tmpdir } from "node:os"; | |
| 9 | import { join } from "node:path"; | |
| 10 | import { createApp } from "../src/app.ts"; | |
| 11 | import { awakeForMemory, capacityValue, CAPACITY_KEYS } from "../src/core/capacity.ts"; | |
| 12 | import { makeRegistry, cellChildren, type CellChild } from "../src/spawn/cell.ts"; | |
| 13 | import { roomFor } from "../src/spawn/host.ts"; | |
| 14 | import { quietLogs } from "./helpers/globals.ts"; | |
| 15 | ||
| 16 | quietLogs(); | |
| 17 | const tmp = mkdtempSync(join(tmpdir(), "awake-room-")); | |
| 18 | afterAll(() => rmSync(tmp, { recursive: true, force: true })); | |
| 19 | const GIB = 1024 ** 3; | |
| 20 | ||
| 21 | test("the ceiling is a capacity setting, sized from memory: 180 GiB holds about 250", () => { | |
| 22 | expect(CAPACITY_KEYS).toContain("children.awake"); | |
| 23 | expect(awakeForMemory(180 * GIB)).toBe(253); | |
| 24 | expect(awakeForMemory(4 * GIB)).toBe(8); | |
| 25 | }); | |
| 26 | ||
| 27 | test("the longest idle forge is the one put to sleep; one in a request is never", async () => { | |
| 28 | const parent = createApp({ dataDir: join(tmp, "a"), url: "https://f.test" }); | |
| 29 | const stopped: string[] = []; | |
| 30 | const procs: ReturnType<typeof Bun.spawn>[] = []; | |
| 31 | const cells = makeRegistry(parent, async (input) => { | |
| 32 | const proc = Bun.spawn(["sleep", "30"], { stdin: "ignore", stdout: "ignore", stderr: "ignore" }); | |
| 33 | procs.push(proc); | |
| 34 | return { ...input, port: 1, limits: {} as CellChild["limits"], pid: proc.pid, exited: proc.exited }; | |
| 35 | }); | |
| 36 | const real = cells.stop.bind(cells); | |
| 37 | cells.stop = async (id) => void (stopped.push(id), await real(id)); | |
| 38 | try { | |
| 39 | for (const id of ["a", "b", "c"]) await cells.start({ bodyId: id, name: id, size: "small" }); | |
| 40 | const t = Date.now(); | |
| 41 | cells.touch("a", t + 3000); | |
| 42 | cells.touch("b", t + 1000); // the longest idle | |
| 43 | cells.touch("c", t + 2000); | |
| 44 | const end = cells.hold("b"); | |
| 45 | // All three are new and were asked for: none may go. | |
| 46 | expect(await cells.sleepLongestIdle(t + 5000, 0)).toBeUndefined(); | |
| 47 | // Past their first ten minutes, b is in a request, so c is the longest idle that is free. | |
| 48 | const later = t + 11 * 60_000; | |
| 49 | expect(await cells.sleepLongestIdle(later)).toBe("c"); | |
| 50 | end(); | |
| 51 | expect(await cells.sleepLongestIdle(later)).toBe("b"); | |
| 52 | expect(stopped).toEqual(["c", "b"]); | |
| 53 | } finally { | |
| 54 | for (const p of procs) p.kill("SIGKILL"); | |
| 55 | parent.close(); | |
| 56 | } | |
| 57 | }); | |
| 58 | ||
| 59 | test("a new forge nobody has asked for since its birth may be put to sleep", async () => { | |
| 60 | const parent = createApp({ dataDir: join(tmp, "n"), url: "https://f.test" }); | |
| 61 | const cells = makeRegistry(parent, async (input) => ({ ...input, port: 1, limits: {} as CellChild["limits"], pid: 0, exited: new Promise<number>(() => {}) })); | |
| 62 | cells.stop = async () => {}; | |
| 63 | try { | |
| 64 | await cells.start({ bodyId: "x", name: "x", size: "small" }); | |
| 65 | // Not one just born and served within the grace: that is in use. | |
| 66 | expect(await cells.sleepLongestIdle(Date.now(), 10_000)).toBeUndefined(); | |
| 67 | expect(await cells.sleepLongestIdle(Date.now() + 11_000, 10_000)).toBe("x"); | |
| 68 | } finally { | |
| 69 | parent.close(); | |
| 70 | } | |
| 71 | }); | |
| 72 | ||
| 73 | test("at the ceiling a wake makes room and goes ahead; with every forge in use it is refused", async () => { | |
| 74 | const prior = process.env["FORGE_CHILD_AWAKE_MAX"]; | |
| 75 | process.env["FORGE_CHILD_AWAKE_MAX"] = "2"; | |
| 76 | const parent = createApp({ dataDir: join(tmp, "r"), url: "https://f.test" }); | |
| 77 | const awake = new Set(["p", "q"]); | |
| 78 | const fake = (id: string) => ({ bodyId: id, name: id, port: 1 }) as unknown as CellChild; | |
| 79 | const cells = cellChildren(parent); | |
| 80 | cells.list = () => [...awake].map(fake); | |
| 81 | cells.get = (id) => (awake.has(id) ? fake(id) : undefined); | |
| 82 | let inUse = false; | |
| 83 | cells.sleepLongestIdle = async () => { | |
| 84 | if (inUse) return undefined; | |
| 85 | awake.delete("p"); | |
| 86 | return "p"; | |
| 87 | }; | |
| 88 | try { | |
| 89 | expect(capacityValue(parent.db, "children.awake")).toBe("2"); | |
| 90 | expect(await roomFor(parent)).toBeUndefined(); | |
| 91 | expect(awake.has("p")).toBe(false); | |
| 92 | awake.add("p"); | |
| 93 | inUse = true; | |
| 94 | expect(await roomFor(parent)).toBe("the forge is full, try again shortly"); | |
| 95 | } finally { | |
| 96 | parent.close(); | |
| 97 | if (prior === undefined) delete process.env["FORGE_CHILD_AWAKE_MAX"]; | |
| 98 | else process.env["FORGE_CHILD_AWAKE_MAX"] = prior; | |
| 99 | } | |
| 100 | }); | |
| 101 | ||
| 102 | test("two wakes at the ceiling at once put one forge to sleep, not two", async () => { | |
| 103 | const prior = process.env["FORGE_CHILD_AWAKE_MAX"]; | |
| 104 | process.env["FORGE_CHILD_AWAKE_MAX"] = "2"; | |
| 105 | const parent = createApp({ dataDir: join(tmp, "c"), url: "https://f.test" }); | |
| 106 | const awake = new Set(["p", "q"]); | |
| 107 | const fake = (id: string) => ({ bodyId: id, name: id, port: 1 }) as unknown as CellChild; | |
| 108 | const cells = cellChildren(parent); | |
| 109 | cells.list = () => [...awake].map(fake); | |
| 110 | cells.get = (id) => (awake.has(id) ? fake(id) : undefined); | |
| 111 | let slept = 0; | |
| 112 | cells.sleepLongestIdle = async () => { | |
| 113 | slept++; | |
| 114 | await Bun.sleep(10); // real-timer: lets the clock tick so the two timestamps differ | |
| 115 | awake.delete([...awake][0] as string); | |
| 116 | return "x"; | |
| 117 | }; | |
| 118 | try { | |
| 119 | const both = await Promise.all([roomFor(parent), roomFor(parent)]); | |
| 120 | expect(both).toEqual([undefined, undefined]); | |
| 121 | expect(slept).toBe(1); | |
| 122 | expect(awake.size).toBe(1); | |
| 123 | } finally { | |
| 124 | parent.close(); | |
| 125 | if (prior === undefined) delete process.env["FORGE_CHILD_AWAKE_MAX"]; | |
| 126 | else process.env["FORGE_CHILD_AWAKE_MAX"] = prior; | |
| 127 | } | |
| 128 | }); |
new above
offline