Public: anyone can read this; the crew can change it.
The limits that bound a surge of new forges are constants and env values, not settings, and one of them is silent #13835
Check · Review · Land · Live done — Landed and live in production.
mason/forge:mason/13822-the-limits-that-bound-a-surge-of-new-for → main 22 commits fork
34 files
changed +768
−60
AGENTS.md +1 −1
| @@ -57,7 +57,7 @@ The capacity setting `children.awake` (boot default `FORGE_CHILD_AWAKE_MAX`) cou | ||
| 57 | 57 | |
| 58 | 58 | An app token's requests go behind people's and repeat reads are 304s (`src/http/app_turn.ts`, `repeat_reads.ts`, `perf/asking.ts`): see `docs/features/app-asking.md`. |
| 59 | 59 | `src/core/reports.ts`: `closedSet`, the ids of repositories whose owner closed reports from outside (#12321), read once per database and kept by `setReportsClosed`, the only writer, so every repository page's head link costs no statement; at most one id per closed repository, gone on restart (the rows are `report_closed`). `src/core/contribute_token.ts`: `openSet`, the ids of repositories that allow outside agents, read once per database and kept by `setContributeOpen`, the only writer, at most one id per allowing repository, gone on restart; a raw SQL write to `contribute_open` bypasses it. The terminal door for reports is `r` on the outside agents screen (`setOutsideReports`, admins). |
| 60 | ||
| 60 | `src/spawn/births.ts`: `states`, per forge, the births in flight, the spawns waiting their turn (first in, first out, signed-in people ahead of strangers; at most 4 × `spawn.births_at_once` wait, strangers at most `spawn.births_at_once` of those, the rest are answered at once with their place; each waits 15 s at most, then is answered with its place) and the mean of the births measured (#13822); a birth is counted until its forge answers or 5 minutes pass; at most `spawn.births_at_once` in flight, gone on restart. The limits that bound a surge (`spawn.*`, `forges.wakes_per_forge_hour`) are settings on /admin/limits. | |
| 61 | 61 | `src/web/studio/waiting_on_me.ts`: `waitingRows`, each viewer's issues and pulls that wait for them, held 10 seconds per viewer (at most 500 viewers, gone on restart) so the board's live lanes share one `decisionsFor` read; the terminal door is `src/tui/screens/waiting_for_me.ts` (palette entry for any person). |
| 62 | 62 | `src/crew/beat.ts`: `lastBeat`, when each repository's beat last ran, and `declared`, the `beat` last read from each default branch with the commit it came from (#13709); at most one entry each per repository, gone on restart (a restart beats once at once; the declaration is a file in git). |
| 63 | 63 | `src/perf/job_alloc.ts`: `counted`, called by `during` around each job's synchronous slice, adds the JS heap's growth to the job's name (at most 128 names); the minute memory line reads and empties it (`memory allocated by jobs:`, #13924), so a timer that allocates with no request running is named. `bench/idle.ts` is the idle budget on a seeded forge (heap under 20 MiB a minute, no stall over 100 ms). Gone on restart. |
docs/MCP.md +1 −1
| @@ -148,7 +148,7 @@ bytes in `tools/list`. After changing a tool run `bun scripts/mcp-table.ts`; | ||
| 148 | 148 | | `land_past_gate` | — | Land one pull past a broken gate, with a recorded reason; an owner's only | 321 | |
| 149 | 149 | | `lift` | — | Lift an issue out from behind the Later gate | 240 | |
| 150 | 150 | | `limb` | `defer`, `resume`, `close`, `raise`, `lower`, `replan`, `takeback`, `undo` | Act on a limb by `action` | 532 | |
| 151 | | `limits` | — | Every limit the owner can hit | 1101 | | |
| 151 | | `limits` | — | Every limit the owner can hit | 1246 | | |
| 152 | 152 | | `maintenance` | `status`, `prepare`, `resume`, `pause-duty`, `resume-duty` | Prepare the forge for a power-off, check the drain, or end maintenance (admin) | 308 | |
| 153 | 153 | | `make_it_yours` | — | Fork an app you can see: code and history, following the original | 337 | |
| 154 | 154 | | `map` | — | An app's plan: ready or waiting, critical path, crew, parkedTwice; `done` adds closed; `first: N` ranks N first | 521 | |
docs/features/forge-sleep.md +4
| @@ -25,3 +25,7 @@ The apps a forge runs sleep too, and the profit slider on the economics page say | ||
| 25 | 25 | - **Pin the preview age.** Setting `FORGE_PREVIEW_IDLE_DAYS` on the forge fixes a quiet pull's preview age, whatever the position. `0` keeps every preview warm. Leave it unset to follow the slider. |
| 26 | 26 | |
| 27 | 27 | Code: `src/apps/sleep_window.ts` (windows, `previewIdleMs`), `src/apps/runtime.ts` (the sweep). Tests: `test/sleep-window.test.ts`, `test/apps-sleep-slider.test.ts`, `test/preview-idle-slider.test.ts`. |
| 28 | ||
| 29 | ## A surge of new forges (#13822) | |
| 30 | ||
| 31 | A new forge takes a core and about 1.5 GiB until it answers, and is not put to sleep meanwhile. One rule admits births (`src/spawn/births.ts`): at most `spawn.births_at_once` at a time, with `spawn.birth_memory_mb` free above the memory floor and the disk above its floor. A spawn that does not fit waits up to 15 s in line, then is answered `503` with `error: "queued"`, its `place`, `aboutS` and `retryAfter`, and a `Retry-After` header of the same seconds; the home start form says the place and the wait in a sentence. Each birth in flight counts its full `spawn.birth_memory_mb` against what is free, since it takes that over the minutes until it answers. `/admin/limits` and the terminal limits screen show the births being made and the spawns waiting next to the limits. The anonymous daily counts (`spawn.anon_per_day`, `spawn.anon_global_per_day`), the new-forge timers (`spawn.timers_max`; a forge past them is made but not timed, and the log says so), and wakes of one forge an hour (`forges.wakes_per_forge_hour`) are settings on `/admin/limits`; the environment variable still applies while the last is unset. The awake ceiling is the capacity setting `children.awake`. |
forge.json +2 −2
| @@ -5,8 +5,8 @@ | ||
| 5 | 5 | "lab--forge-trevato-apps.open-platform.sh" |
| 6 | 6 | ], |
| 7 | 7 | "app": { |
| 8 | "looks": ["/", "/docs", "/explore", "/board", "/notes", "/review", "/releases", "/apps", "/security", "/system", "/maintenance", "/talk", "/flow", "/resilience", "/boards?help", "/settings/crew", "/settings/crew?help", "/settings/security/controls", "/settings/grants", "/preview-admin/sample", "/preview-admin/sample/drop", "/preview-admin/sample/data-move", "/preview-admin/sample/cut-over", "/preview-admin/sample/pulls/3", "/preview-admin/sample/next?pick", "/preview-admin/sample/bets", "/preview-admin/sample/waves", "/preview-admin/sample/space", "/looks/missing", "/looks/failed", "/nope/nothing"], | |
| 9 | "looksAsOwner": ["/", "/board", "/flow", "/boards?help", "/settings/crew", "/settings/crew?help", "/settings/security/controls", "/settings/grants", "/talk", "/system", "/maintenance", "/resilience", "/preview-admin/sample", "/preview-admin/sample/drop", "/preview-admin/sample/data-move", "/preview-admin/sample/cut-over", "/preview-admin/sample/pulls/3", "/preview-admin/sample/next?pick", "/preview-admin/sample/bets", "/preview-admin/sample/waves", "/preview-admin/sample/space"], | |
| 8 | "looks": ["/", "/docs", "/explore", "/board", "/notes", "/review", "/releases", "/apps", "/security", "/system", "/admin/limits", "/maintenance", "/talk", "/flow", "/resilience", "/boards?help", "/settings/crew", "/settings/crew?help", "/settings/security/controls", "/settings/grants", "/preview-admin/sample", "/preview-admin/sample/drop", "/preview-admin/sample/data-move", "/preview-admin/sample/cut-over", "/preview-admin/sample/pulls/3", "/preview-admin/sample/next?pick", "/preview-admin/sample/bets", "/preview-admin/sample/waves", "/preview-admin/sample/space", "/looks/missing", "/looks/failed", "/nope/nothing"], | |
| 9 | "looksAsOwner": ["/", "/board", "/flow", "/boards?help", "/settings/crew", "/settings/crew?help", "/settings/security/controls", "/settings/grants", "/talk", "/system", "/admin/limits", "/maintenance", "/resilience", "/preview-admin/sample", "/preview-admin/sample/drop", "/preview-admin/sample/data-move", "/preview-admin/sample/cut-over", "/preview-admin/sample/pulls/3", "/preview-admin/sample/next?pick", "/preview-admin/sample/bets", "/preview-admin/sample/waves", "/preview-admin/sample/space"], | |
| 10 | 10 | "run": "bun run src/main.ts", |
| 11 | 11 | "install": "", |
| 12 | 12 | "health": "/healthz", |
scripts/unused-exports-allow.json +3
| @@ -155,6 +155,9 @@ | ||
| 155 | 155 | "src/runs/log.ts runLogTapCount", |
| 156 | 156 | "src/runs/stoppable.ts runningRun", |
| 157 | 157 | "src/snapshot/drill_job.ts markerLeft", |
| 158 | "src/spawn/births.ts provideHoldForTest", | |
| 159 | "src/spawn/births.ts provideMemoryForTest", | |
| 160 | "src/spawn/door.ts useWatchProbeByDefault", | |
| 158 | 161 | "src/spawn/child-apps.ts childRuntime", |
| 159 | 162 | "src/spawn/child_db.ts childDbReaders", |
| 160 | 163 | "src/spawn/clients.ts monthsSettled", |
src/actions/limits.ts +9 −1
| @@ -9,6 +9,7 @@ import type { App } from "../app.ts"; | ||
| 9 | 9 | import type { User } from "../core/users.ts"; |
| 10 | 10 | import { err, ok, type Result } from "../kernel/result.ts"; |
| 11 | 11 | import { keepFloor } from "../apps/cgroup.ts"; |
| 12 | import { birthCounts } from "../spawn/births.ts"; | |
| 12 | 13 | import { |
| 13 | 14 | LIMITS, |
| 14 | 15 | LIMIT_KEYS, |
| @@ -37,7 +38,7 @@ function admin(actor: User | undefined): Result<User, "not_found"> { | ||
| 37 | 38 | export function listLimits( |
| 38 | 39 | app: App, |
| 39 | 40 | actor: User | undefined, |
| 40 | ): Result<{ limits: LimitRow[]; changes: LimitChange[] }, "not_found"> { | |
| 41 | ): Result<{ limits: LimitRow[]; changes: LimitChange[]; births: { inFlight: number; waiting: number } }, "not_found"> { | |
| 41 | 42 | const who = admin(actor); |
| 42 | 43 | if (!who.ok) return who; |
| 43 | 44 | const states = limitStates(app.db, LIMIT_KEYS); |
| @@ -48,6 +49,8 @@ export function listLimits( | ||
| 48 | 49 | ...states.get(key)!, |
| 49 | 50 | })), |
| 50 | 51 | changes: limitChanges(app.db), |
| 52 | // What the births limits hold back now (`spawn/births.ts`), so strangers queued have a why. | |
| 53 | births: birthCounts(app), | |
| 51 | 54 | }); |
| 52 | 55 | } |
| 53 | 56 | |
| @@ -56,6 +59,11 @@ export type ChangeLimitError = | ||
| 56 | 59 | | { readonly kind: "no_such_limit" } |
| 57 | 60 | | { readonly kind: "out_of_bounds"; readonly message: string }; |
| 58 | 61 | |
| 62 | /** The births line in words, as both limits screens show it. */ | |
| 63 | export function birthsWords(b: { inFlight: number; waiting: number }): string { | |
| 64 | return `New forges now: ${b.inFlight} being made, ${b.waiting} waiting their turn.`; | |
| 65 | } | |
| 66 | ||
| 59 | 67 | /** Set `key` to `value`; `null` goes back to the default. */ |
| 60 | 68 | export function changeLimit( |
| 61 | 69 | app: App, |
src/http/api/forges.ts +2 −2
| @@ -12,7 +12,7 @@ import { usdWords } from "../../spawn/credit.ts"; | ||
| 12 | 12 | import { fundingOf, fundingWords, refreshFunding } from "../../spawn/funding.ts"; |
| 13 | 13 | import { notePhase, takeSessionTime, timePhase, timePhaseAsync } from "../../core/route_timing.ts"; |
| 14 | 14 | import { clientRows, listedKept } from "../../spawn/clients.ts"; |
| 15 | import { previewDoor, reissueDoor, spawnDoor } from "../../spawn/door.ts"; | |
| 15 | import { doorHeaders, previewDoor, reissueDoor, spawnDoor } from "../../spawn/door.ts"; | |
| 16 | 16 | import { checkMove } from "../../spawn/move_check.ts"; |
| 17 | 17 | import { moveToCell } from "../../spawn/move_cell.ts"; |
| 18 | 18 | import { deleteRecipe, listRecipes, saveRecipe } from "../../spawn/recipes.ts"; |
| @@ -102,7 +102,7 @@ export function forgeRoutes(router: Router<Handler>): void { | ||
| 102 | 102 | if (typeof input !== "object" || input === null || Array.isArray(input)) |
| 103 | 103 | return apiError("invalid", 422, "send a JSON object: name, apps, crew, dryRun"); |
| 104 | 104 | const door = await spawnDoor(ctx, ctx.auth.user, input as Record<string, unknown>); |
| 105 | return json(door.body, door.status); | |
| 105 | return json(door.body, door.status, doorHeaders(door)); | |
| 106 | 106 | }); |
| 107 | 107 | |
| 108 | 108 | // A person's saved recipes (`spawn/recipes.ts`): the forge configs they spawn again. |
src/settings/limits.ts +54
| @@ -49,6 +49,12 @@ export type LimitKey = | ||
| 49 | 49 | | "forges.per_person" |
| 50 | 50 | | "forges.wakes_per_person_hour" |
| 51 | 51 | | "forges.storage_mb" |
| 52 | | "spawn.anon_per_day" | |
| 53 | | "spawn.anon_global_per_day" | |
| 54 | | "spawn.births_at_once" | |
| 55 | | "spawn.birth_memory_mb" | |
| 56 | | "spawn.timers_max" | |
| 57 | | "forges.wakes_per_forge_hour" | |
| 52 | 58 | | "forge.memory_floor_mb" |
| 53 | 59 | | "guest.storage_mb" |
| 54 | 60 | | "guest.fork_storage_mb" |
| @@ -234,6 +240,54 @@ export const LIMITS: Record<LimitKey, LimitDef> = { | ||
| 234 | 240 | max: 1_000, |
| 235 | 241 | reason: "Every pull asks a person to read it, so a guest's pulls have a daily count.", |
| 236 | 242 | }, |
| 243 | "spawn.anon_per_day": { | |
| 244 | label: "Forges one stranger may start a day", | |
| 245 | unit: "forges", | |
| 246 | default: 100, | |
| 247 | min: 1, | |
| 248 | max: 100_000, | |
| 249 | reason: "One address is not one person, so this is only enough to stop a client looping; a signed-in person is not counted here.", | |
| 250 | }, | |
| 251 | "spawn.anon_global_per_day": { | |
| 252 | label: "Forges all strangers together may start a day", | |
| 253 | unit: "forges", | |
| 254 | default: 2_000, | |
| 255 | min: 1, | |
| 256 | max: 1_000_000, | |
| 257 | reason: "A day the link is shared must not fill the disk; what bounds the rush at any moment is the births at once, not this count.", | |
| 258 | }, | |
| 259 | "spawn.births_at_once": { | |
| 260 | label: "Forges being made at once", | |
| 261 | unit: "forges", | |
| 262 | default: 8, | |
| 263 | min: 1, | |
| 264 | max: 1_000, | |
| 265 | reason: "A new forge takes a core and over a gigabyte of memory until it answers and cannot be put to sleep meanwhile, so a surge waits its turn in a queue instead of starving the machine.", | |
| 266 | }, | |
| 267 | "spawn.birth_memory_mb": { | |
| 268 | label: "Memory a forge being made needs free", | |
| 269 | unit: "MiB", | |
| 270 | default: 1_536, | |
| 271 | min: 0, | |
| 272 | max: 1_048_576, | |
| 273 | reason: "A birth starts only while the machine has this much left above the memory the forge keeps for itself; 0 does not look.", | |
| 274 | }, | |
| 275 | "spawn.timers_max": { | |
| 276 | label: "New-forge timers at once", | |
| 277 | unit: "timers", | |
| 278 | default: 20, | |
| 279 | min: 1, | |
| 280 | max: 10_000, | |
| 281 | reason: "Each new forge is timed from the spawn until it answers; past this many, a forge is made but not timed, and the log says so.", | |
| 282 | }, | |
| 283 | "forges.wakes_per_forge_hour": { | |
| 284 | label: "Wakes of one forge an hour", | |
| 285 | unit: "wakes", | |
| 286 | default: 20, | |
| 287 | min: 1, | |
| 288 | max: 10_000, | |
| 289 | reason: "A stranger's visits can wake a sleeping forge again and again; one forge is woken this often at most. FORGE_CHILD_WAKES_PER_HOUR is read only while this is not set.", | |
| 290 | }, | |
| 237 | 291 | "forges.per_person": { |
| 238 | 292 | label: "Hosted forges one person may own", |
| 239 | 293 | unit: "forges", |
src/spawn/births.ts added +180
| @@ -0,0 +1,180 @@ | ||
| 1 | /** | |
| 2 | * One admission rule for births: a new forge takes a core and about 1.5 GiB | |
| 3 | * for the minutes until it answers, and is immune to sleep meanwhile, so what | |
| 4 | * bounds a surge is how many are being made at once, the memory left above | |
| 5 | * the floor the forge keeps (`forge.memory_floor_mb`) and the disk above its | |
| 6 | * floor (`spawn/disk_floor.ts`). A spawn that does not fit waits its turn in | |
| 7 | * a first-in, first-out queue for a short while, and past that is answered | |
| 8 | * with its place and about how long, never with a bare refusal. In process | |
| 9 | * memory: a restart empties it, and no birth is then in flight. | |
| 10 | */ | |
| 11 | ||
| 12 | import { freemem } from "node:os"; | |
| 13 | import { during } from "../perf/jobs.ts"; | |
| 14 | import type { App } from "../app.ts"; | |
| 15 | import { limitValue } from "../settings/limits.ts"; | |
| 16 | import { floorRefusal } from "./disk_floor.ts"; | |
| 17 | ||
| 18 | const MIB = 1024 * 1024; | |
| 19 | /** How long a spawn waits for its turn before it is answered with its place. */ | |
| 20 | const HOLD_MS = 15_000; | |
| 21 | let hold = HOLD_MS; | |
| 22 | /** Test seam: how long a stranger waits for a place before being told its number. */ | |
| 23 | export function provideHoldForTest(ms: number | undefined): void { | |
| 24 | hold = ms ?? HOLD_MS; | |
| 25 | } | |
| 26 | /** A birth is counted in flight for at most this long, answered or not. */ | |
| 27 | export const BIRTH_MAX_MS = 5 * 60_000; | |
| 28 | /** Spawns that may wait, as a multiple of the births at once; strangers may fill one such share. */ | |
| 29 | const QUEUE_FACTOR = 4; | |
| 30 | /** How often the head of the queue looks again, since memory frees without a release. */ | |
| 31 | const LOOK_MS = 1000; | |
| 32 | /** What a birth takes before a first measurement: four minutes. */ | |
| 33 | const FIRST_BIRTH_MS = 240_000; | |
| 34 | ||
| 35 | export type Why = "births" | "memory" | "disk"; | |
| 36 | ||
| 37 | export interface Waiting { | |
| 38 | /** 1 is next. */ | |
| 39 | readonly place: number; | |
| 40 | /** About how many seconds until this spawn would start. */ | |
| 41 | readonly aboutS: number; | |
| 42 | readonly why: Why; | |
| 43 | readonly message: string; | |
| 44 | } | |
| 45 | ||
| 46 | interface State { | |
| 47 | inFlight: number; | |
| 48 | /** The mean of the births measured, ms. */ | |
| 49 | meanMs: number; | |
| 50 | readonly queue: { go: () => void; anon: boolean }[]; | |
| 51 | timer: ReturnType<typeof setInterval> | undefined; | |
| 52 | } | |
| 53 | ||
| 54 | const states = new WeakMap<App, State>(); | |
| 55 | let memoryReader: () => number = freemem; | |
| 56 | ||
| 57 | /** Tests give the machine's free memory in bytes; `undefined` puts the machine's own back. */ | |
| 58 | export function provideMemoryForTest(r: (() => number) | undefined): void { | |
| 59 | memoryReader = r ?? freemem; | |
| 60 | } | |
| 61 | ||
| 62 | const stateOf = (app: App): State => { | |
| 63 | let s = states.get(app); | |
| 64 | if (!s) states.set(app, (s = { inFlight: 0, meanMs: FIRST_BIRTH_MS, queue: [], timer: undefined })); | |
| 65 | return s; | |
| 66 | } | |
| 67 | ||
| 68 | /** Births in flight and spawns waiting now, as the limits screens show them. */ | |
| 69 | export function birthCounts(app: App): { inFlight: number; waiting: number } { | |
| 70 | const s = stateOf(app); | |
| 71 | return { inFlight: s.inFlight, waiting: s.queue.length }; | |
| 72 | } | |
| 73 | ||
| 74 | /** Why one more birth does not fit now, or undefined when it does. */ | |
| 75 | export function whyNot(app: App): { why: Why; message: string } | undefined { | |
| 76 | const s = stateOf(app); | |
| 77 | const max = limitValue(app.db, "spawn.births_at_once"); | |
| 78 | if (s.inFlight >= max) return { why: "births", message: `${s.inFlight} forges are being made, the most at once` }; | |
| 79 | const perMiB = limitValue(app.db, "spawn.birth_memory_mb"); | |
| 80 | const floor = limitValue(app.db, "forge.memory_floor_mb"); | |
| 81 | // A birth takes its share over the minutes until it answers, so one reading of what is free does | |
| 82 | // not yet hold what the births in flight will take: each is set against it in full. | |
| 83 | const left = memoryReader() / MIB - floor - s.inFlight * perMiB; | |
| 84 | if (left < perMiB) return { why: "memory", message: `the machine has ${Math.max(0, Math.floor(left))} MiB left above its floor and a new forge needs ${perMiB} MiB` }; | |
| 85 | const disk = floorRefusal(app.config.dataDir); | |
| 86 | if (disk !== undefined) return { why: "disk", message: disk }; | |
| 87 | return undefined; | |
| 88 | } | |
| 89 | ||
| 90 | function drain(app: App, s: State): void { | |
| 91 | while (s.queue.length > 0 && whyNot(app) === undefined) { | |
| 92 | s.inFlight++; | |
| 93 | s.queue.shift()!.go(); | |
| 94 | } | |
| 95 | if (s.queue.length === 0 && s.timer !== undefined) { | |
| 96 | clearInterval(s.timer); | |
| 97 | s.timer = undefined; | |
| 98 | } | |
| 99 | } | |
| 100 | ||
| 101 | function look(app: App, s: State): void { | |
| 102 | if (s.timer !== undefined) return; | |
| 103 | s.timer = setInterval(() => void during("births-look", () => drain(app, s), undefined, app), LOOK_MS); | |
| 104 | s.timer.unref?.(); | |
| 105 | } | |
| 106 | ||
| 107 | function placeFor(app: App, s: State, place: number, why: Why, message: string): Waiting { | |
| 108 | const max = Math.max(1, limitValue(app.db, "spawn.births_at_once")); | |
| 109 | const rounds = Math.ceil(place / max); | |
| 110 | return { place, aboutS: Math.max(1, Math.round((rounds * s.meanMs) / 1000)), why, message }; | |
| 111 | } | |
| 112 | ||
| 113 | /** A place taken; `release` gives it back (once) and says how long the birth took. */ | |
| 114 | export type Place = { readonly ok: true; release: () => void } | { readonly ok: false; readonly waiting: Waiting }; | |
| 115 | ||
| 116 | function take(app: App, s: State): Place { | |
| 117 | const since = Date.now(); | |
| 118 | let released = false; | |
| 119 | let cap: ReturnType<typeof setTimeout> | undefined; | |
| 120 | const release = (): void => { | |
| 121 | if (released) return; | |
| 122 | released = true; | |
| 123 | if (cap !== undefined) clearTimeout(cap); | |
| 124 | s.inFlight = Math.max(0, s.inFlight - 1); | |
| 125 | const took = Date.now() - since; | |
| 126 | // Only a birth that ran its course teaches the mean; one given back at once does not. | |
| 127 | if (took > 1000) s.meanMs = Math.round(s.meanMs * 0.8 + Math.min(took, BIRTH_MAX_MS) * 0.2); | |
| 128 | drain(app, s); | |
| 129 | }; | |
| 130 | cap = setTimeout(() => void during("births-cap", release, undefined, app), BIRTH_MAX_MS); | |
| 131 | cap.unref?.(); | |
| 132 | return { ok: true, release }; | |
| 133 | } | |
| 134 | ||
| 135 | /** | |
| 136 | * Ask for a place to make a forge. Granted at once when the machine has room | |
| 137 | * and no one is ahead; else the spawn waits in line up to `holdMs`, and past | |
| 138 | * that is told its place and about how many seconds. | |
| 139 | */ | |
| 140 | export async function takePlace(app: App, opts: { holdMs?: number; anon?: boolean } = {}): Promise<Place> { | |
| 141 | const holdMs = opts.holdMs ?? hold; | |
| 142 | const anon = opts.anon === true; | |
| 143 | const s = stateOf(app); | |
| 144 | if (s.queue.length === 0 && whyNot(app) === undefined) { | |
| 145 | s.inFlight++; | |
| 146 | return take(app, s); | |
| 147 | } | |
| 148 | // The line is bounded, and a stranger's share of it is a quarter, so a loop of spawns holds at | |
| 149 | // most so many requests open and never starves a signed-in person. | |
| 150 | const max = Math.max(1, limitValue(app.db, "spawn.births_at_once")); | |
| 151 | const strangers = s.queue.filter((q) => q.anon).length; | |
| 152 | if (s.queue.length >= max * QUEUE_FACTOR || (anon && strangers >= max)) { | |
| 153 | const not = whyNot(app) ?? { why: "births" as Why, message: "forges are being made ahead of this one" }; | |
| 154 | return { ok: false, waiting: placeFor(app, s, s.queue.length + 1, not.why, not.message) }; | |
| 155 | } | |
| 156 | const turn = Promise.withResolvers<void>(); | |
| 157 | const entry = { go: () => turn.resolve(), anon }; | |
| 158 | // A signed-in person goes ahead of strangers, behind the signed-in already waiting. | |
| 159 | const firstStranger = s.queue.findIndex((q) => q.anon); | |
| 160 | if (anon || firstStranger < 0) s.queue.push(entry); | |
| 161 | else s.queue.splice(firstStranger, 0, entry); | |
| 162 | look(app, s); | |
| 163 | let timer: ReturnType<typeof setTimeout> | undefined; | |
| 164 | const timedOut = new Promise<"late">((r) => { | |
| 165 | timer = setTimeout(() => void during("births-hold", () => r("late"), undefined, app), holdMs); | |
| 166 | }); | |
| 167 | const got = await Promise.race([turn.promise.then(() => "turn" as const), timedOut]); | |
| 168 | if (timer !== undefined) clearTimeout(timer); | |
| 169 | if (got === "turn") return take(app, s); | |
| 170 | const at = s.queue.indexOf(entry); | |
| 171 | if (at < 0) return take(app, s); // granted as the wait ended | |
| 172 | const not = whyNot(app) ?? { why: "births" as Why, message: "forges are being made ahead of this one" }; | |
| 173 | const waiting = placeFor(app, s, at + 1, not.why, not.message); | |
| 174 | s.queue.splice(at, 1); | |
| 175 | if (s.queue.length === 0 && s.timer !== undefined) { | |
| 176 | clearInterval(s.timer); | |
| 177 | s.timer = undefined; | |
| 178 | } | |
| 179 | return { ok: false, waiting }; | |
| 180 | } |
src/spawn/door.ts +48 −4
| @@ -49,7 +49,8 @@ import { cellChildren, cellPlacement, ownsBody } from "./cell.ts"; | ||
| 49 | 49 | import { cellsAvailable } from "../apps/contain.ts"; |
| 50 | 50 | import { isSealed } from "./sealed.ts"; |
| 51 | 51 | import { isProber } from "../release/prober.ts"; |
| 52 | import { ANON_SPAWNS_PER_DAY, checkAnonSpawn, noteAnonSpawn } from "./rate.ts"; | |
| 52 | import { takePlace } from "./births.ts"; | |
| 53 | import { checkAnonSpawn, noteAnonSpawn } from "./rate.ts"; | |
| 53 | 54 | import { readdirSync, readFileSync } from "node:fs"; |
| 54 | 55 | import { readdir, stat } from "node:fs/promises"; |
| 55 | 56 | import { join } from "node:path"; |
| @@ -63,6 +64,12 @@ export interface Door { | ||
| 63 | 64 | readonly body: Record<string, unknown>; |
| 64 | 65 | } |
| 65 | 66 | |
| 67 | /** The `Retry-After` a queued answer carries, in the seconds its body names; none for any other. */ | |
| 68 | export function doorHeaders(door: Door): Record<string, string> { | |
| 69 | const after = door.body["retryAfter"]; | |
| 70 | return door.status === 503 && typeof after === "number" ? { "retry-after": String(after) } : {}; | |
| 71 | } | |
| 72 | ||
| 66 | 73 | const MAX_CREW = 100; |
| 67 | 74 | const placements = new WeakMap<App, Placement>(); |
| 68 | 75 | const probes = new WeakMap<App, (url: string) => Probe>(); |
| @@ -76,6 +83,16 @@ export function probeFor(app: App, url: string, timeoutMs: number): Probe { | ||
| 76 | 83 | return swapped ? swapped(url) : probeOf(url, timeoutMs); |
| 77 | 84 | } |
| 78 | 85 | |
| 86 | let watchProbe: (url: string) => Probe = probeOf; | |
| 87 | /** | |
| 88 | * Tests: how a new child is asked whether it is live, for every forge, unless `useProbe` gave one. | |
| 89 | * The suite's children answer nowhere, so each would keep its place among the births for the whole | |
| 90 | * cap; `test/helpers/preload.ts` has them answer at once. | |
| 91 | */ | |
| 92 | export function useWatchProbeByDefault(probe: ((url: string) => Probe) | undefined): void { | |
| 93 | watchProbe = probe ?? probeOf; | |
| 94 | } | |
| 95 | ||
| 79 | 96 | /** Tests swap how a new child is asked whether it is live. */ |
| 80 | 97 | export function useProbe(app: App, probe: (url: string) => Probe): void { |
| 81 | 98 | probes.set(app, probe); |
| @@ -445,10 +462,26 @@ export async function spawnDoor( | ||
| 445 | 462 | status: 429, |
| 446 | 463 | body: { |
| 447 | 464 | error: "rate_limited", |
| 448 | message: `anonymous spawns are limited to ${ANON_SPAWNS_PER_DAY} per day; try again after ${limit.resetsAt}`, | |
| 465 | message: `anonymous spawns are limited to ${limit.perDay} per day; try again after ${limit.resetsAt}`, | |
| 449 | 466 | }, |
| 450 | 467 | }; |
| 451 | 468 | } |
| 469 | // One rule for births (`spawn/births.ts`): the machine's own figures decide how many are made at | |
| 470 | // once, and a spawn that does not fit waits its turn, then is told its place and how long. | |
| 471 | const place = await takePlace(ctx.app, { anon }); | |
| 472 | if (!place.ok) | |
| 473 | return { | |
| 474 | status: 503, | |
| 475 | body: { | |
| 476 | error: "queued", | |
| 477 | // The machine's figures are for the log and the admin, not for a stranger. | |
| 478 | message: `a forge is being made for you soon: you are ${place.waiting.place} in line, about ${place.waiting.aboutS} s${anon ? "" : ` (${place.waiting.message})`}; try again in ${Math.min(30, place.waiting.aboutS)} s`, | |
| 479 | place: place.waiting.place, | |
| 480 | aboutS: place.waiting.aboutS, | |
| 481 | retryAfter: Math.min(30, place.waiting.aboutS), | |
| 482 | why: place.waiting.why, | |
| 483 | }, | |
| 484 | }; | |
| 452 | 485 | // The use is spent just before the forge is made, in one statement; it goes back if no forge is |
| 453 | 486 | // made. |
| 454 | 487 | let gift: Code | undefined; |
| @@ -461,6 +494,8 @@ export async function spawnDoor( | ||
| 461 | 494 | // The use is given back unless the forge was made and the redemption recorded: a throw settles it |
| 462 | 495 | // too. |
| 463 | 496 | let settled = false; |
| 497 | // A forge that was made keeps its place until it answers; any other gives it back at once. | |
| 498 | let born = false; | |
| 464 | 499 | let made: Awaited<ReturnType<typeof spawnForge>>; |
| 465 | 500 | try { |
| 466 | 501 | made = await spawnForge(ctx.app, user, input, source, { |
| @@ -471,12 +506,14 @@ export async function spawnDoor( | ||
| 471 | 506 | ...(typeof modelKey === "string" ? { modelKey } : {}), |
| 472 | 507 | ...(modelEndpoint === undefined ? {} : { modelEndpoint }), |
| 473 | 508 | }); |
| 509 | born = made.ok && !made.value.existed; | |
| 474 | 510 | if (made.ok && gift !== undefined && !made.value.existed) { |
| 475 | 511 | settleGift(ctx.app, gift, made.value.name); |
| 476 | 512 | settled = true; |
| 477 | 513 | } |
| 478 | 514 | } finally { |
| 479 | 515 | if (gift !== undefined && !settled) unclaimGift(ctx.app, gift); |
| 516 | if (!born) place.release(); | |
| 480 | 517 | } |
| 481 | 518 | if (!made.ok) return spawnWords(made.error); |
| 482 | 519 | if (anon && counted && !made.value.existed) noteAnonSpawn(ctx.app.db, ctx.request); |
| @@ -485,8 +522,15 @@ export async function spawnDoor( | ||
| 485 | 522 | // Its apps come up now, so the first visit after a claim finds them awake. |
| 486 | 523 | let up: Promise<void> | undefined; |
| 487 | 524 | if (!made.value.existed) { |
| 488 | up = bootHostedChild(ctx.app, made.value.name); | |
| 489 | startWatch(ctx.app, { name: made.value.name, startedAt, ...templateOfInput(raw["template"]), probe: (probes.get(ctx.app) ?? probeOf)(made.value.url) }); | |
| 525 | // The place is held until the watch settles; a watch that never starts, or a throw before it, | |
| 526 | // gives it back here. | |
| 527 | let watching = false; | |
| 528 | try { | |
| 529 | up = bootHostedChild(ctx.app, made.value.name); | |
| 530 | watching = startWatch(ctx.app, { name: made.value.name, startedAt, ...templateOfInput(raw["template"]), probe: (probes.get(ctx.app) ?? watchProbe)(made.value.url), onSettled: place.release }); | |
| 531 | } finally { | |
| 532 | if (!watching) place.release(); | |
| 533 | } | |
| 490 | 534 | } |
| 491 | 535 | // Proven from outside before it is called done: the child's home and each |
| 492 | 536 | // app's public address, each with what came back. What does not answer yet |
src/spawn/host.ts +1 −1
| @@ -676,7 +676,7 @@ export async function wakeGate(parent: App, request: Request): Promise<Response | ||
| 676 | 676 | const wait = own ? 0 : people.waitS(budget.key, max, at); |
| 677 | 677 | if (wait > 0) |
| 678 | 678 | return wakingResponse(request, name, budgetRefusal({ name, budget, max, waitS: wait, limitsUrl: `${parentOrigin(parent.config)}/admin/limits` }), wait); |
| 679 | const why = (await roomFor(parent)) ?? (known ? undefined : rates.refusal(ledger.bodyId, at)); | |
| 679 | const why = (await roomFor(parent)) ?? (known ? undefined : rates.refusal(ledger.bodyId, at, process.env, parent.db)); | |
| 680 | 680 | if (why !== undefined) { |
| 681 | 681 | const stop = lastStop(parent, ledger.bodyId); |
| 682 | 682 | return wakingResponse(request, name, stop === undefined ? why : stopSentence(stop)); |
src/spawn/live.ts +18 −5
| @@ -8,13 +8,14 @@ | ||
| 8 | 8 | * with no `live` or `failed`, and the boards show that gap as unfinished. |
| 9 | 9 | */ |
| 10 | 10 | import type { App } from "../app.ts"; |
| 11 | import { limitValue } from "../settings/limits.ts"; | |
| 11 | 12 | |
| 12 | 13 | export const POLL_MS = 1000; |
| 13 | 14 | /** After this long without an answer, poll every `SLOW_POLL_MS` instead. */ |
| 14 | 15 | export const SLOW_AFTER_MS = 30_000; |
| 15 | 16 | export const SLOW_POLL_MS = 5000; |
| 16 | /** At most this many watchers at once per forge; a spawn past it is made but not timed. */ | |
| 17 | export const MAX_WATCHERS = 20; | |
| 17 | // At most `spawn.timers_max` watchers at once per forge (/admin/limits); a spawn past it is made | |
| 18 | // but not timed, and the log says so. | |
| 18 | 19 | const open = new WeakMap<App, number>(); |
| 19 | 20 | /** |
| 20 | 21 | * Give up after this long: well past the 3-minute budget, so a slow forge still counts as live, |
| @@ -54,14 +55,26 @@ export interface Watch { | ||
| 54 | 55 | readonly pollMs?: number; |
| 55 | 56 | readonly giveUpMs?: number; |
| 56 | 57 | readonly now?: () => number; |
| 58 | /** Called once when the watch ends, whatever the end. */ | |
| 59 | readonly onSettled?: () => void; | |
| 57 | 60 | } |
| 58 | 61 | |
| 59 | /** Start a watcher unless this forge already runs `MAX_WATCHERS`; true when started. */ | |
| 62 | /** | |
| 63 | * Start a watcher unless this forge already runs `spawn.timers_max`; true when started, else | |
| 64 | * logged. | |
| 65 | */ | |
| 60 | 66 | export function startWatch(app: App, w: Watch): boolean { |
| 61 | 67 | const n = open.get(app) ?? 0; |
| 62 | if (n >= MAX_WATCHERS) return false; | |
| 68 | const max = limitValue(app.db, "spawn.timers_max"); | |
| 69 | if (n >= max) { | |
| 70 | app.log.warn("a new forge is made but not timed: every new-forge timer is taken", { name: w.name, timers: n, max }); | |
| 71 | return false; | |
| 72 | } | |
| 63 | 73 | open.set(app, n + 1); |
| 64 | void watchLive(app, w).finally(() => open.set(app, (open.get(app) ?? 1) - 1)); | |
| 74 | void watchLive(app, w).finally(() => { | |
| 75 | open.set(app, (open.get(app) ?? 1) - 1); | |
| 76 | w.onSettled?.(); | |
| 77 | }); | |
| 65 | 78 | return true; |
| 66 | 79 | } |
| 67 | 80 |
src/spawn/rate.ts +9 −12
| @@ -8,16 +8,11 @@ | ||
| 8 | 8 | |
| 9 | 9 | import type { Database } from "bun:sqlite"; |
| 10 | 10 | import { clientBucket } from "../http/peer.ts"; |
| 11 | import { limitValue } from "../settings/limits.ts"; | |
| 11 | 12 | |
| 12 | /** Anonymous spawns allowed per client in any 24 hours. */ | |
| 13 | // 3 turned away the third person behind one office address, and the release proof's own | |
| 14 | // stranger after three releases in a day (0.7.576 read "unproven" for it): an address is | |
| 15 | // not a person. Enough to stop one client looping, no more. | |
| 16 | export const ANON_SPAWNS_PER_DAY = 100; | |
| 17 | /** Anonymous spawns allowed from everyone together in any 24 hours. */ | |
| 18 | // 30 turned strangers away on any day the link was shared; what bounds a surge is the machine | |
| 19 | // (forges awake at once, and the sleep that follows), not a daily count. | |
| 20 | export const ANON_SPAWNS_GLOBAL_PER_DAY = 2_000; | |
| 13 | // The two counts are settings (`spawn.anon_per_day`, `spawn.anon_global_per_day`, /admin/limits), | |
| 14 | // read on every check. A day's count is no bound on a surge: that is `spawn/births.ts`, which holds | |
| 15 | // how many forges are being made at once. | |
| 21 | 16 | const DAY_MS = 86_400_000; |
| 22 | 17 | |
| 23 | 18 | export interface RateLimit { |
| @@ -25,6 +20,8 @@ export interface RateLimit { | ||
| 25 | 20 | readonly allowed: boolean; |
| 26 | 21 | /** When the oldest counted spawn leaves the window, ISO. */ |
| 27 | 22 | readonly resetsAt: string; |
| 23 | /** The per-client count, for the sentence that refuses. */ | |
| 24 | readonly perDay: number; | |
| 28 | 25 | } |
| 29 | 26 | |
| 30 | 27 | /** Whether this request's client may anonymously spawn now. */ |
| @@ -38,10 +35,10 @@ export function checkAnonSpawn(db: Database, request: Request, now = Date.now()) | ||
| 38 | 35 | n: number; |
| 39 | 36 | oldest: number | null; |
| 40 | 37 | }; |
| 41 | const clientFull = mine.n >= ANON_SPAWNS_PER_DAY; | |
| 42 | const globalFull = all.n >= ANON_SPAWNS_GLOBAL_PER_DAY; | |
| 38 | const clientFull = mine.n >= limitValue(db, "spawn.anon_per_day"); | |
| 39 | const globalFull = all.n >= limitValue(db, "spawn.anon_global_per_day"); | |
| 43 | 40 | const oldest = (clientFull ? mine.oldest : globalFull ? all.oldest : null) ?? now; |
| 44 | return { allowed: !clientFull && !globalFull, resetsAt: new Date(oldest + DAY_MS).toISOString() }; | |
| 41 | return { allowed: !clientFull && !globalFull, resetsAt: new Date(oldest + DAY_MS).toISOString(), perDay: limitValue(db, "spawn.anon_per_day") }; | |
| 45 | 42 | } |
| 46 | 43 | |
| 47 | 44 | /** Count an anonymous spawn against this request's client. */ |
src/spawn/sleep.ts +10 −4
| @@ -15,7 +15,7 @@ import type { Database } from "bun:sqlite"; | ||
| 15 | 15 | import { readdirSync, readFileSync, statSync } from "node:fs"; |
| 16 | 16 | import { join } from "node:path"; |
| 17 | 17 | import { maxAwake } from "../core/capacity.ts"; |
| 18 | import { LIMITS, limitValue } from "../settings/limits.ts"; | |
| 18 | import { LIMITS, limitIsSet, limitValue, type LimitKey } from "../settings/limits.ts"; | |
| 19 | 19 | |
| 20 | 20 | type Env = Record<string, string | undefined>; |
| 21 | 21 | |
| @@ -26,6 +26,11 @@ function whole(env: Env, name: string, fallback: number): number { | ||
| 26 | 26 | return n > 0 ? n : fallback; |
| 27 | 27 | } |
| 28 | 28 | |
| 29 | /** An admin's setting when one is set, else the environment's (or its default). */ | |
| 30 | function settled(db: Database | undefined, key: LimitKey, env: Env, name: string, fallback: number): number { | |
| 31 | return db !== undefined && limitIsSet(db, key) ? limitValue(db, key) : whole(env, name, fallback); | |
| 32 | } | |
| 33 | ||
| 29 | 34 | /** A forge quiet for this long sleeps: FORGE_CHILD_IDLE_S, ten minutes by default. */ |
| 30 | 35 | export const idleAfterMs = (env: Env = process.env): number => whole(env, "FORGE_CHILD_IDLE_S", 600) * 1000; |
| 31 | 36 | /** |
| @@ -34,7 +39,8 @@ export const idleAfterMs = (env: Env = process.env): number => whole(env, "FORGE | ||
| 34 | 39 | */ |
| 35 | 40 | export const awakeCeiling = (env: Env = process.env): number => whole(env, "FORGE_CHILD_AWAKE_MAX", 8); |
| 36 | 41 | /** Most wakes of one forge in an hour: FORGE_CHILD_WAKES_PER_HOUR. */ |
| 37 | export const wakesPerHour = (env: Env = process.env): number => whole(env, "FORGE_CHILD_WAKES_PER_HOUR", 20); | |
| 42 | export const wakesPerHour = (env: Env = process.env, db?: Database): number => | |
| 43 | settled(db, "forges.wakes_per_forge_hour", env, "FORGE_CHILD_WAKES_PER_HOUR", 20); | |
| 38 | 44 | |
| 39 | 45 | const HOUR_MS = 3_600_000; |
| 40 | 46 | |
| @@ -47,8 +53,8 @@ export class WakeRate { | ||
| 47 | 53 | return kept; |
| 48 | 54 | } |
| 49 | 55 | /** Why this forge may not be woken now, or undefined. */ |
| 50 | refusal(bodyId: string, now = Date.now(), env: Env = process.env): string | undefined { | |
| 51 | const max = wakesPerHour(env); | |
| 56 | refusal(bodyId: string, now = Date.now(), env: Env = process.env, db?: Database): string | undefined { | |
| 57 | const max = wakesPerHour(env, db); | |
| 52 | 58 | return this.recent(bodyId, now).length >= max |
| 53 | 59 | ? `this forge has been woken ${max} times in the last hour, the most it may be; try again in a little while` |
| 54 | 60 | : undefined; |
src/tui/screens/limits.ts +4 −2
| @@ -7,7 +7,7 @@ | ||
| 7 | 7 | * an admin who is not an agent; anyone else sees nothing and changes nothing. |
| 8 | 8 | */ |
| 9 | 9 | |
| 10 | import { changeLimit, listLimits, type LimitRow } from "../../actions/limits.ts"; | |
| 10 | import { birthsWords, changeLimit, listLimits, type LimitRow } from "../../actions/limits.ts"; | |
| 11 | 11 | import { limitBounds, validLimit } from "../../settings/limits.ts"; |
| 12 | 12 | import type { Ctx, Screen } from "../app/app.ts"; |
| 13 | 13 | import { binding } from "../app/keys.ts"; |
| @@ -77,7 +77,7 @@ export function limitsScreen(f: Session, ctx: Ctx): Screen { | ||
| 77 | 77 | keys: [binding(["j", "k"], "move cursor"), binding(["e"], "change")], |
| 78 | 78 | render(area, s) { |
| 79 | 79 | const c = ctx.theme.c; |
| 80 | const [head, list] = split(area, "v", [len(1), fill(1)]); | |
| 80 | const [head, births, list] = split(area, "v", [len(1), len(1), fill(1)]); | |
| 81 | 81 | const r = read(); |
| 82 | 82 | const text = |
| 83 | 83 | note !== "" |
| @@ -86,6 +86,8 @@ export function limitsScreen(f: Session, ctx: Ctx): Screen { | ||
| 86 | 86 | ? "What the forge's owner can hit. e changes one." |
| 87 | 87 | : "Limits are set by a forge admin."; |
| 88 | 88 | s.text(head!.x, head!.y, truncate(text, head!.w), style(c.mutedFg, c.bg), head!.x + head!.w); |
| 89 | // What the births limits hold back now, so strangers queued have a why. | |
| 90 | if (r.ok) s.text(births!.x, births!.y, truncate(birthsWords(r.value.births), births!.w), style(c.mutedFg, c.bg), births!.x + births!.w); | |
| 89 | 91 | drawList(s, list!, c, state, rows(r.ok ? r.value.limits : [])); |
| 90 | 92 | }, |
| 91 | 93 | key(e) { |
src/tui/screens/startforge.ts +2 −2
| @@ -10,7 +10,7 @@ | ||
| 10 | 10 | |
| 11 | 11 | import { notePeer } from "../../http/peer.ts"; |
| 12 | 12 | import { spawnDoor } from "../../spawn/door.ts"; |
| 13 | import { START_REFUSALS, startRefusalOf } from "../../web/start_refusal.ts"; | |
| 13 | import { START_REFUSALS, startRefusalOf, startRefusalWords, startWaitOf } from "../../web/start_refusal.ts"; | |
| 14 | 14 | import type { Ctx, Screen } from "../app/app.ts"; |
| 15 | 15 | import { binding } from "../app/keys.ts"; |
| 16 | 16 | import { style } from "../term/style.ts"; |
| @@ -39,7 +39,7 @@ export function startForgeScreen(f: Session, ctx: Ctx): Screen { | ||
| 39 | 39 | const made = await spawnDoor(hctx, undefined, { name: name.trim().toLowerCase(), dryRun: false }); |
| 40 | 40 | const link = made.body["claimUrl"]; |
| 41 | 41 | if (made.status >= 400 || typeof link !== "string") { |
| 42 | error = START_REFUSALS[made.status >= 400 ? startRefusalOf(made) : "taken"]; | |
| 42 | error = made.status >= 400 ? startRefusalWords(startRefusalOf(made), startWaitOf(made)) : START_REFUSALS.taken; | |
| 43 | 43 | return; |
| 44 | 44 | } |
| 45 | 45 | ctx.toast(`${String(made.body["name"] ?? name)} is ready`, "success"); |
src/web/pages/admin-limits.ts +3 −2
| @@ -9,7 +9,7 @@ import { sshOn } from "../../settings/ssh.ts"; | ||
| 9 | 9 | import { usdWords } from "../../spawn/credit.ts"; |
| 10 | 10 | import { worstCaseOf } from "../../spawn/included_credit.ts"; |
| 11 | 11 | import { QUOTA_INVALID } from "../../actions/admin_limits_words.ts"; |
| 12 | import { changeLimit, listLimits } from "../../actions/limits.ts"; | |
| 12 | import { birthsWords, changeLimit, listLimits } from "../../actions/limits.ts"; | |
| 13 | 13 | import { changeQuotas, readQuotas } from "../../actions/quotas.ts"; |
| 14 | 14 | import { inviteLink, list as listInviteLinks, mint as mintInviteLink, revoke as revokeInviteLink } from "../../actions/invites.ts"; |
| 15 | 15 | import { INVITE_NOT_OPEN } from "../../actions/sever_invite_words.ts"; |
| @@ -40,7 +40,7 @@ export function limitsPage(ctx: Ctx): Response { | ||
| 40 | 40 | const billing = readBillingLinks(ctx.app, ctx.auth.user); |
| 41 | 41 | const invites = timePhase(ctx.request, "invites", () => listInviteLinks(ctx.app, ctx.auth.user)); |
| 42 | 42 | const rendering = performance.now(); |
| 43 | const { limits, changes } = listed.value; | |
| 43 | const { limits, changes, births } = listed.value; | |
| 44 | 44 | const rows = limits.map( |
| 45 | 45 | (l) => html`<tr> |
| 46 | 46 | <td> |
| @@ -220,6 +220,7 @@ export function limitsPage(ctx: Ctx): Response { | ||
| 220 | 220 | ${historyBlock("Invite links made", inviteRows)} |
| 221 | 221 | <section class="settings-block mt"> |
| 222 | 222 | <h2>Limits</h2> |
| 223 | <p class="small muted mb" id="births-now">${birthsWords(births)}</p> | |
| 223 | 224 | <p class="small muted mb" id="included-credit-worst">Worst case a day for the crew credit new forges come with: ${usdWords(worstCaseOf((k) => limits.find((l) => l.key === k)?.value ?? 0))} (new accounts an hour × 24 × forges a person may own × the credit).</p> |
| 224 | 225 | <div class="table-scroll"> |
| 225 | 226 | <table class="table setting-table" data-dense> |
src/web/pages/new-forge.ts +7 −3
| @@ -18,7 +18,7 @@ import { notePhase, takeSessionTime, timePhase } from "../../core/route_timing.t | ||
| 18 | 18 | import { giftOf, REDEEM_WORDS } from "../../actions/codes.ts"; |
| 19 | 19 | import { limitValue } from "../../settings/limits.ts"; |
| 20 | 20 | import { quotaFull } from "../../spawn/quota.ts"; |
| 21 | import { childHost, childUrlOf, noRoom, spawnDoor, type Door } from "../../spawn/door.ts"; | |
| 21 | import { childHost, childUrlOf, doorHeaders, noRoom, spawnDoor, type Door } from "../../spawn/door.ts"; | |
| 22 | 22 | import { listRecipes, saveRecipe, type ForgeConfig, type Recipe } from "../../spawn/recipes.ts"; |
| 23 | 23 | import { |
| 24 | 24 | APP_MODES, |
| @@ -716,8 +716,8 @@ export async function newForgeSubmit(ctx: Ctx): Promise<Response> { | ||
| 716 | 716 | { ctx, title: "New forge", body: formView(ctx, user, { ...values, template: DEFAULT_TEMPLATE }, { form: String(door.body["message"]) }, templateRows(templateList(ctx.app.db, user?.kind === "user" ? user : undefined)), readsFor(ctx, user)) }, |
| 717 | 717 | 404, |
| 718 | 718 | ); |
| 719 | if (door.status >= 400) | |
| 720 | return page( | |
| 719 | if (door.status >= 400) { | |
| 720 | const refused = page( | |
| 721 | 721 | { |
| 722 | 722 | ctx, |
| 723 | 723 | title: "New forge", |
| @@ -726,6 +726,10 @@ export async function newForgeSubmit(ctx: Ctx): Promise<Response> { | ||
| 726 | 726 | }, |
| 727 | 727 | door.status === 422 ? 422 : door.status, |
| 728 | 728 | ); |
| 729 | // A queued spawn says its place and wait in the form's words, and when to ask in the header. | |
| 730 | for (const [k, v] of Object.entries(doorHeaders(door))) refused.headers.set(k, v); | |
| 731 | return refused; | |
| 732 | } | |
| 729 | 733 | if (planOnly) |
| 730 | 734 | return page({ |
| 731 | 735 | ctx, |
src/web/pages/start-forge.ts +4 −3
| @@ -16,7 +16,7 @@ import { agentsPage } from "./agents.ts"; | ||
| 16 | 16 | import { madeBody, madeJs } from "../forge_made.ts"; |
| 17 | 17 | import { page, redirect } from "../layout.ts"; |
| 18 | 18 | import { notFoundPage } from "../chrome.ts"; |
| 19 | import { START_REFUSALS, startRefusalHref, startRefusalOf } from "../start_refusal.ts"; | |
| 19 | import { startRefusalHref, startRefusalOf, startRefusalWords, startWaitOf } from "../start_refusal.ts"; | |
| 20 | 20 | |
| 21 | 21 | /** The home's live name check: asks `GET /forges/name/<name>` and says free, taken, or the route's own reason; a 429 or a failure says nothing. */ |
| 22 | 22 | export const startNameJs = `(function(d){ |
| @@ -47,10 +47,11 @@ export async function startForgeSubmit(ctx: Ctx): Promise<Response> { | ||
| 47 | 47 | const door = await spawnDoor(ctx, undefined, { name, dryRun: false }, { page: true }); |
| 48 | 48 | if (door.status >= 400) { |
| 49 | 49 | const why = startRefusalOf(door); |
| 50 | const wait = startWaitOf(door); | |
| 50 | 51 | // The /agents form is answered on /agents, the name kept and the reason beside it. |
| 51 | if (field(form, "from") === "agents") return agentsPage(ctx, refused(form, { name: START_REFUSALS[why] })); | |
| 52 | if (field(form, "from") === "agents") return agentsPage(ctx, refused(form, { name: startRefusalWords(why, wait) })); | |
| 52 | 53 | // Said on the home page, with the name kept (#11238). |
| 53 | return redirect(startRefusalHref(why, typed)); | |
| 54 | return redirect(startRefusalHref(why, typed, wait)); | |
| 54 | 55 | } |
| 55 | 56 | const link = door.body["claimUrl"]; |
| 56 | 57 | if (typeof link !== "string") return redirect(startRefusalHref("taken", name)); |
src/web/start_refusal.ts +43 −4
| @@ -6,12 +6,19 @@ | ||
| 6 | 6 | * except the name, escaped, back in its field. |
| 7 | 7 | */ |
| 8 | 8 | |
| 9 | export type StartRefusal = "rate" | "taken" | "name" | "full" | "other"; | |
| 9 | export type StartRefusal = "rate" | "taken" | "name" | "full" | "queued" | "other"; | |
| 10 | ||
| 11 | /** A queued spawn's place in line and the seconds to wait before pressing again. */ | |
| 12 | export interface StartWait { | |
| 13 | readonly place: number; | |
| 14 | readonly afterS: number; | |
| 15 | } | |
| 10 | 16 | |
| 11 | 17 | export const START_REFUSALS: Readonly<Record<StartRefusal, string>> = { |
| 12 | 18 | rate: "Too many forges have been started from your connection lately. Wait a few minutes, or try again tomorrow, and press Start my forge again.", |
| 13 | 19 | taken: "That name is already taken. Change it a little, for example add your own name or a number, and press Start my forge again.", |
| 14 | 20 | name: "That name can't be used. Use 1 to 39 lowercase letters, digits or dashes, not starting or ending with a dash, and press Start my forge again.", |
| 21 | queued: "Many forges are being made right now. Wait a minute, then press Start my forge again.", | |
| 15 | 22 | full: "This forge has no room for another forge right now. Try again later, or come back tomorrow.", |
| 16 | 23 | other: "We couldn't start your forge this time. Press Start my forge to try again, or come back in a few minutes.", |
| 17 | 24 | }; |
| @@ -21,6 +28,7 @@ export function startRefusalOf(door: { readonly status: number; readonly body: R | ||
| 21 | 28 | const error = door.body["error"]; |
| 22 | 29 | const message = String(door.body["message"] ?? ""); |
| 23 | 30 | if (door.status === 429 || error === "rate_limited") return "rate"; |
| 31 | if (error === "queued") return "queued"; | |
| 24 | 32 | if (error === "exists" || (error === "refused" && /already exists|being made/.test(message))) return "taken"; |
| 25 | 33 | if (error === "invalid" && /^name: /.test(message)) return "name"; |
| 26 | 34 | if (error === "refused" && /forges sent|room|quota|deep|capacity|limit/i.test(message)) return "full"; |
| @@ -32,7 +40,38 @@ export function startRefusalIn(value: string | null): StartRefusal | undefined { | ||
| 32 | 40 | return value !== null && Object.hasOwn(START_REFUSALS, value) ? (value as StartRefusal) : undefined; |
| 33 | 41 | } |
| 34 | 42 | |
| 35 | /** Where the home page is told of a refusal: the kind and the name, kept. */ | |
| 36 | export function startRefusalHref(kind: StartRefusal, name: string): string { | |
| 37 | return `/?start=${kind}&name=${encodeURIComponent(name.slice(0, 39))}#start-forge`; | |
| 43 | /** A whole number from `low` to `high`, or undefined: only digits are read. */ | |
| 44 | function count(raw: unknown, low: number, high: number): number | undefined { | |
| 45 | const n = typeof raw === "number" ? raw : typeof raw === "string" && /^\d{1,6}$/.test(raw) ? Number(raw) : NaN; | |
| 46 | return Number.isInteger(n) && n >= low && n <= high ? n : undefined; | |
| 47 | } | |
| 48 | ||
| 49 | const wait = (place: unknown, afterS: unknown): StartWait | undefined => { | |
| 50 | const p = count(place, 1, 10_000); | |
| 51 | const a = count(afterS, 1, 3_600); | |
| 52 | return p === undefined || a === undefined ? undefined : { place: p, afterS: a }; | |
| 53 | }; | |
| 54 | ||
| 55 | /** The place and wait a queued door answer names (`spawn/door.ts`), or undefined. */ | |
| 56 | export function startWaitOf(door: { readonly body: Record<string, unknown> }): StartWait | undefined { | |
| 57 | return door.body["error"] === "queued" ? wait(door.body["place"], door.body["retryAfter"]) : undefined; | |
| 58 | } | |
| 59 | ||
| 60 | /** `?place=<n>&after=<s>` back to a wait, or undefined unless both are whole numbers in bounds. */ | |
| 61 | export function startWaitIn(params: URLSearchParams): StartWait | undefined { | |
| 62 | return wait(params.get("place"), params.get("after")); | |
| 63 | } | |
| 64 | ||
| 65 | /** The sentence for a refusal; a queued one with its wait names the place and the seconds. */ | |
| 66 | export function startRefusalWords(kind: StartRefusal, w?: StartWait): string { | |
| 67 | if (kind !== "queued" || w === undefined) return START_REFUSALS[kind]; | |
| 68 | return `Many forges are being made right now, and yours is number ${w.place} in line. Try again in ${w.afterS} s: press Start my forge again.`; | |
| 69 | } | |
| 70 | ||
| 71 | /** | |
| 72 | * Where the home page is told of a refusal: the kind and the name, kept, and a queued one's wait. | |
| 73 | */ | |
| 74 | export function startRefusalHref(kind: StartRefusal, name: string, w?: StartWait): string { | |
| 75 | const waitQuery = kind === "queued" && w !== undefined ? `&place=${w.place}&after=${w.afterS}` : ""; | |
| 76 | return `/?start=${kind}&name=${encodeURIComponent(name.slice(0, 39))}${waitQuery}#start-forge`; | |
| 38 | 77 | } |
src/web/studio/home.ts +2 −2
| @@ -39,7 +39,7 @@ import { field, readForm, type Form } from "../form.ts"; | ||
| 39 | 39 | import { liveRegionsJs } from "../client.ts"; |
| 40 | 40 | import { html, when, type Html } from "../html.ts"; |
| 41 | 41 | import { fragment, page, redirect } from "../layout.ts"; |
| 42 | import { START_REFUSALS, startRefusalIn } from "../start_refusal.ts"; | |
| 42 | import { startRefusalIn, startRefusalWords, startWaitIn } from "../start_refusal.ts"; | |
| 43 | 43 | import { SITE_WHAT } from "../site_line.ts"; |
| 44 | 44 | import { siteStructuredData } from "../structured_data.ts"; |
| 45 | 45 | import { running } from "../../kernel/version.ts"; |
| @@ -741,7 +741,7 @@ export function studioDoor(ctx: Ctx): Response { | ||
| 741 | 741 | <div class="door-lead"> |
| 742 | 742 | <p class="door-support">Say what you need and an AI crew builds it in your own forge, free to try.</p> |
| 743 | 743 | <form method="post" action="/forges/start" class="door-start" id="start-forge"> |
| 744 | ${refusal === undefined ? null : html`<p class="error" id="start-refused">${START_REFUSALS[refusal]}</p>`} | |
| 744 | ${refusal === undefined ? null : html`<p class="error" id="start-refused">${startRefusalWords(refusal, startWaitIn(ctx.url.searchParams))}</p>`} | |
| 745 | 745 | <div class="field"> |
| 746 | 746 | <label for="start-name">Forge name</label> |
| 747 | 747 | <input class="input" id="start-name" name="name" type="text" value="${keptName}" aria-describedby="${refusal === undefined ? "start-name-says" : "start-refused start-name-says"}" ${refusal === undefined ? null : html`aria-invalid="true"`} required autocomplete="off" autocapitalize="none" spellcheck="false" /> |
test/brief-hot.test.ts +2 −1
| @@ -1,4 +1,4 @@ | ||
| 1 | import { beforeEach, describe, expect, test } from "bun:test"; | |
| 1 | import { afterAll, beforeEach, describe, expect, test } from "bun:test"; | |
| 2 | 2 | import { taskPrompt, type TaskInput } from "../src/crew/brief.ts"; |
| 3 | 3 | import { rememberPullFiles } from "../src/crew/hot_files.ts"; |
| 4 | 4 | import { HOT_PULLS } from "../src/flow/hot_trend.ts"; |
| @@ -21,6 +21,7 @@ const pulls = (files: string[], count: number) => | ||
| 21 | 21 | |
| 22 | 22 | describe("a planner's brief names the hot files", () => { |
| 23 | 23 | beforeEach(() => rememberPullFiles("repo-hot", [])); |
| 24 | afterAll(() => rememberPullFiles("repo-hot", [])); | |
| 24 | 25 | |
| 25 | 26 | test("hottest first, a builder's brief without the line", () => { |
| 26 | 27 | rememberPullFiles("repo-hot", [ |
test/forge-looks.test.ts +1 −1
| @@ -192,7 +192,7 @@ test("`/` signed out is its own look at phone and desktop, taken with no session | ||
| 192 | 192 | test("a look below the top level is the preview seed's, taken as the owner", () => { |
| 193 | 193 | const { looks, looksAsOwner } = manifest(); |
| 194 | 194 | for (const p of looks.filter((p) => !isTopLevel(p))) { |
| 195 | expect(p.startsWith("/preview-admin/sample") || p === "/settings/crew" || p === "/settings/security/controls" || p === "/settings/grants" || p === "/looks/missing" || p === "/looks/failed" || p === "/nope/nothing" || isHelpLook(p)).toBe(true); | |
| 195 | expect(p.startsWith("/preview-admin/sample") || p === "/settings/crew" || p === "/settings/security/controls" || p === "/settings/grants" || p === "/admin/limits" || p === "/looks/missing" || p === "/looks/failed" || p === "/nope/nothing" || isHelpLook(p)).toBe(true); | |
| 196 | 196 | // the not-found and failed pages are the same page to everyone: no owner needed |
| 197 | 197 | if (!p.startsWith("/looks/") && p !== "/nope/nothing") expect(looksAsOwner).toContain(p); |
| 198 | 198 | } |
test/helpers/preload.ts +8
| @@ -50,3 +50,11 @@ if (process.env.FORGE_SQL_LOG) { | ||
| 50 | 50 | }; |
| 51 | 51 | } |
| 52 | 52 | } |
| 53 | ||
| 54 | // A suite's machine has room for a birth whatever the one running it has free (`spawn/births.ts`). | |
| 55 | const { provideMemoryForTest } = await import("../../src/spawn/births.ts"); | |
| 56 | provideMemoryForTest(() => 64 * 1024 ** 3); | |
| 57 | ||
| 58 | // A new child of the suite answers its first look, so its place among the births is given back. | |
| 59 | const { useWatchProbeByDefault } = await import("../../src/spawn/door.ts"); | |
| 60 | useWatchProbeByDefault(() => async () => true); |
test/helpers/roomy-disk.ts +4 −1
| @@ -9,4 +9,7 @@ | ||
| 9 | 9 | */ |
| 10 | 10 | import { provideDiskForTest } from "../../src/kernel/disk_reading.ts"; |
| 11 | 11 | |
| 12 | provideDiskForTest(() => ({ freeBytes: 900 * 1024 ** 3, totalBytes: 1000 * 1024 ** 3 })); | |
| 12 | /** The suite's reading; a test that gave its own puts this one back, never the machine's. */ | |
| 13 | export const roomyDisk = () => ({ freeBytes: 900 * 1024 ** 3, totalBytes: 1000 * 1024 ** 3 }); | |
| 14 | ||
| 15 | provideDiskForTest(roomyDisk); |
test/home-start-forge.test.ts +27
| @@ -131,6 +131,27 @@ test("no capacity lands on / with the sentence, and no raw message", async () => | ||
| 131 | 131 | expect(body).toContain('value="roomy"'); |
| 132 | 132 | }); |
| 133 | 133 | |
| 134 | test("a queued start lands on / with its place and how long to wait, in words", async () => { | |
| 135 | const { setLimit } = await import("../src/settings/limits.ts"); | |
| 136 | const { takePlace, provideHoldForTest } = await import("../src/spawn/births.ts"); | |
| 137 | refusal = undefined; | |
| 138 | resetAnonSpawns(app.db); | |
| 139 | provideHoldForTest(50); | |
| 140 | setLimit(app.db, "spawn.births_at_once", 1, "test"); | |
| 141 | const held = await takePlace(app); | |
| 142 | try { | |
| 143 | const { to, body } = await landing("waiter"); | |
| 144 | expect(to).toStartWith("/?start=queued"); | |
| 145 | expect(sentence(body)).toMatch(/^Many forges are being made right now, and yours is number 1 in line\. Try again in \d+ s: press Start my forge again\.$/); | |
| 146 | expect(body).toContain('value="waiter"'); | |
| 147 | expect(body).not.toContain("retryAfter"); | |
| 148 | } finally { | |
| 149 | provideHoldForTest(undefined); | |
| 150 | if (held.ok) held.release(); | |
| 151 | setLimit(app.db, "spawn.births_at_once", null, "test"); | |
| 152 | } | |
| 153 | }); | |
| 154 | ||
| 134 | 155 | test("the rate limit lands on / with the sentence, and no code or time", async () => { |
| 135 | 156 | refusal = undefined; |
| 136 | 157 | useProbe(app, () => async () => false); // every forge started here is still booting: a 200 waiting page, then the 303 refusal |
| @@ -149,6 +170,12 @@ test("the rate limit lands on / with the sentence, and no code or time", async ( | ||
| 149 | 170 | expect(last.body).not.toContain("rate_limited"); |
| 150 | 171 | }); |
| 151 | 172 | |
| 173 | test("a queued landing reads only whole numbers for its place and wait", async () => { | |
| 174 | const forged = await (await fetch(`${servers.url}/?start=queued&name=x&place=<b>1&after=9e9`)).text(); | |
| 175 | expect(sentence(forged)).toBe(START_REFUSALS.queued); | |
| 176 | expect(forged).not.toContain("<b>1"); | |
| 177 | }); | |
| 178 | ||
| 152 | 179 | test("the name is escaped and an unknown kind shows nothing", async () => { |
| 153 | 180 | const hostile = await (await fetch(`${servers.url}/?start=name&name=${encodeURIComponent('"><script>alert(1)</script>')}`)).text(); |
| 154 | 181 | expect(hostile).not.toContain("<script>alert(1)"); |
test/job-slices.test.ts +3
| @@ -303,6 +303,9 @@ const UNTIMED: Record<string, string> = { | ||
| 303 | 303 | "cell-duty-resume": "sends SIGCONT to a cell tree the duty pause stopped: needs cells, skipped in this sandbox mode", |
| 304 | 304 | "apps-restart": "restarts an app process after it exits; the fixture has no apps", |
| 305 | 305 | "app-tty-idle": "closes an app terminal left idle; the fixture has no app terminals", |
| 306 | "births-look": "an interval in spawn/births.ts, armed only while a hosted forge is being born: one look at the child over the network; the fixture has no births", | |
| 307 | "births-cap": "a one-shot timer in spawn/births.ts that releases a birth held past its maximum; the fixture has no births", | |
| 308 | "births-hold": "a one-shot timer in spawn/births.ts that answers a held birth as late; the fixture has no births", | |
| 306 | 309 | "route-timing-flush": "a debounced write of route timings: runs after requests, the jobs here make none", |
| 307 | 310 | "usage-count-flush": "a debounced write of usage counts: runs after requests, the jobs here make none", |
| 308 | 311 | "access-flush": "a debounced write of the access log: runs after requests, the jobs here make none", |
test/limits.test.ts +2
| @@ -102,6 +102,8 @@ describe("limits", () => { | ||
| 102 | 102 | expect(text).toContain(key); |
| 103 | 103 | expect(text).toContain(LIMITS[key].reason.replace(/'/g, "'")); |
| 104 | 104 | } |
| 105 | // What the births limits hold back now sits beside them (`spawn/births.ts`). | |
| 106 | expect(text).toMatch(/id="births-now">New forges now: \d+ being made, \d+ waiting their turn\.</); | |
| 105 | 107 | }); |
| 106 | 108 | |
| 107 | 109 | test("a value out of bounds is refused with its bound named; a cross-origin post is refused", async () => { |
test/looks-owner.test.ts +3 −3
| @@ -80,7 +80,7 @@ describe("app.looksAsOwner in forge.json", () => { | ||
| 80 | 80 | |
| 81 | 81 | test("the forge's own looks take its admin pages as the owner", () => { |
| 82 | 82 | const own = parseManifest(readFileSync(join(import.meta.dir, "..", "forge.json"), "utf8"), undefined); |
| 83 | expect(own.ok && own.value.looksAsOwner).toEqual(expect.arrayContaining(["/resilience", "/system"])); | |
| 83 | expect(own.ok && own.value.looksAsOwner).toEqual(expect.arrayContaining(["/resilience", "/system", "/admin/limits"])); | |
| 84 | 84 | }); |
| 85 | 85 | }); |
| 86 | 86 | |
| @@ -252,10 +252,10 @@ describe("a preview hears its parent's admin as the preview admin", () => { | ||
| 252 | 252 | expect(turned.status).toBe(403); |
| 253 | 253 | }); |
| 254 | 254 | |
| 255 | test("/resilience and /system answer the preview admin, with no store said in one line and no credential in the page", async () => { | |
| 255 | test("/resilience, /system and /admin/limits answer the preview admin, with no store said in one line and no credential in the page", async () => { | |
| 256 | 256 | const preview = servers[0]!; |
| 257 | 257 | const headers = identity("admin"); |
| 258 | for (const path of ["/resilience", "/system"]) { | |
| 258 | for (const path of ["/resilience", "/system", "/admin/limits"]) { | |
| 259 | 259 | const res = await fetch(`${preview.url}${path}`, { headers, redirect: "manual" }); |
| 260 | 260 | expect(res.status).toBe(200); |
| 261 | 261 | expect(res.headers.get("set-cookie") ?? "").not.toContain("forge_session"); |
test/release-prober.test.ts +3 −1
| @@ -13,9 +13,11 @@ import { createApp, type App } from "../src/app.ts"; | ||
| 13 | 13 | import { serve, type Servers } from "../src/http/server.ts"; |
| 14 | 14 | import { ok } from "../src/kernel/result.ts"; |
| 15 | 15 | import { usePlacement, useProbe } from "../src/spawn/door.ts"; |
| 16 | import { ANON_SPAWNS_GLOBAL_PER_DAY, noteAnonSpawn, resetAnonSpawns } from "../src/spawn/rate.ts"; | |
| 16 | import { noteAnonSpawn, resetAnonSpawns } from "../src/spawn/rate.ts"; | |
| 17 | 17 | import { isProber, PROBER_HEADER, proberKey } from "../src/release/prober.ts"; |
| 18 | 18 | import { walkJourney } from "../src/release/journey.ts"; |
| 19 | import { LIMITS } from "../src/settings/limits.ts"; | |
| 20 | const ANON_SPAWNS_GLOBAL_PER_DAY = LIMITS["spawn.anon_global_per_day"].default; | |
| 19 | 21 | |
| 20 | 22 | let dataDir: string; |
| 21 | 23 | let app: App; |
test/spawn-anon.test.ts +51 −1
| @@ -12,13 +12,16 @@ import { createApp, type App } from "../src/app.ts"; | ||
| 12 | 12 | import { commitToBranch, createRepo } from "../src/actions/repos.ts"; |
| 13 | 13 | import { usePlacement, useProbe } from "../src/spawn/door.ts"; |
| 14 | 14 | import { Database } from "bun:sqlite"; |
| 15 | import { ANON_SPAWNS_GLOBAL_PER_DAY, ANON_SPAWNS_PER_DAY, checkAnonSpawn, noteAnonSpawn, resetAnonSpawns } from "../src/spawn/rate.ts"; | |
| 15 | import { checkAnonSpawn, noteAnonSpawn, resetAnonSpawns } from "../src/spawn/rate.ts"; | |
| 16 | 16 | import { createToken } from "../src/core/tokens.ts"; |
| 17 | 17 | import { createUser } from "../src/core/users.ts"; |
| 18 | 18 | import { serve, type Servers } from "../src/http/server.ts"; |
| 19 | 19 | import { childDirOf } from "../src/actions/spawn.ts"; |
| 20 | 20 | import { bootHostedChild } from "../src/spawn/host.ts"; |
| 21 | 21 | import { ok } from "../src/kernel/result.ts"; |
| 22 | import { LIMITS } from "../src/settings/limits.ts"; | |
| 23 | const ANON_SPAWNS_PER_DAY = LIMITS["spawn.anon_per_day"].default; | |
| 24 | const ANON_SPAWNS_GLOBAL_PER_DAY = LIMITS["spawn.anon_global_per_day"].default; | |
| 22 | 25 | |
| 23 | 26 | describe("anonymous spawn", () => { |
| 24 | 27 | let dataDir: string; |
| @@ -190,6 +193,33 @@ describe("anonymous spawn", () => { | ||
| 190 | 193 | resetAnonSpawns(app.db); |
| 191 | 194 | }); |
| 192 | 195 | |
| 196 | test("a surge past the births at once is told its place and about how long, not refused", async () => { | |
| 197 | const { setLimit } = await import("../src/settings/limits.ts"); | |
| 198 | const { takePlace, provideHoldForTest } = await import("../src/spawn/births.ts"); | |
| 199 | provideHoldForTest(50); | |
| 200 | setLimit(app.db, "spawn.births_at_once", 1, "test"); | |
| 201 | const held = await takePlace(app); | |
| 202 | try { | |
| 203 | const res = await post({ name: "queued1", dryRun: false }); | |
| 204 | expect(res.status).toBe(503); | |
| 205 | const body = (await res.json()) as any; | |
| 206 | expect(body).toEqual(expect.objectContaining({ error: "queued", place: 1, why: "births" })); | |
| 207 | expect(body.aboutS).toBeGreaterThan(0); | |
| 208 | expect(body.message).toContain("1 in line"); | |
| 209 | expect(body.message).toMatch(/try again in \d+ s$/); | |
| 210 | // The header names the same wait the body does, so a client need not read the body. | |
| 211 | expect(res.headers.get("retry-after")).toBe(String(body.retryAfter)); | |
| 212 | // A stranger is told no figure of the machine. | |
| 213 | expect(body.message).not.toMatch(/MiB|being made, the most|floor/); | |
| 214 | // Nothing was made, and nothing was counted against the stranger. | |
| 215 | expect(app.db.query("SELECT COUNT(*) AS n FROM anon_spawn").get()).toEqual({ n: 0 }); | |
| 216 | } finally { | |
| 217 | provideHoldForTest(undefined); | |
| 218 | if (held.ok) held.release(); | |
| 219 | setLimit(app.db, "spawn.births_at_once", null, "test"); | |
| 220 | } | |
| 221 | }, 30_000); | |
| 222 | ||
| 193 | 223 | test("the counts survive a restart and a global cap holds", async () => { |
| 194 | 224 | const req = new Request("http://x/", { headers: { "x-forwarded-for": "203.0.113.9" } }); |
| 195 | 225 | for (let i = 0; i < ANON_SPAWNS_PER_DAY; i++) noteAnonSpawn(app.db, req); |
| @@ -226,4 +256,24 @@ describe("anonymous spawn", () => { | ||
| 226 | 256 | } |
| 227 | 257 | expect(await readdir(join(dataDir, "children")).catch(() => [])).toHaveLength(4); |
| 228 | 258 | }); |
| 259 | ||
| 260 | test("a forge made but not timed gives its place back at once", async () => { | |
| 261 | const { setLimit } = await import("../src/settings/limits.ts"); | |
| 262 | const { birthCounts } = await import("../src/spawn/births.ts"); | |
| 263 | const { startWatch } = await import("../src/spawn/live.ts"); | |
| 264 | setLimit(app.db, "spawn.timers_max", 1, "test"); | |
| 265 | let open!: () => void; | |
| 266 | const gate = new Promise<boolean>((r) => (open = () => r(true))); | |
| 267 | // The one timer there is stays taken until the spawn has been answered. | |
| 268 | startWatch(app, { name: "taken", startedAt: Date.now(), pollMs: 5, probe: () => gate }); | |
| 269 | try { | |
| 270 | const res = await post({ name: "untimed", dryRun: false }, rw); | |
| 271 | expect(res.status).toBe(201); | |
| 272 | expect(birthCounts(app).inFlight).toBe(0); | |
| 273 | await settle("untimed"); | |
| 274 | } finally { | |
| 275 | open(); | |
| 276 | setLimit(app.db, "spawn.timers_max", null, "test"); | |
| 277 | } | |
| 278 | }); | |
| 229 | 279 | }); |
test/spawn-births.test.ts added +164
| @@ -0,0 +1,164 @@ | ||
| 1 | /** | |
| 2 | * One admission rule for births (`spawn/births.ts`): births in flight, memory left above the floor | |
| 3 | * and disk above its floor decide whether a spawn starts, waits its turn, or is told its place. | |
| 4 | */ | |
| 5 | import { afterEach, expect, test } from "bun:test"; | |
| 6 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 7 | import { tmpdir } from "node:os"; | |
| 8 | import { join } from "node:path"; | |
| 9 | import { createApp } from "../src/app.ts"; | |
| 10 | import { provideDiskForTest, readDisk } from "../src/kernel/disk_reading.ts"; | |
| 11 | import { birthCounts, provideMemoryForTest, takePlace, whyNot } from "../src/spawn/births.ts"; | |
| 12 | import { setLimit } from "../src/settings/limits.ts"; | |
| 13 | import { roomyDisk } from "./helpers/roomy-disk.ts"; | |
| 14 | ||
| 15 | const GIB = 1024 ** 3; | |
| 16 | const made: string[] = []; | |
| 17 | afterEach(async () => { | |
| 18 | provideMemoryForTest(() => 64 * GIB); | |
| 19 | // The preload's roomy reading, not the machine's: later files in this process read it too. | |
| 20 | provideDiskForTest(roomyDisk); | |
| 21 | for (const dir of made.splice(0)) await rm(dir, { recursive: true, force: true }); | |
| 22 | }); | |
| 23 | ||
| 24 | const fresh = async () => { | |
| 25 | const dataDir = await mkdtemp(join(tmpdir(), "forge-births-")); | |
| 26 | made.push(dataDir); | |
| 27 | return createApp({ dataDir, port: 0 }); | |
| 28 | }; | |
| 29 | ||
| 30 | test("births past the setting wait, are told their place, and a release lets the next go", async () => { | |
| 31 | const app = await fresh(); | |
| 32 | setLimit(app.db, "spawn.births_at_once", 2, "t"); | |
| 33 | const a = await takePlace(app); | |
| 34 | const b = await takePlace(app); | |
| 35 | expect([a.ok, b.ok]).toEqual([true, true]); | |
| 36 | expect(birthCounts(app).inFlight).toBe(2); | |
| 37 | const late = await takePlace(app, { holdMs: 30 }); | |
| 38 | expect(late.ok).toBe(false); | |
| 39 | if (!late.ok) { | |
| 40 | expect(late.waiting.place).toBe(1); | |
| 41 | expect(late.waiting.why).toBe("births"); | |
| 42 | expect(late.waiting.aboutS).toBeGreaterThan(0); | |
| 43 | } | |
| 44 | expect(birthCounts(app).waiting).toBe(0); | |
| 45 | const next = takePlace(app, { holdMs: 2_000 }); | |
| 46 | await Bun.sleep(10); // real-timer: the queue is a real in-process timer race | |
| 47 | expect(birthCounts(app).waiting).toBe(1); | |
| 48 | if (a.ok) a.release(); | |
| 49 | const got = await next; | |
| 50 | expect(got.ok).toBe(true); | |
| 51 | expect(birthCounts(app).inFlight).toBe(2); | |
| 52 | if (a.ok) a.release(); // once only | |
| 53 | expect(birthCounts(app).inFlight).toBe(2); | |
| 54 | app.close(); | |
| 55 | }); | |
| 56 | ||
| 57 | test("the queue is first in, first out", async () => { | |
| 58 | const app = await fresh(); | |
| 59 | setLimit(app.db, "spawn.births_at_once", 1, "t"); | |
| 60 | const first = await takePlace(app); | |
| 61 | const order: string[] = []; | |
| 62 | const x = takePlace(app, { holdMs: 2_000 }).then((p) => (order.push("x"), p)); | |
| 63 | await Bun.sleep(5); // real-timer: the queue is a real in-process timer race | |
| 64 | const y = takePlace(app, { holdMs: 2_000 }).then((p) => (order.push("y"), p)); | |
| 65 | await Bun.sleep(5); // real-timer: the queue is a real in-process timer race | |
| 66 | if (first.ok) first.release(); | |
| 67 | const px = await x; | |
| 68 | if (px.ok) px.release(); | |
| 69 | await y; | |
| 70 | expect(order).toEqual(["x", "y"]); | |
| 71 | app.close(); | |
| 72 | }); | |
| 73 | ||
| 74 | test("memory left above the floor decides, and a forge given room starts", async () => { | |
| 75 | const app = await fresh(); | |
| 76 | setLimit(app.db, "forge.memory_floor_mb", 1024, "t"); | |
| 77 | provideMemoryForTest(() => 2 * GIB); // 1 GiB above the floor, a birth needs 1.5 | |
| 78 | expect(whyNot(app)?.why).toBe("memory"); | |
| 79 | expect((await takePlace(app, { holdMs: 20 })).ok).toBe(false); | |
| 80 | provideMemoryForTest(() => 4 * GIB); | |
| 81 | expect(whyNot(app)).toBeUndefined(); | |
| 82 | expect((await takePlace(app, { holdMs: 20 })).ok).toBe(true); | |
| 83 | setLimit(app.db, "spawn.birth_memory_mb", 0, "t"); | |
| 84 | provideMemoryForTest(() => 1 * GIB); | |
| 85 | expect(whyNot(app)).toBeUndefined(); | |
| 86 | app.close(); | |
| 87 | }); | |
| 88 | ||
| 89 | test("a birth in flight is set against the memory it has not yet taken", async () => { | |
| 90 | const app = await fresh(); | |
| 91 | setLimit(app.db, "spawn.births_at_once", 8, "t"); | |
| 92 | setLimit(app.db, "forge.memory_floor_mb", 1024, "t"); | |
| 93 | provideMemoryForTest(() => 3 * GIB); // 2 GiB above the floor: room for one birth of 1.5, not two | |
| 94 | const first = await takePlace(app); | |
| 95 | expect(first.ok).toBe(true); | |
| 96 | // The reading has not moved, as it does not in the minutes a birth is taking its share. | |
| 97 | const second = await takePlace(app, { holdMs: 20 }); | |
| 98 | expect(second.ok).toBe(false); | |
| 99 | if (!second.ok) expect(second.waiting.why).toBe("memory"); | |
| 100 | expect(birthCounts(app).inFlight).toBe(1); | |
| 101 | // Two in line on one reading: given back, the first lets exactly one more go. | |
| 102 | const x = takePlace(app, { holdMs: 2_000 }); | |
| 103 | const y = takePlace(app, { holdMs: 300 }); | |
| 104 | await Bun.sleep(10); // real-timer: the queue is a real in-process timer race | |
| 105 | if (first.ok) first.release(); | |
| 106 | const [px, py] = await Promise.all([x, y]); | |
| 107 | expect(px.ok).toBe(true); | |
| 108 | expect(py.ok).toBe(false); | |
| 109 | if (!py.ok) expect(py.waiting.why).toBe("memory"); | |
| 110 | expect(birthCounts(app).inFlight).toBe(1); | |
| 111 | if (px.ok) px.release(); | |
| 112 | app.close(); | |
| 113 | }); | |
| 114 | ||
| 115 | test("disk under its floor holds births", async () => { | |
| 116 | const app = await fresh(); | |
| 117 | provideDiskForTest(() => ({ freeBytes: 5 * GIB, totalBytes: 100 * GIB })); | |
| 118 | expect(whyNot(app)?.why).toBe("disk"); | |
| 119 | provideDiskForTest(() => ({ freeBytes: 50 * GIB, totalBytes: 100 * GIB })); | |
| 120 | expect(whyNot(app)).toBeUndefined(); | |
| 121 | app.close(); | |
| 122 | }); | |
| 123 | ||
| 124 | test("a test that gave its own disk reading leaves the suite's roomy one, not the machine's", () => { | |
| 125 | // The test before this one read a disk under its floor; files after this one share the process. | |
| 126 | expect(readDisk(tmpdir())).toEqual(roomyDisk()); | |
| 127 | }); | |
| 128 | ||
| 129 | test("the line is bounded: past four times the births at once, a spawn is answered at once", async () => { | |
| 130 | const app = await fresh(); | |
| 131 | setLimit(app.db, "spawn.births_at_once", 1, "t"); | |
| 132 | const held = await takePlace(app); | |
| 133 | const waiting = Array.from({ length: 4 }, () => takePlace(app, { holdMs: 300 })); | |
| 134 | await Bun.sleep(10); // real-timer: the queue is a real in-process timer race | |
| 135 | expect(birthCounts(app).waiting).toBe(4); | |
| 136 | const at = Date.now(); | |
| 137 | const over = await takePlace(app, { holdMs: 5_000 }); | |
| 138 | expect(over.ok).toBe(false); | |
| 139 | expect(Date.now() - at).toBeLessThan(200); | |
| 140 | expect(birthCounts(app).waiting).toBe(4); | |
| 141 | await Promise.all(waiting); | |
| 142 | if (held.ok) held.release(); | |
| 143 | app.close(); | |
| 144 | }); | |
| 145 | ||
| 146 | test("strangers hold a quarter of the line and a signed-in person goes ahead of them", async () => { | |
| 147 | const app = await fresh(); | |
| 148 | setLimit(app.db, "spawn.births_at_once", 1, "t"); | |
| 149 | const held = await takePlace(app); | |
| 150 | const order: string[] = []; | |
| 151 | const stranger = takePlace(app, { holdMs: 2_000, anon: true }).then((p) => (order.push("stranger"), p)); | |
| 152 | await Bun.sleep(5); // real-timer: the queue is a real in-process timer race | |
| 153 | const refused = await takePlace(app, { holdMs: 2_000, anon: true }); | |
| 154 | expect(refused.ok).toBe(false); // a second stranger finds the strangers' share full | |
| 155 | const person = takePlace(app, { holdMs: 2_000 }).then((p) => (order.push("person"), p)); | |
| 156 | await Bun.sleep(5); // real-timer: the queue is a real in-process timer race | |
| 157 | if (held.ok) held.release(); | |
| 158 | const p = await person; | |
| 159 | if (p.ok) p.release(); | |
| 160 | const s = await stranger; | |
| 161 | if (s.ok) s.release(); | |
| 162 | expect(order).toEqual(["person", "stranger"]); | |
| 163 | app.close(); | |
| 164 | }); |
test/spawn-limits-listed.test.ts added +65
| @@ -0,0 +1,65 @@ | ||
| 1 | /** | |
| 2 | * The terminal limits screen shows the new-forge limits with the rest, and a line of what they hold | |
| 3 | * back now: births in flight and spawns waiting (`spawn/births.ts`), so an admin who sees strangers | |
| 4 | * queued can see why. | |
| 5 | */ | |
| 6 | import { afterAll, beforeAll, expect, test } from "bun:test"; | |
| 7 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 8 | import { tmpdir } from "node:os"; | |
| 9 | import { join } from "node:path"; | |
| 10 | import { createApp, type App as Forge } from "../src/app.ts"; | |
| 11 | import { createUser, getUserByName } from "../src/core/users.ts"; | |
| 12 | import { setLimit } from "../src/settings/limits.ts"; | |
| 13 | import { takePlace } from "../src/spawn/births.ts"; | |
| 14 | import { createForgeApp } from "../src/tui/forge.ts"; | |
| 15 | import { headlessTerminal } from "../src/tui/term/terminal.ts"; | |
| 16 | import { quietLogs } from "./helpers/globals.ts"; | |
| 17 | import { settleApp } from "./helpers/tui.ts"; | |
| 18 | ||
| 19 | quietLogs(); | |
| 20 | ||
| 21 | let dataDir: string; | |
| 22 | let forge: Forge; | |
| 23 | beforeAll(async () => { | |
| 24 | dataDir = await mkdtemp(join(tmpdir(), "forge-spawn-limits-")); | |
| 25 | forge = createApp({ dataDir, port: 0 }); | |
| 26 | const made = await createUser(forge.db, { origin: "seed", trusted: true, username: "alice", email: "[email protected]", password: "a-good-passphrase", adminIfFirst: true }); | |
| 27 | if (!made.ok) throw new Error(made.error); | |
| 28 | }); | |
| 29 | afterAll(async () => { | |
| 30 | forge.close(); | |
| 31 | await rm(dataDir, { recursive: true, force: true }); | |
| 32 | }); | |
| 33 | ||
| 34 | async function limitsText(): Promise<string> { | |
| 35 | // Tall enough that every limit is on the screen at once. | |
| 36 | const terminal = headlessTerminal(160, 200); | |
| 37 | const user = getUserByName(forge.db, "alice")!; | |
| 38 | const { app } = createForgeApp(forge, { sshUsername: "t", remoteAddress: "127.0.0.1", user }, { terminal, exitHooks: false }); | |
| 39 | app.start(); | |
| 40 | const key = (k: string, ctrl = false) => app.dispatch({ type: "key", key: k, ctrl, alt: false, shift: false }); | |
| 41 | key("k", true); | |
| 42 | for (const ch of "Limits") key(ch); | |
| 43 | key("enter"); | |
| 44 | await settleApp(app); | |
| 45 | const text = app.screen.toText(); | |
| 46 | app.stop(); | |
| 47 | return text; | |
| 48 | } | |
| 49 | ||
| 50 | test("the terminal limits screen lists the new-forge limits and the births line", async () => { | |
| 51 | const idle = await limitsText(); | |
| 52 | for (const key of ["spawn.births_at_once", "spawn.birth_memory_mb", "spawn.timers_max", "forges.wakes_per_forge_hour", "spawn.anon_per_day", "spawn.anon_global_per_day"]) | |
| 53 | expect(idle).toContain(key); | |
| 54 | expect(idle).toContain("New forges now: 0 being made, 0 waiting their turn."); | |
| 55 | ||
| 56 | setLimit(forge.db, "spawn.births_at_once", 1, "test"); | |
| 57 | const held = await takePlace(forge); | |
| 58 | const waiting = takePlace(forge, { holdMs: 2_000, anon: true }); | |
| 59 | await Bun.sleep(10); // real-timer: the queue is a real in-process timer race | |
| 60 | expect(await limitsText()).toContain("New forges now: 1 being made, 1 waiting their turn."); | |
| 61 | if (held.ok) held.release(); | |
| 62 | const next = await waiting; | |
| 63 | if (next.ok) next.release(); | |
| 64 | setLimit(forge.db, "spawn.births_at_once", null, "test"); | |
| 65 | }); |
test/spawn-live.test.ts +28 −1
| @@ -9,7 +9,9 @@ import { join } from "node:path"; | ||
| 9 | 9 | import { createApp } from "../src/app.ts"; |
| 10 | 10 | import { runQuery } from "../src/boards/query.ts"; |
| 11 | 11 | import { STARTING_BOARDS } from "../src/boards/starting.ts"; |
| 12 | import { MAX_WATCHERS, startWatch, watchLive } from "../src/spawn/live.ts"; | |
| 12 | import { startWatch, watchLive } from "../src/spawn/live.ts"; | |
| 13 | import { LIMITS, setLimit } from "../src/settings/limits.ts"; | |
| 14 | const MAX_WATCHERS = LIMITS["spawn.timers_max"].default; | |
| 13 | 15 | |
| 14 | 16 | let dir: string; |
| 15 | 17 | let app: ReturnType<typeof createApp>; |
| @@ -112,3 +114,28 @@ test("after 30 seconds the poll slows to five", async () => { | ||
| 112 | 114 | await pending; |
| 113 | 115 | expect(calls).toBe(1); |
| 114 | 116 | }); |
| 117 | ||
| 118 | test("a spawn past the timers is logged as made but not timed, and the setting moves the ceiling", async () => { | |
| 119 | const mine = createApp({ dataDir: await mkdtemp(join(tmpdir(), "forge-spawn-untimed-")), port: 0 }); | |
| 120 | const lines: string[] = []; | |
| 121 | const warn = mine.log.warn.bind(mine.log); | |
| 122 | mine.log.warn = ((msg: string, ...rest: unknown[]) => { | |
| 123 | lines.push(msg); | |
| 124 | return warn(msg, ...(rest as [])); | |
| 125 | }) as typeof mine.log.warn; | |
| 126 | expect(setLimit(mine.db, "spawn.timers_max", 2, "test").ok).toBe(true); | |
| 127 | const w = { name: "c", startedAt: Date.now(), pollMs: 10_000, probe: async () => false }; | |
| 128 | expect([startWatch(mine, w), startWatch(mine, w), startWatch(mine, w)]).toEqual([true, true, false]); | |
| 129 | expect(lines.filter((l) => l.includes("not timed")).length).toBe(1); | |
| 130 | mine.close(); | |
| 131 | }); | |
| 132 | ||
| 133 | test("onSettled is called when a watch ends", async () => { | |
| 134 | let settled = 0; | |
| 135 | const got = await new Promise<string>((done) => { | |
| 136 | const mine = createApp({ dataDir: dir + "-s", port: 0 }); | |
| 137 | startWatch(mine, { name: "s", startedAt: Date.now(), pollMs: 1, probe: async () => true, onSettled: () => { settled++; mine.close(); done("ok"); } }); | |
| 138 | }); | |
| 139 | expect(got).toBe("ok"); | |
| 140 | expect(settled).toBe(1); | |
| 141 | }); |
new above
offline