trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRunsg rEventsFlows

Hosted forges have a wake rate per person and a forge-wide ceiling on how many run at once; a sleeping forge doesn't count #9616

Merged torquecrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/9582-hosted-forges-have-a-wake-rate-per-perso → main 3 commits fork

  • Merged. torque/9582-hosted-forges-have-a-wake-rate-per-perso was merged into main as 09bc2d7.

Review

Sign in to review this change.
6 files changed +94 −12
docs/features/forge-sleep.md +1 −1
@@ -3,7 +3,7 @@
33 A person (or their assistant, over the MCP `spawn` tool with their consent) makes a forge at `/forges/new`, the address `<name>--<this host>`. Free within the limits below; no payment path yet.
44
55 - **Sleep.** A forge with no request for `FORGE_CHILD_IDLE_S` (600) sleeps: a hosted child's App and apps close, a cell child's process stops. Data stays under `children/<bodyId>`. The next page load, `git push` or API call wakes it. A page for a sleeping cell forge answers a plain "waking up" page at once (git and API calls wait up to 4 s) (503, `retry-after`, reloads itself); a hosted one opens in-process.
6 - **Limits, each refused in words.** The admin limits `forges.per_person` (3) and `forges.storage_mb` (1000 MB), set at `/admin/limits` (the same list is on the terminal's limits screen), each refused in a sentence naming the limit and the person's use (site admins exempt), `FORGE_CHILD_AWAKE_MAX` (8 awake at once; sleeping forges do not count), `FORGE_CHILD_WAKES_PER_HOUR` (20 per forge), plus the existing size/quota limits (`FORGE_CHILD_*_MIB`), and each cell's own disk cap.
6 - **Limits, each refused in words.** The admin limits `forges.per_person` (3) and `forges.storage_mb` (1000 MB), set at `/admin/limits` (the same list is on the terminal's limits screen), each refused in a sentence naming the limit and the person's use (site admins exempt), `FORGE_CHILD_AWAKE_MAX` (8 awake at once; sleeping forges do not count), `FORGE_CHILD_WAKES_PER_HOUR` (20 per forge), the admin limit `forges.wakes_per_person_hour` (30 wakes an hour across one person's forges, refused with "waking too often, try in N s"), plus the existing size/quota limits (`FORGE_CHILD_*_MIB`), and each cell's own disk cap.
77 - **Owner.** `/forges` shows awake/asleep, size, last activity, with Stop (sleep now) and Remove (ledger and data).
88 - **Secrets.** A child starts from `childEnv` and never sees this forge's model credential; the person brings their own or its crew stays off.
99
src/settings/limits.ts +9
@@ -42,6 +42,7 @@ export type LimitKey =
4242 | "guest.forks_forge_per_day"
4343 | "guest.pulls_per_day"
4444 | "forges.per_person"
45 | "forges.wakes_per_person_hour"
4546 | "forges.storage_mb"
4647 | "guest.storage_mb"
4748 | "guest.fork_storage_mb"
@@ -221,6 +222,14 @@ export const LIMITS: Record<LimitKey, LimitDef> = {
221222 max: 1_000,
222223 reason: "Every hosted forge holds a database and its repositories on this machine's disk; site admins are not counted.",
223224 },
225 "forges.wakes_per_person_hour": {
226 label: "Wakes per person per hour",
227 unit: "wakes",
228 default: 30,
229 min: 1,
230 max: 10_000,
231 reason: "Waking a sleeping forge starts a process on this machine; a person who wakes theirs too often is asked to wait.",
232 },
224233 "forges.storage_mb": {
225234 label: "Storage a person can hold in hosted forges",
226235 unit: "MB",
src/spawn/host.ts +15 −2
@@ -25,6 +25,7 @@ import {
2525 import { DEFAULT_SIZE, isSize, type Size } from "./sizes.ts";
2626 import { cellChildren, forward, forwardSocket, type CellChild } from "./cell.ts";
2727 import { cellsAvailable } from "../apps/contain.ts";
28 import { limitValue } from "../settings/limits.ts";
2829 import { awakeRefusal, idleAfterMs, WakeRate } from "./sleep.ts";
2930 import { wakingResponse } from "../web/pages/waking.ts";
3031 import { takeDoor } from "../http/strangers.ts";
@@ -35,6 +36,7 @@ import { join } from "node:path";
3536 interface ChildLedger {
3637 readonly name: string;
3738 readonly bodyId: string;
39 readonly ownerId: string | undefined;
3840 readonly done: boolean;
3941 readonly placement: PlacementKind;
4042 readonly size: Size;
@@ -46,6 +48,7 @@ function readLedgerSync(app: App, name: string): ChildLedger | undefined {
4648 const parsed = JSON.parse(raw) as {
4749 name?: unknown;
4850 bodyId?: unknown;
51 ownerId?: unknown;
4952 done?: unknown;
5053 placement?: unknown;
5154 size?: unknown;
@@ -54,6 +57,7 @@ function readLedgerSync(app: App, name: string): ChildLedger | undefined {
5457 return {
5558 name: parsed.name,
5659 bodyId: parsed.bodyId,
60 ownerId: typeof parsed.ownerId === "string" ? parsed.ownerId : undefined,
5761 done: parsed.done === true,
5862 placement: parsed.placement === "cell" ? "cell" : "hosted",
5963 size: isSize(parsed.size) ? parsed.size : DEFAULT_SIZE,
@@ -102,6 +106,9 @@ export function hostedChildOpen(parent: App, child: App): boolean {
102106 /** When a request last reached each open hosted child, by body id, with the name it goes by. */
103107 const touched = new WeakMap<App, Map<string, { name: string; at: number }>>();
104108 const wakeRates = new WeakMap<App, WakeRate>();
109 const personWakes = new WeakMap<App, WakeRate>();
110 /** A clock the gate reads; a test sets it to a fake. */
111 export const gateClock = { now: () => Date.now() };
105112 const sweepers = new WeakMap<App, ReturnType<typeof setInterval>>();
106113
107114 function touch(parent: App, ledger: { bodyId: string; name: string }, now = Date.now()): void {
@@ -268,9 +275,15 @@ export function wakeGate(parent: App, request: Request): Response | undefined {
268275 if (!verdict.allowed) return wakingResponse(parent, request, name, "too many requests to sleeping forges from your address; try again in a minute");
269276 let rates = wakeRates.get(parent);
270277 if (!rates) wakeRates.set(parent, (rates = new WakeRate()));
271 const why = awakeRefusal(awakeCount(parent)) ?? (known ? undefined : rates.refusal(ledger.bodyId));
278 const at = gateClock.now();
279 let people = personWakes.get(parent);
280 if (!people) personWakes.set(parent, (people = new WakeRate()));
281 const wait = ledger.ownerId === undefined ? 0 : people.waitS(ledger.ownerId, limitValue(parent.db, "forges.wakes_per_person_hour"), at);
282 if (wait > 0) return wakingResponse(parent, request, name, `waking too often, try in ${wait} s`, wait);
283 const why = awakeRefusal(awakeCount(parent)) ?? (known ? undefined : rates.refusal(ledger.bodyId, at));
272284 if (why !== undefined) return wakingResponse(parent, request, name, why);
273 if (!known) rates.note(ledger.bodyId);
285 if (!known) rates.note(ledger.bodyId, at);
286 if (ledger.ownerId !== undefined) people.note(ledger.ownerId, at);
274287 reserved.set(ledger.bodyId, Date.now());
275288 touch(parent, ledger);
276289 return undefined;
src/spawn/sleep.ts +13 −3
@@ -49,6 +49,16 @@ export class WakeRate {
4949 ? `this forge has been woken ${max} times in the last hour, the most it may be; try again in a little while`
5050 : undefined;
5151 }
52 /**
53 * Seconds until `key` (a person) may wake another forge, 0 when it may now:
54 * `max` wakes in any hour; the oldest leaves the window at its time + 1 h.
55 */
56 waitS(key: string, max: number, now = Date.now()): number {
57 const r = this.recent(key, now);
58 if (r.length < max) return 0;
59 const free = (r[r.length - max] as number) + HOUR_MS;
60 return Math.max(1, Math.ceil((free - now) / 1000));
61 }
5262 note(bodyId: string, now = Date.now()): void {
5363 this.recent(bodyId, now).push(now);
5464 }
@@ -58,7 +68,7 @@ export class WakeRate {
5868 export function awakeRefusal(awake: number, env: Env = process.env): string | undefined {
5969 const max = awakeCeiling(env);
6070 return awake >= max
61 ? `${awake} forges are awake on this host right now, the most it runs at once; try again in a minute`
71 ? "the forge is full, try again shortly"
6272 : undefined;
6373 }
6474
@@ -142,10 +152,10 @@ export function personRefusal(db: Database, dataDir: string, owner: { id: string
142152 }
143153
144154 /** The plain-text "waking up" or refusal answer: for a git client or an API call (a page is `web/pages/waking.ts`). */
145 export function wakingText(name: string, why?: string): Response {
155 export function wakingText(name: string, why?: string, retryS = 5): Response {
146156 const message = why ?? `${name} was asleep and is waking up; this takes a few seconds`;
147157 return new Response(message, {
148158 status: why === undefined ? 503 : 429,
149 headers: { "retry-after": "5", "cache-control": "no-store", "content-type": "text/plain; charset=utf-8" },
159 headers: { "retry-after": String(retryS), "cache-control": "no-store", "content-type": "text/plain; charset=utf-8" },
150160 });
151161 }
src/web/pages/waking.ts +3 −3
@@ -13,8 +13,8 @@ import { html } from "../html.ts";
1313 import { page } from "../layout.ts";
1414
1515 /** Waking (`why` undefined, 503) or refused (`why` a sentence, 429), as a page when the visitor asks for HTML. */
16 export function wakingResponse(app: App, request: Request, name: string, why?: string): Response {
17 if (!(request.headers.get("accept") ?? "").includes("text/html")) return wakingText(name, why);
16 export function wakingResponse(app: App, request: Request, name: string, why?: string, retryS = 5): Response {
17 if (!(request.headers.get("accept") ?? "").includes("text/html")) return wakingText(name, why, retryS);
1818 const url = new URL(request.url);
1919 const ctx: Ctx = { app, request, url, params: {}, auth: { user: undefined, scope: undefined, via: "none" }, theme: undefined };
2020 const response = page(
@@ -33,7 +33,7 @@ export function wakingResponse(app: App, request: Request, name: string, why?: s
3333 },
3434 why === undefined ? 503 : 429,
3535 );
36 response.headers.set("retry-after", "5");
36 response.headers.set("retry-after", String(retryS));
3737 response.headers.set("cache-control", "no-store");
3838 if (why === undefined) response.headers.set("refresh", "3");
3939 return response;
test/forge-sleep.test.ts +53 −3
@@ -15,7 +15,7 @@ import { createApp, type App } from "../src/app.ts";
1515 import { createSession } from "../src/core/sessions.ts";
1616 import { createUser } from "../src/core/users.ts";
1717 import { serve, type Servers } from "../src/http/server.ts";
18 import { awakeCount, childAwake, holdChild, resolveHostedChild, sleepIdleChildren, wakeGate } from "../src/spawn/host.ts";
18 import { awakeCount, childAwake, gateClock, holdChild, resolveHostedChild, sleepIdleChildren, wakeGate } from "../src/spawn/host.ts";
1919 import { removeForge, stopForge } from "../src/spawn/manage.ts";
2020 import { setLimit } from "../src/settings/limits.ts";
2121 import { awakeRefusal, personRefusal, WakeRate } from "../src/spawn/sleep.ts";
@@ -135,7 +135,7 @@ describe("a hosted forge scales to zero", () => {
135135
136136 test("a wake that would pass a limit is refused in plain words, and a sleeping forge is not counted", async () => {
137137 expect(awakeRefusal(1, { FORGE_CHILD_AWAKE_MAX: "2" })).toBeUndefined();
138 expect(awakeRefusal(2, { FORGE_CHILD_AWAKE_MAX: "2" })).toContain("try again in a minute");
138 expect(awakeRefusal(2, { FORGE_CHILD_AWAKE_MAX: "2" })).toContain("the forge is full, try again shortly");
139139 const rates = new WakeRate();
140140 const env = { FORGE_CHILD_WAKES_PER_HOUR: "2" };
141141 rates.note("b", 1000);
@@ -185,7 +185,7 @@ describe("a hosted forge scales to zero", () => {
185185 expect(gate()).toBeUndefined(); // the first wake is granted and holds the one slot
186186 expect(awakeCount(parent)).toBe(1);
187187 // Another forge's wake now would pass the ceiling: the slot is reserved though nothing is open yet.
188 expect(awakeRefusal(awakeCount(parent))).toContain("try again in a minute");
188 expect(awakeRefusal(awakeCount(parent))).toContain("the forge is full, try again shortly");
189189 } finally {
190190 if (was === undefined) delete process.env["FORGE_CHILD_AWAKE_MAX"];
191191 else process.env["FORGE_CHILD_AWAKE_MAX"] = was;
@@ -331,6 +331,56 @@ describe("a hosted forge scales to zero", () => {
331331 expect(existsSync(join(dir, "spawn", "from-agents.json"))).toBe(true);
332332 });
333333
334 test("a person's wakes are rate limited with a fake clock: refused with a wait in seconds, freed as the hour passes", async () => {
335 const rates = new WakeRate();
336 rates.note("p", 1_000_000);
337 rates.note("p", 1_010_000);
338 expect(rates.waitS("p", 3, 1_020_000)).toBe(0);
339 expect(rates.waitS("p", 2, 1_020_000)).toBe(3_580); // the oldest leaves at 1_000_000 + 3_600_000
340 expect(rates.waitS("p", 2, 1_000_000 + 3_600_000)).toBe(0); // the first has left the window, one wake remains of two
341 expect(rates.waitS("q", 1, 1_020_000)).toBe(0);
342
343 await asleepAgain();
344 const t0 = Date.now() + 10_000_000;
345 expect(setLimit(parent.db, "forges.wakes_per_person_hour", 1, "seed").ok).toBe(true);
346 try {
347 gateClock.now = () => t0;
348 expect(gate()).toBeUndefined(); // the one wake this hour
349 await fetch(`${servers.url}/healthz`, { headers: { host: CHILD_HOST } });
350 await asleepAgain();
351 gateClock.now = () => t0 + 60_000;
352 const refused = gate({ accept: "text/html" });
353 expect(refused?.status).toBe(429);
354 expect(await refused?.text()).toContain("waking too often, try in 3540 s");
355 expect(refused?.headers.get("retry-after")).toBe("3540");
356 gateClock.now = () => t0 + 3_600_000;
357 expect(gate()).toBeUndefined();
358 } finally {
359 gateClock.now = () => Date.now();
360 setLimit(parent.db, "forges.wakes_per_person_hour", null, "seed");
361 }
362 });
363
364 test("a wake past the ceiling is told the forge is full; sleep frees the slot", async () => {
365 await fetch(`${servers.url}/healthz`, { headers: { host: CHILD_HOST } });
366 expect(childAwake(parent, "homie")).toBe(true);
367 const was = process.env["FORGE_CHILD_AWAKE_MAX"];
368 process.env["FORGE_CHILD_AWAKE_MAX"] = "1";
369 try {
370 const awake = awakeCount(parent);
371 expect(awake).toBeGreaterThanOrEqual(1);
372 expect(awakeRefusal(awake)).toBe("the forge is full, try again shortly");
373 await sleepIdleChildren(parent, 600_000, later());
374 expect(awakeCount(parent)).toBe(0);
375 expect(awakeRefusal(awakeCount(parent))).toBeUndefined();
376 process.env["FORGE_CHILD_AWAKE_MAX"] = "1";
377 expect(awakeRefusal(1)).toContain("the forge is full");
378 } finally {
379 if (was === undefined) delete process.env["FORGE_CHILD_AWAKE_MAX"];
380 else process.env["FORGE_CHILD_AWAKE_MAX"] = was;
381 }
382 });
383
334384 test("the owner stops a forge, then removes it with its data", async () => {
335385 expect((await stopForge(parent, "homie")).says).toContain("asleep");
336386 expect(childAwake(parent, "homie")).toBe(false);

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs