Public: anyone can read this; the crew can change it.

An app cannot stall the forge: its requests wait behind people's, repeat read… #12268

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/12265-an-app-cannot-stall-the-forge-its-reques → main 16 commits fork 2 threads · 2 unresolved

  • Merged. scout/12265-an-app-cannot-stall-the-forge-its-reques was merged into main as d7f840c.

Review

Sign in to review this change.
16 files changed +679 −9
AGENTS.md +1
@@ -42,4 +42,5 @@ State shared between modules through process memory, not the database, is named
4242 ## Awake ceiling and the boot
4343 `FORGE_CHILD_AWAKE_MAX` counts forges in use. Hosted children the boot opens (so their apps come back) are exempt from `awakeCount` until a credentialed request reaches them or they sleep (`bootOpened` in `src/spawn/host.ts`). The exempt set holds only body ids of existing finished children and is emptied when the parent closes, so it is bounded by the number of children. Cell children are always counted; they are not opened by the boot.
4444
45 An app token's requests go behind people's and repeat reads are 304s (`src/http/app_turn.ts`, `repeat_reads.ts`, `perf/asking.ts`): see `docs/features/app-asking.md`.
4546 `src/core/reports.ts`: `closedSet`, the ids of repositories whose owner closed reports from outside (#12321), read once per database and kept by `setReportsClosed`, the only writer, so every repository page's head link costs no statement; at most one id per closed repository, gone on restart (the rows are `report_closed`). `src/core/contribute_token.ts`: `openSet`, the ids of repositories that allow outside agents, read once per database and kept by `setContributeOpen`, the only writer, at most one id per allowing repository, gone on restart; a raw SQL write to `contribute_open` bypasses it. The terminal door for reports is `r` on the outside agents screen (`setOutsideReports`, admins).
docs/APPS.md +9
@@ -782,6 +782,15 @@ lines: no id, never replayed, and dropped while the reader is behind.
782782 `examples/forge-live` is the worked example, and the acceptance test
783783 (`test/apps-token.test.ts`) deploys it.
784784
785 **Events before polls.** To be told when something changes, open
786 `GET $FORGE_URL/events` once and follow it; do not ask again on a timer. The
787 app template has no polling loop, and its `AGENTS.md` says so. When a read is
788 the right tool, send `If-None-Match` with the `ETag` of the last answer and
789 keep your copy on a `304`: every read an app token makes is answered from the
790 forge's write clock without running its query while nothing was written
791 (`src/http/repeat_reads.ts`, at most 30 s old). Two more things the forge does
792 about an app that asks too much: it waits behind people's requests, and the owner is told which token asks the most. Neither refuses a request; see `docs/features/app-asking.md`.
793
785794 That is production's token: production runs what a person landed. Every
786795 other environment (a pull's preview, a build, a variant) runs code nobody has
787796 landed yet, so its token is bound to the app's one repository, as a crew
docs/features/app-asking.md added +12
@@ -0,0 +1,12 @@
1 # An app cannot stall the forge (#12265)
2
3 The forge serves everyone on one thread, so an app that polls it on a timer must not slow people down. Nothing an app asks is refused; it only goes last, and the forge says who asks the most.
4
5 - **Events before polls.** To be told when something changes, open `GET $FORGE_URL/events` once and follow it. When a read is the right tool, send `If-None-Match` with the last `ETag` and keep your copy on a `304`.
6 - **Repeat reads are free** (`src/http/repeat_reads.ts`). An app token's last 200 tag per URL and `Accept` is kept (under the token's hash, never the secret), at most 512 per forge and 30 s old. While the write clock has not moved, the same request with that tag is answered 304 without running its query. Any write ends it.
7 - **People first** (`src/http/app_turn.ts`). An app token's request waits while a person's request begun in the last second is open, or one was answered in the last 50 ms, for at most one second. With nobody else asking it is served at once.
8 - **Who is asking** (`src/perf/asking.ts`). Any one token over a fifth of the last hour's requests or of the serving thread's time is named, with the route it asks most and how often, with the choice: open the app's page, or put it to sleep, or revoke the token. It is shown on /system and in the terminal's System screen, to the owner only (`canSeeMemory`), and in the admin's needs-you stack. An always-on app nobody opened for a day that is still calling the forge sleeps like any idle app.
9
10 The reading is one in-process map (61 minute buckets, 64 tokens each); it is gone on restart.
11
12 Tests: `test/app-turn.test.ts` (the turn on a fake clock, the 304, the gate on /system), `test/tui-system.test.ts`.
src/actions/health.ts +20 −1
@@ -16,6 +16,7 @@
1616 * One answer for the API (`GET /api/v1/health`) and the MCP `health` tool.
1717 */
1818
19 import { asking } from "../perf/asking.ts";
1920 import { flakeLine, flakeList, flakeNeedLine, flakeNeedRows, type FlakeRow } from "../checks/flake.ts";
2021 import type { App } from "../app.ts";
2122 import { appsRuntime } from "../apps/runtime.ts";
@@ -70,7 +71,7 @@ function gatherVitals(app: App): Vitals {
7071 const envs = runtime
7172 ? listApps(app.db).flatMap((record) => runtime.status(record))
7273 : [];
73 return vitals({
74 const base = vitals({
7475 db: app.db,
7576 dataDir: app.config.dataDir,
7677 ...(app.config.disk ? { disk: app.config.disk } : {}),
@@ -81,6 +82,24 @@ function gatherVitals(app: App): Vitals {
8182 snapshots: snapshotPlace(app),
8283 forgesMemory: memoryOverLine(forgesMemory(app)),
8384 });
85 return withHeavyAskers(base);
86 }
87
88 /**
89 * Any one token that made more than a fifth of the last hour's requests or of the serving
90 * thread's time is a need of its own (#12265): named, with what to do. It is a `todo`: it does
91 * not make the forge unwell (`ok` stays), only asks to be looked at.
92 */
93 export function withHeavyAskers(v: Vitals, now: number = Date.now()): Vitals {
94 const heavy = asking.heavy();
95 if (heavy.length === 0) return v;
96 const since = new Date(now).toISOString();
97 return {
98 ...v,
99 needs: [...v.needs, ...heavy.map((a) => a.sentence)],
100 todo: [...v.todo, ...heavy.map((a) => a.sentence)],
101 degraded: [...v.degraded, ...heavy.map((a) => ({ cause: `asking:${a.token}`, sentence: a.sentence, since, href: "/system#asking" }))],
102 };
84103 }
85104
86105 /**
src/apps/runtime.ts +15 −4
@@ -35,7 +35,8 @@ import { neededNames, resolveAppSecrets } from "../core/app_secrets.ts";
3535 import { featureVarsAt } from "./feature_env.ts";
3636 import { getVariant, listVariants, variantOfBranch } from "../core/variants.ts";
3737 import { readVariantValues } from "./variant_file.ts";
38 import { mintAppToken, renewAppToken, revokeAppToken } from "./token.ts";
38 import { appTokenName, mintAppToken, renewAppToken, revokeAppToken } from "./token.ts";
39 import { asking } from "../perf/asking.ts";
3940 import type { ForgeEvent } from "../core/events.ts";
4041 import { announceRun } from "../actions/runs.ts";
4142 import { requeueAbandonedChecks } from "../checks/service.ts";
@@ -218,6 +219,9 @@ export type EnvState =
218219 | "asleep"
219220 | "failed";
220221
222 /** How long an always-on app may go unopened while it keeps calling the forge. */
223 export const UNOPENED_MS = 24 * 60 * 60_000;
224
221225 /** Idle windows before an environment sleeps: the profit slider's 50 (`core/slider.ts`). */
222226 export const SLEEP_AFTER_MS = { production: 24 * 3_600_000, preview: 30 * 60_000 };
223227 /** Days a quiet pull keeps its preview by default: the slider's 50 too. */
@@ -2299,9 +2303,16 @@ export class AppsRuntime {
22992303 state.env === "production" || envVariant(state.env) !== undefined,
23002304 live.manifest.alwaysOn === true,
23012305 );
2302 const limit = window.ms;
2303 if (limit === null) continue;
23042306 const last = state.lastRequestAt ?? Date.parse(live.since);
2307 // An always-on app nobody has opened for a day, still calling the forge, is idle like any
2308 // other: its own calls are not a visit (`touch` is the proxy's), and they do not keep it
2309 // awake (#12265).
2310 const limit =
2311 window.ms ??
2312 (now - last >= UNOPENED_MS && asking.calling(appTokenName(state.app.slug, state.env), 60 * 60_000, now)
2313 ? UNOPENED_MS
2314 : null);
2315 if (limit === null) continue;
23052316 if (now - last < limit) continue;
23062317 // Idle by the clock; asked whether it is mid-work, and spared while
23072318 // it is — but only for one grace past the window, never forever.
@@ -2335,7 +2346,7 @@ export class AppsRuntime {
23352346 this.app.config.dataDir,
23362347 live.runId,
23372348 busy === undefined
2338 ? asleepWords({ ms: limit, why: window.why })
2349 ? asleepWords({ ms: limit, why: window.ms === null ? "always_on, but nobody opened it and it kept calling the forge" : window.why })
23392350 : `asleep: still busy ${Math.round(grace / 60_000)} minutes past its idle window, which is as long as busy keeps an app awake; the next request wakes it`,
23402351 );
23412352 log.info("asleep", { app: state.app.slug, env: state.env });
src/http/app_turn.ts added +85
@@ -0,0 +1,85 @@
1 /**
2 * An app's turn on the serving thread (#12265): people first, never a cap.
3 *
4 * Requests are served in the forge's ladder: a person's page or form, live
5 * updates, git, an agent's call a person waits on, then an app's own token.
6 * One thread serves them all, so the last rung waits here, after its token is
7 * known and before its route runs, while the thread is wanted by anyone above
8 * it: a person's request begun in the last `YOUNG_MS` and still open, or one
9 * answered in the last `BURST_MS` (a page is a burst, and the next request is
10 * on its way). It looks again every `LOOK_MS` and goes when the thread is free
11 * or when it has waited `BOUND_MS`: deferred, never refused, never starved.
12 * With nobody else asking it goes at once, without a timer.
13 *
14 * Streams (a long poll, `/events`) are older than `YOUNG_MS` and hold a
15 * connection, not the thread. Plain data, one object per process.
16 */
17
18 export const YOUNG_MS = 1_000;
19 export const BURST_MS = 50;
20 export const LOOK_MS = 20;
21 export const BOUND_MS = 1_000;
22
23 export interface Turn {
24 /** A request arrived: it is a person's until it says otherwise. */
25 begin(key: object, at?: number): void;
26 /** Auth showed it to be an app's: it stops counting as a person's. */
27 machine(key: object): void;
28 /** It is answered. */
29 end(key: object, at?: number): void;
30 /** Resolve at once when nobody above is waiting, else when they are done or after the bound. */
31 wait(): Promise<void>;
32 /** Totals since the process began, for /system. */
33 readonly stats: { waited: number; bounded: number; longestMs: number };
34 }
35
36 const sleepReal = (ms: number): Promise<void> => new Promise((r) => setTimeout(r, ms));
37
38 export function createTurn(
39 now: () => number = () => performance.now(),
40 bound = BOUND_MS,
41 sleep: (ms: number) => Promise<void> = sleepReal,
42 ): Turn {
43 const open = new Map<object, number>();
44 let lastEnd = -Infinity;
45 const stats = { waited: 0, bounded: 0, longestMs: 0 };
46 const wanted = (): boolean => {
47 const t = now();
48 if (t - lastEnd < BURST_MS) return true;
49 let young = false;
50 // A request nobody ended (a response that left by another door) is forgotten once it is old.
51 for (const [key, at] of open) {
52 if (t - at < YOUNG_MS) young = true;
53 else open.delete(key);
54 }
55 return young;
56 };
57 return {
58 begin(key, at = now()) {
59 open.set(key, at);
60 },
61 machine(key) {
62 open.delete(key);
63 },
64 end(key, at = now()) {
65 if (open.delete(key)) lastEnd = at;
66 },
67 async wait() {
68 if (!wanted()) return;
69 const since = now();
70 stats.waited++;
71 while (wanted()) {
72 if (now() - since >= bound) {
73 stats.bounded++;
74 break;
75 }
76 await sleep(LOOK_MS);
77 }
78 stats.longestMs = Math.max(stats.longestMs, now() - since);
79 },
80 stats,
81 };
82 }
83
84 /** The forge's one gate: a hosted child answers on the same thread. */
85 export const turn: Turn = createTurn();
src/http/repeat_reads.ts added +73
@@ -0,0 +1,73 @@
1 /**
2 * Repeat reads are free (#12265): an app token that asks again for what it
3 * holds is answered 304 from the write clock, before the route runs a query.
4 *
5 * After a 200 that carries an ETag, the answer's tag is kept under the
6 * token, the user, the scope and the full URL, with the write clock as it was
7 * read BEFORE the route ran (`core/write_clock.ts`). A later GET with that
8 * tag in `If-None-Match` while the clock has not moved (and the entry is
9 * under `MAX_AGE_MS` old, which bounds anything the clock cannot see, such as
10 * telemetry) is the same answer: 304 with the same headers, no handler.
11 * Access and token changes are writes, so they move the clock and end it.
12 *
13 * Plain data, bounded to `MAX_ENTRIES` per forge, oldest dropped.
14 */
15
16 import type { Database } from "bun:sqlite";
17 import { holds } from "./etag.ts";
18
19 export const MAX_AGE_MS = 30_000;
20 export const MAX_ENTRIES = 512;
21
22 interface Kept {
23 stamp: string;
24 at: number;
25 etag: string;
26 headers: Record<string, string>;
27 }
28
29 const kept = new WeakMap<Database, Map<string, Kept>>();
30
31 /**
32 * What the same asker is asking for: one key per token, account, scope, URL and
33 * representation (`Accept`). The token is held as its hash, never as the secret.
outdated

Raw bearer token is the map key; hash it. Add the Accept header (or Vary values) to the key.

34 */
35 export const repeatKey = (credential: string, userId: string, scope: string, url: URL, accept: string | null = null): string =>
36 `${new Bun.CryptoHasher("sha256").update(credential).digest("hex")}|${userId}|${scope}|${accept ?? ""}|${url.pathname}${url.search}`;
37
38 /** The 304 this asker is owed, or undefined when the route must run. */
39 export function repeatRead(
40 db: Database,
41 key: string,
42 request: Request,
43 stamp: string | undefined,
44 now: number = Date.now(),
45 ): Response | undefined {
46 if (stamp === undefined || request.method !== "GET") return undefined;
47 const held = kept.get(db)?.get(key);
48 if (held === undefined || held.stamp !== stamp || now - held.at >= MAX_AGE_MS) return undefined;
49 if (!holds(request, held.etag)) return undefined;
50 return new Response(null, { status: 304, headers: held.headers });
51 }
52
53 /** Keep a 200's tag against the clock read before its route ran. */
54 export function keepRead(
55 db: Database,
56 key: string,
57 stamp: string | undefined,
58 response: Response,
59 now: number = Date.now(),
60 ): void {
61 const etag = response.headers.get("etag");
62 if (stamp === undefined || etag === null || response.status !== 200) return;
63 let map = kept.get(db);
64 if (map === undefined) kept.set(db, (map = new Map()));
65 map.delete(key);
66 const headers: Record<string, string> = { etag };
67 for (const name of ["cache-control", "vary"]) {
68 const value = response.headers.get(name);
69 if (value !== null) headers[name] = value;
70 }
71 map.set(key, { stamp, at: now, etag, headers });
72 if (map.size > MAX_ENTRIES) map.delete(map.keys().next().value as string);
73 }
src/http/server.ts +24
@@ -572,6 +572,10 @@ import { createReadWriter, readOf } from "../perf/reads.ts";
572572 import { parseServerTiming } from "../perf/record.ts";
573573 import { currentLoopLag, LAG_WINDOW_MS, startLoopWorst } from "../perf/loop_worst.ts";
574574 import { methodOf, noteUsage, roleOf, surfaceOf, takeUsage } from "./usage_tag.ts";
575 import { asking, noteAsker, noteWaited, takeAsker } from "../perf/asking.ts";
576 import { turn } from "./app_turn.ts";
577 import { keepRead, repeatKey, repeatRead } from "./repeat_reads.ts";
578 import { writeStamp } from "../core/write_clock.ts";
575579 import { bearerChallenge, oauthRoutes } from "./oauth.ts";
576580 import { flushAccess, recordAccess } from "./access_log.ts";
577581 import { carryPeer, clientAddress, notePeer, wasInternal, wasViaEdge, type ProxyTrust } from "./peer.ts";
@@ -2695,6 +2699,9 @@ export async function serve(app: App): Promise<Servers> {
26952699 ): void => {
26962700 const tag = takeUsage(request);
26972701 const ms = performance.now() - started;
2702 turn.end(request);
2703 const asker = takeAsker(request);
2704 asking.note(asker?.token, tag.pattern, Math.max(0, ms - (asker?.waitedMs ?? 0)));
26982705 finishedStrangerDeep(owner, request);
26992706 try {
27002707 const header = serverTiming(ms, takeDbTime(request), takePhases(request));
@@ -2802,6 +2809,7 @@ export async function serve(app: App): Promise<Servers> {
28022809 ): Promise<Response | undefined> => {
28032810 const started = performance.now();
28042811 noteStart(request, started);
2812 turn.begin(request, started);
28052813 // A read-only standby refuses every mutation here, before any route.
28062814 if (app.config.readOnly === true) {
28072815 const refused = readOnlyRefusal(request);
@@ -3160,6 +3168,7 @@ async function answerGet(
31603168 noteSessionTime(request, performance.now() - authStart);
31613169 if (auth.retryAfter !== undefined) return tooManyAttempts(auth.retryAfter);
31623170 noteUsage(request, { pattern: matched.pattern, role: roleOf(auth) });
3171 if (auth.tokenName !== undefined) noteAsker(request, auth.tokenName);
31633172 // From here a body's cap depends on who is asking (`http/limits.ts`), and
31643173 // a stream this request opens is the account's, to end with it.
31653174 if (auth.user !== undefined) {
@@ -3296,6 +3305,20 @@ async function answerGet(
32963305 door: auth.tokenName === undefined ? auth.via : `token:${auth.tokenName}`,
32973306 ...(auth.through === undefined ? {} : { via: auth.through }),
32983307 };
3308 // An app's own token goes last on the ladder (#12265): a repeat of what it
3309 // holds is answered 304 from the write clock at once; anything else waits its
3310 // turn behind people's requests, for a bound, then runs.
3311 const appAsks = auth.tokenName?.startsWith("app:") === true && auth.user !== undefined && credential !== undefined;
3312 const repeat = appAsks ? repeatKey(credential!, auth.user!.id, auth.scope ?? "ro", url, request.headers.get("accept")) : undefined;
3313 const stamp = repeat === undefined ? undefined : writeStamp(app.db);
3314 if (repeat !== undefined) {
3315 const same = repeatRead(app.db, repeat, request, stamp);
3316 if (same !== undefined) return withRateHeaders(same, verdict);
3317 turn.machine(request);
3318 const waitFrom = performance.now();
3319 await turn.wait();
3320 noteWaited(request, performance.now() - waitFrom);
3321 }
32993322 // And it reads as its caller: in a fork whose parent that caller may not
33003323 // read, what git is asked for by id is held to the fork's own refs, at every
33013324 // handler's reads and at no handler's cost (`git/own_refs.ts`).
@@ -3306,6 +3329,7 @@ async function answerGet(
33063329 : withIdempotency(app, request, url, replayAs, invoke),
33073330 ),
33083331 );
3332 if (repeat !== undefined) keepRead(app.db, repeat, stamp, response);
33093333 // A page route's HTML 404 (`/:owner` claims `/nope`) is a refusal an
33103334 // agent cannot read; one that asked for JSON gets it as JSON.
33113335 if (
src/perf/asking.ts added +179
@@ -0,0 +1,179 @@
1 /**
2 * Who is asking the forge for how much, over the last hour (#12265).
3 *
4 * Every response is noted once: its route and the milliseconds it held the
5 * serving thread, under the name of the token that asked (a person's page
6 * has none, and only adds to the totals). Minute buckets, 61 kept, at most
7 * `NAMES` names a bucket, so the memory is fixed however many tokens ask.
8 *
9 * `heavy()` names any one token that made more than a fifth of the hour's
10 * requests, or of the thread's time, with the route it asked most and how
11 * often: "foundry made 63% of requests this hour: every 2.5 s to /runs".
12 * `calling()` says whether a token asked at all lately, which is what lets
13 * the runtime stop treating an app nobody opened as awake for its own calls.
14 *
15 * One map per process, gone on restart: it is a reading, not state.
16 */
17
18 /** The share (of requests or of the thread's time) above which a token is named. */
19 export const HEAVY_SHARE = 0.2;
20 /** Below this many requests an hour in all, nobody is named: a quiet forge has no heavy asker. */
21 export const MIN_TOTAL = 200;
22 /** A token must have asked at least this often to be named. */
23 export const MIN_OWN = 60;
24 const MINUTE = 60_000;
25 const KEPT = 61;
26 const NAMES = 64;
27 const ROUTES = 8;
28
29 export interface Asker {
30 /** The token's name as its owner gave it (`app:foundry/production`). */
31 readonly token: string;
32 /** What to call it: the app's name for an app token. */
33 readonly name: string;
34 readonly app: boolean;
35 readonly requests: number;
36 /** The larger of its share of requests and of the thread's time. */
37 readonly share: number;
38 /** The route it asked most, with the app's prefix left off. */
39 readonly route: string;
40 /** Milliseconds between its asks of that route, or null when too few to say. */
41 readonly everyMs: number | null;
42 readonly sentence: string;
43 }
44
45 interface Own {
46 first: number;
47 last: number;
48 n: number;
49 ms: number;
50 routes: Map<string, number>;
51 }
52 interface Bucket {
53 minute: number;
54 n: number;
55 ms: number;
56 by: Map<string, Own>;
57 }
58
59 /** `app:foundry/production` is the app `foundry`; any other token is its own name. */
60 export function askerName(token: string): { name: string; app: boolean } {
61 const m = /^app:([^/]+)\//.exec(token);
62 const name = m === null ? token : m[1]!;
63 return { name: name.length > 40 ? `${name.slice(0, 39)}…` : name, app: m !== null };
64 }
65
66 /** "2.5 s", "40 ms", "3 min": the gap as a person says it. */
67 export function gapWords(ms: number): string {
68 if (ms < 1_000) return `${Math.max(1, Math.round(ms))} ms`;
69 if (ms < 60_000) return `${Number((ms / 1_000).toFixed(1))} s`;
70 return `${Number((ms / 60_000).toFixed(1))} min`;
71 }
72
73 const shortRoute = (pattern: string): string => pattern.replace(/^\/api\/v1\/repos\/:owner\/:name/, "") || pattern;
74
75 export interface Asking {
76 /** One response: who asked (a token's name, if any), which route, how long it held the thread. */
77 note(token: string | undefined, pattern: string, ms: number, now?: number): void;
78 heavy(now?: number): Asker[];
79 /** Whether `token` asked at all in the last `withinMs`. */
80 calling(token: string, withinMs: number, now?: number): boolean;
81 }
82
83 export function createAsking(limits: { minTotal?: number; minOwn?: number } = {}): Asking {
84 let buckets: Bucket[] = [];
85 const bucketAt = (now: number): Bucket => {
86 const minute = Math.floor(now / MINUTE);
87 const last = buckets[buckets.length - 1];
88 if (last !== undefined && last.minute === minute) return last;
89 const fresh: Bucket = { minute, n: 0, ms: 0, by: new Map() };
90 buckets.push(fresh);
91 if (buckets.length > KEPT) buckets = buckets.slice(-KEPT);
92 return fresh;
93 };
94 return {
95 note(token, pattern, ms, now = Date.now()) {
96 const b = bucketAt(now);
97 b.n++;
98 b.ms += ms;
99 if (token === undefined) return;
100 let own = b.by.get(token);
101 if (own === undefined) {
102 if (b.by.size >= NAMES) return;
103 own = { first: now, last: now, n: 0, ms: 0, routes: new Map() };
104 b.by.set(token, own);
105 }
106 own.n++;
107 own.last = now;
108 own.ms += ms;
109 if (own.routes.has(pattern) || own.routes.size < ROUTES) own.routes.set(pattern, (own.routes.get(pattern) ?? 0) + 1);
110 },
111 heavy(now = Date.now()) {
112 const from = Math.floor(now / MINUTE) - (KEPT - 1);
113 const live = buckets.filter((b) => b.minute >= from);
114 let total = 0;
115 let totalMs = 0;
116 const sum = new Map<string, { n: number; ms: number; routes: Map<string, number>; first: number; last: number }>();
117 for (const b of live) {
118 total += b.n;
119 totalMs += b.ms;
120 for (const [token, own] of b.by) {
121 let s = sum.get(token);
122 if (s === undefined) sum.set(token, (s = { n: 0, ms: 0, routes: new Map(), first: own.first, last: own.last }));
123 s.n += own.n;
124 s.ms += own.ms;
125 s.first = Math.min(s.first, own.first);
126 s.last = Math.max(s.last, own.last);
127 for (const [r, c] of own.routes) s.routes.set(r, (s.routes.get(r) ?? 0) + c);
128 }
129 }
130 const minTotal = limits.minTotal ?? MIN_TOTAL;
131 const minOwn = limits.minOwn ?? MIN_OWN;
132 if (total < minTotal) return [];
133 const out: Asker[] = [];
134 for (const [token, s] of sum) {
135 const share = s.n / total;
136 const timeShare = totalMs > 0 ? s.ms / totalMs : 0;
137 if (s.n < minOwn || (share <= HEAVY_SHARE && timeShare <= HEAVY_SHARE)) continue;
138 const [route, count] = [...s.routes].sort((a, b) => b[1] - a[1])[0] ?? ["", 0];
139 const everyMs = count >= 10 ? (s.last - s.first) / (count - 1) : null;
140 const who = askerName(token);
141 const pct = Math.round(Math.max(share, timeShare) * 100);
142 const of = share >= timeShare ? "requests" : "the serving thread's time";
143 const sentence =
144 `${who.name} made ${pct}% of ${of} this hour` +
145 (route === "" ? "" : `: ${everyMs === null ? "mostly" : `every ${gapWords(everyMs)}`} to ${shortRoute(route)}`) +
146 (who.app ? ". Open its page, or put it to sleep." : ". Revoke the token if nobody means it to ask this much.");
147 out.push({ token, ...who, requests: s.n, share: Math.max(share, timeShare), route: shortRoute(route), everyMs, sentence });
148 }
149 return out.sort((a, b) => b.share - a.share);
150 },
151 calling(token, withinMs, now = Date.now()) {
152 const from = Math.floor((now - withinMs) / MINUTE);
153 return buckets.some((b) => b.minute >= from && b.by.has(token));
154 },
155 };
156 }
157
158 /** The forge's one reading. */
159 export const asking: Asking = createAsking();
160
161 const askedBy = new WeakMap<Request, { token: string; waitedMs: number }>();
162
163 /** Auth showed this request to carry a token: `counted` notes it under that name. */
164 export function noteAsker(request: Request, token: string): void {
165 askedBy.set(request, { token, waitedMs: 0 });
166 }
167
168 /** The request waited its turn this long: not the thread's time, so it is left out of its share. */
169 export function noteWaited(request: Request, ms: number): void {
170 const held = askedBy.get(request);
171 if (held !== undefined) held.waitedMs = ms;
172 }
173
174 /** The token that asked and the wait to leave out, once, when the response leaves. */
175 export function takeAsker(request: Request): { token: string; waitedMs: number } | undefined {
176 const held = askedBy.get(request);
177 askedBy.delete(request);
178 return held;
179 }
src/tui/screens/system.ts +20
@@ -36,6 +36,8 @@ import type { Ctx, Screen } from "../app/app.ts";
3636 import type { SystemView } from "../../system/view.ts";
3737 import { rolloutLine, type ForgeHealth } from "../../kernel/guard.ts";
3838 import { canSeeMemory, memoryHours } from "../../boards/memory.ts";
39 import { asking } from "../../perf/asking.ts";
40 import { turn } from "../../http/app_turn.ts";
3941 import { telemetryReader } from "../../actions/telemetry.ts";
4042 import { historyOver } from "../../system/telemetry.ts";
4143 import { binding } from "../app/keys.ts";
@@ -418,6 +420,22 @@ export function memoryLines(session: Session, now = Date.now()): string[] {
418420 ];
419421 }
420422
423 /**
424 * "Who is asking" as /system says it, for the owner alone (`canSeeMemory`); empty for anyone
425 * else, or when nothing is heavy and nothing has waited.
426 */
427 export function askingLines(session: Session, inner = WIDTH - 6): string[] {
428 if (!canSeeMemory(session.user)) return [];
429 const { waited, bounded, longestMs } = turn.stats;
430 const askers = asking.heavy();
431 if (askers.length === 0 && waited === 0) return [];
432 return [
433 "Who is asking",
434 ...askers.map((a) => truncate(` ${a.sentence}`, inner)),
435 ...(waited === 0 ? [] : [truncate(` ${waited} app calls waited behind people's, ${bounded} to the bound (${Math.round(longestMs)} ms the longest).`, inner)]),
436 ];
437 }
438
421439 export function memoryScreen(session: Session, ctx: Ctx): Screen {
422440 const scroll = new Scroll();
423441 return {
@@ -497,9 +515,11 @@ export const entry: PaletteEntry = {
497515 run: (session, ctx) => {
498516 const self = selfLines(selfUse(realSelf(session.app.config.dataDir)));
499517 const kids = childLines(childrenUse(listChildrenNow(session.app), (bodyId) => childDirOf(session.app, bodyId)));
518 const who = askingLines(session);
500519 ctx.push(systemScreen(ctx, [
501520 ...forgeLines(readHealth(session.app, session.user)),
502521 ...egressLines(session),
522 ...(who.length === 0 ? [] : ["", ...who]),
503523 ...engineLines(session.app.config.engine),
504524 ...(self.length === 0 ? [] : ["", ...self]),
505525 ...(kids.length === 0 ? [] : ["", ...kids]),
src/web/pages/system.ts +21
@@ -21,6 +21,8 @@
2121 * line each for memory, CPU and disk; a parent lists each child's use
2222 * against its size (#10853, `spawn/usage.ts`).
2323 */
24 import { asking, type Asker } from "../../perf/asking.ts";
25 import { turn } from "../../http/app_turn.ts";
2426 import { mayReadRefusals } from "../../core/first_ask_refusals.ts";
2527 import { canSeeMemory, memoryHours } from "../../boards/memory.ts";
2628 import { telemetryReader } from "../../actions/telemetry.ts";
@@ -391,6 +393,24 @@ export function stallSection(rows: readonly RouteStall[], worst: Html = html``):
391393 </table></div>`}${worst}</section>`;
392394 }
393395
396 /**
397 * Who asked the forge for the most this hour (#12265): any one token over a fifth of the
398 * requests or of the thread's time, named with what to do; and how many app requests waited
399 * behind people's. Nothing drawn when nobody is over a fifth and nothing has waited.
400 */
401 export function askingSection(askers: readonly Asker[], waits: { waited: number; bounded: number; longestMs: number }): Html {
402 if (askers.length === 0 && waits.waited === 0) return html``;
403 return html`<section class="settings-block" id="asking"><h2>Who is asking</h2>${
404 askers.length === 0
405 ? html``
406 : html`<ul>${askers.map((a) => html`<li data-asker="${a.name}">${a.sentence}</li>`)}</ul>`
407 }${
408 waits.waited === 0
409 ? html``
410 : html`<p class="small muted">Apps' own calls go behind people's: ${String(waits.waited)} waited their turn since the forge started, ${String(waits.bounded)} to the bound (${String(Math.round(waits.longestMs))} ms the longest). Nothing is refused.</p>`
411 }</section>`;
412 }
413
394414 /** How many of the last hour's worst stalls the page lists (#11069); the terminal lists as many. */
395415 export { WORST_STALLS };
396416
@@ -617,6 +637,7 @@ export async function systemPage(ctx: Ctx): Promise<Response> {
617637 ${featuresSection(timePhase(ctx.request, "features", () => featureHealth(ctx.app, reader)))}
618638 ${lagSection(lagMinutes.readings())}
619639 <div id="sys-stalls">${stallSection(stalls, worstStallsTable(worst, nowMono, Date.now()))}</div>
640 ${owner ? askingSection(asking.heavy(), turn.stats) : html``}
620641 <div id="sys-slow-pages">${slowPagesSection(timePhase(ctx.request, "slowpages", () => slowPagesCached(ctx.app.db)))}</div>
621642 <div id="sys-readers">${readersSection(readerPaces())}</div>
622643 <div id="sys-caches">${cachesSection(pageCaches())}</div>
templates/app-template.json +1 −1

Diff not shown — a line 15,877 characters wide (limit 2,000). +1 −1

View file
test/app-turn.test.ts added +186
@@ -0,0 +1,186 @@
1 /**
2 * An app cannot stall the forge (#12265): its requests wait behind people's,
3 * repeat reads cost nothing, and the forge names an app that asks too much.
4 */
5 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
6 import { mkdtemp, rm } from "node:fs/promises";
7 import { tmpdir } from "node:os";
8 import { join } from "node:path";
9 import { createApp, type App } from "../src/app.ts";
10 import { createRepo } from "../src/actions/repos.ts";
11 import { openIssue } from "../src/actions/issues.ts";
12 import { forgeVitals } from "../src/actions/health.ts";
13 import { getRepo } from "../src/core/repos.ts";
14 import { createToken } from "../src/core/tokens.ts";
15 import { createUser } from "../src/core/users.ts";
16 import { serve, type Servers } from "../src/http/server.ts";
17 import { createTurn } from "../src/http/app_turn.ts";
18 import { asking, createAsking, gapWords } from "../src/perf/asking.ts";
19
20 describe("the turn", () => {
21 test("nobody else asking: served at once; a person's request open: waits, to a bound, never refused", async () => {
22 let t = 0;
23 const turn = createTurn(() => t, 100, async (ms) => void (t += ms));
24 await turn.wait();
25 expect(turn.stats.waited).toBe(0);
26 const person = {};
27 turn.begin(person, t);
28 // A person still being answered: the wait ends at the bound, on the fake clock.
29 await turn.wait();
30 expect(turn.stats.waited).toBe(1);
31 expect(turn.stats.bounded).toBe(1);
32 expect(turn.stats.longestMs).toBe(100);
33 // Answered: a burst follows, then free.
34 turn.end(person, t);
35 await turn.wait();
36 expect(turn.stats.bounded).toBe(1);
37 expect(turn.stats.waited).toBe(2);
38 t += 1_000;
39 await turn.wait();
40 expect(turn.stats.waited).toBe(2);
41 });
42
43 test("a person answered while the app waits lets it go before the bound", async () => {
44 let t = 0;
45 const person = {};
46 const turn = createTurn(() => t, 1_000, async (ms) => {
47 t += ms;
48 if (t >= 40) turn.end(person, t);
49 });
50 turn.begin(person, t);
51 await turn.wait();
52 expect(turn.stats.bounded).toBe(0);
53 expect(turn.stats.longestMs).toBeLessThan(1_000);
54 });
55
56 test("an app's own request is no one's reason to wait", async () => {
57 let t = 0;
58 const turn = createTurn(() => t, 100);
59 const app = {};
60 turn.begin(app, t);
61 turn.machine(app);
62 await turn.wait();
63 expect(turn.stats.waited).toBe(0);
64 });
65
66 test("a person's request older than a second holds a connection, not the thread", async () => {
67 let t = 0;
68 const turn = createTurn(() => t, 100);
69 turn.begin({}, 0);
70 t = 1_500;
71 await turn.wait();
72 expect(turn.stats.waited).toBe(0);
73 });
74 });
75
76 describe("who is asking", () => {
77 test("a token over a fifth of the hour's requests is named, with how often and where", () => {
78 const a = createAsking({ minTotal: 100, minOwn: 20 });
79 const t0 = 1_000_000_000_000;
80 for (let i = 0; i < 60; i++) a.note("app:foundry/production", "/api/v1/repos/:owner/:name/runs", 2, t0 + i * 2_500);
81 for (let i = 0; i < 40; i++) a.note(undefined, "/", 5, t0 + i * 3_000);
82 const [heavy] = a.heavy(t0 + 150_000);
83 expect(heavy?.name).toBe("foundry");
84 expect(heavy?.share).toBeCloseTo(0.6, 1);
85 expect(heavy?.sentence).toContain("foundry made 60% of requests this hour: every 2.5 s to /runs");
86 expect(heavy?.sentence).toContain("Open its page, or put it to sleep");
87 expect(a.calling("app:foundry/production", 60_000, t0 + 150_000)).toBe(true);
88 expect(a.calling("app:foundry/production", 60_000, t0 + 7_200_000)).toBe(false);
89 expect(a.heavy(t0 + 7_200_000)).toEqual([]);
90 });
91
92 test("a quiet forge, or a token under a fifth, names nobody", () => {
93 const a = createAsking({ minTotal: 100, minOwn: 5 });
94 for (let i = 0; i < 10; i++) a.note("app:a/production", "/x", 1, 5_000 + i);
95 expect(a.heavy(10_000)).toEqual([]);
96 for (let i = 0; i < 100; i++) a.note(undefined, "/", 1, 6_000 + i);
97 expect(a.heavy(10_000)).toEqual([]);
98 });
99
100 test("gaps are said as a person says them", () => {
101 expect(gapWords(2_500)).toBe("2.5 s");
102 expect(gapWords(40)).toBe("40 ms");
103 expect(gapWords(180_000)).toBe("3 min");
104 });
105 });
106
107 describe("the forge answers an app token", () => {
108 let app: App;
109 let servers: Servers;
110 let dir: string;
111 let secret: string;
112 const mine = (path: string, extra: Record<string, string> = {}) =>
113 fetch(servers.url + path, { headers: { authorization: `Bearer ${secret}`, ...extra } });
114
115 beforeAll(async () => {
116 dir = await mkdtemp(join(tmpdir(), "forge-turn-"));
117 app = createApp({ dataDir: dir, port: 0 });
118 servers = await serve(app);
119 const made = await createUser(app.db, { origin: "seed", trusted: true, username: "alice", email: "[email protected]", password: "a-good-passphrase", adminIfFirst: true });
120 if (!made.ok) throw new Error(made.error);
121 const repo = await createRepo(app, made.value, { name: "yard", visibility: "private" });
122 if (!repo.ok) throw new Error(String(repo.error));
123 secret = createToken(app.db, made.value.id, "app:poller/production", "ro").token;
124 (globalThis as { __alice?: unknown }).__alice = made.value;
125 });
126 afterAll(async () => {
127 await servers.stop();
128 await rm(dir, { recursive: true, force: true });
129 });
130
131 test("a repeat read is a 304 from the write clock, without its query; a write ends it", async () => {
132 const first = await mine("/api/v1/repos/alice/yard/runs");
133 expect(first.status).toBe(200);
134 const etag = first.headers.get("etag")!;
135 expect(etag).toBeTruthy();
136 await first.text();
137 // The route's own 304 ran its tag; the clock's answer ran nothing.
138 const warm = await mine("/api/v1/repos/alice/yard/runs", { "if-none-match": etag });
139 expect(warm.status).toBe(304);
140 const again = await mine("/api/v1/repos/alice/yard/runs", { "if-none-match": etag });
141 expect(again.status).toBe(304);
142 expect(again.headers.get("server-timing") ?? "").not.toContain("tag");
143 expect(again.headers.get("etag")).toBe(etag);
144 // Anything written moves the clock: the route runs again.
145 const alice = (globalThis as { __alice?: never }).__alice!;
146 const issue = await openIssue(app, alice, getRepo(app.db, "alice", "yard")!, { title: "x", body: "" });
147 expect(issue.ok).toBe(true);
148 const after = await mine("/api/v1/repos/alice/yard/runs", { "if-none-match": etag });
149 expect(after.headers.get("server-timing") ?? "").toContain("tag");
150 });
151
152 test("a stranger's token is not given another's tag", async () => {
153 const first = await mine("/api/v1/repos/alice/yard/runs");
154 const etag = first.headers.get("etag")!;
155 const anon = await fetch(servers.url + "/api/v1/repos/alice/yard/runs", { headers: { "if-none-match": etag } });
156 expect(anon.status).toBe(404);
157 });
158
159 test("a token over a fifth of the hour is a need of the forge's vitals, named with what to do", () => {
160 for (let i = 0; i < 300; i++) asking.note("app:poller/production", "/api/v1/repos/:owner/:name/runs", 1);
161 const line = forgeVitals(app).degraded.find((d) => d.cause.startsWith("asking:"));
162 expect(line?.sentence).toContain("poller made");
163 expect(line?.sentence).toContain("to /runs");
164 expect(line?.sentence).toContain("put it to sleep");
165 });
166
167 test("the same URL asked for in another representation is not given the first's 304", async () => {
168 const first = await mine("/api/v1/repos/alice/yard/runs?limit=7", { accept: "application/json" });
169 const etag = first.headers.get("etag")!;
170 await first.text();
171 const same = await mine("/api/v1/repos/alice/yard/runs?limit=7", { accept: "application/json", "if-none-match": etag });
172 expect(same.status).toBe(304);
173 const other = await mine("/api/v1/repos/alice/yard/runs?limit=7", { accept: "text/plain", "if-none-match": etag });
174 expect(other.headers.get("server-timing") ?? "").toContain("tag");
175 });
176
177 test("/system names who is asking to the owner only", async () => {
178 const bob = await createUser(app.db, { origin: "seed", trusted: true, username: "bob", email: "[email protected]", password: "a-good-passphrase" });
179 if (!bob.ok) throw new Error(bob.error);
180 const bobToken = createToken(app.db, bob.value.id, "bob-cli", "ro").token;
181 const page = (token: string) => fetch(servers.url + "/system", { headers: { authorization: `Bearer ${token}` } }).then((r) => r.text());
182 expect(await page(secret)).toContain('data-asker="poller"');
183 expect(await page(bobToken)).not.toContain("Who is asking");
184 expect(await page(bobToken)).not.toContain("poller");
185 });
186 });
test/apps-sleep-slider.test.ts +13
@@ -17,6 +17,7 @@ import type { Repo } from "../src/core/repos.ts";
1717 import { listRuns, type Run } from "../src/core/runs.ts";
1818 import { createUser, type User } from "../src/core/users.ts";
1919 import { serve, type Servers } from "../src/http/server.ts";
20 import { asking } from "../src/perf/asking.ts";
2021 import { readRunLog } from "../src/runs/log.ts";
2122 import { setEconomics } from "../src/settings/economics.ts";
2223 import { createForgeApp } from "../src/tui/forge.ts";
@@ -142,4 +143,16 @@ describe("sleep windows follow the slider", () => {
142143 tui.app.stop();
143144 expect(text).toContain("never sleeps (app.always_on)");
144145 }, 120_000);
146 test("an always-on app nobody opened for a day sleeps once it is seen calling the forge, and not before", async () => {
147 ahead = 0;
148 expect((await viaHost(`hello.${DOMAIN}`)).status).toBe(200);
149 expect(production()?.alwaysOn).toBe(true);
150 ahead = 25 * 60 * 60_000;
151 await Bun.sleep(600); // real-timer: the sleeper loop must get its turn
152 // Unopened for a day, but quiet toward the forge: always-on keeps it.
153 expect(production()?.state).toBe("running");
154 asking.note("app:hello/production", "/api/v1/repos/:owner/:name/runs", 1, Date.now() + ahead);
155 await until(() => (production()?.state === "asleep" ? true : undefined), 5_000);
156 expect(log(production()!.runId!)).toContain("asleep: nothing asked for it in 24 h (always_on, but nobody opened it and it kept calling the forge)");
157 }, 30_000);
145158 });
test/timers-named.test.ts +8 −2
@@ -74,6 +74,7 @@ const ALLOWED: readonly Allowed[] = [
7474 ...allow("setTimeout", `a shell session's resize debounce, ${PER_CONNECTION}`, [["src/machines/shell.ts", "() => session.size === resize"]]),
7575 ...allow("setTimeout", `an ssh login's auth deadline, ${PER_CONNECTION}`, [["src/ssh/server.ts", "() => { this.transport.disconnect("]]),
7676 ...allow("setTimeout", "a maintenance duty's deadline on one call: it rejects a promise and does no work on the loop", [["src/maintain/runner.ts", "() => reject(new Error(\"timed out\"))"]]),
77 ...allow("setTimeout", "an app request's look-again sleep in the ladder: it holds one request, does no work on the loop, and ends within the bound", [["src/http/app_turn.ts", "r"]]),
7778 ...allow("setTimeout", `a media stream's idle deadline, ${PER_CONNECTION}`, [["src/git/read.ts", "() => { finish(); controller.error("]]),
7879 ...allow("setTimeout", `a git upload's idle deadline, ${PER_CONNECTION}`, [["src/git/smart-http.ts", "stop"]]),
7980 ...allow("setTimeout", "the loop-lag meter's own tick: it measures how late the loop runs, and a span around it would count that lateness as its own work", [["src/kernel/loop_lag.ts", "() => { // Early"]]),
@@ -179,7 +180,11 @@ function sources(dir: string, out: string[] = []): string[] {
179180
180181 /** The first argument of each `call(` in `text`: its text, whitespace collapsed, and whether its work runs inside `during`. */
181182 export function callbacks(text: string, call: Call): { callback: string; named: boolean; line: number }[] {
182 const source = ts.createSourceFile("x.ts", text, ts.ScriptTarget.Latest, true);
183 return callbacksIn(ts.createSourceFile("x.ts", text, ts.ScriptTarget.Latest, false), call);
184 }
185
186 /** The same, from a file already parsed: the scan parses each file once for both calls. */
187 function callbacksIn(source: ts.SourceFile, call: Call): { callback: string; named: boolean; line: number }[] {
183188 const found: { callback: string; named: boolean; line: number }[] = [];
184189 const visit = (node: ts.Node): void => {
185190 if (ts.isCallExpression(node) && ts.isIdentifier(node.expression) && node.expression.text === call) {
@@ -272,7 +277,8 @@ function scan(): Timer[] {
272277 for (const file of sources(SRC)) {
273278 const text = readFileSync(join(ROOT, file), "utf8");
274279 if (!/\bset(Interval|Timeout)\(/.test(text)) continue;
275 for (const call of ["setInterval", "setTimeout"] as const) for (const c of callbacks(text, call)) out.push({ file, call, ...c });
280 const source = ts.createSourceFile("x.ts", text, ts.ScriptTarget.Latest, false);
281 for (const call of ["setInterval", "setTimeout"] as const) for (const c of callbacksIn(source, call)) out.push({ file, call, ...c });
276282 }
277283 return out;
278284 }
test/tui-system.test.ts +12 −1
@@ -16,12 +16,13 @@ import { createForgeApp, forgeCommands, Session } from "../src/tui/forge.ts";
1616 import { setTelemetryGrant } from "../src/actions/telemetry.ts";
1717 import { hasGrant } from "../src/core/telemetry.ts";
1818 import { countRead, resetTelemetryReads } from "../src/system/telemetry_reads.ts";
19 import { dbSizeLines, dbSizeScreen, footprintLines, forgeLines, historyLines, largestLines, readerLines, memoryLines, memoryScreen, systemScreen } from "../src/tui/screens/system.ts";
19 import { askingLines, dbSizeLines, dbSizeScreen, footprintLines, forgeLines, historyLines, largestLines, readerLines, memoryLines, memoryScreen, systemScreen } from "../src/tui/screens/system.ts";
2020 import type { DbSize } from "../src/system/db_size.ts";
2121 import { changeCapacity } from "../src/actions/capacity.ts";
2222 import { EGRESS_REPORTED } from "../src/system/egress_line.ts";
2323 import { keepFootprint } from "../src/system/footprint.ts";
2424 import { headlessTerminal } from "../src/tui/term/terminal.ts";
25 import { asking } from "../src/perf/asking.ts";
2526 import { quietLogs } from "./helpers/globals.ts";
2627 import { settleApp } from "./helpers/tui.ts";
2728
@@ -108,6 +109,16 @@ describe("tui system memory", () => {
108109 });
109110 });
110111
112 describe("tui system who is asking (#12265)", () => {
113 test("the owner reads who asks the most; nobody else reads a name", () => {
114 for (let i = 0; i < 300; i++) asking.note("app:tui-poller/production", "/api/v1/repos/:owner/:name/runs", 1);
115 const lines = askingLines(new Session(app, identity(carol))).join("\n");
116 expect(lines).toContain("Who is asking");
117 expect(lines).toContain("tui-poller made");
118 for (const who of [bob, bot, undefined]) expect(askingLines(new Session(app, identity(who)))).toEqual([]);
119 });
120 });
121
111122 describe("tui system largest directories (#11800)", () => {
112123 const found = { at: new Date(Date.now() - 3 * 60_000).toISOString(), dirs: [{ name: "repos", bytes: 2 * 1024 ** 3 }, { name: "apps", bytes: 5 * MiB }] };
113124

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs