Public: anyone can read this; the crew can change it.

The serving process stops keeping every request an app’s host answers and 64 graphs of the busiest repository; what it holds is named each minute #13099

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

trevato/heap-stays-bounded → main 2 commits 1 thread · 1 unresolved

  • Merged. trevato/heap-stays-bounded was merged into main as e9ac3d7.

Review

Sign in to review this change.
19 files changed +497 −17
AGENTS.md +2
@@ -37,6 +37,8 @@ State shared between modules through process memory, not the database, is named
3737
3838 `src/perf/memory_watch.ts`: the memory watch's first-hour sums and baseline per owner (the main VM and each worker by name, at most 64), and each owner's latest reading, fed each minute by the sampler and the workers' `forge-memory` channel (#12951); gone with the process, which is when the baseline should be.
3939
40 `src/perf/held.ts`: `heldOf(app)` and `threadHeld`, the name and the size getter of each long-lived Map, Set or list a forge or the thread keeps (#12901), written when the holder is made or its module loads, never by a request, and read each minute by the `memory holds:` log line (`src/boards/memory.ts`; a worker's own on `memory worker <name> holds:`). A new long-lived collection names itself there, with a bound and a test of its eviction.
41
4042 `src/apps/timer_fire.ts`: `inFlight`, the ids of timers being fired in this process, so a slow app's timer is never fired twice at once; at most the timers due, gone when each settles. The timers themselves are rows (`timer`), read at every boot, never memory.
4143
4244 `src/actions/jobs.ts`: `jobReach.override`, a test seam only (a test hands in its own way to reach an app); the real one is `runtimeReach`. A durable job's runs are the app's own rows (`src/jobs.ts`, carried in the app template bundle `templates/app-template.json`; the template repository `trevato/app-template` must take the same files before a forge that reads that repository hands them out); the template's `Jobs` keeps in process memory only the ids of runs being advanced now (`active`), so a restart forgets nothing it needs.
docs/features/memory-growth.md +12 −2
@@ -5,6 +5,7 @@ Since Bun 1.4.4 the serving process has held several GiB after hours. This chang
55 ## Where it shows
66
77 - **The minute line.** Each minute the log carries `memory main: rss …, heap …, malloc … (peak …, committed peak …), up … min`. Every worker VM writes `memory worker <name>: heap …, malloc …, process rss …` on its own minute (#12925). RSS rising while the main heap stays flat is native memory; a worker line rising is that worker's. Read them in `/system`'s log or the process log.
8 - **What is held.** Beside the minute line the log carries `memory holds: <name> <count>, …`: the forge's long-lived Maps, Sets and lists by name with how many entries each holds now, largest first, twelve at most (`src/perf/held.ts`). A worker whose thread holds anything writes `memory worker <name> holds: …`. A name whose count only rises from line to line is the holder to read.
89 - **The stall line.** Each loop-stall line ends with `heap X (was Y N s ago; …)`, the heap now against the last minute's sample.
910 - `fell: a collection ran` means the heap shrank by more than a tenth since the sample, so the collector ran inside that stall. Bun 1.4.4 reports no collector time, so this is inferred, not measured.
1011 - `did not fall: no collection seen` means the stall was not the collector's.
@@ -15,6 +16,7 @@ Since Bun 1.4.4 the serving process has held several GiB after hours. This chang
1516 ## What to do when the need shows
1617
1718 1. The need names the VM and the part. To see the curve, read the `memory main` and `memory worker` lines around the time named: heap up means JS objects, heap flat with RSS up means native memory, a worker line up means that worker.
19 Then read the `memory holds:` lines across the same minutes: a holder that grew with the heap is named there with its count.
1820 2. Check the stall lines for `fell: a collection ran`: stalls that are collections point at the heap.
1921 3. Restart the forge to release it, and report the lines with the issue.
2022
@@ -24,6 +26,14 @@ The need clears by itself once RSS falls under twice the baseline. A process tha
2426
2527 `src/perf/memory_watch.ts` owns the readings (`heapSize()` and `memoryUsage()`, constant time; never `heapStats()`, which held the loop ~280 ms a minute, #11118). `vitals()` takes the watch as `watch`, as it takes `disk`. Each worker entry calls `workerMemory(<name>)` once at start; `FORGE_WORKER_MEMORY_MS` shortens the minute for tests. The same minute timer that writes a worker's line posts its reading (`{ name, heap, current }`) on the `forge-memory` `BroadcastChannel` (`tellReading`); the serving VM's memory sampler listens (`hearWorkers`), validates each message (`workerReadingOf`: a lower-case name, not `main`, finite byte counts) and notes it. Asks carry nothing extra, and at most 64 workers are kept. Tests: `test/memory-watch.test.ts`, `test/worker-memory-lines.test.ts`, `test/vitals.test.ts`, `test/tui-memory-growth.test.ts`.
2628
27 ## Not done
29 ## What grew, and what holds it now
2830
29 Finding what grows and fixing it.
31 Found on a copy of production's data with no network, production's own work on (apps, crew, checks, the picture keeper) and a replay of pages, API and MCP calls, event streams and pushes; object counts by type from `bun:jsc` `heapStats()` after a full collection, and heap snapshots, at 5, 20 and 60 minutes.
32
33 - **The turn gate kept every request an app's host answered** (`src/http/app_turn.ts`, in since 0.7.598, one release before Bun 1.4.4). The listener began a turn for every request and only the forge's own routes ended it; a request answered ahead of the routes (an app's host through the relay, a refusal) stayed in the gate's Map, with its `Request`, `Headers` and `AbortSignal`, until an app's own token next waited on a quiet thread, which on production is never. On the copy, with nobody visiting an app, the looks' Chromium alone asked previews for about 450 requests a minute. The listener now leaves the gate in its `finally`, whatever door answered: the gate holds what is in flight. Bun holds nothing here: a served `Request` is collected on 1.3.14 and 1.4.4 alike once nothing of ours keeps it.
34 - **The plan and lane caches kept answers nobody could ask for again** (`src/web/map_cache.ts`, `src/view/cache.ts`). A plan was keyed by its stamp, which moves with every run that starts or spends, and a lane by its repository's latest event; an entry past its minute was only dropped when the same key was read again or the count (64 plans, 128 lanes) pushed it out. A plan holds its repository's whole graph: the copy held ten graphs of the busiest repository (4,593 issues each) after an hour of page views, and the cache had room for 64. Now there is one plan per repository, spend visibility and day, and one lane per repository, scope and day, each replaced when its stamp moves; and every view cache drops what is past its time on the first write after `SWEEP_MS` (30 s).
35 - **A hosted forge that slept left its request counter and route timing behind** (`src/http/server.ts`), one more of each with every wake, each keeping the closed database and a flush that retried against it. They now go when the forge closes, written out first.
36
37 A new long-lived collection names itself in `heldOf(app)` (a forge's) or `threadHeld` (the thread's), with a bound and a test of its eviction. Named today: `turn`, `requests in flight`, `event streams`, `request counters`, `route timings`, `map plans`, `map lanes`, `map layouts`, and four the crew keeps per pull or per check that nothing sweeps yet (`autonomy conflicted pulls`, `finishing said`, `checks retried`, `app links said`): each is a short string per entry, and its line says when one is worth a sweep.
38
39 Tests: `test/app-turn.test.ts`, `test/cache-holds.test.ts`, `test/forge-sleep.test.ts`, `test/held.test.ts`.
src/boards/memory.ts +5 −1
@@ -5,7 +5,9 @@
55 * directory), as a count, a sum and a max. RSS rising while the
66 * heap stays flat is native memory growth. A week of hours is kept.
77 *
8 * Each tick is also one log line (`perf/memory_watch.ts`, #12901).
8 * Each tick is also one log line (`perf/memory_watch.ts`, #12901), and a second that
9 * names what the forge holds, each holder with its entry count (`perf/held.ts`): a Map
10 * that only grows reads there by its name.
911 *
1012 * `heapSize()`, never `heapStats().heapSize`: the same byte count, but
1113 * `heapStats()` first counts every object on the heap by type, which held the
@@ -15,6 +17,7 @@ import type { Database } from "bun:sqlite";
1517 import type { App } from "../app.ts";
1618 import { quietly } from "../core/write_clock.ts";
1719 import { during } from "../perf/jobs.ts";
20 import { heldLine, heldNow } from "../perf/held.ts";
1821 import { hearWorkers, memoryLine, memoryWatch, readNow } from "../perf/memory_watch.ts";
1922
2023 export const MEMORY_EVERY_MS = 60_000;
@@ -74,6 +77,7 @@ export function startMemorySampler(app: App, everyMs = MEMORY_EVERY_MS): MemoryS
7477 recordMemory(app.db, reading.rss, reading.heap, now);
7578 memoryWatch.note(reading.rss, reading.heap, now, reading.current);
7679 app.log.info(memoryLine(reading, process.uptime() * 1000));
80 app.log.info(heldLine(heldNow(app)));

No test fails if this line is removed. Add a sampler test that expects a memory holds: line each tick.

Added in 57e2566: test/held.test.ts, "each tick writes what the forge holds beside its memory line, counted at that tick". It starts the sampler on a forge with a named holder and expects, for both ticks, memory main then memory holds with the holder at 3 and then at 4. With this line removed it is red.

7781 } catch (error) {
7882 app.log.warn("memory sample failed", { error: String(error) });
7983 }
src/crew/autonomy.ts +2
@@ -157,6 +157,7 @@ import { passLanded } from "./box_pass.ts";
157157 import { reviewersForRepos, reviewsOff, sendToReview, sweepReviews } from "./review.ts";
158158 import { lastWaits, noteWaits, tallyWaits, type Wait, waitWords } from "./waits.ts";
159159 import { bootCounts } from "../kernel/boots.ts";
160 import { threadHeld } from "../perf/held.ts";
160161
161162 const log = createLog("autonomy");
162163
@@ -1597,6 +1598,7 @@ async function nextOf(
15971598
15981599 /** Pull id → the `head:base` tips a merge last conflicted between. */
15991600 const conflicted = new Map<string, string>();
1601 threadHeld.add("autonomy conflicted pulls", () => conflicted.size);
16001602
16011603 /** A merge refusal in words a log can carry. */
16021604 function mergeWhy(error: AccessError | MergePullError): string {
src/crew/dispatch.ts +2
@@ -140,6 +140,7 @@ import { STALE_AFTER_MS } from "../runs/lease.ts";
140140 import { stoppedMark } from "../spawn/stopped.ts";
141141 import { abortAgentRun } from "./running.ts";
142142 import { during } from "../perf/jobs.ts";
143 import { threadHeld } from "../perf/held.ts";
143144
144145 const log = createLog("crew");
145146
@@ -1171,6 +1172,7 @@ function recheckWaiting(app: App, repo: Repo, state: unknown): void {
11711172
11721173 /** Checks already run again once for a head; a second red is real. */
11731174 const retriedChecks = new Set<string>();
1175 threadHeld.add("checks retried", () => retriedChecks.size);
11741176
11751177 /** What a failed check led to, once the crew had read it. */
11761178 export type CheckAnswer = "reran" | "reworked" | "parked" | "waiting";
src/crew/finishing.ts +2
@@ -61,6 +61,7 @@ import { HANDOVER_NOTE } from "./guard.ts";
6161 import { interruptionOf } from "./interrupted.ts";
6262 import { pickBuilder } from "./route.ts";
6363 import { NO_BUILDER_ON_CREW, WAITS_FOR_BUILDER_SLOT } from "./words.ts";
64 import { threadHeld } from "../perf/held.ts";
6465
6566 const log = createLog("crew");
6667
@@ -344,6 +345,7 @@ function handOver(app: App, repo: Repo, pull: Pull, issue: Issue, files: readonl
344345
345346 /** The last why-not said for a pull, so a stall is said once until it changes. */
346347 const said = new Map<string, string>();
348 threadHeld.add("finishing said", () => said.size);
347349
348350 /** What each repository's pass last said it read, so its entry is said when that changes. */
349351 const entered = new Map<string, string>();
src/http/app_turn.ts +34 −1
@@ -13,12 +13,26 @@
1313 *
1414 * Streams (a long poll, `/events`) are older than `YOUNG_MS` and hold a
1515 * connection, not the thread. Plain data, one object per process.
16 *
17 * The gate holds a request only while it is open (#12901): every door ends or leaves it,
18 * the listener's own `finally` included, so what it holds is what is in flight, and past
19 * `OPEN_MAX` it forgets the ones too old to matter, whoever forgot to end them. It held
20 * every request an app's host answered (the relay never ended one) until an app's own
21 * token next asked on a quiet thread: 27,000 an hour on a copy of production, from the
22 * looks' Chromium alone, each with its Request, Headers and AbortSignal.
1623 */
1724
25 import { heldOf } from "../perf/held.ts";
26
1827 export const YOUNG_MS = 1_000;
1928 export const BURST_MS = 50;
2029 export const LOOK_MS = 20;
2130 export const BOUND_MS = 1_000;
31 /**
32 * The most requests the gate keeps before it forgets the old ones itself: a request
33 * older than `YOUNG_MS` holds a connection, not the thread, so it is nobody's reason to wait.
34 */
35 export const OPEN_MAX = 4_096;
2236
2337 export interface Turn {
2438 /** A request arrived: it is a person's until it says otherwise. */
@@ -27,6 +41,10 @@ export interface Turn {
2741 machine(key: object): void;
2842 /** It is answered. */
2943 end(key: object, at?: number): void;
44 /** It left by a door that does not count it (an app's host, a refusal): no burst follows. */
45 leave(key: object): void;
46 /** How many requests the gate holds now: the open ones. */
47 size(): number;
3048 /**
3149 * Resolve at once when nobody above is waiting, else when they are done or after the bound.
3250 * The wait is counted in `into`: the gate's own totals, or the forge's whose request it was.
@@ -65,6 +83,13 @@ export function createTurn(
6583 };
6684 return {
6785 begin(key, at = now()) {
86 // Begun in time order, so the walk ends at the first young one: a full gate costs
87 // one look at what it forgets, never a scan of what it keeps.
88 if (open.size >= OPEN_MAX)
89 for (const [held, since] of open) {
90 if (at - since < YOUNG_MS) break;
91 open.delete(held);
92 }
6893 open.set(key, at);
6994 },
7095 machine(key) {
@@ -73,6 +98,10 @@ export function createTurn(
7398 end(key, at = now()) {
7499 if (open.delete(key)) lastEnd = at;
75100 },
101 leave(key) {
102 open.delete(key);
103 },
104 size: () => open.size,
76105 async wait(into = stats) {
77106 if (!wanted()) return;
78107 const since = now();
@@ -99,7 +128,11 @@ export function createTurn(
99128 const gates = new WeakMap<object, Turn>();
100129 export function turnOf(app: object): Turn {
101130 let gate = gates.get(app);
102 if (gate === undefined) gates.set(app, (gate = createTurn()));
131 if (gate === undefined) {
132 const made = createTurn();
133 gates.set(app, (gate = made));
134 heldOf(app).add("turn", made.size);
135 }
103136 return gate;
104137 }
105138
src/http/server.ts +26 −2
@@ -560,6 +560,7 @@ import {
560560 type RouteTiming,
561561 } from "../core/route_timing.ts";
562562 import { captureStore, inFlight, liveOf, readLedger } from "../perf/live.ts";
563 import { heldOf } from "../perf/held.ts";
563564 import { childLabel, markChildRequest, noteChildRequest } from "../perf/child_time.ts";
564565 import { parseHostedChildName } from "../spawn/placement.ts";
565566 import { childHost } from "../spawn/door.ts";
@@ -2704,6 +2705,23 @@ export async function serve(app: App): Promise<Servers> {
27042705 // map increment; a refused event would be a programming error, not worth a 500.
27052706 const counters = new Map<Database, UsageCounter>();
27062707 const timings = new Map<Database, RouteTiming>();
2708 heldOf(app).add("request counters", () => counters.size);
2709 heldOf(app).add("route timings", () => timings.size);
2710 // A hosted forge that closes takes its counter and its timing with it, written out
2711 // while its database is still open (#12901): each was kept for as long as the parent
2712 // ran, one more with every wake, its flush retrying against a closed database.
2713 const unwatchCounts = watchHostedChildClose(app, (child) => {
2714 for (const kept of [counters, timings]) {
2715 const one = kept.get(child.db);
2716 if (one === undefined) continue;
2717 kept.delete(child.db);
2718 try {
2719 one.stop();
2720 } catch (error) {
2721 app.log.warn("could not flush a closing forge's request counts", { error: String(error) });
2722 }
2723 }
2724 });
27072725 // The minute ledger's history: one write a minute, never on a request.
27082726 const trafficWriter = createTrafficWriter(app.db, liveOf(app).perfRecord.ledger, app.log);
27092727 // Page views are buffered by the beacon doors and written here; GitHub's traffic is pulled daily when a token is set.
@@ -3025,8 +3043,13 @@ export async function serve(app: App): Promise<Servers> {
30253043 const request = parseableRequest(raw);
30263044 const bound = request === raw ? server : answeringFor(server, request, raw);
30273045 return inFlight.during(request, performance.now(), async () => {
3028 const response = await handle(request, bound);
3029 return handedOver && response !== undefined ? withHeader(response, "connection", "close") : response;
3046 try {
3047 const response = await handle(request, bound);
3048 return handedOver && response !== undefined ? withHeader(response, "connection", "close") : response;
3049 } finally {
3050 // Whatever door answered: the gate holds no request past its answer (#12901).
3051 turnOf(app).leave(request);
3052 }
30303053 });
30313054 },
30323055 // A last resort: `handle` answers a throw itself, so it is counted.
@@ -3085,6 +3108,7 @@ export async function serve(app: App): Promise<Servers> {
30853108 for (const key of [...new Set([...childHttps.keys(), ...childInternals.keys()])]) letGo(key);
30863109 closeHostedChildren(app);
30873110 unwatchChildren();
3111 unwatchCounts();
30883112 void stopCellChildren(app);
30893113 flushAccess();
30903114 web.stop(true);
src/http/sse.ts +2
@@ -26,6 +26,7 @@ import { mayReadLogOfRun } from "../actions/runs.ts";
2626 import { getRun } from "../core/runs.ts";
2727 import { frameOf, tapRunLogs } from "../runs/log.ts";
2828 import type { ForgeEvent } from "../core/events.ts";
29 import { threadHeld } from "../perf/held.ts";
2930
3031 const HEARTBEAT_MS = 25_000;
3132
@@ -66,6 +67,7 @@ export const resetSseDroppedCount = (): void => {
6667 * replaced this one (docs/ARCHITECTURE.md, "Two processes at once").
6768 */
6869 const openStreams = new Set<() => void>();
70 threadHeld.add("event streams", () => openStreams.size);
6971
7072 /** How soon a drained stream's client reconnects, in milliseconds. */
7173 export const DRAIN_RETRY_MS = 1000;
src/perf/held.ts added +82
@@ -0,0 +1,82 @@
1 /**
2 * What the forge holds in memory, by name (#12901): every long-lived Map, Set or list
3 * names itself here with a way to count it, and the minute memory line
4 * (`boards/memory.ts`) writes the largest counts, so a holder that only grows reads in
5 * the log by its name. A holder that belongs to a forge names itself in that forge's
6 * registry when it is made (`heldOf(app)`); one a module keeps once for the thread names
7 * itself as the module loads (`threadHeld`). The serving forge's line reads both, and a
8 * worker's line reads its own thread's (`perf/memory_watch.ts`).
9 *
10 * A count is a `size` read, constant time; nothing here walks a collection, and the
11 * registry keeps only the name and the getter. A name given again replaces the earlier
12 * getter, so the registry is as large as the names in the source, never as the work done.
13 */
14
15 export interface HeldCount {
16 readonly name: string;
17 /** Entries now; -1 when the getter threw. */
18 readonly size: number;
19 }
20
21 export interface Held {
22 /** Name a holder and how to count it. The same name again replaces the earlier count. */
23 add(name: string, size: () => number): void;
24 /** Every holder's count now, largest first, then by name. */
25 read(): HeldCount[];
26 }
27
28 export function createHeld(): Held {
29 const sizes = new Map<string, () => number>();
30 return {
31 add(name, size) {
32 sizes.set(name, size);
33 },
34 read() {
35 const out: HeldCount[] = [];
36 for (const [name, size] of sizes) {
37 let n = -1;
38 try {
39 const read = size();
40 if (Number.isFinite(read)) n = read;
41 } catch {
42 // A holder that cannot be counted says so with -1 and never fails the line.
43 }
44 out.push({ name, size: n });
45 }
46 return out.sort((a, b) => b.size - a.size || (a.name < b.name ? -1 : 1));
47 },
48 };
49 }
50
51 const registries = new WeakMap<object, Held>();
52
53 /** This forge's registry, made at its first use. */
54 export function heldOf(app: object): Held {
55 let held = registries.get(app);
56 if (held === undefined) registries.set(app, (held = createHeld()));
57 return held;
58 }
59
60 /**
61 * What a module keeps once for the thread, named as the module loads. The names are the
62 * source's, so this is written at load and read each minute, never by a request.
63 */
64 export const threadHeld: Held = createHeld();
65
66 /** A forge's own holders and the thread's, as one reading: largest first. */
67 export function heldNow(app: object): HeldCount[] {
68 return [...heldOf(app).read(), ...threadHeld.read()].sort((a, b) => b.size - a.size || (a.name < b.name ? -1 : 1));
69 }
70
71 /** How many holders the minute line names. */
72 export const HELD_TOP = 12;
73
74 /**
75 * The minute line's words: `memory holds: sse 14, turn 3, in-flight 3` for the `top`
76 * largest holders that hold anything; `memory holds: nothing` when none does.
77 */
78 export function heldLine(counts: readonly HeldCount[], top: number = HELD_TOP): string {
79 const named = counts.filter((c) => c.size !== 0).slice(0, top);
80 if (named.length === 0) return "memory holds: nothing";
81 return `memory holds: ${named.map((c) => `${c.name} ${c.size < 0 ? "unread" : c.size}`).join(", ")}`;
82 }
src/perf/live.ts +2
@@ -15,9 +15,11 @@ import { createReadLedger } from "./reads.ts";
1515 import { createStallBlame } from "./stall_blame.ts";
1616 import { createPerfRecord } from "./record.ts";
1717 import { createLagMinutes } from "./lag_minutes.ts";
18 import { threadHeld } from "./held.ts";
1819
1920 /** The requests the server is answering now (#11478): begun and ended around each one in `http/server.ts`. */
2021 export const inFlight = createInFlight<Request>((request) => childRequestLabel(request) ?? patternOf(request));
22 threadHeld.add("requests in flight", inFlight.size);
2123 export const gitRecord = createGitRecord();
2224 /** The worst stall in this process's first five minutes, read by the release keeper (#8848). */
2325 export const bootStall = createBootStall();
src/perf/memory_watch.ts +20 −1
@@ -12,6 +12,9 @@
1212 * posts to from the same minute timer that writes its line (`tellReading`, `hearWorkers`),
1313 * so no ask and no hot-path message carries it; a worker silent for five minutes is not blamed.
1414 *
15 * Beside each line, what is held by name (`perf/held.ts`): the serving forge's holders
16 * and the thread's on `memory holds:`, a worker's own thread's on its second line.
17 *
1518 * `heapStats()` counts every object on the heap by type and held the loop ~280 ms a
1619 * minute on a 2M-object heap (#11118), so the minute line never calls it: `heapSize()`
1720 * and `memoryUsage()` are constant-time. Bun 1.4.4 exposes no collector time (a
@@ -21,6 +24,7 @@
2124 */
2225 import { heapSize, memoryUsage as jscMemory } from "bun:jsc";
2326 import { createLog } from "../kernel/log.ts";
27 import { heldLine, threadHeld, type HeldCount } from "./held.ts";
2428
2529 const MIB = 1048576;
2630 export const FIRST_HOUR_MS = 3_600_000;
@@ -309,8 +313,23 @@ export function hearWorkers(watch: Pick<MemoryWatch, "noteWorker"> = memoryWatch
309313 };
310314 }
311315
316 /**
317 * A worker's second line, when its thread holds anything by name (`perf/held.ts`):
318 * `memory worker engine holds: checks retried 12`. Undefined when it holds nothing.
319 */
320 export function workerHeldLine(name: string, counts: readonly HeldCount[] = threadHeld.read()): string | undefined {
321 const line = heldLine(counts);
322 return line === heldLine([]) ? undefined : line.replace(/^memory holds:/, `memory worker ${name} holds:`);
323 }
324
312325 /** Start a worker's own minute line (an unref'd timer inside the worker). Returns the stop. */
313 export function startWorkerMemory(name: string, write: (line: string) => void, everyMs = 60_000): () => void {
326 export function startWorkerMemory(name: string, out: (line: string) => void, everyMs = 60_000): () => void {
327 // The reading's line, then what the thread holds by name, when it holds anything.
328 const write = (line: string): void => {
329 out(line);
330 const held = workerHeldLine(name);
331 if (held !== undefined) out(held);
332 };
314333 const timer = setInterval(() => write(workerMemoryLine(name, tellReading(name, readNow()))), everyMs);
315334 (timer as { unref?: () => void }).unref?.();
316335 return () => clearInterval(timer);
src/view/cache.ts
Binary file not shown.
src/web/map_cache.ts +24 −9
@@ -10,6 +10,7 @@
1010 */
1111
1212 import type { Database } from "bun:sqlite";
13 import { threadHeld } from "../perf/held.ts";
1314 import { ViewCache } from "../view/cache.ts";
1415 import type { Graph } from "../core/dependencies.ts";
1516
@@ -20,6 +21,8 @@ const LAYOUT_TTL_MS = 60 * 60_000;
2021
2122 const lanes = new ViewCache({ maxEntries: 128 });
2223 const layouts = new ViewCache({ maxEntries: 256 });
24 threadHeld.add("map lanes", () => lanes.stats().entries);
25 threadHeld.add("map layouts", () => layouts.stats().entries);
2326
2427 /** What `sealedScope` computed, as a string that goes into a key. */
2528 export function scopeKey(scope: { sealed?: true; sealedBy?: string }): string {
@@ -59,20 +62,25 @@ const utcDay = (now: Date): string => now.toISOString().slice(0, 10);
5962 /**
6063 * A lane's cached build, or `build()`'s result kept for next time. `null`
6164 * is a cached answer too: a repository with nothing to draw.
65 *
66 * One lane per repository, scope and day (#12901): the head it was built at is
67 * kept beside it, and a new head replaces it. Keyed by the head, every event
68 * left the lane before it in the cache until 128 newer ones pushed it out.
6269 */
6370 export function cachedLane<T>(
6471 key: { repoId: string; head: number; scope: string; now: Date },
6572 build: () => T,
6673 ): T {
67 const k = `lane:${key.repoId}:${key.head}:${key.scope}:${utcDay(key.now)}`;
68 const kept = lanes.get<{ value: T }>(k);
69 if (kept !== undefined) return kept.value;
74 const k = `lane:${key.repoId}:${key.scope}:${utcDay(key.now)}`;
75 const kept = lanes.get<{ head: number; value: T }>(k);
76 if (kept !== undefined && kept.head === key.head) return kept.value;
7077 const value = build();
71 lanes.set(k, { value }, LANE_TTL_MS);
78 lanes.set(k, { head: key.head, value }, LANE_TTL_MS);
7279 return value;
7380 }
7481
7582 const plans = new ViewCache({ maxEntries: 64 });
83 threadHeld.add("map plans", () => plans.stats().entries);
7684 let planBuilds = 0;
7785
7886 /**
@@ -82,6 +90,12 @@ let planBuilds = 0;
8290 * lifetime. The plan holds no
8391 * viewer: who may read the repository and the seats are the caller's, every
8492 * request.
93 *
94 * One plan per repository, spend visibility and day (#12901): the stamp is kept
95 * beside it, and a new stamp replaces it. A plan holds the repository's whole
96 * graph, and the stamp moves with every run that starts or spends: keyed by
97 * the stamp, the cache held 64 graphs of the busiest repository, 63 of them
98 * for a stamp that could not come back.
8599 */
86100 export function cachedPlan<T>(
87101 key: { repoId: string; stamp: string | undefined; spend: boolean; now: Date },
@@ -92,12 +106,12 @@ export function cachedPlan<T>(
92106 planBuilds++;
93107 return build();
94108 }
95 const k = `plan:${key.repoId}:${key.stamp}:${key.spend ? "spend" : "plain"}:${utcDay(key.now)}`;
96 const kept = plans.get<{ value: T }>(k);
97 if (kept !== undefined) return kept.value;
109 const k = `plan:${key.repoId}:${key.spend ? "spend" : "plain"}:${utcDay(key.now)}`;
110 const kept = plans.get<{ stamp: string; value: T }>(k);
111 if (kept !== undefined && kept.stamp === key.stamp) return kept.value;
98112 planBuilds++;
99113 const value = build();
100 plans.set(k, { value }, LANE_TTL_MS);
114 plans.set(k, { stamp: key.stamp, value }, LANE_TTL_MS);
101115 return value;
102116 }
103117
@@ -157,4 +171,5 @@ export function clearMapCache(): void {
157171 export const mapCacheStats = (): {
158172 lanes: ReturnType<ViewCache["stats"]>;
159173 layouts: ReturnType<ViewCache["stats"]>;
160 } => ({ lanes: lanes.stats(), layouts: layouts.stats() });
174 plans: ReturnType<ViewCache["stats"]>;
175 } => ({ lanes: lanes.stats(), layouts: layouts.stats(), plans: plans.stats() });
test/app-turn.test.ts +68 −1
@@ -14,7 +14,7 @@ import { getRepo } from "../src/core/repos.ts";
1414 import { createToken } from "../src/core/tokens.ts";
1515 import { createUser } from "../src/core/users.ts";
1616 import { serve, type Servers } from "../src/http/server.ts";
17 import { createTurn, turnStatsOf } from "../src/http/app_turn.ts";
17 import { createTurn, OPEN_MAX, turnOf, turnStatsOf, YOUNG_MS } from "../src/http/app_turn.ts";
1818 import { askingOf, createAsking, gapWords } from "../src/perf/asking.ts";
1919
2020 describe("the turn", () => {
@@ -72,6 +72,50 @@ describe("the turn", () => {
7272 expect(turn.stats.waited).toBe(0);
7373 });
7474
75 test("the gate holds a request only while it is open: ended or left, it is gone (#12901)", async () => {
76 let t = 0;
77 const turn = createTurn(() => t, 100, async (ms) => void (t += ms));
78 // Ten thousand requests an app's host answered: begun, and gone by the relay's door.
79 for (let i = 0; i < 10_000; i++) {
80 const relayed = {};
81 turn.begin(relayed, t);
82 expect(turn.size()).toBe(1);
83 turn.leave(relayed);
84 t += 1;
85 }
86 expect(turn.size()).toBe(0);
87 // Leaving is not an answer a burst follows: an app's own call goes at once.
88 await turn.wait();
89 expect(turn.stats.waited).toBe(0);
90 // An answer ends it the same way, and a key the gate never held is ignored.
91 const person = {};
92 turn.begin(person, t);
93 turn.end(person, t);
94 turn.leave(person);
95 turn.leave({});
96 expect(turn.size()).toBe(0);
97 });
98
99 test("a gate nobody ends is bounded all the same: past its size the old ones are forgotten", async () => {
100 let t = 0;
101 const turn = createTurn(() => t, 100, async (ms) => void (t += ms));
102 // A hundred thousand requests begun and never ended, a millisecond apart.
103 for (let i = 0; i < 100_000; i++) {
104 turn.begin({}, t);
105 t += 1;
106 expect(turn.size()).toBeLessThanOrEqual(OPEN_MAX);
107 }
108 // What is left began in the last second or filled the gate since its last sweep.
109 expect(turn.size()).toBeGreaterThan(0);
110 // The young ones still count: an app's call waits for them, to its bound.
111 await turn.wait();
112 expect(turn.stats.waited).toBe(1);
113 // A second of quiet, and the next request finds a gate with nobody in it to wait for.
114 t += YOUNG_MS;
115 await turn.wait();
116 expect(turn.stats.waited).toBe(1);
117 });
118
75119 test("the gate is the thread's, the count is the forge's: a wait is counted for the forge whose request it was", async () => {
76120 let t = 0;
77121 const turn = createTurn(() => t, 100, async (ms) => void (t += ms));
@@ -129,6 +173,29 @@ describe("who is asking", () => {
129173 });
130174 });
131175
176 describe("the gate behind the listener", () => {
177 test("a request answered by a door that counts nothing is not held past its answer (#12901)", async () => {
178 // A read-only forge refuses a write before any route: that door never ended the turn,
179 // like an app's host and every other answer made ahead of the routes.
180 const dir = await mkdtemp(join(tmpdir(), "forge-turn-door-"));
181 const app = createApp({ dataDir: dir, port: 0, readOnly: true });
182 const servers = await serve(app);
183 try {
184 for (let i = 0; i < 5; i++) {
185 const refused = await fetch(servers.url + "/api/v1/repos", { method: "POST", body: "{}" });
186 expect(refused.status).toBe(503);
187 await refused.text();
188 }
189 const page = await fetch(servers.url + "/healthz");
190 await page.text();
191 expect(turnOf(app).size()).toBe(0);
192 } finally {
193 await servers.stop();
194 await rm(dir, { recursive: true, force: true });
195 }
196 });
197 });
198
132199 describe("the forge answers an app token", () => {
133200 let app: App;
134201 let servers: Servers;
test/cache-holds.test.ts added +86
@@ -0,0 +1,86 @@
1 /**
2 * A cache holds what can still be asked for (#12901): an entry past its time goes on the next
3 * sweep, and a plan or a lane is one per repository, replaced when its stamp moves.
4 */
5 import { beforeEach, describe, expect, test } from "bun:test";
6 import { SWEEP_MS, ViewCache } from "../src/view/cache.ts";
7 import { cachedLane, cachedPlan, clearMapCache, mapCacheStats, planBuildCount } from "../src/web/map_cache.ts";
8 import { threadHeld } from "../src/perf/held.ts";
9
10 describe("the view cache drops what is past its time", () => {
11 test("entries nobody reads again go on the first write after the sweep is due, not at the count", () => {
12 let t = 1_000_000;
13 const cache = new ViewCache({ maxEntries: 10_000, now: () => t });
14 // A thousand answers under keys that never come back, each good for a minute.
15 for (let i = 0; i < 1_000; i++) cache.set(`answer:${i}`, { i }, 60_000, 10);
16 expect(cache.stats()).toMatchObject({ entries: 1_000, bytes: 10_000 });
17 // Past their time, and no write yet: still held.
18 t += 61_000;
19 expect(cache.stats().entries).toBe(1_000);
20 cache.set("fresh", { i: -1 }, 60_000, 10);
21 expect(cache.stats()).toMatchObject({ entries: 1, bytes: 10 });
22 expect(cache.get<{ i: number }>("fresh")).toEqual({ i: -1 });
23 });
24
25 test("a sweep leaves what is still good, and is one walk each SWEEP_MS however many writes come", () => {
26 let t = 0;
27 const cache = new ViewCache({ maxEntries: 10_000, now: () => t });
28 cache.set("short", 1, 1_000);
29 cache.set("long", 2, 10 * SWEEP_MS);
30 // Inside the sweep's interval an expired entry waits; the count and the bytes still bound it.
31 t += 2_000;
32 cache.set("another", 3, 10 * SWEEP_MS);
33 expect(cache.stats().entries).toBe(3);
34 t += SWEEP_MS;
35 cache.set("last", 4, 10 * SWEEP_MS);
36 expect(cache.stats().entries).toBe(3);
37 expect(cache.get<number>("short")).toBeUndefined();
38 expect(cache.get<number>("long")).toBe(2);
39 expect(cache.get<number>("another")).toBe(3);
40 });
41 });
42
43 describe("a plan and a lane are one per repository", () => {
44 beforeEach(() => clearMapCache());
45 const now = new Date("2026-11-15T08:00:00Z");
46
47 test("a hundred stamps of one repository's plan leave one plan, the newest", () => {
48 const before = planBuildCount();
49 for (let stamp = 0; stamp < 100; stamp++) {
50 const plan = cachedPlan({ repoId: "r1", stamp: `s${stamp}`, spend: false, now }, () => ({ stamp }));
51 expect(plan.stamp).toBe(stamp);
52 expect(mapCacheStats().plans.entries).toBe(1);
53 }
54 expect(planBuildCount()).toBe(before + 100);
55 // The same stamp again is served; a stamp that moved back is built again, never served stale.
56 expect(cachedPlan({ repoId: "r1", stamp: "s99", spend: false, now }, () => ({ stamp: -1 })).stamp).toBe(99);
57 expect(cachedPlan({ repoId: "r1", stamp: "s98", spend: false, now }, () => ({ stamp: 980 })).stamp).toBe(980);
58 expect(planBuildCount()).toBe(before + 101);
59 // Each spend visibility and each repository keeps its own.
60 cachedPlan({ repoId: "r1", stamp: "s98", spend: true, now }, () => ({ stamp: 1 }));
61 cachedPlan({ repoId: "r2", stamp: "s98", spend: false, now }, () => ({ stamp: 2 }));
62 expect(mapCacheStats().plans.entries).toBe(3);
63 expect(threadHeld.read().find((h) => h.name === "map plans")?.size).toBe(3);
64 // Inside a transaction there is no stamp, and nothing is kept.
65 cachedPlan({ repoId: "r3", stamp: undefined, spend: false, now }, () => ({ stamp: 3 }));
66 expect(mapCacheStats().plans.entries).toBe(3);
67 });
68
69 test("a hundred events on one repository leave one lane per scope, built once per head", () => {
70 let built = 0;
71 for (let head = 1; head <= 100; head++) {
72 for (const scope of ["all", "none"]) {
73 const lane = cachedLane({ repoId: "r1", head, scope, now }, () => ({ head, n: ++built }));
74 expect(lane.head).toBe(head);
75 // Asked again at the same head: the kept lane, not a new build.
76 expect(cachedLane({ repoId: "r1", head, scope, now }, () => ({ head: -1, n: ++built }))).toBe(lane);
77 }
78 expect(mapCacheStats().lanes.entries).toBe(2);
79 }
80 expect(built).toBe(200);
81 // A repository with nothing to draw is a kept answer too.
82 expect(cachedLane({ repoId: "r2", head: 5, scope: "all", now }, () => null)).toBeNull();
83 expect(cachedLane({ repoId: "r2", head: 5, scope: "all", now }, () => ({ head: 5, n: -1 }))).toBeNull();
84 expect(threadHeld.read().find((h) => h.name === "map lanes")?.size).toBe(3);
85 });
86 });
test/forge-sleep.test.ts +20
@@ -16,6 +16,7 @@ import { createSession } from "../src/core/sessions.ts";
1616 import { createUser } from "../src/core/users.ts";
1717 import { serve, type Servers } from "../src/http/server.ts";
1818 import { awakeCount, childAwake, gateClock, holdChild, resolveHostedChild, sleepIdleChildren, wakeGate } from "../src/spawn/host.ts";
19 import { heldOf } from "../src/perf/held.ts";
1920 import { removeForge, stopForge } from "../src/spawn/manage.ts";
2021 import { setLimit } from "../src/settings/limits.ts";
2122 import { awakeRefusal, personRefusal, WakeRate } from "../src/spawn/sleep.ts";
@@ -109,6 +110,25 @@ describe("a hosted forge scales to zero", () => {
109110 expect(repo.status).toBe(200);
110111 });
111112
113 test("a forge that sleeps takes its request counter and route timing with it, however often it wakes (#12901)", async () => {
114 const kept = (name: string) => heldOf(parent).read().find((h) => h.name === name)?.size ?? 0;
115 await sleepIdleChildren(parent, 600_000, later());
116 // Asleep: what the parent keeps for itself, one per database it has counted in.
117 const counters = kept("request counters");
118 const timings = kept("route timings");
119 for (let wake = 0; wake < 3; wake++) {
120 expect((await asChild("/api/v1/repos/bob/app")).status).toBe(200);
121 expect(childAwake(parent, "homie")).toBe(true);
122 expect(kept("request counters")).toBe(counters + 1);
123 expect(kept("route timings")).toBe(timings + 1);
124 expect(await sleepIdleChildren(parent, 600_000, later())).toEqual(["homie"]);
125 expect(kept("request counters")).toBe(counters);
126 expect(kept("route timings")).toBe(timings);
127 }
128 // And it wakes again: what it counted while awake was written before its database closed.
129 expect((await asChild("/api/v1/repos/bob/app")).status).toBe(200);
130 });
131
112132 test("wakes on a git push, and the data from before the sleep is still there", async () => {
113133 const repo = join(work, "app");
114134 expect((await git(["init", "-q", "-b", "main", repo], work)).code).toBe(0);
test/held.test.ts added +106
@@ -0,0 +1,106 @@
1 /**
2 * What the forge holds in memory is named and counted (#12901): one registry per forge,
3 * read by the minute memory line.
4 */
5 import { describe, expect, spyOn, test } from "bun:test";
6 import { mkdtemp, rm } from "node:fs/promises";
7 import { tmpdir } from "node:os";
8 import { join } from "node:path";
9 import { createApp } from "../src/app.ts";
10 import { startMemorySampler } from "../src/boards/memory.ts";
11 import { createHeld, heldLine, heldNow, heldOf, HELD_TOP, threadHeld } from "../src/perf/held.ts";
12 import { workerHeldLine } from "../src/perf/memory_watch.ts";
13
14 describe("the registry of what is held", () => {
15 test("each holder is counted when read, largest first, and a name given again replaces the earlier count", () => {
16 const held = createHeld();
17 const a = new Map<number, number>();
18 const b = new Set<number>();
19 held.add("a", () => a.size);
20 held.add("b", () => b.size);
21 for (let i = 0; i < 3; i++) a.set(i, i);
22 for (let i = 0; i < 7; i++) b.add(i);
23 expect(held.read()).toEqual([{ name: "b", size: 7 }, { name: "a", size: 3 }]);
24 a.clear();
25 expect(held.read()).toEqual([{ name: "b", size: 7 }, { name: "a", size: 0 }]);
26 // The registry is as large as the names, however often a holder is made again.
27 for (let i = 0; i < 1_000; i++) held.add("a", () => i);
28 expect(held.read()).toEqual([{ name: "a", size: 999 }, { name: "b", size: 7 }]);
29 });
30
31 test("a holder that cannot be counted reads -1 and never fails the read", () => {
32 const held = createHeld();
33 held.add("broken", () => {
34 throw new Error("gone");
35 });
36 held.add("nan", () => Number.NaN);
37 held.add("fine", () => 2);
38 expect(held.read()).toEqual([{ name: "fine", size: 2 }, { name: "broken", size: -1 }, { name: "nan", size: -1 }]);
39 });
40
41 test("each forge has its own registry", () => {
42 const one = {};
43 const other = {};
44 heldOf(one).add("x", () => 1);
45 expect(heldOf(one)).toBe(heldOf(one));
46 expect(heldOf(one).read()).toEqual([{ name: "x", size: 1 }]);
47 expect(heldOf(other).read()).toEqual([]);
48 });
49
50 test("a forge's line reads its own holders and the thread's as one, largest first", () => {
51 const forge = {};
52 const mine = new Set<number>([1, 2, 3]);
53 heldOf(forge).add("mine", () => mine.size);
54 const names = heldNow(forge).map((h) => h.name);
55 expect(names).toContain("mine");
56 // What a module keeps once for the thread named itself as it loaded.
57 for (const h of threadHeld.read()) expect(names).toContain(h.name);
58 const sizes = heldNow(forge).map((h) => h.size);
59 expect(sizes).toEqual([...sizes].sort((a, b) => b - a));
60 });
61
62 test("a worker's line names what its thread holds, and is not written when it holds nothing", () => {
63 expect(workerHeldLine("engine", [])).toBeUndefined();
64 expect(workerHeldLine("engine", [{ name: "checks retried", size: 0 }])).toBeUndefined();
65 expect(workerHeldLine("engine", [{ name: "checks retried", size: 12 }])).toBe("memory worker engine holds: checks retried 12");
66 });
67
68 test("the line names the largest holders that hold anything", () => {
69 expect(heldLine([])).toBe("memory holds: nothing");
70 expect(heldLine([{ name: "turn", size: 0 }])).toBe("memory holds: nothing");
71 expect(heldLine([{ name: "sse", size: 14 }, { name: "turn", size: 3 }, { name: "idle", size: 0 }, { name: "broken", size: -1 }])).toBe(
72 "memory holds: sse 14, turn 3, broken unread",
73 );
74 const many = Array.from({ length: 40 }, (_, i) => ({ name: `h${i}`, size: 100 - i }));
75 expect(heldLine(many).split(", ")).toHaveLength(HELD_TOP);
76 expect(heldLine(many, 2)).toBe("memory holds: h0 100, h1 99");
77 });
78 });
79
80 describe("the memory sampler", () => {
81 test("each tick writes what the forge holds beside its memory line, counted at that tick", async () => {
82 const dir = await mkdtemp(join(tmpdir(), "forge-held-"));
83 const app = createApp({ dataDir: dir, port: 0 });
84 const info = spyOn(app.log, "info").mockImplementation(() => {});
85 const probe = new Set<number>([1, 2, 3]);
86 heldOf(app).add("probe", () => probe.size);
87 // An hour between ticks: the sampler ticks once as it starts, and the test asks for the next.
88 const sampler = startMemorySampler(app, 3_600_000);
89 try {
90 probe.add(4);
91 sampler.tick();
92 const lines = info.mock.calls.map((call) => String(call[0]));
93 expect(lines.map((l) => l.slice(0, l.indexOf(":")))).toEqual(["memory main", "memory holds", "memory main", "memory holds"]);
94 expect(lines[1]).toStartWith("memory holds: ");
95 expect(lines[1]!.split(": ")[1]!.split(", ")).toContain("probe 3");
96 expect(lines[3]!.split(": ")[1]!.split(", ")).toContain("probe 4");
97 // The thread's holders are on the same line as the forge's own.
98 for (const h of threadHeld.read().filter((h) => h.size > 0).slice(0, 3)) expect(lines[3]).toContain(`${h.name} ${h.size}`);
99 } finally {
100 sampler.stop();
101 info.mockRestore();
102 app.close();
103 await rm(dir, { recursive: true, force: true });
104 }
105 });
106 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs