Public: anyone can read this; the crew can change it.
The serving process stops keeping every request an app’s host answers and 64 graphs of the busiest repository; what it holds is named each minute #13099
Check · Review · Land · Live done — Landed and live in production.
trevato/heap-stays-bounded → main 2 commits 1 thread · 1 unresolved
19 files
changed +497
−17
AGENTS.md +2
| @@ -37,6 +37,8 @@ State shared between modules through process memory, not the database, is named | ||
| 37 | 37 | |
| 38 | 38 | `src/perf/memory_watch.ts`: the memory watch's first-hour sums and baseline per owner (the main VM and each worker by name, at most 64), and each owner's latest reading, fed each minute by the sampler and the workers' `forge-memory` channel (#12951); gone with the process, which is when the baseline should be. |
| 39 | 39 | |
| 40 | `src/perf/held.ts`: `heldOf(app)` and `threadHeld`, the name and the size getter of each long-lived Map, Set or list a forge or the thread keeps (#12901), written when the holder is made or its module loads, never by a request, and read each minute by the `memory holds:` log line (`src/boards/memory.ts`; a worker's own on `memory worker <name> holds:`). A new long-lived collection names itself there, with a bound and a test of its eviction. | |
| 41 | ||
| 40 | 42 | `src/apps/timer_fire.ts`: `inFlight`, the ids of timers being fired in this process, so a slow app's timer is never fired twice at once; at most the timers due, gone when each settles. The timers themselves are rows (`timer`), read at every boot, never memory. |
| 41 | 43 | |
| 42 | 44 | `src/actions/jobs.ts`: `jobReach.override`, a test seam only (a test hands in its own way to reach an app); the real one is `runtimeReach`. A durable job's runs are the app's own rows (`src/jobs.ts`, carried in the app template bundle `templates/app-template.json`; the template repository `trevato/app-template` must take the same files before a forge that reads that repository hands them out); the template's `Jobs` keeps in process memory only the ids of runs being advanced now (`active`), so a restart forgets nothing it needs. |
docs/features/memory-growth.md +12 −2
| @@ -5,6 +5,7 @@ Since Bun 1.4.4 the serving process has held several GiB after hours. This chang | ||
| 5 | 5 | ## Where it shows |
| 6 | 6 | |
| 7 | 7 | - **The minute line.** Each minute the log carries `memory main: rss …, heap …, malloc … (peak …, committed peak …), up … min`. Every worker VM writes `memory worker <name>: heap …, malloc …, process rss …` on its own minute (#12925). RSS rising while the main heap stays flat is native memory; a worker line rising is that worker's. Read them in `/system`'s log or the process log. |
| 8 | - **What is held.** Beside the minute line the log carries `memory holds: <name> <count>, …`: the forge's long-lived Maps, Sets and lists by name with how many entries each holds now, largest first, twelve at most (`src/perf/held.ts`). A worker whose thread holds anything writes `memory worker <name> holds: …`. A name whose count only rises from line to line is the holder to read. | |
| 8 | 9 | - **The stall line.** Each loop-stall line ends with `heap X (was Y N s ago; …)`, the heap now against the last minute's sample. |
| 9 | 10 | - `fell: a collection ran` means the heap shrank by more than a tenth since the sample, so the collector ran inside that stall. Bun 1.4.4 reports no collector time, so this is inferred, not measured. |
| 10 | 11 | - `did not fall: no collection seen` means the stall was not the collector's. |
| @@ -15,6 +16,7 @@ Since Bun 1.4.4 the serving process has held several GiB after hours. This chang | ||
| 15 | 16 | ## What to do when the need shows |
| 16 | 17 | |
| 17 | 18 | 1. The need names the VM and the part. To see the curve, read the `memory main` and `memory worker` lines around the time named: heap up means JS objects, heap flat with RSS up means native memory, a worker line up means that worker. |
| 19 | Then read the `memory holds:` lines across the same minutes: a holder that grew with the heap is named there with its count. | |
| 18 | 20 | 2. Check the stall lines for `fell: a collection ran`: stalls that are collections point at the heap. |
| 19 | 21 | 3. Restart the forge to release it, and report the lines with the issue. |
| 20 | 22 | |
| @@ -24,6 +26,14 @@ The need clears by itself once RSS falls under twice the baseline. A process tha | ||
| 24 | 26 | |
| 25 | 27 | `src/perf/memory_watch.ts` owns the readings (`heapSize()` and `memoryUsage()`, constant time; never `heapStats()`, which held the loop ~280 ms a minute, #11118). `vitals()` takes the watch as `watch`, as it takes `disk`. Each worker entry calls `workerMemory(<name>)` once at start; `FORGE_WORKER_MEMORY_MS` shortens the minute for tests. The same minute timer that writes a worker's line posts its reading (`{ name, heap, current }`) on the `forge-memory` `BroadcastChannel` (`tellReading`); the serving VM's memory sampler listens (`hearWorkers`), validates each message (`workerReadingOf`: a lower-case name, not `main`, finite byte counts) and notes it. Asks carry nothing extra, and at most 64 workers are kept. Tests: `test/memory-watch.test.ts`, `test/worker-memory-lines.test.ts`, `test/vitals.test.ts`, `test/tui-memory-growth.test.ts`. |
| 26 | 28 | |
| 27 | ## Not done | |
| 29 | ## What grew, and what holds it now | |
| 28 | 30 | |
| 29 | Finding what grows and fixing it. | |
| 31 | Found on a copy of production's data with no network, production's own work on (apps, crew, checks, the picture keeper) and a replay of pages, API and MCP calls, event streams and pushes; object counts by type from `bun:jsc` `heapStats()` after a full collection, and heap snapshots, at 5, 20 and 60 minutes. | |
| 32 | ||
| 33 | - **The turn gate kept every request an app's host answered** (`src/http/app_turn.ts`, in since 0.7.598, one release before Bun 1.4.4). The listener began a turn for every request and only the forge's own routes ended it; a request answered ahead of the routes (an app's host through the relay, a refusal) stayed in the gate's Map, with its `Request`, `Headers` and `AbortSignal`, until an app's own token next waited on a quiet thread, which on production is never. On the copy, with nobody visiting an app, the looks' Chromium alone asked previews for about 450 requests a minute. The listener now leaves the gate in its `finally`, whatever door answered: the gate holds what is in flight. Bun holds nothing here: a served `Request` is collected on 1.3.14 and 1.4.4 alike once nothing of ours keeps it. | |
| 34 | - **The plan and lane caches kept answers nobody could ask for again** (`src/web/map_cache.ts`, `src/view/cache.ts`). A plan was keyed by its stamp, which moves with every run that starts or spends, and a lane by its repository's latest event; an entry past its minute was only dropped when the same key was read again or the count (64 plans, 128 lanes) pushed it out. A plan holds its repository's whole graph: the copy held ten graphs of the busiest repository (4,593 issues each) after an hour of page views, and the cache had room for 64. Now there is one plan per repository, spend visibility and day, and one lane per repository, scope and day, each replaced when its stamp moves; and every view cache drops what is past its time on the first write after `SWEEP_MS` (30 s). | |
| 35 | - **A hosted forge that slept left its request counter and route timing behind** (`src/http/server.ts`), one more of each with every wake, each keeping the closed database and a flush that retried against it. They now go when the forge closes, written out first. | |
| 36 | ||
| 37 | A new long-lived collection names itself in `heldOf(app)` (a forge's) or `threadHeld` (the thread's), with a bound and a test of its eviction. Named today: `turn`, `requests in flight`, `event streams`, `request counters`, `route timings`, `map plans`, `map lanes`, `map layouts`, and four the crew keeps per pull or per check that nothing sweeps yet (`autonomy conflicted pulls`, `finishing said`, `checks retried`, `app links said`): each is a short string per entry, and its line says when one is worth a sweep. | |
| 38 | ||
| 39 | Tests: `test/app-turn.test.ts`, `test/cache-holds.test.ts`, `test/forge-sleep.test.ts`, `test/held.test.ts`. |
src/boards/memory.ts +5 −1
| @@ -5,7 +5,9 @@ | ||
| 5 | 5 | * directory), as a count, a sum and a max. RSS rising while the |
| 6 | 6 | * heap stays flat is native memory growth. A week of hours is kept. |
| 7 | 7 | * |
| 8 | * Each tick is also one log line (`perf/memory_watch.ts`, #12901). | |
| 8 | * Each tick is also one log line (`perf/memory_watch.ts`, #12901), and a second that | |
| 9 | * names what the forge holds, each holder with its entry count (`perf/held.ts`): a Map | |
| 10 | * that only grows reads there by its name. | |
| 9 | 11 | * |
| 10 | 12 | * `heapSize()`, never `heapStats().heapSize`: the same byte count, but |
| 11 | 13 | * `heapStats()` first counts every object on the heap by type, which held the |
| @@ -15,6 +17,7 @@ import type { Database } from "bun:sqlite"; | ||
| 15 | 17 | import type { App } from "../app.ts"; |
| 16 | 18 | import { quietly } from "../core/write_clock.ts"; |
| 17 | 19 | import { during } from "../perf/jobs.ts"; |
| 20 | import { heldLine, heldNow } from "../perf/held.ts"; | |
| 18 | 21 | import { hearWorkers, memoryLine, memoryWatch, readNow } from "../perf/memory_watch.ts"; |
| 19 | 22 | |
| 20 | 23 | export const MEMORY_EVERY_MS = 60_000; |
| @@ -74,6 +77,7 @@ export function startMemorySampler(app: App, everyMs = MEMORY_EVERY_MS): MemoryS | ||
| 74 | 77 | recordMemory(app.db, reading.rss, reading.heap, now); |
| 75 | 78 | memoryWatch.note(reading.rss, reading.heap, now, reading.current); |
| 76 | 79 | app.log.info(memoryLine(reading, process.uptime() * 1000)); |
| 80 | app.log.info(heldLine(heldNow(app))); | |
Added in 57e2566: |
||
| 77 | 81 | } catch (error) { |
| 78 | 82 | app.log.warn("memory sample failed", { error: String(error) }); |
| 79 | 83 | } |
src/crew/autonomy.ts +2
| @@ -157,6 +157,7 @@ import { passLanded } from "./box_pass.ts"; | ||
| 157 | 157 | import { reviewersForRepos, reviewsOff, sendToReview, sweepReviews } from "./review.ts"; |
| 158 | 158 | import { lastWaits, noteWaits, tallyWaits, type Wait, waitWords } from "./waits.ts"; |
| 159 | 159 | import { bootCounts } from "../kernel/boots.ts"; |
| 160 | import { threadHeld } from "../perf/held.ts"; | |
| 160 | 161 | |
| 161 | 162 | const log = createLog("autonomy"); |
| 162 | 163 | |
| @@ -1597,6 +1598,7 @@ async function nextOf( | ||
| 1597 | 1598 | |
| 1598 | 1599 | /** Pull id → the `head:base` tips a merge last conflicted between. */ |
| 1599 | 1600 | const conflicted = new Map<string, string>(); |
| 1601 | threadHeld.add("autonomy conflicted pulls", () => conflicted.size); | |
| 1600 | 1602 | |
| 1601 | 1603 | /** A merge refusal in words a log can carry. */ |
| 1602 | 1604 | function mergeWhy(error: AccessError | MergePullError): string { |
src/crew/dispatch.ts +2
| @@ -140,6 +140,7 @@ import { STALE_AFTER_MS } from "../runs/lease.ts"; | ||
| 140 | 140 | import { stoppedMark } from "../spawn/stopped.ts"; |
| 141 | 141 | import { abortAgentRun } from "./running.ts"; |
| 142 | 142 | import { during } from "../perf/jobs.ts"; |
| 143 | import { threadHeld } from "../perf/held.ts"; | |
| 143 | 144 | |
| 144 | 145 | const log = createLog("crew"); |
| 145 | 146 | |
| @@ -1171,6 +1172,7 @@ function recheckWaiting(app: App, repo: Repo, state: unknown): void { | ||
| 1171 | 1172 | |
| 1172 | 1173 | /** Checks already run again once for a head; a second red is real. */ |
| 1173 | 1174 | const retriedChecks = new Set<string>(); |
| 1175 | threadHeld.add("checks retried", () => retriedChecks.size); | |
| 1174 | 1176 | |
| 1175 | 1177 | /** What a failed check led to, once the crew had read it. */ |
| 1176 | 1178 | export type CheckAnswer = "reran" | "reworked" | "parked" | "waiting"; |
src/crew/finishing.ts +2
| @@ -61,6 +61,7 @@ import { HANDOVER_NOTE } from "./guard.ts"; | ||
| 61 | 61 | import { interruptionOf } from "./interrupted.ts"; |
| 62 | 62 | import { pickBuilder } from "./route.ts"; |
| 63 | 63 | import { NO_BUILDER_ON_CREW, WAITS_FOR_BUILDER_SLOT } from "./words.ts"; |
| 64 | import { threadHeld } from "../perf/held.ts"; | |
| 64 | 65 | |
| 65 | 66 | const log = createLog("crew"); |
| 66 | 67 | |
| @@ -344,6 +345,7 @@ function handOver(app: App, repo: Repo, pull: Pull, issue: Issue, files: readonl | ||
| 344 | 345 | |
| 345 | 346 | /** The last why-not said for a pull, so a stall is said once until it changes. */ |
| 346 | 347 | const said = new Map<string, string>(); |
| 348 | threadHeld.add("finishing said", () => said.size); | |
| 347 | 349 | |
| 348 | 350 | /** What each repository's pass last said it read, so its entry is said when that changes. */ |
| 349 | 351 | const entered = new Map<string, string>(); |
src/http/app_turn.ts +34 −1
| @@ -13,12 +13,26 @@ | ||
| 13 | 13 | * |
| 14 | 14 | * Streams (a long poll, `/events`) are older than `YOUNG_MS` and hold a |
| 15 | 15 | * connection, not the thread. Plain data, one object per process. |
| 16 | * | |
| 17 | * The gate holds a request only while it is open (#12901): every door ends or leaves it, | |
| 18 | * the listener's own `finally` included, so what it holds is what is in flight, and past | |
| 19 | * `OPEN_MAX` it forgets the ones too old to matter, whoever forgot to end them. It held | |
| 20 | * every request an app's host answered (the relay never ended one) until an app's own | |
| 21 | * token next asked on a quiet thread: 27,000 an hour on a copy of production, from the | |
| 22 | * looks' Chromium alone, each with its Request, Headers and AbortSignal. | |
| 16 | 23 | */ |
| 17 | 24 | |
| 25 | import { heldOf } from "../perf/held.ts"; | |
| 26 | ||
| 18 | 27 | export const YOUNG_MS = 1_000; |
| 19 | 28 | export const BURST_MS = 50; |
| 20 | 29 | export const LOOK_MS = 20; |
| 21 | 30 | export const BOUND_MS = 1_000; |
| 31 | /** | |
| 32 | * The most requests the gate keeps before it forgets the old ones itself: a request | |
| 33 | * older than `YOUNG_MS` holds a connection, not the thread, so it is nobody's reason to wait. | |
| 34 | */ | |
| 35 | export const OPEN_MAX = 4_096; | |
| 22 | 36 | |
| 23 | 37 | export interface Turn { |
| 24 | 38 | /** A request arrived: it is a person's until it says otherwise. */ |
| @@ -27,6 +41,10 @@ export interface Turn { | ||
| 27 | 41 | machine(key: object): void; |
| 28 | 42 | /** It is answered. */ |
| 29 | 43 | end(key: object, at?: number): void; |
| 44 | /** It left by a door that does not count it (an app's host, a refusal): no burst follows. */ | |
| 45 | leave(key: object): void; | |
| 46 | /** How many requests the gate holds now: the open ones. */ | |
| 47 | size(): number; | |
| 30 | 48 | /** |
| 31 | 49 | * Resolve at once when nobody above is waiting, else when they are done or after the bound. |
| 32 | 50 | * The wait is counted in `into`: the gate's own totals, or the forge's whose request it was. |
| @@ -65,6 +83,13 @@ export function createTurn( | ||
| 65 | 83 | }; |
| 66 | 84 | return { |
| 67 | 85 | begin(key, at = now()) { |
| 86 | // Begun in time order, so the walk ends at the first young one: a full gate costs | |
| 87 | // one look at what it forgets, never a scan of what it keeps. | |
| 88 | if (open.size >= OPEN_MAX) | |
| 89 | for (const [held, since] of open) { | |
| 90 | if (at - since < YOUNG_MS) break; | |
| 91 | open.delete(held); | |
| 92 | } | |
| 68 | 93 | open.set(key, at); |
| 69 | 94 | }, |
| 70 | 95 | machine(key) { |
| @@ -73,6 +98,10 @@ export function createTurn( | ||
| 73 | 98 | end(key, at = now()) { |
| 74 | 99 | if (open.delete(key)) lastEnd = at; |
| 75 | 100 | }, |
| 101 | leave(key) { | |
| 102 | open.delete(key); | |
| 103 | }, | |
| 104 | size: () => open.size, | |
| 76 | 105 | async wait(into = stats) { |
| 77 | 106 | if (!wanted()) return; |
| 78 | 107 | const since = now(); |
| @@ -99,7 +128,11 @@ export function createTurn( | ||
| 99 | 128 | const gates = new WeakMap<object, Turn>(); |
| 100 | 129 | export function turnOf(app: object): Turn { |
| 101 | 130 | let gate = gates.get(app); |
| 102 | if (gate === undefined) gates.set(app, (gate = createTurn())); | |
| 131 | if (gate === undefined) { | |
| 132 | const made = createTurn(); | |
| 133 | gates.set(app, (gate = made)); | |
| 134 | heldOf(app).add("turn", made.size); | |
| 135 | } | |
| 103 | 136 | return gate; |
| 104 | 137 | } |
| 105 | 138 |
src/http/server.ts +26 −2
| @@ -560,6 +560,7 @@ import { | ||
| 560 | 560 | type RouteTiming, |
| 561 | 561 | } from "../core/route_timing.ts"; |
| 562 | 562 | import { captureStore, inFlight, liveOf, readLedger } from "../perf/live.ts"; |
| 563 | import { heldOf } from "../perf/held.ts"; | |
| 563 | 564 | import { childLabel, markChildRequest, noteChildRequest } from "../perf/child_time.ts"; |
| 564 | 565 | import { parseHostedChildName } from "../spawn/placement.ts"; |
| 565 | 566 | import { childHost } from "../spawn/door.ts"; |
| @@ -2704,6 +2705,23 @@ export async function serve(app: App): Promise<Servers> { | ||
| 2704 | 2705 | // map increment; a refused event would be a programming error, not worth a 500. |
| 2705 | 2706 | const counters = new Map<Database, UsageCounter>(); |
| 2706 | 2707 | const timings = new Map<Database, RouteTiming>(); |
| 2708 | heldOf(app).add("request counters", () => counters.size); | |
| 2709 | heldOf(app).add("route timings", () => timings.size); | |
| 2710 | // A hosted forge that closes takes its counter and its timing with it, written out | |
| 2711 | // while its database is still open (#12901): each was kept for as long as the parent | |
| 2712 | // ran, one more with every wake, its flush retrying against a closed database. | |
| 2713 | const unwatchCounts = watchHostedChildClose(app, (child) => { | |
| 2714 | for (const kept of [counters, timings]) { | |
| 2715 | const one = kept.get(child.db); | |
| 2716 | if (one === undefined) continue; | |
| 2717 | kept.delete(child.db); | |
| 2718 | try { | |
| 2719 | one.stop(); | |
| 2720 | } catch (error) { | |
| 2721 | app.log.warn("could not flush a closing forge's request counts", { error: String(error) }); | |
| 2722 | } | |
| 2723 | } | |
| 2724 | }); | |
| 2707 | 2725 | // The minute ledger's history: one write a minute, never on a request. |
| 2708 | 2726 | const trafficWriter = createTrafficWriter(app.db, liveOf(app).perfRecord.ledger, app.log); |
| 2709 | 2727 | // Page views are buffered by the beacon doors and written here; GitHub's traffic is pulled daily when a token is set. |
| @@ -3025,8 +3043,13 @@ export async function serve(app: App): Promise<Servers> { | ||
| 3025 | 3043 | const request = parseableRequest(raw); |
| 3026 | 3044 | const bound = request === raw ? server : answeringFor(server, request, raw); |
| 3027 | 3045 | return inFlight.during(request, performance.now(), async () => { |
| 3028 | const response = await handle(request, bound); | |
| 3029 | return handedOver && response !== undefined ? withHeader(response, "connection", "close") : response; | |
| 3046 | try { | |
| 3047 | const response = await handle(request, bound); | |
| 3048 | return handedOver && response !== undefined ? withHeader(response, "connection", "close") : response; | |
| 3049 | } finally { | |
| 3050 | // Whatever door answered: the gate holds no request past its answer (#12901). | |
| 3051 | turnOf(app).leave(request); | |
| 3052 | } | |
| 3030 | 3053 | }); |
| 3031 | 3054 | }, |
| 3032 | 3055 | // A last resort: `handle` answers a throw itself, so it is counted. |
| @@ -3085,6 +3108,7 @@ export async function serve(app: App): Promise<Servers> { | ||
| 3085 | 3108 | for (const key of [...new Set([...childHttps.keys(), ...childInternals.keys()])]) letGo(key); |
| 3086 | 3109 | closeHostedChildren(app); |
| 3087 | 3110 | unwatchChildren(); |
| 3111 | unwatchCounts(); | |
| 3088 | 3112 | void stopCellChildren(app); |
| 3089 | 3113 | flushAccess(); |
| 3090 | 3114 | web.stop(true); |
src/http/sse.ts +2
| @@ -26,6 +26,7 @@ import { mayReadLogOfRun } from "../actions/runs.ts"; | ||
| 26 | 26 | import { getRun } from "../core/runs.ts"; |
| 27 | 27 | import { frameOf, tapRunLogs } from "../runs/log.ts"; |
| 28 | 28 | import type { ForgeEvent } from "../core/events.ts"; |
| 29 | import { threadHeld } from "../perf/held.ts"; | |
| 29 | 30 | |
| 30 | 31 | const HEARTBEAT_MS = 25_000; |
| 31 | 32 | |
| @@ -66,6 +67,7 @@ export const resetSseDroppedCount = (): void => { | ||
| 66 | 67 | * replaced this one (docs/ARCHITECTURE.md, "Two processes at once"). |
| 67 | 68 | */ |
| 68 | 69 | const openStreams = new Set<() => void>(); |
| 70 | threadHeld.add("event streams", () => openStreams.size); | |
| 69 | 71 | |
| 70 | 72 | /** How soon a drained stream's client reconnects, in milliseconds. */ |
| 71 | 73 | export const DRAIN_RETRY_MS = 1000; |
src/perf/held.ts added +82
| @@ -0,0 +1,82 @@ | ||
| 1 | /** | |
| 2 | * What the forge holds in memory, by name (#12901): every long-lived Map, Set or list | |
| 3 | * names itself here with a way to count it, and the minute memory line | |
| 4 | * (`boards/memory.ts`) writes the largest counts, so a holder that only grows reads in | |
| 5 | * the log by its name. A holder that belongs to a forge names itself in that forge's | |
| 6 | * registry when it is made (`heldOf(app)`); one a module keeps once for the thread names | |
| 7 | * itself as the module loads (`threadHeld`). The serving forge's line reads both, and a | |
| 8 | * worker's line reads its own thread's (`perf/memory_watch.ts`). | |
| 9 | * | |
| 10 | * A count is a `size` read, constant time; nothing here walks a collection, and the | |
| 11 | * registry keeps only the name and the getter. A name given again replaces the earlier | |
| 12 | * getter, so the registry is as large as the names in the source, never as the work done. | |
| 13 | */ | |
| 14 | ||
| 15 | export interface HeldCount { | |
| 16 | readonly name: string; | |
| 17 | /** Entries now; -1 when the getter threw. */ | |
| 18 | readonly size: number; | |
| 19 | } | |
| 20 | ||
| 21 | export interface Held { | |
| 22 | /** Name a holder and how to count it. The same name again replaces the earlier count. */ | |
| 23 | add(name: string, size: () => number): void; | |
| 24 | /** Every holder's count now, largest first, then by name. */ | |
| 25 | read(): HeldCount[]; | |
| 26 | } | |
| 27 | ||
| 28 | export function createHeld(): Held { | |
| 29 | const sizes = new Map<string, () => number>(); | |
| 30 | return { | |
| 31 | add(name, size) { | |
| 32 | sizes.set(name, size); | |
| 33 | }, | |
| 34 | read() { | |
| 35 | const out: HeldCount[] = []; | |
| 36 | for (const [name, size] of sizes) { | |
| 37 | let n = -1; | |
| 38 | try { | |
| 39 | const read = size(); | |
| 40 | if (Number.isFinite(read)) n = read; | |
| 41 | } catch { | |
| 42 | // A holder that cannot be counted says so with -1 and never fails the line. | |
| 43 | } | |
| 44 | out.push({ name, size: n }); | |
| 45 | } | |
| 46 | return out.sort((a, b) => b.size - a.size || (a.name < b.name ? -1 : 1)); | |
| 47 | }, | |
| 48 | }; | |
| 49 | } | |
| 50 | ||
| 51 | const registries = new WeakMap<object, Held>(); | |
| 52 | ||
| 53 | /** This forge's registry, made at its first use. */ | |
| 54 | export function heldOf(app: object): Held { | |
| 55 | let held = registries.get(app); | |
| 56 | if (held === undefined) registries.set(app, (held = createHeld())); | |
| 57 | return held; | |
| 58 | } | |
| 59 | ||
| 60 | /** | |
| 61 | * What a module keeps once for the thread, named as the module loads. The names are the | |
| 62 | * source's, so this is written at load and read each minute, never by a request. | |
| 63 | */ | |
| 64 | export const threadHeld: Held = createHeld(); | |
| 65 | ||
| 66 | /** A forge's own holders and the thread's, as one reading: largest first. */ | |
| 67 | export function heldNow(app: object): HeldCount[] { | |
| 68 | return [...heldOf(app).read(), ...threadHeld.read()].sort((a, b) => b.size - a.size || (a.name < b.name ? -1 : 1)); | |
| 69 | } | |
| 70 | ||
| 71 | /** How many holders the minute line names. */ | |
| 72 | export const HELD_TOP = 12; | |
| 73 | ||
| 74 | /** | |
| 75 | * The minute line's words: `memory holds: sse 14, turn 3, in-flight 3` for the `top` | |
| 76 | * largest holders that hold anything; `memory holds: nothing` when none does. | |
| 77 | */ | |
| 78 | export function heldLine(counts: readonly HeldCount[], top: number = HELD_TOP): string { | |
| 79 | const named = counts.filter((c) => c.size !== 0).slice(0, top); | |
| 80 | if (named.length === 0) return "memory holds: nothing"; | |
| 81 | return `memory holds: ${named.map((c) => `${c.name} ${c.size < 0 ? "unread" : c.size}`).join(", ")}`; | |
| 82 | } |
src/perf/live.ts +2
| @@ -15,9 +15,11 @@ import { createReadLedger } from "./reads.ts"; | ||
| 15 | 15 | import { createStallBlame } from "./stall_blame.ts"; |
| 16 | 16 | import { createPerfRecord } from "./record.ts"; |
| 17 | 17 | import { createLagMinutes } from "./lag_minutes.ts"; |
| 18 | import { threadHeld } from "./held.ts"; | |
| 18 | 19 | |
| 19 | 20 | /** The requests the server is answering now (#11478): begun and ended around each one in `http/server.ts`. */ |
| 20 | 21 | export const inFlight = createInFlight<Request>((request) => childRequestLabel(request) ?? patternOf(request)); |
| 22 | threadHeld.add("requests in flight", inFlight.size); | |
| 21 | 23 | export const gitRecord = createGitRecord(); |
| 22 | 24 | /** The worst stall in this process's first five minutes, read by the release keeper (#8848). */ |
| 23 | 25 | export const bootStall = createBootStall(); |
src/perf/memory_watch.ts +20 −1
| @@ -12,6 +12,9 @@ | ||
| 12 | 12 | * posts to from the same minute timer that writes its line (`tellReading`, `hearWorkers`), |
| 13 | 13 | * so no ask and no hot-path message carries it; a worker silent for five minutes is not blamed. |
| 14 | 14 | * |
| 15 | * Beside each line, what is held by name (`perf/held.ts`): the serving forge's holders | |
| 16 | * and the thread's on `memory holds:`, a worker's own thread's on its second line. | |
| 17 | * | |
| 15 | 18 | * `heapStats()` counts every object on the heap by type and held the loop ~280 ms a |
| 16 | 19 | * minute on a 2M-object heap (#11118), so the minute line never calls it: `heapSize()` |
| 17 | 20 | * and `memoryUsage()` are constant-time. Bun 1.4.4 exposes no collector time (a |
| @@ -21,6 +24,7 @@ | ||
| 21 | 24 | */ |
| 22 | 25 | import { heapSize, memoryUsage as jscMemory } from "bun:jsc"; |
| 23 | 26 | import { createLog } from "../kernel/log.ts"; |
| 27 | import { heldLine, threadHeld, type HeldCount } from "./held.ts"; | |
| 24 | 28 | |
| 25 | 29 | const MIB = 1048576; |
| 26 | 30 | export const FIRST_HOUR_MS = 3_600_000; |
| @@ -309,8 +313,23 @@ export function hearWorkers(watch: Pick<MemoryWatch, "noteWorker"> = memoryWatch | ||
| 309 | 313 | }; |
| 310 | 314 | } |
| 311 | 315 | |
| 316 | /** | |
| 317 | * A worker's second line, when its thread holds anything by name (`perf/held.ts`): | |
| 318 | * `memory worker engine holds: checks retried 12`. Undefined when it holds nothing. | |
| 319 | */ | |
| 320 | export function workerHeldLine(name: string, counts: readonly HeldCount[] = threadHeld.read()): string | undefined { | |
| 321 | const line = heldLine(counts); | |
| 322 | return line === heldLine([]) ? undefined : line.replace(/^memory holds:/, `memory worker ${name} holds:`); | |
| 323 | } | |
| 324 | ||
| 312 | 325 | /** Start a worker's own minute line (an unref'd timer inside the worker). Returns the stop. */ |
| 313 | export function startWorkerMemory(name: string, write: (line: string) => void, everyMs = 60_000): () => void { | |
| 326 | export function startWorkerMemory(name: string, out: (line: string) => void, everyMs = 60_000): () => void { | |
| 327 | // The reading's line, then what the thread holds by name, when it holds anything. | |
| 328 | const write = (line: string): void => { | |
| 329 | out(line); | |
| 330 | const held = workerHeldLine(name); | |
| 331 | if (held !== undefined) out(held); | |
| 332 | }; | |
| 314 | 333 | const timer = setInterval(() => write(workerMemoryLine(name, tellReading(name, readNow()))), everyMs); |
| 315 | 334 | (timer as { unref?: () => void }).unref?.(); |
| 316 | 335 | return () => clearInterval(timer); |
src/spawn/app_links.ts +2
| @@ -57,6 +57,7 @@ import { repoDir } from "../git/repos.ts"; | ||
| 57 | 57 | import { readPrivate, writePrivate } from "../kernel/file_modes.ts"; |
| 58 | 58 | import { seal } from "../kernel/secret.ts"; |
| 59 | 59 | import { DEFAULT_MODE, originOf } from "./chain.ts"; |
| 60 | import { threadHeld } from "../perf/held.ts"; | |
| 60 | 61 | |
| 61 | 62 | /** One app in `parent.json`, as the two sides write it. */ |
| 62 | 63 | export interface AppEntry { |
| @@ -529,6 +530,7 @@ export function reasonOf(error: unknown): string { | ||
| 529 | 530 | |
| 530 | 531 | /** What was said in the log already: a state that holds is said once, not each minute. */ |
| 531 | 532 | const said = new Set<string>(); |
| 533 | threadHeld.add("app links said", () => said.size); | |
| 532 | 534 | |
| 533 | 535 | /** Say one app's condition once, while it holds; the same words again are not repeated. */ |
| 534 | 536 | function say(app: App, level: "info" | "warn", forge: string, upstream: string, words: string): void { |
src/view/cache.ts
Binary file not shown.
src/web/map_cache.ts +24 −9
| @@ -10,6 +10,7 @@ | ||
| 10 | 10 | */ |
| 11 | 11 | |
| 12 | 12 | import type { Database } from "bun:sqlite"; |
| 13 | import { threadHeld } from "../perf/held.ts"; | |
| 13 | 14 | import { ViewCache } from "../view/cache.ts"; |
| 14 | 15 | import type { Graph } from "../core/dependencies.ts"; |
| 15 | 16 | |
| @@ -20,6 +21,8 @@ const LAYOUT_TTL_MS = 60 * 60_000; | ||
| 20 | 21 | |
| 21 | 22 | const lanes = new ViewCache({ maxEntries: 128 }); |
| 22 | 23 | const layouts = new ViewCache({ maxEntries: 256 }); |
| 24 | threadHeld.add("map lanes", () => lanes.stats().entries); | |
| 25 | threadHeld.add("map layouts", () => layouts.stats().entries); | |
| 23 | 26 | |
| 24 | 27 | /** What `sealedScope` computed, as a string that goes into a key. */ |
| 25 | 28 | export function scopeKey(scope: { sealed?: true; sealedBy?: string }): string { |
| @@ -59,20 +62,25 @@ const utcDay = (now: Date): string => now.toISOString().slice(0, 10); | ||
| 59 | 62 | /** |
| 60 | 63 | * A lane's cached build, or `build()`'s result kept for next time. `null` |
| 61 | 64 | * is a cached answer too: a repository with nothing to draw. |
| 65 | * | |
| 66 | * One lane per repository, scope and day (#12901): the head it was built at is | |
| 67 | * kept beside it, and a new head replaces it. Keyed by the head, every event | |
| 68 | * left the lane before it in the cache until 128 newer ones pushed it out. | |
| 62 | 69 | */ |
| 63 | 70 | export function cachedLane<T>( |
| 64 | 71 | key: { repoId: string; head: number; scope: string; now: Date }, |
| 65 | 72 | build: () => T, |
| 66 | 73 | ): T { |
| 67 | const k = `lane:${key.repoId}:${key.head}:${key.scope}:${utcDay(key.now)}`; | |
| 68 | const kept = lanes.get<{ value: T }>(k); | |
| 69 | if (kept !== undefined) return kept.value; | |
| 74 | const k = `lane:${key.repoId}:${key.scope}:${utcDay(key.now)}`; | |
| 75 | const kept = lanes.get<{ head: number; value: T }>(k); | |
| 76 | if (kept !== undefined && kept.head === key.head) return kept.value; | |
| 70 | 77 | const value = build(); |
| 71 | lanes.set(k, { value }, LANE_TTL_MS); | |
| 78 | lanes.set(k, { head: key.head, value }, LANE_TTL_MS); | |
| 72 | 79 | return value; |
| 73 | 80 | } |
| 74 | 81 | |
| 75 | 82 | const plans = new ViewCache({ maxEntries: 64 }); |
| 83 | threadHeld.add("map plans", () => plans.stats().entries); | |
| 76 | 84 | let planBuilds = 0; |
| 77 | 85 | |
| 78 | 86 | /** |
| @@ -82,6 +90,12 @@ let planBuilds = 0; | ||
| 82 | 90 | * lifetime. The plan holds no |
| 83 | 91 | * viewer: who may read the repository and the seats are the caller's, every |
| 84 | 92 | * request. |
| 93 | * | |
| 94 | * One plan per repository, spend visibility and day (#12901): the stamp is kept | |
| 95 | * beside it, and a new stamp replaces it. A plan holds the repository's whole | |
| 96 | * graph, and the stamp moves with every run that starts or spends: keyed by | |
| 97 | * the stamp, the cache held 64 graphs of the busiest repository, 63 of them | |
| 98 | * for a stamp that could not come back. | |
| 85 | 99 | */ |
| 86 | 100 | export function cachedPlan<T>( |
| 87 | 101 | key: { repoId: string; stamp: string | undefined; spend: boolean; now: Date }, |
| @@ -92,12 +106,12 @@ export function cachedPlan<T>( | ||
| 92 | 106 | planBuilds++; |
| 93 | 107 | return build(); |
| 94 | 108 | } |
| 95 | const k = `plan:${key.repoId}:${key.stamp}:${key.spend ? "spend" : "plain"}:${utcDay(key.now)}`; | |
| 96 | const kept = plans.get<{ value: T }>(k); | |
| 97 | if (kept !== undefined) return kept.value; | |
| 109 | const k = `plan:${key.repoId}:${key.spend ? "spend" : "plain"}:${utcDay(key.now)}`; | |
| 110 | const kept = plans.get<{ stamp: string; value: T }>(k); | |
| 111 | if (kept !== undefined && kept.stamp === key.stamp) return kept.value; | |
| 98 | 112 | planBuilds++; |
| 99 | 113 | const value = build(); |
| 100 | plans.set(k, { value }, LANE_TTL_MS); | |
| 114 | plans.set(k, { stamp: key.stamp, value }, LANE_TTL_MS); | |
| 101 | 115 | return value; |
| 102 | 116 | } |
| 103 | 117 | |
| @@ -157,4 +171,5 @@ export function clearMapCache(): void { | ||
| 157 | 171 | export const mapCacheStats = (): { |
| 158 | 172 | lanes: ReturnType<ViewCache["stats"]>; |
| 159 | 173 | layouts: ReturnType<ViewCache["stats"]>; |
| 160 | } => ({ lanes: lanes.stats(), layouts: layouts.stats() }); | |
| 174 | plans: ReturnType<ViewCache["stats"]>; | |
| 175 | } => ({ lanes: lanes.stats(), layouts: layouts.stats(), plans: plans.stats() }); |
test/app-turn.test.ts +68 −1
| @@ -14,7 +14,7 @@ import { getRepo } from "../src/core/repos.ts"; | ||
| 14 | 14 | import { createToken } from "../src/core/tokens.ts"; |
| 15 | 15 | import { createUser } from "../src/core/users.ts"; |
| 16 | 16 | import { serve, type Servers } from "../src/http/server.ts"; |
| 17 | import { createTurn, turnStatsOf } from "../src/http/app_turn.ts"; | |
| 17 | import { createTurn, OPEN_MAX, turnOf, turnStatsOf, YOUNG_MS } from "../src/http/app_turn.ts"; | |
| 18 | 18 | import { askingOf, createAsking, gapWords } from "../src/perf/asking.ts"; |
| 19 | 19 | |
| 20 | 20 | describe("the turn", () => { |
| @@ -72,6 +72,50 @@ describe("the turn", () => { | ||
| 72 | 72 | expect(turn.stats.waited).toBe(0); |
| 73 | 73 | }); |
| 74 | 74 | |
| 75 | test("the gate holds a request only while it is open: ended or left, it is gone (#12901)", async () => { | |
| 76 | let t = 0; | |
| 77 | const turn = createTurn(() => t, 100, async (ms) => void (t += ms)); | |
| 78 | // Ten thousand requests an app's host answered: begun, and gone by the relay's door. | |
| 79 | for (let i = 0; i < 10_000; i++) { | |
| 80 | const relayed = {}; | |
| 81 | turn.begin(relayed, t); | |
| 82 | expect(turn.size()).toBe(1); | |
| 83 | turn.leave(relayed); | |
| 84 | t += 1; | |
| 85 | } | |
| 86 | expect(turn.size()).toBe(0); | |
| 87 | // Leaving is not an answer a burst follows: an app's own call goes at once. | |
| 88 | await turn.wait(); | |
| 89 | expect(turn.stats.waited).toBe(0); | |
| 90 | // An answer ends it the same way, and a key the gate never held is ignored. | |
| 91 | const person = {}; | |
| 92 | turn.begin(person, t); | |
| 93 | turn.end(person, t); | |
| 94 | turn.leave(person); | |
| 95 | turn.leave({}); | |
| 96 | expect(turn.size()).toBe(0); | |
| 97 | }); | |
| 98 | ||
| 99 | test("a gate nobody ends is bounded all the same: past its size the old ones are forgotten", async () => { | |
| 100 | let t = 0; | |
| 101 | const turn = createTurn(() => t, 100, async (ms) => void (t += ms)); | |
| 102 | // A hundred thousand requests begun and never ended, a millisecond apart. | |
| 103 | for (let i = 0; i < 100_000; i++) { | |
| 104 | turn.begin({}, t); | |
| 105 | t += 1; | |
| 106 | expect(turn.size()).toBeLessThanOrEqual(OPEN_MAX); | |
| 107 | } | |
| 108 | // What is left began in the last second or filled the gate since its last sweep. | |
| 109 | expect(turn.size()).toBeGreaterThan(0); | |
| 110 | // The young ones still count: an app's call waits for them, to its bound. | |
| 111 | await turn.wait(); | |
| 112 | expect(turn.stats.waited).toBe(1); | |
| 113 | // A second of quiet, and the next request finds a gate with nobody in it to wait for. | |
| 114 | t += YOUNG_MS; | |
| 115 | await turn.wait(); | |
| 116 | expect(turn.stats.waited).toBe(1); | |
| 117 | }); | |
| 118 | ||
| 75 | 119 | test("the gate is the thread's, the count is the forge's: a wait is counted for the forge whose request it was", async () => { |
| 76 | 120 | let t = 0; |
| 77 | 121 | const turn = createTurn(() => t, 100, async (ms) => void (t += ms)); |
| @@ -129,6 +173,29 @@ describe("who is asking", () => { | ||
| 129 | 173 | }); |
| 130 | 174 | }); |
| 131 | 175 | |
| 176 | describe("the gate behind the listener", () => { | |
| 177 | test("a request answered by a door that counts nothing is not held past its answer (#12901)", async () => { | |
| 178 | // A read-only forge refuses a write before any route: that door never ended the turn, | |
| 179 | // like an app's host and every other answer made ahead of the routes. | |
| 180 | const dir = await mkdtemp(join(tmpdir(), "forge-turn-door-")); | |
| 181 | const app = createApp({ dataDir: dir, port: 0, readOnly: true }); | |
| 182 | const servers = await serve(app); | |
| 183 | try { | |
| 184 | for (let i = 0; i < 5; i++) { | |
| 185 | const refused = await fetch(servers.url + "/api/v1/repos", { method: "POST", body: "{}" }); | |
| 186 | expect(refused.status).toBe(503); | |
| 187 | await refused.text(); | |
| 188 | } | |
| 189 | const page = await fetch(servers.url + "/healthz"); | |
| 190 | await page.text(); | |
| 191 | expect(turnOf(app).size()).toBe(0); | |
| 192 | } finally { | |
| 193 | await servers.stop(); | |
| 194 | await rm(dir, { recursive: true, force: true }); | |
| 195 | } | |
| 196 | }); | |
| 197 | }); | |
| 198 | ||
| 132 | 199 | describe("the forge answers an app token", () => { |
| 133 | 200 | let app: App; |
| 134 | 201 | let servers: Servers; |
test/cache-holds.test.ts added +86
| @@ -0,0 +1,86 @@ | ||
| 1 | /** | |
| 2 | * A cache holds what can still be asked for (#12901): an entry past its time goes on the next | |
| 3 | * sweep, and a plan or a lane is one per repository, replaced when its stamp moves. | |
| 4 | */ | |
| 5 | import { beforeEach, describe, expect, test } from "bun:test"; | |
| 6 | import { SWEEP_MS, ViewCache } from "../src/view/cache.ts"; | |
| 7 | import { cachedLane, cachedPlan, clearMapCache, mapCacheStats, planBuildCount } from "../src/web/map_cache.ts"; | |
| 8 | import { threadHeld } from "../src/perf/held.ts"; | |
| 9 | ||
| 10 | describe("the view cache drops what is past its time", () => { | |
| 11 | test("entries nobody reads again go on the first write after the sweep is due, not at the count", () => { | |
| 12 | let t = 1_000_000; | |
| 13 | const cache = new ViewCache({ maxEntries: 10_000, now: () => t }); | |
| 14 | // A thousand answers under keys that never come back, each good for a minute. | |
| 15 | for (let i = 0; i < 1_000; i++) cache.set(`answer:${i}`, { i }, 60_000, 10); | |
| 16 | expect(cache.stats()).toMatchObject({ entries: 1_000, bytes: 10_000 }); | |
| 17 | // Past their time, and no write yet: still held. | |
| 18 | t += 61_000; | |
| 19 | expect(cache.stats().entries).toBe(1_000); | |
| 20 | cache.set("fresh", { i: -1 }, 60_000, 10); | |
| 21 | expect(cache.stats()).toMatchObject({ entries: 1, bytes: 10 }); | |
| 22 | expect(cache.get<{ i: number }>("fresh")).toEqual({ i: -1 }); | |
| 23 | }); | |
| 24 | ||
| 25 | test("a sweep leaves what is still good, and is one walk each SWEEP_MS however many writes come", () => { | |
| 26 | let t = 0; | |
| 27 | const cache = new ViewCache({ maxEntries: 10_000, now: () => t }); | |
| 28 | cache.set("short", 1, 1_000); | |
| 29 | cache.set("long", 2, 10 * SWEEP_MS); | |
| 30 | // Inside the sweep's interval an expired entry waits; the count and the bytes still bound it. | |
| 31 | t += 2_000; | |
| 32 | cache.set("another", 3, 10 * SWEEP_MS); | |
| 33 | expect(cache.stats().entries).toBe(3); | |
| 34 | t += SWEEP_MS; | |
| 35 | cache.set("last", 4, 10 * SWEEP_MS); | |
| 36 | expect(cache.stats().entries).toBe(3); | |
| 37 | expect(cache.get<number>("short")).toBeUndefined(); | |
| 38 | expect(cache.get<number>("long")).toBe(2); | |
| 39 | expect(cache.get<number>("another")).toBe(3); | |
| 40 | }); | |
| 41 | }); | |
| 42 | ||
| 43 | describe("a plan and a lane are one per repository", () => { | |
| 44 | beforeEach(() => clearMapCache()); | |
| 45 | const now = new Date("2026-11-15T08:00:00Z"); | |
| 46 | ||
| 47 | test("a hundred stamps of one repository's plan leave one plan, the newest", () => { | |
| 48 | const before = planBuildCount(); | |
| 49 | for (let stamp = 0; stamp < 100; stamp++) { | |
| 50 | const plan = cachedPlan({ repoId: "r1", stamp: `s${stamp}`, spend: false, now }, () => ({ stamp })); | |
| 51 | expect(plan.stamp).toBe(stamp); | |
| 52 | expect(mapCacheStats().plans.entries).toBe(1); | |
| 53 | } | |
| 54 | expect(planBuildCount()).toBe(before + 100); | |
| 55 | // The same stamp again is served; a stamp that moved back is built again, never served stale. | |
| 56 | expect(cachedPlan({ repoId: "r1", stamp: "s99", spend: false, now }, () => ({ stamp: -1 })).stamp).toBe(99); | |
| 57 | expect(cachedPlan({ repoId: "r1", stamp: "s98", spend: false, now }, () => ({ stamp: 980 })).stamp).toBe(980); | |
| 58 | expect(planBuildCount()).toBe(before + 101); | |
| 59 | // Each spend visibility and each repository keeps its own. | |
| 60 | cachedPlan({ repoId: "r1", stamp: "s98", spend: true, now }, () => ({ stamp: 1 })); | |
| 61 | cachedPlan({ repoId: "r2", stamp: "s98", spend: false, now }, () => ({ stamp: 2 })); | |
| 62 | expect(mapCacheStats().plans.entries).toBe(3); | |
| 63 | expect(threadHeld.read().find((h) => h.name === "map plans")?.size).toBe(3); | |
| 64 | // Inside a transaction there is no stamp, and nothing is kept. | |
| 65 | cachedPlan({ repoId: "r3", stamp: undefined, spend: false, now }, () => ({ stamp: 3 })); | |
| 66 | expect(mapCacheStats().plans.entries).toBe(3); | |
| 67 | }); | |
| 68 | ||
| 69 | test("a hundred events on one repository leave one lane per scope, built once per head", () => { | |
| 70 | let built = 0; | |
| 71 | for (let head = 1; head <= 100; head++) { | |
| 72 | for (const scope of ["all", "none"]) { | |
| 73 | const lane = cachedLane({ repoId: "r1", head, scope, now }, () => ({ head, n: ++built })); | |
| 74 | expect(lane.head).toBe(head); | |
| 75 | // Asked again at the same head: the kept lane, not a new build. | |
| 76 | expect(cachedLane({ repoId: "r1", head, scope, now }, () => ({ head: -1, n: ++built }))).toBe(lane); | |
| 77 | } | |
| 78 | expect(mapCacheStats().lanes.entries).toBe(2); | |
| 79 | } | |
| 80 | expect(built).toBe(200); | |
| 81 | // A repository with nothing to draw is a kept answer too. | |
| 82 | expect(cachedLane({ repoId: "r2", head: 5, scope: "all", now }, () => null)).toBeNull(); | |
| 83 | expect(cachedLane({ repoId: "r2", head: 5, scope: "all", now }, () => ({ head: 5, n: -1 }))).toBeNull(); | |
| 84 | expect(threadHeld.read().find((h) => h.name === "map lanes")?.size).toBe(3); | |
| 85 | }); | |
| 86 | }); |
test/forge-sleep.test.ts +20
| @@ -16,6 +16,7 @@ import { createSession } from "../src/core/sessions.ts"; | ||
| 16 | 16 | import { createUser } from "../src/core/users.ts"; |
| 17 | 17 | import { serve, type Servers } from "../src/http/server.ts"; |
| 18 | 18 | import { awakeCount, childAwake, gateClock, holdChild, resolveHostedChild, sleepIdleChildren, wakeGate } from "../src/spawn/host.ts"; |
| 19 | import { heldOf } from "../src/perf/held.ts"; | |
| 19 | 20 | import { removeForge, stopForge } from "../src/spawn/manage.ts"; |
| 20 | 21 | import { setLimit } from "../src/settings/limits.ts"; |
| 21 | 22 | import { awakeRefusal, personRefusal, WakeRate } from "../src/spawn/sleep.ts"; |
| @@ -109,6 +110,25 @@ describe("a hosted forge scales to zero", () => { | ||
| 109 | 110 | expect(repo.status).toBe(200); |
| 110 | 111 | }); |
| 111 | 112 | |
| 113 | test("a forge that sleeps takes its request counter and route timing with it, however often it wakes (#12901)", async () => { | |
| 114 | const kept = (name: string) => heldOf(parent).read().find((h) => h.name === name)?.size ?? 0; | |
| 115 | await sleepIdleChildren(parent, 600_000, later()); | |
| 116 | // Asleep: what the parent keeps for itself, one per database it has counted in. | |
| 117 | const counters = kept("request counters"); | |
| 118 | const timings = kept("route timings"); | |
| 119 | for (let wake = 0; wake < 3; wake++) { | |
| 120 | expect((await asChild("/api/v1/repos/bob/app")).status).toBe(200); | |
| 121 | expect(childAwake(parent, "homie")).toBe(true); | |
| 122 | expect(kept("request counters")).toBe(counters + 1); | |
| 123 | expect(kept("route timings")).toBe(timings + 1); | |
| 124 | expect(await sleepIdleChildren(parent, 600_000, later())).toEqual(["homie"]); | |
| 125 | expect(kept("request counters")).toBe(counters); | |
| 126 | expect(kept("route timings")).toBe(timings); | |
| 127 | } | |
| 128 | // And it wakes again: what it counted while awake was written before its database closed. | |
| 129 | expect((await asChild("/api/v1/repos/bob/app")).status).toBe(200); | |
| 130 | }); | |
| 131 | ||
| 112 | 132 | test("wakes on a git push, and the data from before the sleep is still there", async () => { |
| 113 | 133 | const repo = join(work, "app"); |
| 114 | 134 | expect((await git(["init", "-q", "-b", "main", repo], work)).code).toBe(0); |
test/held.test.ts added +106
| @@ -0,0 +1,106 @@ | ||
| 1 | /** | |
| 2 | * What the forge holds in memory is named and counted (#12901): one registry per forge, | |
| 3 | * read by the minute memory line. | |
| 4 | */ | |
| 5 | import { describe, expect, spyOn, test } from "bun:test"; | |
| 6 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 7 | import { tmpdir } from "node:os"; | |
| 8 | import { join } from "node:path"; | |
| 9 | import { createApp } from "../src/app.ts"; | |
| 10 | import { startMemorySampler } from "../src/boards/memory.ts"; | |
| 11 | import { createHeld, heldLine, heldNow, heldOf, HELD_TOP, threadHeld } from "../src/perf/held.ts"; | |
| 12 | import { workerHeldLine } from "../src/perf/memory_watch.ts"; | |
| 13 | ||
| 14 | describe("the registry of what is held", () => { | |
| 15 | test("each holder is counted when read, largest first, and a name given again replaces the earlier count", () => { | |
| 16 | const held = createHeld(); | |
| 17 | const a = new Map<number, number>(); | |
| 18 | const b = new Set<number>(); | |
| 19 | held.add("a", () => a.size); | |
| 20 | held.add("b", () => b.size); | |
| 21 | for (let i = 0; i < 3; i++) a.set(i, i); | |
| 22 | for (let i = 0; i < 7; i++) b.add(i); | |
| 23 | expect(held.read()).toEqual([{ name: "b", size: 7 }, { name: "a", size: 3 }]); | |
| 24 | a.clear(); | |
| 25 | expect(held.read()).toEqual([{ name: "b", size: 7 }, { name: "a", size: 0 }]); | |
| 26 | // The registry is as large as the names, however often a holder is made again. | |
| 27 | for (let i = 0; i < 1_000; i++) held.add("a", () => i); | |
| 28 | expect(held.read()).toEqual([{ name: "a", size: 999 }, { name: "b", size: 7 }]); | |
| 29 | }); | |
| 30 | ||
| 31 | test("a holder that cannot be counted reads -1 and never fails the read", () => { | |
| 32 | const held = createHeld(); | |
| 33 | held.add("broken", () => { | |
| 34 | throw new Error("gone"); | |
| 35 | }); | |
| 36 | held.add("nan", () => Number.NaN); | |
| 37 | held.add("fine", () => 2); | |
| 38 | expect(held.read()).toEqual([{ name: "fine", size: 2 }, { name: "broken", size: -1 }, { name: "nan", size: -1 }]); | |
| 39 | }); | |
| 40 | ||
| 41 | test("each forge has its own registry", () => { | |
| 42 | const one = {}; | |
| 43 | const other = {}; | |
| 44 | heldOf(one).add("x", () => 1); | |
| 45 | expect(heldOf(one)).toBe(heldOf(one)); | |
| 46 | expect(heldOf(one).read()).toEqual([{ name: "x", size: 1 }]); | |
| 47 | expect(heldOf(other).read()).toEqual([]); | |
| 48 | }); | |
| 49 | ||
| 50 | test("a forge's line reads its own holders and the thread's as one, largest first", () => { | |
| 51 | const forge = {}; | |
| 52 | const mine = new Set<number>([1, 2, 3]); | |
| 53 | heldOf(forge).add("mine", () => mine.size); | |
| 54 | const names = heldNow(forge).map((h) => h.name); | |
| 55 | expect(names).toContain("mine"); | |
| 56 | // What a module keeps once for the thread named itself as it loaded. | |
| 57 | for (const h of threadHeld.read()) expect(names).toContain(h.name); | |
| 58 | const sizes = heldNow(forge).map((h) => h.size); | |
| 59 | expect(sizes).toEqual([...sizes].sort((a, b) => b - a)); | |
| 60 | }); | |
| 61 | ||
| 62 | test("a worker's line names what its thread holds, and is not written when it holds nothing", () => { | |
| 63 | expect(workerHeldLine("engine", [])).toBeUndefined(); | |
| 64 | expect(workerHeldLine("engine", [{ name: "checks retried", size: 0 }])).toBeUndefined(); | |
| 65 | expect(workerHeldLine("engine", [{ name: "checks retried", size: 12 }])).toBe("memory worker engine holds: checks retried 12"); | |
| 66 | }); | |
| 67 | ||
| 68 | test("the line names the largest holders that hold anything", () => { | |
| 69 | expect(heldLine([])).toBe("memory holds: nothing"); | |
| 70 | expect(heldLine([{ name: "turn", size: 0 }])).toBe("memory holds: nothing"); | |
| 71 | expect(heldLine([{ name: "sse", size: 14 }, { name: "turn", size: 3 }, { name: "idle", size: 0 }, { name: "broken", size: -1 }])).toBe( | |
| 72 | "memory holds: sse 14, turn 3, broken unread", | |
| 73 | ); | |
| 74 | const many = Array.from({ length: 40 }, (_, i) => ({ name: `h${i}`, size: 100 - i })); | |
| 75 | expect(heldLine(many).split(", ")).toHaveLength(HELD_TOP); | |
| 76 | expect(heldLine(many, 2)).toBe("memory holds: h0 100, h1 99"); | |
| 77 | }); | |
| 78 | }); | |
| 79 | ||
| 80 | describe("the memory sampler", () => { | |
| 81 | test("each tick writes what the forge holds beside its memory line, counted at that tick", async () => { | |
| 82 | const dir = await mkdtemp(join(tmpdir(), "forge-held-")); | |
| 83 | const app = createApp({ dataDir: dir, port: 0 }); | |
| 84 | const info = spyOn(app.log, "info").mockImplementation(() => {}); | |
| 85 | const probe = new Set<number>([1, 2, 3]); | |
| 86 | heldOf(app).add("probe", () => probe.size); | |
| 87 | // An hour between ticks: the sampler ticks once as it starts, and the test asks for the next. | |
| 88 | const sampler = startMemorySampler(app, 3_600_000); | |
| 89 | try { | |
| 90 | probe.add(4); | |
| 91 | sampler.tick(); | |
| 92 | const lines = info.mock.calls.map((call) => String(call[0])); | |
| 93 | expect(lines.map((l) => l.slice(0, l.indexOf(":")))).toEqual(["memory main", "memory holds", "memory main", "memory holds"]); | |
| 94 | expect(lines[1]).toStartWith("memory holds: "); | |
| 95 | expect(lines[1]!.split(": ")[1]!.split(", ")).toContain("probe 3"); | |
| 96 | expect(lines[3]!.split(": ")[1]!.split(", ")).toContain("probe 4"); | |
| 97 | // The thread's holders are on the same line as the forge's own. | |
| 98 | for (const h of threadHeld.read().filter((h) => h.size > 0).slice(0, 3)) expect(lines[3]).toContain(`${h.name} ${h.size}`); | |
| 99 | } finally { | |
| 100 | sampler.stop(); | |
| 101 | info.mockRestore(); | |
| 102 | app.close(); | |
| 103 | await rm(dir, { recursive: true, force: true }); | |
| 104 | } | |
| 105 | }); | |
| 106 | }); |
new above
offline
No test fails if this line is removed. Add a sampler test that expects a
memory holds:line each tick.