Public: anyone can read this; the crew can change it.
An app cannot stall the forge: its requests wait behind people's, repeat read… #12268
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/12265-an-app-cannot-stall-the-forge-its-reques → main 16 commits fork 2 threads · 2 unresolved
16 files
changed +679
−9
AGENTS.md +1
| @@ -42,4 +42,5 @@ State shared between modules through process memory, not the database, is named | ||
| 42 | 42 | ## Awake ceiling and the boot |
| 43 | 43 | `FORGE_CHILD_AWAKE_MAX` counts forges in use. Hosted children the boot opens (so their apps come back) are exempt from `awakeCount` until a credentialed request reaches them or they sleep (`bootOpened` in `src/spawn/host.ts`). The exempt set holds only body ids of existing finished children and is emptied when the parent closes, so it is bounded by the number of children. Cell children are always counted; they are not opened by the boot. |
| 44 | 44 | |
| 45 | An app token's requests go behind people's and repeat reads are 304s (`src/http/app_turn.ts`, `repeat_reads.ts`, `perf/asking.ts`): see `docs/features/app-asking.md`. | |
| 45 | 46 | `src/core/reports.ts`: `closedSet`, the ids of repositories whose owner closed reports from outside (#12321), read once per database and kept by `setReportsClosed`, the only writer, so every repository page's head link costs no statement; at most one id per closed repository, gone on restart (the rows are `report_closed`). `src/core/contribute_token.ts`: `openSet`, the ids of repositories that allow outside agents, read once per database and kept by `setContributeOpen`, the only writer, at most one id per allowing repository, gone on restart; a raw SQL write to `contribute_open` bypasses it. The terminal door for reports is `r` on the outside agents screen (`setOutsideReports`, admins). |
docs/APPS.md +9
| @@ -782,6 +782,15 @@ lines: no id, never replayed, and dropped while the reader is behind. | ||
| 782 | 782 | `examples/forge-live` is the worked example, and the acceptance test |
| 783 | 783 | (`test/apps-token.test.ts`) deploys it. |
| 784 | 784 | |
| 785 | **Events before polls.** To be told when something changes, open | |
| 786 | `GET $FORGE_URL/events` once and follow it; do not ask again on a timer. The | |
| 787 | app template has no polling loop, and its `AGENTS.md` says so. When a read is | |
| 788 | the right tool, send `If-None-Match` with the `ETag` of the last answer and | |
| 789 | keep your copy on a `304`: every read an app token makes is answered from the | |
| 790 | forge's write clock without running its query while nothing was written | |
| 791 | (`src/http/repeat_reads.ts`, at most 30 s old). Two more things the forge does | |
| 792 | about an app that asks too much: it waits behind people's requests, and the owner is told which token asks the most. Neither refuses a request; see `docs/features/app-asking.md`. | |
| 793 | ||
| 785 | 794 | That is production's token: production runs what a person landed. Every |
| 786 | 795 | other environment (a pull's preview, a build, a variant) runs code nobody has |
| 787 | 796 | landed yet, so its token is bound to the app's one repository, as a crew |
docs/features/app-asking.md added +12
| @@ -0,0 +1,12 @@ | ||
| 1 | # An app cannot stall the forge (#12265) | |
| 2 | ||
| 3 | The forge serves everyone on one thread, so an app that polls it on a timer must not slow people down. Nothing an app asks is refused; it only goes last, and the forge says who asks the most. | |
| 4 | ||
| 5 | - **Events before polls.** To be told when something changes, open `GET $FORGE_URL/events` once and follow it. When a read is the right tool, send `If-None-Match` with the last `ETag` and keep your copy on a `304`. | |
| 6 | - **Repeat reads are free** (`src/http/repeat_reads.ts`). An app token's last 200 tag per URL and `Accept` is kept (under the token's hash, never the secret), at most 512 per forge and 30 s old. While the write clock has not moved, the same request with that tag is answered 304 without running its query. Any write ends it. | |
| 7 | - **People first** (`src/http/app_turn.ts`). An app token's request waits while a person's request begun in the last second is open, or one was answered in the last 50 ms, for at most one second. With nobody else asking it is served at once. | |
| 8 | - **Who is asking** (`src/perf/asking.ts`). Any one token over a fifth of the last hour's requests or of the serving thread's time is named, with the route it asks most and how often, with the choice: open the app's page, or put it to sleep, or revoke the token. It is shown on /system and in the terminal's System screen, to the owner only (`canSeeMemory`), and in the admin's needs-you stack. An always-on app nobody opened for a day that is still calling the forge sleeps like any idle app. | |
| 9 | ||
| 10 | The reading is one in-process map (61 minute buckets, 64 tokens each); it is gone on restart. | |
| 11 | ||
| 12 | Tests: `test/app-turn.test.ts` (the turn on a fake clock, the 304, the gate on /system), `test/tui-system.test.ts`. |
src/actions/health.ts +20 −1
| @@ -16,6 +16,7 @@ | ||
| 16 | 16 | * One answer for the API (`GET /api/v1/health`) and the MCP `health` tool. |
| 17 | 17 | */ |
| 18 | 18 | |
| 19 | import { asking } from "../perf/asking.ts"; | |
| 19 | 20 | import { flakeLine, flakeList, flakeNeedLine, flakeNeedRows, type FlakeRow } from "../checks/flake.ts"; |
| 20 | 21 | import type { App } from "../app.ts"; |
| 21 | 22 | import { appsRuntime } from "../apps/runtime.ts"; |
| @@ -70,7 +71,7 @@ function gatherVitals(app: App): Vitals { | ||
| 70 | 71 | const envs = runtime |
| 71 | 72 | ? listApps(app.db).flatMap((record) => runtime.status(record)) |
| 72 | 73 | : []; |
| 73 | return vitals({ | |
| 74 | const base = vitals({ | |
| 74 | 75 | db: app.db, |
| 75 | 76 | dataDir: app.config.dataDir, |
| 76 | 77 | ...(app.config.disk ? { disk: app.config.disk } : {}), |
| @@ -81,6 +82,24 @@ function gatherVitals(app: App): Vitals { | ||
| 81 | 82 | snapshots: snapshotPlace(app), |
| 82 | 83 | forgesMemory: memoryOverLine(forgesMemory(app)), |
| 83 | 84 | }); |
| 85 | return withHeavyAskers(base); | |
| 86 | } | |
| 87 | ||
| 88 | /** | |
| 89 | * Any one token that made more than a fifth of the last hour's requests or of the serving | |
| 90 | * thread's time is a need of its own (#12265): named, with what to do. It is a `todo`: it does | |
| 91 | * not make the forge unwell (`ok` stays), only asks to be looked at. | |
| 92 | */ | |
| 93 | export function withHeavyAskers(v: Vitals, now: number = Date.now()): Vitals { | |
| 94 | const heavy = asking.heavy(); | |
| 95 | if (heavy.length === 0) return v; | |
| 96 | const since = new Date(now).toISOString(); | |
| 97 | return { | |
| 98 | ...v, | |
| 99 | needs: [...v.needs, ...heavy.map((a) => a.sentence)], | |
| 100 | todo: [...v.todo, ...heavy.map((a) => a.sentence)], | |
| 101 | degraded: [...v.degraded, ...heavy.map((a) => ({ cause: `asking:${a.token}`, sentence: a.sentence, since, href: "/system#asking" }))], | |
| 102 | }; | |
| 84 | 103 | } |
| 85 | 104 | |
| 86 | 105 | /** |
src/apps/runtime.ts +15 −4
| @@ -35,7 +35,8 @@ import { neededNames, resolveAppSecrets } from "../core/app_secrets.ts"; | ||
| 35 | 35 | import { featureVarsAt } from "./feature_env.ts"; |
| 36 | 36 | import { getVariant, listVariants, variantOfBranch } from "../core/variants.ts"; |
| 37 | 37 | import { readVariantValues } from "./variant_file.ts"; |
| 38 | import { mintAppToken, renewAppToken, revokeAppToken } from "./token.ts"; | |
| 38 | import { appTokenName, mintAppToken, renewAppToken, revokeAppToken } from "./token.ts"; | |
| 39 | import { asking } from "../perf/asking.ts"; | |
| 39 | 40 | import type { ForgeEvent } from "../core/events.ts"; |
| 40 | 41 | import { announceRun } from "../actions/runs.ts"; |
| 41 | 42 | import { requeueAbandonedChecks } from "../checks/service.ts"; |
| @@ -218,6 +219,9 @@ export type EnvState = | ||
| 218 | 219 | | "asleep" |
| 219 | 220 | | "failed"; |
| 220 | 221 | |
| 222 | /** How long an always-on app may go unopened while it keeps calling the forge. */ | |
| 223 | export const UNOPENED_MS = 24 * 60 * 60_000; | |
| 224 | ||
| 221 | 225 | /** Idle windows before an environment sleeps: the profit slider's 50 (`core/slider.ts`). */ |
| 222 | 226 | export const SLEEP_AFTER_MS = { production: 24 * 3_600_000, preview: 30 * 60_000 }; |
| 223 | 227 | /** Days a quiet pull keeps its preview by default: the slider's 50 too. */ |
| @@ -2299,9 +2303,16 @@ export class AppsRuntime { | ||
| 2299 | 2303 | state.env === "production" || envVariant(state.env) !== undefined, |
| 2300 | 2304 | live.manifest.alwaysOn === true, |
| 2301 | 2305 | ); |
| 2302 | const limit = window.ms; | |
| 2303 | if (limit === null) continue; | |
| 2304 | 2306 | const last = state.lastRequestAt ?? Date.parse(live.since); |
| 2307 | // An always-on app nobody has opened for a day, still calling the forge, is idle like any | |
| 2308 | // other: its own calls are not a visit (`touch` is the proxy's), and they do not keep it | |
| 2309 | // awake (#12265). | |
| 2310 | const limit = | |
| 2311 | window.ms ?? | |
| 2312 | (now - last >= UNOPENED_MS && asking.calling(appTokenName(state.app.slug, state.env), 60 * 60_000, now) | |
| 2313 | ? UNOPENED_MS | |
| 2314 | : null); | |
| 2315 | if (limit === null) continue; | |
| 2305 | 2316 | if (now - last < limit) continue; |
| 2306 | 2317 | // Idle by the clock; asked whether it is mid-work, and spared while |
| 2307 | 2318 | // it is — but only for one grace past the window, never forever. |
| @@ -2335,7 +2346,7 @@ export class AppsRuntime { | ||
| 2335 | 2346 | this.app.config.dataDir, |
| 2336 | 2347 | live.runId, |
| 2337 | 2348 | busy === undefined |
| 2338 | ? asleepWords({ ms: limit, why: window.why }) | |
| 2349 | ? asleepWords({ ms: limit, why: window.ms === null ? "always_on, but nobody opened it and it kept calling the forge" : window.why }) | |
| 2339 | 2350 | : `asleep: still busy ${Math.round(grace / 60_000)} minutes past its idle window, which is as long as busy keeps an app awake; the next request wakes it`, |
| 2340 | 2351 | ); |
| 2341 | 2352 | log.info("asleep", { app: state.app.slug, env: state.env }); |
src/http/app_turn.ts added +85
| @@ -0,0 +1,85 @@ | ||
| 1 | /** | |
| 2 | * An app's turn on the serving thread (#12265): people first, never a cap. | |
| 3 | * | |
| 4 | * Requests are served in the forge's ladder: a person's page or form, live | |
| 5 | * updates, git, an agent's call a person waits on, then an app's own token. | |
| 6 | * One thread serves them all, so the last rung waits here, after its token is | |
| 7 | * known and before its route runs, while the thread is wanted by anyone above | |
| 8 | * it: a person's request begun in the last `YOUNG_MS` and still open, or one | |
| 9 | * answered in the last `BURST_MS` (a page is a burst, and the next request is | |
| 10 | * on its way). It looks again every `LOOK_MS` and goes when the thread is free | |
| 11 | * or when it has waited `BOUND_MS`: deferred, never refused, never starved. | |
| 12 | * With nobody else asking it goes at once, without a timer. | |
| 13 | * | |
| 14 | * Streams (a long poll, `/events`) are older than `YOUNG_MS` and hold a | |
| 15 | * connection, not the thread. Plain data, one object per process. | |
| 16 | */ | |
| 17 | ||
| 18 | export const YOUNG_MS = 1_000; | |
| 19 | export const BURST_MS = 50; | |
| 20 | export const LOOK_MS = 20; | |
| 21 | export const BOUND_MS = 1_000; | |
| 22 | ||
| 23 | export interface Turn { | |
| 24 | /** A request arrived: it is a person's until it says otherwise. */ | |
| 25 | begin(key: object, at?: number): void; | |
| 26 | /** Auth showed it to be an app's: it stops counting as a person's. */ | |
| 27 | machine(key: object): void; | |
| 28 | /** It is answered. */ | |
| 29 | end(key: object, at?: number): void; | |
| 30 | /** Resolve at once when nobody above is waiting, else when they are done or after the bound. */ | |
| 31 | wait(): Promise<void>; | |
| 32 | /** Totals since the process began, for /system. */ | |
| 33 | readonly stats: { waited: number; bounded: number; longestMs: number }; | |
| 34 | } | |
| 35 | ||
| 36 | const sleepReal = (ms: number): Promise<void> => new Promise((r) => setTimeout(r, ms)); | |
| 37 | ||
| 38 | export function createTurn( | |
| 39 | now: () => number = () => performance.now(), | |
| 40 | bound = BOUND_MS, | |
| 41 | sleep: (ms: number) => Promise<void> = sleepReal, | |
| 42 | ): Turn { | |
| 43 | const open = new Map<object, number>(); | |
| 44 | let lastEnd = -Infinity; | |
| 45 | const stats = { waited: 0, bounded: 0, longestMs: 0 }; | |
| 46 | const wanted = (): boolean => { | |
| 47 | const t = now(); | |
| 48 | if (t - lastEnd < BURST_MS) return true; | |
| 49 | let young = false; | |
| 50 | // A request nobody ended (a response that left by another door) is forgotten once it is old. | |
| 51 | for (const [key, at] of open) { | |
| 52 | if (t - at < YOUNG_MS) young = true; | |
| 53 | else open.delete(key); | |
| 54 | } | |
| 55 | return young; | |
| 56 | }; | |
| 57 | return { | |
| 58 | begin(key, at = now()) { | |
| 59 | open.set(key, at); | |
| 60 | }, | |
| 61 | machine(key) { | |
| 62 | open.delete(key); | |
| 63 | }, | |
| 64 | end(key, at = now()) { | |
| 65 | if (open.delete(key)) lastEnd = at; | |
| 66 | }, | |
| 67 | async wait() { | |
| 68 | if (!wanted()) return; | |
| 69 | const since = now(); | |
| 70 | stats.waited++; | |
| 71 | while (wanted()) { | |
| 72 | if (now() - since >= bound) { | |
| 73 | stats.bounded++; | |
| 74 | break; | |
| 75 | } | |
| 76 | await sleep(LOOK_MS); | |
| 77 | } | |
| 78 | stats.longestMs = Math.max(stats.longestMs, now() - since); | |
| 79 | }, | |
| 80 | stats, | |
| 81 | }; | |
| 82 | } | |
| 83 | ||
| 84 | /** The forge's one gate: a hosted child answers on the same thread. */ | |
| 85 | export const turn: Turn = createTurn(); |
src/http/repeat_reads.ts added +73
| @@ -0,0 +1,73 @@ | ||
| 1 | /** | |
| 2 | * Repeat reads are free (#12265): an app token that asks again for what it | |
| 3 | * holds is answered 304 from the write clock, before the route runs a query. | |
| 4 | * | |
| 5 | * After a 200 that carries an ETag, the answer's tag is kept under the | |
| 6 | * token, the user, the scope and the full URL, with the write clock as it was | |
| 7 | * read BEFORE the route ran (`core/write_clock.ts`). A later GET with that | |
| 8 | * tag in `If-None-Match` while the clock has not moved (and the entry is | |
| 9 | * under `MAX_AGE_MS` old, which bounds anything the clock cannot see, such as | |
| 10 | * telemetry) is the same answer: 304 with the same headers, no handler. | |
| 11 | * Access and token changes are writes, so they move the clock and end it. | |
| 12 | * | |
| 13 | * Plain data, bounded to `MAX_ENTRIES` per forge, oldest dropped. | |
| 14 | */ | |
| 15 | ||
| 16 | import type { Database } from "bun:sqlite"; | |
| 17 | import { holds } from "./etag.ts"; | |
| 18 | ||
| 19 | export const MAX_AGE_MS = 30_000; | |
| 20 | export const MAX_ENTRIES = 512; | |
| 21 | ||
| 22 | interface Kept { | |
| 23 | stamp: string; | |
| 24 | at: number; | |
| 25 | etag: string; | |
| 26 | headers: Record<string, string>; | |
| 27 | } | |
| 28 | ||
| 29 | const kept = new WeakMap<Database, Map<string, Kept>>(); | |
| 30 | ||
| 31 | /** | |
| 32 | * What the same asker is asking for: one key per token, account, scope, URL and | |
| 33 | * representation (`Accept`). The token is held as its hash, never as the secret. | |
| 34 | */ | |
| 35 | export const repeatKey = (credential: string, userId: string, scope: string, url: URL, accept: string | null = null): string => | |
| 36 | `${new Bun.CryptoHasher("sha256").update(credential).digest("hex")}|${userId}|${scope}|${accept ?? ""}|${url.pathname}${url.search}`; | |
| 37 | ||
| 38 | /** The 304 this asker is owed, or undefined when the route must run. */ | |
| 39 | export function repeatRead( | |
| 40 | db: Database, | |
| 41 | key: string, | |
| 42 | request: Request, | |
| 43 | stamp: string | undefined, | |
| 44 | now: number = Date.now(), | |
| 45 | ): Response | undefined { | |
| 46 | if (stamp === undefined || request.method !== "GET") return undefined; | |
| 47 | const held = kept.get(db)?.get(key); | |
| 48 | if (held === undefined || held.stamp !== stamp || now - held.at >= MAX_AGE_MS) return undefined; | |
| 49 | if (!holds(request, held.etag)) return undefined; | |
| 50 | return new Response(null, { status: 304, headers: held.headers }); | |
| 51 | } | |
| 52 | ||
| 53 | /** Keep a 200's tag against the clock read before its route ran. */ | |
| 54 | export function keepRead( | |
| 55 | db: Database, | |
| 56 | key: string, | |
| 57 | stamp: string | undefined, | |
| 58 | response: Response, | |
| 59 | now: number = Date.now(), | |
| 60 | ): void { | |
| 61 | const etag = response.headers.get("etag"); | |
| 62 | if (stamp === undefined || etag === null || response.status !== 200) return; | |
| 63 | let map = kept.get(db); | |
| 64 | if (map === undefined) kept.set(db, (map = new Map())); | |
| 65 | map.delete(key); | |
| 66 | const headers: Record<string, string> = { etag }; | |
| 67 | for (const name of ["cache-control", "vary"]) { | |
| 68 | const value = response.headers.get(name); | |
| 69 | if (value !== null) headers[name] = value; | |
| 70 | } | |
| 71 | map.set(key, { stamp, at: now, etag, headers }); | |
| 72 | if (map.size > MAX_ENTRIES) map.delete(map.keys().next().value as string); | |
| 73 | } |
src/http/server.ts +24
| @@ -572,6 +572,10 @@ import { createReadWriter, readOf } from "../perf/reads.ts"; | ||
| 572 | 572 | import { parseServerTiming } from "../perf/record.ts"; |
| 573 | 573 | import { currentLoopLag, LAG_WINDOW_MS, startLoopWorst } from "../perf/loop_worst.ts"; |
| 574 | 574 | import { methodOf, noteUsage, roleOf, surfaceOf, takeUsage } from "./usage_tag.ts"; |
| 575 | import { asking, noteAsker, noteWaited, takeAsker } from "../perf/asking.ts"; | |
| 576 | import { turn } from "./app_turn.ts"; | |
| 577 | import { keepRead, repeatKey, repeatRead } from "./repeat_reads.ts"; | |
| 578 | import { writeStamp } from "../core/write_clock.ts"; | |
| 575 | 579 | import { bearerChallenge, oauthRoutes } from "./oauth.ts"; |
| 576 | 580 | import { flushAccess, recordAccess } from "./access_log.ts"; |
| 577 | 581 | import { carryPeer, clientAddress, notePeer, wasInternal, wasViaEdge, type ProxyTrust } from "./peer.ts"; |
| @@ -2695,6 +2699,9 @@ export async function serve(app: App): Promise<Servers> { | ||
| 2695 | 2699 | ): void => { |
| 2696 | 2700 | const tag = takeUsage(request); |
| 2697 | 2701 | const ms = performance.now() - started; |
| 2702 | turn.end(request); | |
| 2703 | const asker = takeAsker(request); | |
| 2704 | asking.note(asker?.token, tag.pattern, Math.max(0, ms - (asker?.waitedMs ?? 0))); | |
| 2698 | 2705 | finishedStrangerDeep(owner, request); |
| 2699 | 2706 | try { |
| 2700 | 2707 | const header = serverTiming(ms, takeDbTime(request), takePhases(request)); |
| @@ -2802,6 +2809,7 @@ export async function serve(app: App): Promise<Servers> { | ||
| 2802 | 2809 | ): Promise<Response | undefined> => { |
| 2803 | 2810 | const started = performance.now(); |
| 2804 | 2811 | noteStart(request, started); |
| 2812 | turn.begin(request, started); | |
| 2805 | 2813 | // A read-only standby refuses every mutation here, before any route. |
| 2806 | 2814 | if (app.config.readOnly === true) { |
| 2807 | 2815 | const refused = readOnlyRefusal(request); |
| @@ -3160,6 +3168,7 @@ async function answerGet( | ||
| 3160 | 3168 | noteSessionTime(request, performance.now() - authStart); |
| 3161 | 3169 | if (auth.retryAfter !== undefined) return tooManyAttempts(auth.retryAfter); |
| 3162 | 3170 | noteUsage(request, { pattern: matched.pattern, role: roleOf(auth) }); |
| 3171 | if (auth.tokenName !== undefined) noteAsker(request, auth.tokenName); | |
| 3163 | 3172 | // From here a body's cap depends on who is asking (`http/limits.ts`), and |
| 3164 | 3173 | // a stream this request opens is the account's, to end with it. |
| 3165 | 3174 | if (auth.user !== undefined) { |
| @@ -3296,6 +3305,20 @@ async function answerGet( | ||
| 3296 | 3305 | door: auth.tokenName === undefined ? auth.via : `token:${auth.tokenName}`, |
| 3297 | 3306 | ...(auth.through === undefined ? {} : { via: auth.through }), |
| 3298 | 3307 | }; |
| 3308 | // An app's own token goes last on the ladder (#12265): a repeat of what it | |
| 3309 | // holds is answered 304 from the write clock at once; anything else waits its | |
| 3310 | // turn behind people's requests, for a bound, then runs. | |
| 3311 | const appAsks = auth.tokenName?.startsWith("app:") === true && auth.user !== undefined && credential !== undefined; | |
| 3312 | const repeat = appAsks ? repeatKey(credential!, auth.user!.id, auth.scope ?? "ro", url, request.headers.get("accept")) : undefined; | |
| 3313 | const stamp = repeat === undefined ? undefined : writeStamp(app.db); | |
| 3314 | if (repeat !== undefined) { | |
| 3315 | const same = repeatRead(app.db, repeat, request, stamp); | |
| 3316 | if (same !== undefined) return withRateHeaders(same, verdict); | |
| 3317 | turn.machine(request); | |
| 3318 | const waitFrom = performance.now(); | |
| 3319 | await turn.wait(); | |
| 3320 | noteWaited(request, performance.now() - waitFrom); | |
| 3321 | } | |
| 3299 | 3322 | // And it reads as its caller: in a fork whose parent that caller may not |
| 3300 | 3323 | // read, what git is asked for by id is held to the fork's own refs, at every |
| 3301 | 3324 | // handler's reads and at no handler's cost (`git/own_refs.ts`). |
| @@ -3306,6 +3329,7 @@ async function answerGet( | ||
| 3306 | 3329 | : withIdempotency(app, request, url, replayAs, invoke), |
| 3307 | 3330 | ), |
| 3308 | 3331 | ); |
| 3332 | if (repeat !== undefined) keepRead(app.db, repeat, stamp, response); | |
| 3309 | 3333 | // A page route's HTML 404 (`/:owner` claims `/nope`) is a refusal an |
| 3310 | 3334 | // agent cannot read; one that asked for JSON gets it as JSON. |
| 3311 | 3335 | if ( |
src/perf/asking.ts added +179
| @@ -0,0 +1,179 @@ | ||
| 1 | /** | |
| 2 | * Who is asking the forge for how much, over the last hour (#12265). | |
| 3 | * | |
| 4 | * Every response is noted once: its route and the milliseconds it held the | |
| 5 | * serving thread, under the name of the token that asked (a person's page | |
| 6 | * has none, and only adds to the totals). Minute buckets, 61 kept, at most | |
| 7 | * `NAMES` names a bucket, so the memory is fixed however many tokens ask. | |
| 8 | * | |
| 9 | * `heavy()` names any one token that made more than a fifth of the hour's | |
| 10 | * requests, or of the thread's time, with the route it asked most and how | |
| 11 | * often: "foundry made 63% of requests this hour: every 2.5 s to /runs". | |
| 12 | * `calling()` says whether a token asked at all lately, which is what lets | |
| 13 | * the runtime stop treating an app nobody opened as awake for its own calls. | |
| 14 | * | |
| 15 | * One map per process, gone on restart: it is a reading, not state. | |
| 16 | */ | |
| 17 | ||
| 18 | /** The share (of requests or of the thread's time) above which a token is named. */ | |
| 19 | export const HEAVY_SHARE = 0.2; | |
| 20 | /** Below this many requests an hour in all, nobody is named: a quiet forge has no heavy asker. */ | |
| 21 | export const MIN_TOTAL = 200; | |
| 22 | /** A token must have asked at least this often to be named. */ | |
| 23 | export const MIN_OWN = 60; | |
| 24 | const MINUTE = 60_000; | |
| 25 | const KEPT = 61; | |
| 26 | const NAMES = 64; | |
| 27 | const ROUTES = 8; | |
| 28 | ||
| 29 | export interface Asker { | |
| 30 | /** The token's name as its owner gave it (`app:foundry/production`). */ | |
| 31 | readonly token: string; | |
| 32 | /** What to call it: the app's name for an app token. */ | |
| 33 | readonly name: string; | |
| 34 | readonly app: boolean; | |
| 35 | readonly requests: number; | |
| 36 | /** The larger of its share of requests and of the thread's time. */ | |
| 37 | readonly share: number; | |
| 38 | /** The route it asked most, with the app's prefix left off. */ | |
| 39 | readonly route: string; | |
| 40 | /** Milliseconds between its asks of that route, or null when too few to say. */ | |
| 41 | readonly everyMs: number | null; | |
| 42 | readonly sentence: string; | |
| 43 | } | |
| 44 | ||
| 45 | interface Own { | |
| 46 | first: number; | |
| 47 | last: number; | |
| 48 | n: number; | |
| 49 | ms: number; | |
| 50 | routes: Map<string, number>; | |
| 51 | } | |
| 52 | interface Bucket { | |
| 53 | minute: number; | |
| 54 | n: number; | |
| 55 | ms: number; | |
| 56 | by: Map<string, Own>; | |
| 57 | } | |
| 58 | ||
| 59 | /** `app:foundry/production` is the app `foundry`; any other token is its own name. */ | |
| 60 | export function askerName(token: string): { name: string; app: boolean } { | |
| 61 | const m = /^app:([^/]+)\//.exec(token); | |
| 62 | const name = m === null ? token : m[1]!; | |
| 63 | return { name: name.length > 40 ? `${name.slice(0, 39)}…` : name, app: m !== null }; | |
| 64 | } | |
| 65 | ||
| 66 | /** "2.5 s", "40 ms", "3 min": the gap as a person says it. */ | |
| 67 | export function gapWords(ms: number): string { | |
| 68 | if (ms < 1_000) return `${Math.max(1, Math.round(ms))} ms`; | |
| 69 | if (ms < 60_000) return `${Number((ms / 1_000).toFixed(1))} s`; | |
| 70 | return `${Number((ms / 60_000).toFixed(1))} min`; | |
| 71 | } | |
| 72 | ||
| 73 | const shortRoute = (pattern: string): string => pattern.replace(/^\/api\/v1\/repos\/:owner\/:name/, "") || pattern; | |
| 74 | ||
| 75 | export interface Asking { | |
| 76 | /** One response: who asked (a token's name, if any), which route, how long it held the thread. */ | |
| 77 | note(token: string | undefined, pattern: string, ms: number, now?: number): void; | |
| 78 | heavy(now?: number): Asker[]; | |
| 79 | /** Whether `token` asked at all in the last `withinMs`. */ | |
| 80 | calling(token: string, withinMs: number, now?: number): boolean; | |
| 81 | } | |
| 82 | ||
| 83 | export function createAsking(limits: { minTotal?: number; minOwn?: number } = {}): Asking { | |
| 84 | let buckets: Bucket[] = []; | |
| 85 | const bucketAt = (now: number): Bucket => { | |
| 86 | const minute = Math.floor(now / MINUTE); | |
| 87 | const last = buckets[buckets.length - 1]; | |
| 88 | if (last !== undefined && last.minute === minute) return last; | |
| 89 | const fresh: Bucket = { minute, n: 0, ms: 0, by: new Map() }; | |
| 90 | buckets.push(fresh); | |
| 91 | if (buckets.length > KEPT) buckets = buckets.slice(-KEPT); | |
| 92 | return fresh; | |
| 93 | }; | |
| 94 | return { | |
| 95 | note(token, pattern, ms, now = Date.now()) { | |
| 96 | const b = bucketAt(now); | |
| 97 | b.n++; | |
| 98 | b.ms += ms; | |
| 99 | if (token === undefined) return; | |
| 100 | let own = b.by.get(token); | |
| 101 | if (own === undefined) { | |
| 102 | if (b.by.size >= NAMES) return; | |
| 103 | own = { first: now, last: now, n: 0, ms: 0, routes: new Map() }; | |
| 104 | b.by.set(token, own); | |
| 105 | } | |
| 106 | own.n++; | |
| 107 | own.last = now; | |
| 108 | own.ms += ms; | |
| 109 | if (own.routes.has(pattern) || own.routes.size < ROUTES) own.routes.set(pattern, (own.routes.get(pattern) ?? 0) + 1); | |
| 110 | }, | |
| 111 | heavy(now = Date.now()) { | |
| 112 | const from = Math.floor(now / MINUTE) - (KEPT - 1); | |
| 113 | const live = buckets.filter((b) => b.minute >= from); | |
| 114 | let total = 0; | |
| 115 | let totalMs = 0; | |
| 116 | const sum = new Map<string, { n: number; ms: number; routes: Map<string, number>; first: number; last: number }>(); | |
| 117 | for (const b of live) { | |
| 118 | total += b.n; | |
| 119 | totalMs += b.ms; | |
| 120 | for (const [token, own] of b.by) { | |
| 121 | let s = sum.get(token); | |
| 122 | if (s === undefined) sum.set(token, (s = { n: 0, ms: 0, routes: new Map(), first: own.first, last: own.last })); | |
| 123 | s.n += own.n; | |
| 124 | s.ms += own.ms; | |
| 125 | s.first = Math.min(s.first, own.first); | |
| 126 | s.last = Math.max(s.last, own.last); | |
| 127 | for (const [r, c] of own.routes) s.routes.set(r, (s.routes.get(r) ?? 0) + c); | |
| 128 | } | |
| 129 | } | |
| 130 | const minTotal = limits.minTotal ?? MIN_TOTAL; | |
| 131 | const minOwn = limits.minOwn ?? MIN_OWN; | |
| 132 | if (total < minTotal) return []; | |
| 133 | const out: Asker[] = []; | |
| 134 | for (const [token, s] of sum) { | |
| 135 | const share = s.n / total; | |
| 136 | const timeShare = totalMs > 0 ? s.ms / totalMs : 0; | |
| 137 | if (s.n < minOwn || (share <= HEAVY_SHARE && timeShare <= HEAVY_SHARE)) continue; | |
| 138 | const [route, count] = [...s.routes].sort((a, b) => b[1] - a[1])[0] ?? ["", 0]; | |
| 139 | const everyMs = count >= 10 ? (s.last - s.first) / (count - 1) : null; | |
| 140 | const who = askerName(token); | |
| 141 | const pct = Math.round(Math.max(share, timeShare) * 100); | |
| 142 | const of = share >= timeShare ? "requests" : "the serving thread's time"; | |
| 143 | const sentence = | |
| 144 | `${who.name} made ${pct}% of ${of} this hour` + | |
| 145 | (route === "" ? "" : `: ${everyMs === null ? "mostly" : `every ${gapWords(everyMs)}`} to ${shortRoute(route)}`) + | |
| 146 | (who.app ? ". Open its page, or put it to sleep." : ". Revoke the token if nobody means it to ask this much."); | |
| 147 | out.push({ token, ...who, requests: s.n, share: Math.max(share, timeShare), route: shortRoute(route), everyMs, sentence }); | |
| 148 | } | |
| 149 | return out.sort((a, b) => b.share - a.share); | |
| 150 | }, | |
| 151 | calling(token, withinMs, now = Date.now()) { | |
| 152 | const from = Math.floor((now - withinMs) / MINUTE); | |
| 153 | return buckets.some((b) => b.minute >= from && b.by.has(token)); | |
| 154 | }, | |
| 155 | }; | |
| 156 | } | |
| 157 | ||
| 158 | /** The forge's one reading. */ | |
| 159 | export const asking: Asking = createAsking(); | |
| 160 | ||
| 161 | const askedBy = new WeakMap<Request, { token: string; waitedMs: number }>(); | |
| 162 | ||
| 163 | /** Auth showed this request to carry a token: `counted` notes it under that name. */ | |
| 164 | export function noteAsker(request: Request, token: string): void { | |
| 165 | askedBy.set(request, { token, waitedMs: 0 }); | |
| 166 | } | |
| 167 | ||
| 168 | /** The request waited its turn this long: not the thread's time, so it is left out of its share. */ | |
| 169 | export function noteWaited(request: Request, ms: number): void { | |
| 170 | const held = askedBy.get(request); | |
| 171 | if (held !== undefined) held.waitedMs = ms; | |
| 172 | } | |
| 173 | ||
| 174 | /** The token that asked and the wait to leave out, once, when the response leaves. */ | |
| 175 | export function takeAsker(request: Request): { token: string; waitedMs: number } | undefined { | |
| 176 | const held = askedBy.get(request); | |
| 177 | askedBy.delete(request); | |
| 178 | return held; | |
| 179 | } |
src/tui/screens/system.ts +20
| @@ -36,6 +36,8 @@ import type { Ctx, Screen } from "../app/app.ts"; | ||
| 36 | 36 | import type { SystemView } from "../../system/view.ts"; |
| 37 | 37 | import { rolloutLine, type ForgeHealth } from "../../kernel/guard.ts"; |
| 38 | 38 | import { canSeeMemory, memoryHours } from "../../boards/memory.ts"; |
| 39 | import { asking } from "../../perf/asking.ts"; | |
| 40 | import { turn } from "../../http/app_turn.ts"; | |
| 39 | 41 | import { telemetryReader } from "../../actions/telemetry.ts"; |
| 40 | 42 | import { historyOver } from "../../system/telemetry.ts"; |
| 41 | 43 | import { binding } from "../app/keys.ts"; |
| @@ -418,6 +420,22 @@ export function memoryLines(session: Session, now = Date.now()): string[] { | ||
| 418 | 420 | ]; |
| 419 | 421 | } |
| 420 | 422 | |
| 423 | /** | |
| 424 | * "Who is asking" as /system says it, for the owner alone (`canSeeMemory`); empty for anyone | |
| 425 | * else, or when nothing is heavy and nothing has waited. | |
| 426 | */ | |
| 427 | export function askingLines(session: Session, inner = WIDTH - 6): string[] { | |
| 428 | if (!canSeeMemory(session.user)) return []; | |
| 429 | const { waited, bounded, longestMs } = turn.stats; | |
| 430 | const askers = asking.heavy(); | |
| 431 | if (askers.length === 0 && waited === 0) return []; | |
| 432 | return [ | |
| 433 | "Who is asking", | |
| 434 | ...askers.map((a) => truncate(` ${a.sentence}`, inner)), | |
| 435 | ...(waited === 0 ? [] : [truncate(` ${waited} app calls waited behind people's, ${bounded} to the bound (${Math.round(longestMs)} ms the longest).`, inner)]), | |
| 436 | ]; | |
| 437 | } | |
| 438 | ||
| 421 | 439 | export function memoryScreen(session: Session, ctx: Ctx): Screen { |
| 422 | 440 | const scroll = new Scroll(); |
| 423 | 441 | return { |
| @@ -497,9 +515,11 @@ export const entry: PaletteEntry = { | ||
| 497 | 515 | run: (session, ctx) => { |
| 498 | 516 | const self = selfLines(selfUse(realSelf(session.app.config.dataDir))); |
| 499 | 517 | const kids = childLines(childrenUse(listChildrenNow(session.app), (bodyId) => childDirOf(session.app, bodyId))); |
| 518 | const who = askingLines(session); | |
| 500 | 519 | ctx.push(systemScreen(ctx, [ |
| 501 | 520 | ...forgeLines(readHealth(session.app, session.user)), |
| 502 | 521 | ...egressLines(session), |
| 522 | ...(who.length === 0 ? [] : ["", ...who]), | |
| 503 | 523 | ...engineLines(session.app.config.engine), |
| 504 | 524 | ...(self.length === 0 ? [] : ["", ...self]), |
| 505 | 525 | ...(kids.length === 0 ? [] : ["", ...kids]), |
src/web/pages/system.ts +21
| @@ -21,6 +21,8 @@ | ||
| 21 | 21 | * line each for memory, CPU and disk; a parent lists each child's use |
| 22 | 22 | * against its size (#10853, `spawn/usage.ts`). |
| 23 | 23 | */ |
| 24 | import { asking, type Asker } from "../../perf/asking.ts"; | |
| 25 | import { turn } from "../../http/app_turn.ts"; | |
| 24 | 26 | import { mayReadRefusals } from "../../core/first_ask_refusals.ts"; |
| 25 | 27 | import { canSeeMemory, memoryHours } from "../../boards/memory.ts"; |
| 26 | 28 | import { telemetryReader } from "../../actions/telemetry.ts"; |
| @@ -391,6 +393,24 @@ export function stallSection(rows: readonly RouteStall[], worst: Html = html``): | ||
| 391 | 393 | </table></div>`}${worst}</section>`; |
| 392 | 394 | } |
| 393 | 395 | |
| 396 | /** | |
| 397 | * Who asked the forge for the most this hour (#12265): any one token over a fifth of the | |
| 398 | * requests or of the thread's time, named with what to do; and how many app requests waited | |
| 399 | * behind people's. Nothing drawn when nobody is over a fifth and nothing has waited. | |
| 400 | */ | |
| 401 | export function askingSection(askers: readonly Asker[], waits: { waited: number; bounded: number; longestMs: number }): Html { | |
| 402 | if (askers.length === 0 && waits.waited === 0) return html``; | |
| 403 | return html`<section class="settings-block" id="asking"><h2>Who is asking</h2>${ | |
| 404 | askers.length === 0 | |
| 405 | ? html`` | |
| 406 | : html`<ul>${askers.map((a) => html`<li data-asker="${a.name}">${a.sentence}</li>`)}</ul>` | |
| 407 | }${ | |
| 408 | waits.waited === 0 | |
| 409 | ? html`` | |
| 410 | : html`<p class="small muted">Apps' own calls go behind people's: ${String(waits.waited)} waited their turn since the forge started, ${String(waits.bounded)} to the bound (${String(Math.round(waits.longestMs))} ms the longest). Nothing is refused.</p>` | |
| 411 | }</section>`; | |
| 412 | } | |
| 413 | ||
| 394 | 414 | /** How many of the last hour's worst stalls the page lists (#11069); the terminal lists as many. */ |
| 395 | 415 | export { WORST_STALLS }; |
| 396 | 416 | |
| @@ -617,6 +637,7 @@ export async function systemPage(ctx: Ctx): Promise<Response> { | ||
| 617 | 637 | ${featuresSection(timePhase(ctx.request, "features", () => featureHealth(ctx.app, reader)))} |
| 618 | 638 | ${lagSection(lagMinutes.readings())} |
| 619 | 639 | <div id="sys-stalls">${stallSection(stalls, worstStallsTable(worst, nowMono, Date.now()))}</div> |
| 640 | ${owner ? askingSection(asking.heavy(), turn.stats) : html``} | |
| 620 | 641 | <div id="sys-slow-pages">${slowPagesSection(timePhase(ctx.request, "slowpages", () => slowPagesCached(ctx.app.db)))}</div> |
| 621 | 642 | <div id="sys-readers">${readersSection(readerPaces())}</div> |
| 622 | 643 | <div id="sys-caches">${cachesSection(pageCaches())}</div> |
templates/app-template.json +1 −1
Diff not shown — a line 15,877 characters wide (limit 2,000). +1 −1
View filetest/app-turn.test.ts added +186
| @@ -0,0 +1,186 @@ | ||
| 1 | /** | |
| 2 | * An app cannot stall the forge (#12265): its requests wait behind people's, | |
| 3 | * repeat reads cost nothing, and the forge names an app that asks too much. | |
| 4 | */ | |
| 5 | import { afterAll, beforeAll, describe, expect, test } from "bun:test"; | |
| 6 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 7 | import { tmpdir } from "node:os"; | |
| 8 | import { join } from "node:path"; | |
| 9 | import { createApp, type App } from "../src/app.ts"; | |
| 10 | import { createRepo } from "../src/actions/repos.ts"; | |
| 11 | import { openIssue } from "../src/actions/issues.ts"; | |
| 12 | import { forgeVitals } from "../src/actions/health.ts"; | |
| 13 | import { getRepo } from "../src/core/repos.ts"; | |
| 14 | import { createToken } from "../src/core/tokens.ts"; | |
| 15 | import { createUser } from "../src/core/users.ts"; | |
| 16 | import { serve, type Servers } from "../src/http/server.ts"; | |
| 17 | import { createTurn } from "../src/http/app_turn.ts"; | |
| 18 | import { asking, createAsking, gapWords } from "../src/perf/asking.ts"; | |
| 19 | ||
| 20 | describe("the turn", () => { | |
| 21 | test("nobody else asking: served at once; a person's request open: waits, to a bound, never refused", async () => { | |
| 22 | let t = 0; | |
| 23 | const turn = createTurn(() => t, 100, async (ms) => void (t += ms)); | |
| 24 | await turn.wait(); | |
| 25 | expect(turn.stats.waited).toBe(0); | |
| 26 | const person = {}; | |
| 27 | turn.begin(person, t); | |
| 28 | // A person still being answered: the wait ends at the bound, on the fake clock. | |
| 29 | await turn.wait(); | |
| 30 | expect(turn.stats.waited).toBe(1); | |
| 31 | expect(turn.stats.bounded).toBe(1); | |
| 32 | expect(turn.stats.longestMs).toBe(100); | |
| 33 | // Answered: a burst follows, then free. | |
| 34 | turn.end(person, t); | |
| 35 | await turn.wait(); | |
| 36 | expect(turn.stats.bounded).toBe(1); | |
| 37 | expect(turn.stats.waited).toBe(2); | |
| 38 | t += 1_000; | |
| 39 | await turn.wait(); | |
| 40 | expect(turn.stats.waited).toBe(2); | |
| 41 | }); | |
| 42 | ||
| 43 | test("a person answered while the app waits lets it go before the bound", async () => { | |
| 44 | let t = 0; | |
| 45 | const person = {}; | |
| 46 | const turn = createTurn(() => t, 1_000, async (ms) => { | |
| 47 | t += ms; | |
| 48 | if (t >= 40) turn.end(person, t); | |
| 49 | }); | |
| 50 | turn.begin(person, t); | |
| 51 | await turn.wait(); | |
| 52 | expect(turn.stats.bounded).toBe(0); | |
| 53 | expect(turn.stats.longestMs).toBeLessThan(1_000); | |
| 54 | }); | |
| 55 | ||
| 56 | test("an app's own request is no one's reason to wait", async () => { | |
| 57 | let t = 0; | |
| 58 | const turn = createTurn(() => t, 100); | |
| 59 | const app = {}; | |
| 60 | turn.begin(app, t); | |
| 61 | turn.machine(app); | |
| 62 | await turn.wait(); | |
| 63 | expect(turn.stats.waited).toBe(0); | |
| 64 | }); | |
| 65 | ||
| 66 | test("a person's request older than a second holds a connection, not the thread", async () => { | |
| 67 | let t = 0; | |
| 68 | const turn = createTurn(() => t, 100); | |
| 69 | turn.begin({}, 0); | |
| 70 | t = 1_500; | |
| 71 | await turn.wait(); | |
| 72 | expect(turn.stats.waited).toBe(0); | |
| 73 | }); | |
| 74 | }); | |
| 75 | ||
| 76 | describe("who is asking", () => { | |
| 77 | test("a token over a fifth of the hour's requests is named, with how often and where", () => { | |
| 78 | const a = createAsking({ minTotal: 100, minOwn: 20 }); | |
| 79 | const t0 = 1_000_000_000_000; | |
| 80 | for (let i = 0; i < 60; i++) a.note("app:foundry/production", "/api/v1/repos/:owner/:name/runs", 2, t0 + i * 2_500); | |
| 81 | for (let i = 0; i < 40; i++) a.note(undefined, "/", 5, t0 + i * 3_000); | |
| 82 | const [heavy] = a.heavy(t0 + 150_000); | |
| 83 | expect(heavy?.name).toBe("foundry"); | |
| 84 | expect(heavy?.share).toBeCloseTo(0.6, 1); | |
| 85 | expect(heavy?.sentence).toContain("foundry made 60% of requests this hour: every 2.5 s to /runs"); | |
| 86 | expect(heavy?.sentence).toContain("Open its page, or put it to sleep"); | |
| 87 | expect(a.calling("app:foundry/production", 60_000, t0 + 150_000)).toBe(true); | |
| 88 | expect(a.calling("app:foundry/production", 60_000, t0 + 7_200_000)).toBe(false); | |
| 89 | expect(a.heavy(t0 + 7_200_000)).toEqual([]); | |
| 90 | }); | |
| 91 | ||
| 92 | test("a quiet forge, or a token under a fifth, names nobody", () => { | |
| 93 | const a = createAsking({ minTotal: 100, minOwn: 5 }); | |
| 94 | for (let i = 0; i < 10; i++) a.note("app:a/production", "/x", 1, 5_000 + i); | |
| 95 | expect(a.heavy(10_000)).toEqual([]); | |
| 96 | for (let i = 0; i < 100; i++) a.note(undefined, "/", 1, 6_000 + i); | |
| 97 | expect(a.heavy(10_000)).toEqual([]); | |
| 98 | }); | |
| 99 | ||
| 100 | test("gaps are said as a person says them", () => { | |
| 101 | expect(gapWords(2_500)).toBe("2.5 s"); | |
| 102 | expect(gapWords(40)).toBe("40 ms"); | |
| 103 | expect(gapWords(180_000)).toBe("3 min"); | |
| 104 | }); | |
| 105 | }); | |
| 106 | ||
| 107 | describe("the forge answers an app token", () => { | |
| 108 | let app: App; | |
| 109 | let servers: Servers; | |
| 110 | let dir: string; | |
| 111 | let secret: string; | |
| 112 | const mine = (path: string, extra: Record<string, string> = {}) => | |
| 113 | fetch(servers.url + path, { headers: { authorization: `Bearer ${secret}`, ...extra } }); | |
| 114 | ||
| 115 | beforeAll(async () => { | |
| 116 | dir = await mkdtemp(join(tmpdir(), "forge-turn-")); | |
| 117 | app = createApp({ dataDir: dir, port: 0 }); | |
| 118 | servers = await serve(app); | |
| 119 | const made = await createUser(app.db, { origin: "seed", trusted: true, username: "alice", email: "[email protected]", password: "a-good-passphrase", adminIfFirst: true }); | |
| 120 | if (!made.ok) throw new Error(made.error); | |
| 121 | const repo = await createRepo(app, made.value, { name: "yard", visibility: "private" }); | |
| 122 | if (!repo.ok) throw new Error(String(repo.error)); | |
| 123 | secret = createToken(app.db, made.value.id, "app:poller/production", "ro").token; | |
| 124 | (globalThis as { __alice?: unknown }).__alice = made.value; | |
| 125 | }); | |
| 126 | afterAll(async () => { | |
| 127 | await servers.stop(); | |
| 128 | await rm(dir, { recursive: true, force: true }); | |
| 129 | }); | |
| 130 | ||
| 131 | test("a repeat read is a 304 from the write clock, without its query; a write ends it", async () => { | |
| 132 | const first = await mine("/api/v1/repos/alice/yard/runs"); | |
| 133 | expect(first.status).toBe(200); | |
| 134 | const etag = first.headers.get("etag")!; | |
| 135 | expect(etag).toBeTruthy(); | |
| 136 | await first.text(); | |
| 137 | // The route's own 304 ran its tag; the clock's answer ran nothing. | |
| 138 | const warm = await mine("/api/v1/repos/alice/yard/runs", { "if-none-match": etag }); | |
| 139 | expect(warm.status).toBe(304); | |
| 140 | const again = await mine("/api/v1/repos/alice/yard/runs", { "if-none-match": etag }); | |
| 141 | expect(again.status).toBe(304); | |
| 142 | expect(again.headers.get("server-timing") ?? "").not.toContain("tag"); | |
| 143 | expect(again.headers.get("etag")).toBe(etag); | |
| 144 | // Anything written moves the clock: the route runs again. | |
| 145 | const alice = (globalThis as { __alice?: never }).__alice!; | |
| 146 | const issue = await openIssue(app, alice, getRepo(app.db, "alice", "yard")!, { title: "x", body: "" }); | |
| 147 | expect(issue.ok).toBe(true); | |
| 148 | const after = await mine("/api/v1/repos/alice/yard/runs", { "if-none-match": etag }); | |
| 149 | expect(after.headers.get("server-timing") ?? "").toContain("tag"); | |
| 150 | }); | |
| 151 | ||
| 152 | test("a stranger's token is not given another's tag", async () => { | |
| 153 | const first = await mine("/api/v1/repos/alice/yard/runs"); | |
| 154 | const etag = first.headers.get("etag")!; | |
| 155 | const anon = await fetch(servers.url + "/api/v1/repos/alice/yard/runs", { headers: { "if-none-match": etag } }); | |
| 156 | expect(anon.status).toBe(404); | |
| 157 | }); | |
| 158 | ||
| 159 | test("a token over a fifth of the hour is a need of the forge's vitals, named with what to do", () => { | |
| 160 | for (let i = 0; i < 300; i++) asking.note("app:poller/production", "/api/v1/repos/:owner/:name/runs", 1); | |
| 161 | const line = forgeVitals(app).degraded.find((d) => d.cause.startsWith("asking:")); | |
| 162 | expect(line?.sentence).toContain("poller made"); | |
| 163 | expect(line?.sentence).toContain("to /runs"); | |
| 164 | expect(line?.sentence).toContain("put it to sleep"); | |
| 165 | }); | |
| 166 | ||
| 167 | test("the same URL asked for in another representation is not given the first's 304", async () => { | |
| 168 | const first = await mine("/api/v1/repos/alice/yard/runs?limit=7", { accept: "application/json" }); | |
| 169 | const etag = first.headers.get("etag")!; | |
| 170 | await first.text(); | |
| 171 | const same = await mine("/api/v1/repos/alice/yard/runs?limit=7", { accept: "application/json", "if-none-match": etag }); | |
| 172 | expect(same.status).toBe(304); | |
| 173 | const other = await mine("/api/v1/repos/alice/yard/runs?limit=7", { accept: "text/plain", "if-none-match": etag }); | |
| 174 | expect(other.headers.get("server-timing") ?? "").toContain("tag"); | |
| 175 | }); | |
| 176 | ||
| 177 | test("/system names who is asking to the owner only", async () => { | |
| 178 | const bob = await createUser(app.db, { origin: "seed", trusted: true, username: "bob", email: "[email protected]", password: "a-good-passphrase" }); | |
| 179 | if (!bob.ok) throw new Error(bob.error); | |
| 180 | const bobToken = createToken(app.db, bob.value.id, "bob-cli", "ro").token; | |
| 181 | const page = (token: string) => fetch(servers.url + "/system", { headers: { authorization: `Bearer ${token}` } }).then((r) => r.text()); | |
| 182 | expect(await page(secret)).toContain('data-asker="poller"'); | |
| 183 | expect(await page(bobToken)).not.toContain("Who is asking"); | |
| 184 | expect(await page(bobToken)).not.toContain("poller"); | |
| 185 | }); | |
| 186 | }); |
test/apps-sleep-slider.test.ts +13
| @@ -17,6 +17,7 @@ import type { Repo } from "../src/core/repos.ts"; | ||
| 17 | 17 | import { listRuns, type Run } from "../src/core/runs.ts"; |
| 18 | 18 | import { createUser, type User } from "../src/core/users.ts"; |
| 19 | 19 | import { serve, type Servers } from "../src/http/server.ts"; |
| 20 | import { asking } from "../src/perf/asking.ts"; | |
| 20 | 21 | import { readRunLog } from "../src/runs/log.ts"; |
| 21 | 22 | import { setEconomics } from "../src/settings/economics.ts"; |
| 22 | 23 | import { createForgeApp } from "../src/tui/forge.ts"; |
| @@ -142,4 +143,16 @@ describe("sleep windows follow the slider", () => { | ||
| 142 | 143 | tui.app.stop(); |
| 143 | 144 | expect(text).toContain("never sleeps (app.always_on)"); |
| 144 | 145 | }, 120_000); |
| 146 | test("an always-on app nobody opened for a day sleeps once it is seen calling the forge, and not before", async () => { | |
| 147 | ahead = 0; | |
| 148 | expect((await viaHost(`hello.${DOMAIN}`)).status).toBe(200); | |
| 149 | expect(production()?.alwaysOn).toBe(true); | |
| 150 | ahead = 25 * 60 * 60_000; | |
| 151 | await Bun.sleep(600); // real-timer: the sleeper loop must get its turn | |
| 152 | // Unopened for a day, but quiet toward the forge: always-on keeps it. | |
| 153 | expect(production()?.state).toBe("running"); | |
| 154 | asking.note("app:hello/production", "/api/v1/repos/:owner/:name/runs", 1, Date.now() + ahead); | |
| 155 | await until(() => (production()?.state === "asleep" ? true : undefined), 5_000); | |
| 156 | expect(log(production()!.runId!)).toContain("asleep: nothing asked for it in 24 h (always_on, but nobody opened it and it kept calling the forge)"); | |
| 157 | }, 30_000); | |
| 145 | 158 | }); |
test/timers-named.test.ts +8 −2
| @@ -74,6 +74,7 @@ const ALLOWED: readonly Allowed[] = [ | ||
| 74 | 74 | ...allow("setTimeout", `a shell session's resize debounce, ${PER_CONNECTION}`, [["src/machines/shell.ts", "() => session.size === resize"]]), |
| 75 | 75 | ...allow("setTimeout", `an ssh login's auth deadline, ${PER_CONNECTION}`, [["src/ssh/server.ts", "() => { this.transport.disconnect("]]), |
| 76 | 76 | ...allow("setTimeout", "a maintenance duty's deadline on one call: it rejects a promise and does no work on the loop", [["src/maintain/runner.ts", "() => reject(new Error(\"timed out\"))"]]), |
| 77 | ...allow("setTimeout", "an app request's look-again sleep in the ladder: it holds one request, does no work on the loop, and ends within the bound", [["src/http/app_turn.ts", "r"]]), | |
| 77 | 78 | ...allow("setTimeout", `a media stream's idle deadline, ${PER_CONNECTION}`, [["src/git/read.ts", "() => { finish(); controller.error("]]), |
| 78 | 79 | ...allow("setTimeout", `a git upload's idle deadline, ${PER_CONNECTION}`, [["src/git/smart-http.ts", "stop"]]), |
| 79 | 80 | ...allow("setTimeout", "the loop-lag meter's own tick: it measures how late the loop runs, and a span around it would count that lateness as its own work", [["src/kernel/loop_lag.ts", "() => { // Early"]]), |
| @@ -179,7 +180,11 @@ function sources(dir: string, out: string[] = []): string[] { | ||
| 179 | 180 | |
| 180 | 181 | /** The first argument of each `call(` in `text`: its text, whitespace collapsed, and whether its work runs inside `during`. */ |
| 181 | 182 | export function callbacks(text: string, call: Call): { callback: string; named: boolean; line: number }[] { |
| 182 | const source = ts.createSourceFile("x.ts", text, ts.ScriptTarget.Latest, true); | |
| 183 | return callbacksIn(ts.createSourceFile("x.ts", text, ts.ScriptTarget.Latest, false), call); | |
| 184 | } | |
| 185 | ||
| 186 | /** The same, from a file already parsed: the scan parses each file once for both calls. */ | |
| 187 | function callbacksIn(source: ts.SourceFile, call: Call): { callback: string; named: boolean; line: number }[] { | |
| 183 | 188 | const found: { callback: string; named: boolean; line: number }[] = []; |
| 184 | 189 | const visit = (node: ts.Node): void => { |
| 185 | 190 | if (ts.isCallExpression(node) && ts.isIdentifier(node.expression) && node.expression.text === call) { |
| @@ -272,7 +277,8 @@ function scan(): Timer[] { | ||
| 272 | 277 | for (const file of sources(SRC)) { |
| 273 | 278 | const text = readFileSync(join(ROOT, file), "utf8"); |
| 274 | 279 | if (!/\bset(Interval|Timeout)\(/.test(text)) continue; |
| 275 | for (const call of ["setInterval", "setTimeout"] as const) for (const c of callbacks(text, call)) out.push({ file, call, ...c }); | |
| 280 | const source = ts.createSourceFile("x.ts", text, ts.ScriptTarget.Latest, false); | |
| 281 | for (const call of ["setInterval", "setTimeout"] as const) for (const c of callbacksIn(source, call)) out.push({ file, call, ...c }); | |
| 276 | 282 | } |
| 277 | 283 | return out; |
| 278 | 284 | } |
test/tui-system.test.ts +12 −1
| @@ -16,12 +16,13 @@ import { createForgeApp, forgeCommands, Session } from "../src/tui/forge.ts"; | ||
| 16 | 16 | import { setTelemetryGrant } from "../src/actions/telemetry.ts"; |
| 17 | 17 | import { hasGrant } from "../src/core/telemetry.ts"; |
| 18 | 18 | import { countRead, resetTelemetryReads } from "../src/system/telemetry_reads.ts"; |
| 19 | import { dbSizeLines, dbSizeScreen, footprintLines, forgeLines, historyLines, largestLines, readerLines, memoryLines, memoryScreen, systemScreen } from "../src/tui/screens/system.ts"; | |
| 19 | import { askingLines, dbSizeLines, dbSizeScreen, footprintLines, forgeLines, historyLines, largestLines, readerLines, memoryLines, memoryScreen, systemScreen } from "../src/tui/screens/system.ts"; | |
| 20 | 20 | import type { DbSize } from "../src/system/db_size.ts"; |
| 21 | 21 | import { changeCapacity } from "../src/actions/capacity.ts"; |
| 22 | 22 | import { EGRESS_REPORTED } from "../src/system/egress_line.ts"; |
| 23 | 23 | import { keepFootprint } from "../src/system/footprint.ts"; |
| 24 | 24 | import { headlessTerminal } from "../src/tui/term/terminal.ts"; |
| 25 | import { asking } from "../src/perf/asking.ts"; | |
| 25 | 26 | import { quietLogs } from "./helpers/globals.ts"; |
| 26 | 27 | import { settleApp } from "./helpers/tui.ts"; |
| 27 | 28 | |
| @@ -108,6 +109,16 @@ describe("tui system memory", () => { | ||
| 108 | 109 | }); |
| 109 | 110 | }); |
| 110 | 111 | |
| 112 | describe("tui system who is asking (#12265)", () => { | |
| 113 | test("the owner reads who asks the most; nobody else reads a name", () => { | |
| 114 | for (let i = 0; i < 300; i++) asking.note("app:tui-poller/production", "/api/v1/repos/:owner/:name/runs", 1); | |
| 115 | const lines = askingLines(new Session(app, identity(carol))).join("\n"); | |
| 116 | expect(lines).toContain("Who is asking"); | |
| 117 | expect(lines).toContain("tui-poller made"); | |
| 118 | for (const who of [bob, bot, undefined]) expect(askingLines(new Session(app, identity(who)))).toEqual([]); | |
| 119 | }); | |
| 120 | }); | |
| 121 | ||
| 111 | 122 | describe("tui system largest directories (#11800)", () => { |
| 112 | 123 | const found = { at: new Date(Date.now() - 3 * 60_000).toISOString(), dirs: [{ name: "repos", bytes: 2 * 1024 ** 3 }, { name: "apps", bytes: 5 * MiB }] }; |
| 113 | 124 |
new above
offline
Raw bearer token is the map key; hash it. Add the Accept header (or Vary values) to the key.