trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns15g rEventsFlows

A gate whose install downloads from the npm registry says so in its log with the bytes, and /capacity names the repository that fetched most in the last hour #8394

Merged scoutcrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

scout/forge:scout/8382-a-gate-whose-install-downloads-from-the → main 2 commits fork

  • Merged. scout/8382-a-gate-whose-install-downloads-from-the was merged into main as 623ed31.

Review

Sign in to review this change.
14 files changed +373 −83
docs/MCP.md +1 −1
@@ -62,7 +62,7 @@ for a tunnel and a forge that sleeps.
6262 | `board_write` | `{ slug, board }` (person admins): write a whole board under a slug, made or replaced: `{ title, range, refresh, panels: [{ title, chart, query, w, h }] }` | yes |
6363 | `boards` | `{ slug?, panel?, from?, to?, step? }` (person admins): without `slug`, lists the forge's boards (slug, title, range, refresh, panels); with `slug`, that board; with `panel` (place from 0, up to 23), that panel's data over the `from`/`to` window and `step` | |
6464 | `branches` | `{ repo, name?, action?, from?, kind? }`: without `name`, lists the branches; `kind: "tags"` lists the tags instead (as GET `/api/v1/repos/:owner/:name/tags` does, newest first; no `name` or `action`); with `name` and `action` (`create` from `from`, or `delete`) it does what the branches page does: the default branch is never deleted, only a writer may act | yes |
65 | `capacity` | `{ key?, value?, dryRun? }` (forge admin): the forge's capacity and guards, live (crew and checks concurrency, check shards, cell egress, the new-account guard, the sign-in edge, queue depth, child forges, `egress.npmBytesLastHour`; only the first six are settable by `key`); with `key` and `value` (`""` restores the default), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token | yes |
65 | `capacity` | `{ key?, value?, dryRun? }` (forge admin): the forge's capacity and guards, live (crew and checks concurrency, check shards, cell egress, the new-account guard, the sign-in edge, queue depth, child forges, `egress.npmBytesLastHour` and `npmTopRepoLastHour`; only the first six are settable by `key`); with `key` and `value` (`""` restores the default), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token | yes |
6666 | `claim_link` | `{ name }`: a new claim link for a child you spawned that nobody has claimed; the old one stops working. An admin's | yes |
6767 | `close` | `{ repo, number, state?, reason?, into? }`: close an issue or a pull (`state` closed by default) or reopen it (`open`); the same rules and refusals as `PATCH …/issues/:number`: an agent closes only what the rules give it and reopens nothing; answers `state` and `closeReason` | yes |
6868 | `cloud_account` | `{ verb, provider, name, value?, to?, allowed?, repo? }`: change a cloud account: `attach` or `rotate` a token (`value`), `verify` it, `rename` it (`to`), `remove` it, `allow_writes` (`allowed` true or false) or `audit` (`repo` owner/name); a forge admin with an admin-scope token; answers a one-line `says` | yes |
docs/METRICS.md +13 −5
@@ -76,12 +76,20 @@ by run, so the series stay one per host). They show the bun cache working
7676 stays near zero, and a cold one is hundreds of megabytes an hour (#7541).
7777 Without Prometheus, the same hour is read four ways:
7878
79 - MCP `capacity` (forge admin) answers `egress.npmBytesLastHour` and the ten
80 busiest `egress.destinations` with their totals and last hour.
81 - `/settings/capacity` prints it in the Egress connections card.
79 - MCP `capacity` (forge admin) answers `egress.npmBytesLastHour`,
80 `egress.npmTopRepoLastHour` (the repository whose contained installs the
81 registry sent most this hour, attributed by the cell's own meter, never by
82 install output; null when none) and the ten busiest `egress.destinations`
83 with their totals and last hour.
84 - `/settings/capacity` prints it, with the top repository, in the Egress
85 connections card.
8286 - The terminal's Egress connections screen (About, `c`, site admin) prints the
83 same line above the open connections.
87 same line, with the top repository, above the open connections; all three
88 read `egressHour`.
8489 - Each check's install step in a cell logs one line, `install fetched 0 B from
8590 the registry` on a warm cache: what the cell's egress proxy received. A check
8691 outside a cell logs none: its repository's bun cache is shared, so no figure
87 there would be its own.
92 there would be its own. A contained check, gate or app install that received
93 more than 1 MiB from `registry.npmjs.org` ends its log with `install fetched
94 N MiB from the npm registry; the shared cache missed` (#8382), failed or
95 aborted installs included.
src/actions/capacity.ts +14 −3
@@ -22,7 +22,8 @@ import {
2222 type CapacityChange,
2323 type CapacityKey,
2424 } from "../core/capacity.ts";
25 import { egressTotals, npmBytesLastHour, type EgressTotal } from "../apps/egress.ts";
25 import { egressTotals, npmBytesLastHour, npmTopRepoLastHour, type EgressTotal } from "../apps/egress.ts";
26 import { getRepoById } from "../core/repos.ts";
2627 import { passkeyIds } from "../core/passkeys.ts";
2728 import type { User } from "../core/users.ts";
2829 import { err, ok, type Result } from "../kernel/result.ts";
@@ -57,13 +58,23 @@ export interface CapacityRow {
5758 */
5859 export interface EgressHour {
5960 readonly npmBytesLastHour: number;
61 /**
62 * The repository whose contained installs the registry sent most this
63 * hour (#8382), as `owner/name`, attributed by the cell that fetched;
64 * null when no metered install fetched, or its repository is gone.
65 */
66 readonly npmTopRepoLastHour: { readonly repo: string; readonly bytes: number } | null;
6067 /** The ten destinations that sent most this hour, with their totals since boot. */
6168 readonly destinations: readonly EgressTotal[];
6269 }
6370
64 export function egressHour(now = Date.now()): EgressHour {
71 export function egressHour(db: Database, now = Date.now()): EgressHour {
72 const top = npmTopRepoLastHour(now);
73 const repo = top === undefined ? undefined : getRepoById(db, top.repoId);
6574 return {
6675 npmBytesLastHour: npmBytesLastHour(now),
76 npmTopRepoLastHour:
77 top === undefined || repo === undefined ? null : { repo: `${repo.ownerName}/${repo.name}`, bytes: top.bytes },
6778 destinations: egressTotals(now).slice(0, 10),
6879 };
6980 }
@@ -104,7 +115,7 @@ export function listCapacity(
104115 ...changedOf(app.db, key),
105116 })),
106117 changes: capacityChanges(app.db),
107 egress: egressHour(),
118 egress: egressHour(app.db),
108119 });
109120 }
110121
src/apps/egress.ts +96 −19
@@ -129,8 +129,51 @@ interface Moved {
129129 const MOVED_MAX = 256;
130130 const HOUR_MINUTES = 60;
131131 const moved = new Map<string, Moved>();
132 /** label -> bytes in, for the labels someone is metering now (`meterEgress`). */
133 const meters = new Map<string, { bytesIn: number; holders: number }>();
132 /**
133 * label -> bytes in, for the labels someone is metering now (`meterEgress`),
134 * and the repository whose cell the label is: the one thing that says whose
135 * registry bytes these are (#8382), set by the caller that opened the cell.
136 */
137 const meters = new Map<string, { bytesIn: number; npmIn: number; holders: number; repoId: string | undefined }>();
138 /**
139 * Registry bytes per repository, minute by minute over the last hour
140 * (#8382). Only a metered install with a repository lands here; past
141 * `NPM_REPOS_MAX` repositories the quietest is dropped first.
142 */
143 const npmByRepo = new Map<string, Moved>();
144 const NPM_REPOS_MAX = 256;
145
146 function ring(): Moved {
147 return { bytesIn: 0, bytesOut: 0, minute: Array(HOUR_MINUTES).fill(-1), minuteIn: Array(HOUR_MINUTES).fill(0) };
148 }
149
150 /** Add `bytesIn` to `m` at minute `minute`. */
151 function addMinute(m: Moved, minute: number, bytesIn: number): void {
152 const slot = minute % HOUR_MINUTES;
153 if (m.minute[slot] !== minute) {
154 m.minute[slot] = minute;
155 m.minuteIn[slot] = 0;
156 }
157 m.minuteIn[slot] = (m.minuteIn[slot] ?? 0) + bytesIn;
158 }
159
160 function recordNpmRepo(repoId: string, bytesIn: number, minute: number): void {
161 let m = npmByRepo.get(repoId);
162 if (m === undefined) {
163 if (npmByRepo.size >= NPM_REPOS_MAX) {
164 let quietest: string | undefined;
165 let least = Infinity;
166 for (const [id, r] of npmByRepo) {
167 const hour = lastHour(r, minute);
168 if (hour < least) [quietest, least] = [id, hour];
169 }
170 if (quietest !== undefined) npmByRepo.delete(quietest);
171 }
172 npmByRepo.set(repoId, (m = ring()));
173 }
174 m.bytesIn += bytesIn;
175 addMinute(m, minute, bytesIn);
176 }
134177
135178 /** Count `bytesIn` received from and `bytesOut` sent to `destination`. */
136179 export function recordEgress(
@@ -145,19 +188,17 @@ export function recordEgress(
145188 ? destination
146189 : "other";
147190 let m = moved.get(key);
148 if (m === undefined)
149 moved.set(key, (m = { bytesIn: 0, bytesOut: 0, minute: Array(HOUR_MINUTES).fill(-1), minuteIn: Array(HOUR_MINUTES).fill(0) }));
191 if (m === undefined) moved.set(key, (m = ring()));
150192 m.bytesIn += bytesIn;
151193 m.bytesOut += bytesOut;
152194 const minute = Math.floor(now / 60_000);
153 const slot = minute % HOUR_MINUTES;
154 if (m.minute[slot] !== minute) {
155 m.minute[slot] = minute;
156 m.minuteIn[slot] = 0;
157 }
158 m.minuteIn[slot] = (m.minuteIn[slot] ?? 0) + bytesIn;
195 addMinute(m, minute, bytesIn);
159196 const meter = meters.get(label);
160 if (meter !== undefined) meter.bytesIn += bytesIn;
197 if (meter === undefined) return;
198 meter.bytesIn += bytesIn;
199 if (destination !== NPM_REGISTRY || bytesIn === 0) return;
200 meter.npmIn += bytesIn;
201 if (meter.repoId !== undefined) recordNpmRepo(meter.repoId, bytesIn, minute);
161202 }
162203
163204 export interface EgressTotal {
@@ -198,23 +239,58 @@ export function npmBytesLastHour(now = Date.now()): number {
198239 return m === undefined ? 0 : lastHour(m, Math.floor(now / 60_000));
199240 }
200241
242 /** The repository that received most from the npm registry over the last hour, by id. */
243 export interface NpmTopRepo {
244 readonly repoId: string;
245 readonly bytes: number;
246 }
247
248 /** The metered repository the registry sent most to in the last hour; none when no metered install fetched. */
249 export function npmTopRepoLastHour(now = Date.now()): NpmTopRepo | undefined {
250 const minute = Math.floor(now / 60_000);
251 let top: NpmTopRepo | undefined;
252 for (const [repoId, m] of npmByRepo) {
253 const bytes = lastHour(m, minute);
254 if (bytes > 0 && (top === undefined || bytes > top.bytes || (bytes === top.bytes && repoId < top.repoId)))
255 top = { repoId, bytes };
256 }
257 return top;
258 }
259
260 export interface EgressMeter {
261 /** Stop counting; answers the bytes received from every destination. */
262 stop(): number;
263 /** Bytes received from the npm registry alone, up to `stop()`. */
264 npm(): number;
265 }
266
201267 /**
202268 * Count what `label`'s proxy receives from now until `stop()`, which answers
203269 * the bytes in. Only labels being metered are kept, and each goes with its
204 * last holder, so a run label never outlives the step that asked.
270 * last holder, so a run label never outlives the step that asked. `repoId`
271 * is the repository whose cell `label` names, from the caller that opened
272 * it: the registry bytes are attributed to it, never to anything the
273 * install printed.
205274 */
206 export function meterEgress(label: string): { stop(): number } {
207 const meter = meters.get(label) ?? { bytesIn: 0, holders: 0 };
275 export function meterEgress(label: string, repoId?: string): EgressMeter {
276 const meter = meters.get(label) ?? { bytesIn: 0, npmIn: 0, holders: 0, repoId };
277 // An earlier holder that knew no repository must not lose a later one's.
278 meter.repoId ??= repoId;
208279 meters.set(label, meter);
209280 meter.holders++;
210281 const start = meter.bytesIn;
211 let stopped = false;
282 const npmStart = meter.npmIn;
283 let stopped: { bytes: number; npm: number } | undefined;
212284 return {
213285 stop() {
214 const bytes = meter.bytesIn - start;
215 if (!stopped && --meter.holders === 0) meters.delete(label);
216 stopped = true;
217 return bytes;
286 if (stopped === undefined) {
287 stopped = { bytes: meter.bytesIn - start, npm: meter.npmIn - npmStart };
288 if (--meter.holders === 0) meters.delete(label);
289 }
290 return stopped.bytes;
291 },
292 npm() {
293 return stopped?.npm ?? meter.npmIn - npmStart;
218294 },
219295 };
220296 }
@@ -222,6 +298,7 @@ export function meterEgress(label: string): { stop(): number } {
222298 /** For tests: forget every cumulative count. */
223299 export function resetEgressTotals(): void {
224300 moved.clear();
301 npmByRepo.clear();
225302 }
226303
227304 /** The ports a declared host may be reached on when its entry names none. */
src/apps/process.ts +11 −4
@@ -19,7 +19,7 @@ import { isReservedAppEnvName } from "../core/app_env.ts";
1919 import { err, ok, type Result } from "../kernel/result.ts";
2020 import { spellings, stripForgeDirs } from "../runs/paths.ts";
2121 import { seconds } from "./timing.ts";
22 import { cellInstall, prefillInstallCache, withLock } from "../checks/install.ts";
22 import { cellInstall, meteredInstall, prefillInstallCache, withLock } from "../checks/install.ts";
2323 import {
2424 commandPid,
2525 killTree,
@@ -96,6 +96,8 @@ export interface CellBase {
9696 readonly egressMode?: "enforce" | "report" | undefined;
9797 /** Whose cell, for the machine sampler: `app:<slug>/<env>`. */
9898 readonly label?: string | undefined;
99 /** Whose repository the cell runs, for the npm registry tally (`apps/egress.ts`). */
100 readonly repoId?: string | undefined;
99101 }
100102
101103 export type StartStage = "install" | "test" | "start" | "ready";
@@ -451,9 +453,14 @@ async function install(
451453 const plan = installPlan(command, input, env, filled);
452454 const step = (): Promise<Result<void, string>> =>
453455 runStep("install", plan.command, { cwd: input.dir, env: plan.env, log: input.log, signal: input.signal, cell: plan.cell });
454 if (plan.lock === undefined) return step();
455 const held = await withLock(plan.lock, step, { signal: input.signal });
456 return held.ok ? held.value : held;
456 if (plan.lock !== undefined) {
457 const held = await withLock(plan.lock, step, { signal: input.signal });
458 return held.ok ? held.value : held;
459 }
460 // A contained install is metered by its cell's label (#8382): more than
461 // a MiB from the registry is the shared cache missing, and says so, even
462 // when the install failed or was aborted.
463 return meteredInstall(plan.cell?.label, input.cell?.repoId, step, input.log);
457464 }
458465
459466 /**
src/apps/runtime.ts +1
@@ -1485,6 +1485,7 @@ export class AppsRuntime {
14851485 // Likewise an admin's flip of the egress mode.
14861486 egressMode: egressSetting(this.app.db),
14871487 label: `app:${record.slug}/${env}`,
1488 repoId: record.repoId,
14881489 };
14891490 }
14901491
src/checks/install.ts +41
@@ -18,6 +18,7 @@
1818
1919 import { cpSync, existsSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
2020 import { tmpdir } from "node:os";
21 import { meterEgress } from "../apps/egress.ts";
2122 import { dirname, join } from "node:path";
2223 import { formatBytes } from "../core/storage.ts";
2324 import { err, ok, type Result } from "../kernel/result.ts";
@@ -46,6 +47,46 @@ export function installFetchedLine(bytes: number): string {
4647 return `install fetched ${formatBytes(Math.max(0, bytes))} from the registry`;
4748 }
4849
50 /** More than this from the npm registry in one install is the shared cache missing (#8382). */
51 export const NPM_MISS_BYTES = 1024 * 1024;
52
53 /**
54 * The line a contained install's log ends with when its cell received more
55 * than `NPM_MISS_BYTES` from the npm registry; none under it.
56 */
57 export function npmMissLine(npmBytes: number): string | undefined {
58 if (!(npmBytes > NPM_MISS_BYTES)) return undefined;
59 const mib = npmBytes / NPM_MISS_BYTES;
60 return `install fetched ${mib < 10 ? mib.toFixed(1) : Math.round(mib)} MiB from the npm registry; the shared cache missed`;
61 }
62
63 /**
64 * Run a contained install `step` with its cell's egress proxy metered by
65 * `label`, the registry's bytes attributed to `repoId` (#8382). After the
66 * step — in `finally`, so a failed, aborted or throwing install logs too —
67 * `write` gets `installFetchedLine` when `fetched` asks for it (a check's
68 * #7546 line), then `npmMissLine` when the registry sent more than a MiB.
69 * With no `label` the install is uncontained: nothing is metered or logged.
70 */
71 export async function meteredInstall<T>(
72 label: string | undefined,
73 repoId: string | undefined,
74 step: () => Promise<T>,
75 write: (line: string) => void,
76 options: { readonly fetched?: boolean } = {},
77 ): Promise<T> {
78 if (label === undefined) return step();
79 const meter = meterEgress(label, repoId);
80 try {
81 return await step();
82 } finally {
83 const fetched = meter.stop();
84 if (options.fetched === true) write(installFetchedLine(fetched));
85 const missed = npmMissLine(meter.npm());
86 if (missed !== undefined) write(missed);
87 }
88 }
89
4990 export type InstallOutcome =
5091 | { readonly ok: true }
5192 | { readonly ok: false; readonly error: string; readonly infrastructure: boolean };
src/checks/service.ts +42 −43
@@ -110,13 +110,12 @@ import { JOB_VANISHED, runJobStep } from "./job_step.ts";
110110 import {
111111 cellInstall,
112112 checkInstallCache,
113 installFetchedLine,
113 meteredInstall,
114114 installWithRetry,
115115 prefillInstallCache,
116116 withLock,
117117 } from "./install.ts";
118118 import { evictSoon, lend } from "./install_evict.ts";
119 import { meterEgress } from "../apps/egress.ts";
120119 import { checkStage, type GateStage } from "./gate_reason.ts";
121120 import { proofsInFlight } from "./proof.ts";
122121 import { extraGates, GATE_DEADLINE_MS, gateQueued, gateRunId, gatesInFlight, watchGates } from "./queue.ts";
@@ -1232,48 +1231,48 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
12321231 // What the cell's proxy received, for the one line that shows the cache
12331232 // working (#7546). Uncontained, nothing is logged: the shared cache
12341233 // is not metered, and its growth is wrong under concurrent checks.
1235 // Stopped in `finally`, so a throw never leaves the label metered.
1236 const meter = cell === undefined ? undefined : meterEgress(`check:${run.id}`);
1237 let fetched: number | undefined;
1238 let installed: Awaited<ReturnType<typeof installWithRetry>>;
1239 try {
1240 installed = await installWithRetry(
1241 async (log) => {
1242 if (shared === undefined) {
1243 if (warm !== undefined) {
1244 const fill = await prefillInstallCache({
1245 cache: warm,
1246 dir,
1247 env,
1248 log,
1249 signal,
1250 timeoutMs: manifest.timeout * 1000,
1251 });
1252 if (!fill.ok) return fill;
1253 filled = fill.value;
1254 void evictSoon(warm, process.env);
1234 // Stopped in `finally`, so a throw never leaves the label metered;
1235 // over a MiB from the npm registry adds the miss line (#8382).
1236 const installed = await meteredInstall(
1237 cell === undefined ? undefined : `check:${run.id}`,
1238 repo.id,
1239 () =>
1240 installWithRetry(
1241 async (log) => {
1242 if (shared === undefined) {
1243 if (warm !== undefined) {
1244 const fill = await prefillInstallCache({
1245 cache: warm,
1246 dir,
1247 env,
1248 log,
1249 signal,
1250 timeoutMs: manifest.timeout * 1000,
1251 });
1252 if (!fill.ok) return fill;
1253 filled = fill.value;
1254 void evictSoon(warm, process.env);
1255 }
1256 return once(log);
12551257 }
1256 return once(log);
1257 }
1258 mkdirSync(shared, { recursive: true });
1259 const held = await withLock(
1260 `${shared}.lock`,
1261 async () => {
1262 const done = await once(log);
1263 if (done.ok) lend(shared, dir);
1264 return done;
1265 },
1266 { signal },
1267 );
1268 if (held.ok && held.value.ok) void evictSoon(shared, process.env);
1269 return held.ok ? held.value : held;
1270 },
1271 { log: write, signal, retryMs: options.installRetryMs },
1272 );
1273 } finally {
1274 fetched = meter?.stop();
1275 }
1276 if (fetched !== undefined) write(installFetchedLine(fetched));
1258 mkdirSync(shared, { recursive: true });
1259 const held = await withLock(
1260 `${shared}.lock`,
1261 async () => {
1262 const done = await once(log);
1263 if (done.ok) lend(shared, dir);
1264 return done;
1265 },
1266 { signal },
1267 );
1268 if (held.ok && held.value.ok) void evictSoon(shared, process.env);
1269 return held.ok ? held.value : held;
1270 },
1271 { log: write, signal, retryMs: options.installRetryMs },
1272 ),
1273 write,
1274 { fetched: true },
1275 );
12771276 if (!installed.ok)
12781277 return {
12791278 ok: false,
src/http/mcp/tools/capacity.ts +1 −1
@@ -5,7 +5,7 @@ import { good, bad, type Tool } from "../shared.ts";
55
66 export const tool: Tool = {
77 name: "capacity",
8 doc: "`{ key?, value?, dryRun? }` (forge admin): the forge's capacity and guards, live (crew and checks concurrency, check shards, cell egress, the new-account guard, the sign-in edge, queue depth, child forges, `egress.npmBytesLastHour`; only the first six are settable by `key`); with `key` and `value` (`\"\"` restores the default), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token",
8 doc: "`{ key?, value?, dryRun? }` (forge admin): the forge's capacity and guards, live (crew and checks concurrency, check shards, cell egress, the new-account guard, the sign-in edge, queue depth, child forges, `egress.npmBytesLastHour` and `npmTopRepoLastHour`; only the first six are settable by `key`); with `key` and `value` (`\"\"` restores the default), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token",
99 description:
1010 "Forge capacity and guards, live. With `key` and `value` (\"\" restores default), plans.",
1111 inputSchema: {
src/tui/screens/egress.ts +8 −3
@@ -11,7 +11,8 @@ import {
1111 revokeAppEgress,
1212 type EgressRow,
1313 } from "../../actions/egress_grants.ts";
14 import { egressSnapshot, npmBytesLastHour } from "../../apps/egress.ts";
14 import { egressHour } from "../../actions/capacity.ts";
15 import { egressSnapshot } from "../../apps/egress.ts";
1516 import { bytes } from "../../system/format.ts";
1617 import type { Repo } from "../../core/repos.ts";
1718 import type { Ctx, Screen } from "../app/app.ts";
@@ -117,7 +118,8 @@ export function egressScreen(f: Session, ctx: Ctx, repo: Repo): Screen {
117118 /**
118119 * The egress connections open right now, per app and destination: the
119120 * Capacity page's card, for a site admin, under the bytes the npm registry
120 * sent over the last hour (#7546). Read again on every draw, so it moves as
121 * sent over the last hour (#7546) and the repository it sent most to (#8382),
122 * both from `egressHour`, as the page and MCP read them. Read again on every draw, so it moves as
121123 * connections do; anyone else is offered nothing.
122124 */
123125 export function connectionsScreen(f: Session, ctx: Ctx): Screen {
@@ -138,7 +140,10 @@ export function connectionsScreen(f: Session, ctx: Ctx): Screen {
138140 state.clamp(rows.length);
139141 const [npm, head, list] = split(area, "v", [len(1), len(1), fill(1)]);
140142 if (f.user?.isAdmin === true) {
141 const sent = `The npm registry sent ${bytes(npmBytesLastHour())} over the last hour.`;
143 const hour = egressHour(f.app.db);
144 const top = hour.npmTopRepoLastHour;
145 const most = top === null ? "." : `, most to ${top.repo} (${bytes(top.bytes)}).`;
146 const sent = `The npm registry sent ${bytes(hour.npmBytesLastHour)} over the last hour${most}`;
142147 s.text(npm!.x, npm!.y, truncate(sent, npm!.w), style(c.mutedFg, c.bg), npm!.x + npm!.w);
143148 }
144149 const note =
src/web/pages/capacity.ts +7 −1
@@ -160,7 +160,13 @@ export function capacityPage(ctx: Ctx): Response {
160160 const open = egressSnapshot();
161161 // The bun cache at work (#7546): a warm cache keeps this near zero.
162162 const npm = html`The npm registry sent
163 <span class="mono">${bytes(egress.npmBytesLastHour)}</span> over the last hour.`;
163 <span class="mono">${bytes(egress.npmBytesLastHour)}</span> over the last hour${
164 egress.npmTopRepoLastHour === null
165 ? "."
166 : html`, most to
167 <span class="mono">${egress.npmTopRepoLastHour.repo}</span>
168 (<span class="mono">${bytes(egress.npmTopRepoLastHour.bytes)}</span>).`
169 }`;
164170 const connections = card(
165171 open.length === 0
166172 ? html`<div class="card-body">
test/capacity.test.ts +19 −1
@@ -18,7 +18,7 @@ import { addToCrew, createAgent } from "../src/actions/crew.ts";
1818 import { assignIssue, openIssue } from "../src/actions/issues.ts";
1919 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
2020 import { enableApp } from "../src/actions/apps.ts";
21 import { NPM_REGISTRY, recordEgress, resetEgressTotals, startEgress } from "../src/apps/egress.ts";
21 import { meterEgress, NPM_REGISTRY, recordEgress, resetEgressTotals, startEgress } from "../src/apps/egress.ts";
2222 import { AppsRuntime } from "../src/apps/runtime.ts";
2323 import { startChecks, type ChecksHandle } from "../src/checks/service.ts";
2424 import {
@@ -161,6 +161,24 @@ describe("capacity: only a forge admin reads or changes it", () => {
161161 resetEgressTotals();
162162 });
163163
164 test("MCP capacity and the page name the repository the registry sent most this hour (#8382)", async () => {
165 resetEgressTotals();
166 const made = await createRepo(app, alice, { name: "npm-heavy", visibility: "public" });
167 if (!made.ok) throw new Error(String(made.error));
168 const meter = meterEgress("check:heavy", made.value.id);
169 recordEgress("check:heavy", NPM_REGISTRY, 5 * 1024 * 1024, 0);
170 meter.stop();
171 const read = await mcp(aliceRo, {});
172 const egress = (JSON.parse(read.text) as { egress: { npmTopRepoLastHour: { repo: string; bytes: number } | null } }).egress;
173 expect(egress.npmTopRepoLastHour).toEqual({ repo: "alice/npm-heavy", bytes: 5 * 1024 * 1024 });
174 const cookie = await login("alice");
175 const page = await (await fetch(`${servers.url}/settings/capacity`, { headers: { cookie } })).text();
176 expect(page).toMatch(/id="egress-npm"[\s\S]*?alice\/npm-heavy[\s\S]*?5\.0 MiB/);
177 resetEgressTotals();
178 const none = await mcp(aliceRo, {});
179 expect((JSON.parse(none.text) as { egress: { npmTopRepoLastHour: unknown } }).egress.npmTopRepoLastHour).toBeNull();
180 });
181
164182 test("a member, an agent and nobody are told it is not there, at every door", async () => {
165183 expect(listCapacity(app, bob).ok).toBe(false);
166184 expect(listCapacity(app, agent).ok).toBe(false);
test/egress-admin.test.ts +5 −2
@@ -13,7 +13,7 @@ import { commitToBranch, createRepo } from "../src/actions/repos.ts";
1313 import { listEgressGrants } from "../src/core/egress_grants.ts";
1414 import type { Repo } from "../src/core/repos.ts";
1515 import { createUser, type User } from "../src/core/users.ts";
16 import { NPM_REGISTRY, recordEgress, resetEgressTotals, startEgress } from "../src/apps/egress.ts";
16 import { meterEgress, NPM_REGISTRY, recordEgress, resetEgressTotals, startEgress } from "../src/apps/egress.ts";
1717 import { createForgeApp } from "../src/tui/forge.ts";
1818 import { headlessTerminal } from "../src/tui/term/terminal.ts";
1919 import { quietLogs } from "./helpers/globals.ts";
@@ -144,7 +144,9 @@ describe("egress grants in the terminal", () => {
144144 client.write(`CONNECT 127.0.0.1:${camera.port} HTTP/1.1\r\n\r\n`);
145145 while (!reply.includes("200")) await Bun.sleep(10);
146146 resetEgressTotals();
147 const meter = meterEgress("check:r", repo.id);
147148 recordEgress("check:r", NPM_REGISTRY, 3_000_000, 0);
149 meter.stop();
148150 const admin = boot(ada);
149151 const other = boot(bob);
150152 try {
@@ -153,7 +155,8 @@ describe("egress grants in the terminal", () => {
153155 await settleUntil(admin.tui, "app:cam/main");
154156 expect(admin.text()).toContain(`127.0.0.1:${camera.port} · 1 open`);
155157 // The registry's hour, as the Capacity page and MCP `capacity` read it (#7546).
156 expect(admin.text()).toContain("The npm registry sent 2.9 MiB over the last hour.");
158 // …and the repository it sent most to, from the same `egressHour` (#8382).
159 expect(admin.text()).toContain("The npm registry sent 2.9 MiB over the last hour, most to ada/arm (2.9 MiB).");
157160 await other.openAbout();
158161 await other.press("c");
159162 expect(other.text()).not.toContain("Egress connections");
test/egress-npm-repo.test.ts added +114
@@ -0,0 +1,114 @@
1 /**
2 * A contained install that fetches more than a MiB from the npm registry
3 * says the shared cache missed, and the registry's bytes are kept per
4 * repository for the last hour (#8382), attributed by the cell's meter —
5 * never by anything the install printed.
6 */
7
8 import { afterEach, beforeEach, expect, test } from "bun:test";
9 import { meterEgress, NPM_REGISTRY, npmTopRepoLastHour, recordEgress, resetEgressTotals } from "../src/apps/egress.ts";
10 import { meteredInstall, npmMissLine } from "../src/checks/install.ts";
11
12 beforeEach(resetEgressTotals);
13 afterEach(resetEgressTotals);
14
15 const T = Date.UTC(2026, 10, 2, 12, 0, 0);
16 const MIN = 60_000;
17 const MiB = 1024 * 1024;
18
19 /** One metered install: what its proxy received, then the line its log ends with. */
20 function install(label: string, repoId: string | undefined, received: [string, number][], now = T): string | undefined {
21 const meter = meterEgress(label, repoId);
22 for (const [destination, bytes] of received) recordEgress(label, destination, bytes, 0, now);
23 meter.stop();
24 return npmMissLine(meter.npm());
25 }
26
27 test("an install over a MiB from the registry ends with the miss line; one under it does not", () => {
28 expect(install("check:big", "r1", [[NPM_REGISTRY, 3 * MiB]])).toBe(
29 "install fetched 3.0 MiB from the npm registry; the shared cache missed",
30 );
31 expect(install("check:small", "r1", [[NPM_REGISTRY, 512 * 1024]])).toBeUndefined();
32 expect(install("check:edge", "r1", [[NPM_REGISTRY, MiB]])).toBeUndefined();
33 });
34
35 test("only the registry's bytes count toward the miss, not other hosts'", () => {
36 expect(install("check:gh", "r1", [["github.com:443", 50 * MiB], [NPM_REGISTRY, 10]])).toBeUndefined();
37 });
38
39 test("bytes after stop() are not the install's", () => {
40 const meter = meterEgress("check:late", "r1");
41 recordEgress("check:late", NPM_REGISTRY, 100, 0, T);
42 meter.stop();
43 recordEgress("check:late", NPM_REGISTRY, 9 * MiB, 0, T);
44 expect(meter.npm()).toBe(100);
45 });
46
47 test("two repositories' installs are attributed to each, and the top one is named", () => {
48 install("check:a", "repo-a", [[NPM_REGISTRY, 2 * MiB]]);
49 install("app:b/pr-1", "repo-b", [[NPM_REGISTRY, 3 * MiB]]);
50 install("check:a2", "repo-a", [[NPM_REGISTRY, 2 * MiB]]);
51 expect(npmTopRepoLastHour(T)).toEqual({ repoId: "repo-a", bytes: 4 * MiB });
52 // An hour on, repo-a's minute has aged out while repo-b fetched again.
53 install("app:b/pr-1", "repo-b", [[NPM_REGISTRY, 1]], T + 60 * MIN);
54 expect(npmTopRepoLastHour(T + 60 * MIN)).toEqual({ repoId: "repo-b", bytes: 1 });
55 });
56
57 test("unmetered or repository-less traffic is attributed to no repository", () => {
58 recordEgress("check:nobody", NPM_REGISTRY, 9 * MiB, 0, T);
59 install("check:anon", undefined, [[NPM_REGISTRY, 9 * MiB]]);
60 expect(npmTopRepoLastHour(T)).toBeUndefined();
61 });
62
63 test("a later holder's repository is kept when the first holder of the label had none", () => {
64 const first = meterEgress("app:x/main");
65 const second = meterEgress("app:x/main", "repo-x");
66 recordEgress("app:x/main", NPM_REGISTRY, 2 * MiB, 0, T);
67 second.stop();
68 first.stop();
69 expect(npmTopRepoLastHour(T)).toEqual({ repoId: "repo-x", bytes: 2 * MiB });
70 });
71
72 // The step both the check/gate site (`checks/service.ts`) and the app and
73 // preview site (`apps/process.ts`) run their contained install through.
74 test("a metered install's log ends with the miss line, after the fetched line a check asks for", async () => {
75 const lines: string[] = [];
76 const result = await meteredInstall(
77 "check:run-1",
78 "repo-c",
79 async () => {
80 recordEgress("check:run-1", NPM_REGISTRY, 5 * MiB, 0, T);
81 return "done";
82 },
83 (line) => lines.push(line),
84 { fetched: true },
85 );
86 expect(result).toBe("done");
87 expect(lines).toEqual([
88 "install fetched 5.2 MB from the registry",
89 "install fetched 5.0 MiB from the npm registry; the shared cache missed",
90 ]);
91 expect(npmTopRepoLastHour(T)).toEqual({ repoId: "repo-c", bytes: 5 * MiB });
92 });
93
94 test("an app install logs only the miss line, and logs it when the install throws", async () => {
95 const lines: string[] = [];
96 const failing = meteredInstall(
97 "app:y/pr-2",
98 "repo-y",
99 async () => {
100 recordEgress("app:y/pr-2", NPM_REGISTRY, 2 * MiB, 0, T);
101 throw new Error("aborted");
102 },
103 (line) => lines.push(line),
104 );
105 await expect(failing).rejects.toThrow("aborted");
106 expect(lines).toEqual(["install fetched 2.0 MiB from the npm registry; the shared cache missed"]);
107 });
108
109 test("an uncontained install is not metered and logs nothing", async () => {
110 const lines: string[] = [];
111 await meteredInstall(undefined, "repo-z", async () => recordEgress("", NPM_REGISTRY, 9 * MiB, 0, T), (l) => lines.push(l), { fetched: true });
112 expect(lines).toEqual([]);
113 expect(npmTopRepoLastHour(T)).toBeUndefined();
114 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs