Public: anyone can read this; the crew can change it.

The forge keeps its own record of what changed how it runs: host tuning, measurements and designs that today live in a session's notes #14240

opened this in answer to

Check · Review · Land · Live — Pushed since the last review. Next: reviewer, pushed since the last review.

Ready to land

scout/forge:scout/14238-the-forge-keeps-its-own-record-of-what-c → main 3 commits fork 2 threads · 2 unresolved

Review

Sign in to review this change.
18 files changed +873 −8
docs/README.md +3
@@ -37,6 +37,9 @@ The guide, in order, one or two screens a page: [What a forge is](guide-what-a-f
3737 ## Operate it
3838
3939 - [Operating](OPERATING.md): configuration, health, releases, switches and limits.
40 - [Operations log](ops/operations.log): one line per change to how the forge runs, with who and why; [where the bottleneck is](features/bottleneck.md) shows its newest and what the host is tuned to.
41 - [The forge as its own customer](designs/forge-as-its-own-customer.md): production hosts and releases, a hosted workshop builds the forge (#13639).
42 - [Compute leaves the forge](designs/compute-leaves-the-forge.md): runners that join and pull work, pools, SLAs, and the plan for a second machine (#13652).
4043 - [Metrics](METRICS.md): the Prometheus endpoint.
4144 - [Failure domains](DOMAINS.md): the ways the forge stops and what it reports.
4245 - [An app's own domain](features/app-own-domain.md): cut-over, certificate, and the way back.
docs/designs/compute-leaves-the-forge.md added +145
@@ -0,0 +1,145 @@
1 <!-- Source: issue #13652, kept here so the design lives in the repository (#14238). -->
2
3 The owner's direction, 2026-11-15, formalized. An epic beside #13639 (the forge as its own customer). Step R0 is the first issue to cut from it: name the seam in the check queue with no behaviour change.
4
5 # Part two: compute leaves the forge
6
7 2026-11-15, from the owner: "most forges separate the compute. github actions runners, forgejo
8 runners … in the platform we used KEDA to scale. what if my mac could join and help get through
9 checks when on internet and power above 80%? … i want users to be able to scale up and down as
10 they need. part of that will be SLAs … what about those inevitable users who have thousands of
11 agents going?"
12
13 ## The answer in five lines
14
15 1. Yes: the forge becomes a **control plane** (repositories, issues, the queue, the record of what
16 ran and what it proved) and work runs on **runners** that join it. Today both are one process on
17 one machine, which is why checks hold 50 of 72 cores beside the page thread.
18 2. A runner **pulls** work: it asks the forge for a job, gets a commit and a recipe, runs it in the
19 sandbox the forge already has, and hands back a signed result. Nothing dials into the runner, so
20 a laptop behind any network can be one.
21 3. Checks and crew runs are the same kind of job. One seam serves both, so "thousands of agents"
22 is more runners, not a bigger forge.
23 4. The forge never needs a particular runner. Its own machine is always a runner; everything else
24 is extra capacity that may come and go. The Mac helps when it is plugged in and online and is
25 never missed when it is not.
26 5. An SLA is then a statement about two separate things, which is what makes it sellable: the
27 control plane's availability (where your data lives) and the capacity you have reserved (how
28 fast work clears).
29
30 ## Why pull, and why now
31
32 - **It is the proven shape.** Actions runners, Forgejo runners and Buildkite agents all long-poll a
33 queue with a registration token. It needs no inbound port, survives a sleeping laptop, and
34 scales by counting queue depth (what KEDA did on the platform: scale on the queue, not on CPU).
35 - **The pieces exist.** The forge has a check queue with ranks, a cell that sandboxes a run, an
36 impact map and kept results that are content-addressed, a model-runner seam (#12068), tokens,
37 and hosted forges that are already separate processes. What is missing is the wire between a
38 queue here and a sandbox there.
39 - **It is the same move as part one.** The workshop takes development off production; runners take
40 compute off whichever forge is asking.
41
42 ## The runner, formally
43
44 **Joining.** A person runs one command on a machine with a join token from the forge (scoped to a
45 forge, an owner or one repository; revocable; shown once). The runner registers what it is: CPUs,
46 memory, OS and architecture, what it may run (checks, crew runs, looks), and its **conditions**.
47
48 **Conditions** are the runner's own rules for when it offers itself, evaluated on the runner:
49 on mains power or battery above a threshold, on an unmetered network, idle for some minutes,
50 inside set hours, at most N cores. The Mac's line is "on power or battery above 80%, online, at
51 most 6 cores". A runner that stops meeting its conditions finishes or hands back its job and
52 says so.
53
54 **A job** is: a commit (fetched from the forge's git door with a short-lived token), a recipe
55 (the repository's own `forge.json` check steps), the cache keys it may read, a time budget, and
56 the secrets the repository was granted for that kind of run and nothing else.
57
58 **Running.** The same cell the forge uses today (bubblewrap on Linux; a container or the system
59 sandbox on macOS, with the honest note that a Mac result is for macOS-compatible steps: a check
60 that needs Linux goes to a Linux runner). The runner streams the log back and returns the result
61 with the test list it ran and kept.
62
63 **Trust.** A result is signed by the runner's key and recorded with who ran it. Three classes:
64 the forge's own machine (trusted); a runner the owner enrolled (trusted for that owner's
65 repositories); anyone else's (its green is evidence, and the landing gate re-runs on a trusted
66 runner). A pull from a stranger never runs on a runner that holds anyone's secrets: the outside
67 lane the forge already has.
68
69 **Losing a runner.** A job has a lease. A runner that goes silent loses the lease and the job is
70 queued again without counting as a try (the rule that now holds for restarts).
71
72 **Caches.** Installs, the impact map and kept results are content-addressed, so a runner fetches
73 what it lacks from the forge by hash and can keep a local store. A cold runner is slower, never
74 wrong.
75
76 ## Scaling up and down
77
78 - The queue's depth and age, by class of job, is the one signal. A hosted customer sets "at most N
79 runners, at least M" and the host adds or removes cells or machines to hold the queue's age under
80 a target. That is the KEDA idea with the forge's own queue as the metric.
81 - **Thousands of agents.** A crew run is a job: the model call is the provider's, the tool calls
82 and the workspace are the runner's. A customer with a thousand agents has a control plane sized
83 for writes (issues, pulls, events: this is where the database work matters) and a runner pool
84 sized for the agents. The forge's per-forge limits become per-pool limits.
85 - **Many forges, one pool.** An organisation buys one pool that serves all its people's forges.
86 That is the enterprise shape: a forge per person or team, a shared and metered runner pool, one
87 bill.
88
89 ## SLAs, as things that can be measured and sold
90
91 | What is promised | What it rests on | Who can promise it |
92 |---|---|---|
93 | The forge answers (control plane) | where the forge process and its data run; backups and a proven restore | the host of that machine |
94 | Work starts within N minutes | reserved runners in the pool | whoever provides the runners |
95 | Work finishes at a given rate | runner count and the size of the repository's checks | the customer's choice of pool |
96
97 Your vxrail gives you the first row for your own forge and for customers you host: you control the
98 machine. The honest ceiling today is one machine on one power feed and one uplink; a battery and a
99 second uplink are what turn "I sleep next to it" into a number. A customer who wants more nines
100 than one site can give takes a second control-plane site (the restore you already prove nightly is
101 the seed of that), and that is a tier, not a redesign.
102
103 ## What this does to the forge's architecture, in one picture
104
105 ```
106 people and agents
107 |
108 +----------v-----------+
109 | control plane | repositories, issues, pulls, queue, records,
110 | (a forge) | pages, MCP, git door; small CPU, careful writes
111 +----+------------+----+
112 | jobs (pull)| results (signed)
113 +-------v--+ +-----v-----+ +-----------+
114 | runner: | | runner: | | runner: | checks, crew runs, looks;
115 | its own | | a pool | | the Mac, | join and leave freely;
116 | machine | | (hosted) | | when able | sandboxed; no secrets it was not given
117 +----------+ +-----------+ +-----------+
118 ```
119
120 Part one says who builds the forge and where. Part two says where anything runs. Together: a
121 forge is small and always there; the work it asks for happens wherever there is room.
122
123 ## Steps, each reversible
124
125 | Step | What | Proves |
126 |---|---|---|
127 | R0 | Name the seam in code: the check queue hands a job to "a runner" that today is the same process. No behaviour change. | the job is fully described by data |
128 | R1 | A runner as a separate process on the same machine, pulling over HTTP with a token. Checks only. | the wire; P's page thread without checks beside it |
129 | R2 | The Mac as a second runner with conditions (power, network, cores). Mac-compatible steps only at first. | join, leave, lease, cache by hash |
130 | R3 | Crew runs through the same seam (#12068). | "many agents" is a pool size |
131 | R4 | Pools for hosted forges: min, max, scale on queue age; metered. | scale up and down as a product |
132 | R5 | SLA tiers written against measured numbers; an outside probe for the first row. | something to sell |
133
134 ## Risks
135
136 - **Security is the whole game.** A runner executes untrusted code and holds a token. Scope every
137 token to one job; give secrets only to trusted runners; treat a volunteer runner's green as
138 evidence, never as proof.
139 - **macOS is not Linux.** The forge's checks assume Linux cells. The Mac can run the large share
140 that is plain `bun test`; the rest stays on Linux. Do not chase parity.
141 - **A distributed queue has failure modes one process does not**: double runs, lost results,
142 clock skew. Leases, idempotency keys (already used for schedules) and "a rerun is always safe"
143 cover them; they need tests of their own.
144 - **Do not build a second scheduler.** The check queue with ranks is the scheduler. Runners are
145 its hands.
docs/designs/forge-as-its-own-customer.md added +188
@@ -0,0 +1,188 @@
1 <!-- Source: issue #13639, kept here so the design lives in the repository (#14238). -->
2
3 The owner's direction, 2026-11-15, formalized. This is an epic: each step in the plan's table becomes its own issue as it is reached; steps 0 and 1 are being piloted now.
4
5 # The forge as its own customer
6
7 2026-11-15. Written from the owner's direction: "why don't we dogfood our own forge?
8 forge.open-platform.sh could be just the hosted forge. production. only certain users can
9 propose changes upstream and it will take a ton of work off of the main forge … I think this
10 also gets to your point of the packages and modularization."
11
12 ## The answer in five lines
13
14 1. Yes. Today one process is three things at once: the product's production site, the workshop
15 where the forge is built, and the owner's personal forge. Every problem of the last three days
16 came from that overlap.
17 2. Split by role, not by code: **Production** hosts and sells and holds the canonical repository.
18 **The workshop** is a hosted forge, a customer like any other, where the crew builds the forge.
19 3. The workshop proposes upstream. Only named maintainers may. Production lands what arrives
20 already proven and releases it on a train: the workshop runs each candidate first.
21 4. This makes us the heaviest customer of our own hosting, which is the fastest way to find what
22 hosted forges lack.
23 5. Modularization follows the same line: a small kernel, and features that live in their own
24 repositories and are absorbed, each with its own tests. That is what makes checks cheap.
25
26 ## What is true today (measured, 2026-11-13 to 11-15)
27
28 `forge.open-platform.sh` is one container on one machine (72 of 80 CPUs, 251 GB). In it:
29
30 | Role | What it does there | What it costs the other roles |
31 | -------------- | -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
32 | Production | front door, sign-up, pricing, 16 hosted forges in cells, `gretchy.store`, `open-platform.sh` | must never be down |
33 | Workshop | 8 builders, a planner, ~3,150 test files per check, 30 to 60 pulls opened an hour | checks hold 50 to 62 cores; the serving thread stalls 5 to 20 times a minute; the database took 2.4M events |
34 | Personal forge | the owner's apps and their crews | its home page fills with the workshop's noise |
35
36 Incidents that were this overlap and nothing else:
37
38 - **Five outages in three days** (2 min, 2.5 min, three Bun crashes, a 4.5 min crash loop). Each was
39 development reaching production: a helper opening the live database, a profiler, a worker the
40 crew added, a default the crew changed. Customers' forges ride the same restarts.
41 - **About ten releases a day**, each a restart of the host of every customer, because the workshop
42 wants its own fixes live.
43 - **Restarts cut a quarter of checks** one night. The workshop's releases interrupted the
44 workshop's own work.
45 - **The staging rig cannot see production's shape** (no crew, no apps, no checks), so two bad
46 releases passed it.
47 - **The week's model allowance** is spent by a planner that opens general work on the production
48 forge.
49
50 What already exists and is unused for this: hosted forges in their own cells, each with its own
51 crew, budget and checks; origins and forks between forges; `propose_to_parent_forge` (a child
52 offers a landed change to its parent as a pull); following an upstream and taking updates; a
53 release keeper per forge; absorbed features (`feature` in `forge.json`, as the morning paper is).
54 The contribution loop was proven end to end between forges on 2026-11-13.
55
56 ## The model
57
58 ### Three roles, three places
59
60 **Production (P): `forge.open-platform.sh`.** Hosts forges, sells them, serves the sites, holds
61 the canonical `forge` repository and its releases. It runs no builders and no planner for the
62 forge's own code. Its engine's only work on that repository is admitting proposals and cutting
63 releases. It restarts when a release is promoted: once or twice a day, at a chosen hour.
64
65 **The workshop (W): a hosted forge on P, e.g. `workshop--forge.open-platform.sh`.** A customer.
66 It holds a fork of `forge`, the backlog, the planner, the builders, the checks. It runs the newest
67 candidate build at all times (its release keeper is on), so it is also the canary. It can crash,
68 restart ten times an hour and fill its own database, and P does not feel it.
69
70 **Personal forges.** The owner's apps move to a forge of his own (`trevato--forge…`), also a
71 customer. P's home page is then the product's, and his home page is his kanban.
72
73 ### The flow of a change
74
75 1. An issue is opened on W (or arrives from P: a customer's feedback, a stranger's report, routed
76 to W as an issue with its origin kept).
77 2. W's crew builds it and W's check passes on the merge with W's main. W lands it.
78 3. W proposes upstream: one proposal per landed change, or one per train (see open questions).
79 The proposal carries W's attestation: the commit, the base, the check run, its result, signed
80 by W's key that P holds because P hosts W.
81 4. P admits it: the proposer is on the maintainer list; the attestation verifies; it merges
82 cleanly; P's own small gate passes on the exact tree that becomes main (rule scans and a boot,
83 minutes not an hour). P does not re-run 3,150 tests.
84 5. A release candidate is cut from P's main. W takes it first, automatically. After a soak on W
85 under real crew load (a number of hours with no restart, no fatal line, stalls and memory
86 inside their first-hour figures), P promotes it at its release hour. A failed soak never
87 reaches P.
88
89 ### Who may propose upstream
90
91 A short list on P's `forge` repository: the owner and W's landing identity. Everyone else (Ben,
92 grokbot, a stranger) forks from P to a forge of their own and proposes to **W**, where a crew and
93 a full check can receive them. P stays the root of the fork tree, so "contribute anywhere" keeps
94 its shape: you fork the canonical repository, and your proposal is routed to the workshop.
95
96 ### What P keeps
97
98 Accounts and billing; the hosted forges and their cells; the front door, pricing, docs, releases
99 and changelog; the canonical repository (read by everyone, written by admitted proposals only);
100 intake of feedback; the edge. Its database is the customers', not the workshop's: it stops
101 growing by a million rows a week.
102
103 ## Why this is right
104
105 - **Blast radius.** A bad commit can only hurt W until a human-free soak has passed. Every outage
106 above would have been W's.
107 - **The rig becomes real.** W is production-shaped by construction: crew, checks, apps, a full
108 database. The 0.7.599 crash loop needs W's config to appear; it would have appeared on W.
109 - **Production gets calm.** No checks on its CPUs, a release a day, a small database. The page
110 thread's stalls were the workshop's work.
111 - **We find our own product's holes first.** A workshop needs a size above "large" (4 CPUs, 8 GB),
112 checks that can run a forge's own end-to-end tests inside a cell, egress for installs, backups
113 of a hosted forge, a way to carry a repository with its history to another forge, a model
114 account on a hosted forge. Customers need every one of those. Today nobody at home feels them.
115 - **It is the vision already written down.** More forges is how this scales; a forge divides when
116 heavy; what is not carried is archived; the kernel is small and what it generates is large.
117
118 ## What it costs, honestly
119
120 1. **Compute does not shrink by moving.** W's checks still want 50 cores, on the same machine.
121 What changes is that they are in a cell with a CPU set of its own, apart from P's serving cores.
122 Real relief on check time is the modularization below, not the move.
123 2. **A hosted forge cannot yet run the forge's own suite.** Tests that spawn cells, take looks
124 with Chromium or install packages fail or are skipped when nested in a cell (seen on 11-14 and
125 11-15). Either cells learn to nest, or those tests run in a lane P provides to W. This is the
126 first thing the pilot must measure.
127 3. **The backlog lives on P.** 17,700 issues and 9,000 pulls. Moving them needs "carry a
128 repository with its history to another forge": one file per repository as an export format
129 (measured as the right use of the per-repository database idea on 11-15). Until it exists, W
130 starts with a fork and new work, and the history stays readable on P.
131 4. **Two forges to keep well.** W must take updates by itself and be backed up like a customer.
132 That is the product working, or a list of what to fix.
133 5. **Admission by attestation is new trust code.** If it is wrong, an unproven change reaches
134 production's main. Mitigation: P's own gate on the exact tree, the soak on W, and the 75-second
135 watch with rollback that already exists.
136 6. **Latency of a fix.** A change reaches customers in hours, not minutes. An emergency lane (the
137 owner promotes now) keeps the old speed for the rare case.
138 7. **One machine.** W, P and the customers share a host. A second machine for W is the real
139 isolation, and the design does not depend on it.
140
141 ## Modularization: what "packages" should mean here
142
143 The measured ceiling on checks: a pull's fast tier still selects about 2,670 of 3,150 test files,
144 because 567 source files are each executed by more than 1,000 tests. Most tests boot a whole
145 forge. Selection and caching cannot fix that; boundaries can.
146
147 1. **Kernel**: database, migrations, log, ids, events, config, the git door, auth. Small, slow to
148 change, the only thing every test may boot.
149 2. **Engines**, each a package with its own tests against the kernel only: checks, crew and
150 autonomy, apps runtime and cells, spawn and hosting, snapshot and restore, flow and pictures.
151 3. **Surfaces**: web pages, MCP, terminal. Tested against engines' interfaces with fixtures, not
152 a booted forge.
153 4. **Features** in their own repositories, absorbed through the `feature` block that already
154 exists: the morning paper today; billing, the changelog, boards, the map and talk are
155 candidates. A feature's check is its own repository's check: minutes.
156
157 Order: draw the boundaries inside the one repository first (workspace packages, an import rule
158 per package, a test tier per package), measure the fast tier per package, and only then move a
159 feature out. A package boundary is a rule test, like the ones the gate already has.
160
161 The workshop and the packages are the same idea at two sizes: work lives in a unit small enough
162 to check quickly, and proposes to the unit above it.
163
164 ## The plan, in reversible steps
165
166 | Step | What | Proves | Undo |
167 | ---- | ------------------------------------------------------------------------------------------------------------------- | --------------------------------- | ----------------------------- |
168 | 0 | Stand up W as a hosted forge with a fork of `forge` and a crew on lent credit. Run the forge's check inside W once. | what a hosted forge cannot do yet | remove W |
169 | 1 | W builds one real issue and proposes it upstream with `propose_to_parent_forge`; P lands it by today's path. | the loop, with numbers | none needed |
170 | 2 | A workshop size (CPUs and memory a crew's forge needs) and a CPU set for cells apart from P's serving cores. | stalls on P fall | size and pin are settings |
171 | 3 | W's release keeper on: W runs every candidate. P promotes after a soak. | the canary | turn the keeper off |
172 | 4 | Attested admission on P; the maintainer list. P's builders for `forge` off. | P does no development | list and switch |
173 | 5 | Carry `forge`'s backlog to W in the per-repository export format. | history moves | P keeps its copy until proven |
174 | 6 | The owner's apps to his own forge. | P's home is the product's | same carry, back |
175 | 7 | Package boundaries, then the first feature moved out. | checks in minutes | boundaries are rules |
176
177 Steps 0 to 3 change nothing a customer can see and can each be undone the same day.
178
179 ## Open questions for the owner
180
181 1. **One proposal per change, or one per train?** Per change keeps history and blame simple and
182 is what the tools do today. Per train is fewer admissions and matches releases. Recommended:
183 per change, admitted in batches.
184 2. **Does W live on this machine?** Recommended: yes to start, pinned apart; a second machine when
185 one is at hand.
186 3. **Where do customers' feedback and strangers' reports land?** Recommended: P takes them and
187 routes them to W with the origin kept, so nobody has to know W exists.
188 4. **Who is on the maintainer list besides you and W?**
docs/features/bottleneck.md +18 −1
@@ -17,4 +17,21 @@ it. Two doors, one reading (`readBottleneck`):
1717 page through the public address against loopback, bytes split by sender, and the host's traffic
1818 shaping.
1919
20 Code: `src/system/bottleneck.ts`, `src/web/pages/bottleneck.ts`.
20 - **What the host is tuned to** is declared, not measured, and not yet confirmed: `src/system/host_tuning.ts`
21 holds the traffic shaper (cake at 9 Mbit, four lanes, LAN unshaped) and the edge proxy's
22 large-or-slow-response log and crawler lane. The figures were reconstructed from issue #14238; the page
23 says "Not yet" under *Checked against the host* until the owner has compared them with `tc qdisc show`
24 and the edge's nginx.conf and set `confirmed` in that file. The marks that would put pages ahead of
25 backups are not set by anything here, and the page does not claim a priority order.
26 `bun scripts/host-tune.ts --apply --dev <uplink>` applies the shaper (run it again after a reboot);
27 `--edge` prints the proxy snippet to include.
28 - **Changes to how the forge runs** are `docs/ops/operations.log`, one line each
29 (`day | setting|host|incident | who | what | why`; append one in the pull that makes the change), with
30 the settings changed on /settings/capacity read from their events. The newest twelve show as dated rows
31 on the page, in the terminal and in the MCP `bottleneck` reading (`host`: `{label, value}` rows, `ops`:
32 `{day, kind, who, what, why}`); the morning paper says how many came since its last line.
33 - **In the terminal** (admins, as the page): the palette's *Where the bottleneck is* shows the sentence,
34 a line a row, what the host is tuned to and the changes; `j`/`k` scroll, `g` reads again, `esc` goes back.
35
36 Code: `src/system/bottleneck.ts`, `src/system/ops_log.ts`, `src/system/host_tuning.ts`, `src/web/pages/bottleneck.ts`,
37 `src/tui/screens/bottleneck.ts`.
docs/ops/operations.log added +10
@@ -0,0 +1,10 @@
1 # One line per change to how the forge runs, newest last:
2 # <YYYY-MM-DD> | <setting|host|incident> | <who> | <what> | <why>
3 # Setting changes made on /settings/capacity are kept by the forge itself as
4 # `capacity.changed` events and read from there; this file holds the rest.
5 # Append a line in the same pull that makes the change. Pipes are not allowed inside a field.
6 # The first lines were written from issue #14238, not from the host: only what it states is here.
7 2026-11-15 | host | owner | Traffic shaper on the uplink: cake at 9 Mbit, diffserv4, LAN not shaped (from #14238, not yet read from the host) | Set by hand on the server and lost on reboot; now declared in src/system/host_tuning.ts
8 2026-11-15 | host | owner | Edge proxy logs large or slow responses to a separate access log (from #14238, not yet read from nginx.conf) | Declared in src/system/host_tuning.ts
9 2026-11-15 | host | owner | Edge proxy paces crawlers in their own lane (from #14238, not yet read from nginx.conf) | Declared in src/system/host_tuning.ts
10 2026-11-15 | incident | owner | Restarts cut a quarter of a night's checks | See docs/designs/forge-as-its-own-customer.md
scripts/host-tune.ts added +29
@@ -0,0 +1,29 @@
1 #!/usr/bin/env bun
2 /**
3 * Applies or shows what the host is tuned to (#14238; src/system/host_tuning.ts is the
4 * declaration).
5 * bun scripts/host-tune.ts --dev eth0 print the tc commands
6 * bun scripts/host-tune.ts --apply --dev eth0 run them (root, on the host)
7 * bun scripts/host-tune.ts --edge print the edge proxy's snippet
8 * Each command is an argument list built from the declaration and a validated device name; no
9 * text is ever handed to a shell. Run it again after a reboot: `replace` makes it safe to repeat.
10 */
11 import { edgeSnippet, shaperCommands, validDevice } from "../src/system/host_tuning.ts";
12
13 const args = process.argv.slice(2);
14 if (args.includes("--edge")) {
15 process.stdout.write(edgeSnippet());
16 process.exit(0);
17 }
18 const at = args.indexOf("--dev");
19 const dev = at >= 0 ? (args[at + 1] ?? "") : "";
20 if (!validDevice(dev)) {
21 console.error("usage: bun scripts/host-tune.ts [--apply] --dev <uplink interface> | --edge");
22 process.exit(2);
23 }
24 for (const argv of shaperCommands(dev)) {
25 console.log(argv.join(" "));
26 if (!args.includes("--apply")) continue;
27 const done = Bun.spawnSync(argv, { stdout: "inherit", stderr: "inherit" });
28 if (done.exitCode !== 0) process.exit(done.exitCode ?? 1);
29 }
src/crew/routine_facts_story.ts +9 −2
@@ -9,12 +9,18 @@ import { publicForgeRepo } from "../core/forge_repo.ts";
99 import { mergedByPillar, pillarsOf, servedBy } from "../core/pillars.ts";
1010 import { readChangelog } from "../release/changelog.ts";
1111 import { repoDir } from "../git/repos.ts";
12 import { opLine, opsSince, readOps } from "../system/ops_log.ts";
1213
1314 export interface Story {
1415 /** Versions cut since the last check-in, oldest first. */
1516 readonly releases: readonly string[];
1617 /** Pillars that merged work since then, with how many pulls. */
1718 readonly pillarsMoved: readonly { readonly pillar: string; readonly merged: number }[];
19 /**
20 * Changes to how the forge runs since then (docs/ops/operations.log, kept settings), newest
21 * first.
22 */
23 readonly ops?: readonly string[];
1824 }
1925
2026 export const NO_STORY: Story = { releases: [], pillarsMoved: [] };
@@ -29,8 +35,9 @@ export function releasesSince(entries: readonly { version: string; date: string
2935
3036 /** The story since `sinceMs`; empty when the forge has no public source repository. */
3137 export async function storyFacts(app: App, sinceMs: number): Promise<Story> {
38 const ops = opsSince(readOps(app.db), sinceMs).map(opLine);
3239 const repo = publicForgeRepo(app);
33 if (repo === undefined) return NO_STORY;
40 if (repo === undefined) return ops.length === 0 ? NO_STORY : { ...NO_STORY, ops };
3441 const entries = await readChangelog(repoDir(app.reposRoot, repo.ownerName, repo.name), repo.defaultBranch);
3542 const set = pillarsOf(app.db, repo);
3643 const moved: { pillar: string; merged: number }[] = [];
@@ -40,5 +47,5 @@ export async function storyFacts(app: App, sinceMs: number): Promise<Story> {
4047 }
4148 moved.sort((x, y) => y.merged - x.merged);
4249 }
43 return { releases: releasesSince(entries, sinceMs), pillarsMoved: moved };
50 return { releases: releasesSince(entries, sinceMs), pillarsMoved: moved, ops };
4451 }
src/crew/routine_report.ts +2
@@ -50,6 +50,8 @@ export function storyText(f: Story): string {
5050 parts.push(`released ${f.releases.length > 3 ? `${f.releases.length} (to ${f.releases.at(-1)})` : shown}`);
5151 }
5252 if (f.pillarsMoved.length > 0) parts.push(`moved: ${f.pillarsMoved.slice(0, 3).map((p) => `${p.pillar} ${p.merged}`).join(", ")}`);
53 const ops = f.ops ?? [];
54 if (ops.length > 0) parts.push(`${plural(ops.length, "change")} to how it runs (${ops[0]!.replace(/^\d{4}-\d{2}-\d{2} /, "").slice(0, 80)})`);
5355 return parts.join("; ");
5456 }
5557
src/system/bottleneck.ts +15 −2
@@ -10,6 +10,8 @@
1010 import type { Database } from "bun:sqlite";
1111 import { capacityValue } from "../core/capacity.ts";
1212 import { runningAgentRuns } from "../crew/running.ts";
13 import { hostFacts, type HostFact } from "./host_tuning.ts";
14 import { readOps, type Op } from "./ops_log.ts";
1315 import type { SystemView } from "./view.ts";
1416
1517 export type RowId = "line" | "cpu" | "checks" | "crew" | "memory" | "disk";
@@ -29,7 +31,15 @@ export type Row = {
2931 lever: { label: string; href: string } | null;
3032 };
3133
32 export type Bottleneck = { at: string; says: string; limited: RowId | null; rows: Row[]; unseen: string[] };
34 export type Bottleneck = { at: string; says: string; limited: RowId | null; rows: Row[]; unseen: string[];
35 /** What the host is tuned to, declared (src/system/host_tuning.ts). */
36 host: HostFact[];
37 /** The newest changes to how the forge runs (docs/ops/operations.log and the kept settings). */
38 ops: Op[];
39 };
40
41 /** How many changes to how the forge runs a reading carries. */
42 export const OPS_SHOWN = 12;
3343
3444 export type Input = {
3545 view: SystemView;
@@ -37,6 +47,8 @@ export type Input = {
3747 lineMbit: number | null;
3848 checks: { running: number; slots: number; queued: number };
3949 crew: { running: number; slots: number; queued: number };
50 /** The changes to how the forge runs to carry along; none when left out. */
51 ops?: readonly Op[];
4052 };
4153
4254 /** A row at or past this share of its ceiling is what limits the forge. */
@@ -136,7 +148,7 @@ export function judge(input: Input, at = new Date()): Bottleneck {
136148
137149 let worst: Row | null = null;
138150 for (const r of rows) if (r.fraction !== null && r.fraction >= LIMIT && (worst === null || r.fraction > (worst.fraction ?? 0))) worst = r;
139 return { at: at.toISOString(), says: worst === null ? "Nothing is at its limit." : sentence(worst, input, tx), limited: worst?.id ?? null, rows, unseen: [...UNSEEN] };
151 return { at: at.toISOString(), says: worst === null ? "Nothing is at its limit." : sentence(worst, input, tx), limited: worst?.id ?? null, rows, unseen: [...UNSEEN], host: hostFacts(), ops: [...(input.ops ?? [])] };
140152 }
141153
142154 function sentence(r: Row, input: Input, tx: number): string {
@@ -160,5 +172,6 @@ export function readBottleneck(db: Database, view: SystemView, env: string | und
160172 lineMbit: lineCeiling(env),
161173 checks: { running: count(db, "check", "running"), slots: Number(capacityValue(db, "checks.concurrency")), queued: count(db, "check", "queued") },
162174 crew: { running: runningAgentRuns(), slots: Number(capacityValue(db, "crew.concurrency")), queued: count(db, "agent", "queued") },
175 ops: readOps(db).slice(0, OPS_SHOWN),
163176 });
164177 }
src/system/host_tuning.ts added +79
@@ -0,0 +1,79 @@
1 /**
2 * What the host is tuned to, declared (#14238): the traffic shaper and the edge proxy's lanes were
3 * set by hand on the server and lost on a reboot. This is the one description;
4 * `scripts/host-tune.ts` prints or applies it and the bottleneck page shows it. Plain data in,
5 * plain data out: nothing here touches the machine. The figures were reconstructed from the issue,
6 * not read from the host: `confirmed` stays false until the owner has checked them against
7 * `tc qdisc show` and the edge's nginx.conf, and the page says so while it is.
8 */
9
10 export const HOST_TUNING = {
11 /** Set to true only once the owner has read these figures off the host itself. */
12 confirmed: false as boolean,
13 shaper: {
14 /** Megabits a second the shaper lets out; just under the 16.3 Mbit/s the line carries. */
15 mbit: 9,
16 lanUnshaped: true,
17 },
18 edge: {
19 /** The edge proxy's configuration on the host. */
20 conf: "/srv/forge/edge/nginx.conf",
21 /** A response this large, in bytes, or this slow, in seconds, goes to the separate log. */
22 logBytes: 1_000_000,
23 logSeconds: 1,
24 /** Crawlers are paced to this many requests a second, in their own lane. */
25 crawlerPerSecond: 2,
26 },
27 } as const;
28
29 /** An interface name the kernel accepts and no shell would read as anything else. */
30 export const validDevice = (s: string): boolean => /^[a-z][a-z0-9._-]{0,14}$/i.test(s);
31
32 /** The `tc` argument lists that apply the shaper to `dev`: argv, never a shell line. */
33 export function shaperCommands(dev: string, t = HOST_TUNING.shaper): string[][] {
34 if (!validDevice(dev)) return [];
35 // cake's diffserv4 sorts packets into four tins by their marks; nothing in this repository sets
36 // the marks, so which traffic rides which tin is not declared here. The LAN is left unshaped by
37 // shaping only the uplink.
38 return [["tc", "qdisc", "replace", "dev", dev, "root", "cake", "bandwidth", `${t.mbit}mbit`, "diffserv4", "nat", "wash"]];
39 }
40
41 /** The edge proxy's two additions, as text to include from nginx.conf (`http` context). */
42 export function edgeSnippet(t = HOST_TUNING.edge): string {
43 return [
44 `# Declared by the forge (src/system/host_tuning.ts); include it from ${t.conf}.`,
45 `map "$body_bytes_sent:$request_time" $forge_loud {`,
46 ` default 0;`,
47 ` "~^\\d{${String(t.logBytes).length},}:" 1;`,
48 ` "~^\\d+:(?:[${t.logSeconds}-9]|\\d{2,})\\." 1;`,
49 `}`,
50 `access_log /var/log/nginx/forge-loud.log combined if=$forge_loud;`,
51 `map $http_user_agent $forge_crawler {`,
52 ` default "";`,
53 ` "~*(bot|crawl|spider|slurp)" $binary_remote_addr;`,
54 `}`,
55 `limit_req_zone $forge_crawler zone=forge_crawlers:10m rate=${t.crawlerPerSecond}r/s;`,
56 `# and, in the server block: limit_req zone=forge_crawlers burst=10;`,
57 ``,
58 ].join("\n");
59 }
60
61 export type HostFact = { label: string; value: string };
62
63 /** The declaration as label and value rows, in plain words, for the page, the terminal and MCP. */
64 export function hostFacts(t = HOST_TUNING): HostFact[] {
65 return [
66 {
67 label: "Checked against the host",
68 value: t.confirmed ? "Yes" : "Not yet: these figures come from the issue, not from the host's own settings",
69 },
70 { label: "Uploads", value: `Held to ${t.shaper.mbit} Mbit/s, just under the 16.3 Mbit/s the line carries` },
71 { label: "Sorting of traffic", value: "Four lanes by the mark on each packet; nothing in this repository sets the marks yet" },
72 { label: "Local network", value: t.shaper.lanUnshaped ? "Not slowed" : "Slowed like the rest" },
73 {
74 label: "Large or slow responses",
75 value: `Written to a separate log when over ${t.edge.logBytes.toLocaleString("en-US")} bytes or ${t.edge.logSeconds} s`,
76 },
77 { label: "Crawlers", value: `Share one lane of ${t.edge.crawlerPerSecond} requests a second` },
78 ];
79 }
src/system/ops_log.ts added +101
@@ -0,0 +1,101 @@
1 /**
2 * The operations log (#14238): one line per change to how the forge runs, with who and why.
3 * `docs/ops/operations.log` holds host and incident lines (a pull appends them);
4 * settings changed on /settings/capacity are the forge's own `capacity.changed` events and are
5 * merged in, so nothing is written twice. Parsing is pure; the bottleneck page, the MCP tool and
6 * the morning paper read `readOps`.
7 */
8 import type { Database } from "bun:sqlite";
9 import { existsSync, readFileSync, statSync } from "node:fs";
10 import { join } from "node:path";
11
12 export const OP_KINDS = ["setting", "host", "incident"] as const;
13 export type OpKind = (typeof OP_KINDS)[number];
14 export type Op = { day: string; kind: OpKind; who: string; what: string; why: string };
15
16 const FILE = join(import.meta.dir, "..", "..", "docs", "ops", "operations.log");
17 const MAX_BYTES = 256 * 1024;
18
19 /** Lines that do not parse are left out, never guessed at. */
20 export function parseOps(text: string): Op[] {
21 const out: Op[] = [];
22 for (const raw of text.split("\n")) {
23 const line = raw.trim();
24 if (line === "" || line.startsWith("#")) continue;
25 const [day, kind, who, what, ...rest] = line.split("|").map((s) => s.trim());
26 const why = rest.join(" | ");
27 if (day === undefined || !/^\d{4}-\d{2}-\d{2}$/.test(day) || Number.isNaN(Date.parse(day))) continue;
28 if (!(OP_KINDS as readonly string[]).includes(kind ?? "")) continue;
29 if (!who || !what) continue;
30 out.push({ day, kind: kind as OpKind, who, what, why });
31 }
32 return out;
33 }
34
35 /** The last parsed file, kept until its path, mtime or size changes. */
36 let kept: { file: string; stamp: string; ops: Op[] } | null = null;
37
38 export function readOpsFile(file = FILE): Op[] {
39 try {
40 if (!existsSync(file)) return [];
41 const st = statSync(file);
42 if (st.size > MAX_BYTES) return [];
43 const stamp = `${st.mtimeMs}:${st.size}`;
44 if (kept && kept.file === file && kept.stamp === stamp) return kept.ops;
45 const ops = parseOps(readFileSync(file, "utf8"));
46 kept = { file, stamp, ops };
47 return ops;
48 } catch {
49 return [];
50 }
51 }
52
53 /** Setting changes the forge kept itself, as ops. */
54 export function settingOps(db: Database, limit = 50): Op[] {
55 const rows = db
56 .query<{ at: string; payload: string; name: string | null }, [number]>(
57 "SELECT e.created_at AS at, e.payload AS payload, u.username AS name FROM event e LEFT JOIN user u ON u.id = e.actor_id WHERE e.kind = 'capacity.changed' ORDER BY e.id DESC LIMIT ?1",
58 )
59 .all(limit);
60 const out: Op[] = [];
61 for (const r of rows) {
62 try {
63 const p = JSON.parse(r.payload) as { key?: unknown; from?: unknown; to?: unknown; reason?: unknown };
64 if (typeof p.key !== "string") continue;
65 out.push({
66 day: r.at.slice(0, 10),
67 kind: "setting",
68 who: r.name ?? "unknown",
69 what: `${p.key}: ${String(p.from ?? "default")} to ${String(p.to ?? "default")}`,
70 why: typeof p.reason === "string" ? p.reason : "",
71 });
72 } catch {
73 /* a payload that is not ours is not a change */
74 }
75 }
76 return out;
77 }
78
79 /**
80 * Newest first (on one day, the later entry first), the file's lines and the kept settings
81 * together.
82 */
83 export function readOps(db: Database, file = FILE): Op[] {
84 let settings: Op[] = [];
85 try {
86 settings = settingOps(db);
87 } catch {
88 /* an old database without the event table's shape reads as no settings */
89 }
90 // Both are oldest first, so a later index is a later entry.
91 const all = [...readOpsFile(file), ...settings.reverse()];
92 return all.map((o, i) => ({ o, i })).sort((a, b) => (a.o.day < b.o.day ? 1 : a.o.day > b.o.day ? -1 : b.i - a.i)).map((x) => x.o);
93 }
94
95 export const opLine = (o: Op): string => `${o.day} ${o.kind}: ${o.what} (${o.who}${o.why === "" ? "" : `; ${o.why}`})`;
96
97 /** Ops from `sinceMs` on (by UTC day), newest first. */
98 export function opsSince(ops: readonly Op[], sinceMs: number): Op[] {
99 const day = new Date(sinceMs).toISOString().slice(0, 10);
100 return ops.filter((o) => o.day >= day);
101 }
src/tui/screens/bottleneck.ts added +73
@@ -0,0 +1,73 @@
1 /**
2 * Where the bottleneck is, in the terminal (#14238): the sentence, a line a row, what the host is
3 * tuned to and the changes to how the forge runs, as /system/bottleneck reads them
4 * (`readBottleneck`). Gated as the page is: a forge admin who is a person. Read-only.
5 */
6
7 import { telemetryReader } from "../../actions/telemetry.ts";
8 import { readBottleneck, type Bottleneck } from "../../system/bottleneck.ts";
9 import { live } from "../../system/live.ts";
10 import type { Ctx, Screen } from "../app/app.ts";
11 import { binding } from "../app/keys.ts";
12 import type { Session } from "../session.ts";
13 import { style } from "../term/style.ts";
14 import { pad } from "../term/width.ts";
15 import { Scroll, drawLines, hitScroll } from "./common.ts";
16 import type { PaletteEntry } from "./entries.ts";
17
18 const mayRead = (session: Session): boolean => session.user?.kind !== "agent" && telemetryReader(session.app, session.user)?.kind === "owner";
19
20 /** The screen's lines; empty for anyone the gate refuses. */
21 export function bottleneckLines(session: Session, b?: Bottleneck): string[] {
22 if (!mayRead(session)) return [];
23 const r = b ?? readBottleneck(session.app.db, live().read());
24 const lines = [r.says, ""];
25 for (const row of r.rows) {
26 const of = row.ceiling === null ? "" : ` of ${row.ceiling}`;
27 lines.push(`${pad(row.label, 14)}${row.now}${of}${row.id === r.limited ? " (at its limit)" : ""}`);
28 }
29 lines.push("", "What the host is tuned to");
30 for (const f of r.host) lines.push(` ${pad(f.label, 26)}${f.value}`);
31 lines.push("", "Changes to how the forge runs");
32 if (r.ops.length === 0) lines.push(" None kept yet.");
33 for (const o of r.ops) {
34 lines.push(` ${o.day} ${pad(o.kind, 9)}${o.what} (${o.who})`);
35 if (o.why !== "") lines.push(`${pad("", 23)}${o.why}`);
36 }
37 return lines;
38 }
39
40 export function bottleneckScreen(session: Session, ctx: Ctx): Screen {
41 const scroll = new Scroll();
42 let lines = bottleneckLines(session);
43 return {
44 title: "Where the bottleneck is",
45 keys: [binding(["j", "k"], "scroll"), binding(["g"], "read again"), binding(["esc", "q"], "back")],
46 render(area, s) {
47 const c = ctx.theme.c;
48 drawLines(s, area, c, scroll, lines.map((text) => [{ text, style: style(text.startsWith(" ") ? c.mutedFg : c.fg, c.bg) }]));
49 hitScroll(ctx, area, scroll, lines.length);
50 },
51 key(e) {
52 if (scroll.key(e, lines.length)) return true;
53 if (e.key === "g") {
54 lines = bottleneckLines(session);
55 return true;
56 }
57 if (e.key === "esc" || e.key === "q") {
58 ctx.pop();
59 return true;
60 }
61 return false;
62 },
63 };
64 }
65
66 export const entry: PaletteEntry = {
67 id: "bottleneck",
68 title: "Where the bottleneck is",
69 hint: "what limits the forge, what the host is tuned to, what changed",
70 after: ["system"],
71 offered: mayRead,
72 run: (session, ctx) => ctx.push(bottleneckScreen(session, ctx)),
73 };
src/tui/screens/entries.ts +2
@@ -19,6 +19,7 @@ import { entry as audit } from "./audit.ts";
1919 import { entry as appTemplates } from "./app_templates.ts";
2020 import { entry as dotfiles } from "./dotfiles.ts";
2121 import { entry as devices } from "./devices.ts";
22 import { entry as bottleneck } from "./bottleneck.ts";
2223 import { entry as bets } from "./bets.ts";
2324 import { entry as bills } from "./bills.ts";
2425 import { entry as payments } from "./payments.ts";
@@ -142,6 +143,7 @@ export const ENTRIES: readonly PaletteEntry[] = [
142143 bugs,
143144 homePins,
144145 system,
146 bottleneck,
145147 healthVoice,
146148 loopStalls,
147149 footing,
src/web/pages/bottleneck.ts +13 −1
@@ -8,6 +8,7 @@ import { notePhase, takeSessionTime } from "../../core/route_timing.ts";
88 import { telemetryReader } from "../../actions/telemetry.ts";
99 import type { Ctx } from "../../http/context.ts";
1010 import { readBottleneck, type Row } from "../../system/bottleneck.ts";
11 import type { Op } from "../../system/ops_log.ts";
1112 import { live } from "../../system/live.ts";
1213 import { notFoundPage, pageHead, signInFirst } from "../chrome.ts";
1314 import { html, type Html } from "../html.ts";
@@ -21,6 +22,13 @@ function rowHtml(r: Row, limited: boolean): Html {
2122 return statusRow(r.label, state, r.lever === null ? null : html`<a href="${r.lever.href}">${r.lever.label}</a>`, `bn-${r.id}`);
2223 }
2324
25 const KIND_WORDS: Record<Op["kind"], string> = { setting: "Setting", host: "Host", incident: "Incident" };
26
27 function opRow(o: Op): Html {
28 const why = o.why === "" ? html`` : html`<div class="small muted">${o.why}</div>`;
29 return statusRow(`${o.day} · ${KIND_WORDS[o.kind]}`, html`${o.what}${why}`, html`<span class="small muted">${o.who}</span>`);
30 }
31
2432 export async function bottleneckPage(ctx: Ctx): Promise<Response> {
2533 notePhase(ctx.request, "session", takeSessionTime(ctx.request));
2634 if (!ctx.auth.user) return signInFirst(ctx);
@@ -32,7 +40,11 @@ export async function bottleneckPage(ctx: Ctx): Promise<Response> {
3240 <p id="bn-says"><strong>${b.says}</strong></p>
3341 <dl class="status-rows" id="bn-rows">${b.rows.map((r) => rowHtml(r, r.id === b.limited))}</dl>
3442 <section class="settings-block"><h2>Not measured here</h2>
35 <ul>${b.unseen.map((u) => html`<li class="small">${u}</li>`)}</ul></section>`;
43 <ul>${b.unseen.map((u) => html`<li class="small">${u}</li>`)}</ul></section>
44 <section class="settings-block" id="bn-host"><h2>What the host is tuned to</h2>
45 <dl class="status-rows">${b.host.map((f) => statusRow(f.label, html`${f.value}`))}</dl></section>
46 <section class="settings-block" id="bn-ops"><h2>Changes to how the forge runs</h2>
47 ${b.ops.length === 0 ? html`<p class="small muted">None kept yet.</p>` : html`<dl class="status-rows">${b.ops.map(opRow)}</dl>`}</section>`;
3648 const response = page({ ctx, title: "Bottleneck", body });
3749 notePhase(ctx.request, "render", performance.now() - rendering);
3850 return response;
test/bottleneck.test.ts +45 −1
@@ -7,6 +7,10 @@ import { createApp, type App } from "../src/app.ts";
77 import { createToken } from "../src/core/tokens.ts";
88 import { createUser, getUserByName } from "../src/core/users.ts";
99 import { serve, type Servers } from "../src/http/server.ts";
10 import { createForgeApp, forgeCommands, Session } from "../src/tui/forge.ts";
11 import { bottleneckLines } from "../src/tui/screens/bottleneck.ts";
12 import { headlessTerminal } from "../src/tui/term/terminal.ts";
13 import type { User } from "../src/core/users.ts";
1014 import { judge, lineCeiling, type Input } from "../src/system/bottleneck.ts";
1115 import type { SystemView } from "../src/system/view.ts";
1216 import { quietLogs } from "./helpers/globals.ts";
@@ -118,6 +122,13 @@ describe("the page", () => {
118122 expect(res.status).toBe(200);
119123 const text = await res.text();
120124 expect(text).toContain('id="bn-says"');
125 const host = text.slice(text.indexOf('id="bn-host"'), text.indexOf('id="bn-ops"'));
126 expect(host).toContain("<dt>Checked against the host</dt>");
127 expect(host).toContain("Held to 9 Mbit/s");
128 expect(host).not.toMatch(/bun scripts|diffserv4|lanUnshaped|<li/);
129 const ops = text.slice(text.indexOf('id="bn-ops"'));
130 expect(ops).toContain("<dt>2026-11-15 · Host</dt>");
131 expect(ops).not.toContain("<li");
121132 expect(text).toContain('href="/settings/capacity"');
122133 const out = await fetch(`${servers.url}/system/bottleneck`, { redirect: "manual" });
123134 expect(await out.text()).not.toContain("bn-says");
@@ -148,6 +159,39 @@ describe("the page", () => {
148159 expect(refused.content[0].text).toContain("forbidden");
149160 const ok = await call(alice.id);
150161 expect(ok.isError).not.toBe(true);
151 expect(JSON.parse(ok.content[0].text).rows.map((r: any) => r.id)).toEqual(["line", "cpu", "checks", "crew", "memory", "disk"]);
162 const reading = JSON.parse(ok.content[0].text);
163 expect(reading.rows.map((r: any) => r.id)).toEqual(["line", "cpu", "checks", "crew", "memory", "disk"]);
164 expect(reading.host[0]).toEqual({ label: "Checked against the host", value: expect.stringContaining("Not yet") });
165 expect(reading.host.map((f: any) => f.label)).toContain("Uploads");
166 expect(reading.ops.length).toBeGreaterThan(0);
167 expect(Object.keys(reading.ops[0]).sort()).toEqual(["day", "kind", "what", "who", "why"]);
168 });
169 test("a setting changed on /settings/capacity shows as a row, escaped, and in the terminal's lines", () => {
170 app.db
171 .query("INSERT INTO event (id, kind, actor_id, payload, created_at) VALUES ('ev-bn', 'capacity.changed', ?1, ?2, '2026-11-16T09:00:00Z')")
172 .run(getUserByName(app.db, "alice")!.id, JSON.stringify({ key: "checks.concurrency", from: "4", to: "6", reason: "<b>runners</b>" }));
173 const alice = getUserByName(app.db, "alice")!;
174 const lines = bottleneckLines(new Session(app, { sshUsername: "alice", remoteAddress: "127.0.0.1", fingerprint: "SHA256:t", user: alice }));
175 expect(lines).toContain("What the host is tuned to");
176 expect(lines.join("\n")).toContain("2026-11-16 setting checks.concurrency: 4 to 6 (alice)");
177 return fetch(`${servers.url}/system/bottleneck`, { headers: { cookie } }).then(async (res) => {
178 const text = await res.text();
179 expect(text).toContain("&lt;b&gt;runners&lt;/b&gt;");
180 expect(text).not.toContain("<b>runners");
181 });
182 });
183 test("the terminal door is the page's: an admin is offered it, a non-admin and nobody are not and read nothing", () => {
184 const identity = (user?: User) => ({ sshUsername: user?.username ?? "git", remoteAddress: "127.0.0.1", fingerprint: "SHA256:t", user });
185 const doors = (user?: User): string[] => {
186 const { app: tui, session } = createForgeApp(app, identity(user), { terminal: headlessTerminal(80, 24), exitHooks: false });
187 const ids = forgeCommands(session)(tui.ctx).map((c) => c.id);
188 tui.stop?.();
189 return ids;
190 };
191 expect(doors(getUserByName(app.db, "alice")!)).toContain("bottleneck");
192 for (const who of [getUserByName(app.db, "bob") ?? undefined, undefined]) {
193 expect(bottleneckLines(new Session(app, identity(who)))).toEqual([]);
194 expect(doors(who)).not.toContain("bottleneck");
195 }
152196 });
153197 });
test/ops-log.test.ts added +127
@@ -0,0 +1,127 @@
1 /** The operations log and the declared host tuning (#14238). */
2 import { afterEach, beforeEach, describe, expect, test } from "bun:test";
3 import { mkdtempSync, readFileSync, rmSync, utimesSync, writeFileSync } from "node:fs";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { Database } from "bun:sqlite";
7 import { migrationFiles } from "../src/core/schema.ts";
8 import { migrate } from "../src/kernel/migrate.ts";
9 import { storyText } from "../src/crew/routine_report.ts";
10 import { edgeSnippet, hostFacts, HOST_TUNING, shaperCommands, validDevice } from "../src/system/host_tuning.ts";
11 import { opLine, opsSince, parseOps, readOps, readOpsFile, settingOps } from "../src/system/ops_log.ts";
12 import { judge, type Input } from "../src/system/bottleneck.ts";
13
14 describe("parseOps", () => {
15 test("reads the three kinds and skips what does not parse", () => {
16 const ops = parseOps(["# c", "2026-11-15 | host | owner | shaper | why | more", "bad | host | a | b | c", "2026-11-15 | nope | a | b | c", "2026-11-15 | release | a | b | c", "2026-11-14 | incident | a | | c", ""].join("\n"));
17 expect(ops).toEqual([{ day: "2026-11-15", kind: "host", who: "owner", what: "shaper", why: "why | more" }]);
18 });
19 test("the checked-in log parses line for line", () => {
20 const text = readFileSync("docs/ops/operations.log", "utf8");
21 const lines = text.split("\n").filter((l) => l.trim() !== "" && !l.startsWith("#"));
22 expect(lines.length).toBeGreaterThan(0);
23 expect(readOpsFile().length).toBe(lines.length);
24 });
25 test("a missing file reads as none", () => expect(readOpsFile("/nonexistent/x.log")).toEqual([]));
26 test("opsSince keeps the day and after", () => {
27 const ops = parseOps("2026-11-10 | host | a | old | x\n2026-11-15 | host | a | new | x");
28 expect(opsSince(ops, Date.parse("2026-11-15T03:00:00Z")).map((o) => o.what)).toEqual(["new"]);
29 expect(opLine(ops[1]!)).toBe("2026-11-15 host: new (a; x)");
30 });
31 });
32
33 describe("reading the file and the settings together", () => {
34 let dir: string;
35 let db: Database;
36 const event = (id: string, at: string, key: string, to: string) =>
37 db.query("INSERT INTO event (id, kind, actor_id, payload, created_at) VALUES (?1, 'capacity.changed', 'u1', ?2, ?3)").run(id, JSON.stringify({ key, from: "4", to, reason: "r" }), at);
38 beforeEach(() => {
39 dir = mkdtempSync(join(tmpdir(), "forge-ops-"));
40 db = new Database(":memory:");
41 migrate(db, migrationFiles);
42 db.query("INSERT INTO user (id, username, email, password_hash) VALUES ('u1', 'owner', '[email protected]', 'x')").run();
43 });
44 afterEach(() => {
45 db.close();
46 rmSync(dir, { recursive: true, force: true });
47 });
48
49 test("a capacity.changed event is a setting line with its reason; other events are not", () => {
50 event("e1", "2026-11-15T10:00:00Z", "checks.concurrency", "6");
51 db.query("INSERT INTO event (id, kind, actor_id, payload, created_at) VALUES ('e2', 'other', 'u1', '{}', '2026-11-15T10:00:00Z')").run();
52 expect(settingOps(db).map(opLine)).toEqual(["2026-11-15 setting: checks.concurrency: 4 to 6 (owner; r)"]);
53 });
54 test("newest day first; on one day the later entry first, file lines before settings of that day", () => {
55 const file = join(dir, "ops.log");
56 writeFileSync(file, "2026-11-10 | host | a | old | x\n2026-11-14 | host | a | first-file | x\n2026-11-14 | host | a | second-file | x\n");
57 event("e1", "2026-11-14T08:00:00Z", "k.one", "1");
58 event("e2", "2026-11-14T09:00:00Z", "k.two", "2");
59 event("e3", "2026-11-12T09:00:00Z", "k.three", "3");
60 expect(readOps(db, file).map((o) => o.what)).toEqual(["k.two: 4 to 2", "k.one: 4 to 1", "second-file", "first-file", "k.three: 4 to 3", "old"]);
61 });
62 test("the parsed file is kept until it changes", () => {
63 const file = join(dir, "ops.log");
64 writeFileSync(file, "2026-11-10 | host | a | one | x\n");
65 expect(readOpsFile(file).map((o) => o.what)).toEqual(["one"]);
66 writeFileSync(file, "2026-11-10 | host | a | one | x\n2026-11-11 | host | a | two | x\n");
67 utimesSync(file, new Date(), new Date(Date.now() + 5000));
68 expect(readOpsFile(file).map((o) => o.what)).toEqual(["one", "two"]);
69 expect(readOpsFile(file)).toBe(readOpsFile(file));
70 });
71 });
72
73 describe("host tuning", () => {
74 test("the shaper is one argv list, and a device that is not a name gets nothing", () => {
75 expect(shaperCommands("eth0")).toEqual([["tc", "qdisc", "replace", "dev", "eth0", "root", "cake", "bandwidth", "9mbit", "diffserv4", "nat", "wash"]]);
76 for (const bad of ["", "eth0; rm -rf /", "-x", "a b", "$(x)", "x".repeat(16)]) {
77 expect(validDevice(bad)).toBe(false);
78 expect(shaperCommands(bad)).toEqual([]);
79 }
80 });
81 test("the edge snippet declares the loud log and the crawler lane", () => {
82 const s = edgeSnippet();
83 expect(s).toContain("limit_req_zone");
84 expect(s).toContain("rate=2r/s");
85 expect(s).toContain("access_log /var/log/nginx/forge-loud.log combined if=$forge_loud;");
86 });
87 test("the facts say they are unconfirmed, claim no priority order, and carry no shell command", () => {
88 const facts = hostFacts();
89 expect(facts[0]).toEqual({ label: "Checked against the host", value: expect.stringContaining("Not yet") });
90 const text = facts.map((f) => `${f.label} ${f.value}`).join("\n");
91 expect(text).toContain("Held to 9 Mbit/s");
92 expect(text).toContain("1,000,000 bytes");
93 expect(text).not.toMatch(/pages first|backups|bulk|bun scripts|tc qdisc/);
94 expect(hostFacts({ ...HOST_TUNING, confirmed: true })[0]!.value).toBe("Yes");
95 });
96 test("the script prints the commands without applying", () => {
97 const p = Bun.spawnSync(["bun", "scripts/host-tune.ts", "--dev", "eth0"]);
98 expect(p.stdout.toString()).toContain("cake bandwidth 9mbit");
99 expect(Bun.spawnSync(["bun", "scripts/host-tune.ts", "--dev", "a;b"]).exitCode).toBe(2);
100 expect(Bun.spawnSync(["bun", "scripts/host-tune.ts", "--edge"]).stdout.toString()).toContain("forge_crawlers");
101 });
102 });
103
104 describe("the reading", () => {
105 const input = (over: Partial<Input> = {}): Input => ({
106 view: { at: "", seconds: 1, cpu: { busyPct: 0, cores: [], limitCores: 1, history: [] }, temps: [], hottestC: null, load: null, memory: null, disks: [], net: [], io: [], processes: 0, groups: [], duplicates: [] },
107 lineMbit: null,
108 checks: { running: 0, slots: 1, queued: 0 },
109 crew: { running: 0, slots: 1, queued: 0 },
110 ...over,
111 });
112 test("judge carries the host facts and the ops it is given, none when left out", () => {
113 const ops = parseOps("2026-11-15 | host | owner | shaper | why");
114 const b = judge(input({ ops }));
115 expect(b.host).toEqual(hostFacts());
116 expect(b.ops).toEqual(ops);
117 expect(judge(input()).ops).toEqual([]);
118 });
119 });
120
121 describe("the paper", () => {
122 test("a change to how the forge runs is said once, with its newest line", () => {
123 const f = { releases: [], pillarsMoved: [], ops: ["2026-11-15 host: shaper (owner; x)", "2026-11-15 host: edge (owner; y)"] };
124 expect(storyText(f)).toBe("2 changes to how it runs (host: shaper (owner; x))");
125 expect(storyText({ releases: [], pillarsMoved: [] })).toBe("");
126 });
127 });
test/palette-roster.test.ts +2 −1
@@ -130,10 +130,11 @@ test("a site-admin person is offered the whole roster", () => {
130130 ["pulls-to-land", "Pulls to land", "approved pulls, stale crew pulls, unproposed work"],
131131 ["valves", "Valves", "the week's budget as a valve per repository; turn one and the crew obeys"],
132132 ["waiting-for-me", "Waiting for me", "issues and pulls that wait for you"],
133 ["tell-undo", "Take a tell back", "a tell you sent in the last minute, while nobody has read it"],
133134 ["refused-asks", "Refused first asks", "the last 20 first asks turned away"],
134135 ["disks", "Disks", "machines' disk readings: 30-day write rate and projected rated-TBW date"],
135136 ["telemetry", "Telemetry", "which agents may read flow and system; enter flips the grant"],
136 ["tell-undo", "Take a tell back", "a tell you sent in the last minute, while nobody has read it"],
137 ["bottleneck", "Where the bottleneck is", "what limits the forge, what the host is tuned to, what changed"],
137138 ["health-voice", "Health — spoken", "how the forge is, in two plain sentences"],
138139 ["stalls", "Main-thread stalls", "the last hour's stall ms by route"],
139140 ["footing", "What the forge stands on", "each piece on another machine, in the order it would break the forge"],
test/talk-checkin-story.test.ts +12
@@ -91,6 +91,18 @@ describe("storyFacts", () => {
9191 });
9292 });
9393
94 describe("storyFacts with changes to how the forge runs", () => {
95 test("a kept setting change inside the window is in the story, and the story still has its pillars", async () => {
96 app.db
97 .query("INSERT INTO event (id, kind, actor_id, payload, created_at) VALUES ('ev-ops-1', 'capacity.changed', ?1, ?2, ?3)")
98 .run(alice.id, JSON.stringify({ key: "checks.concurrency", from: "4", to: "6", reason: "runners" }), new Date(T - 60_000).toISOString());
99 const s = await storyFacts(app, T - H);
100 expect(s.ops).toContain(`${new Date(T - 60_000).toISOString().slice(0, 10)} setting: checks.concurrency: 4 to 6 (alice; runners)`);
101 expect(s.pillarsMoved).toEqual([{ pillar: "P0 Trust", merged: 1 }]);
102 app.db.query("DELETE FROM event WHERE id = 'ev-ops-1'").run();
103 });
104 });
105
94106 describe("runRoutine's window is the last check-in", () => {
95107 test("the second run does not repeat the first's releases and pillars", async () => {
96108 const first = new Date(T + 60_000);

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs