Public: anyone can read this; the crew can change it.
The forge keeps its own record of what changed how it runs: host tuning, measurements and designs that today live in a session's notes #14240
Check · Review · Land · Live — Pushed since the last review.
scout/forge:scout/14238-the-forge-keeps-its-own-record-of-what-c → main 3 commits fork 2 threads · 2 unresolved
18 files
changed +873
−8
docs/README.md +3
| @@ -37,6 +37,9 @@ The guide, in order, one or two screens a page: [What a forge is](guide-what-a-f | ||
| 37 | 37 | ## Operate it |
| 38 | 38 | |
| 39 | 39 | - [](OPERATING.md): configuration, health, releases, switches and limits. |
| 40 | - [](ops/operations.log): one line per change to how the forge runs, with who and why; [](features/bottleneck.md) shows its newest and what the host is tuned to. | |
| 41 | - [](designs/forge-as-its-own-customer.md): production hosts and releases, a hosted workshop builds the forge (#13639). | |
| 42 | - [](designs/compute-leaves-the-forge.md): runners that join and pull work, pools, SLAs, and the plan for a second machine (#13652). | |
| 40 | 43 | - [](METRICS.md): the Prometheus endpoint. |
| 41 | 44 | - [](DOMAINS.md): the ways the forge stops and what it reports. |
| 42 | 45 | - [](features/app-own-domain.md): cut-over, certificate, and the way back. |
docs/designs/compute-leaves-the-forge.md added +145
| @@ -0,0 +1,145 @@ | ||
| 1 | <!-- Source: issue #13652, kept here so the design lives in the repository (#14238). --> | |
| 2 | ||
| 3 | The owner's direction, 2026-11-15, formalized. An epic beside #13639 (the forge as its own customer). Step R0 is the first issue to cut from it: name the seam in the check queue with no behaviour change. | |
| 4 | ||
| 5 | # Part two: compute leaves the forge | |
| 6 | ||
| 7 | 2026-11-15, from the owner: "most forges separate the compute. github actions runners, forgejo | |
| 8 | runners … in the platform we used KEDA to scale. what if my mac could join and help get through | |
| 9 | checks when on internet and power above 80%? … i want users to be able to scale up and down as | |
| 10 | they need. part of that will be SLAs … what about those inevitable users who have thousands of | |
| 11 | agents going?" | |
| 12 | ||
| 13 | ## The answer in five lines | |
| 14 | ||
| 15 | 1. Yes: the forge becomes a **control plane** (repositories, issues, the queue, the record of what | |
| 16 | ran and what it proved) and work runs on **runners** that join it. Today both are one process on | |
| 17 | one machine, which is why checks hold 50 of 72 cores beside the page thread. | |
| 18 | 2. A runner **pulls** work: it asks the forge for a job, gets a commit and a recipe, runs it in the | |
| 19 | sandbox the forge already has, and hands back a signed result. Nothing dials into the runner, so | |
| 20 | a laptop behind any network can be one. | |
| 21 | 3. Checks and crew runs are the same kind of job. One seam serves both, so "thousands of agents" | |
| 22 | is more runners, not a bigger forge. | |
| 23 | 4. The forge never needs a particular runner. Its own machine is always a runner; everything else | |
| 24 | is extra capacity that may come and go. The Mac helps when it is plugged in and online and is | |
| 25 | never missed when it is not. | |
| 26 | 5. An SLA is then a statement about two separate things, which is what makes it sellable: the | |
| 27 | control plane's availability (where your data lives) and the capacity you have reserved (how | |
| 28 | fast work clears). | |
| 29 | ||
| 30 | ## Why pull, and why now | |
| 31 | ||
| 32 | - **It is the proven shape.** Actions runners, Forgejo runners and Buildkite agents all long-poll a | |
| 33 | queue with a registration token. It needs no inbound port, survives a sleeping laptop, and | |
| 34 | scales by counting queue depth (what KEDA did on the platform: scale on the queue, not on CPU). | |
| 35 | - **The pieces exist.** The forge has a check queue with ranks, a cell that sandboxes a run, an | |
| 36 | impact map and kept results that are content-addressed, a model-runner seam (#12068), tokens, | |
| 37 | and hosted forges that are already separate processes. What is missing is the wire between a | |
| 38 | queue here and a sandbox there. | |
| 39 | - **It is the same move as part one.** The workshop takes development off production; runners take | |
| 40 | compute off whichever forge is asking. | |
| 41 | ||
| 42 | ## The runner, formally | |
| 43 | ||
| 44 | **Joining.** A person runs one command on a machine with a join token from the forge (scoped to a | |
| 45 | forge, an owner or one repository; revocable; shown once). The runner registers what it is: CPUs, | |
| 46 | memory, OS and architecture, what it may run (checks, crew runs, looks), and its **conditions**. | |
| 47 | ||
| 48 | **Conditions** are the runner's own rules for when it offers itself, evaluated on the runner: | |
| 49 | on mains power or battery above a threshold, on an unmetered network, idle for some minutes, | |
| 50 | inside set hours, at most N cores. The Mac's line is "on power or battery above 80%, online, at | |
| 51 | most 6 cores". A runner that stops meeting its conditions finishes or hands back its job and | |
| 52 | says so. | |
| 53 | ||
| 54 | **A job** is: a commit (fetched from the forge's git door with a short-lived token), a recipe | |
| 55 | (the repository's own `forge.json` check steps), the cache keys it may read, a time budget, and | |
| 56 | the secrets the repository was granted for that kind of run and nothing else. | |
| 57 | ||
| 58 | **Running.** The same cell the forge uses today (bubblewrap on Linux; a container or the system | |
| 59 | sandbox on macOS, with the honest note that a Mac result is for macOS-compatible steps: a check | |
| 60 | that needs Linux goes to a Linux runner). The runner streams the log back and returns the result | |
| 61 | with the test list it ran and kept. | |
| 62 | ||
| 63 | **Trust.** A result is signed by the runner's key and recorded with who ran it. Three classes: | |
| 64 | the forge's own machine (trusted); a runner the owner enrolled (trusted for that owner's | |
| 65 | repositories); anyone else's (its green is evidence, and the landing gate re-runs on a trusted | |
| 66 | runner). A pull from a stranger never runs on a runner that holds anyone's secrets: the outside | |
| 67 | lane the forge already has. | |
| 68 | ||
| 69 | **Losing a runner.** A job has a lease. A runner that goes silent loses the lease and the job is | |
| 70 | queued again without counting as a try (the rule that now holds for restarts). | |
| 71 | ||
| 72 | **Caches.** Installs, the impact map and kept results are content-addressed, so a runner fetches | |
| 73 | what it lacks from the forge by hash and can keep a local store. A cold runner is slower, never | |
| 74 | wrong. | |
| 75 | ||
| 76 | ## Scaling up and down | |
| 77 | ||
| 78 | - The queue's depth and age, by class of job, is the one signal. A hosted customer sets "at most N | |
| 79 | runners, at least M" and the host adds or removes cells or machines to hold the queue's age under | |
| 80 | a target. That is the KEDA idea with the forge's own queue as the metric. | |
| 81 | - **Thousands of agents.** A crew run is a job: the model call is the provider's, the tool calls | |
| 82 | and the workspace are the runner's. A customer with a thousand agents has a control plane sized | |
| 83 | for writes (issues, pulls, events: this is where the database work matters) and a runner pool | |
| 84 | sized for the agents. The forge's per-forge limits become per-pool limits. | |
| 85 | - **Many forges, one pool.** An organisation buys one pool that serves all its people's forges. | |
| 86 | That is the enterprise shape: a forge per person or team, a shared and metered runner pool, one | |
| 87 | bill. | |
| 88 | ||
| 89 | ## SLAs, as things that can be measured and sold | |
| 90 | ||
| 91 | | What is promised | What it rests on | Who can promise it | | |
| 92 | |---|---|---| | |
| 93 | | The forge answers (control plane) | where the forge process and its data run; backups and a proven restore | the host of that machine | | |
| 94 | | Work starts within N minutes | reserved runners in the pool | whoever provides the runners | | |
| 95 | | Work finishes at a given rate | runner count and the size of the repository's checks | the customer's choice of pool | | |
| 96 | ||
| 97 | Your vxrail gives you the first row for your own forge and for customers you host: you control the | |
| 98 | machine. The honest ceiling today is one machine on one power feed and one uplink; a battery and a | |
| 99 | second uplink are what turn "I sleep next to it" into a number. A customer who wants more nines | |
| 100 | than one site can give takes a second control-plane site (the restore you already prove nightly is | |
| 101 | the seed of that), and that is a tier, not a redesign. | |
| 102 | ||
| 103 | ## What this does to the forge's architecture, in one picture | |
| 104 | ||
| 105 | ||
| 106 | people and agents | |
| 107 | | | |
| 108 | +----------v-----------+ | |
| 109 | | control plane | repositories, issues, pulls, queue, records, | |
| 110 | | (a forge) | pages, MCP, git door; small CPU, careful writes | |
| 111 | +----+------------+----+ | |
| 112 | | jobs (pull)| results (signed) | |
| 113 | +-------v--+ +-----v-----+ +-----------+ | |
| 114 | | runner: | | runner: | | runner: | checks, crew runs, looks; | |
| 115 | | its own | | a pool | | the Mac, | join and leave freely; | |
| 116 | | machine | | (hosted) | | when able | sandboxed; no secrets it was not given | |
| 117 | +----------+ +-----------+ +-----------+ | |
| 118 | ||
| 119 | ||
| 120 | Part one says who builds the forge and where. Part two says where anything runs. Together: a | |
| 121 | forge is small and always there; the work it asks for happens wherever there is room. | |
| 122 | ||
| 123 | ## Steps, each reversible | |
| 124 | ||
| 125 | | Step | What | Proves | | |
| 126 | |---|---|---| | |
| 127 | | R0 | Name the seam in code: the check queue hands a job to "a runner" that today is the same process. No behaviour change. | the job is fully described by data | | |
| 128 | | R1 | A runner as a separate process on the same machine, pulling over HTTP with a token. Checks only. | the wire; P's page thread without checks beside it | | |
| 129 | | R2 | The Mac as a second runner with conditions (power, network, cores). Mac-compatible steps only at first. | join, leave, lease, cache by hash | | |
| 130 | | R3 | Crew runs through the same seam (#12068). | "many agents" is a pool size | | |
| 131 | | R4 | Pools for hosted forges: min, max, scale on queue age; metered. | scale up and down as a product | | |
| 132 | | R5 | SLA tiers written against measured numbers; an outside probe for the first row. | something to sell | | |
| 133 | ||
| 134 | ## Risks | |
| 135 | ||
| 136 | - **Security is the whole game.** A runner executes untrusted code and holds a token. Scope every | |
| 137 | token to one job; give secrets only to trusted runners; treat a volunteer runner's green as | |
| 138 | evidence, never as proof. | |
| 139 | - **macOS is not Linux.** The forge's checks assume Linux cells. The Mac can run the large share | |
| 140 | that is plain `bun test`; the rest stays on Linux. Do not chase parity. | |
| 141 | - **A distributed queue has failure modes one process does not**: double runs, lost results, | |
| 142 | clock skew. Leases, idempotency keys (already used for schedules) and "a rerun is always safe" | |
| 143 | cover them; they need tests of their own. | |
| 144 | - **Do not build a second scheduler.** The check queue with ranks is the scheduler. Runners are | |
| 145 | its hands. |
docs/designs/forge-as-its-own-customer.md added +188
| @@ -0,0 +1,188 @@ | ||
| 1 | <!-- Source: issue #13639, kept here so the design lives in the repository (#14238). --> | |
| 2 | ||
| 3 | The owner's direction, 2026-11-15, formalized. This is an epic: each step in the plan's table becomes its own issue as it is reached; steps 0 and 1 are being piloted now. | |
| 4 | ||
| 5 | # The forge as its own customer | |
| 6 | ||
| 7 | 2026-11-15. Written from the owner's direction: "why don't we dogfood our own forge? | |
| 8 | forge.open-platform.sh could be just the hosted forge. production. only certain users can | |
| 9 | propose changes upstream and it will take a ton of work off of the main forge … I think this | |
| 10 | also gets to your point of the packages and modularization." | |
| 11 | ||
| 12 | ## The answer in five lines | |
| 13 | ||
| 14 | 1. Yes. Today one process is three things at once: the product's production site, the workshop | |
| 15 | where the forge is built, and the owner's personal forge. Every problem of the last three days | |
| 16 | came from that overlap. | |
| 17 | 2. Split by role, not by code: **Production** hosts and sells and holds the canonical repository. | |
| 18 | **The workshop** is a hosted forge, a customer like any other, where the crew builds the forge. | |
| 19 | 3. The workshop proposes upstream. Only named maintainers may. Production lands what arrives | |
| 20 | already proven and releases it on a train: the workshop runs each candidate first. | |
| 21 | 4. This makes us the heaviest customer of our own hosting, which is the fastest way to find what | |
| 22 | hosted forges lack. | |
| 23 | 5. Modularization follows the same line: a small kernel, and features that live in their own | |
| 24 | repositories and are absorbed, each with its own tests. That is what makes checks cheap. | |
| 25 | ||
| 26 | ## What is true today (measured, 2026-11-13 to 11-15) | |
| 27 | ||
| 28 | `forge.open-platform.sh` is one container on one machine (72 of 80 CPUs, 251 GB). In it: | |
| 29 | ||
| 30 | | Role | What it does there | What it costs the other roles | | |
| 31 | | -------------- | -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | |
| 32 | | Production | front door, sign-up, pricing, 16 hosted forges in cells, `gretchy.store`, `open-platform.sh` | must never be down | | |
| 33 | | Workshop | 8 builders, a planner, ~3,150 test files per check, 30 to 60 pulls opened an hour | checks hold 50 to 62 cores; the serving thread stalls 5 to 20 times a minute; the database took 2.4M events | | |
| 34 | | Personal forge | the owner's apps and their crews | its home page fills with the workshop's noise | | |
| 35 | ||
| 36 | Incidents that were this overlap and nothing else: | |
| 37 | ||
| 38 | - **Five outages in three days** (2 min, 2.5 min, three Bun crashes, a 4.5 min crash loop). Each was | |
| 39 | development reaching production: a helper opening the live database, a profiler, a worker the | |
| 40 | crew added, a default the crew changed. Customers' forges ride the same restarts. | |
| 41 | - **About ten releases a day**, each a restart of the host of every customer, because the workshop | |
| 42 | wants its own fixes live. | |
| 43 | - **Restarts cut a quarter of checks** one night. The workshop's releases interrupted the | |
| 44 | workshop's own work. | |
| 45 | - **The staging rig cannot see production's shape** (no crew, no apps, no checks), so two bad | |
| 46 | releases passed it. | |
| 47 | - **The week's model allowance** is spent by a planner that opens general work on the production | |
| 48 | forge. | |
| 49 | ||
| 50 | What already exists and is unused for this: hosted forges in their own cells, each with its own | |
| 51 | crew, budget and checks; origins and forks between forges; `propose_to_parent_forge` (a child | |
| 52 | offers a landed change to its parent as a pull); following an upstream and taking updates; a | |
| 53 | release keeper per forge; absorbed features (`feature` in `forge.json`, as the morning paper is). | |
| 54 | The contribution loop was proven end to end between forges on 2026-11-13. | |
| 55 | ||
| 56 | ## The model | |
| 57 | ||
| 58 | ### Three roles, three places | |
| 59 | ||
| 60 | **Production (P): `forge.open-platform.sh`.** Hosts forges, sells them, serves the sites, holds | |
| 61 | the canonical `forge` repository and its releases. It runs no builders and no planner for the | |
| 62 | forge's own code. Its engine's only work on that repository is admitting proposals and cutting | |
| 63 | releases. It restarts when a release is promoted: once or twice a day, at a chosen hour. | |
| 64 | ||
| 65 | **The workshop (W): a hosted forge on P, e.g. `workshop--forge.open-platform.sh`.** A customer. | |
| 66 | It holds a fork of `forge`, the backlog, the planner, the builders, the checks. It runs the newest | |
| 67 | candidate build at all times (its release keeper is on), so it is also the canary. It can crash, | |
| 68 | restart ten times an hour and fill its own database, and P does not feel it. | |
| 69 | ||
| 70 | **Personal forges.** The owner's apps move to a forge of his own (`trevato--forge…`), also a | |
| 71 | customer. P's home page is then the product's, and his home page is his kanban. | |
| 72 | ||
| 73 | ### The flow of a change | |
| 74 | ||
| 75 | 1. An issue is opened on W (or arrives from P: a customer's feedback, a stranger's report, routed | |
| 76 | to W as an issue with its origin kept). | |
| 77 | 2. W's crew builds it and W's check passes on the merge with W's main. W lands it. | |
| 78 | 3. W proposes upstream: one proposal per landed change, or one per train (see open questions). | |
| 79 | The proposal carries W's attestation: the commit, the base, the check run, its result, signed | |
| 80 | by W's key that P holds because P hosts W. | |
| 81 | 4. P admits it: the proposer is on the maintainer list; the attestation verifies; it merges | |
| 82 | cleanly; P's own small gate passes on the exact tree that becomes main (rule scans and a boot, | |
| 83 | minutes not an hour). P does not re-run 3,150 tests. | |
| 84 | 5. A release candidate is cut from P's main. W takes it first, automatically. After a soak on W | |
| 85 | under real crew load (a number of hours with no restart, no fatal line, stalls and memory | |
| 86 | inside their first-hour figures), P promotes it at its release hour. A failed soak never | |
| 87 | reaches P. | |
| 88 | ||
| 89 | ### Who may propose upstream | |
| 90 | ||
| 91 | A short list on P's `forge` repository: the owner and W's landing identity. Everyone else (Ben, | |
| 92 | grokbot, a stranger) forks from P to a forge of their own and proposes to **W**, where a crew and | |
| 93 | a full check can receive them. P stays the root of the fork tree, so "contribute anywhere" keeps | |
| 94 | its shape: you fork the canonical repository, and your proposal is routed to the workshop. | |
| 95 | ||
| 96 | ### What P keeps | |
| 97 | ||
| 98 | Accounts and billing; the hosted forges and their cells; the front door, pricing, docs, releases | |
| 99 | and changelog; the canonical repository (read by everyone, written by admitted proposals only); | |
| 100 | intake of feedback; the edge. Its database is the customers', not the workshop's: it stops | |
| 101 | growing by a million rows a week. | |
| 102 | ||
| 103 | ## Why this is right | |
| 104 | ||
| 105 | - **Blast radius.** A bad commit can only hurt W until a human-free soak has passed. Every outage | |
| 106 | above would have been W's. | |
| 107 | - **The rig becomes real.** W is production-shaped by construction: crew, checks, apps, a full | |
| 108 | database. The 0.7.599 crash loop needs W's config to appear; it would have appeared on W. | |
| 109 | - **Production gets calm.** No checks on its CPUs, a release a day, a small database. The page | |
| 110 | thread's stalls were the workshop's work. | |
| 111 | - **We find our own product's holes first.** A workshop needs a size above "large" (4 CPUs, 8 GB), | |
| 112 | checks that can run a forge's own end-to-end tests inside a cell, egress for installs, backups | |
| 113 | of a hosted forge, a way to carry a repository with its history to another forge, a model | |
| 114 | account on a hosted forge. Customers need every one of those. Today nobody at home feels them. | |
| 115 | - **It is the vision already written down.** More forges is how this scales; a forge divides when | |
| 116 | heavy; what is not carried is archived; the kernel is small and what it generates is large. | |
| 117 | ||
| 118 | ## What it costs, honestly | |
| 119 | ||
| 120 | 1. **Compute does not shrink by moving.** W's checks still want 50 cores, on the same machine. | |
| 121 | What changes is that they are in a cell with a CPU set of its own, apart from P's serving cores. | |
| 122 | Real relief on check time is the modularization below, not the move. | |
| 123 | 2. **A hosted forge cannot yet run the forge's own suite.** Tests that spawn cells, take looks | |
| 124 | with Chromium or install packages fail or are skipped when nested in a cell (seen on 11-14 and | |
| 125 | 11-15). Either cells learn to nest, or those tests run in a lane P provides to W. This is the | |
| 126 | first thing the pilot must measure. | |
| 127 | 3. **The backlog lives on P.** 17,700 issues and 9,000 pulls. Moving them needs "carry a | |
| 128 | repository with its history to another forge": one file per repository as an export format | |
| 129 | (measured as the right use of the per-repository database idea on 11-15). Until it exists, W | |
| 130 | starts with a fork and new work, and the history stays readable on P. | |
| 131 | 4. **Two forges to keep well.** W must take updates by itself and be backed up like a customer. | |
| 132 | That is the product working, or a list of what to fix. | |
| 133 | 5. **Admission by attestation is new trust code.** If it is wrong, an unproven change reaches | |
| 134 | production's main. Mitigation: P's own gate on the exact tree, the soak on W, and the 75-second | |
| 135 | watch with rollback that already exists. | |
| 136 | 6. **Latency of a fix.** A change reaches customers in hours, not minutes. An emergency lane (the | |
| 137 | owner promotes now) keeps the old speed for the rare case. | |
| 138 | 7. **One machine.** W, P and the customers share a host. A second machine for W is the real | |
| 139 | isolation, and the design does not depend on it. | |
| 140 | ||
| 141 | ## Modularization: what "packages" should mean here | |
| 142 | ||
| 143 | The measured ceiling on checks: a pull's fast tier still selects about 2,670 of 3,150 test files, | |
| 144 | because 567 source files are each executed by more than 1,000 tests. Most tests boot a whole | |
| 145 | forge. Selection and caching cannot fix that; boundaries can. | |
| 146 | ||
| 147 | 1. **Kernel**: database, migrations, log, ids, events, config, the git door, auth. Small, slow to | |
| 148 | change, the only thing every test may boot. | |
| 149 | 2. **Engines**, each a package with its own tests against the kernel only: checks, crew and | |
| 150 | autonomy, apps runtime and cells, spawn and hosting, snapshot and restore, flow and pictures. | |
| 151 | 3. **Surfaces**: web pages, MCP, terminal. Tested against engines' interfaces with fixtures, not | |
| 152 | a booted forge. | |
| 153 | 4. **Features** in their own repositories, absorbed through the `feature` block that already | |
| 154 | exists: the morning paper today; billing, the changelog, boards, the map and talk are | |
| 155 | candidates. A feature's check is its own repository's check: minutes. | |
| 156 | ||
| 157 | Order: draw the boundaries inside the one repository first (workspace packages, an import rule | |
| 158 | per package, a test tier per package), measure the fast tier per package, and only then move a | |
| 159 | feature out. A package boundary is a rule test, like the ones the gate already has. | |
| 160 | ||
| 161 | The workshop and the packages are the same idea at two sizes: work lives in a unit small enough | |
| 162 | to check quickly, and proposes to the unit above it. | |
| 163 | ||
| 164 | ## The plan, in reversible steps | |
| 165 | ||
| 166 | | Step | What | Proves | Undo | | |
| 167 | | ---- | ------------------------------------------------------------------------------------------------------------------- | --------------------------------- | ----------------------------- | | |
| 168 | | 0 | Stand up W as a hosted forge with a fork of `forge` and a crew on lent credit. Run the forge's check inside W once. | what a hosted forge cannot do yet | remove W | | |
| 169 | | 1 | W builds one real issue and proposes it upstream with `propose_to_parent_forge`; P lands it by today's path. | the loop, with numbers | none needed | | |
| 170 | | 2 | A workshop size (CPUs and memory a crew's forge needs) and a CPU set for cells apart from P's serving cores. | stalls on P fall | size and pin are settings | | |
| 171 | | 3 | W's release keeper on: W runs every candidate. P promotes after a soak. | the canary | turn the keeper off | | |
| 172 | | 4 | Attested admission on P; the maintainer list. P's builders for `forge` off. | P does no development | list and switch | | |
| 173 | | 5 | Carry `forge`'s backlog to W in the per-repository export format. | history moves | P keeps its copy until proven | | |
| 174 | | 6 | The owner's apps to his own forge. | P's home is the product's | same carry, back | | |
| 175 | | 7 | Package boundaries, then the first feature moved out. | checks in minutes | boundaries are rules | | |
| 176 | ||
| 177 | Steps 0 to 3 change nothing a customer can see and can each be undone the same day. | |
| 178 | ||
| 179 | ## Open questions for the owner | |
| 180 | ||
| 181 | 1. **One proposal per change, or one per train?** Per change keeps history and blame simple and | |
| 182 | is what the tools do today. Per train is fewer admissions and matches releases. Recommended: | |
| 183 | per change, admitted in batches. | |
| 184 | 2. **Does W live on this machine?** Recommended: yes to start, pinned apart; a second machine when | |
| 185 | one is at hand. | |
| 186 | 3. **Where do customers' feedback and strangers' reports land?** Recommended: P takes them and | |
| 187 | routes them to W with the origin kept, so nobody has to know W exists. | |
| 188 | 4. **Who is on the maintainer list besides you and W?** |
docs/features/bottleneck.md +18 −1
| @@ -17,4 +17,21 @@ it. Two doors, one reading (`readBottleneck`): | ||
| 17 | 17 | page through the public address against loopback, bytes split by sender, and the host's traffic |
| 18 | 18 | shaping. |
| 19 | 19 | |
| 20 | Code: `src/system/bottleneck.ts`, `src/web/pages/bottleneck.ts`. | |
| 20 | - **What the host is tuned to** is declared, not measured, and not yet confirmed: `src/system/host_tuning.ts` | |
| 21 | holds the traffic shaper (cake at 9 Mbit, four lanes, LAN unshaped) and the edge proxy's | |
| 22 | large-or-slow-response log and crawler lane. The figures were reconstructed from issue #14238; the page | |
| 23 | says "Not yet" under *Checked against the host* until the owner has compared them with `tc qdisc show` | |
| 24 | and the edge's nginx.conf and set `confirmed` in that file. The marks that would put pages ahead of | |
| 25 | backups are not set by anything here, and the page does not claim a priority order. | |
| 26 | `bun scripts/host-tune.ts --apply --dev <uplink>` applies the shaper (run it again after a reboot); | |
| 27 | `--edge` prints the proxy snippet to include. | |
| 28 | - **Changes to how the forge runs** are `docs/ops/operations.log`, one line each | |
| 29 | (`day | setting|host|incident | who | what | why`; append one in the pull that makes the change), with | |
| 30 | the settings changed on /settings/capacity read from their events. The newest twelve show as dated rows | |
| 31 | on the page, in the terminal and in the MCP `bottleneck` reading (`host`: `{label, value}` rows, `ops`: | |
| 32 | `{day, kind, who, what, why}`); the morning paper says how many came since its last line. | |
| 33 | - **In the terminal** (admins, as the page): the palette's *Where the bottleneck is* shows the sentence, | |
| 34 | a line a row, what the host is tuned to and the changes; `j`/`k` scroll, `g` reads again, `esc` goes back. | |
| 35 | ||
| 36 | Code: `src/system/bottleneck.ts`, `src/system/ops_log.ts`, `src/system/host_tuning.ts`, `src/web/pages/bottleneck.ts`, | |
| 37 | `src/tui/screens/bottleneck.ts`. |
docs/ops/operations.log added +10
| @@ -0,0 +1,10 @@ | ||
| 1 | # One line per change to how the forge runs, newest last: | |
| 2 | # <YYYY-MM-DD> | <setting|host|incident> | <who> | <what> | <why> | |
| 3 | # Setting changes made on /settings/capacity are kept by the forge itself as | |
| 4 | # `capacity.changed` events and read from there; this file holds the rest. | |
| 5 | # Append a line in the same pull that makes the change. Pipes are not allowed inside a field. | |
| 6 | # The first lines were written from issue #14238, not from the host: only what it states is here. | |
| 7 | 2026-11-15 | host | owner | Traffic shaper on the uplink: cake at 9 Mbit, diffserv4, LAN not shaped (from #14238, not yet read from the host) | Set by hand on the server and lost on reboot; now declared in src/system/host_tuning.ts | |
| 8 | 2026-11-15 | host | owner | Edge proxy logs large or slow responses to a separate access log (from #14238, not yet read from nginx.conf) | Declared in src/system/host_tuning.ts | |
| 9 | 2026-11-15 | host | owner | Edge proxy paces crawlers in their own lane (from #14238, not yet read from nginx.conf) | Declared in src/system/host_tuning.ts | |
| 10 | 2026-11-15 | incident | owner | Restarts cut a quarter of a night's checks | See docs/designs/forge-as-its-own-customer.md |
scripts/host-tune.ts added +29
| @@ -0,0 +1,29 @@ | ||
| 1 | #!/usr/bin/env bun | |
| 2 | /** | |
| 3 | * Applies or shows what the host is tuned to (#14238; src/system/host_tuning.ts is the | |
| 4 | * declaration). | |
| 5 | * bun scripts/host-tune.ts --dev eth0 print the tc commands | |
| 6 | * bun scripts/host-tune.ts --apply --dev eth0 run them (root, on the host) | |
| 7 | * bun scripts/host-tune.ts --edge print the edge proxy's snippet | |
| 8 | * Each command is an argument list built from the declaration and a validated device name; no | |
| 9 | * text is ever handed to a shell. Run it again after a reboot: `replace` makes it safe to repeat. | |
| 10 | */ | |
| 11 | import { edgeSnippet, shaperCommands, validDevice } from "../src/system/host_tuning.ts"; | |
| 12 | ||
| 13 | const args = process.argv.slice(2); | |
| 14 | if (args.includes("--edge")) { | |
| 15 | process.stdout.write(edgeSnippet()); | |
| 16 | process.exit(0); | |
| 17 | } | |
| 18 | const at = args.indexOf("--dev"); | |
| 19 | const dev = at >= 0 ? (args[at + 1] ?? "") : ""; | |
| 20 | if (!validDevice(dev)) { | |
| 21 | console.error("usage: bun scripts/host-tune.ts [--apply] --dev <uplink interface> | --edge"); | |
| 22 | process.exit(2); | |
| 23 | } | |
| 24 | for (const argv of shaperCommands(dev)) { | |
| 25 | console.log(argv.join(" ")); | |
| 26 | if (!args.includes("--apply")) continue; | |
| 27 | const done = Bun.spawnSync(argv, { stdout: "inherit", stderr: "inherit" }); | |
| 28 | if (done.exitCode !== 0) process.exit(done.exitCode ?? 1); | |
| 29 | } |
src/crew/routine_facts_story.ts +9 −2
| @@ -9,12 +9,18 @@ import { publicForgeRepo } from "../core/forge_repo.ts"; | ||
| 9 | 9 | import { mergedByPillar, pillarsOf, servedBy } from "../core/pillars.ts"; |
| 10 | 10 | import { readChangelog } from "../release/changelog.ts"; |
| 11 | 11 | import { repoDir } from "../git/repos.ts"; |
| 12 | import { opLine, opsSince, readOps } from "../system/ops_log.ts"; | |
| 12 | 13 | |
| 13 | 14 | export interface Story { |
| 14 | 15 | /** Versions cut since the last check-in, oldest first. */ |
| 15 | 16 | readonly releases: readonly string[]; |
| 16 | 17 | /** Pillars that merged work since then, with how many pulls. */ |
| 17 | 18 | readonly pillarsMoved: readonly { readonly pillar: string; readonly merged: number }[]; |
| 19 | /** | |
| 20 | * Changes to how the forge runs since then (docs/ops/operations.log, kept settings), newest | |
| 21 | * first. | |
| 22 | */ | |
| 23 | readonly ops?: readonly string[]; | |
| 18 | 24 | } |
| 19 | 25 | |
| 20 | 26 | export const NO_STORY: Story = { releases: [], pillarsMoved: [] }; |
| @@ -29,8 +35,9 @@ export function releasesSince(entries: readonly { version: string; date: string | ||
| 29 | 35 | |
| 30 | 36 | /** The story since `sinceMs`; empty when the forge has no public source repository. */ |
| 31 | 37 | export async function storyFacts(app: App, sinceMs: number): Promise<Story> { |
| 38 | const ops = opsSince(readOps(app.db), sinceMs).map(opLine); | |
| 32 | 39 | const repo = publicForgeRepo(app); |
| 33 | if (repo === undefined) return NO_STORY; | |
| 40 | if (repo === undefined) return ops.length === 0 ? NO_STORY : { ...NO_STORY, ops }; | |
| 34 | 41 | const entries = await readChangelog(repoDir(app.reposRoot, repo.ownerName, repo.name), repo.defaultBranch); |
| 35 | 42 | const set = pillarsOf(app.db, repo); |
| 36 | 43 | const moved: { pillar: string; merged: number }[] = []; |
| @@ -40,5 +47,5 @@ export async function storyFacts(app: App, sinceMs: number): Promise<Story> { | ||
| 40 | 47 | } |
| 41 | 48 | moved.sort((x, y) => y.merged - x.merged); |
| 42 | 49 | } |
| 43 | return { releases: releasesSince(entries, sinceMs), pillarsMoved: moved }; | |
| 50 | return { releases: releasesSince(entries, sinceMs), pillarsMoved: moved, ops }; | |
| 44 | 51 | } |
src/crew/routine_report.ts +2
| @@ -50,6 +50,8 @@ export function storyText(f: Story): string { | ||
| 50 | 50 | parts.push(`released ${f.releases.length > 3 ? `${f.releases.length} (to ${f.releases.at(-1)})` : shown}`); |
| 51 | 51 | } |
| 52 | 52 | if (f.pillarsMoved.length > 0) parts.push(`moved: ${f.pillarsMoved.slice(0, 3).map((p) => `${p.pillar} ${p.merged}`).join(", ")}`); |
| 53 | const ops = f.ops ?? []; | |
| 54 | if (ops.length > 0) parts.push(`${plural(ops.length, "change")} to how it runs (${ops[0]!.replace(/^\d{4}-\d{2}-\d{2} /, "").slice(0, 80)})`); | |
| 53 | 55 | return parts.join("; "); |
| 54 | 56 | } |
| 55 | 57 |
src/system/bottleneck.ts +15 −2
| @@ -10,6 +10,8 @@ | ||
| 10 | 10 | import type { Database } from "bun:sqlite"; |
| 11 | 11 | import { capacityValue } from "../core/capacity.ts"; |
| 12 | 12 | import { runningAgentRuns } from "../crew/running.ts"; |
| 13 | import { hostFacts, type HostFact } from "./host_tuning.ts"; | |
| 14 | import { readOps, type Op } from "./ops_log.ts"; | |
| 13 | 15 | import type { SystemView } from "./view.ts"; |
| 14 | 16 | |
| 15 | 17 | export type RowId = "line" | "cpu" | "checks" | "crew" | "memory" | "disk"; |
| @@ -29,7 +31,15 @@ export type Row = { | ||
| 29 | 31 | lever: { label: string; href: string } | null; |
| 30 | 32 | }; |
| 31 | 33 | |
| 32 | export type Bottleneck = { at: string; says: string; limited: RowId | null; rows: Row[]; unseen: string[] }; | |
| 34 | export type Bottleneck = { at: string; says: string; limited: RowId | null; rows: Row[]; unseen: string[]; | |
| 35 | /** What the host is tuned to, declared (src/system/host_tuning.ts). */ | |
| 36 | host: HostFact[]; | |
| 37 | /** The newest changes to how the forge runs (docs/ops/operations.log and the kept settings). */ | |
| 38 | ops: Op[]; | |
| 39 | }; | |
| 40 | ||
| 41 | /** How many changes to how the forge runs a reading carries. */ | |
| 42 | export const OPS_SHOWN = 12; | |
| 33 | 43 | |
| 34 | 44 | export type Input = { |
| 35 | 45 | view: SystemView; |
| @@ -37,6 +47,8 @@ export type Input = { | ||
| 37 | 47 | lineMbit: number | null; |
| 38 | 48 | checks: { running: number; slots: number; queued: number }; |
| 39 | 49 | crew: { running: number; slots: number; queued: number }; |
| 50 | /** The changes to how the forge runs to carry along; none when left out. */ | |
| 51 | ops?: readonly Op[]; | |
| 40 | 52 | }; |
| 41 | 53 | |
| 42 | 54 | /** A row at or past this share of its ceiling is what limits the forge. */ |
| @@ -136,7 +148,7 @@ export function judge(input: Input, at = new Date()): Bottleneck { | ||
| 136 | 148 | |
| 137 | 149 | let worst: Row | null = null; |
| 138 | 150 | for (const r of rows) if (r.fraction !== null && r.fraction >= LIMIT && (worst === null || r.fraction > (worst.fraction ?? 0))) worst = r; |
| 139 | return { at: at.toISOString(), says: worst === null ? "Nothing is at its limit." : sentence(worst, input, tx), limited: worst?.id ?? null, rows, unseen: [...UNSEEN] }; | |
| 151 | return { at: at.toISOString(), says: worst === null ? "Nothing is at its limit." : sentence(worst, input, tx), limited: worst?.id ?? null, rows, unseen: [...UNSEEN], host: hostFacts(), ops: [...(input.ops ?? [])] }; | |
| 140 | 152 | } |
| 141 | 153 | |
| 142 | 154 | function sentence(r: Row, input: Input, tx: number): string { |
| @@ -160,5 +172,6 @@ export function readBottleneck(db: Database, view: SystemView, env: string | und | ||
| 160 | 172 | lineMbit: lineCeiling(env), |
| 161 | 173 | checks: { running: count(db, "check", "running"), slots: Number(capacityValue(db, "checks.concurrency")), queued: count(db, "check", "queued") }, |
| 162 | 174 | crew: { running: runningAgentRuns(), slots: Number(capacityValue(db, "crew.concurrency")), queued: count(db, "agent", "queued") }, |
| 175 | ops: readOps(db).slice(0, OPS_SHOWN), | |
| 163 | 176 | }); |
| 164 | 177 | } |
src/system/host_tuning.ts added +79
| @@ -0,0 +1,79 @@ | ||
| 1 | /** | |
| 2 | * What the host is tuned to, declared (#14238): the traffic shaper and the edge proxy's lanes were | |
| 3 | * set by hand on the server and lost on a reboot. This is the one description; | |
| 4 | * `scripts/host-tune.ts` prints or applies it and the bottleneck page shows it. Plain data in, | |
| 5 | * plain data out: nothing here touches the machine. The figures were reconstructed from the issue, | |
| 6 | * not read from the host: `confirmed` stays false until the owner has checked them against | |
| 7 | * `tc qdisc show` and the edge's nginx.conf, and the page says so while it is. | |
| 8 | */ | |
| 9 | ||
| 10 | export const HOST_TUNING = { | |
| 11 | /** Set to true only once the owner has read these figures off the host itself. */ | |
| 12 | confirmed: false as boolean, | |
| 13 | shaper: { | |
| 14 | /** Megabits a second the shaper lets out; just under the 16.3 Mbit/s the line carries. */ | |
| 15 | mbit: 9, | |
| 16 | lanUnshaped: true, | |
| 17 | }, | |
| 18 | edge: { | |
| 19 | /** The edge proxy's configuration on the host. */ | |
| 20 | conf: "/srv/forge/edge/nginx.conf", | |
| 21 | /** A response this large, in bytes, or this slow, in seconds, goes to the separate log. */ | |
| 22 | logBytes: 1_000_000, | |
| 23 | logSeconds: 1, | |
| 24 | /** Crawlers are paced to this many requests a second, in their own lane. */ | |
| 25 | crawlerPerSecond: 2, | |
| 26 | }, | |
| 27 | } as const; | |
| 28 | ||
| 29 | /** An interface name the kernel accepts and no shell would read as anything else. */ | |
| 30 | export const validDevice = (s: string): boolean => /^[a-z][a-z0-9._-]{0,14}$/i.test(s); | |
| 31 | ||
| 32 | /** The `tc` argument lists that apply the shaper to `dev`: argv, never a shell line. */ | |
| 33 | export function shaperCommands(dev: string, t = HOST_TUNING.shaper): string[][] { | |
| 34 | if (!validDevice(dev)) return []; | |
| 35 | // cake's diffserv4 sorts packets into four tins by their marks; nothing in this repository sets | |
| 36 | // the marks, so which traffic rides which tin is not declared here. The LAN is left unshaped by | |
| 37 | // shaping only the uplink. | |
| 38 | return [["tc", "qdisc", "replace", "dev", dev, "root", "cake", "bandwidth", `${t.mbit}mbit`, "diffserv4", "nat", "wash"]]; | |
| 39 | } | |
| 40 | ||
| 41 | /** The edge proxy's two additions, as text to include from nginx.conf (`http` context). */ | |
| 42 | export function edgeSnippet(t = HOST_TUNING.edge): string { | |
| 43 | return [ | |
| 44 | `# Declared by the forge (src/system/host_tuning.ts); include it from ${t.conf}.`, | |
| 45 | `map "$body_bytes_sent:$request_time" $forge_loud {`, | |
| 46 | ` default 0;`, | |
| 47 | ` "~^\\d{${String(t.logBytes).length},}:" 1;`, | |
| 48 | ` "~^\\d+:(?:[${t.logSeconds}-9]|\\d{2,})\\." 1;`, | |
| 49 | `}`, | |
| 50 | `access_log /var/log/nginx/forge-loud.log combined if=$forge_loud;`, | |
| 51 | `map $http_user_agent $forge_crawler {`, | |
| 52 | ` default "";`, | |
| 53 | ` "~*(bot|crawl|spider|slurp)" $binary_remote_addr;`, | |
| 54 | `}`, | |
| 55 | `limit_req_zone $forge_crawler zone=forge_crawlers:10m rate=${t.crawlerPerSecond}r/s;`, | |
| 56 | `# and, in the server block: limit_req zone=forge_crawlers burst=10;`, | |
| 57 | ``, | |
| 58 | ].join("\n"); | |
| 59 | } | |
| 60 | ||
| 61 | export type HostFact = { label: string; value: string }; | |
| 62 | ||
| 63 | /** The declaration as label and value rows, in plain words, for the page, the terminal and MCP. */ | |
| 64 | export function hostFacts(t = HOST_TUNING): HostFact[] { | |
| 65 | return [ | |
| 66 | { | |
| 67 | label: "Checked against the host", | |
| 68 | value: t.confirmed ? "Yes" : "Not yet: these figures come from the issue, not from the host's own settings", | |
| 69 | }, | |
| 70 | { label: "Uploads", value: `Held to ${t.shaper.mbit} Mbit/s, just under the 16.3 Mbit/s the line carries` }, | |
| 71 | { label: "Sorting of traffic", value: "Four lanes by the mark on each packet; nothing in this repository sets the marks yet" }, | |
| 72 | { label: "Local network", value: t.shaper.lanUnshaped ? "Not slowed" : "Slowed like the rest" }, | |
| 73 | { | |
| 74 | label: "Large or slow responses", | |
| 75 | value: `Written to a separate log when over ${t.edge.logBytes.toLocaleString("en-US")} bytes or ${t.edge.logSeconds} s`, | |
| 76 | }, | |
| 77 | { label: "Crawlers", value: `Share one lane of ${t.edge.crawlerPerSecond} requests a second` }, | |
| 78 | ]; | |
| 79 | } |
src/system/ops_log.ts added +101
| @@ -0,0 +1,101 @@ | ||
| 1 | /** | |
| 2 | * The operations log (#14238): one line per change to how the forge runs, with who and why. | |
| 3 | * `docs/ops/operations.log` holds host and incident lines (a pull appends them); | |
| 4 | * settings changed on /settings/capacity are the forge's own `capacity.changed` events and are | |
| 5 | * merged in, so nothing is written twice. Parsing is pure; the bottleneck page, the MCP tool and | |
| 6 | * the morning paper read `readOps`. | |
| 7 | */ | |
| 8 | import type { Database } from "bun:sqlite"; | |
| 9 | import { existsSync, readFileSync, statSync } from "node:fs"; | |
| 10 | import { join } from "node:path"; | |
| 11 | ||
| 12 | export const OP_KINDS = ["setting", "host", "incident"] as const; | |
| 13 | export type OpKind = (typeof OP_KINDS)[number]; | |
| 14 | export type Op = { day: string; kind: OpKind; who: string; what: string; why: string }; | |
| 15 | ||
| 16 | const FILE = join(import.meta.dir, "..", "..", "docs", "ops", "operations.log"); | |
| 17 | const MAX_BYTES = 256 * 1024; | |
| 18 | ||
| 19 | /** Lines that do not parse are left out, never guessed at. */ | |
| 20 | export function parseOps(text: string): Op[] { | |
| 21 | const out: Op[] = []; | |
| 22 | for (const raw of text.split("\n")) { | |
| 23 | const line = raw.trim(); | |
| 24 | if (line === "" || line.startsWith("#")) continue; | |
| 25 | const [day, kind, who, what, ...rest] = line.split("|").map((s) => s.trim()); | |
| 26 | const why = rest.join(" | "); | |
| 27 | if (day === undefined || !/^\d{4}-\d{2}-\d{2}$/.test(day) || Number.isNaN(Date.parse(day))) continue; | |
| 28 | if (!(OP_KINDS as readonly string[]).includes(kind ?? "")) continue; | |
| 29 | if (!who || !what) continue; | |
| 30 | out.push({ day, kind: kind as OpKind, who, what, why }); | |
| 31 | } | |
| 32 | return out; | |
| 33 | } | |
| 34 | ||
| 35 | /** The last parsed file, kept until its path, mtime or size changes. */ | |
| 36 | let kept: { file: string; stamp: string; ops: Op[] } | null = null; | |
| 37 | ||
| 38 | export function readOpsFile(file = FILE): Op[] { | |
| 39 | try { | |
| 40 | if (!existsSync(file)) return []; | |
| 41 | const st = statSync(file); | |
| 42 | if (st.size > MAX_BYTES) return []; | |
| 43 | const stamp = `${st.mtimeMs}:${st.size}`; | |
| 44 | if (kept && kept.file === file && kept.stamp === stamp) return kept.ops; | |
| 45 | const ops = parseOps(readFileSync(file, "utf8")); | |
| 46 | kept = { file, stamp, ops }; | |
| 47 | return ops; | |
| 48 | } catch { | |
| 49 | return []; | |
| 50 | } | |
| 51 | } | |
| 52 | ||
| 53 | /** Setting changes the forge kept itself, as ops. */ | |
| 54 | export function settingOps(db: Database, limit = 50): Op[] { | |
| 55 | const rows = db | |
| 56 | .query<{ at: string; payload: string; name: string | null }, [number]>( | |
| 57 | "SELECT e.created_at AS at, e.payload AS payload, u.username AS name FROM event e LEFT JOIN user u ON u.id = e.actor_id WHERE e.kind = 'capacity.changed' ORDER BY e.id DESC LIMIT ?1", | |
| 58 | ) | |
| 59 | .all(limit); | |
| 60 | const out: Op[] = []; | |
| 61 | for (const r of rows) { | |
| 62 | try { | |
| 63 | const p = JSON.parse(r.payload) as { key?: unknown; from?: unknown; to?: unknown; reason?: unknown }; | |
| 64 | if (typeof p.key !== "string") continue; | |
| 65 | out.push({ | |
| 66 | day: r.at.slice(0, 10), | |
| 67 | kind: "setting", | |
| 68 | who: r.name ?? "unknown", | |
| 69 | what: `${p.key}: ${String(p.from ?? "default")} to ${String(p.to ?? "default")}`, | |
| 70 | why: typeof p.reason === "string" ? p.reason : "", | |
| 71 | }); | |
| 72 | } catch { | |
| 73 | /* a payload that is not ours is not a change */ | |
| 74 | } | |
| 75 | } | |
| 76 | return out; | |
| 77 | } | |
| 78 | ||
| 79 | /** | |
| 80 | * Newest first (on one day, the later entry first), the file's lines and the kept settings | |
| 81 | * together. | |
| 82 | */ | |
| 83 | export function readOps(db: Database, file = FILE): Op[] { | |
| 84 | let settings: Op[] = []; | |
| 85 | try { | |
| 86 | settings = settingOps(db); | |
| 87 | } catch { | |
| 88 | /* an old database without the event table's shape reads as no settings */ | |
| 89 | } | |
| 90 | // Both are oldest first, so a later index is a later entry. | |
| 91 | const all = [...readOpsFile(file), ...settings.reverse()]; | |
| 92 | return all.map((o, i) => ({ o, i })).sort((a, b) => (a.o.day < b.o.day ? 1 : a.o.day > b.o.day ? -1 : b.i - a.i)).map((x) => x.o); | |
| 93 | } | |
| 94 | ||
| 95 | export const opLine = (o: Op): string => `${o.day} ${o.kind}: ${o.what} (${o.who}${o.why === "" ? "" : `; ${o.why}`})`; | |
| 96 | ||
| 97 | /** Ops from `sinceMs` on (by UTC day), newest first. */ | |
| 98 | export function opsSince(ops: readonly Op[], sinceMs: number): Op[] { | |
| 99 | const day = new Date(sinceMs).toISOString().slice(0, 10); | |
| 100 | return ops.filter((o) => o.day >= day); | |
| 101 | } |
src/tui/screens/bottleneck.ts added +73
| @@ -0,0 +1,73 @@ | ||
| 1 | /** | |
| 2 | * Where the bottleneck is, in the terminal (#14238): the sentence, a line a row, what the host is | |
| 3 | * tuned to and the changes to how the forge runs, as /system/bottleneck reads them | |
| 4 | * (`readBottleneck`). Gated as the page is: a forge admin who is a person. Read-only. | |
| 5 | */ | |
| 6 | ||
| 7 | import { telemetryReader } from "../../actions/telemetry.ts"; | |
| 8 | import { readBottleneck, type Bottleneck } from "../../system/bottleneck.ts"; | |
| 9 | import { live } from "../../system/live.ts"; | |
| 10 | import type { Ctx, Screen } from "../app/app.ts"; | |
| 11 | import { binding } from "../app/keys.ts"; | |
| 12 | import type { Session } from "../session.ts"; | |
| 13 | import { style } from "../term/style.ts"; | |
| 14 | import { pad } from "../term/width.ts"; | |
| 15 | import { Scroll, drawLines, hitScroll } from "./common.ts"; | |
| 16 | import type { PaletteEntry } from "./entries.ts"; | |
| 17 | ||
| 18 | const mayRead = (session: Session): boolean => session.user?.kind !== "agent" && telemetryReader(session.app, session.user)?.kind === "owner"; | |
| 19 | ||
| 20 | /** The screen's lines; empty for anyone the gate refuses. */ | |
| 21 | export function bottleneckLines(session: Session, b?: Bottleneck): string[] { | |
| 22 | if (!mayRead(session)) return []; | |
| 23 | const r = b ?? readBottleneck(session.app.db, live().read()); | |
| 24 | const lines = [r.says, ""]; | |
| 25 | for (const row of r.rows) { | |
| 26 | const of = row.ceiling === null ? "" : ` of ${row.ceiling}`; | |
| 27 | lines.push(`${pad(row.label, 14)}${row.now}${of}${row.id === r.limited ? " (at its limit)" : ""}`); | |
| 28 | } | |
| 29 | lines.push("", "What the host is tuned to"); | |
| 30 | for (const f of r.host) lines.push(` ${pad(f.label, 26)}${f.value}`); | |
| 31 | lines.push("", "Changes to how the forge runs"); | |
| 32 | if (r.ops.length === 0) lines.push(" None kept yet."); | |
| 33 | for (const o of r.ops) { | |
| 34 | lines.push(` ${o.day} ${pad(o.kind, 9)}${o.what} (${o.who})`); | |
| 35 | if (o.why !== "") lines.push(`${pad("", 23)}${o.why}`); | |
| 36 | } | |
| 37 | return lines; | |
| 38 | } | |
| 39 | ||
| 40 | export function bottleneckScreen(session: Session, ctx: Ctx): Screen { | |
| 41 | const scroll = new Scroll(); | |
| 42 | let lines = bottleneckLines(session); | |
| 43 | return { | |
| 44 | title: "Where the bottleneck is", | |
| 45 | keys: [binding(["j", "k"], "scroll"), binding(["g"], "read again"), binding(["esc", "q"], "back")], | |
| 46 | render(area, s) { | |
| 47 | const c = ctx.theme.c; | |
| 48 | drawLines(s, area, c, scroll, lines.map((text) => [{ text, style: style(text.startsWith(" ") ? c.mutedFg : c.fg, c.bg) }])); | |
| 49 | hitScroll(ctx, area, scroll, lines.length); | |
| 50 | }, | |
| 51 | key(e) { | |
| 52 | if (scroll.key(e, lines.length)) return true; | |
| 53 | if (e.key === "g") { | |
| 54 | lines = bottleneckLines(session); | |
| 55 | return true; | |
| 56 | } | |
| 57 | if (e.key === "esc" || e.key === "q") { | |
| 58 | ctx.pop(); | |
| 59 | return true; | |
| 60 | } | |
| 61 | return false; | |
| 62 | }, | |
| 63 | }; | |
| 64 | } | |
| 65 | ||
| 66 | export const entry: PaletteEntry = { | |
| 67 | id: "bottleneck", | |
| 68 | title: "Where the bottleneck is", | |
| 69 | hint: "what limits the forge, what the host is tuned to, what changed", | |
| 70 | after: ["system"], | |
| 71 | offered: mayRead, | |
| 72 | run: (session, ctx) => ctx.push(bottleneckScreen(session, ctx)), | |
| 73 | }; |
src/tui/screens/entries.ts +2
| @@ -19,6 +19,7 @@ import { entry as audit } from "./audit.ts"; | ||
| 19 | 19 | import { entry as appTemplates } from "./app_templates.ts"; |
| 20 | 20 | import { entry as dotfiles } from "./dotfiles.ts"; |
| 21 | 21 | import { entry as devices } from "./devices.ts"; |
| 22 | import { entry as bottleneck } from "./bottleneck.ts"; | |
| 22 | 23 | import { entry as bets } from "./bets.ts"; |
| 23 | 24 | import { entry as bills } from "./bills.ts"; |
| 24 | 25 | import { entry as payments } from "./payments.ts"; |
| @@ -142,6 +143,7 @@ export const ENTRIES: readonly PaletteEntry[] = [ | ||
| 142 | 143 | bugs, |
| 143 | 144 | homePins, |
| 144 | 145 | system, |
| 146 | bottleneck, | |
| 145 | 147 | healthVoice, |
| 146 | 148 | loopStalls, |
| 147 | 149 | footing, |
src/web/pages/bottleneck.ts +13 −1
| @@ -8,6 +8,7 @@ import { notePhase, takeSessionTime } from "../../core/route_timing.ts"; | ||
| 8 | 8 | import { telemetryReader } from "../../actions/telemetry.ts"; |
| 9 | 9 | import type { Ctx } from "../../http/context.ts"; |
| 10 | 10 | import { readBottleneck, type Row } from "../../system/bottleneck.ts"; |
| 11 | import type { Op } from "../../system/ops_log.ts"; | |
| 11 | 12 | import { live } from "../../system/live.ts"; |
| 12 | 13 | import { notFoundPage, pageHead, signInFirst } from "../chrome.ts"; |
| 13 | 14 | import { html, type Html } from "../html.ts"; |
| @@ -21,6 +22,13 @@ function rowHtml(r: Row, limited: boolean): Html { | ||
| 21 | 22 | return statusRow(r.label, state, r.lever === null ? null : html`<a href="${r.lever.href}">${r.lever.label}</a>`, `bn-${r.id}`); |
| 22 | 23 | } |
| 23 | 24 | |
| 25 | const KIND_WORDS: Record<Op["kind"], string> = { setting: "Setting", host: "Host", incident: "Incident" }; | |
| 26 | ||
| 27 | function opRow(o: Op): Html { | |
| 28 | const why = o.why === "" ? html`` : html`<div class="small muted">${o.why}</div>`; | |
| 29 | return statusRow(`${o.day} · ${KIND_WORDS[o.kind]}`, html`${o.what}${why}`, html`<span class="small muted">${o.who}</span>`); | |
| 30 | } | |
| 31 | ||
| 24 | 32 | export async function bottleneckPage(ctx: Ctx): Promise<Response> { |
| 25 | 33 | notePhase(ctx.request, "session", takeSessionTime(ctx.request)); |
| 26 | 34 | if (!ctx.auth.user) return signInFirst(ctx); |
| @@ -32,7 +40,11 @@ export async function bottleneckPage(ctx: Ctx): Promise<Response> { | ||
| 32 | 40 | <p id="bn-says"><strong>${b.says}</strong></p> |
| 33 | 41 | <dl class="status-rows" id="bn-rows">${b.rows.map((r) => rowHtml(r, r.id === b.limited))}</dl> |
| 34 | 42 | <section class="settings-block"><h2>Not measured here</h2> |
| 35 | <ul>${b.unseen.map((u) => html`<li class="small">${u}</li>`)}</ul></section>`; | |
| 43 | <ul>${b.unseen.map((u) => html`<li class="small">${u}</li>`)}</ul></section> | |
| 44 | <section class="settings-block" id="bn-host"><h2>What the host is tuned to</h2> | |
| 45 | <dl class="status-rows">${b.host.map((f) => statusRow(f.label, html`${f.value}`))}</dl></section> | |
| 46 | <section class="settings-block" id="bn-ops"><h2>Changes to how the forge runs</h2> | |
| 47 | ${b.ops.length === 0 ? html`<p class="small muted">None kept yet.</p>` : html`<dl class="status-rows">${b.ops.map(opRow)}</dl>`}</section>`; | |
| 36 | 48 | const response = page({ ctx, title: "Bottleneck", body }); |
| 37 | 49 | notePhase(ctx.request, "render", performance.now() - rendering); |
| 38 | 50 | return response; |
test/bottleneck.test.ts +45 −1
| @@ -7,6 +7,10 @@ import { createApp, type App } from "../src/app.ts"; | ||
| 7 | 7 | import { createToken } from "../src/core/tokens.ts"; |
| 8 | 8 | import { createUser, getUserByName } from "../src/core/users.ts"; |
| 9 | 9 | import { serve, type Servers } from "../src/http/server.ts"; |
| 10 | import { createForgeApp, forgeCommands, Session } from "../src/tui/forge.ts"; | |
| 11 | import { bottleneckLines } from "../src/tui/screens/bottleneck.ts"; | |
| 12 | import { headlessTerminal } from "../src/tui/term/terminal.ts"; | |
| 13 | import type { User } from "../src/core/users.ts"; | |
| 10 | 14 | import { judge, lineCeiling, type Input } from "../src/system/bottleneck.ts"; |
| 11 | 15 | import type { SystemView } from "../src/system/view.ts"; |
| 12 | 16 | import { quietLogs } from "./helpers/globals.ts"; |
| @@ -118,6 +122,13 @@ describe("the page", () => { | ||
| 118 | 122 | expect(res.status).toBe(200); |
| 119 | 123 | const text = await res.text(); |
| 120 | 124 | expect(text).toContain('id="bn-says"'); |
| 125 | const host = text.slice(text.indexOf('id="bn-host"'), text.indexOf('id="bn-ops"')); | |
| 126 | expect(host).toContain("<dt>Checked against the host</dt>"); | |
| 127 | expect(host).toContain("Held to 9 Mbit/s"); | |
| 128 | expect(host).not.toMatch(/bun scripts|diffserv4|lanUnshaped|<li/); | |
| 129 | const ops = text.slice(text.indexOf('id="bn-ops"')); | |
| 130 | expect(ops).toContain("<dt>2026-11-15 · Host</dt>"); | |
| 131 | expect(ops).not.toContain("<li"); | |
| 121 | 132 | expect(text).toContain('href="/settings/capacity"'); |
| 122 | 133 | const out = await fetch(`${servers.url}/system/bottleneck`, { redirect: "manual" }); |
| 123 | 134 | expect(await out.text()).not.toContain("bn-says"); |
| @@ -148,6 +159,39 @@ describe("the page", () => { | ||
| 148 | 159 | expect(refused.content[0].text).toContain("forbidden"); |
| 149 | 160 | const ok = await call(alice.id); |
| 150 | 161 | expect(ok.isError).not.toBe(true); |
| 151 | expect(JSON.parse(ok.content[0].text).rows.map((r: any) => r.id)).toEqual(["line", "cpu", "checks", "crew", "memory", "disk"]); | |
| 162 | const reading = JSON.parse(ok.content[0].text); | |
| 163 | expect(reading.rows.map((r: any) => r.id)).toEqual(["line", "cpu", "checks", "crew", "memory", "disk"]); | |
| 164 | expect(reading.host[0]).toEqual({ label: "Checked against the host", value: expect.stringContaining("Not yet") }); | |
| 165 | expect(reading.host.map((f: any) => f.label)).toContain("Uploads"); | |
| 166 | expect(reading.ops.length).toBeGreaterThan(0); | |
| 167 | expect(Object.keys(reading.ops[0]).sort()).toEqual(["day", "kind", "what", "who", "why"]); | |
| 168 | }); | |
| 169 | test("a setting changed on /settings/capacity shows as a row, escaped, and in the terminal's lines", () => { | |
| 170 | app.db | |
| 171 | .query("INSERT INTO event (id, kind, actor_id, payload, created_at) VALUES ('ev-bn', 'capacity.changed', ?1, ?2, '2026-11-16T09:00:00Z')") | |
| 172 | .run(getUserByName(app.db, "alice")!.id, JSON.stringify({ key: "checks.concurrency", from: "4", to: "6", reason: "<b>runners</b>" })); | |
| 173 | const alice = getUserByName(app.db, "alice")!; | |
| 174 | const lines = bottleneckLines(new Session(app, { sshUsername: "alice", remoteAddress: "127.0.0.1", fingerprint: "SHA256:t", user: alice })); | |
| 175 | expect(lines).toContain("What the host is tuned to"); | |
| 176 | expect(lines.join("\n")).toContain("2026-11-16 setting checks.concurrency: 4 to 6 (alice)"); | |
| 177 | return fetch(`${servers.url}/system/bottleneck`, { headers: { cookie } }).then(async (res) => { | |
| 178 | const text = await res.text(); | |
| 179 | expect(text).toContain("<b>runners</b>"); | |
| 180 | expect(text).not.toContain("<b>runners"); | |
| 181 | }); | |
| 182 | }); | |
| 183 | test("the terminal door is the page's: an admin is offered it, a non-admin and nobody are not and read nothing", () => { | |
| 184 | const identity = (user?: User) => ({ sshUsername: user?.username ?? "git", remoteAddress: "127.0.0.1", fingerprint: "SHA256:t", user }); | |
| 185 | const doors = (user?: User): string[] => { | |
| 186 | const { app: tui, session } = createForgeApp(app, identity(user), { terminal: headlessTerminal(80, 24), exitHooks: false }); | |
| 187 | const ids = forgeCommands(session)(tui.ctx).map((c) => c.id); | |
| 188 | tui.stop?.(); | |
| 189 | return ids; | |
| 190 | }; | |
| 191 | expect(doors(getUserByName(app.db, "alice")!)).toContain("bottleneck"); | |
| 192 | for (const who of [getUserByName(app.db, "bob") ?? undefined, undefined]) { | |
| 193 | expect(bottleneckLines(new Session(app, identity(who)))).toEqual([]); | |
| 194 | expect(doors(who)).not.toContain("bottleneck"); | |
| 195 | } | |
| 152 | 196 | }); |
| 153 | 197 | }); |
test/ops-log.test.ts added +127
| @@ -0,0 +1,127 @@ | ||
| 1 | /** The operations log and the declared host tuning (#14238). */ | |
| 2 | import { afterEach, beforeEach, describe, expect, test } from "bun:test"; | |
| 3 | import { mkdtempSync, readFileSync, rmSync, utimesSync, writeFileSync } from "node:fs"; | |
| 4 | import { tmpdir } from "node:os"; | |
| 5 | import { join } from "node:path"; | |
| 6 | import { Database } from "bun:sqlite"; | |
| 7 | import { migrationFiles } from "../src/core/schema.ts"; | |
| 8 | import { migrate } from "../src/kernel/migrate.ts"; | |
| 9 | import { storyText } from "../src/crew/routine_report.ts"; | |
| 10 | import { edgeSnippet, hostFacts, HOST_TUNING, shaperCommands, validDevice } from "../src/system/host_tuning.ts"; | |
| 11 | import { opLine, opsSince, parseOps, readOps, readOpsFile, settingOps } from "../src/system/ops_log.ts"; | |
| 12 | import { judge, type Input } from "../src/system/bottleneck.ts"; | |
| 13 | ||
| 14 | describe("parseOps", () => { | |
| 15 | test("reads the three kinds and skips what does not parse", () => { | |
| 16 | const ops = parseOps(["# c", "2026-11-15 | host | owner | shaper | why | more", "bad | host | a | b | c", "2026-11-15 | nope | a | b | c", "2026-11-15 | release | a | b | c", "2026-11-14 | incident | a | | c", ""].join("\n")); | |
| 17 | expect(ops).toEqual([{ day: "2026-11-15", kind: "host", who: "owner", what: "shaper", why: "why | more" }]); | |
| 18 | }); | |
| 19 | test("the checked-in log parses line for line", () => { | |
| 20 | const text = readFileSync("docs/ops/operations.log", "utf8"); | |
| 21 | const lines = text.split("\n").filter((l) => l.trim() !== "" && !l.startsWith("#")); | |
| 22 | expect(lines.length).toBeGreaterThan(0); | |
| 23 | expect(readOpsFile().length).toBe(lines.length); | |
| 24 | }); | |
| 25 | test("a missing file reads as none", () => expect(readOpsFile("/nonexistent/x.log")).toEqual([])); | |
| 26 | test("opsSince keeps the day and after", () => { | |
| 27 | const ops = parseOps("2026-11-10 | host | a | old | x\n2026-11-15 | host | a | new | x"); | |
| 28 | expect(opsSince(ops, Date.parse("2026-11-15T03:00:00Z")).map((o) => o.what)).toEqual(["new"]); | |
| 29 | expect(opLine(ops[1]!)).toBe("2026-11-15 host: new (a; x)"); | |
| 30 | }); | |
| 31 | }); | |
| 32 | ||
| 33 | describe("reading the file and the settings together", () => { | |
| 34 | let dir: string; | |
| 35 | let db: Database; | |
| 36 | const event = (id: string, at: string, key: string, to: string) => | |
| 37 | db.query("INSERT INTO event (id, kind, actor_id, payload, created_at) VALUES (?1, 'capacity.changed', 'u1', ?2, ?3)").run(id, JSON.stringify({ key, from: "4", to, reason: "r" }), at); | |
| 38 | beforeEach(() => { | |
| 39 | dir = mkdtempSync(join(tmpdir(), "forge-ops-")); | |
| 40 | db = new Database(":memory:"); | |
| 41 | migrate(db, migrationFiles); | |
| 42 | db.query("INSERT INTO user (id, username, email, password_hash) VALUES ('u1', 'owner', '[email protected]', 'x')").run(); | |
| 43 | }); | |
| 44 | afterEach(() => { | |
| 45 | db.close(); | |
| 46 | rmSync(dir, { recursive: true, force: true }); | |
| 47 | }); | |
| 48 | ||
| 49 | test("a capacity.changed event is a setting line with its reason; other events are not", () => { | |
| 50 | event("e1", "2026-11-15T10:00:00Z", "checks.concurrency", "6"); | |
| 51 | db.query("INSERT INTO event (id, kind, actor_id, payload, created_at) VALUES ('e2', 'other', 'u1', '{}', '2026-11-15T10:00:00Z')").run(); | |
| 52 | expect(settingOps(db).map(opLine)).toEqual(["2026-11-15 setting: checks.concurrency: 4 to 6 (owner; r)"]); | |
| 53 | }); | |
| 54 | test("newest day first; on one day the later entry first, file lines before settings of that day", () => { | |
| 55 | const file = join(dir, "ops.log"); | |
| 56 | writeFileSync(file, "2026-11-10 | host | a | old | x\n2026-11-14 | host | a | first-file | x\n2026-11-14 | host | a | second-file | x\n"); | |
| 57 | event("e1", "2026-11-14T08:00:00Z", "k.one", "1"); | |
| 58 | event("e2", "2026-11-14T09:00:00Z", "k.two", "2"); | |
| 59 | event("e3", "2026-11-12T09:00:00Z", "k.three", "3"); | |
| 60 | expect(readOps(db, file).map((o) => o.what)).toEqual(["k.two: 4 to 2", "k.one: 4 to 1", "second-file", "first-file", "k.three: 4 to 3", "old"]); | |
| 61 | }); | |
| 62 | test("the parsed file is kept until it changes", () => { | |
| 63 | const file = join(dir, "ops.log"); | |
| 64 | writeFileSync(file, "2026-11-10 | host | a | one | x\n"); | |
| 65 | expect(readOpsFile(file).map((o) => o.what)).toEqual(["one"]); | |
| 66 | writeFileSync(file, "2026-11-10 | host | a | one | x\n2026-11-11 | host | a | two | x\n"); | |
| 67 | utimesSync(file, new Date(), new Date(Date.now() + 5000)); | |
| 68 | expect(readOpsFile(file).map((o) => o.what)).toEqual(["one", "two"]); | |
| 69 | expect(readOpsFile(file)).toBe(readOpsFile(file)); | |
| 70 | }); | |
| 71 | }); | |
| 72 | ||
| 73 | describe("host tuning", () => { | |
| 74 | test("the shaper is one argv list, and a device that is not a name gets nothing", () => { | |
| 75 | expect(shaperCommands("eth0")).toEqual([["tc", "qdisc", "replace", "dev", "eth0", "root", "cake", "bandwidth", "9mbit", "diffserv4", "nat", "wash"]]); | |
| 76 | for (const bad of ["", "eth0; rm -rf /", "-x", "a b", "$(x)", "x".repeat(16)]) { | |
| 77 | expect(validDevice(bad)).toBe(false); | |
| 78 | expect(shaperCommands(bad)).toEqual([]); | |
| 79 | } | |
| 80 | }); | |
| 81 | test("the edge snippet declares the loud log and the crawler lane", () => { | |
| 82 | const s = edgeSnippet(); | |
| 83 | expect(s).toContain("limit_req_zone"); | |
| 84 | expect(s).toContain("rate=2r/s"); | |
| 85 | expect(s).toContain("access_log /var/log/nginx/forge-loud.log combined if=$forge_loud;"); | |
| 86 | }); | |
| 87 | test("the facts say they are unconfirmed, claim no priority order, and carry no shell command", () => { | |
| 88 | const facts = hostFacts(); | |
| 89 | expect(facts[0]).toEqual({ label: "Checked against the host", value: expect.stringContaining("Not yet") }); | |
| 90 | const text = facts.map((f) => `${f.label} ${f.value}`).join("\n"); | |
| 91 | expect(text).toContain("Held to 9 Mbit/s"); | |
| 92 | expect(text).toContain("1,000,000 bytes"); | |
| 93 | expect(text).not.toMatch(/pages first|backups|bulk|bun scripts|tc qdisc/); | |
| 94 | expect(hostFacts({ ...HOST_TUNING, confirmed: true })[0]!.value).toBe("Yes"); | |
| 95 | }); | |
| 96 | test("the script prints the commands without applying", () => { | |
| 97 | const p = Bun.spawnSync(["bun", "scripts/host-tune.ts", "--dev", "eth0"]); | |
| 98 | expect(p.stdout.toString()).toContain("cake bandwidth 9mbit"); | |
| 99 | expect(Bun.spawnSync(["bun", "scripts/host-tune.ts", "--dev", "a;b"]).exitCode).toBe(2); | |
| 100 | expect(Bun.spawnSync(["bun", "scripts/host-tune.ts", "--edge"]).stdout.toString()).toContain("forge_crawlers"); | |
| 101 | }); | |
| 102 | }); | |
| 103 | ||
| 104 | describe("the reading", () => { | |
| 105 | const input = (over: Partial<Input> = {}): Input => ({ | |
| 106 | view: { at: "", seconds: 1, cpu: { busyPct: 0, cores: [], limitCores: 1, history: [] }, temps: [], hottestC: null, load: null, memory: null, disks: [], net: [], io: [], processes: 0, groups: [], duplicates: [] }, | |
| 107 | lineMbit: null, | |
| 108 | checks: { running: 0, slots: 1, queued: 0 }, | |
| 109 | crew: { running: 0, slots: 1, queued: 0 }, | |
| 110 | ...over, | |
| 111 | }); | |
| 112 | test("judge carries the host facts and the ops it is given, none when left out", () => { | |
| 113 | const ops = parseOps("2026-11-15 | host | owner | shaper | why"); | |
| 114 | const b = judge(input({ ops })); | |
| 115 | expect(b.host).toEqual(hostFacts()); | |
| 116 | expect(b.ops).toEqual(ops); | |
| 117 | expect(judge(input()).ops).toEqual([]); | |
| 118 | }); | |
| 119 | }); | |
| 120 | ||
| 121 | describe("the paper", () => { | |
| 122 | test("a change to how the forge runs is said once, with its newest line", () => { | |
| 123 | const f = { releases: [], pillarsMoved: [], ops: ["2026-11-15 host: shaper (owner; x)", "2026-11-15 host: edge (owner; y)"] }; | |
| 124 | expect(storyText(f)).toBe("2 changes to how it runs (host: shaper (owner; x))"); | |
| 125 | expect(storyText({ releases: [], pillarsMoved: [] })).toBe(""); | |
| 126 | }); | |
| 127 | }); |
test/palette-roster.test.ts +2 −1
| @@ -130,10 +130,11 @@ test("a site-admin person is offered the whole roster", () => { | ||
| 130 | 130 | ["pulls-to-land", "Pulls to land", "approved pulls, stale crew pulls, unproposed work"], |
| 131 | 131 | ["valves", "Valves", "the week's budget as a valve per repository; turn one and the crew obeys"], |
| 132 | 132 | ["waiting-for-me", "Waiting for me", "issues and pulls that wait for you"], |
| 133 | ["tell-undo", "Take a tell back", "a tell you sent in the last minute, while nobody has read it"], | |
| 133 | 134 | ["refused-asks", "Refused first asks", "the last 20 first asks turned away"], |
| 134 | 135 | ["disks", "Disks", "machines' disk readings: 30-day write rate and projected rated-TBW date"], |
| 135 | 136 | ["telemetry", "Telemetry", "which agents may read flow and system; enter flips the grant"], |
| 136 | ["tell-undo", "Take a tell back", "a tell you sent in the last minute, while nobody has read it"], | |
| 137 | ["bottleneck", "Where the bottleneck is", "what limits the forge, what the host is tuned to, what changed"], | |
| 137 | 138 | ["health-voice", "Health — spoken", "how the forge is, in two plain sentences"], |
| 138 | 139 | ["stalls", "Main-thread stalls", "the last hour's stall ms by route"], |
| 139 | 140 | ["footing", "What the forge stands on", "each piece on another machine, in the order it would break the forge"], |
test/talk-checkin-story.test.ts +12
| @@ -91,6 +91,18 @@ describe("storyFacts", () => { | ||
| 91 | 91 | }); |
| 92 | 92 | }); |
| 93 | 93 | |
| 94 | describe("storyFacts with changes to how the forge runs", () => { | |
| 95 | test("a kept setting change inside the window is in the story, and the story still has its pillars", async () => { | |
| 96 | app.db | |
| 97 | .query("INSERT INTO event (id, kind, actor_id, payload, created_at) VALUES ('ev-ops-1', 'capacity.changed', ?1, ?2, ?3)") | |
| 98 | .run(alice.id, JSON.stringify({ key: "checks.concurrency", from: "4", to: "6", reason: "runners" }), new Date(T - 60_000).toISOString()); | |
| 99 | const s = await storyFacts(app, T - H); | |
| 100 | expect(s.ops).toContain(`${new Date(T - 60_000).toISOString().slice(0, 10)} setting: checks.concurrency: 4 to 6 (alice; runners)`); | |
| 101 | expect(s.pillarsMoved).toEqual([{ pillar: "P0 Trust", merged: 1 }]); | |
| 102 | app.db.query("DELETE FROM event WHERE id = 'ev-ops-1'").run(); | |
| 103 | }); | |
| 104 | }); | |
| 105 | ||
| 94 | 106 | describe("runRoutine's window is the last check-in", () => { |
| 95 | 107 | test("the second run does not repeat the first's releases and pillars", async () => { |
| 96 | 108 | const first = new Date(T + 60_000); |
new above
offline