Developing the forge
This document is for people and agents who change the forge. It covers the tests, the design check, the type check and the gate that every push must pass.
bun run check is the gate. It is the same command the forge runs against every
push, so a green local run predicts the result. Documentation follows
WRITING.md. The principles behind the code are in
ARCHITECTURE.md.
Commands and the gate
Driving the forge from a shell — every forge command, its flags and one example: docs/CLI.md.
bun test # protocol-level + adversarial suite
bun scripts/design-lint.ts # no colour or off-scale font-size beyond docs/design/baseline.json, no class docs/DESIGN.md §5 does not name
bun scripts/cold-read.ts <url> [--repo owner/name] [--write [--allow-text-only]] [--file-issues N [--file-into owner/name]] # cold-read audit (#5882): a no-context model reads the top pages (signed in with FORGE_TOKEN, sent only to FORGE_URL's forge; the signed-in pages go to the model endpoint, so use a throwaway account) at 390 and 1440 px; needs ANTHROPIC_API_KEY
bun x tsc --noEmit # zero type errors is the standard
bun run check # exactly what the forge runs against every push
bun run check:record # the same, and rewrites scripts/test-times.json
The design check (test/design-lint.test.ts, run in the gate) fails a pull that
adds a colour literal or an off-scale font-size beyond
docs/design/baseline.json, or defines a class docs/DESIGN.md §5 does not
name; the message names the file or the class. The way out is to use a token
(var(--…)) or a component §5 names; a new component is named in its own
docs/design/areas/<area>.md. When a change lowers the counts, run bun scripts/design-lint.ts
--update and commit the baseline; when it deletes the last rule for a class
in the §5d register, strike the name there too. See docs/design/plan.md.
docs/DESIGN.md is a spine: each of its sections is its own file under
docs/design/sections/ (§5d is 5d-page-local-classes.md), linked from the
spine in order, and the check reads them in that order. A new section is a
new file and a new line in the spine.
bun run check is the gate: the same command forge.json declares, so what
it says here is what the deploy will say. It type-checks once, then runs the
test files scripts/gate.ts admits — every test/*.test.ts not in its
EXCLUDED — as parallel bun test shards
(scripts/test-shards.ts): one per CPU the process may use, no more than
one per 512 MiB of the memory its cgroup allows (16 on the 16-CPU, 8 GiB
check cell; the old ceiling of eight left half of it idle, #2038), and
FORGE_CHECK_SHARDS=2 turns that down on a small or busy host. Each
shard's output is written whole as it finishes; the log ends with every
failing test from every shard, or on green with one line — how many tests,
across how many files, in how long, then the slowest and fastest shard.
A shard of several files that goes red gets a second chance for its failing
files, first three only: each is run once alone, same flags and env. A file
that passes alone was red only beside others and no longer fails the check;
the shard's log (and the tail) carries test-shards: fails only beside
others: <file> (shard-mates: …). Only failed tests clear: if the report
holds an error (a file that threw while loading, an unhandled error, a
crash before the summary) nothing in that shard is retried, and a file that
fails alone stays red. --record, --impact and one-file-per-process runs
never retry. The cleared file is reported in the log only; filing it as an
issue is not built yet (#2480).
Two things keep a test's setup in milliseconds (#2038). bunfig.toml
preloads test/helpers/preload.ts, which hashes passwords at argon2id's
floor (8 KiB, one pass) instead of Bun's default 64 MiB, two passes:
making an account and signing it in was ~700 ms of a typical beforeAll
and is now ~15 ms; the forge itself never loads it. And openDb keeps
the empty database each migration list makes: the second fresh database
in a process is a copy of it rather than ninety-odd migrations again
(createApp ~100 ms → ~6 ms). No test file runs one subject long
enough to set the gate's floor alone: the studio's and the apps
adversarial suites are split by subject into test/studio*.test.ts and
test/apps-adversarial*.test.ts.
The check's first step is the gate-list audit (bun scripts/gate.ts), so an
EXCLUDED entry for a file that is gone, or one with no reason or issue,
fails in under a second,
before install-heavy work. Comment width is formatting, not review, and never
fails a check: bun scripts/gate.ts --fix rewraps every comment line over
100 columns in the files the pull changes (src/kernel/comment_wrap.ts;
--all for the whole tree) and then runs the same audit. A pull that skips
it gets one gate: note: line in its check log and lands as it is; the
nightly rewrap job (docs/features/nightly.md) proposes the files main
changed in the last two days, rewrapped, as one pull. tsc then runs --incremental with a 2 GB heap,
its build info in .forge-cache/ (made first with mkdir -p: tsc will not
create the directory, and a checkout with no restored cache failed on it,
#1881). It is TypeScript 7, the native compiler (a
Go binary, so the bound is GOMEMLIMIT=2GiB): on this tree a cold check is
~3 s and ~17 CPU-s where 5.9 took ~30 s and ~50 CPU-s (#1864). 5.9 stays as
the dev dependency typescript-api only because scripts/hollow.ts walks
its AST, which 7.0 has no stable JS API for. The forge restores
.forge-cache/ from the base's own green check and keeps it only from green checks of the default
branch (src/checks/cache.ts), so a pull's code never fills a cache another
check reads. The full gate also writes every file's time there, and the next
split weighs every file by measurement rather than the 41 the table named.
The fast tier picks by measurement. The full gate runs each file in a
bun test of its own under Bun's coverage (--impact; coverage is per
process, so a shard of many files would blur theirs together) and writes
the impact map: test file → the src/ files whose functions it called.
A file merely imported — its top level run, none of its functions — does
not count; that is the import graph again (#1794). Nor does what every
test process runs while it loads the suite (bunfig.toml's preload and
all it imports: a function in 893 files on 2026-11-14, which made every
entry 893 files long and the map 125 MB, more than the forge keeps, so no
check had one, #12178): the run first covers test/suite-loads.test.ts,
which only loads, and an entry is what a test runs beyond that. Bun's
coverage is by block, not by count, so a test calling again a function
loading already called shows nothing: the files loading runs a function
of are that test's own entry, and a change to one is judged by imports as
well as by coverage. The file is packed (packImpact: the files listed
once, a test's entry one string) and a run kept going writes it every
hundred files, the never-measured first, so a run cut short leaves what
it measured. Under the forge the map
is handed over as $FORGE_CHECK_IMPACT in the checkout: every check is
seeded with the one the last green default-branch run kept (under
the forge's data directory, checks/impact/, src/checks/impact-store.ts), and
only such a run, or the full gate on land, keeps a new one. By hand it is
scripts/test-impact.json. A diff intersected with it is the set of tests
the change reaches; the plan line says by coverage: N changed files reach
M tests; src/x.ts reaches none (a coverage gap). A measured test is
guessed by imports only against the changes the map cannot see; tests the
map has never seen fall back to the import-hop guess, and a change to
forge.json, package.json, scripts/ or src/kernel/** runs
everything — except a scan's allow-list (scripts/<scan>-allow.json),
which only that scan's script and test read
(test/check-allow-lists.test.ts holds each to it). With no map yet the
line says by imports (no impact map).
Kept results (#12178). A selected file whose inputs a default-branch
measuring run already proved green is not run again: the log says
kept from <commit>. The key is the Bun, the tooling files
(KEPT_TOOLING: the runner scripts, forge.json, the preload among
them), the test, the files its impact entry names and its imports. Only a
file green on its first try is kept (never one a retry made green), only
the default branch's run hands results back ($FORGE_CHECK_KEPT,
src/checks/kept-store.ts), a test that opens the tree is never kept, and
one kept file in ten is run anyway (sampled) until an audit exists.
A test that reads the whole tree — every test file, every migration, the
README — is reached by no diff, yet any file can break it, so every fast
tier runs it whatever else the diff holds (#7138) — a diff of only the
timing or impact table breaks no rule and runs none. Such a test says
@tree-scan in its header comment (src/checks/scans.ts, #10304); that
mark is the contract. The plain cases are found without it: a test whose
source, or a scripts/ module it imports, lists the repository (a Glob,
git ls-files, a readdir handed the root or a name bound to it). A
walk's parameter is not inferred, so a test that walks through one
declares itself. One the reading mistakes for a scan (it lists only its
own temp directories) says @not-a-tree-scan and why. RULE_TESTS in
src/checks/tiers.ts is the floor the forge reads without a tree; a new
scan needs no line there. Together the scans may cost half the fast
tier's budget (test/tree-scans.test.ts).
Write the rule with breaches() from test/helpers/rule.ts: one red test
per breach, named path:line: …, so the check's summary names the rule
and the line.
A fast tier red only on test files the pull did not change is asked of
main: each is run once more on the base's tree, and when it is red there
too with the same tests the log says … is red on main at <base> too: not
counted against this pull and the check is green
(src/checks/base_red.ts). The pull's check summary keeps it — check
passed in 40s; red on main at abcdef0, not this pull's: test/a.test.ts —
so the masked red is seen on the pull's page after the log is gone. Main's
red itself is recorded by main's own checks: the red-main hold
(src/checks/red_main.ts) and its vitals card, and the revert the forge
proposes (src/checks/revert_red.ts).
Before you push, ask what the fast tier would run for your change:
files=$(bun scripts/fast.ts) && bun test $files. bun scripts/fast.ts
[--base origin/main] [--why | --null | --json] lists the picked test files
for the diff from the base's merge-base to the working tree (committed,
uncommitted, untracked), one per line on stdout and nothing else there;
--why adds a tab and the reason, --null ends each path with NUL (for
xargs -0 -r bun test), --json is {full, files:[{file, why}]}. Exit 0:
files listed. Exit 3: the change reaches no test (stdout empty). Exit 2: the
diff cannot be bounded (forge.json, package.json, scripts/**…, said on
stderr as full tier: … changed), or the arguments or git failed. The
&& is what keeps an empty or unbounded answer from becoming a bare
bun test, which would run the whole suite.
Files go to shards by how long they take, not how big they are: longest
first onto the least-loaded shard, weighed from scripts/test-times.json
(file → seconds, from one recorded run). A file the table does not name
weighs its bytes at the table's own seconds-per-byte. If the table is
malformed, the log says it was refused and the split falls back to bytes.
A stale table only makes the split less even; it never drops a file.
Refresh it with one command after adding a slow test file, or when the
log's slowest shard is well over 1.5× its fastest:
bun run check:record # the gate's own command, with test-shards.ts --record
--record runs each file as its own bun test, with as many running at
once as the gate runs shards. It rewrites the table only when every file
is green. Files it did not run keep their old times.
Every file is independent (its own forge, its own database, port 0), so a test that fails only beside another is a bug in that test. Two things it will not do:
- It never borrows your git configuration. A test drives a real
git
against a forge it booted a second ago, and builds that child's
environment rather than inheriting yours (test/helpers/git-client.ts) —
otherwise a GIT_CONFIG_COUNT in your shell (a crew run sets one, holding
its own forge credential) would send that credential to the test's server
and every push in the suite would come back 401.
- It skips what this machine cannot run, out loud. The OpenSSH interop
tests need ssh and ssh-keygen; without them the log says SKIPPING:
and names the tool (test/helpers/tools.ts). Install OpenSSH and they run
again — the skip is conditional on the binary, never on anything else.
Every test/*.test.ts is in the gate unless EXCLUDED names it with a
one-line reason and the issue that owns admitting it, so a new test file is
gated the moment it exists and no pull edits a shared list for it (#2037).
test/check-list.test.ts goes red on an exclusion named twice, without a
reason or issue, or for a file that does not exist. An excluded file rejoins
the gate by deleting its EXCLUDED entry in a pull whose check is green and
whose description carries ten green runs in random order under load:
bun scripts/flake-hunt.ts admit test/x.test.ts runs the file ten times, each
its own bun test --randomize with a fresh seed beside busy loops, prints the
record and exits non-zero unless all ten are green.
Every gated file is held to a budget (#1242, scripts/test-budget.ts): the
file alone in its own bun test under 20 s, and each test in it under 2 s
(read from the JUnit report bun test writes beside its console output; a
beforeAll counts toward the file, not the test). Over is red on the whole
gate, and on the fast tier for a test file the pull itself changed, as a
(fail) line saying the fix; a slice run by hand names it and stays green.
A breach is timed once more alone, without coverage, and
judged by that run, so one busy moment on the box is not a red. A file that
cannot be brought under goes in SLOW in scripts/gate.ts with a reason and
the issue that owns it; bun scripts/gate.ts refuses a stale, doubled or
unowned entry. A green full gate writes each test's time to
.forge-cache/test-times-tests.json beside the per-file table, and its
summary names the five slowest tests and their share of the run, then
ends with every breach, slowest first, so a log kept by its tail still says
what to fix or name in SLOW.
A brand-new test file does not need that record: it joins the gate on its
own green check, not ten runs. That is accepted on purpose — a flaky new
test can turn the gate red once before anyone sees it — and the nightly
flake hunt below is the backstop that files it and quarantines it. One
shape is refused on sight, new file or not, because ten runs on a fast box
miss it: a wait that runs out of time and carries on — a loop that polls a
condition at most N times or until a deadline, and whose next statement
neither checks the outcome nor returns it (fallThroughWaits in
scripts/flake.ts). admit refuses such a file before any run, naming the
line, and test/flake.test.ts refuses it in every test file except the 47
that held one when the rule began (FALL_THROUGH_OWED, owned by #9025; the
rule goes red on a listed file that no longer breaches, so the list only
shrinks). The fix is one line after the loop: fail saying what never came.
How tui-backups and conversation-queue got in (#9017)
Neither passed an admission run; neither was ever given one. Both were new
files, so each joined the gate on its pull's own green check: no seed (the
gate runs every file's tests in written order, without --randomize), the
check cell's own load, one run.
test/tui-backups.test.tsjoined with #7278 (6e48d8c0). Its tests read
the destination an earlier test sealed, and its key check read the frames
every earlier test drew, so it was green only in written order. Run
through admit as it landed, it is red in 9 of 10 runs (seeds 2899264984
… 2895068531, the one green seed 3988233074); as #8995 left it, 0 of 10,
and refused first for its prompt wait. #9017 made each test make its own
destination and check every frame as it is drawn: 10 of 10 green.
test/conversation-queue.test.tsjoined with #8893 (65654ceb). Its
until polled 200 × 20 ms and then went on as if the condition had
come, so on a slow box the test stopped a turn before the forge had read
its partial answer. Under admit as it landed it was red once in 10
(seed 795951491), so one admission run could have passed it; it is now
refused before any run, at its line 48. As #8995 left it, it passes:
10 of 10 green.
No flaky tests. Nightly at 02:17 UTC the forge itself (src/nightly/,
a cell on a checkout of main) runs every gate file five times on a loaded box, each run a fresh seed that orders both
the files and (through --randomize) the tests inside them, nothing stopped
early. A file red in some runs and green in others is flaky: the hunt files
a Flaky: test/x.test.ts issue carrying the failure text, the seed and the
run, with the usual root causes to tick (a shared port, a real clock, order
dependence, a leaked server, a timeout set near the real duration, a tight
wall-clock bound); then
bun scripts/flake-hunt.ts quarantine moves every file an open Flaky:
issue the hunt itself filed (its author is the token's user and the forge
has not marked it untrusted) into EXCLUDED under that issue and proposes
the move as a pull; a file the issue names that no longer exists is skipped,
since its exclusion would turn the audit red. Nobody hand-edits the list. A file red in every run is broken,
not flaky, and the check already says so. The fix is the test's shape, never
a retry: test/flake.test.ts fails on --retry, --rerun-each, a bunfig
retry or a { retry: N } anywhere the gate runs, and on any test that
bounds a measured time under 200 ms (a loopback fetch on a loaded runner
takes tens of ms): assert relative to a baseline from the same run, or bound
only the failure the test names. The hunt takes the gate in
20 slices, one after another, each cut at 30 minutes, and ends by proposing the
quarantine as a pull on flake-hunt/<date>. Worst case, all 20 slices hit
their cut: 20 x 30 minutes, plus the install and the quarantine (10 each), so
about 10 h 20 min, ending near 12:40 UTC. Jobs run one at a time, so the
outside run (03:47) queues behind a long hunt and starts late; it still keeps
its own slot, which is what the record and the owed-on-waking rule read. A
stop (a release) aborts the running step at once. The forge needs FORGE_TOKEN (the
hunt's author) and cells; a preview runs none, and a forge without either says
so in its log and runs none. A job whose slot passed while the forge was down
runs on waking; one restarted mid-run is not run twice (nightly/<job>.json
in FORGE_APP_DATA). What each last did, and what it filed (filed #12,
closed #4, quarantined test/x.test.ts), is on Resilience and in the
terminal's Resilience dialog; the run's whole output is
/resilience/nightly/<job> (nightly/<job>.log), the run every issue it
files names, as the Forgejo jobs named their Actions run
(docs/features/nightly.md).
No unseen red outside the gate. Nightly at 03:47 UTC, the forge itself
runs bun scripts/flake-hunt.ts outside --file-issues: every file in
EXCLUDED (the complement of the gate, from the same list, so a file moved
into the gate is no longer run there), once, one at a time, with the gate's
60 s timeout. A file that newly fails opens one Red: test/x.test.ts issue
naming its first failing test; still red on another test, it updates that
issue; still red on the same test, it says nothing, so the owner's asks show
it once; passing again, it closes the issue. Same FORGE_TOKEN as the hunt.
The Forgejo workflows these replace, and sync-env, are covered in
docs/features/nightly.md.
Can the Forgejo jobs be switched off? Not yet. They go (#9020) once one
night of each side has been set side by side and agrees:
bun scripts/flake-compare.ts --forge hunt.log --forge outside.log --forgejo
slice-1.log … --forgejo outside.log reads what each side's run printed (the
forge's /resilience/nightly/<job>, with the job's Resilience line pasted in
so its quarantine pull on flake-hunt/<date> counts; each Forgejo step log)
and prints what each filed — Red: issues opened, updated or closed, Flaky:
issues, broken and quarantined files, the quarantine pull — then every
difference, naming the file; it exits 1 when there is one
(test/flake-compare.test.ts). One side silent where the other filed is the
dedupe, not a difference. A flaky file only one side found can be the seed;
a file red outside the gate on one side and green on the other is the cell
differing from the isolated runner, and blocks the switch until explained.
No such night has been compared yet: the logs are admin-only on both sides,
so an admin runs it and posts the output on #9045.
Bun's own --randomize --seed=N drives every hunt and admission run. Not
used: --rerun-each (one seed for every rerun, so ten reruns explore one
order) and --parallel (it changes the load the gate is judged under).
The gate also refuses hollow work. test/hollow.test.ts runs
scripts/hollow.ts, which names an exported function in src/actions/ that
nothing that runs reaches — no route's handler, MCP tool, crew worker, CLI
command or terminal screen, followed from where the program starts — and a
screen — a GET path with no parameter that draws a page — that no reachable
page links to. scripts/hollow-allow.ts lists what main
carries today, each entry under the issue that owns it; an entry that is no
longer hollow turns the check red, so the list only shrinks.
bun scripts/hollow.ts prints the same lines on their own.
Shards
Each shard gets its own
TMPDIR and FORGE_APP_DATA (nested under the cell's, when it set one),
removed when the shard ends.