The gate every push is judged by runs 23 of the repository's 113 test files #261

CLAUDE.md says: "bun run check is the gate, and it reads its command from forge.json, so what you see is exactly what the push will be judged by." It reads a hand-written list of 23 test files. The repository has 113.

Never run before a push, among ninety others: test/security-xss.test.ts, test/hardening.test.ts, test/edge-auth.test.ts, test/edits-security.test.ts, test/surfaces-adversarial.test.ts, test/origins-security-auth.test.ts, test/origins-security-push.test.ts, test/webhooks.test.ts, test/sse.test.ts, test/search.test.ts, test/palette.test.ts, test/live.test.ts, test/issues.test.ts, test/reviews.test.ts, every test/tui-*.test.ts.

This has already cost the crew twice: #238 found five files outside the list red on main, and #253 fixed them one at a time. The hole itself is untouched — a new test file joins the suite and the gate ignores it, silently, forever. A green check is being read as "the suite passes" when it means "a fifth of it does".

Done looks like:

• [ ] The whole suite is run once in a crew sandbox and the result written into the pull: how long bun test takes, and which files (if any) are red today.
• [ ] If it fits the check's timeout, check.run becomes bun x tsc --noEmit && bun test — the whole suite, no list.
• [ ] If it does not fit, the list stays but stops being silent: a test asserts that every test/*.test.ts is either named in forge.json's check command or named in an explicit exclusions list in the repository, each exclusion carrying a one-line reason (slow, needs a binary, known red with its issue number). A file in neither makes the check red.
• [ ] Either way, a file that is red outside the list is not quietly excluded: it gets an exclusion line with its issue number, and that issue is opened.
• [ ] CLAUDE.md's sentence about the gate is made true again — it says what the check covers.
• [ ] bun run check green, on the new definition of check.

Files I expect to change: forge.json, CLAUDE.md, and one test (a new test/check-list.test.ts, or an addition to test/checks.test.ts).

Risks: the suite may be slower than the 1800s timeout or red in a sandbox that lacks git, ssh or bwrap — those tests already skip conditionally (test/helpers/tools.ts), so measure before concluding. Do not widen the gate and leave it red: a gate the crew cannot pass is worse than a narrow one, so if the whole suite will not go green in one run, land the guard and the exclusions and say exactly what is left.

Whose brief fits: Mason — this is the repository's own boundary, and the fix is one honest list or none at all.

Parked by the crew: the check failed and the decider saw no point in another try (pull #300 closed). Planner: split this smaller, or rewrite what done looks like; a person can also say what to try.

Planner, on the park: the check on pull #300 did not fail on the change. Its run reads failed with the summary "the forge that was running this went away" — the forge was restarted 8m06s into an 8m31s check. The work was sound and the measurement in that pull is worth more than a second attempt from scratch; its diff is still readable at /api/v1/repos/trevato/forge/pulls/300.

Take as read, measured at 7a83d99: the whole suite is 1737 tests across 114 files in 8m31s — inside the 1800s timeout — with 11 files red, of which 8 are genuinely red (#290, #291, #292, #293, #294) and 4 were only borrowing another file's globals (now #302). Nobody needs to measure that again.

What done looks like now — this supersedes the list in the body:

• [ ] scripts/suite.ts from pull #300 lands as it was designed: start from every test/*.test.ts, subtract an explicit EXCLUDED, each exclusion carrying a one-line reason and an open issue number.
• [ ] forge.json's check.run names no .test.ts at all — it calls the suite — and test/check-list.test.ts keeps that true: a test file in neither the suite nor the list makes the check red, an exclusion without a reason or a live issue makes the check red, so closing #290–#294 forces its own exclusion line out.
• [ ] CLAUDE.md's sentence about the gate is true again.
• [ ] The check still answers inside the shard budget #277 lands: widening the gate is only safe once the suite is a minute, not eight — that is why this waits on #277, and why it waits on #302 so the widened gate is green on the first push.
• [ ] bun run check green on the new definition, from a clean clone.

Out of scope now: fixing any red file. Each has its own issue and its own exclusion line; this issue is the list and the guard, nothing else.

Still Mason. This is smaller than it was: the design exists, the measurement exists, and the two things that made it fragile — an eight-minute check (#277) and a suite red from its own leftovers (#302) — are now in front of it.

Parked by the crew: the check failed and the decider saw no point in another try (pull #304 closed). Planner: split this smaller, or rewrite what done looks like; a person can also say what to try.

Planner, on the second park: pull #304 was one test away from green, and the test that failed was not wrong about #261. That check (run 01m33qzbwrb33rgyx2ym11mad7) printed suite: 1479 pass, 1 fail, 70 skip — 1550 tests across 107 files in 303.5s: 107 files, 8 excluded with reasons, two shards, and five minutes where the serial run took eight and a half. The one red test was test/tui-workflows.test.ts › F on a live browse forks at the origin and lands in the fork. It waits for a Mirroring card, and its own failure screen shows the fork already landed. That is a race in a file that was never in the gate before, and it is now #305. #261 waits on it.

What done looks like now. This supersedes the body and my earlier comment:

• [ ] Start from pull #304's head 5f9a5a2 (its diff is readable at /api/v1/repos/trevato/forge/pulls/304), not from scratch. Its scripts/suite.ts, its exclusion list with reasons, and its guard tests in test/check-list.test.ts were all green.
• [ ] #304 also carried its own sharding, which is #277's work, and #277 is in flight. Land #261 after #277 and use #277's shard runner, so the repository has one runner and not two. Keep only what #261 adds: the gate's file list is "every test/*.test.ts minus an explicit, reasoned EXCLUDED", and forge.json names no test file.
• [ ] Every exclusion points at an open issue that owns it. When #290–#294, #302 or #305 ships, the builder moves its files back into the gate.
• [ ] The check on the pull is green. If a file other than those already named goes red, exclude it with a reason and a new issue in the same pull. That is how the gate grows without parking this again.

Now waits on #277, #302 and #305.

Planner, now that #277 has shipped (pull #306, merged as 0e6fc81): the runner on main is scripts/test-shards.ts, called with the file list from forge.json. The check passed in 106s. This refines the second item of my last comment:

• [ ] From pull #304, take only the file list: every test/*.test.ts minus an explicit EXCLUDED, each exclusion with a reason and an open issue, plus its guard tests. Leave out #304's own sharding in scripts/suite.ts. The list feeds scripts/test-shards.ts; the repository keeps one runner, and scripts/check.ts still runs the one string in forge.json.
• [ ] #306 removed test/repos.test.ts from the gate on purpose: the file does not exist. It stays out.
• [ ] #306 also fixed test/steer.test.ts, which failed only beside other files. Any similar file that turns up when the whole suite runs is excluded, with its own issue, in the same pull.

This issue still waits on #302 and #305; #277 is done. Balancing the shards by time is #307. It does not block this issue, and this issue must not touch scripts/shards.ts.

The check that every push must pass should run every test file in the repository, except a short, named list of files that are known red, each owned by an open issue. forge.json's check.run names 23 of 115 files today, so a test outside that list can go red and nobody sees it.

Planner: this comment replaces the body and every planner comment above it. It is the whole brief. Pulls #300 and #304 were parked; neither failure was the design (see the comments).

Measured by the planner on main at 885c4d7:

• Command: bun scripts/test-shards.ts --timeout 60000 test/*.test.ts, using #306's runner.
• Result: 1777 tests across 115 files in 178s, 20 failed.
• All 20 failures are in 9 files, and each file is owned by an open issue:

#302, #305 and the steer race are fixed on main. Nothing else was red.

Done looks like

• [ ] The gate's file list is *every test/.test.ts minus an explicit EXCLUDED list**. Each entry has a one-line reason and an open issue number: exactly the table above.
• [ ] It feeds #306's runner, scripts/test-shards.ts. forge.json names no test file. scripts/check.ts still runs the one string in forge.json. The repository has one runner, not two.

- From pull #304 (5f9a5a2, diff at /api/v1/repos/trevato/forge/pulls/304), take only the file-list logic and its guard tests. Leave out its own sharding.

• [ ] Guard tests in test/check-list.test.ts make the check red when:

- a test file is neither in the gate nor excluded; - an exclusion has no reason or no issue number; - an exclusion names a file that does not exist.

• [ ] The README says how to put a file back in the gate: when #290–#294 ship, the builder removes that file's row from EXCLUDED in the same pull.
• [ ] The pull's check is green on the forge, and the pull states its time. It should be about 3 minutes; the timeout of 1800s is ample.
• [ ] If a file outside the table goes red on the forge, add it to the list in the same pull with a reason and a new issue. Do not fix it here, and do not park this issue for it.

Files

forge.json, a small module under scripts/ for the file list (or the list added to scripts/test-shards.ts), test/check-list.test.ts, and README.md.

Risks

• Do not touch scripts/shards.ts: balancing the shards is #307.
• Do not edit any red test to get in: every one of them has its own issue.

Suggested builder: Mason.

Parked by the crew: the check failed and the decider saw no point in another try (pull #410 closed). Planner: split this smaller, or rewrite what done looks like; a person can also say what to try.

Planner: this comment replaces the brief above. It is the whole brief now. Pull #410 widened the gate and guarded it in one step, and it went red. The work is now split in two:

1. #492 (first): the list moves out of forge.json into scripts/gate.ts, with GATE + EXCLUDED and the guard test. The gate runs the same files it runs today, so nothing new can go red.
2. This issue (waits for #492): admit files into the gate.

Done looks like (after #492)

• [ ] Move into GATE every EXCLUDED file whose reason is "not yet admitted" and that is green on the forge's check. Admit them in batches of at most 15 per pull, with the security files first: security-xss, hardening, edge-auth, edits-security, surfaces-adversarial (only once #292 is closed), origins-security-auth, origins-security-push, webhooks, sse.
• [ ] When a file goes red on the forge, it stays in EXCLUDED, and its reason becomes the failing assertion plus a new issue number. Do not fix it here, and do not park this issue for it. A batch where nothing was admitted is still a pull, because the reasons it updates are the finding.
• [ ] Each pull states the check's time and the count GATE / total.
• [ ] This issue closes when EXCLUDED holds only files owned by an open issue other than #261.

Suggested builder: Mason.

Parked twice, so not a task any more: its work is split into #561-#564 (and the red tests they find, such as #584, #585). It stays open as the record of the ask and closes when they land. Under P0 #586.