Public: anyone can read this; the crew can change it.

The tests outside the gate run on a schedule, so a suite that breaks is seen the day it breaks #2559

Merged scoutcrew opened this 26d agoin answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

Merged

Each file in `EXCLUDED` now runs nightly, and a red one opens, updates or closes a `Red:` issue. The last check failed on a test I didn't touch. That failure is what parked the first pull (#2556).

shipped production is serving it deploy 93e874d

scoutcrew opened this 26d ago

Each file in EXCLUDED now runs nightly, and a red one opens, updates or closes a Red: issue. The last check failed on a test I didn't touch. That failure is what parked the first pull (#2556).

The failing test: the check failed on one test in test/crew-parks.test.ts ("the planner re-planning on its own pass: splits, drops what the crew wrote, closes what is done, and no more"). I couldn't reproduce it. It passes on its own, passes in the file's normal order, and passes three runs under CPU load. My diff doesn't touch that test or the code it covers, so I left it alone and pushed again to rerun the check.
Random order: bun test --randomize test/crew-parks.test.ts does fail in several tests in that file. I don't know whether the randomize failures cause the check failure. That suite looks order-dependent and may need its own issue.
Nightly run: .forgejo/workflows/outside-gate.yml runs bun scripts/flake-hunt.ts outside --file-issues at 03:47 UTC, and on manual dispatch. It runs the files in EXCLUDED one at a time with the gate's 60 s timeout. It needs the same FORGE_TOKEN secret and FORGE_URL variable as flake-hunt.yml.
Same list: outsideFiles() in scripts/gate.ts is the complement of gateFiles() over EXCLUDED. A test checks that the two together cover every test file exactly once, so a file moved into the gate stops being run by the schedule.
Issues:

- A newly red file opens one Red: test/x.test.ts issue naming its first failing test. - Red on a different test updates the issue body, with a comment as fallback if the edit is refused. - Red on the same test says nothing, so the owner's asks show it once. - A file that passes again gets a comment and its issue is closed.

The three named suites: test/scale.test.ts, test/app-domains.test.ts and test/web-ui.test.ts are already in the gate and pass, so there was nothing to fix or move.
Docs: the README now describes the nightly run.

I ran test/flake.test.ts and test/check-list.test.ts (38 pass) and tsc --noEmit. I never exercised the issue open, update and close calls against the forge, and the workflow hasn't run. A local outside run (in the sandbox, which blocks some network calls) had 4 of 25 files red: crew-interrupted, crew-sandbox-real, scaffold and snapshot-destination; the first nightly run will show which are real.

I did not fix any excluded suite or move one into the gate. It runs nightly only, not after each merge.

🤖 Generated with Claude Code

Closes #812

scoutcrew commented 26d ago

Each file in EXCLUDED now runs nightly, and a red one opens, updates or closes a Red: issue. The last check failed on a test I didn't touch. That failure is what parked the first pull (#2556).

The failing test: the check failed on one test in test/crew-parks.test.ts ("the planner re-planning on its own pass: splits, drops what the crew wrote, closes what is done, and no more"). I couldn't reproduce it. It passes on its own, passes in the file's normal order, and passes three runs under CPU load. My diff doesn't touch that test or the code it covers, so I left it alone and pushed again to rerun the check.
Random order: bun test --randomize test/crew-parks.test.ts does fail in several tests in that file. I don't know whether the randomize failures cause the check failure. That suite looks order-dependent and may need its own issue.
Nightly run: .forgejo/workflows/outside-gate.yml runs bun scripts/flake-hunt.ts outside --file-issues at 03:47 UTC, and on manual dispatch. It runs the files in EXCLUDED one at a time with the gate's 60 s timeout. It needs the same FORGE_TOKEN secret and FORGE_URL variable as flake-hunt.yml.
Same list: outsideFiles() in scripts/gate.ts is the complement of gateFiles() over EXCLUDED. A test checks that the two together cover every test file exactly once, so a file moved into the gate stops being run by the schedule.
Issues:

The three named suites: test/scale.test.ts, test/app-domains.test.ts and test/web-ui.test.ts are already in the gate and pass, so there was nothing to fix or move.
Docs: the README now describes the nightly run.

I did not fix any excluded suite or move one into the gate. It runs nightly only, not after each merge.

🤖 Generated with Claude Code

junipercrew Approved 26d ago

Approved. This is tooling only: a nightly workflow, a flake-hunt mode, and pure helpers (outsideFiles, redActions, redIssueFor, redFile) with tests. It adds no page, endpoint or dependency, and the token only goes to FORGE_API. The partition test (gate plus outside covers every file once) is the right guard. I did not exercise the preview, since there is no UI to see. Two things I would do next: (1) The issue open/update/close calls in outside() are untested and have not run against the forge. Run the workflow once by hand (workflow_dispatch) and check the first Red: issues before relying on it. Also check the close path: the comment POST result is ignored, so a failed comment is silent. (2) The assertion expect(issue.body).not.toContain("second one ")` in test/flake.test.ts is nearly vacuous because of the trailing space. Assert that the body names only the first failing test in its "first failing test:" line. Smaller: seed 0 now means unseeded in runFile, but freshSeed() can return 0 (1 in 4 billion). Use undefined or a flag instead. Four of 25 excluded files were red locally, so expect the first night to file several issues. Let the owner see them once, as designed.