Flake hunt and outside gate run from the forge on their schedule and file what the Forgejo jobs filed #9029

The flake hunt and the outside-gate run already ran from the forge's own timers in a cell (#8957). This pull makes them report the way the Forgejo jobs did. Before it, a Flaky: or Red: issue named no run (FORGE_RUN_URL was the repository's address), the run's output existed only in the process log, and the record never said what was filed. Without those, one forge night could not be compared with one Forgejo night.

What changed

• src/nightly/output.ts (new): each job's newest output is kept in FORGE_APP_DATA/nightly/<job>.log. Every step's command and what it printed go there, along with why a run did not start. The next run replaces it, and it is cut at 1 MB with a line saying so. filedIn reads what was filed (filed #12, updated #7, closed #4, quarantined test/x.test.ts) from the lines scripts/flake-hunt.ts already prints.
• src/nightly/run.ts: FORGE_RUN_URL is now <forge>/resilience/nightly/<job>, so every issue the hunt files links to its run, as the Actions run link did before. Ran gains filed.
• src/nightly/service.ts: the record now says what was filed, e.g. every step finished; filed #12, closed #4. or filed nothing.
• GET /resilience/nightly/:job (src/web/pages/resilience.ts, registered in src/http/server.ts): shows the job's record and its output, escaped, to admins only, like /resilience. Anyone else, or a job name that doesn't exist, gets not-found. Each nightly line on Resilience links to it.
• Docs: docs/features/nightly.md gets "Comparing one night": what to set side by side and what counts as agreeing. It also states a limit: the two sides only avoid filing the same issue twice when both tokens act as the same user. The Forgejo hunt's issues were filed as Mason. docs/DEVELOP.md now says where the output and the "what it filed" line are.

Tests

• New test/nightly-output.test.ts covers:

- reading what was filed, checked against the strings the script really prints; - the output being replaced each run and cut at 1 MB; - the record's sentence; - the page: escaped output for an admin, links from Resilience, a job that hasn't run yet, and not-found for non-admins and for an unknown job.

• test/nightly.test.ts fixtures gain filed: [] because Ran has a new field; no assertion changed.
• test/security-floor-routes.test.ts lists the new route in UNSEEDED, with the reason, as every route with a parameter the seed can't fill is listed.

Checks run: bun scripts/gate.ts, bun x tsc --noEmit and bun scripts/design-lint.ts are clean. The nightly, resilience, security-floor, router, admin-gate, anonymous and timing-sweep test files pass (54 tests). scripts/fast.ts named 1294 files because the change touches server.ts, so I picked those files instead.

Not done

• The one-night comparison (the second "Done when" item) is still open. My sandbox can't reach Forgejo, and I can't read production's FORGE_APP_DATA. No Red: issue has ever been filed, and the newest Flaky: issues are from Oct 16, so there was nothing filed to compare yet. After the next 02:17 and 03:47 UTC slots, an admin can compare /resilience/nightly/flake-hunt and /resilience/nightly/outside-gate with that night's Actions runs, following the new doc section, and post the result here.
• Whose token production uses: I couldn't check that production has FORGE_TOKEN and working cells. Without either, the jobs stay off and the log says so. I also couldn't check whether that token acts as the same user as the mirror's secret.
• Deleting the workflows is #9020's.
• I couldn't keep the plan on the issue: there is no keep_plan tool in this run.

To try it: as an admin, open /resilience and follow "Output" on either nightly line, or run bun test test/nightly-output.test.ts.

🤖 Generated with Claude Code

Closes #9019

The flake hunt and the outside-gate run already ran from the forge's own timers in a cell (#8957). This pull makes them report the way the Forgejo jobs did. Before it, a Flaky: or Red: issue named no run (FORGE_RUN_URL was the repository's address), the run's output existed only in the process log, and the record never said what was filed. Without those, one forge night could not be compared with one Forgejo night.

What changed

• src/nightly/output.ts (new): each job's newest output is kept in FORGE_APP_DATA/nightly/<job>.log. Every step's command and what it printed go there, along with why a run did not start. The next run replaces it, and it is cut at 1 MB with a line saying so. filedIn reads what was filed (filed #12, updated #7, closed #4, quarantined test/x.test.ts) from the lines scripts/flake-hunt.ts already prints.
• src/nightly/run.ts: FORGE_RUN_URL is now <forge>/resilience/nightly/<job>, so every issue the hunt files links to its run, as the Actions run link did before. Ran gains filed.
• src/nightly/service.ts: the record now says what was filed, e.g. every step finished; filed #12, closed #4. or filed nothing.
• GET /resilience/nightly/:job (src/web/pages/resilience.ts, registered in src/http/server.ts): shows the job's record and its output, escaped, to admins only, like /resilience. Anyone else, or a job name that doesn't exist, gets not-found. Each nightly line on Resilience links to it.
• Docs: docs/features/nightly.md gets "Comparing one night": what to set side by side and what counts as agreeing. It also states a limit: the two sides only avoid filing the same issue twice when both tokens act as the same user. The Forgejo hunt's issues were filed as Mason. docs/DEVELOP.md now says where the output and the "what it filed" line are.

Tests

• New test/nightly-output.test.ts covers:

- reading what was filed, checked against the strings the script really prints; - the output being replaced each run and cut at 1 MB; - the record's sentence; - the page: escaped output for an admin, links from Resilience, a job that hasn't run yet, and not-found for non-admins and for an unknown job.

• test/nightly.test.ts fixtures gain filed: [] because Ran has a new field; no assertion changed.
• test/security-floor-routes.test.ts lists the new route in UNSEEDED, with the reason, as every route with a parameter the seed can't fill is listed.

Checks run: bun scripts/gate.ts, bun x tsc --noEmit and bun scripts/design-lint.ts are clean. The nightly, resilience, security-floor, router, admin-gate, anonymous and timing-sweep test files pass (54 tests). scripts/fast.ts named 1294 files because the change touches server.ts, so I picked those files instead.

Not done

• The one-night comparison (the second "Done when" item) is still open. My sandbox can't reach Forgejo, and I can't read production's FORGE_APP_DATA. No Red: issue has ever been filed, and the newest Flaky: issues are from Oct 16, so there was nothing filed to compare yet. After the next 02:17 and 03:47 UTC slots, an admin can compare /resilience/nightly/flake-hunt and /resilience/nightly/outside-gate with that night's Actions runs, following the new doc section, and post the result here.
• Whose token production uses: I couldn't check that production has FORGE_TOKEN and working cells. Without either, the jobs stay off and the log says so. I also couldn't check whether that token acts as the same user as the mirror's secret.
• Deleting the workflows is #9020's.
• I couldn't keep the plan on the issue: there is no keep_plan tool in this run.

To try it: as an admin, open /resilience and follow "Output" on either nightly line, or run bun test test/nightly-output.test.ts.

🤖 Generated with Claude Code

The flake hunt and the outside-gate run already ran on the forge's timers in a cell (#8957). This pull makes them report the way the Forgejo jobs did: each run's output is kept, the issues it files link to that output, and the run's record says what it filed. The check's gate failed because no commit had a Try: line. The new commit 28b5627e adds one and changes no files.

What changed

• src/nightly/output.ts (new): each job's newest output is saved to FORGE_APP_DATA/nightly/<job>.log. It holds every step's command, what it printed, and why a run did not start. The next run replaces it, and it is cut at 1 MB with a line saying so. filedIn reads what was filed (filed #12, updated #7, closed #4, quarantined test/x.test.ts) from the lines scripts/flake-hunt.ts already prints.
• src/nightly/run.ts: FORGE_RUN_URL was the repository's address. It is now <forge>/resilience/nightly/<job>, so every Flaky: or Red: issue links to its run, as the Forgejo Actions run link did. Ran gains a filed field.
• src/nightly/service.ts: the record now says what the run filed, e.g. every step finished; filed #12, closed #4., or filed nothing.
• GET /resilience/nightly/:job (src/web/pages/resilience.ts, registered in src/http/server.ts): shows the job's record and its output, escaped, to admins only, like /resilience. Anyone else, and any name that is not a job, gets not-found. Each nightly line on Resilience links to it.
• Docs: docs/features/nightly.md gains "Comparing one night", which lists what to set side by side and what counts as agreeing. It also notes that the forge and Forgejo only avoid filing the same issue twice when both tokens act as the same user; the Forgejo hunt's issues were filed as Mason. docs/DEVELOP.md says where the output and the line saying what was filed are.

Tests

• New test/nightly-output.test.ts covers:

- reading what was filed, checked against the strings the script really prints; - the output being replaced by each run and cut at 1 MB; - the record's sentence; - the page: escaped output for an admin, the links from Resilience, a job that has not run yet, and not-found for non-admins and unknown jobs.

• test/nightly.test.ts: fixtures gain filed: [] because Ran has a new field. No assertion changed.
• test/security-floor-routes.test.ts: the new route is added to UNSEEDED with its reason, as the test requires for a route parameter it cannot fill.

Checks run here: the gate passes against the merge-base with origin/main. tsc and design-lint passed on the first push, and no file has changed since. Locally, the nightly, nightly-output and security-floor-routes tests pass. The resilience, router, admin-gate, anonymous and timing-sweep tests passed before the empty commit. scripts/fast.ts named 1294 files because server.ts changed, so I picked the test files that cover the change instead.

Not done

• The one-night comparison (the second "Done when" item) is still open. My sandbox cannot reach Forgejo or read production's FORGE_APP_DATA. There was also nothing filed to compare: no Red: issue has ever been filed, and the newest Flaky: issues are from Oct 16. After the next 02:17 and 03:47 UTC slots, an admin can compare the two output pages with that night's Actions runs, following the new doc section.
• Production's setup: I could not confirm that production has FORGE_TOKEN and working cells; without either the jobs stay off and the log says so. I also could not confirm that its token acts as the same user as the mirror's secret.
• Deleting the workflows belongs to #9020.
• The plan on the issue: I could not keep it there, because there was no keep_plan tool in this run.

Try: as an admin on the preview, open /resilience and follow "Output" on the flake-hunt or outside-gate line. A preview runs no nightly job, so the page says "No output here yet". /resilience/nightly/sync-env is not found.

🤖 Generated with Claude Code

Each nightly job (the flake hunt and the outside gate) now keeps its newest output under FORGE_APP_DATA/nightly/<job>.log. It is shown to admins at /resilience/nightly/<job>, and the Flaky: and Red: issues a job files now point to that page through FORGE_RUN_URL instead of the repository. A run's record also lists what it filed (filed #12, quarantined test/x.test.ts), read from what scripts/flake-hunt.ts prints. The output is capped at 1 MB, each run replaces the last, and every line is escaped. docs/DEVELOP.md and docs/features/nightly.md say where a night's output is and how to compare it with the same night on Forgejo.

The last commit fixes the two tests that failed the previous check. Neither failure came from the nightly code:

• test/mcp-write-scope.test.ts: the test counted rows in every table before and after the refused writes. On a slow machine, the forge's counts of the test's own requests (route_timing, usage_count) are flushed on a timer mid-test, which looked like a write. Slowing the test down on purpose made it fail every time. Those two tables are now left out of the count, as test/write-clock-quiet.test.ts already does. The event log is still compared row for row, so "nothing was written" is still tested.
• test/oauth-connect-browser.test.ts: it timed out at its 15 s limit on a heavily loaded machine. The check log was cut short, so the exact step that stalled can't be confirmed. In test/helpers/chrome.ts, a click now retries until the page has the element to click, where before a click on a slow page failed silently and the test waited out the clock. This test now allows 30 s for its two page loads.

Neither test's assertions changed. These tests pass locally: the MCP test (also slowed on purpose), the nightly tests, bun scripts/gate.ts and bun x tsc --noEmit. The browser test could not run here because Chromium cannot start in this sandbox, so the forge's check is the first real run of that change.

To try it on the preview, sign in as an admin, open /resilience and follow a job's Output link. Before any night has run, the page says it has no output yet.

🤖 Generated with Claude Code

This round makes all five changes from review.

1. The terminal can read a run's output. In the Resilience dialog (src/tui/screens/resilience.ts), o opens a "Nightly output" dialog. For each job it shows the job's line, then the last 12 lines of its newest output, read through outputTail in src/nightly/output.ts. outputTail reads at most the last 16 KB of the file, so a 1 MB output is never loaded whole. It uses the same gate as /resilience/nightly/<job>: o is offered and works only for a site admin. test/tui-nightly-output.test.ts checks that an admin sees the tail and not the lines before it, that a job that never ran says so, and that someone who isn't an admin gets no o and nothing opens.
2. Scope. The changes to test/helpers/chrome.ts, test/mcp-write-scope.test.ts and test/oauth-connect-browser.test.ts are taken back out (a487d759). They are filed as #9031 with the fix that was tried, b55f16c6, and a note that the retry loop needs a test that fails without it. They will go in their own pull.
3. Docs. In docs/features/nightly.md, the sync-env.yml paragraph is back on its own under "What the Forgejo workflows were", before "Comparing one night". "Seeing it" now also names the terminal's o key and says the filed line is advisory.
4. Filed line. readsFiled(step) limits filedIn to steps that run bun scripts/flake-hunt.ts, so the install step is never read. A test checks this against JOBS. The hunt and outside modes already capture the output of the tests they run instead of printing it, so a test cannot print a line into the record. The module comment and the docs still call the record advisory, because it is what the default branch's script says it did; the issues themselves are the truth.
5. Writes off the request path. startOutput now returns { write, end }. Lines are buffered and appended asynchronously, in order, once 64 KB is waiting or a second has passed. runJob waits on end() in a finally, so every way out of a run leaves the file whole. A test checks that 100 lines cause no write until end(), then arrive in order.

What to look at: the runJob/runSteps split in src/nightly/run.ts and the flush queue in startOutput. If writing to the disk fails, output is dropped quietly and the run carries on; that is deliberate.

Checks run: bun scripts/gate.ts, bun x tsc --noEmit and bun scripts/design-lint.ts are clean. bun scripts/fast.ts named 1300 files because src/http/server.ts changed, so I ran the ten files that cover this change: the nightly, terminal-nightly-output, terminal-resilience, resilience-page and security-floor tests. 61 pass. The sandbox has no git binary, so one log line reports posix_spawn 'git' ENOENT; no test failed on it. Because the flake fixes are out again, the check may hit those two flaky tests until #9031 lands.

To try it on the preview, sign in as an admin and open /resilience/nightly/outside-gate, or press o in the terminal's Resilience dialog. Until a job has run on the forge, both say there is no output yet.

🤖 Generated with Claude Code

Left #9019 open: 1 unticked item remain (One night's output from the forge and from Forgejo is compared on the pull.).