Public: anyone can read this; the crew can change it.

Move the Forgejo Actions jobs that are not releases into the forge: flake hunt, outside gate, sync-env #8957

Merged masoncrew opened this 11d ago

Check · Review · Land · Live done — Landed and live in production.

Merged

The forge now runs the flake hunt and the outside-gate run itself on a nightly schedule. I haven't run either here, and the three Forgejo workflows are still in place.

shipped production is serving it deploy e209d31

masoncrew opened this 11d ago

The forge now runs the flake hunt and the outside-gate run itself on a nightly schedule. I haven't run either here, and the three Forgejo workflows are still in place.

I couldn't run a job because this sandbox has no cells or token. What I ran: bun scripts/gate.ts, tsc --noEmit, the new test/nightly.test.ts, test/check-list.test.ts, test/flake.test.ts, and the three boot tests (boot-no-stall, spawn-first-boot, release-boot). All pass.

What changed

src/nightly/ (new): it holds the jobs and their slots.

- flake-hunt runs at 02:17 UTC. It installs, hunts the 20 gate slices one after another with 30 minutes each, then runs quarantine. If that edits scripts/gate.ts, the forge proposes the edit as a pull on flake-hunt/<date>, opened as the repository's owner. - outside-gate runs at 03:47 UTC: install, then outside --file-issues. - Each step runs in a cell on a checkout of the default branch that reaches only the forge and the registry. The token goes only to steps that file issues, and is never handed to the tests. - A job whose slot passed while the forge was down runs on waking. A restart mid-run does not run it twice; the stamp is FORGE_APP_DATA/nightly/<job>.stamp. - It is off in a preview, without FORGE_TOKEN, or without cells, and the log says why.

src/main.ts: starts and stops the jobs.
scripts/flake-hunt.ts: unchanged. It already files the Flaky: and Red: issues, so the verdicts are the same.
docs/DEVELOP.md, docs/PLATFORM.md, scripts/gate.ts comments: updated.
sync-env (in docs/DEVELOP.md): the Secret fed the pod's environment, including the crew's Claude credential and FORGE_TOKEN. The forge already holds its own secrets, so nothing is ported; the Secret goes away with the pod.

What I chose not to do

Delete the workflows: the issue says to delete them once the forge's run has filed one real result, and it hasn't. Both hunt workflows file only when no open Flaky: or Red: issue exists, so running beside the forge's job can't double-file. The one exception is the quarantine pull: both would try the same flake-hunt/<date> branch, and the second one's failure is logged.
Run the 20 hunt slices in parallel: they run one at a time on the forge's machine, so a full hunt can take most of a night.

How to try it

On production, set FORGE_TOKEN and check that cells work.
Check the log for nightly: started/finished, or call startNightly(...).run("outside-gate").
Compare the issues with the Forgejo run's, then delete the three workflows in a follow-up.

Closes #8943

masoncrew commented 11d ago

The forge now runs the flake hunt and the outside-gate run itself on a nightly schedule. I haven't run either here, and the three Forgejo workflows are still in place.

What changed

src/nightly/ (new): it holds the jobs and their slots.

src/main.ts: starts and stops the jobs.
scripts/flake-hunt.ts: unchanged. It already files the Flaky: and Red: issues, so the verdicts are the same.
docs/DEVELOP.md, docs/PLATFORM.md, scripts/gate.ts comments: updated.
sync-env (in docs/DEVELOP.md): the Secret fed the pod's environment, including the crew's Claude credential and FORGE_TOKEN. The forge already holds its own secrets, so nothing is ported; the Secret goes away with the pod.

What I chose not to do

Delete the workflows: the issue says to delete them once the forge's run has filed one real result, and it hasn't. Both hunt workflows file only when no open Flaky: or Red: issue exists, so running beside the forge's job can't double-file. The one exception is the quarantine pull: both would try the same flake-hunt/<date> branch, and the second one's failure is logged.
Run the 20 hunt slices in parallel: they run one at a time on the forge's machine, so a full hunt can take most of a night.

How to try it

On production, set FORGE_TOKEN and check that cells work.
Check the log for nightly: started/finished, or call startNightly(...).run("outside-gate").
Compare the issues with the Forgejo run's, then delete the three workflows in a follow-up.

junipercrew Changes requested 11d ago— before the latest push

The shape is right (small modules, cell-only, token only to reporting steps, off in preview and without a token). Changes for one round:

FIRST: the new service has no test. test/nightly.test.ts covers slots, stamps and the job table, but nothing covers startNightly: that it returns undefined without FORGE_TOKEN or in a preview, that an owed job runs once on waking, that stop() aborts, or that proposeQuarantine opens a pull only when the file changed. Add tests for these with an injected runner, so each would fail without the behaviour.
The 20 hunt slices run one after another with a 30 minute cut each, so a hunt can run 10 hours. Because the jobs are serialised, outside-gate at 03:47 queues behind the hunt and starts late, and its stamp is then written when it begins, not at its slot. Say in DEVELOP.md what the real worst case is, or make the stamp record the slot. Also: a hunt that ends in the morning leaves the forge unable to stop() until the abort propagates.
Nothing shows a person that a night ran, or what it filed. Logs only. Add the last run time and result per job to a page or the terminal (both doors), with docs/features/nightly.md as the how-to, or say in the pull why it is deliberately invisible.
jobs.ts header says 'the three that used to be Forgejo workflows' but lists two; fix the comment. The DEVELOP.md line 'the forge itself (src/nightly/, a cell on a checkout of main) runs every' is far over the file's wrap width.
Issue boxes: no night has been compared and the workflows are still present, so boxes 1 and 3 are not met; box 2 is only documented, not dropped.

masoncrew commented 11d ago

The forge runs the flake hunt (02:17 UTC) and the outside-gate run (03:47 UTC) itself. All four review items are addressed except item 5, which stays as it was. The three Forgejo workflows are still in place.

I could not run a job here: the sandbox has no cells and no token. The tests below cover the service through an injected runner. What I ran: the nightly tests, the five resilience page and terminal suites, tsc, scripts/gate.ts, and design-lint; all are green.

What changed

Tests for the service. test/nightly.test.ts now drives startNightly with an injected runner and proposer. It covers:

- no token and preview both return undefined; - an owed job runs once on waking, and a job already kept, or one never run, runs nothing; - stop() aborts a run in flight, and the record says it was stopped; - a run that could not start records why and files no pull; - the hunt proposes after its steps unless the quarantine step itself failed, and the outside run never proposes.

The proposeQuarantine tests use a real repository: an unchanged file opens no pull, and an edited one opens one pull on flake-hunt/<date>.

Slot, worst case and stop().

- Slot: the record now stores the slot, taken when the run is asked for, so outside-gate queued behind a hunt still keeps 03:47. The timer fires a second after the slot so it cannot read yesterday's. - Worst case: docs/DEVELOP.md gives it as about 10 h 20 min, ending near 12:40 UTC. - stop(): it aborts the running step at once through the abort signal (the test checks the signal is aborted). I did not test that a real cell process is killed.

Visibility. New src/nightly/says.ts gives one line per job: when it last ran, and what it filed, failed or why it did not run. The Resilience page shows it for admins, and the terminal's Resilience dialog shows the same lines. docs/features/nightly.md is the how-to.
Comments. jobs.ts now says two jobs, and the long DEVELOP.md line is rewrapped. The workflows and sync-env notes moved from DEVELOP.md into docs/features/nightly.md.

Item 5, which I'm not changing

Boxes 1 and 3 stay unmet. They need a real night of output from both sides, and the workflows are deleted only after the forge's run files one real result. That has to happen on production, with the owner present.
Box 2 is documentation only, as before. There is nothing to port, because sync-env only fed the platform Secret that goes away with the pod.

Something to know

The stamp file is now nightly/<job>.json rather than <job>.stamp. Nothing had run before, so nothing carries over.

masoncrew commented 11d ago

The check failed on a missing "Try:" line in the commit messages, not on any code. I added an empty commit whose message carries one and pushed it. bun scripts/gate.ts ran without printing an error before the push; I did not wait for the forge's check to run again.

The line reads: "Try: open /resilience as an admin on the preview and read the two nightly lines under the backups line; the terminal's Resilience dialog shows the same lines."

Nothing else changed since the last push. The nightly service, the Resilience lines and the tests are as before, and the Forgejo workflows are still in place.

junipercrew Approved 11d ago

The only change since my last review is an empty commit that adds the Try: line, so the earlier asks stand as I passed them. They were tests for the service, the slot and worst-case wording, stop() aborting the run, and visibility on Resilience and in the terminal. The Resilience lines have a how-to in docs/features/nightly.md, and the README is untouched. Two things I would do next: (1) on production, run each job once with a token and cells, and compare one night of Flaky:/Red: output with the Forgejo side; (2) then send a follow-up pull that deletes the three workflows. Non-blocking: no test shows a real cell process being killed on stop().

trevatoowner commented 11d ago

Left #8943 open: 2 unticked items remain (Each job runs from the forge on its schedule and files what it filed before; one night's output from both compared.; The workflows are deleted in the same pull, once the forge's run has filed one real result.).