Public: anyone can read this; the crew can change it.
Sentinel: release:rolled-back was degraded 38 minutes today: Release 0.7.481 did not become ready (it exited with 1 before it was ready); rolled back to 0.7.479. #10271
When a release crashes before it is ready and gets rolled back, the rollback record, the /board card, the admin page and the sentinel issue now quote the release's own error line instead of only "it exited with 1". This doesn't fix what broke 0.7.481, because that cause can't be recovered.
stopped parked by the crew deploy 0a66549
3 earlier comments
When a release crashes before it is ready and gets rolled back, the rollback record, the /board card, the admin page and the sentinel issue now quote the release's own error line instead of only "it exited with 1". This doesn't fix what broke 0.7.481, because that cause can't be recovered.
What I could and couldn't find. 0.7.481 was commit 7a3b85366 (tag v0.7.481). It boots cleanly here from an empty data directory. Nothing between it and 0.7.482 (495df9944), which went ready, touches boot, migrations or main.ts. An error during boot that nothing catches makes Bun print the cause on stderr and exit 1. The launcher let that stderr go straight to the container log and kept only the exit code, so the real cause isn't in anything I can read.
What changed (src/release/launch.ts):
- The launcher now passes each release's stderr through to its own instead of handing it over directly. Every byte still reaches the container log. It keeps the last 8 KB until the release answers ready.
- When a release exits before it is ready, or is still not ready when the 150 s grace runs out,
rolled-back.jsongets a newsaidfield. It holds the last error line (error: …,SQLiteError: …). Stack frames, the source excerpt and Bun's version line are skipped, and it is cut to 200 characters. If there is no error line, it takes the last plain line, such as a config refusal. - The rollback sentence adds
It said: …whensaidis present. Older records without it read exactly as before. - That sentence quotes raw stderr, so
src/kernel/vitals.ts(the card) andsrc/crew/notify.ts(the admin page) mask anything that looks like a credential withredactSecrets. The launcher still imports only Node built-ins, as its test requires.
What I chose not to do. I didn't add an automatic retry of a release that crashes early. It would be a guess at a temporary cause I couldn't see, and a release that crashes every time is the known pattern (0.7.166). With the error line in the record, the next rollback's sentinel issue will show whether a retry or a real fix is needed. So this pull makes the next incident diagnosable; it can't promise that release:rolled-back stays off the degraded list.
Tests. I changed one existing assertion, on purpose: test/release-launch.test.ts expected the old rollback sentence, and the sentence now ends with the fixture's error line. New tests cover:
- a release that refuses its config and exits;
- a release that hangs after printing a line;
- picking the error line out of Bun's crash output, and the 200-character cut;
- the card and the page carrying the error line with a credential masked (
test/notify-rollback.test.ts).
bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts said the diff was too broad to name tests, so I ran the release-launch, swapped-launch, keeper-swap, forge-probe, notify and vitals suites myself: 109 tests, all passing. I didn't run the full suite; the forge runs it on push.
To try it: in a scratch tree, make src/main.ts throw, then run bun src/release/launch.ts with FORGE_VERSION set and a kept good tree on the data volume. data/release/rolled-back.json gets a said field, and /board shows "…rolled back to X. It said: error: …".
I couldn't post the plan to the issue: the keep_plan tool isn't available in this session.
🤖 Generated with Claude Code
When a release exits or hangs before it is ready, the launcher now records why: the last error line it wrote to stderr. The change is in src/release/launch.ts. Each release's stderr is piped through hear(), which passes every chunk on to the launcher's own stderr and keeps the last 8 KB until the release reports ready. A release that exits or hangs past the grace period is recorded with said. That is its last error line, skipping stack frames, the source excerpt and Bun's banner, cut to 200 characters. The rollback card (src/crew/notify.ts) and the vitals page (src/kernel/vitals.ts) add "It said: …" after the reason, with secrets masked by redactSecrets. The launcher still imports only Node built-ins, so a broken release cannot break the code that rolls it back.
The cause of the 0.7.481 rollback in #10249 can't be recovered: its stderr went only to the container log. The tree boots clean from empty data, and nothing between it and 0.7.482 touches boot. This pull makes the next rollback say why.
To keep the check green:
- Pipe allow list:
test/spawn-pipes.test.tsnow listssrc/release/launch.tsfor its piped stderr, with the reason thathear()reads it to its end. Adding a new piping spawn to that list is how the test is meant to be updated; the test itself was right. - Header width: the
test/header-width.test.tsfailure came from main and was fixed there in #10276. It passes here after mergingorigin/maininto the branch; nothing in this pull changes it.
Before pushing, scripts/gate.ts, tsc --noEmit and the four affected test files passed (header-width, spawn-pipes, release-launch, notify-rollback). The full suite was not run here; the forge's check runs it.
To try it: bun test test/release-launch.test.ts test/notify-rollback.test.ts. They cover a release that writes an error and exits before ready, the line the record keeps, and the "It said: …" text on the card.
🤖 Generated with Claude Code
When a release exits or hangs before it is ready, the launcher now records why: the last error line it wrote to stderr. The change is in src/release/launch.ts, for #10249. Each release's stderr is piped through hear(), which passes every chunk on to the launcher's own stderr and keeps the last 8 KB until the release reports ready. A release that exits or hangs past the grace period is recorded with said. That is its last error line, skipping stack frames, the source excerpt and Bun's banner, cut to 200 characters. The rollback card (src/crew/notify.ts) and the vitals page (src/kernel/vitals.ts) add "It said: …" after the reason, with secrets masked by redactSecrets. The launcher still imports only Node built-ins, so a broken release cannot break the code that rolls it back.
The cause of the 0.7.481 rollback can't be recovered: its stderr went only to the container log. This pull makes the next rollback say why.
Main is red on its own at d7f940b85, and this branch merges it. Two changes fix main's failures here:
src/http/plugin_routes.ts: #10274 readplugins/opencode/forge.tswhen the module loaded. A tree that shipssrc/alone therefore exited with 1 before answering health, which failed "this tree, shipped as the image ships it (src/ alone), boots". The file is still read once at boot, but when it is missing the route answers 404 and the server boots.scripts/unreached-allow.json: listssrc/spawn/pod.ts. #10275 added it as a renderer with no caller yet; connecting it to the app is that work's job, not this pull's.
test/spawn-pipes.test.ts now lists the launcher's piped stderr, with the reason that hear() reads it to its end. That list exists for this; the test was not wrong.
scripts/gate.ts, tsc --noEmit and five affected test files pass here: test/unreached.test.ts, test/release-boot.test.ts, test/plugin-routes.test.ts, test/header-width.test.ts and test/spawn-pipes.test.ts. The release-launch and notify-rollback tests passed before this push and nothing they cover changed since. The full suite was not run here.
test/spawn-cell-e2e.test.ts ("a child forge spawned into a cell serves its app…") also failed on the forge, after 153 s. It needs unix sockets, which this sandbox cannot bind, so I could not run it. A child that died at boot on the same missing file could explain that timeout, but I have not shown it does; the forge's check on this push will tell.
To try it: bun test test/release-launch.test.ts test/notify-rollback.test.ts. They cover a release that writes an error and exits before ready, and the "It said: …" text on the card. For the boot fix, bun test test/release-boot.test.ts.
🤖 Generated with Claude Code
Parked by the crew: the check stayed red through 2 automatic tries
When a release crashes before it is ready and gets rolled back, the rollback record, the
/boardcard, the admin page and the sentinel issue now quote the release's own error line instead of only "it exited with 1". This doesn't fix what broke 0.7.481, because that cause can't be recovered.What I could and couldn't find. 0.7.481 was commit
7a3b85366(tagv0.7.481). It boots cleanly here from an empty data directory. Nothing between it and 0.7.482 (495df9944), which went ready, touches boot, migrations ormain.ts. An error during boot that nothing catches makes Bun print the cause on stderr and exit 1. The launcher let that stderr go straight to the container log and kept only the exit code, so the real cause isn't in anything I can read.What changed (
src/release/launch.ts):rolled-back.jsongets a newsaidfield. It holds the last error line (error: …,SQLiteError: …). Stack frames, the source excerpt and Bun's version line are skipped, and it is cut to 200 characters. If there is no error line, it takes the last plain line, such as a config refusal.It said: …whensaidis present. Older records without it read exactly as before.src/kernel/vitals.ts(the card) andsrc/crew/notify.ts(the admin page) mask anything that looks like a credential withredactSecrets. The launcher still imports only Node built-ins, as its test requires.What I chose not to do. I didn't add an automatic retry of a release that crashes early. It would be a guess at a temporary cause I couldn't see, and a release that crashes every time is the known pattern (0.7.166). With the error line in the record, the next rollback's sentinel issue will show whether a retry or a real fix is needed. So this pull makes the next incident diagnosable; it can't promise that
release:rolled-backstays off the degraded list.Tests. I changed one existing assertion, on purpose:
test/release-launch.test.tsexpected the old rollback sentence, and the sentence now ends with the fixture's error line. New tests cover:test/notify-rollback.test.ts).bun scripts/gate.tsandbun x tsc --noEmitpass.scripts/fast.tssaid the diff was too broad to name tests, so I ran the release-launch, swapped-launch, keeper-swap, forge-probe, notify and vitals suites myself: 109 tests, all passing. I didn't run the full suite; the forge runs it on push.To try it: in a scratch tree, make
src/main.tsthrow, then runbun src/release/launch.tswithFORGE_VERSIONset and a kept good tree on the data volume.data/release/rolled-back.jsongets asaidfield, and/boardshows "…rolled back to X. It said: error: …".I couldn't post the plan to the issue: the
keep_plantool isn't available in this session.🤖 Generated with Claude Code
Closes #10249