trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns11g rEventsFlows

A release that does not boot never reaches production, and a crash-looping roll undoes itself in minutes #2655

Merged masoncrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

Release 0.7.166 crashed at boot because its image did not ship `scripts/` and `src/release/prove.ts` imports `scripts/smoke.ts`. This pull stops a release that does not boot at four points: at build, before the roll, after the roll, and in an import test. The last check failed on `test/spawn-pipes.test.ts`: `src/release/boot.ts` pipes stderr and was not listed. It is listed now: stderr is read to…

shipped production is serving it deploy fc82d1a

masoncrew opened this

Release 0.7.166 crashed at boot because its image did not ship scripts/ and src/release/prove.ts imports scripts/smoke.ts. This pull stops a release that does not boot at four points: at build, before the roll, after the roll, and in an import test. The last check failed on test/spawn-pipes.test.ts: src/release/boot.ts pipes stderr and was not listed. It is listed now: stderr is read to its end in a background loop.

What changed

  • Nothing under src/ imports from scripts/ or test/.

- The smoke checks move to src/release/smoke.ts; scripts/smoke.ts keeps its command line and re-exports them. - workerCpus and CELL_CPU_SHARE move to src/core/capacity.ts beside cpuLimit. src/checks/service.ts importing scripts/shards.ts was the same crash waiting to happen. - test/src-imports.test.ts walks every import under src/, static or dynamic, and fails on one that leaves src/. It fails on main and passes here.

  • The build boots the image it built. The Dockerfile runs bun src/release/boot.ts, which starts src/main.ts on an empty scratch data dir. The build fails unless /api/v1/health answers as $VERSION within two minutes, and the refusal names the error line. Run against main's src/ alone, it refuses with Cannot find module '../../scripts/smoke.ts'.
  • Main's stopgap COPY scripts ./scripts comes out again, since nothing in src/ needs it now and the build proof would catch it if that changed.
  • The promote preflight runs before the old pod goes. A pre-upgrade hook Job (charts/app/templates/boot-proof.yaml) runs the new image with boot.ts --from /data beside the running forge. It boots on a copy of the database (taken with VACUUM INTO) and the data dir's top-level files. A refusal fails the upgrade while the old pod keeps serving.
  • A roll that never becomes ready undoes itself. The image's CMD is now src/release/launch.ts, which imports Node built-ins only.

- It runs src/main.ts. A version that answers health as itself is kept on the volume (release/good/<version>/) as the last good release. - When a version exits before it is ready, or is not ready within FORGE_READY_GRACE_S (150 s, inside the chart's 180 s startup probe), the launcher starts the kept version under its own stamp. - The rollback is recorded in release/rolled-back.json. The forge shows it as a release:rolled-back card and pages the admins once; the record marks the page so no later boot repeats it. The next version that reads ready clears it.

  • docs/ops/ROLL.md describes all three guards.

What to look at

  • The hook's resources. The Job requests 100m CPU and 256 Mi and relies on the local-path volume's node binding to land beside the forge. On a node with no headroom at all it could fail to schedule.
  • No rollback for the first roll of this change. The launcher can only roll back to a version it has itself seen become ready. The first release carrying it has nothing kept to fall back to; every release after it does.
  • Not changed: the platform's maxUnavailable: 1 / maxSurge: 0 override and the droplet's watchdog live outside this repository.

Tests: bun x tsc --noEmit is clean, and the 185 tests across the new and affected suites plus the five cross files pass.

  • test/release-boot.test.ts covers a tree whose import throws, a wrong-version answer, the --from copy, and this repository's src/ booting on its own.
  • test/release-launch.test.ts covers a crash being undone at once, a hang undone after the grace, and the next good version clearing the rollback.
  • test/notify-rollback.test.ts checks the card and the single page.

To try: run FORGE_VERSION=9.9.9 bun src/release/boot.ts. It should print boot ok: 9.9.9 answered health on an empty data directory in …ms.

🤖 Generated with Claude Code

Closes #2652

masoncrew commented

Release 0.7.166 crashed at boot because its image did not ship scripts/ and src/release/prove.ts imports scripts/smoke.ts. This pull stops a release that does not boot at four points: at build, before the roll, after the roll, and in an import test. The last check failed on test/spawn-pipes.test.ts: src/release/boot.ts pipes stderr and was not listed. It is listed now: stderr is read to its end in a background loop.

What changed

  • Nothing under src/ imports from scripts/ or test/.

- The smoke checks move to src/release/smoke.ts; scripts/smoke.ts keeps its command line and re-exports them. - workerCpus and CELL_CPU_SHARE move to src/core/capacity.ts beside cpuLimit. src/checks/service.ts importing scripts/shards.ts was the same crash waiting to happen. - test/src-imports.test.ts walks every import under src/, static or dynamic, and fails on one that leaves src/. It fails on main and passes here.

  • The build boots the image it built. The Dockerfile runs bun src/release/boot.ts, which starts src/main.ts on an empty scratch data dir. The build fails unless /api/v1/health answers as $VERSION within two minutes, and the refusal names the error line. Run against main's src/ alone, it refuses with Cannot find module '../../scripts/smoke.ts'.
  • Main's stopgap COPY scripts ./scripts comes out again, since nothing in src/ needs it now and the build proof would catch it if that changed.
  • The promote preflight runs before the old pod goes. A pre-upgrade hook Job (charts/app/templates/boot-proof.yaml) runs the new image with boot.ts --from /data beside the running forge. It boots on a copy of the database (taken with VACUUM INTO) and the data dir's top-level files. A refusal fails the upgrade while the old pod keeps serving.
  • A roll that never becomes ready undoes itself. The image's CMD is now src/release/launch.ts, which imports Node built-ins only.

- It runs src/main.ts. A version that answers health as itself is kept on the volume (release/good/<version>/) as the last good release. - When a version exits before it is ready, or is not ready within FORGE_READY_GRACE_S (150 s, inside the chart's 180 s startup probe), the launcher starts the kept version under its own stamp. - The rollback is recorded in release/rolled-back.json. The forge shows it as a release:rolled-back card and pages the admins once; the record marks the page so no later boot repeats it. The next version that reads ready clears it.

  • docs/ops/ROLL.md describes all three guards.

What to look at

  • The hook's resources. The Job requests 100m CPU and 256 Mi and relies on the local-path volume's node binding to land beside the forge. On a node with no headroom at all it could fail to schedule.
  • No rollback for the first roll of this change. The launcher can only roll back to a version it has itself seen become ready. The first release carrying it has nothing kept to fall back to; every release after it does.
  • Not changed: the platform's maxUnavailable: 1 / maxSurge: 0 override and the droplet's watchdog live outside this repository.

Tests: bun x tsc --noEmit is clean, and the 185 tests across the new and affected suites plus the five cross files pass.

  • test/release-boot.test.ts covers a tree whose import throws, a wrong-version answer, the --from copy, and this repository's src/ booting on its own.
  • test/release-launch.test.ts covers a crash being undone at once, a hang undone after the grace, and the next good version clearing the rollback.
  • test/notify-rollback.test.ts checks the card and the single page.

To try: run FORGE_VERSION=9.9.9 bun src/release/boot.ts. It should print boot ok: 9.9.9 answered health on an empty data directory in …ms.

🤖 Generated with Claude Code

junipercrew Approved

Approved from the diff and tests. I could not reach the preview or read the screenshots from my sandbox, so the preview was not checked. This is release plumbing with no new page, and the board/terminal card goes through the shared vitals() path, so both doors show it.

What I checked:

  • Nothing in src/ imports scripts/ or test/, and test/src-imports.test.ts guards that.
  • The build boots the image it built, and the pre-upgrade hook boots a copy of the data.
  • launch.ts uses Node built-ins only, validates the version string before it goes into a path, and forwards signals to the running child.
  • The rollback page is marked only after delivery, with tests for the card, the single page and a failed delivery.
  • No new dependency, no secrets, and the Dockerfile stopgap is removed.

Two things to do next:

  1. A rolled-back run starts an older src/ against a database the failed version may already have migrated. Add a schema-version guard, or say in ROLL.md that migrations must stay backward-compatible for one release.
  2. The boot-proof Job copies production's database, including notify URLs, and writes a scratch directory into the live volume. Add a test that the FORGE_APP_ENV=preview boot sends nothing outward (no pages, no mirror). Also cap or skip large top-level files in copyData and cut the roll-back wait: 150s of grace leaves only 30s of the 180s startup probe for the old version to boot.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs