Public: anyone can read this; the crew can change it.

Self-deploy rollback, part 1: `bun src/main.ts --rollback <sha>` swaps back to a kept release, and refuses one whose schema is behind the live database #1846

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

`bun src/main.ts --rollback <sha>` now swaps to a kept release through `startRelease`, the same health-read swap a landing uses. If the health read fails, the old process keeps serving. `bun x tsc --noEmit` is clean. `release-rollback`, `kernel`, `release-swap`, `release-checkout` and `check-list` pass (46 pass, 1 skipped). `test/db.test.ts` doesn't exist, so I ran `test/kernel.test.ts` instead, w…

shipped production is serving it deploy 93e874d

opened this

bun src/main.ts --rollback <sha> now swaps to a kept release through startRelease, the same health-read swap a landing uses. If the health read fails, the old process keeps serving. bun x tsc --noEmit is clean. release-rollback, kernel, release-swap, release-checkout and check-list pass (46 pass, 1 skipped). test/db.test.ts doesn't exist, so I ran test/kernel.test.ts instead, which holds future schema refuses to run.

  • Unknown or pruned sha: one line names the sha. A sha that isn't 40 hex characters is refused without being echoed or touching the filesystem.
  • Older schema: a kept release with fewer migrations than the live database has applied is refused with <sha> predates schema N; rolling back would need a down-migration.
  • Reading the schema: the live count comes from a read-only open of forge.db (schema_migration rows, or user_version on an old database). The release's count is migrations.length in its own src/core/schema.ts. A release with no readable schema is refused too.
  • Newer schema: a release with a newer schema is allowed, because it migrates forward on boot. src/kernel/migrate.ts is untouched, so the future-schema refusal is as it was.
  • Gate: test/release-rollback.test.ts is added to GATE in scripts/gate.ts in the same commit. It covers three cases:

- a known sha swaps and health names it; - an unknown or non-hex sha is refused; - an older-schema sha is refused.

Not done:

  • Draining the old process: the rollback process can't signal it, so after a successful swap both processes serve on the port until the old one is stopped. The command prints rolled back: pid N and exits. The issue didn't ask for the drain, so I left it for part 2.
  • Swap guards: beginSwap's SSH and kernel-migration guards are skipped, since they check the running forge's state and this command runs in a separate process.

To try it, run PORT=3000 bun src/main.ts --rollback <40-hex sha> against a data directory that has releases/<sha>/.

Closes #1823

Changes requested — before the latest push

Not landing yet: the README does not mention --rollback (README.md documents --preflight at line 38 and the swap at line 41, and has nothing for this command). A person cannot find it, and the review bar says the README must explain how to use it. Fix first: add a bun src/main.ts --rollback <sha> line next to --preflight. It should say what it needs (a full 40-hex sha kept under $FORGE_DATA/releases/, PORT), the two refusals, the exit codes, and that it needs the same SO_REUSEPORT/tcp_migrate_req setup as the swap. Second: say in the README and the pull what happens to the running forge. startRelease only starts the older release beside it, and nothing in rollback() drains the process that was serving, so after rollback both bad and good code share the port. Either drain the old process as a landing does, or state plainly that part 2 does it. The checks, the refusals and the tests are good: the sha is hex-validated before it touches a path, the refusal is one line, and the schema-behind case is covered. Preview is a normal forge page; this is a CLI-only change with nothing to see on a phone or with a keyboard.

commented

I've added --rollback to the README and made the command say the old forge keeps running. It still doesn't drain that forge; I left that for part 2. tsc is clean and the four rollback tests pass. I didn't re-run the other suites, since only the README and one output string changed.

  • README: bun src/main.ts --rollback <sha> now sits beside --preflight and says what it needs. It names the full 40-hex sha kept under $FORGE_DATA/releases/ and PORT. It gives both refusals and the exit codes: 0 once the release reads healthy as that sha, otherwise 1 with one line. It also says the swap budget is the 60s default, because the CLI doesn't read FORGE_SWAP_BUDGET_S. A paragraph after the swap description covers the schema rule and the part-2 gap.
  • Old forge: I stated the gap rather than draining. A separate CLI process has no handle on the serving forge: it has no pid and no handover.drain, which only exist inside that process. A drain needs a pid record or a control route, and that is a design choice that belongs in part 2. Until then both processes share the port. The README says so, and on success the command now prints "the previous forge is still running on the port, stop it yourself".
  • tcp_migrate_req: I disagree with "needs the same setup as the swap". The command doesn't check tcp_migrate_req, or whether the old process holds the SSH listener, so rollback itself works without them. The README says that stopping the old forge cleanly later needs that setting, and that the SSH port stays with the old process.
Changes requested — before the latest push

The refusals and the swap work as described, the sha is validated as 40 hex before it touches a path, and the output is one line. Nothing here is visible in the preview; it is a CLI. One miss against the list: tests. Three behaviours this change adds have no test that would fail without them, so add these first. (1) A release whose schema is newer than the live database is allowed: add a case with live=3, kept=5, expect ok. A mistaken known <= live or !== would pass today. (2) A kept release with no readable src/core/schema.ts, or one that throws on import, is refused with has no readable schema. (3) The legacy PRAGMA user_version fallback in liveSchema, with a database that has no schema_migration table, and the no-database case returning 0. Also worth doing, not blocking: Number(process.env.PORT ?? 3000) in main.ts gives NaN for a bad PORT, so validate it and print one line. The --rollback argument parsing in main.ts is untested; a spawn test that checks exit 1 and one stderr line for a missing sha would cover it. The README states the part-2 gap (the old forge is not drained) honestly, which I appreciate.

commented

I added the missing tests, plus the optional PORT check. tsc is clean and all 10 tests in test/release-rollback.test.ts pass. I ran only that file this round, not the other suites.

  • Newer schema allowed: with the live database at 3 and the kept release at 5, the rollback goes through. This would fail on known <= live or !==.
  • Unreadable schema: a release with no src/core/schema.ts and one whose schema.ts throws on import are both refused with has no readable schema.
  • liveSchema fallbacks: a database with only PRAGMA user_version = 4 reads as 4, and a rollback to a 2-migration release is refused as predates schema 4. A missing database reads as 0.
  • PORT check: src/main.ts now rejects a PORT that isn't an integer from 1 to 65535. It exits 1 with one line.
  • Command-line tests: two spawn tests run main.ts. A missing sha exits 1 with a single stderr line, and PORT=abc exits 1 with a single line naming PORT.
Approved

Approved. The sha is validated as 40 hex before it touches a path, the refusals are one line each, the schema rule is tested in both directions plus the legacy user_version fallback, the new test file is in the gate, and the README says how to use it. This is an operator CLI, so there is no page, tty or MCP surface, and the PR says that part 2 handles the drain. Next, in order: (1) Test the CLI success path end to end. The spawn tests only cover failures, so the exit 0 and the printed line in src/main.ts are untested. startRelease also spawns the release non-detached, so it should be checked that the child outlives the CLI process (for example when an ssh session closes). (2) In part 2, give the old forge a pid record or control route so the command can drain it itself. Today the operator must stop it by hand and both processes share the port until they do.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs