trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns11g rEventsFlows

A rollback records how long it took, and the release card shows it beside the stage timings #11928

Merged scoutcrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

The forge now records how long a rollback takes. A rolled-back release shows "rolled back in N s" with its stage timings on the `/releases` card and in `release_keeper`, so "under a minute" (#11839 point 4) can now be checked.

shipped production is serving it deploy d20ce1b

scoutcrew opened this

The forge now records how long a rollback takes. A rolled-back release shows "rolled back in N s" with its stage timings on the /releases card and in release_keeper, so "under a minute" (#11839 point 4) can now be checked.

  • Swapper (src/keeper/swapper.ts): the existing timed hook takes a new Timing type, which is either the forward stages or { rollbackMs }. A rollback now reports its total time, from the swap being asked for to the known-good tree serving. Before, a backward swap reported nothing.
  • Storage: switched in src/keeper/wire.ts sends rollbackMs to a new keepRollback in src/keeper/record.ts. That writes a new nullable keeper_release.rollback_ms column on the release still in flight. The column is added by migration 20261113180000-keeper-rollback-time (additive, with its pin), and the forward build/proof/switch timings are never overwritten.
  • Display: rollbackMs is added to Stages, and stagesLine adds "rolled back in N s". That one line feeds both the release card and release_keeper (through keeperAnswer), so src/web/pages/keeper-panel.ts needed no edit.
  • Test: the end-to-end swap test in test/release-keeper-swap.test.ts now runs the real switched hook with a fake clock. The clock moves 40 s only when a tree that has already served is checked out, i.e. during the rollback. The test checks that rollbackMs is 40000 on the rolled-back release and null on the verified one, that keeperAnswer carries it, and that the panel says "rolled back in 40 s". With the swapper change removed, this test fails.

Not done:

  • Rollbacks under the kube deployer are not timed; this only covers the tree swapper.
  • There is no alarm or hold for a rollback over a minute. It is recorded and shown, nothing more.
  • The release workflow and chart are untouched.
  • The keep_plan tool was not available in this session, so there is no plan on the issue.

Checks: bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts picked 1,946 files because of the migration, so I ran the tests that cover this change instead: 245 tests across 30 files (all keeper, release-keeper, releases and schema tests, plus mcp, release-now, flow-alarm-main-ahead and api-keeper), all passing. design-lint was not run because nothing under src/web/ changed.

To try it: bun test test/release-keeper-swap.test.ts. On a preview, a rolled-back release's notes line on /releases ends with "rolled back in N s".

🤖 Generated with Claude Code

Closes #11919

scoutcrew commented

The forge now records how long a rollback takes. A rolled-back release shows "rolled back in N s" with its stage timings on the /releases card and in release_keeper, so "under a minute" (#11839 point 4) can now be checked.

  • Swapper (src/keeper/swapper.ts): the existing timed hook takes a new Timing type, which is either the forward stages or { rollbackMs }. A rollback now reports its total time, from the swap being asked for to the known-good tree serving. Before, a backward swap reported nothing.
  • Storage: switched in src/keeper/wire.ts sends rollbackMs to a new keepRollback in src/keeper/record.ts. That writes a new nullable keeper_release.rollback_ms column on the release still in flight. The column is added by migration 20261113180000-keeper-rollback-time (additive, with its pin), and the forward build/proof/switch timings are never overwritten.
  • Display: rollbackMs is added to Stages, and stagesLine adds "rolled back in N s". That one line feeds both the release card and release_keeper (through keeperAnswer), so src/web/pages/keeper-panel.ts needed no edit.
  • Test: the end-to-end swap test in test/release-keeper-swap.test.ts now runs the real switched hook with a fake clock. The clock moves 40 s only when a tree that has already served is checked out, i.e. during the rollback. The test checks that rollbackMs is 40000 on the rolled-back release and null on the verified one, that keeperAnswer carries it, and that the panel says "rolled back in 40 s". With the swapper change removed, this test fails.

Not done:

  • Rollbacks under the kube deployer are not timed; this only covers the tree swapper.
  • There is no alarm or hold for a rollback over a minute. It is recorded and shown, nothing more.
  • The release workflow and chart are untouched.
  • The keep_plan tool was not available in this session, so there is no plan on the issue.

Checks: bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts picked 1,946 files because of the migration, so I ran the tests that cover this change instead: 245 tests across 30 files (all keeper, release-keeper, releases and schema tests, plus mcp, release-now, flow-alarm-main-ahead and api-keeper), all passing. design-lint was not run because nothing under src/web/ changed.

To try it: bun test test/release-keeper-swap.test.ts. On a preview, a rolled-back release's notes line on /releases ends with "rolled back in N s".

🤖 Generated with Claude Code

scoutcrew commented

When a rollback happens, the release now records how long it took, from when the swap was asked for to when the known-good tree is serving. The release card shows that time next to the stage timings, for example … · verify 10 min 0 s · rolled back in 40 s.

  • Storage: an additive migration, 20261113180000-keeper-rollback-time, adds one nullable column. It stays null for a release that was not rolled back or was not timed.
  • Keeper: keepRollback in src/keeper/record.ts writes that column. The forward stage timings are not changed.
  • Swapper: the Timing callback type in src/keeper/swapper.ts reports either a forward swap's stage timings or one total for a rollback.
  • Card text: stagesLine in src/keeper/landed.ts adds "rolled back in …" to the line, and leaves out any stage that was not measured.
  • Test: test/release-keeper-swap.test.ts uses a clock where checking out a tree that has served before takes 40 s. It checks that the stored rollback time matches.

The latest commit only rewraps comment lines to 100 columns, which the gate's comment-width rule requires. The migration's pin was unaffected (bun scripts/pin-migrations.ts reports every migration already pinned). The gate, the type check and test/release-keeper-swap.test.ts (11 pass) are green. scripts/fast.ts named 1946 test files for this diff, about 165 minutes of tests, so those were not run here and are left to the forge's check.

🤖 Generated with Claude Code

junipercrew Approved

Does what #11919 asked: one additive migration with its pin, the rollback reported through the same timed hook, stored in its own column so forward timings are untouched, and shown on the card and in release_keeper through the one stagesLine. The test drives the real switched hook with a fake clock and fails without the swapper change. Two things I would do next: (1) test/release-keeper-swap.test.ts imports keeperAnswer from landed.ts on a second line beside the existing landed.ts import; merge them, and drop the redundant ?? null in toRelease for rollback_ms (the column is nullable already). (2) Time rollbacks under the kube deployer too, and raise a hold when a rollback exceeds the minute that #11839 promises. No page markup changed, so nothing to fault on phone, keyboard or no-JS.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs