Public: anyone can read this; the crew can change it.

A change is live in two minutes, proven on a copy before anyone sees it, and… #11839

Closed opened this

Pull #11889 was closed · 9 runs

opened this

A change is live in two minutes, proven on a copy before anyone sees it, and undone in one step

Why. Owner, 2026-11-12, after a release covered every page on phones and the fix took a release cycle to reach him: "can we tighten the release loop more? we let this through so the slowness doesnt seem justified" and "we need to be able to react at a moments notice and trust that when we make changes we arent disrupting everything." Measured from the release log that day: cut → live took 8.5 to 9.5 minutes (image build in Forgejo Actions, boot check, promote), a ten-minute verify follows, cuts are at most every 30 minutes, and every release restarts the forge's pod, which interrupts every crew run and every hosted forge's run. The verify probes status codes; it passed while the page was unusable on a phone.

What.

  1. Fast. The forge has no build step; a release should not take nine minutes. Cut → serving in under two minutes: build only the layer that changed (or ship the source at a commit to a warm base image), on the forge's own machine. Measure and show each stage's time on the release card (cut, build, boot proof, switch, verify) so the slow stage is always named.
  2. Proven before anyone sees it. The new version starts beside the old one on its own port with a copy-on-read of nothing it can damage (read-only against live data, or a snapshot), and the proof runs against it before traffic moves: the status probes, the stranger journey (#11158), and the covered-page check at phone and desktop size (#11831). Only then does traffic switch. A failed proof never reaches a person, and says which assertion failed.
  3. The switch disturbs nothing. In-flight requests finish on the old process; crew runs and hosted forges' runs are not killed by a release: they keep running across it (the work lives outside the process that restarts, or is handed over), or the release waits a bounded time for runs to reach a safe point. Count interrupted runs per release on the release card; the target is zero.
  4. Undone in one step. rollback (tool, page button, and the release tick itself on a failed post-switch check) puts the previous version back in under a minute, because the previous process is kept warm for ten minutes after a switch. Rolling back never needs a build.
  5. A hotfix does not wait. A release can be cut now by the owner or the steward (release now with a reason), outside the half-hour spacing; the spacing itself becomes "when the previous release is proven", not a clock.
  6. Nothing external decides. The tick, its holds (including the disk guard) and their reasons are visible on /flow and in release_keeper, which today answers empty.

Acceptance. On production, with timings pasted from the release card: a one-line change goes from merge to serving in under two minutes; a deliberately broken stylesheet (a full-screen fixed layer) is stopped at the proof and never served; a release during a crew run interrupts zero runs; a rollback completes in under a minute. release_keeper shows the current release, its stage timings and any hold.

3 earlier comments · 2 events
the crew's run was interrupted: restart, resumes on the next pass — cancelled
commented

Parked by the crew: runs on this issue repeated more than three times in ten minutes, so nothing more starts here until a person says what to try.

the crew's run was interrupted: restart, resumes on the next pass — cancelled
commented

The branch has no changes against origin/main. Main already contains the issue's work, since the last merge notes #11841 had landed it.

DECLINED: already done on main in fd4d15b83 (A change is live in two minutes, proven on a copy before anyone sees it, and… #11841) and 72c2f10a6 (release_now asks first like every admin-scope tool, #11839). The branch carries no diff against origin/main, and nothing has been added to the issue since the last run. A person can close #11839.

commented

Parked by the crew: the run ended without a pull: no changes pushed: The branch has no changes against origin/main. Main already contains the issue's work, since the last merge notes #11841 had landed it.

Parked twice since a person last spoke here, so it is the planner's now. Every reason:

  1. runs on this issue repeated more than three times in ten minutes, so nothing more starts here until a person says what to try
  2. the run ended without a pull: no changes pushed: The branch has no changes against origin/main. Main already contains the issue's work, since the last merge notes #11841 had landed it

The planner splits it, replaces it with a fresh issue, or defers it; a person can also say what to try. Nothing starts here until then, and the board lists it.

commented

Split by the planner. Main already carries part of this: fd4d15b83 (#11841) landed release_now with the hotfix spacing, the stage timings (build, proof, switch) on the release card, and the tree swap on the forge's own machine. That is why the last two runs found nothing to push. What is left is now four small issues, each one run: #11916 (a tree that covers the page is refused before the swap), #11917 (count the crew runs a release interrupted), #11918 (a cut waits a bounded five minutes for running crew runs), #11919 (time the rollback). Unlike this issue, none asks a builder to prove the whole acceptance at once. The production proof with pasted timings is the steward's once these land.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs