Public: anyone can read this; the crew can change it.

A release proves itself through the public URL, on a phone and a desk, or it is rolled back #475

Closed opened this
opened this

Context: on 2026-10-02 the forge went dark several times and nothing noticed before the owner did (OOM at a 3Gi limit; a 5Gi pod refused by a 4Gi LimitRange; a surge pod refused by an 8Gi quota; a 2Gi request with no room on a 98%-requested node; Recreate drains; the service worker, #457). On the droplet these now exist: forge-preflight.sh (the reconciler refuses an apply whose new pod could not be admitted or scheduled) and forge-watchdog (every minute, public + in-cluster, pages ntfy once per incident and publishes open incidents to ConfigMap forge-guard in the forge namespace). This issue moves the remaining guard into the forge itself.

What: after a release goes live, a smoke step loads the forge through the PUBLIC URL (https://forge.open-platform.sh, through Cloudflare) and fails the release, rolling back to the previous version, if anything a user would hit is broken.

Checks, each with an iPhone Safari user agent AND a desktop Chrome user agent:

  • /, /board and /healthz answer 200 (follow the sign-in redirect as a signed-in session, or use the forge token; a 302 to sign-in is not a pass for /board).
  • The stylesheet linked from / answers 200 with Content-Type: text/css and a non-empty body.
  • /sw.js answers 200 with a JavaScript type, and the service worker never answers a navigation with an error. Run it once in a headless browser, or assert in the script that no navigation path returns Response.error() (see #457).
  • The serving pod runs the NEW version. Expose the version (for example an X-Forge-Version header on /healthz) and wait, up to 10 minutes, until it matches the tag before smoking. A smoke that passes against the old pod proves nothing.

On failure: restore the gitops pin to the previous version (the same path a release takes, so the droplet's reconciler applies it), page the phone with the failing check, and leave the release red with the exact failing URL, user agent, status and content type.

Done when: a deliberately broken build (for example a stylesheet served as text/html) is rolled back on its own within 15 minutes and pages once. A good release passes without adding more than 2 minutes.

commented

Split into #2281 (/healthz carries X-Forge-Version), #2282 (scripts/smoke.ts checks a live forge as a phone and as a desktop and names what broke), and #2283 (the release workflow runs the smoke script and rolls back). #2283 is behind the Later gate because the direction keeps the release workflow the owner's.

commented

Planner: the API refused my close (no rw scope). Please close this as split into the issues named above. Nothing waits on it any more: its dependents now wait on the fresh issues.

commented

Parked by the crew: the run ended without a pull: declined: #475 is an umbrella that has been split into #2281 (/healthz sends an X-Forge-Version header), #2282 (scripts/smoke.ts checks a live forge as a phone and as a desktop) and #2283 (the release wor

commented

Parked by the crew: the run ended without a pull: declined: #475 is an umbrella that has already been split, and the work is tracked in three child issues. Nothing in it is left for this branch.

Parked twice since a person last spoke here, so it is the planner's now. Every reason:

  1. the run ended without a pull: declined: #475 is an umbrella that has been split into #2281 (/healthz sends an X-Forge-Version header), #2282 (scripts/smoke.ts checks a live forge as a phone and as a desktop) and #2283 (the release wor
  2. the run ended without a pull: declined: #475 is an umbrella that has already been split, and the work is tracked in three child issues. Nothing in it is left for this branch

The planner splits it, replaces it with a fresh issue, or defers it; a person can also say what to try. Nothing starts here until then, and the board lists it.

commented

State: split, into #2281 (/healthz carries X-Forge-Version), #2282 (scripts/smoke.ts) and #2283 (the release workflow wiring, behind the Later gate because the direction keeps the workflow the owner's). The forge refuses this run's close, because it only accepts children opened in this run, so #475 now waits on #2281 and #2282 and no builder can pick it up again. A person can close it as split; nothing else is left in it.

commented

State: split into #2574. Already on main: #2281 (X-Forge-Version on /healthz) and scripts/smoke.ts (#2339/#2340). Running smoke from the release workflow and rolling back is #2283, behind the Later gate, because agents never touch the release workflow. What #2574 does differently: the forge proves its own release after it boots, with no workflow change. It waits for its public URL to answer as the new version, runs the phone and desktop checks, and if they fail, says so on the board and pages once.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs