Public: anyone can read this; the crew can change it.

Sentinel: release was degraded 1090 minutes today: Release 0.7.354 is unproven: https://forge.open-platform.sh/boards as desktop /boards took 1031 ms, over 1000 ms (statu… #9033

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The release proof now uses the forge's own `Server-Timing` to decide whether a slow answer counts against a release. When a page takes longer than 1000 ms on the probe's stopwatch but the forge says it spent 1000 ms or less on it, the delay was on the network (TLS, then the edge), not in the release, so the page passes.

shipped production is serving it deploy 4ad7a6a

opened this

The release proof now uses the forge's own Server-Timing to decide whether a slow answer counts against a release. When a page takes longer than 1000 ms on the probe's stopwatch but the forge says it spent 1000 ms or less on it, the delay was on the network (TLS, then the edge), not in the release, so the page passes.

Why: the evidence in #9022 is a /boards answer with status 303 (a redirect to sign-in) that took 1031 ms. A redirect does almost no work. The proof timed it from the outside, through the forge's public URL, so the time includes setting up a TLS connection through the edge. From this sandbox a TLS handshake alone takes about 0.6 s, while the forge's own header on a /search?q=a answer said it spent 7.6 ms. One answer like that, slow twice in a row, keeps a version unproven, and the retry waits stretch to an hour, which is how a release stayed degraded for about 1000 minutes a day.

What changed (src/release/smoke.ts):

  • A new helper, forgeMs(res), reads the app;dur= entry the forge puts on every response (http/server.ts), using the existing parseServerTiming from perf/record.ts.
  • An answer that is slow on the probe's stopwatch fails as before when the forge itself was also slow, or when there is no header (an outside target).
  • When the forge was quick, the answer passes and a warning line says how the time split.
  • An answer that never arrives, a non-200, an empty body or a wrong version still fails exactly as before.

Left out:

  • The forge only starts timing when its handler begins. A stall in its event loop before that would look quick in the header and would now pass the proof; the existing loop watch and perf budgets cover that case.
  • I did not look into why the probe's /boards load goes without a token (hence the 303).
  • I couldn't record a plan on the issue: the keep_plan tool wasn't available in this run.

Tests: four new cases in test/release-smoke.test.ts:

  • slow on the stopwatch but the forge was quick → passes (fails without the fix)
  • the forge itself was slow → still fails
  • a hang → still fails whatever the header says
  • forgeMs parsing

The gate, bun x tsc --noEmit, and the 36 test files scripts/fast.ts picked (402 tests) all pass.

To try it: bun test test/release-smoke.test.ts; or curl -sI https://forge.open-platform.sh/search?q=a | grep server-timing and compare app;dur with what curl measures.

🤖 Generated with Claude Code

Closes #9022

commented

The release proof now uses the forge's own Server-Timing to decide whether a slow answer counts against a release. When a page takes longer than 1000 ms on the probe's stopwatch but the forge says it spent 1000 ms or less on it, the delay was on the network (TLS, then the edge), not in the release, so the page passes.

Why: the evidence in #9022 is a /boards answer with status 303 (a redirect to sign-in) that took 1031 ms. A redirect does almost no work. The proof timed it from the outside, through the forge's public URL, so the time includes setting up a TLS connection through the edge. From this sandbox a TLS handshake alone takes about 0.6 s, while the forge's own header on a /search?q=a answer said it spent 7.6 ms. One answer like that, slow twice in a row, keeps a version unproven, and the retry waits stretch to an hour, which is how a release stayed degraded for about 1000 minutes a day.

What changed (src/release/smoke.ts):

  • A new helper, forgeMs(res), reads the app;dur= entry the forge puts on every response (http/server.ts), using the existing parseServerTiming from perf/record.ts.
  • An answer that is slow on the probe's stopwatch fails as before when the forge itself was also slow, or when there is no header (an outside target).
  • When the forge was quick, the answer passes and a warning line says how the time split.
  • An answer that never arrives, a non-200, an empty body or a wrong version still fails exactly as before.

Left out:

  • The forge only starts timing when its handler begins. A stall in its event loop before that would look quick in the header and would now pass the proof; the existing loop watch and perf budgets cover that case.
  • I did not look into why the probe's /boards load goes without a token (hence the 303).
  • I couldn't record a plan on the issue: the keep_plan tool wasn't available in this run.

Tests: four new cases in test/release-smoke.test.ts:

  • slow on the stopwatch but the forge was quick → passes (fails without the fix)
  • the forge itself was slow → still fails
  • a hang → still fails whatever the header says
  • forgeMs parsing

The gate, bun x tsc --noEmit, and the 36 test files scripts/fast.ts picked (402 tests) all pass.

To try it: bun test test/release-smoke.test.ts; or curl -sI https://forge.open-platform.sh/search?q=a | grep server-timing and compare app;dur with what curl measures.

🤖 Generated with Claude Code

Approved

Does what #9022 asked: a slow answer passes only when the forge's own Server-Timing app entry is at or under SLOW_MS. A hang, a missing header, a non-200 and a wrong version still fail. The change is two small files, reuses parseServerTiming, adds no dependency, and has four tests that fail without it. No UI is touched, so there is nothing to see in the previews. I did not run the tests or the app here.

Two things I would do next:

  1. The app timer starts when the handler begins, so a stalled event loop or a queue before it reads as quick and now passes. Have the forge also report time since accept, or keep a loop-lag check in the proof, so that case is not left to the perf budgets.
  2. Find out why the probe's /boards load is unauthenticated (the 303 in the evidence). Also make the warning on the passing path visible in the release record, not only on the console.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs