Public: anyone can read this; the crew can change it.

Sentinel: release was degraded 13 minutes today: Release 0.7.531 is unproven: https://forge.open-platform.sh/trevato/forge as desktop /trevato/forge did not answer with… #11266

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

Release proof: an answer that takes more than 2 s now counts as slow, and its `Server-Timing` judges it. It only counts as a hang if nothing comes back within 10 s (#11250).

shipped production is serving it deploy f21b126

opened this

Release proof: an answer that takes more than 2 s now counts as slow, and its Server-Timing judges it. It only counts as a hang if nothing comes back within 10 s (#11250).

What happened. Release 0.7.531 stayed unproven for 13 minutes because /trevato/forge "did not answer within 2000 ms" on both first-load asks. The smoke check in src/release/smoke.ts put the hang line at twice SLOW_MS, so an answer that took a little over 2 s was cut off before it arrived. Its Server-Timing header was never read. Since #9022 that header has been how the proof tells the forge's own slowness from a slow road to the public edge. A cut-off answer carries no header, so it always failed as a hang, and the 1, 2, 3, 4 minute slow-retry schedule held the card open for 13 minutes.

What changed.

  • New hang line: ANSWER_MS is 10 s and can be set per call as Heavy.answerMs. It is now the hang line for the heavy-page loads, the /healthz watch and the unencoded re-ask.
  • Slow answers: an answer between SLOW_MS and ANSWER_MS is slow. It passes when the forge's own app time is within SLOW_MS, and fails as slow otherwise.
  • Real hangs: no answer within 10 s still fails as a hang, retried as before.
  • Failure text: a slow failure now names the forge's share, e.g. "took 2100 ms, over 1000 ms, the forge 450 ms of it". The next card will say whose time it was.

Not done. The cause of 0.7.531's slow /trevato/forge is still unknown. I could not reach forge.open-platform.sh from this sandbox, and the hang left no timing to read. If the forge itself spent the time, the proof still fails, now as slow and with the forge's milliseconds in the sentence. Retry timing and the first-load re-ask are unchanged. A page that never answers costs at most two 10 s waits plus the 3 s settle, well inside the proof's 15-minute hang guard.

Tests. In test/release-smoke.test.ts:

  • An answer at 500 ms (slow line 200 ms, hang line 2 s) whose Server-Timing says 5 ms passes.
  • One whose Server-Timing says 450 ms fails as slow (not hung), naming "the forge 450 ms of it".

Both would fail on the old cutoff, which made them hangs. The file's shared heavy fixture now sets answerMs: 400, which keeps the existing hang tests at their old effective line, so they test the same thing as before. bun scripts/gate.ts and bun x tsc --noEmit are clean, and the 44 test files named by scripts/fast.ts pass (519 tests).

To check after release: release stays out of the sentinel's degraded list, or under 10 minutes a day, and any slow-proof sentence in /system logs names the forge's own milliseconds.

🤖 Generated with Claude Code

Closes #11250

commented

Release proof: an answer that takes more than 2 s now counts as slow, and its Server-Timing judges it. It only counts as a hang if nothing comes back within 10 s (#11250).

What happened. Release 0.7.531 stayed unproven for 13 minutes because /trevato/forge "did not answer within 2000 ms" on both first-load asks. The smoke check in src/release/smoke.ts put the hang line at twice SLOW_MS, so an answer that took a little over 2 s was cut off before it arrived. Its Server-Timing header was never read. Since #9022 that header has been how the proof tells the forge's own slowness from a slow road to the public edge. A cut-off answer carries no header, so it always failed as a hang, and the 1, 2, 3, 4 minute slow-retry schedule held the card open for 13 minutes.

What changed.

  • New hang line: ANSWER_MS is 10 s and can be set per call as Heavy.answerMs. It is now the hang line for the heavy-page loads, the /healthz watch and the unencoded re-ask.
  • Slow answers: an answer between SLOW_MS and ANSWER_MS is slow. It passes when the forge's own app time is within SLOW_MS, and fails as slow otherwise.
  • Real hangs: no answer within 10 s still fails as a hang, retried as before.
  • Failure text: a slow failure now names the forge's share, e.g. "took 2100 ms, over 1000 ms, the forge 450 ms of it". The next card will say whose time it was.

Not done. The cause of 0.7.531's slow /trevato/forge is still unknown. I could not reach forge.open-platform.sh from this sandbox, and the hang left no timing to read. If the forge itself spent the time, the proof still fails, now as slow and with the forge's milliseconds in the sentence. Retry timing and the first-load re-ask are unchanged. A page that never answers costs at most two 10 s waits plus the 3 s settle, well inside the proof's 15-minute hang guard.

Tests. In test/release-smoke.test.ts:

  • An answer at 500 ms (slow line 200 ms, hang line 2 s) whose Server-Timing says 5 ms passes.
  • One whose Server-Timing says 450 ms fails as slow (not hung), naming "the forge 450 ms of it".

Both would fail on the old cutoff, which made them hangs. The file's shared heavy fixture now sets answerMs: 400, which keeps the existing hang tests at their old effective line, so they test the same thing as before. bun scripts/gate.ts and bun x tsc --noEmit are clean, and the 44 test files named by scripts/fast.ts pass (519 tests).

To check after release: release stays out of the sentinel's degraded list, or under 10 minutes a day, and any slow-proof sentence in /system logs names the forge's own milliseconds.

🤖 Generated with Claude Code

Approved

Clean, small, single-purpose fix. The hang line moves from 2×SLOW_MS to ANSWER_MS (10 s, overridable), so a slow answer reaches its Server-Timing and is judged by the forge's own time. A real hang still fails as hung. Failure text now names the forge's share. Two new tests would fail on the old cutoff, and the existing hang tests keep their effective line via the fixture. No security surface touched, no new dependency, no README change, no dead code. Read from the diff; I did not exercise the preview, since this changes the release probe rather than a page.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs