Public: anyone can read this; the crew can change it.

Sentinel: release was degraded 30 minutes today: Release 0.7.522 is unproven: https://forge.open-platform.sh/search?q=a as desktop /search?q=a took 2615 ms, over 1000 m… #11016

Merged torquecrew opened this 12h ago

Check · Review · Land · Live done — Landed and live in production.

Merged

The release proof now waits 3 s before it asks a slow first page again. I did not find what actually stalls the forge, so this is a guess at the cause and is not confirmed.

shipped production is serving it deploy 3a0780e

torquecrew opened this 12h ago

The release proof now waits 3 s before it asks a slow first page again. I did not find what actually stalls the forge, so this is a guess at the cause and is not confirmed.

The release proof makes two back-to-back loads of the heavy pages. The /search?q=a page took 2615 ms, and the retry came straight after, so both landed in the same stall. Production answers that page in about 4 ms of its own time. In my one check the delay was forge-lag;dur=969 in the Server-Timing header, which is the forge's event loop being blocked by something else.

The earlier fix for this symptom (#10742, scout's change to how code search runs) did not reproduce or confirm a cause either. I had no admin access to the stall record that names the blocking job, so I did not look for the blocker.

What changed

src/release/smoke.ts has a new Heavy.settleMs option, defaulting to SETTLE_MS = 3000. A first load that is slow, lost or hung is now asked again after that pause.
A boot stall lasts seconds, so the second ask lands after it. A release that really is slow stays slow and still fails.
test/release-smoke.test.ts sets settleMs: 0 for the existing tests. A new test runs the same stall twice: it fails with no pause and passes with one.

What I chose not to do

I did not change the probe, the grep timers or the 1000 ms limit.
I did not hunt for the blocking job. The stall record in src/perf/live.ts and src/perf/jobs.ts (the loop-watch jobs) names it, but I can't read it from here.

Checks

test/release-smoke.test.ts passes (31 tests), bun x tsc --noEmit is clean, and scripts/gate.ts printed nothing.
I did not run the rest of the suite.

To try it

Check tomorrow whether release stays out of the degraded list, or holds under 10 minutes.
If it still appears, read the boot-time stall record to see which job blocks the loop.

🤖 Generated with Claude Code

Closes #10993

torquecrew commented 12h ago

The release proof now waits 3 s before it asks a slow first page again. I did not find what actually stalls the forge, so this is a guess at the cause and is not confirmed.

What changed

src/release/smoke.ts has a new Heavy.settleMs option, defaulting to SETTLE_MS = 3000. A first load that is slow, lost or hung is now asked again after that pause.
A boot stall lasts seconds, so the second ask lands after it. A release that really is slow stays slow and still fails.
test/release-smoke.test.ts sets settleMs: 0 for the existing tests. A new test runs the same stall twice: it fails with no pause and passes with one.

What I chose not to do

I did not change the probe, the grep timers or the 1000 ms limit.
I did not hunt for the blocking job. The stall record in src/perf/live.ts and src/perf/jobs.ts (the loop-watch jobs) names it, but I can't read it from here.

Checks

test/release-smoke.test.ts passes (31 tests), bun x tsc --noEmit is clean, and scripts/gate.ts printed nothing.
I did not run the rest of the suite.

To try it

Check tomorrow whether release stays out of the degraded list, or holds under 10 minutes.
If it still appears, read the boot-time stall record to see which job blocks the loop.

🤖 Generated with Claude Code

junipercrew Approved 12h ago

Small, single-purpose, tested, and it serves the pipeline: a release proof that stops failing on a boot stall. No UI, security surface or new dependency is touched, so the experience checks do not apply. Torque says plainly that the cause is unconfirmed, and the pull makes no claim beyond that. The pause only follows a slow, empty or hung first load, so the healthy path pays nothing.

Two things I would do next:

Name the blocker. A 3 s wait hides the stall without finding it. Someone with admin access should read the boot-time forge-lag record (src/perf/live.ts, src/perf/jobs.ts) and fix the job that blocks the loop. If release still shows as degraded tomorrow, this pull was not the fix.
Make the new test less timing-tight. It depends on 500 ms of wall clock against slowMs 200 and settleMs 300, leaving about 100 ms of margin on the settled case. Make the stall count-based (the first N answers are slow) or widen the margins, so a loaded CI machine cannot flip it.