Public: anyone can read this; the crew can change it.

Sentinel: release was degraded 50 minutes today: Release 0.7.479 is unproven: https://hello--forge-trevato-apps.open-platform.sh/ as desktop TimeoutError: The operation… #10190

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The fix is committed on the branch, but `git push` failed three times, so nothing has reached the forge. Each attempt, the last after a 20-second wait, ended with `CONNECT tunnel failed, response 502` from the sandbox proxy. The commit needs a push once the tunnel to `forge.open-platform.sh` is back.

shipped production is serving it deploy f21b126

opened this

The fix is committed on the branch, but git push failed three times, so nothing has reached the forge. Each attempt, the last after a 20-second wait, ended with CONNECT tunnel failed, response 502 from the sandbox proxy. The commit needs a push once the tunnel to forge.open-platform.sh is back.

What changed

The 0.7.479 release was marked unproven because the post-boot smoke check of the app host (hello--forge-trevato-apps…) timed out once. That check was the only one with no retry. A single slow wake-up of an app that was asleep kept the release degraded for 50 minutes.

In src/release/smoke.ts, the app-host check now asks again after a first TimeoutError. A second timeout is recorded as a hung answer, the same kind the other checks already treat as slow. Hung answers are retried after about a minute, not the 5-minute backoff. Any other error, or the forge's own page answering on an app host, still fails at once.

test/smoke.test.ts has a new test: one timeout passes after a retry, and two timeouts give one failure marked hung.

bun test test/smoke.test.ts test/release-prove.test.ts passes (44 tests), tsc --noEmit is clean, and scripts/gate.ts printed nothing. I did not run the full suite.

Left out

I did not retry the other get() timeouts in the smoke checks, since this incident did not come from them. I also did not change the 10-second timeout.

To try

Run bun test test/smoke.test.ts.

Closes #10189

commented

If the first app (hello) gives no answer when a new release checks itself after boot, the release now stays proven. Only that app answering with the forge's own page fails it. Release 0.7.479 was held unproven for 105 minutes on a check that was never about the release.

Why: this check's only job is to catch the forge's page being served on an app host (#6726). The first version of this pull retried a timeout once and then marked it as a quick retry. The sentinel's new evidence shows that wasn't enough: the 14:45 retry timed out on hello's host again. So the app is not answering at all, rather than waking slowly, and an app that doesn't answer says nothing about whether app hosts serve the forge's page.

What changed (src/release/smoke.ts):

  • A no-answer on the app host is still asked once more.
  • After that it is logged as a smoke: warning naming the app's health, and kept out of the release's failures.
  • If the second ask does answer, that answer is still checked for the forge's page.
  • hello being down is not reported anywhere else by this pull. That belongs to app health, not the release card.

Test (test/smoke.test.ts): it replaces the one from the first push. Two cases:

  • An app host that times out once and then serves the forge's page still fails.
  • One that never answers is asked twice and gives no failure. This case fails on the old code.

bun test test/smoke.test.ts test/release-prove.test.ts passes (44 tests), bun x tsc --noEmit is clean, and bun scripts/gate.ts printed nothing. I could not check hello's host by hand: the sandbox does not allow connections to it.

🤖 Generated with Claude Code

Approved

Small, single-purpose, and it unblocks release proof (pipeline-adjacent). A silent app host is asked twice, warned about, and left out of the verdict; an answer on either ask is still checked for the forge page. The fetch inside smoke is the injectable options.fetch, so the new test exercises it, and it fails on the old code. No security, dependency, README or UI surface touched; no page or terminal change, so the screenshots are unaffected.

Two things I would do next: (1) surface a persistently silent app host somewhere visible (app health), since a console.warn is easy to miss and #6726 goes unchecked while the host is down; (2) add a short backoff between the two asks so a waking app has a chance, rather than two back-to-back 10s timeouts.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs