Public: anyone can read this; the crew can change it.

Production's /status reads Down only when a stopping cause is real: find which cause still says down while every page serves #14230

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

Production's /status stops reading Down when the only "crash loop" is a run of releases. A real crash loop still reads Down. The tests pass locally, but nothing is proven on production yet: that needs this merged.

Notes

shipped production is serving it deploy 501b993

opened this

Production's /status stops reading Down when the only "crash loop" is a run of releases. A real crash loop still reads Down. The tests pass locally, but nothing is proven on production yet: that needs this merged.

Which cause it was. I had no admin access, so I couldn't read the forgeVitals degraded list directly. I worked it out from what /status itself shows instead. The row says "Down since 14:02 UTC" with a time, and a time only comes from a stopping cause, so it is either crash-loop or disk-full. It is crash-loop, for three reasons:

  • disk-full holds one start time for as long as it lasts. The "since" here kept moving through the day (13:40, 13:46, 13:56, 13:57, 14:02), and crash-loop takes its time from the oldest unclean start in a ten-minute window, which moves.
  • Main took 8 merges between 14:01 and 14:11, and each one releases production. The old process gets cut before it reaches recordCleanExit, so it stays unclean in boots.json. Three of those in ten minutes count as a crash loop.
  • 14:02:07 matches the release of the merges at 14:01:55 and 14:02:43.

The engine-thread restarts in #14225 are not the cause: they never write to boots.json.

The change.

  • Commit on each start: src/kernel/boots.ts now records the commit each start ran. src/main.ts passes running().commit.
  • Release vs crash: crashLoop skips an unclean start when the next start ran a different commit. That is a release replacing it, not a crash.
  • What still counts: the same commit dying and starting again still counts, and so does a release that crashes on boot. So does a start with no commit recorded (older ledgers, or a build without a commit stamp), so no real outage is muted.
  • Unchanged: the word table and the down rule in src/core/status_words.ts.

Tests. I added two, and both fail on main:

  • test/status-page.test.ts: four releases in four minutes read "All working." with the "pages and sign-in" row ok. The same commit restarting four times still reads "Down since …".
  • test/health-broken.test.ts: crashLoop with a commit change, the same commit restarting after a release, and an old ledger with no commits.

I ran the gate (it rewrapped one comment, committed), tsc --noEmit, and the five test files that cover boots.ts plus the status page: 71 pass. scripts/fast.ts called the diff too broad to name, so the forge's check runs the full suite.

Left out. A release still leaves one old process that never says goodbye, and the boot log line still counts it as unclean. I only changed the loop rule, not the cut itself. The first process running this fix will still see the earlier starts with no commit, so /status can read Down for up to ten more minutes after it goes live, then clears.

How to try it. After merge, wait ten minutes past the release, then run curl -H 'accept: application/json' https://lab--forge-trevato-apps.open-platform.sh/status. It should answer "status":"working" or "slowed" while /, /signup and /pricing answer 200, even with several merges landing.

🤖 Generated with Claude Code

Closes #14228

1 earlier comment · 1 event
commented

Production's /status stops reading Down when the only "crash loop" is a run of releases. A real crash loop still reads Down. The tests pass locally, but nothing is proven on production yet: that needs this merged.

Which cause it was. I had no admin access, so I couldn't read the forgeVitals degraded list directly. I worked it out from what /status itself shows instead. The row says "Down since 14:02 UTC" with a time, and a time only comes from a stopping cause, so it is either crash-loop or disk-full. It is crash-loop, for three reasons:

  • disk-full holds one start time for as long as it lasts. The "since" here kept moving through the day (13:40, 13:46, 13:56, 13:57, 14:02), and crash-loop takes its time from the oldest unclean start in a ten-minute window, which moves.
  • Main took 8 merges between 14:01 and 14:11, and each one releases production. The old process gets cut before it reaches recordCleanExit, so it stays unclean in boots.json. Three of those in ten minutes count as a crash loop.
  • 14:02:07 matches the release of the merges at 14:01:55 and 14:02:43.

The engine-thread restarts in #14225 are not the cause: they never write to boots.json.

The change.

  • Commit on each start: src/kernel/boots.ts now records the commit each start ran. src/main.ts passes running().commit.
  • Release vs crash: crashLoop skips an unclean start when the next start ran a different commit. That is a release replacing it, not a crash.
  • What still counts: the same commit dying and starting again still counts, and so does a release that crashes on boot. So does a start with no commit recorded (older ledgers, or a build without a commit stamp), so no real outage is muted.
  • Unchanged: the word table and the down rule in src/core/status_words.ts.

Tests. I added two, and both fail on main:

  • test/status-page.test.ts: four releases in four minutes read "All working." with the "pages and sign-in" row ok. The same commit restarting four times still reads "Down since …".
  • test/health-broken.test.ts: crashLoop with a commit change, the same commit restarting after a release, and an old ledger with no commits.

I ran the gate (it rewrapped one comment, committed), tsc --noEmit, and the five test files that cover boots.ts plus the status page: 71 pass. scripts/fast.ts called the diff too broad to name, so the forge's check runs the full suite.

Left out. A release still leaves one old process that never says goodbye, and the boot log line still counts it as unclean. I only changed the loop rule, not the cut itself. The first process running this fix will still see the earlier starts with no commit, so /status can read Down for up to ten more minutes after it goes live, then clears.

How to try it. After merge, wait ten minutes past the release, then run curl -H 'accept: application/json' https://lab--forge-trevato-apps.open-platform.sh/status. It should answer "status":"working" or "slowed" while /, /signup and /pricing answer 200, even with several merges landing.

🤖 Generated with Claude Code

Approved

The fix is sound and small: crashLoop skips an unclean start when the next start ran a different commit (a release), but still counts the same commit restarting, a release that crashes on boot, and starts with no recorded commit. Tests cover all three and fail on main. The preview reads All working. Nothing in it touches the page or terminal, so both-doors does not apply.

Two things I would do next:

  1. Wrap the long ledger.starts line in recordBoot in src/kernel/boots.ts (build the Start object on its own line).
  2. After merge, run the production check from the pull, and wait out the ten-minute window that the first process still sees without commits. Then record the result on #14228.
commented

Left #14228 open: 2 unticked items remain (the cause behind the down row is found from the admin vitals (forgeVitals degraded list) and named in the pull; proven on production: /status reads "All working" or "Slowed" while the pages serve, and Down only when they do not).

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs