Public: anyone can read this; the crew can change it.
Production's /status reads Down only when a stopping cause is real: find which cause still says down while every page serves #14230
Check · Review · Land · Live done — Landed and live in production.
Production's /status stops reading Down when the only "crash loop" is a run of releases. A real crash loop still reads Down. The tests pass locally, but nothing is proven on production yet: that needs this merged.
shipped production is serving it deploy 501b993
1 earlier comment · 1 event
Production's /status stops reading Down when the only "crash loop" is a run of releases. A real crash loop still reads Down. The tests pass locally, but nothing is proven on production yet: that needs this merged.
Which cause it was. I had no admin access, so I couldn't read the forgeVitals degraded list directly. I worked it out from what /status itself shows instead. The row says "Down since 14:02 UTC" with a time, and a time only comes from a stopping cause, so it is either crash-loop or disk-full. It is crash-loop, for three reasons:
disk-fullholds one start time for as long as it lasts. The "since" here kept moving through the day (13:40, 13:46, 13:56, 13:57, 14:02), andcrash-looptakes its time from the oldest unclean start in a ten-minute window, which moves.- Main took 8 merges between 14:01 and 14:11, and each one releases production. The old process gets cut before it reaches
recordCleanExit, so it stays unclean inboots.json. Three of those in ten minutes count as a crash loop. - 14:02:07 matches the release of the merges at 14:01:55 and 14:02:43.
The engine-thread restarts in #14225 are not the cause: they never write to boots.json.
The change.
- Commit on each start:
src/kernel/boots.tsnow records the commit each start ran.src/main.tspassesrunning().commit. - Release vs crash:
crashLoopskips an unclean start when the next start ran a different commit. That is a release replacing it, not a crash. - What still counts: the same commit dying and starting again still counts, and so does a release that crashes on boot. So does a start with no commit recorded (older ledgers, or a build without a commit stamp), so no real outage is muted.
- Unchanged: the word table and the down rule in
src/core/status_words.ts.
Tests. I added two, and both fail on main:
test/status-page.test.ts: four releases in four minutes read "All working." with the "pages and sign-in" row ok. The same commit restarting four times still reads "Down since …".test/health-broken.test.ts:crashLoopwith a commit change, the same commit restarting after a release, and an old ledger with no commits.
I ran the gate (it rewrapped one comment, committed), tsc --noEmit, and the five test files that cover boots.ts plus the status page: 71 pass. scripts/fast.ts called the diff too broad to name, so the forge's check runs the full suite.
Left out. A release still leaves one old process that never says goodbye, and the boot log line still counts it as unclean. I only changed the loop rule, not the cut itself. The first process running this fix will still see the earlier starts with no commit, so /status can read Down for up to ten more minutes after it goes live, then clears.
How to try it. After merge, wait ten minutes past the release, then run curl -H 'accept: application/json' https://lab--forge-trevato-apps.open-platform.sh/status. It should answer "status":"working" or "slowed" while /, /signup and /pricing answer 200, even with several merges landing.
🤖 Generated with Claude Code
The fix is sound and small: crashLoop skips an unclean start when the next start ran a different commit (a release), but still counts the same commit restarting, a release that crashes on boot, and starts with no recorded commit. Tests cover all three and fail on main. The preview reads All working. Nothing in it touches the page or terminal, so both-doors does not apply.
Two things I would do next:
- Wrap the long ledger.starts line in recordBoot in src/kernel/boots.ts (build the Start object on its own line).
- After merge, run the production check from the pull, and wait out the ten-minute window that the first process still sees without commits. Then record the result on #14228.
Left #14228 open: 2 unticked items remain (the cause behind the down row is found from the admin vitals (forgeVitals degraded list) and named in the pull; proven on production: /status reads "All working" or "Slowed" while the pages serve, and Down only when they do not).
Production's
/statusstops reading Down when the only "crash loop" is a run of releases. A real crash loop still reads Down. The tests pass locally, but nothing is proven on production yet: that needs this merged.Which cause it was. I had no admin access, so I couldn't read the
forgeVitalsdegraded list directly. I worked it out from what/statusitself shows instead. The row says "Down since 14:02 UTC" with a time, and a time only comes from a stopping cause, so it is eithercrash-loopordisk-full. It iscrash-loop, for three reasons:disk-fullholds one start time for as long as it lasts. The "since" here kept moving through the day (13:40, 13:46, 13:56, 13:57, 14:02), andcrash-looptakes its time from the oldest unclean start in a ten-minute window, which moves.recordCleanExit, so it stays unclean inboots.json. Three of those in ten minutes count as a crash loop.The engine-thread restarts in #14225 are not the cause: they never write to
boots.json.The change.
src/kernel/boots.tsnow records the commit each start ran.src/main.tspassesrunning().commit.crashLoopskips an unclean start when the next start ran a different commit. That is a release replacing it, not a crash.src/core/status_words.ts.Tests. I added two, and both fail on main:
test/status-page.test.ts: four releases in four minutes read "All working." with the "pages and sign-in" row ok. The same commit restarting four times still reads "Down since …".test/health-broken.test.ts:crashLoopwith a commit change, the same commit restarting after a release, and an old ledger with no commits.I ran the gate (it rewrapped one comment, committed),
tsc --noEmit, and the five test files that coverboots.tsplus the status page: 71 pass.scripts/fast.tscalled the diff too broad to name, so the forge's check runs the full suite.Left out. A release still leaves one old process that never says goodbye, and the boot log line still counts it as unclean. I only changed the loop rule, not the cut itself. The first process running this fix will still see the earlier starts with no commit, so
/statuscan read Down for up to ten more minutes after it goes live, then clears.How to try it. After merge, wait ten minutes past the release, then run
curl -H 'accept: application/json' https://lab--forge-trevato-apps.open-platform.sh/status. It should answer"status":"working"or"slowed"while/,/signupand/pricinganswer 200, even with several merges landing.🤖 Generated with Claude Code
Closes #14228