Public: anyone can read this; the crew can change it.
/status says down only when pages or sign-in actually fail, not for any broken cause #14222
Check · Review · Land · Live done — Landed and live in production.
/status now says "down" for "pages and sign-in" only when the forge's own health is not ok or a broken cause actually stops it serving: the database not answering, a crash loop, or a full disk refusing writes (database, crash-loop, disk-full).
shipped production is serving it deploy 501b993
2 earlier comments · 2 events
/status now says "down" for "pages and sign-in" only when the forge's own health is not ok or a broken cause actually stops it serving: the database not answering, a crash loop, or a full disk refusing writes (database, crash-loop, disk-full). Every other broken cause makes its row "slow", in words from the fixed table. The sentence follows the rows, so "Down since …" appears only when a row is down. The page, its JSON and the terminal screen all read statusOf, so this one change covers all three. I haven't checked production yet; that needs this change deployed (see "How to try it").
Two causes of the false "down" in src/core/status_words.ts:
- The input to
statusOf:statusWordspassedok: health.ok && v.ok.v.okis false for any broken vital, so even after fixingstatusOfthe page would still have said Down. It now passes onlyhealth.ok, andstatusOfdecides from the causes which ones stop serving. - The cause rule in
statusOf: every broken cause outside the building lane raised the row to down. A newSTOPSlist now holds the three causes that stop serving. Memory under its floor, a red disk (disk-red), a red main (red:*, on the building row) and unknown keys now read as slow, each in its row's fixed words or the generic line. No cause text is ever copied.
Production's 13:40 cause is most likely memory under its floor. That's an inference from which broken causes exist; the status JSON doesn't name the cause. This repo's tests already use that exact case: an app with no free memory. It now reads "Slowed: pages are slower than usual." and no longer "Down since …".
Tests: these are in test/status-page.test.ts, the existing status test, rather than a new test/status-words.test.ts. I changed three existing tests because they asserted the old rule:
- "a broken cause with any key is down" now reads "only a broken cause that stops serving is down". It checks that
crash-loop,disk-fullanddatabaseare down, and thatmemory,disk-red,red:7and an unknown key are slow with the right row and sentence. This test fails on main. - The real failing-vital test (memory under its floor) now expects slowed in the words, the page, the JSON and the terminal.
- In the secrets test, a broken cause alone is now slowed; Down. still shows, with no copied text, when the forge's own health is not ok.
The gate, tsc, and test/status-page.test.ts, status-badge, tell-status, next-steps-ask-status and plan-notice all pass. scripts/fast.ts couldn't narrow this change to a short list, so the forge's full check covers the rest.
Left out: the forge's own health is still false for any platform incident or a stale guard, which also reads Down. That's left as the issue scoped it.
How to try it: after this deploys, curl -H 'accept: application/json' <app>/status while a non-stopping cause such as memory under its floor is raised. It should answer "status":"slowed" with "pages and sign-in" slow, matching the 200s the pages give.
🤖 Generated with Claude Code
The rule is right and well tested, but I can't confirm it works. The preview of this head answers /status with Down since 13:54 UTC., row pages and sign-in down, while its pages answer 200. Production says the same. Fix first: say which condition makes the preview down. If it is health.ok false or a real crash-loop/disk-full, show that in the pull, with the cause named in the reply. If it is another cause, the rule still has a hole. Then:
- STOPS lists
databaseanddb, but vitals.ts emits neither as a cause (it emitsdisk-full,disk-red,disk-floor,crash-loop). Drop the names nothing emits, or point to where the database failure becomes a cause. The test loop assertsdatabaseis down and so pins a cause that never occurs. - KNOWN duplicates the list the old inline
starts(...)held; keep it next to PLAIN, and do not adddatabase/dbto it unless they exist. - Box 4 needs the proof on production after deploy; it is not shown yet.
/status now says "down" for "pages and sign-in" only when the forge's own health is not ok or a broken cause stops serving: a crash loop or a full disk (crash-loop, disk-full). Every other broken cause makes its row slow, in its plain words from the fixed table. The sentence follows the rows: "Down since …" appears only when a row is down. The page, its JSON and the terminal screen all read statusOf in src/core/status_words.ts, so one change covers all three.
There are two fixes:
- Health input:
statusWordspassedok: health.ok && v.ok, andv.okis false for any broken vital. It now passes onlyhealth.ok, andstatusOfdecides from the causes which ones stop serving. - Cause rule: every broken cause used to make the row down. Now only
crash-loopanddisk-fulldo. Memory under its floor, a red disk, a red main and unknown keys make their row slow.
Why the preview of this head still says down (14:02–14:04 UTC): it is the crash-loop cause, not a hole in the rule. I can't read boots.json or the admin vitals from here, so this is inferred from outside, three ways:
/api/v1/healthanswersok:truethere, sohealth.okis true.- A down result with a time but with
health.oktrue can only come from a broken cause inSTOPS. - The
sincemoves along a 10-minute window, which is the crash-loop rule insrc/kernel/boots.ts(LOOP_WINDOW_MS): the oldest of 3 or more unclean starts in the last 10 minutes. Production said 13:40:06 at 13:49, then 13:54:39 at 14:02–14:04. Adisk-fulltime would stay fixed.
So the forge is restarting without clean exits several times in 10 minutes (compare #14208), while it answers 200 in about 5 ms between restarts. The issue classes a crash loop as stopping serving, and I kept that rule. Whether a crash loop that still serves should read as "slow, restarting" is the owner's call.
Review items:
databaseanddbare gone fromSTOPSand from the test: vitals never emit them, and a database that does not answer is/healthz's 503.KNOWNnow sits next toPLAIN, without those names.- The test checks that only
crash-loopanddisk-fullare down.memory,disk-red,red:7and an unknown key are slow, each with its row and sentence; this fails on main. The real memory-under-floor app test now expects "Slowed: pages are slower than usual." on the words, page, JSON and terminal.
Not proven yet: the last box, production after deploy. Production still says down because of that crash loop. It should read "All working." or "Slowed" about 10 minutes after the last unclean start, once this is merged. The release hold on trevato/forge ("production is rolling") was not affected: this only updates the pull's branch.
The gate, bun x tsc --noEmit, and test/status-page.test.ts, status-badge, tell-status, next-steps-ask-status and plan-notice all pass. scripts/fast.ts couldn't name a short test list for this change, so the forge's full check covers the rest.
To try it: curl -H 'accept: application/json' <app>/status. Unless a crash loop or full disk is raised, it should match the pages: "All working." or "Slowed: …".
🤖 Generated with Claude Code
Earlier asks met: database/db are out of STOPS and the test, and KNOWN sits beside PLAIN. The rule is now small and readable: only crash-loop and disk-full (or health not ok) are down; every other broken cause is slow in fixed words, and the sentence follows the rows. Tests cover each case and the secret-not-copied case. The preview still reads Down since 13:54 UTC; the builder attributes that to a live crash-loop, which STOPS says is down by the issue. I could not confirm the cause from outside. Two things I would do next: (1) Non-blocking: decide whether a crash loop that still answers 200 should read slow, restarting rather than down. (2) Non-blocking: once deployed, check /status against production after the 10 minute crash-loop window clears.
Left #14219 open: 1 unticked item remain (proven on production: /status reads the same as what the pages do).
/status now says "down" for "pages and sign-in" only when the forge's own health is not ok or a broken cause actually stops it serving: the database not answering, a crash loop, or a full disk refusing writes (
database,crash-loop,disk-full). Every other broken cause makes its row "slow", in words from the fixed table. The sentence follows the rows, so "Down since …" appears only when a row is down. The page, its JSON and the terminal screen all readstatusOf, so this one change covers all three. I haven't checked production yet; that needs this change deployed (see "How to try it").Two causes of the false "down" in
src/core/status_words.ts:statusOf:statusWordspassedok: health.ok && v.ok.v.okis false for any broken vital, so even after fixingstatusOfthe page would still have said Down. It now passes onlyhealth.ok, andstatusOfdecides from the causes which ones stop serving.statusOf: every broken cause outside the building lane raised the row to down. A newSTOPSlist now holds the three causes that stop serving. Memory under its floor, a red disk (disk-red), a red main (red:*, on the building row) and unknown keys now read as slow, each in its row's fixed words or the generic line. No cause text is ever copied.Production's 13:40 cause is most likely memory under its floor. That's an inference from which broken causes exist; the status JSON doesn't name the cause. This repo's tests already use that exact case: an app with no free memory. It now reads "Slowed: pages are slower than usual." and no longer "Down since …".
Tests: these are in
test/status-page.test.ts, the existing status test, rather than a newtest/status-words.test.ts. I changed three existing tests because they asserted the old rule:crash-loop,disk-fullanddatabaseare down, and thatmemory,disk-red,red:7and an unknown key are slow with the right row and sentence. This test fails on main.The gate,
tsc, andtest/status-page.test.ts,status-badge,tell-status,next-steps-ask-statusandplan-noticeall pass.scripts/fast.tscouldn't narrow this change to a short list, so the forge's full check covers the rest.Left out: the forge's own health is still false for any platform incident or a stale guard, which also reads Down. That's left as the issue scoped it.
How to try it: after this deploys,
curl -H 'accept: application/json' <app>/statuswhile a non-stopping cause such as memory under its floor is raised. It should answer"status":"slowed"with "pages and sign-in" slow, matching the 200s the pages give.🤖 Generated with Claude Code
Closes #14219