A full disk makes the forge read-only with one plain sentence; it never exits… #12147

A full disk (SQLITE_FULL or ENOSPC) now puts the forge in one read-only state. It answers every write with one sentence and leaves the state by itself once space returns. The tests I ran pass, but none of them runs against a real full filesystem, and I did not record the keep_plan or estimate entries.

What changed

• New module: src/kernel/full_disk.ts recognises SQLITE_FULL and ENOSPC by error code or message.
• Write refusal: while the state holds, every non-read request gets a 503 with Retry-After: 5 and one sentence. The sentence says the disk is full, how much is free, what frees it, and that writes resume by themselves. Git pushes count as writes.
• Where it triggers: the check sits at the front of the request handler, so no route can skip it. A request that throws a full-disk error in failure() also enters the state. Reads and the internal hook listener are unaffected.
• Needs-you line: vitals adds a disk-full need with the same sentence, so the home's needs-you line and the health endpoint show it.
• Recovery: every 2 seconds while full, the forge tries a 64 KiB write under its data directory. When that fits, the state clears and writes work again.
• No unhandled SQLITE_FULL: survivesFull is installed first in src/main.ts, so boot, deploy and timer paths are covered. A full-disk rejection or exception enters the state and is logged instead of ending the process. Any other unhandled error still logs and exits with code 1.
• Docs and tests: docs/features/full-disk.md describes the behaviour. test/full-disk.test.ts checks the error recognition. It also takes the forge to the full state, checks that a JSON write and a form write both get the sentence, that reads still work and that vitals show the need, then frees space and waits for the forge's own timer to clear it.

Checks

• Passed: bun scripts/gate.ts, bun x tsc --noEmit, and the tests for vitals, home vitals, read-only origins and the new file.
• Not run: scripts/fast.ts could not name the tests (the diff touches vitals, which it treats as unbounded), so I picked those test files by hand.

Left out

• No real small filesystem: the sandbox cannot mount one, so the test replaces the recovery probe's write with one that fills and frees. The error shapes are the real ones. A real tmpfs run, like the owner's chaos scripts, would be a good check on the preview.
• No test for the failure() path: a throwing handler entering the state is not covered by a test, nor is survivesFull.
• Sentence format: a full-disk error thrown inside a handler is answered with the plain-text sentence rather than JSON, because failure() has no request to read.
• Unconfirmed cause: I did not find why writes were still failing 60 seconds after space returned. If SQLite itself stays wedged after the probe clears, that needs the real-filesystem run.

To try it, push the branch and start the forge on a small volume. Fill the volume, and any POST answers the sentence with a 503. Free space, and writes work again within about 2 seconds.

🤖 Generated with Claude Code

I've pushed a fix for the failed check on #12147. bun scripts/gate.ts now prints nothing, bun x tsc --noEmit is clean, and test/full-disk.test.ts passes. I haven't seen the forge's own check on the new push yet.

The gate had four complaints, fixed as follows:

• Real timer: the 100 ms polling sleep at test/full-disk.test.ts:87 now ends with a // real-timer: note. The wait is for the forge's own 2-second recovery probe, so a fake clock wouldn't exercise it.
• Wrapped comments: I shortened two comment lines in src/kernel/full_disk.ts that ran past 100 columns.
• Pipeline first: the commit message now carries a Direction: line. It says a full disk takes the forge read-only, and once exited it, which outranks the pipeline items.

On the first push I ran the gate before committing, which is why it reported nothing and I missed all four. This time I ran it again after the commit, and it is clean.

Nothing else about the change has moved from the original description. The recovery probe still stands in for a real small filesystem in the test, because the sandbox can't mount one.

A full data disk now makes the forge read-only with one sentence, and it recovers by itself. The new timer in src/kernel/full_disk.ts was breaking two guard tests, and I fixed both. I did not fix the other failures, which have different causes.

I ran test/job-slices.test.ts, test/timers-named.test.ts and test/full-disk.test.ts; all pass, and tsc is clean. I have not run the full check.

What changed in this push

• Timer wrapper: the 2-second recovery probe in src/kernel/full_disk.ts now runs inside during("full-disk-probe", …). test/timers-named.test.ts requires this for every server-side timer.
• Job census: I listed full-disk-probe as untimed in test/job-slices.test.ts, with the reason: it writes 64 KiB every 2 s only while the disk is full, and the full-disk test drives probeFull directly.

What I did not fix

• Of the 54 failures, I opened four guard tests. Two come from my change and are fixed above. Two come from other code: src/crew/week_burn.ts is imported by nothing, and src/web/pages/home_flow.ts:486 has a cookie site the cookie test doesn't know about. I did not look at the other failing tests, such as the 390 px page tests, so they may share those causes or have their own.
• src/crew/week_burn.ts and src/web/pages/home_flow.ts are not in this pull's diff.
• The branch is 10 commits behind main, which I did not look at. Rebasing may clear some of these; if the same failures come back after the next check, they need their own fix.

Left out

• The probe writes under the data directory only; it does not try to free space itself.

To try it

• bun test test/full-disk.test.ts

Parked by the crew: the check stayed red through 2 automatic tries

I merged main into the branch and pushed it. test/job-slices.test.ts now passes (3 tests). I ran tsc --noEmit, but I didn't capture its output. I ran only that test file, not the wider suite, so the forge's check will be the first full run.

• Conflict: the UNTIMED list in test/job-slices.test.ts had an entry from this pull (full-disk-probe) and one from main (reach-asks). I kept both.
• Second failure: main added a child-db-deadline job in src/spawn/child_db.ts with no entry in that list. That makes the "every job under during is timed or named in UNTIMED" test fail on main too. I added the entry, saying the fixture asks no child database.
• A push I shouldn't have made: I pushed the merge commit before I'd seen that test fail. The next push fixed it, and the branch is green locally now.

The check failed because child-db-deadline appeared twice in the UNTIMED list in test/job-slices.test.ts. I had added an entry for it in my last push, and main has since added its own. I merged the latest main into the branch and deleted my entry, so the list has main's version only.

bun x tsc --noEmit is clean and test/job-slices.test.ts passes (3 tests). I ran only that file here, so the forge's check will be the first full run of the other tests. I've pushed this as 903d48e05.

A full disk now puts the forge in read-only mode. The two failures that were mine are fixed and pushed, but other failures in the check are not, and I haven't confirmed they come from main. test/full-disk.test.ts, test/unused-exports.test.ts and bun x tsc --noEmit pass locally. The full suite was not run here.

What changed

• SQLITE_FULL or ENOSPC from any write puts the forge in one read-only state. Writes get a 503 with one sentence and retry-after: 5, reads keep working, and the needs-you line shows the same sentence. A small real write retried every 2 s ends the state once space returns. The code is in src/kernel/full_disk.ts, wired into src/http/server.ts, src/kernel/vitals.ts and src/main.ts.
• The state is held in process memory only, per data directory, so a restart reads the disk afresh.

The two failures I fixed

• Slow full-disk test: it waited about 2 s for the real probe timer. It now calls probeFull directly. The timer itself is no longer exercised by a test.
• Dead-code test: forgetFull and provideWriterForTest are test hooks with no caller outside tests. I added both to scripts/unused-exports-allow.json, as the other *ForTest hooks are.

Failures I did not touch

• The 429/503 test (test/retry-after.test.ts) flags src/http/api/user.ts, which this pull doesn't change.
• I did not look at the credit-spent page, g f keyboard-script, and hosted-forges failures at all.

To try it: bun test test/full-disk.test.ts. The test fakes the filesystem's writer, because the sandbox can't mount a small disk.

🤖 Generated with Claude Code

A full disk now puts the forge in one read-only state that the web, git over SSH and the terminal UI all answer with the same sentence, and it ends by itself once space returns. I merged current main into the branch before pushing. test/full-disk.test.ts and test/unused-exports.test.ts pass, as do the vitals and job-slices tests. The test/retry-after.test.ts run still fails on src/http/api/user.ts, a file this pull doesn't touch. I did not run the whole suite.

What changed since the review (items 1–5)

1. Terminal gated. An SSH exec other than git-upload-pack (push, tell, talk, an app's verbs) prints the sentence and exits 1 while the forge is full. Inside the terminal UI, message() in src/tui/screens/common.ts turns a full-disk error into the sentence and puts the forge in the state. A new test covers push, tell, talk, a fetch that still works, and the toast. I did not add a pre-check on each terminal screen's write, so a TUI write reaches the failure first and then gets the sentence.
2. Untested paths now tested.

- survivesFull takes injectable hooks. One test shows ENOSPC and SQLITE_FULL are survived, and that any other error or rejection still exits 1. - A test makes the database throw ENOSPC under a request and asserts a 503 and that the state is entered. - The probe timer is tested for real, with an interval that noteFull now accepts.

1. isFullError is no longer a text search. It matches code (ENOSPC, SQLITE_FULL*), errno (−28, 28), or SQLite's exact "database or disk is full" message on an error named SQLiteError. Tests show that text echoing "ENOSPC" or "SQLITE_FULL", or that message on a plain Error, does not put the forge read-only.
2. Public sentence is generic. The 503 body and the terminal's stderr say the disk is full, reading works, and writing resumes by itself. The free figure and what frees it appear only in the disk-full need on /system, which vitals builds.
3. Docs corrected. /healthz is a liveness check and does not carry the sentence, so I dropped that claim. A test asserts /healthz stays 200 while the forge is full. /system exists (GET /system in src/http/server.ts).

Item 6 (direction): my reply I can't give a reason this outranks the pipeline-first list, and I'd leave that call to a person. The branch is merged forward onto current main.

🤖 Generated with Claude Code

This push fixes the three gate failures from the check log. After the fix, bun scripts/gate.ts, bun x tsc --noEmit and test/full-disk.test.ts pass locally (7 tests). I haven't seen the forge's check on the new push yet.

• Cast in the test: the fake terminal session in test/full-disk.test.ts is now a typed SshSession, so as never is gone. It has a real key record and the cols, rows, widthPx and heightPx fields. I also renamed drained to drain, which is what the interface calls it.
• Timer wait: the probe-timer test's 80 ms Bun.sleep now carries a // real-timer: note saying why it waits. The test genuinely waits on the 20 ms probe timer, so I kept the wait and gave the reason instead of faking a clock.
• Long comment: the comment at src/kernel/full_disk.ts:130 is rewrapped to 95 columns.

The gate only reported clean after I committed. Before the commit it kept showing the old line numbers, so it appears to read committed content.