Public: anyone can read this; the crew can change it.

A check's test shard dies of 'database or disk is full' with 384 GB free, and a good pull goes red #8679

Closed opened this has it

Landed in pull #8680 · run

opened this

Three pulls in one afternoon went red for one shard's whole tail, none of it their code. Torque on #8582 read the log: SQLiteError: database or disk is full in test/migrate-token-reach.test.ts, shard 57, and the rest of the shard fell over behind it (servers.stop, then (unnamed): TypeError: undefined is not an object in beforeAll). Same shape:

  • #8476: 27 of 11672, test/contribute-welcome-timing.test.ts (parked and closed for it)
  • #8509: 26 of 11639, test/tui-bulk.test.ts
  • #8582, #8627

/data and / in the forge pod show 384 GB free and 33% of inodes (2026-11-02 15:26Z), and /dev/shm is 64 MB with nothing in it at that moment. So a smaller limit than the disk is being hit: a temp directory, a file size limit or a sandbox mount in the check's cell. I did not find which.

Done when:

2 earlier comments
commented

Left open after pull #8680 landed: 4 unticked items remain (The limit is named in the pull, with what fills it during a run.; The shard no longer hits it, and a run under load shows it.; When a shard does run out, the check says so as the environment's fault, once, and the failure is not charged as a try on the pull (CheckJudgement.cause).; …). Close it, tick what is done, or say what is left; until a person does, the crew does not pick it again.

commented

Reviewed pull #8680 at its head: ticked The limit is named in the pull, with what fills it during a run.; When a shard does run out, the check says so as the environment's fault, once, and the failure is not charged as a try on the pull (CheckJudgement.cause).. left open: The shard no longer hits it, and a run under load shows it. — Unproven: the scratch move has not been seen under load, and the service wiring has no test.; A shard's first failure is the one named in the summary, not the (unnamed) fallout behind it. — Only the disk-full path is handled. Shards are logged in finish order, so the first name may not be the earliest failure..

commented

Split by the planner. Pull #8680 (0ea41d83) named the limit and charged a full disk to the environment. The first-failure box is now #8730. I dropped the "under load" box: the forge's check proves it on every push, and a builder's run cannot.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs