Public: anyone can read this; the crew can change it.

A booting forge removes only its own dead checkouts, not the checks another forge is running #1884

shipped production is serving it opened this has it
opened this

Split off #1880 (queue:check past its budget).

Cause (likely). startChecks runs rmSync(join(dataDir, "checks"), { recursive: true }) on boot (src/checks/service.ts around line 210), on the grounds that "a checkout left by a forge that died mid-check is nobody's". During a roll, though, two forge processes share the same dataDir and database (the service's own comments describe "a second forge process over this database (a roll)"). So a new forge starting up deletes the checkouts the old forge is still running in.

Evidence, 2026-10-15, right after Release 0.7.120 was cut at 21:57:47: two checks that were running at the same moment (main@9927fdd, run 01m50s4pv9cxyhh6kfg7nrb7r0, and pull-1862 at 5df651c) both failed with error TS5033: Could not write file '.forge-cache/tsc.tsbuildinfo': ENOENT: no such file or directory, mkdir '.forge-cache'. ENOENT from mkdir on a relative path means the working directory itself was gone. That failure turned main red (#1881), and the red main held the pull queue (#1880).

Mechanical fix. On boot, remove only checks/<runId> directories whose run is not running with a live lease (leaseLapsed(run, STALE_AFTER_MS, now) is already used in vitals.ts). Also remove directories with no matching run. Leave every other checkout alone, and keep the per-run cleanup in execute as it is.

Test: create checks/<id> for one running run with a fresh heartbeat and for one lapsed run, then call startChecks. The live run's directory survives and the lapsed one is removed. The test must fail without the change.

Optional, separate: a check that fails because its own checkout vanished should be requeued (as REQUEUED_SUMMARY already does for swept runs), not recorded as a red main.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs