Concurrent check installs race on the shared bun cache ("Fail extracting tarball"): good pulls look red and stall the merge queue #5094

A check's install no longer turns a pull red when the package manager fails on its own (Fail extracting tarball, EINTEGRITY, ECONNRESET). The new logic is in src/checks/install.ts and is wired into runTier in src/checks/service.ts.

What was actually racing. The failed runs on #4814 (and #4781 and #4911) ran in a cell. A cell has its own home, so its bun cache was already private, and the cache race doesn't explain those failures. The more likely cause is a registry download cut short; the retry below covers that, but I haven't confirmed the root cause. The real shared cache is in uncontained checks, which have no cell: they all used the forge's own ~/.bun cache, across every repository at once.

• Cache: a cell keeps its private cache. I did not share one across cells, because a pull's install scripts could then write into a cache another check reads. An uncontained check now uses its repository's own cache (<data>/check-install/<repo>/bun), and installs into it take turns:

- withLock creates a lock file only if none exists, so this works across forge processes. - Waiters in the same process queue in memory, so a released lock passes to the next one at once. - A lock left by a dead process, or older than 15 minutes, is taken over.

• Retry: an install whose output has one of the three errors is run once more after 2 s. A failure without one is the code's own and is reported straight away.
• Infrastructure:

- Normal check: if the retry fails the same way, the run is cancelled with INFRA_SUMMARY (failed: infrastructure — …; queued again) and queued once more, the same pattern as a vanished cluster Job. - Merge-queue gate: the gate reports it as interrupted, so the queue counts it as cut off, not red. - If the re-run fails the same way: that run fails with a summary starting infrastructure:. I capped re-runs at one so a registry outage can't requeue forever; it does show as a failure on the pull.

• Not done: a job check runs install inside its own pod and gets no lock or retry. Deploys and previews still share the forge's ~/.bun cache when they run without a cell.

Tests: test/check-install.test.ts covers:

• 8 parallel installs on one cache, 20 rounds, zero failures. Each install creates the same directory, so an overlap shows as a failure; without the lock, 140 of the 160 collided.
• Taking over a dead holder's lock, waiting for a live one, and cancelling the wait.
• The retry and infrastructure rules.
• End to end through startChecks: an install that fails once passes on its retry, and one that always fails is queued again, then fails saying infrastructure.

tsc is clean, and the new file, the related check suites and the five cross files pass (221 tests). docs/APPS.md describes the behaviour.

To try it: put "install": "sh install.sh" in a check's forge.json, with an install.sh that prints error: Fail extracting tarball for "x" and exits 1. The run log shows the retry, then the run is queued again.

🤖 Generated with Claude Code

Closes #5093

A check's install no longer turns a pull red when the package manager fails on its own (Fail extracting tarball, EINTEGRITY, ECONNRESET). The new logic is in src/checks/install.ts and is wired into runTier in src/checks/service.ts.

What was actually racing. The failed runs on #4814 (and #4781 and #4911) ran in a cell. A cell has its own home, so its bun cache was already private, and the cache race doesn't explain those failures. The more likely cause is a registry download cut short; the retry below covers that, but I haven't confirmed the root cause. The real shared cache is in uncontained checks, which have no cell: they all used the forge's own ~/.bun cache, across every repository at once.

• Cache: a cell keeps its private cache. I did not share one across cells, because a pull's install scripts could then write into a cache another check reads. An uncontained check now uses its repository's own cache (<data>/check-install/<repo>/bun), and installs into it take turns:

- withLock creates a lock file only if none exists, so this works across forge processes. - Waiters in the same process queue in memory, so a released lock passes to the next one at once. - A lock left by a dead process, or older than 15 minutes, is taken over.

• Retry: an install whose output has one of the three errors is run once more after 2 s. A failure without one is the code's own and is reported straight away.
• Infrastructure:

- Normal check: if the retry fails the same way, the run is cancelled with INFRA_SUMMARY (failed: infrastructure — …; queued again) and queued once more, the same pattern as a vanished cluster Job. - Merge-queue gate: the gate reports it as interrupted, so the queue counts it as cut off, not red. - If the re-run fails the same way: that run fails with a summary starting infrastructure:. I capped re-runs at one so a registry outage can't requeue forever; it does show as a failure on the pull.

• Not done: a job check runs install inside its own pod and gets no lock or retry. Deploys and previews still share the forge's ~/.bun cache when they run without a cell.

Tests: test/check-install.test.ts covers:

• 8 parallel installs on one cache, 20 rounds, zero failures. Each install creates the same directory, so an overlap shows as a failure; without the lock, 140 of the 160 collided.
• Taking over a dead holder's lock, waiting for a live one, and cancelling the wait.
• The retry and infrastructure rules.
• End to end through startChecks: an install that fails once passes on its retry, and one that always fails is queued again, then fails saying infrastructure.

tsc is clean, and the new file, the related check suites and the five cross files pass (221 tests). docs/APPS.md describes the behaviour.

To try it: put "install": "sh install.sh" in a check's forge.json, with an install.sh that prints error: Fail extracting tarball for "x" and exits 1. The run log shows the retry, then the run is queued again.

🤖 Generated with Claude Code

A check's install no longer turns a pull red when the package manager fails on its own (Fail extracting tarball, EINTEGRITY, ECONNRESET). The new logic is in src/checks/install.ts and is wired into runTier in src/checks/service.ts.

What was actually racing. The failed runs on #4814, #4781 and #4911 ran in a cell. A cell has its own home, so its bun cache was already private, and a cache race doesn't explain those failures. The more likely cause is a registry download that was cut short. The retry below covers that, but I haven't confirmed the root cause. The cache that really was shared is in uncontained checks, which have no cell: they all used the forge's own ~/.bun cache, across every repository at once.

• Cache: a cell keeps its private cache. I did not share one cache across cells, because a pull's install scripts could then write into a cache another check reads. An uncontained check now uses its repository's own cache (<data>/check-install/<repo>/bun), and installs into it take turns:

- withLock creates a lock file only if none exists, so the lock holds across forge processes. - Waiters in the same process queue in memory, so a released lock passes straight to the next one. - A lock left by a dead process, or older than 15 minutes, is taken over.

• Retry: an install whose output contains one of the three errors runs once more after 2 s. A failure without one of them comes from the code itself and is reported at once.
• Infrastructure:

- Normal check: if the retry fails the same way, the run is cancelled with INFRA_SUMMARY (failed: infrastructure — …; queued again) and queued once more. This follows the pattern already used for a cluster Job that vanishes. - Merge-queue gate: the gate reports it as interrupted, so the queue counts it as cut off, not red. - Re-run fails the same way: that run fails with a summary starting infrastructure:. Re-runs are capped at one so a registry outage can't requeue forever, so in that case the pull does show a failure.

• Not done:

- A job check runs its install inside its own pod and gets no lock or retry. - Deploys and previews that run without a cell still share the forge's ~/.bun cache.

• Pillar: the issue's triage note says it waits behind the Later gate until a pillar is named. Naming the pillar is for a maintainer, so this push changes no code.

Tests: test/check-install.test.ts covers:

• 8 parallel installs on one cache, 20 rounds, with no failures. Each install creates the same directory, so any overlap shows up as a failure; without the lock, 140 of the 160 installs collided.
• Taking over a dead holder's lock, waiting for a live one, and cancelling the wait.
• The retry and infrastructure rules.
• End to end through startChecks: an install that fails once passes on its retry, and one that always fails is queued again, then fails with an infrastructure summary.

tsc is clean, and bun test test/check-install.test.ts passes (10 tests). The forge check on 2626c973 passed. docs/APPS.md describes the new behaviour.

To try it: put "install": "sh install.sh" in a check's forge.json, with an install.sh that prints error: Fail extracting tarball for "x" and exits 1. The run log shows the retry, and then the run is queued again.

🤖 Generated with Claude Code

Closes #5093