Sentinel: shared:01m23vygyffx5hg2f5vm8r74sr:21ydbn1vqn is past its budget: trevato/forge: "the host holds cells > cells are available" failed on 3 pulls in the last hour — the lane's red, not th… #6059

The cell probe no longer reports a slow machine as one that refuses cells. That should stop "the host holds cells > cells are available" from failing on a loaded check machine. The sandbox can't run cells, so I couldn't test the fix against a real cell. test/spawn-cell.test.ts passes (13 pass, 3 skipped) and the change is pushed. I didn't run tsc: it can't find the bun type definitions here, even after bun install.

Cause. cellsAvailable() in src/apps/contain.ts runs a quick bwrap check and gives it 5 seconds. When the check machine was busy, the timer killed the check, and the result was cached as "this host refuses the namespaces a cell needs". With FORGE_REQUIRE_CELLS=1 set, that turned into a hard failure.

Fix.

• A probe the timer killed is now retried, up to 3 attempts.
• The timeout is 20 seconds instead of 5.
• If every attempt times out, the reason now says "the namespace probe took over 20000 ms", not "refuses the namespaces".
• A real refusal from the host (nonzero exit, missing bwrap or socat, FORGE_CELLS=off) is still reported as a failure straight away, with no retry.

Left out. I added no test that forces the timeout path, because it needs a real bwrap, which the sandbox doesn't have. I didn't change the FORGE_REQUIRE_CELLS test itself, since failing when the check host can't hold cells is what it's there to do.

To try it, push and watch the forge's check on the cell lane. Once it's green on main, shared:01m23vygyffx5hg2f5vm8r74sr:21ydbn1vqn should leave the degraded list.

Closes #6058

The cell probe no longer reports a slow machine as one that refuses cells. That should stop "the host holds cells > cells are available" from failing on a loaded check machine. The sandbox can't run cells, so I couldn't test the fix against a real cell. test/spawn-cell.test.ts passes (13 pass, 3 skipped) and the change is pushed. I didn't run tsc: it can't find the bun type definitions here, even after bun install.

Cause. cellsAvailable() in src/apps/contain.ts runs a quick bwrap check and gives it 5 seconds. When the check machine was busy, the timer killed the check, and the result was cached as "this host refuses the namespaces a cell needs". With FORGE_REQUIRE_CELLS=1 set, that turned into a hard failure.

Fix.

• A probe the timer killed is now retried, up to 3 attempts.
• The timeout is 20 seconds instead of 5.
• If every attempt times out, the reason now says "the namespace probe took over 20000 ms", not "refuses the namespaces".
• A real refusal from the host (nonzero exit, missing bwrap or socat, FORGE_CELLS=off) is still reported as a failure straight away, with no retry.

Left out. I added no test that forces the timeout path, because it needs a real bwrap, which the sandbox doesn't have. I didn't change the FORGE_REQUIRE_CELLS test itself, since failing when the check host can't hold cells is what it's there to do.

To try it, push and watch the forge's check on the cell lane. Once it's green on main, shared:01m23vygyffx5hg2f5vm8r74sr:21ydbn1vqn should leave the degraded list.

The cell probe now retries every failed attempt, not just slow ones. This is a second guess at the cause: I couldn't see the check log, so I don't know why the probe fails on the check machine.

The sentinel still saw "cells are available" failing on 5 pulls after my first push. That means the failures weren't only the probe running out of time. A host that is briefly starved of processes or namespaces while other checks run would also fail the probe.

• Change: in src/apps/contain.ts, cellsAvailable() now makes up to 4 attempts. It pauses 0.5, 1 and 1.5 seconds between them, and only believes a refusal if every attempt fails. The earlier changes (20-second timeout per attempt, and the reason text for a timeout) are still in.
• Tests: test/spawn-cell.test.ts passes locally (3 skipped, since the sandbox can't run cells). I didn't run tsc: it can't find the bun type definitions here.
• Not changed: the FORGE_REQUIRE_CELLS=1 test still fails when the host can't hold cells. That is what it exists to do (added in #6031). If the check machine really can't create namespaces, this change won't fix it. The next failing log should then show the real refusal reason, such as the bwrap error line, and the fix would be on the machine.

I've addressed all three of Juniper's points and pushed. test/spawn-cell.test.ts passes (16 pass, 3 skipped); I haven't run tsc here.

• Testable retry: I split the retry out as askUntilBelieved(ask, pause) in src/apps/contain.ts. probe() calls it with the real bwrap runner. The tests need no bwrap and don't wait on real pauses.
• Tests added (in a new "the cell probe" block):

- Two failed attempts then a success give ok, with three asks and two pauses. This fails without the retry. - Three failed attempts return the last reason. - A host that answers first time is asked only once.

• Dead field: timedOut is now a local inside tryOnce, which returns the verdict directly.
• Cost of a real refusal: you're right that my first push's "no retry on a real refusal" was wrong. A genuine refusal now costs the pauses, 1.5 s in total (0.5 s + 1 s), before it's reported. The comment on askUntilBelieved says so. I didn't try to tell transient exits from real ones: without the failing log I'd be guessing which exit codes or messages count as transient.
• Worst case: each attempt now waits 10 s and there are 3 attempts, so a wedged host fails in about 31.5 s at worst, down from about 81 s.

The root cause is still unconfirmed, as before. If the lane stays red, the reason string in the next failing log should show what the host is actually refusing.