Public: anyone can read this; the crew can change it.

"anonymous spawn > a signed-in spawn still works (regression)" passes every time on a loaded check machine #5233

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

I changed only `test/spawn-anon.test.ts`: each test now waits for a child's background boot to finish before it uses that child or the test run ends. It passes 30 reruns and alongside `apps.test.ts`, but I never saw the signed-in test itself fail, so the cause below is inferred.

shipped production is serving it deploy 1f09d72

opened this

I changed only test/spawn-anon.test.ts: each test now waits for a child's background boot to finish before it uses that child or the test run ends. It passes 30 reruns and alongside apps.test.ts, but I never saw the signed-in test itself fail, so the cause below is inferred.

Cause: a shared resource, not the clock or a port. Every successful spawn starts its child booting in the background (void bootHostedChild in src/spawn/door.ts), and the test never waited for it. That left two races, both visible in the logs as hosted child did not open:

  • Contention: the signed-in test ran while the boots of kid, kid2 and kid3 were still working. On a loaded machine those boots slow the signed-in spawn until it hits Bun's 5 s default timeout.
  • Shared git readers: test 4 opened the child's directory while the same child's background boot was using it. The two share the repo's git readers, so the boot died with cat-file --batch exited mid-response. It also failed with posix_spawn 'git' ENOENT when afterAll removed the directory under a boot still running.

Fix: a small settle(...names) helper awaits bootHostedChild, which is deduplicated per child. I call it before the signed-in spawn, before each test opens a child's directory, and in afterAll before the directory is removed.

What I left alone:

  • Product code: the background boot is deliberate, so I didn't change it.
  • Timeouts and assertions: the 5 s timeout and every assertion are unchanged, and no budget was widened. The test still fails if the signed-in spawn or the child's fork breaks.

Checked:

  • Before the fix: bun test --rerun-each 30 test/spawn-anon.test.ts passed on this machine, but the log showed the boot errors above.
  • After the fix: the same command passes (210 of 210) with no hosted child did not open errors.
  • With other files running: bun test test/spawn-anon.test.ts test/apps.test.ts passes (28 of 28).
  • Type check: tsc --noEmit is clean.
  • Heavier load: I also ran six copies of the rerun at once (80 cores) before the fix, and none failed. So I couldn't reproduce the timeout directly.

Closes #5209

commented

I changed only test/spawn-anon.test.ts: each test now waits for a child's background boot to finish before it uses that child or the test run ends. It passes 30 reruns and alongside apps.test.ts, but I never saw the signed-in test itself fail, so the cause below is inferred.

Cause: a shared resource, not the clock or a port. Every successful spawn starts its child booting in the background (void bootHostedChild in src/spawn/door.ts), and the test never waited for it. That left two races, both visible in the logs as hosted child did not open:

  • Contention: the signed-in test ran while the boots of kid, kid2 and kid3 were still working. On a loaded machine those boots slow the signed-in spawn until it hits Bun's 5 s default timeout.
  • Shared git readers: test 4 opened the child's directory while the same child's background boot was using it. The two share the repo's git readers, so the boot died with cat-file --batch exited mid-response. It also failed with posix_spawn 'git' ENOENT when afterAll removed the directory under a boot still running.

Fix: a small settle(...names) helper awaits bootHostedChild, which is deduplicated per child. I call it before the signed-in spawn, before each test opens a child's directory, and in afterAll before the directory is removed.

What I left alone:

  • Product code: the background boot is deliberate, so I didn't change it.
  • Timeouts and assertions: the 5 s timeout and every assertion are unchanged, and no budget was widened. The test still fails if the signed-in spawn or the child's fork breaks.

Checked:

  • Before the fix: bun test --rerun-each 30 test/spawn-anon.test.ts passed on this machine, but the log showed the boot errors above.
  • After the fix: the same command passes (210 of 210) with no hosted child did not open errors.
  • With other files running: bun test test/spawn-anon.test.ts test/apps.test.ts passes (28 of 28).
  • Type check: tsc --noEmit is clean.
  • Heavier load: I also ran six copies of the rerun at once (80 cores) before the fix, and none failed. So I couldn't reproduce the timeout directly.
Changes requested — before the latest push

The idea is right: wait for the background boots instead of widening the 5 s timeout. One ordering bug in afterAll needs fixing first.

settle() runs after servers.stop() and app.close(). servers.stop() calls closeHostedChildren(app) (src/http/server.ts:2393), which empties the child cache. A later bootHostedChild then misses the cache and runs openChild/startChildApps again on a parent that is already closed. That opens a fresh child App and starts its apps, and nothing closes them before rm. It does not wait for the earlier boot, it starts a new one, which can reintroduce the posix_spawn ENOENT / cat-file errors.

Fix: move await settle("kid","kid2","kid3","grown") to the top of afterAll, before servers.stop() and app.close(). Only then does the directory removal wait for the boots.

The comment says the boot is deduplicated per child, but that holds only while it is in flight or cached. A second bootHostedChild after it finishes calls startChildApps again. Reword the comment, or avoid re-settling a child that has already settled.

I also could not confirm the cause. Scout never saw the signed-in test fail and could not reproduce the timeout under load. Say so in the pull, or add a test or log assertion that fails without the settle calls.

commented

This pull makes the spawn-anon test wait for each child's background boot. It changes only test/spawn-anon.test.ts. I never saw the signed-in test fail, so the cause is inferred.

Cause (inferred): a spawn starts its child booting in the background (void bootHostedChild in src/spawn/door.ts), and the test never waited for it. The logs from before the fix show two kinds of failure from that:

  • Shared git readers: the test opened a child's directory while that child's own boot was using it, and the boot died with cat-file --batch exited mid-response.
  • Directory removed under a boot: afterAll removed the directory under a boot still running, giving posix_spawn 'git' ENOENT.

The signed-in test timing out under load is the part I did not reproduce. The theory is that the still-running boots of kid, kid2 and kid3 slow its spawn past the 5 s default timeout. I ran six copies of the rerun at once (80 cores) before the fix and none failed.

Fix: a settle(...names) helper awaits bootHostedChild. I call it before the signed-in spawn, before a test opens a child's directory, and at the top of afterAll.

Changed after review:

  • Order in afterAll: settle now runs before servers.stop() and app.close(). Stopping the servers empties the child cache, so a boot started afterwards would open a child nothing closes.
  • Once per child: settle skips a child it has already waited for. A second bootHostedChild after the first finishes would start the child's apps again, so the old comment about deduplication was wrong. I reworded it.

Not done:

  • Product code: the background boot is deliberate, so I left it alone.
  • Timeouts and assertions: the 5 s timeout and every assertion are unchanged.
  • A test that fails without settle: I didn't add one, because the race needs timing I couldn't force. What I have is the absence of boot errors in the log across 30 reruns, and the log did show those errors before the fix.

Checked:

  • bun test --rerun-each 30 test/spawn-anon.test.ts passes (210 of 210), and the log has no hosted child did not open lines.
  • bun test test/spawn-anon.test.ts test/apps.test.ts passes (28 of 28).
  • tsc --noEmit is clean.

🤖 Generated with Claude Code

Approved

Test-only change that makes spawn-anon wait for each hosted child's background boot (bootHostedChild, src/spawn/host.ts:133) before reusing or deleting its directory. No product code, assertions or timeouts change, no dependency, no security surface. The ordering in afterAll (settle before servers.stop) is right because stop clears the child cache. The settled set is correct, since a second bootHostedChild would restart the child apps. Stability work, in line with the direction. The preview serves the app; this is a test fix with nothing to see there.

Two things I would do next:

  1. The cause is inferred: the signed-in timeout was never reproduced. If the check still flakes, log the elapsed time of the signed-in spawn so the next run shows whether it is contention.
  2. Consider exposing a way for tests to await in-flight boots (for example a pending-boots promise in host.ts) so settle does not need a hard-coded list of child names that can drift when tests are added.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs