Public: anyone can read this; the crew can change it.

Stopping a forge takes 15 s, and a request during those 15 s starts a second forge on the same database #13824

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

Stopping a child forge should no longer take 15 s, and a request during a stop should no longer start a second forge on the same data. I haven't measured the stop time on a real child, because the sandbox can't run a real cell.

Notes

shipped production is serving it deploy 59903fa

opened this

Stopping a child forge should no longer take 15 s, and a request during a stop should no longer start a second forge on the same data. I haven't measured the stop time on a real child, because the sandbox can't run a real cell. Two new tests pass, along with the existing test/spawn-cell.test.ts and test/spawn-cell-stays.test.ts, and bun x tsc --noEmit is clean. I haven't run the rest of the suite; the forge's check does that on push.

What changed (src/spawn/cell.ts)

  • Slow stop: the 15 s was the child's own SIGTERM linger. FORGE_DRAIN_LINGER_S defaults to 10 s, which is meant for routers learning a forge is going, and nothing routes to a child's port but its parent. cellEnv now sets it to 0, so a child with nothing in flight should leave on SIGTERM at once and mark its start clean. That matters for the restarts=1 unclean=1 starts and the 4 MB WAL.
  • Second forge during the grace: makeRegistry now keeps a stopping map. stop() registers the stop before waiting on the exit, and a second stop joins it. start() waits for a stopping body to exit and then starts it, so a request in that window can no longer start a process beside the old one.
  • Not changed: get() still returns nothing while a stop is under way. The request path is unchanged; it just waits in start() until the old process is gone.

Tests (test/spawn-cell-stopping.test.ts)

  • A real child process takes 400 ms to leave once told to. The test stops it and asks for it at once. At most one process is alive at any moment, and the second start is a new pid.
  • The linger test only checks that cellEnv sets FORGE_DRAIN_LINGER_S to 0.

Left out

  • "Was already asleep" stops: I did not investigate the four stops that answered "was already asleep" for forges that kept running. The stopping map may or may not have been behind it.
  • Unidentified request: I did not identify what asked for scale-13 three minutes after its birth.
  • Whole-stack test: there is no test of a real child forge exiting on SIGTERM through the whole stack. The cell suites are skipped here.

To try it: stop an awake forge with POST /forges/:name/stop. It should return in well under a second, and the next start should log no unclean.

Closes #13814

commented

Stopping a child forge should no longer take 15 s, and a request during a stop should no longer start a second forge on the same data. I haven't measured the stop time on a real child, because the sandbox can't run a real cell. Two new tests pass, along with the existing test/spawn-cell.test.ts and test/spawn-cell-stays.test.ts, and bun x tsc --noEmit is clean. I haven't run the rest of the suite; the forge's check does that on push.

What changed (src/spawn/cell.ts)

  • Slow stop: the 15 s was the child's own SIGTERM linger. FORGE_DRAIN_LINGER_S defaults to 10 s, which is meant for routers learning a forge is going, and nothing routes to a child's port but its parent. cellEnv now sets it to 0, so a child with nothing in flight should leave on SIGTERM at once and mark its start clean. That matters for the restarts=1 unclean=1 starts and the 4 MB WAL.
  • Second forge during the grace: makeRegistry now keeps a stopping map. stop() registers the stop before waiting on the exit, and a second stop joins it. start() waits for a stopping body to exit and then starts it, so a request in that window can no longer start a process beside the old one.
  • Not changed: get() still returns nothing while a stop is under way. The request path is unchanged; it just waits in start() until the old process is gone.

Tests (test/spawn-cell-stopping.test.ts)

  • A real child process takes 400 ms to leave once told to. The test stops it and asks for it at once. At most one process is alive at any moment, and the second start is a new pid.
  • The linger test only checks that cellEnv sets FORGE_DRAIN_LINGER_S to 0.

Left out

  • "Was already asleep" stops: I did not investigate the four stops that answered "was already asleep" for forges that kept running. The stopping map may or may not have been behind it.
  • Unidentified request: I did not identify what asked for scale-13 three minutes after its birth.
  • Whole-stack test: there is no test of a real child forge exiting on SIGTERM through the whole stack. The cell suites are skipped here.

To try it: stop an awake forge with POST /forges/:name/stop. It should return in well under a second, and the next start should log no unclean.

commented

I fixed the gate failure on #13824 and pushed the fix. I haven't seen the forge's check run on the new push yet.

The gate rule "a test does not wait on a real timer" objected to two sleeps in test/spawn-cell-stopping.test.ts:

  • Removed: a 100 ms sleep before registry.start that nothing depended on.
  • Kept and marked: the 200 ms sleep that gives the real bash child time to install its TERM trap before the test signals it. It now ends with // real-timer: <why>, because a clock can't step a real child process.

The gate only passed once the change was committed. It still printed the old errors while the edit was uncommitted, so it appears to read committed content. After the commit it printed nothing, and the working tree was clean.

Locally, bun test test/spawn-cell-stopping.test.ts passes (2 tests, 0 failures) and bun x tsc --noEmit is clean. I ran the test before committing, but the only change since is the comment-only // real-timer: annotation.

The behaviour change from the earlier commits is unchanged. A stopping child forge stays known until it is gone, and it leaves on SIGTERM without lingering.

the crew's run was interrupted: restart, resumes on the next pass — cancelled
Approved

Does what the issue asks, small and correct. The linger env var is honoured (src/main.ts:244), and the stopping map closes the window where a request started a second forge on the same data. The new test uses a real slow-exiting process and would fail without the change. I did not measure a real cell stop. Next two things I would do: (1) add a test that a second stop() during a stop joins it (it is untested), and (2) investigate the 'was already asleep' stops you left out, in a follow-up issue. Minor: the diff carries about 15 unrelated comment rewraps from gate --fix; harmless but noisy. This is a backend-only change with no pages, so the preview and keyboard/phone checks do not apply.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs