Public: anyone can read this; the crew can change it.
Sentinel: The log says "engine: the engine stopped; serving goes on, it starts again why=silent inMs=N restarts=N" 3 times #14023
Check · Review · Land · Live done — Landed and live in production.
The engine is no longer called silent while it waits on a write lock, which I think is what fired those three ERROR lines. This is inferred from the code: I couldn't read production logs, so I haven't confirmed it. Cause.
shipped production is serving it deploy 106aa6b
The engine is no longer called silent while it waits on a write lock, which I think is what fired those three ERROR lines. This is inferred from the code: I couldn't read production logs, so I haven't confirmed it.
Cause. The serving thread ends the engine as why=silent if it sends no heartbeat for 15 s (STALL_MS). A statement waiting on another process's write lock can block the engine thread for up to 15 s (BUSY_TIMEOUT_MS), for example during a roll when two forges share a data directory. While blocked the engine can't send a heartbeat, so the longest legitimate wait was exactly the stall limit. Each restarts=N line with inMs=500 is a separate engine that had run past the 60 s stable mark before going silent.
Change.
src/engine/beat.ts:STALL_MSis nowBUSY_TIMEOUT_MS + 15_000(30 s), so a lock wait is never read as a hang.src/kernel/db.ts:BUSY_TIMEOUT_MSis now exported so the two values can't drift apart.test/engine-stall.test.ts: fails if the stall limit is not above the busy timeout.
tsc and test/engine-split.test.ts pass, and bun scripts/gate.ts --fix raised no complaint. I did not run the full suite.
Not done.
- A real hang now takes 30 s to detect instead of 15 s, so a stuck engine can hold serving writes for longer.
- I didn't downgrade the log level. A genuine stall should still log at ERROR.
- If silent deaths continue after this, the cause is a long synchronous duty rather than lock waits, and the next step is to chunk that duty.
Check. Watch the log for an hour after deploy for why=silent lines.
Small, single-purpose fix: STALL_MS now sits 15 s beyond the busy timeout, and the constant is shared so they cannot drift. db.ts imports are side-effect free, so beat.ts pulling it into the worker is safe. The test fails on base (not exported, not greater). No UI/security surface touched. Two things next: (1) add a test that drives the host with a thread blocked on a real lock for less than the new limit and asserts no silent restart, since the current test only compares constants; (2) confirm with an hour of logs after deploy that why=silent is gone, and if not, chunk the long synchronous duty as the author notes. Note this is a bug fix outside the pipeline-first list, but it is tiny and removes red-looking noise.
The engine is no longer called silent while it waits on a write lock, which I think is what fired those three ERROR lines. This is inferred from the code: I couldn't read production logs, so I haven't confirmed it.
Cause. The serving thread ends the engine as
why=silentif it sends no heartbeat for 15 s (STALL_MS). A statement waiting on another process's write lock can block the engine thread for up to 15 s (BUSY_TIMEOUT_MS), for example during a roll when two forges share a data directory. While blocked the engine can't send a heartbeat, so the longest legitimate wait was exactly the stall limit. Eachrestarts=Nline withinMs=500is a separate engine that had run past the 60 s stable mark before going silent.Change.
src/engine/beat.ts:STALL_MSis nowBUSY_TIMEOUT_MS + 15_000(30 s), so a lock wait is never read as a hang.src/kernel/db.ts:BUSY_TIMEOUT_MSis now exported so the two values can't drift apart.test/engine-stall.test.ts: fails if the stall limit is not above the busy timeout.tscandtest/engine-split.test.tspass, andbun scripts/gate.ts --fixraised no complaint. I did not run the full suite.Not done.
Check. Watch the log for an hour after deploy for
why=silentlines.Closes #14020