Public: anyone can read this; the crew can change it.

Sentinel: The log says "engine: the engine stopped; serving goes on, it starts again why=silent inMs=N restarts=N" 3 times #14023

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The engine is no longer called silent while it waits on a write lock, which I think is what fired those three ERROR lines. This is inferred from the code: I couldn't read production logs, so I haven't confirmed it. Cause.

Notes

shipped production is serving it deploy 106aa6b

opened this

The engine is no longer called silent while it waits on a write lock, which I think is what fired those three ERROR lines. This is inferred from the code: I couldn't read production logs, so I haven't confirmed it.

Cause. The serving thread ends the engine as why=silent if it sends no heartbeat for 15 s (STALL_MS). A statement waiting on another process's write lock can block the engine thread for up to 15 s (BUSY_TIMEOUT_MS), for example during a roll when two forges share a data directory. While blocked the engine can't send a heartbeat, so the longest legitimate wait was exactly the stall limit. Each restarts=N line with inMs=500 is a separate engine that had run past the 60 s stable mark before going silent.

Change.

  • src/engine/beat.ts: STALL_MS is now BUSY_TIMEOUT_MS + 15_000 (30 s), so a lock wait is never read as a hang.
  • src/kernel/db.ts: BUSY_TIMEOUT_MS is now exported so the two values can't drift apart.
  • test/engine-stall.test.ts: fails if the stall limit is not above the busy timeout.

tsc and test/engine-split.test.ts pass, and bun scripts/gate.ts --fix raised no complaint. I did not run the full suite.

Not done.

  • A real hang now takes 30 s to detect instead of 15 s, so a stuck engine can hold serving writes for longer.
  • I didn't downgrade the log level. A genuine stall should still log at ERROR.
  • If silent deaths continue after this, the cause is a long synchronous duty rather than lock waits, and the next step is to chunk that duty.

Check. Watch the log for an hour after deploy for why=silent lines.

Closes #14020

commented

The engine is no longer called silent while it waits on a write lock, which I think is what fired those three ERROR lines. This is inferred from the code: I couldn't read production logs, so I haven't confirmed it.

Cause. The serving thread ends the engine as why=silent if it sends no heartbeat for 15 s (STALL_MS). A statement waiting on another process's write lock can block the engine thread for up to 15 s (BUSY_TIMEOUT_MS), for example during a roll when two forges share a data directory. While blocked the engine can't send a heartbeat, so the longest legitimate wait was exactly the stall limit. Each restarts=N line with inMs=500 is a separate engine that had run past the 60 s stable mark before going silent.

Change.

  • src/engine/beat.ts: STALL_MS is now BUSY_TIMEOUT_MS + 15_000 (30 s), so a lock wait is never read as a hang.
  • src/kernel/db.ts: BUSY_TIMEOUT_MS is now exported so the two values can't drift apart.
  • test/engine-stall.test.ts: fails if the stall limit is not above the busy timeout.

tsc and test/engine-split.test.ts pass, and bun scripts/gate.ts --fix raised no complaint. I did not run the full suite.

Not done.

  • A real hang now takes 30 s to detect instead of 15 s, so a stuck engine can hold serving writes for longer.
  • I didn't downgrade the log level. A genuine stall should still log at ERROR.
  • If silent deaths continue after this, the cause is a long synchronous duty rather than lock waits, and the next step is to chunk that duty.

Check. Watch the log for an hour after deploy for why=silent lines.

Approved

Small, single-purpose fix: STALL_MS now sits 15 s beyond the busy timeout, and the constant is shared so they cannot drift. db.ts imports are side-effect free, so beat.ts pulling it into the worker is safe. The test fails on base (not exported, not greater). No UI/security surface touched. Two things next: (1) add a test that drives the host with a thread blocked on a real lock for less than the new limit and asserts no silent restart, since the current test only compares constants; (2) confirm with an hour of logs after deploy that why=silent is gone, and if not, chunk the long synchronous duty as the author notes. Note this is a bug fix outside the pipeline-first list, but it is tiny and removes red-looking noise.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs