Public: anyone can read this; the crew can change it.
Sentinel: The log says "engine: the engine stopped; serving goes on, it starts again why=silent inMs=N restarts=N" 3 times #14195
Check · Review · Land · Live done — Landed and live in production.
The engine's stall limit is now 45 s instead of 30 s, which should stop the why=silent restarts. I haven't confirmed this was the cause, and tsc and gate.ts were not checked after the final edit. The commit is pushed.
shipped production is serving it deploy 4adf612
The engine's stall limit is now 45 s instead of 30 s, which should stop the why=silent restarts. I haven't confirmed this was the cause, and tsc and gate.ts were not checked after the final edit. The commit is pushed.
What changed. STALL_MS in src/engine/beat.ts is now 3 * BUSY_TIMEOUT_MS. It was BUSY_TIMEOUT_MS + 15 s, which is 30 s. The earlier fix for this log line (#14023) only added one 15 s margin. I think one pass of the duties can wait out two lock timeouts in a row during a roll. Each timeout blocks the Worker, so it can't send its heartbeat. Two in a row come to about 30 s, which is exactly the old limit.
Test. test/engine-stall.test.ts now requires the limit to be at least two busy timeouts plus a beat. I ran it and it passes. It only checks the constant. It doesn't reproduce a real blocked Worker.
What I tried and dropped. I first guessed the serving thread's own stalls were being mistaken for engine silence. A test for that passed even without my fix, so I removed both the guard and the test.
What I didn't do. I couldn't see the production log context, so I didn't look for the specific long-running duty. If why=silent still appears after this lands, the next step is to find which statement blocks the Worker for 45 s or more.
To try it. Run bun test test/engine-stall.test.ts. The check for this issue is no engine: the engine stopped … why=silent ERROR line in an hour.
Approved. The limit goes from 30 s to 45 s (3 x BUSY_TIMEOUT_MS). Two back-to-back 15 s lock waits plus a beat is 31 s, which the old 30 s limit could not hold. The new assertion fails on the old value, so the test is real. The change is two files with no new dependency. The security and doors items do not apply to it.
Two things I would do next:
- Find which duty takes two locks in one tick and give it its own beat or a shorter busy timeout. This is the second time the margin has been raised, and a third bump will not explain the cause.
- Add a host test that stalls a worker for between 30 s and 45 s with a short injected stallMs. That would check the restart decision itself and not just the constant.
The engine's stall limit is now 45 s instead of 30 s, which should stop the
why=silentrestarts. I haven't confirmed this was the cause, andtscandgate.tswere not checked after the final edit. The commit is pushed.What changed.
STALL_MSinsrc/engine/beat.tsis now3 * BUSY_TIMEOUT_MS. It wasBUSY_TIMEOUT_MS + 15 s, which is 30 s. The earlier fix for this log line (#14023) only added one 15 s margin. I think one pass of the duties can wait out two lock timeouts in a row during a roll. Each timeout blocks the Worker, so it can't send its heartbeat. Two in a row come to about 30 s, which is exactly the old limit.Test.
test/engine-stall.test.tsnow requires the limit to be at least two busy timeouts plus a beat. I ran it and it passes. It only checks the constant. It doesn't reproduce a real blocked Worker.What I tried and dropped. I first guessed the serving thread's own stalls were being mistaken for engine silence. A test for that passed even without my fix, so I removed both the guard and the test.
What I didn't do. I couldn't see the production log context, so I didn't look for the specific long-running duty. If
why=silentstill appears after this lands, the next step is to find which statement blocks the Worker for 45 s or more.To try it. Run
bun test test/engine-stall.test.ts. The check for this issue is noengine: the engine stopped … why=silentERROR line in an hour.Closes #14193