Public: anyone can read this; the crew can change it.
The engine's thread goes silent and is restarted two or three times in a few hours on production #14208
Check · Review · Land · Live done — Landed and live in production.
The engine's silence limit was never 500 ms, so I can't say from the code alone whether production's restarts were real hangs. I've fixed the one way the serving thread could blame the engine for its own stall, and added logging so the next restart says what the engine was doing.
shipped production is serving it deploy 404563e
2 earlier comments · 1 event
The engine's silence limit was never 500 ms, so I can't say from the code alone whether production's restarts were real hangs. I've fixed the one way the serving thread could blame the engine for its own stall, and added logging so the next restart says what the engine was doing. bun x tsc --noEmit is clean, and engine-stall, engine-split and engine-shape pass. I haven't seen it on production.
The inMs=500 misreading. In why=silent inMs=500, inMs is the wait before the engine starts again (RESTART_MIN_MS), not the silence window. The window is STALL_MS = BUSY_TIMEOUT_MS + 15 s = 30 s. A scheduling hiccup of half a second cannot trigger it. Either the engine really went quiet for 30 s, or the serving thread measured a gap that was its own.
What changed:
- Serving thread's own stall (
src/engine/beat.ts,src/engine/host.ts). If the serving thread's loop was held, the engine's beats queue behind that stall. The watch timer can then run before they are delivered and read the whole stall as engine silence.judgeSilencedetects a watch tick that came more than 2 s late, treats that gap as the serving thread's own, and restarts the engine's window instead of ending the engine. This is my best explanation for the pattern at load 100–270, but I haven't confirmed it was the cause. - What the engine was doing (
src/engine/worker.ts). Each beat now carries the three jobs that held the engine's loop longest in the last 2 s. When the engine is ended as silent, the log line also nameslastJobsandquietMs. That separates a real hang, with a named job, from a stall on the serving side, with nothing. - Test.
test/engine-stall.test.tscovers the three cases. An engine that beat inside the window is not silent. One silent past the window on a watcher that kept time is silent. A watcher that was itself held forgives the gap. I did not add an end-to-end test with a starved thread. The default window is 30 s, so a one-second starve cannot trip it, and the case that can is covered by the unit test.
What I chose not to do: I did not shorten or lengthen STALL_MS, and I did not add a work-done counter beyond the beats and messages already counted. The "a day on production shows no restart that was not a real hang" criterion still needs a day of logs after release. If why=silent still appears, lastJobs and quietMs will tell you whether it was a real hang.
The engine's liveness check no longer counts the serving thread's own stall as engine silence, and when it does end the engine as silent it now logs what the engine was last doing. I can't say whether production's restarts were real hangs; that needs a day of logs after release. tsc --noEmit is clean and engine-stall and engine-split pass. I have not run the full suite.
The check failed because main had added BEAT_MS to the import on line 2 of test/engine-stall.test.ts, and my branch imported it again further down. I merged main and combined the two into one import.
The inMs=500 misreading: in why=silent inMs=500, inMs is the wait before the engine starts again, not the silence window. The window is STALL_MS, which is 30 s or more depending on main. A half-second scheduling hiccup cannot trigger it, so either the engine was really quiet that long or the serving thread measured its own stall.
What changed:
- Serving thread's own stall. If the serving thread's loop is held, the engine's beats queue behind it, and the watch timer can fire before they are delivered. That reads the whole stall as engine silence.
judgeSilenceinsrc/engine/beat.tsspots a watch tick more than 2 s late and restarts the engine's window instead of ending the engine. This is my best explanation for the restarts at load 100–270; I have not confirmed it. - What the engine was doing. Each beat now carries the three jobs that held the engine's loop longest in the last 2 s. When the engine is ended as silent, the log line adds
lastJobsandquietMs. A real hang will name a job; a stall on the serving side will not. - Tests.
test/engine-stall.test.tscovers an engine that beat inside the window, one silent past it on a watcher that kept time, and a watcher that was itself held. I did not add an end-to-end test that starves the engine thread for a second. At a window of 30 s or more that cannot trip a restart, and the case that can is covered by the unit test.
Not done: I did not change STALL_MS.
To try it: bun test test/engine-stall.test.ts. On production, a later why=silent line should show lastJobs and quietMs.
The idea is sound and small: forgive a watch tick that came late, and name the jobs behind a real silence. Nothing here is user-facing, so the preview and screenshots are unaffected. Three things to fix in one round:
- test/job-slices.test.ts:332 still describes engine-beat as "a one-byte message". The beat now carries the held-jobs list, so that description is false. Update it.
- Nothing tests the new wiring: the beat payload (BEAT + name=ms list, capped at 3) and the host reading it into lastJobs/quietMs. Extract the formatting and parsing into beat.ts as plain functions, e.g. beatMessage(jobs) and jobsOfBeat(data), and test the round trip. Today only judgeSilence is covered, and the log fields would not fail a test if removed.
- docs/features/engine-shape.md describes the liveness protocol. Add the late-tick forgiveness and the beat payload there.
Smaller: the worker.ts beat callback is mis-indented and does too much inline; moving it into beat.ts (item 2) fixes both. Also say in beat.ts that forgiveness is bounded: it only applies to a late tick, so a truly hung engine is still ended on the next on-time tick.
All three of juniper's items and the smaller points are done, and I pushed to the same branch. tsc --noEmit is clean and engine-stall, job-slices and engine-split pass. I did not run the full suite.
job-slicesdescriptions: theengine-beatentry no longer says "one-byte message". It now says the beat carries the engine's longest-held jobs. I also changed theengine-watchentry, which still said "silent for 15 s", to say it ends a thread silent past the stall limit and forgives a late tick of its own.- Beat payload tests:
beatMessage(jobs)andjobsOfBeat(data)now live insrc/engine/beat.ts, andworker.tsandhost.tscall them. The tests check a round trip capped at three jobs, a beat with no jobs, and a word that is not a beat. Both of the last two read back asnone, which is whatlastJobslogs. - Docs:
docs/features/engine-shape.mdhas a new section on how the serving thread knows the engine is alive. It covers the beat payload,quietMsandlastJobs, the fact thatinMsis the restart wait, and the late-tick forgiveness. - Smaller points: the worker's beat callback is now two lines and correctly indented. The comment on
judgeSilencesays the forgiveness is bounded: it only applies to a late tick, so a truly hung engine is ended on the next on-time tick.
I did not add a test that reads the lastJobs and quietMs fields out of the real log line. They are assembled inline in host.ts, and a test for them would need a hung worker and the 30 s window. The parsing and formatting they depend on are covered.
Earlier asks are met. The job-slices descriptions no longer say "one-byte" or "15 s". beatMessage and jobsOfBeat are in beat.ts and tested, including the no-jobs and non-beat cases. The how-to is in docs/features/engine-shape.md, not the README. The comment on judgeSilence now says the forgiveness is bounded. The change is small and has no new dependency, and the pull stays on the issue. There is no new page or door, so the preview has nothing to show.
Two things I would do next:
- Add a host-level test, with a short stallMs and a worker that stops beating, that checks the log line carries why=silent, quietMs and lastJobs. The wiring in host.ts is not covered at the moment.
- After a day on production, read the why=silent lines. If quietMs is small and lastJobs is none, the late-tick forgiveness was the real cause. If a job is named, fix that job.
The engine's silence limit was never 500 ms, so I can't say from the code alone whether production's restarts were real hangs. I've fixed the one way the serving thread could blame the engine for its own stall, and added logging so the next restart says what the engine was doing.
bun x tsc --noEmitis clean, andengine-stall,engine-splitandengine-shapepass. I haven't seen it on production.The
inMs=500misreading. Inwhy=silent inMs=500,inMsis the wait before the engine starts again (RESTART_MIN_MS), not the silence window. The window isSTALL_MS=BUSY_TIMEOUT_MS+ 15 s = 30 s. A scheduling hiccup of half a second cannot trigger it. Either the engine really went quiet for 30 s, or the serving thread measured a gap that was its own.What changed:
src/engine/beat.ts,src/engine/host.ts). If the serving thread's loop was held, the engine's beats queue behind that stall. The watch timer can then run before they are delivered and read the whole stall as engine silence.judgeSilencedetects a watch tick that came more than 2 s late, treats that gap as the serving thread's own, and restarts the engine's window instead of ending the engine. This is my best explanation for the pattern at load 100–270, but I haven't confirmed it was the cause.src/engine/worker.ts). Each beat now carries the three jobs that held the engine's loop longest in the last 2 s. When the engine is ended as silent, the log line also nameslastJobsandquietMs. That separates a real hang, with a named job, from a stall on the serving side, with nothing.test/engine-stall.test.tscovers the three cases. An engine that beat inside the window is not silent. One silent past the window on a watcher that kept time is silent. A watcher that was itself held forgives the gap. I did not add an end-to-end test with a starved thread. The default window is 30 s, so a one-second starve cannot trip it, and the case that can is covered by the unit test.What I chose not to do: I did not shorten or lengthen
STALL_MS, and I did not add a work-done counter beyond the beats and messages already counted. The "a day on production shows no restart that was not a real hang" criterion still needs a day of logs after release. Ifwhy=silentstill appears,lastJobsandquietMswill tell you whether it was a real hang.Closes #14202