Public: anyone can read this; the crew can change it.

The engine's thread goes silent and is restarted two or three times in a few hours on production #14208

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

The engine's silence limit was never 500 ms, so I can't say from the code alone whether production's restarts were real hangs. I've fixed the one way the serving thread could blame the engine for its own stall, and added logging so the next restart says what the engine was doing.

Notes

shipped production is serving it deploy 404563e

opened this

The engine's silence limit was never 500 ms, so I can't say from the code alone whether production's restarts were real hangs. I've fixed the one way the serving thread could blame the engine for its own stall, and added logging so the next restart says what the engine was doing. bun x tsc --noEmit is clean, and engine-stall, engine-split and engine-shape pass. I haven't seen it on production.

The inMs=500 misreading. In why=silent inMs=500, inMs is the wait before the engine starts again (RESTART_MIN_MS), not the silence window. The window is STALL_MS = BUSY_TIMEOUT_MS + 15 s = 30 s. A scheduling hiccup of half a second cannot trigger it. Either the engine really went quiet for 30 s, or the serving thread measured a gap that was its own.

What changed:

  • Serving thread's own stall (src/engine/beat.ts, src/engine/host.ts). If the serving thread's loop was held, the engine's beats queue behind that stall. The watch timer can then run before they are delivered and read the whole stall as engine silence. judgeSilence detects a watch tick that came more than 2 s late, treats that gap as the serving thread's own, and restarts the engine's window instead of ending the engine. This is my best explanation for the pattern at load 100–270, but I haven't confirmed it was the cause.
  • What the engine was doing (src/engine/worker.ts). Each beat now carries the three jobs that held the engine's loop longest in the last 2 s. When the engine is ended as silent, the log line also names lastJobs and quietMs. That separates a real hang, with a named job, from a stall on the serving side, with nothing.
  • Test. test/engine-stall.test.ts covers the three cases. An engine that beat inside the window is not silent. One silent past the window on a watcher that kept time is silent. A watcher that was itself held forgives the gap. I did not add an end-to-end test with a starved thread. The default window is 30 s, so a one-second starve cannot trip it, and the case that can is covered by the unit test.

What I chose not to do: I did not shorten or lengthen STALL_MS, and I did not add a work-done counter beyond the beats and messages already counted. The "a day on production shows no restart that was not a real hang" criterion still needs a day of logs after release. If why=silent still appears, lastJobs and quietMs will tell you whether it was a real hang.

Closes #14202

2 earlier comments · 1 event
commented

The engine's silence limit was never 500 ms, so I can't say from the code alone whether production's restarts were real hangs. I've fixed the one way the serving thread could blame the engine for its own stall, and added logging so the next restart says what the engine was doing. bun x tsc --noEmit is clean, and engine-stall, engine-split and engine-shape pass. I haven't seen it on production.

The inMs=500 misreading. In why=silent inMs=500, inMs is the wait before the engine starts again (RESTART_MIN_MS), not the silence window. The window is STALL_MS = BUSY_TIMEOUT_MS + 15 s = 30 s. A scheduling hiccup of half a second cannot trigger it. Either the engine really went quiet for 30 s, or the serving thread measured a gap that was its own.

What changed:

  • Serving thread's own stall (src/engine/beat.ts, src/engine/host.ts). If the serving thread's loop was held, the engine's beats queue behind that stall. The watch timer can then run before they are delivered and read the whole stall as engine silence. judgeSilence detects a watch tick that came more than 2 s late, treats that gap as the serving thread's own, and restarts the engine's window instead of ending the engine. This is my best explanation for the pattern at load 100–270, but I haven't confirmed it was the cause.
  • What the engine was doing (src/engine/worker.ts). Each beat now carries the three jobs that held the engine's loop longest in the last 2 s. When the engine is ended as silent, the log line also names lastJobs and quietMs. That separates a real hang, with a named job, from a stall on the serving side, with nothing.
  • Test. test/engine-stall.test.ts covers the three cases. An engine that beat inside the window is not silent. One silent past the window on a watcher that kept time is silent. A watcher that was itself held forgives the gap. I did not add an end-to-end test with a starved thread. The default window is 30 s, so a one-second starve cannot trip it, and the case that can is covered by the unit test.

What I chose not to do: I did not shorten or lengthen STALL_MS, and I did not add a work-done counter beyond the beats and messages already counted. The "a day on production shows no restart that was not a real hang" criterion still needs a day of logs after release. If why=silent still appears, lastJobs and quietMs will tell you whether it was a real hang.

commented

The engine's liveness check no longer counts the serving thread's own stall as engine silence, and when it does end the engine as silent it now logs what the engine was last doing. I can't say whether production's restarts were real hangs; that needs a day of logs after release. tsc --noEmit is clean and engine-stall and engine-split pass. I have not run the full suite.

The check failed because main had added BEAT_MS to the import on line 2 of test/engine-stall.test.ts, and my branch imported it again further down. I merged main and combined the two into one import.

The inMs=500 misreading: in why=silent inMs=500, inMs is the wait before the engine starts again, not the silence window. The window is STALL_MS, which is 30 s or more depending on main. A half-second scheduling hiccup cannot trigger it, so either the engine was really quiet that long or the serving thread measured its own stall.

What changed:

  • Serving thread's own stall. If the serving thread's loop is held, the engine's beats queue behind it, and the watch timer can fire before they are delivered. That reads the whole stall as engine silence. judgeSilence in src/engine/beat.ts spots a watch tick more than 2 s late and restarts the engine's window instead of ending the engine. This is my best explanation for the restarts at load 100–270; I have not confirmed it.
  • What the engine was doing. Each beat now carries the three jobs that held the engine's loop longest in the last 2 s. When the engine is ended as silent, the log line adds lastJobs and quietMs. A real hang will name a job; a stall on the serving side will not.
  • Tests. test/engine-stall.test.ts covers an engine that beat inside the window, one silent past it on a watcher that kept time, and a watcher that was itself held. I did not add an end-to-end test that starves the engine thread for a second. At a window of 30 s or more that cannot trip a restart, and the case that can is covered by the unit test.

Not done: I did not change STALL_MS.

To try it: bun test test/engine-stall.test.ts. On production, a later why=silent line should show lastJobs and quietMs.

Changes requested — before the latest push

The idea is sound and small: forgive a watch tick that came late, and name the jobs behind a real silence. Nothing here is user-facing, so the preview and screenshots are unaffected. Three things to fix in one round:

  1. test/job-slices.test.ts:332 still describes engine-beat as "a one-byte message". The beat now carries the held-jobs list, so that description is false. Update it.
  2. Nothing tests the new wiring: the beat payload (BEAT + name=ms list, capped at 3) and the host reading it into lastJobs/quietMs. Extract the formatting and parsing into beat.ts as plain functions, e.g. beatMessage(jobs) and jobsOfBeat(data), and test the round trip. Today only judgeSilence is covered, and the log fields would not fail a test if removed.
  3. docs/features/engine-shape.md describes the liveness protocol. Add the late-tick forgiveness and the beat payload there.

Smaller: the worker.ts beat callback is mis-indented and does too much inline; moving it into beat.ts (item 2) fixes both. Also say in beat.ts that forgiveness is bounded: it only applies to a late tick, so a truly hung engine is still ended on the next on-time tick.

commented

All three of juniper's items and the smaller points are done, and I pushed to the same branch. tsc --noEmit is clean and engine-stall, job-slices and engine-split pass. I did not run the full suite.

  • job-slices descriptions: the engine-beat entry no longer says "one-byte message". It now says the beat carries the engine's longest-held jobs. I also changed the engine-watch entry, which still said "silent for 15 s", to say it ends a thread silent past the stall limit and forgives a late tick of its own.
  • Beat payload tests: beatMessage(jobs) and jobsOfBeat(data) now live in src/engine/beat.ts, and worker.ts and host.ts call them. The tests check a round trip capped at three jobs, a beat with no jobs, and a word that is not a beat. Both of the last two read back as none, which is what lastJobs logs.
  • Docs: docs/features/engine-shape.md has a new section on how the serving thread knows the engine is alive. It covers the beat payload, quietMs and lastJobs, the fact that inMs is the restart wait, and the late-tick forgiveness.
  • Smaller points: the worker's beat callback is now two lines and correctly indented. The comment on judgeSilence says the forgiveness is bounded: it only applies to a late tick, so a truly hung engine is ended on the next on-time tick.

I did not add a test that reads the lastJobs and quietMs fields out of the real log line. They are assembled inline in host.ts, and a test for them would need a hung worker and the 30 s window. The parsing and formatting they depend on are covered.

Approved

Earlier asks are met. The job-slices descriptions no longer say "one-byte" or "15 s". beatMessage and jobsOfBeat are in beat.ts and tested, including the no-jobs and non-beat cases. The how-to is in docs/features/engine-shape.md, not the README. The comment on judgeSilence now says the forgiveness is bounded. The change is small and has no new dependency, and the pull stays on the issue. There is no new page or door, so the preview has nothing to show.

Two things I would do next:

  1. Add a host-level test, with a short stallMs and a worker that stops beating, that checks the log line carries why=silent, quietMs and lastJobs. The wiring in host.ts is not covered at the moment.
  2. After a day on production, read the why=silent lines. If quietMs is small and lastJobs is none, the late-tick forgiveness was the real cause. If a job is named, fix that job.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs