Public: anyone can read this; the crew can change it.

On the stage rig three background jobs meet a locked database at start, every release since the engine moved to its own thread #14206

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

A retention job that meets a locked database now retries in 3 seconds instead of resting an hour, and the lock is no longer logged as a warning. I haven't seen the rig run it, so "the rig logs none of them across a release" still needs a vetted release to confirm. The retention tests pass (38) and tsc is clean.

Notes

shipped production is serving it deploy 404563e

opened this

A retention job that meets a locked database now retries in 3 seconds instead of resting an hour, and the lock is no longer logged as a warning. I haven't seen the rig run it, so "the rig logs none of them across a release" still needs a vetted release to confirm. The retention tests pass (38) and tsc is clean. I didn't run the full suite, and I didn't record a plan with keep_plan; the tool isn't available in this session.

Why the rig sees it. With FORGE_ENGINE=worker, the engine's thread and the serving thread both write to the same database. Two of the jobs, deploy landing backfill and event prune, run on the engine's thread and try the lock once with no wait (withoutWaiting). The sample prune runs on the serving thread, whose lock wait tuneForSplit cuts to 2 seconds. Any slice that hits the other thread's write failed, and work() treated every failure the same, resting that job for FAILED_MS (an hour). On the rig the start-up page crawl writes heavily at the same moment, so the collision is near-certain; production's start is quieter, so I'd call it timing. I haven't confirmed that against production's logs.

What changed (src/system/retention.ts):

  • A slice that fails with a busy or locked error is logged at info level and retried after BUSY_RETRY_MS (3 s). Any other error still warns and rests the hour.
  • The landing backfill and the sample prune now use withoutWaiting, like the other jobs. The sample prune could otherwise hold the serving thread for up to 2 seconds on a lock.
  • test/retention-jobs.test.ts has a new test: a locked database gives no warning and a short retry.

Left out: I didn't give the engine-thread jobs a bounded lock wait. The short retry is simpler and keeps the 50 ms request budget.

To try: bun test test/retention, then watch the rig's first minute after start for the three warnings.

Closes #14203

2 earlier comments · 1 event
commented

A retention job that meets a locked database now retries in 3 seconds instead of resting an hour, and the lock is no longer logged as a warning. I haven't seen the rig run it, so "the rig logs none of them across a release" still needs a vetted release to confirm. The retention tests pass (38) and tsc is clean. I didn't run the full suite, and I didn't record a plan with keep_plan; the tool isn't available in this session.

Why the rig sees it. With FORGE_ENGINE=worker, the engine's thread and the serving thread both write to the same database. Two of the jobs, deploy landing backfill and event prune, run on the engine's thread and try the lock once with no wait (withoutWaiting). The sample prune runs on the serving thread, whose lock wait tuneForSplit cuts to 2 seconds. Any slice that hits the other thread's write failed, and work() treated every failure the same, resting that job for FAILED_MS (an hour). On the rig the start-up page crawl writes heavily at the same moment, so the collision is near-certain; production's start is quieter, so I'd call it timing. I haven't confirmed that against production's logs.

What changed (src/system/retention.ts):

  • A slice that fails with a busy or locked error is logged at info level and retried after BUSY_RETRY_MS (3 s). Any other error still warns and rests the hour.
  • The landing backfill and the sample prune now use withoutWaiting, like the other jobs. The sample prune could otherwise hold the serving thread for up to 2 seconds on a lock.
  • test/retention-jobs.test.ts has a new test: a locked database gives no warning and a short retry.

Left out: I didn't give the engine-thread jobs a bounded lock wait. The short retry is simpler and keeps the 50 ms request budget.

To try: bun test test/retention, then watch the rig's first minute after start for the three warnings.

commented

A retention job that meets a locked database now retries in 3 seconds instead of resting an hour, and the lock is logged at info level, not as a warning. I haven't seen it on the rig, so "the rig logs none of them across a release" is still unconfirmed.

The check failed because my new test waits on a real timer, and the gate wants a marker on such lines. I added the // real-timer: marker with its reason to that line, and bun scripts/gate.ts now passes. The retention-jobs test file passes (6 tests). I haven't re-run the other retention test files or tsc since the earlier all-green run, and I didn't run the full suite.

The gate prints a note that src/system/retention.ts and test/retention-jobs.test.ts have a comment line over 100 columns. It says the nightly rewrap handles it on main, so I left both alone.

Why the rig sees it. With FORGE_ENGINE=worker, the engine's thread and the serving thread both write to the same database. The three jobs try the lock once with no wait, and any failure used to rest that job for an hour. The rig's start-up page crawl writes heavily at that moment, so a collision is near-certain there. Production starts quieter, so I'd call it timing. I haven't checked this against production's logs.

What changed (src/system/retention.ts):

  • A slice that fails on a busy or locked database retries after BUSY_RETRY_MS (3 s) and logs at info. Any other error still warns and rests the hour.
  • The landing backfill and the sample prune now also use withoutWaiting, like the other retention jobs. That stops the sample prune holding the serving thread for up to 2 seconds on a lock.
  • A new test in test/retention-jobs.test.ts covers a locked database: no warning, short retry.

Left out: I didn't give the engine-thread jobs a bounded lock wait. The short retry is simpler and keeps the 50 ms request budget.

To try: bun test test/retention, then watch the rig's first minute after start for the three warnings.

Changes requested — before the latest push

The fix in retention.ts is right: isBusy is the existing classifier, other errors still warn and rest the hour, and withoutWaiting on the two remaining slices keeps the serving thread from blocking. I did not exercise the preview, since this is a background-job change with no page. One round of changes, all in the test.

  1. The new test does not prove the retry. It asserts no warning, then compares two constants (BUSY_RETRY_MS < FAILED_MS/100), which is true whatever work() returns. Assert the delay the job actually gets: export the work function, or drive the job through fake or injected timers, and check that after a busy error the next run is scheduled at BUSY_RETRY_MS, not FAILED_MS. It should also check that the next slice runs and succeeds once the lock clears.
  2. The sample prune change (withoutWaiting around pruneSamples) has no test. Add one where the lock is thrown on the samples slice and the job is retried, or one that shows busy_timeout is 0 during the slice and restored after.
  3. The test sleeps 200 ms on a real timer, which is a flake risk under the direction (fast tests that never flake). Once point 1 is in place, drop the sleep.

Not blocking: a persistent lock logs at info every 3 s. If you want to cap that, log only on the first consecutive busy.

commented

All three test points from the review are addressed and pushed. The retention-jobs test file passes (7 tests), tsc is clean and the gate is clean. I haven't re-run the other retention test files or the full suite.

  • Retry delay (point 1): the test now replaces setTimeout and performance.now with hand-stepped versions and fires the job's timers itself. After a locked-database error on the backfill, it checks that the next timer is armed at exactly BUSY_RETRY_MS, not FAILED_MS, and that nothing is warned. It then clears the lock, runs that retry, and checks the slice succeeds (3 deploy_landed rows written).
  • Sample prune (point 2): a new test shows busy_timeout is 0 on every machine_sample statement inside the slice and restored to its original value afterwards. I didn't confirm that the test fails without the withoutWaiting change.
  • Real sleep (point 3): the 200 ms sleep and its // real-timer: marker are gone.

I didn't take the optional suggestion to log only the first consecutive busy. A persistent lock still logs at info every 3 seconds.

The hold on the desk ("production is rolling, hold changes to it until it settles") didn't touch this change; it's a push to a pull branch, not to production.

Approved

Earlier asks: (1) retry delay asserted by hand-stepped timers at exactly BUSY_RETRY_MS, not FAILED_MS, and the retry succeeds once the lock clears: met. (2) sample prune busy_timeout=0 inside the slice and restored after: met. (3) real sleep removed: met. isBusy matches the error the test throws, so the test exercises the real branch. No UI, terminal or README surface is touched; no new dependency. Next two things I would do: Non-blocking: log only the first consecutive busy at info so a persistent lock does not log every 3 s. Non-blocking: confirm the (2) test fails with withoutWaiting removed, then confirm on a vetted rig release that the three warnings are gone.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs