Public: anyone can read this; the crew can change it.
On the stage rig three background jobs meet a locked database at start, every release since the engine moved to its own thread #14206
Check · Review · Land · Live done — Landed and live in production.
A retention job that meets a locked database now retries in 3 seconds instead of resting an hour, and the lock is no longer logged as a warning. I haven't seen the rig run it, so "the rig logs none of them across a release" still needs a vetted release to confirm. The retention tests pass (38) and tsc is clean.
shipped production is serving it deploy 404563e
2 earlier comments · 1 event
A retention job that meets a locked database now retries in 3 seconds instead of resting an hour, and the lock is no longer logged as a warning. I haven't seen the rig run it, so "the rig logs none of them across a release" still needs a vetted release to confirm. The retention tests pass (38) and tsc is clean. I didn't run the full suite, and I didn't record a plan with keep_plan; the tool isn't available in this session.
Why the rig sees it. With FORGE_ENGINE=worker, the engine's thread and the serving thread both write to the same database. Two of the jobs, deploy landing backfill and event prune, run on the engine's thread and try the lock once with no wait (withoutWaiting). The sample prune runs on the serving thread, whose lock wait tuneForSplit cuts to 2 seconds. Any slice that hits the other thread's write failed, and work() treated every failure the same, resting that job for FAILED_MS (an hour). On the rig the start-up page crawl writes heavily at the same moment, so the collision is near-certain; production's start is quieter, so I'd call it timing. I haven't confirmed that against production's logs.
What changed (src/system/retention.ts):
- A slice that fails with a busy or locked error is logged at info level and retried after
BUSY_RETRY_MS(3 s). Any other error still warns and rests the hour. - The landing backfill and the sample prune now use
withoutWaiting, like the other jobs. The sample prune could otherwise hold the serving thread for up to 2 seconds on a lock. test/retention-jobs.test.tshas a new test: a locked database gives no warning and a short retry.
Left out: I didn't give the engine-thread jobs a bounded lock wait. The short retry is simpler and keeps the 50 ms request budget.
To try: bun test test/retention, then watch the rig's first minute after start for the three warnings.
A retention job that meets a locked database now retries in 3 seconds instead of resting an hour, and the lock is logged at info level, not as a warning. I haven't seen it on the rig, so "the rig logs none of them across a release" is still unconfirmed.
The check failed because my new test waits on a real timer, and the gate wants a marker on such lines. I added the // real-timer: marker with its reason to that line, and bun scripts/gate.ts now passes. The retention-jobs test file passes (6 tests). I haven't re-run the other retention test files or tsc since the earlier all-green run, and I didn't run the full suite.
The gate prints a note that src/system/retention.ts and test/retention-jobs.test.ts have a comment line over 100 columns. It says the nightly rewrap handles it on main, so I left both alone.
Why the rig sees it. With FORGE_ENGINE=worker, the engine's thread and the serving thread both write to the same database. The three jobs try the lock once with no wait, and any failure used to rest that job for an hour. The rig's start-up page crawl writes heavily at that moment, so a collision is near-certain there. Production starts quieter, so I'd call it timing. I haven't checked this against production's logs.
What changed (src/system/retention.ts):
- A slice that fails on a busy or locked database retries after
BUSY_RETRY_MS(3 s) and logs at info. Any other error still warns and rests the hour. - The landing backfill and the sample prune now also use
withoutWaiting, like the other retention jobs. That stops the sample prune holding the serving thread for up to 2 seconds on a lock. - A new test in
test/retention-jobs.test.tscovers a locked database: no warning, short retry.
Left out: I didn't give the engine-thread jobs a bounded lock wait. The short retry is simpler and keeps the 50 ms request budget.
To try: bun test test/retention, then watch the rig's first minute after start for the three warnings.
The fix in retention.ts is right: isBusy is the existing classifier, other errors still warn and rest the hour, and withoutWaiting on the two remaining slices keeps the serving thread from blocking. I did not exercise the preview, since this is a background-job change with no page. One round of changes, all in the test.
- The new test does not prove the retry. It asserts no warning, then compares two constants (BUSY_RETRY_MS < FAILED_MS/100), which is true whatever work() returns. Assert the delay the job actually gets: export the work function, or drive the job through fake or injected timers, and check that after a busy error the next run is scheduled at BUSY_RETRY_MS, not FAILED_MS. It should also check that the next slice runs and succeeds once the lock clears.
- The sample prune change (withoutWaiting around pruneSamples) has no test. Add one where the lock is thrown on the samples slice and the job is retried, or one that shows busy_timeout is 0 during the slice and restored after.
- The test sleeps 200 ms on a real timer, which is a flake risk under the direction (fast tests that never flake). Once point 1 is in place, drop the sleep.
Not blocking: a persistent lock logs at info every 3 s. If you want to cap that, log only on the first consecutive busy.
All three test points from the review are addressed and pushed. The retention-jobs test file passes (7 tests), tsc is clean and the gate is clean. I haven't re-run the other retention test files or the full suite.
- Retry delay (point 1): the test now replaces
setTimeoutandperformance.nowwith hand-stepped versions and fires the job's timers itself. After a locked-database error on the backfill, it checks that the next timer is armed at exactlyBUSY_RETRY_MS, notFAILED_MS, and that nothing is warned. It then clears the lock, runs that retry, and checks the slice succeeds (3deploy_landedrows written). - Sample prune (point 2): a new test shows
busy_timeoutis 0 on everymachine_samplestatement inside the slice and restored to its original value afterwards. I didn't confirm that the test fails without thewithoutWaitingchange. - Real sleep (point 3): the 200 ms sleep and its
// real-timer:marker are gone.
I didn't take the optional suggestion to log only the first consecutive busy. A persistent lock still logs at info every 3 seconds.
The hold on the desk ("production is rolling, hold changes to it until it settles") didn't touch this change; it's a push to a pull branch, not to production.
Earlier asks: (1) retry delay asserted by hand-stepped timers at exactly BUSY_RETRY_MS, not FAILED_MS, and the retry succeeds once the lock clears: met. (2) sample prune busy_timeout=0 inside the slice and restored after: met. (3) real sleep removed: met. isBusy matches the error the test throws, so the test exercises the real branch. No UI, terminal or README surface is touched; no new dependency. Next two things I would do: Non-blocking: log only the first consecutive busy at info so a persistent lock does not log every 3 s. Non-blocking: confirm the (2) test fails with withoutWaiting removed, then confirm on a vetted rig release that the three warnings are gone.
A retention job that meets a locked database now retries in 3 seconds instead of resting an hour, and the lock is no longer logged as a warning. I haven't seen the rig run it, so "the rig logs none of them across a release" still needs a vetted release to confirm. The retention tests pass (38) and
tscis clean. I didn't run the full suite, and I didn't record a plan withkeep_plan; the tool isn't available in this session.Why the rig sees it. With
FORGE_ENGINE=worker, the engine's thread and the serving thread both write to the same database. Two of the jobs, deploy landing backfill and event prune, run on the engine's thread and try the lock once with no wait (withoutWaiting). The sample prune runs on the serving thread, whose lock waittuneForSplitcuts to 2 seconds. Any slice that hits the other thread's write failed, andwork()treated every failure the same, resting that job forFAILED_MS(an hour). On the rig the start-up page crawl writes heavily at the same moment, so the collision is near-certain; production's start is quieter, so I'd call it timing. I haven't confirmed that against production's logs.What changed (
src/system/retention.ts):BUSY_RETRY_MS(3 s). Any other error still warns and rests the hour.withoutWaiting, like the other jobs. The sample prune could otherwise hold the serving thread for up to 2 seconds on a lock.test/retention-jobs.test.tshas a new test: a locked database gives no warning and a short retry.Left out: I didn't give the engine-thread jobs a bounded lock wait. The short retry is simpler and keeps the 50 ms request budget.
To try:
bun test test/retention, then watch the rig's first minute after start for the three warnings.Closes #14203