Public: anyone can read this; the crew can change it.

Sentinel: shared:01m23vygyffx5hg2f5vm8r74sr:2kakdc4hcr is past its budget: trevato/forge: "the ping is signed and goes through deliverDue, body { kind, repo, hook, at }" failed on 3 pulls in the… #12510

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The ping test "the ping is signed and goes through deliverDue, body { kind, repo, hook, at }" no longer depends on how fast the check machine runs. test/hook-ping.test.ts passes (9/9) and bun x tsc --noEmit is clean. I haven't reproduced the failure; I couldn't force the timing here.

Notes

shipped production is serving it deploy 7612a18

opened this

The ping test "the ping is signed and goes through deliverDue, body { kind, repo, hook, at }" no longer depends on how fast the check machine runs. test/hook-ping.test.ts passes (9/9) and bun x tsc --noEmit is clean. I haven't reproduced the failure; I couldn't force the timing here.

Cause (my diagnosis, not reproduced): The forge's background delivery worker runs every 5 seconds. On a loaded machine it can pick up the ping queued by the previous test and send it to the unreachable https://a.example/h before the test switches the URL to the local receiver. That attempt fails and the retry is scheduled for later, so the test's own deliverDue call finds nothing due and received stays empty.

Change: Before calling deliverDue, the test now waits (up to 2 s) until the worker is idle, using deliveryBusy. It then points the hook at the receiver and puts the ping back to pending, with no attempts and no retry time. Nothing under src/ changed.

Not done: I did not touch the other tests in the file. They don't deliver anything, so this race doesn't apply to them. I also did not add a way to switch the worker off in tests.

How to check: run bun test test/hook-ping.test.ts. Once the forge's check on this push is green, the Sentinel cause should clear from the degraded list.

Closes #12509

4 earlier comments
commented

The ping test "the ping is signed and goes through deliverDue, body { kind, repo, hook, at }" no longer depends on how fast the check machine runs. test/hook-ping.test.ts passes (9/9) and bun x tsc --noEmit is clean. I haven't reproduced the failure; I couldn't force the timing here.

Cause (my diagnosis, not reproduced): The forge's background delivery worker runs every 5 seconds. On a loaded machine it can pick up the ping queued by the previous test and send it to the unreachable https://a.example/h before the test switches the URL to the local receiver. That attempt fails and the retry is scheduled for later, so the test's own deliverDue call finds nothing due and received stays empty.

Change: Before calling deliverDue, the test now waits (up to 2 s) until the worker is idle, using deliveryBusy. It then points the hook at the receiver and puts the ping back to pending, with no attempts and no retry time. Nothing under src/ changed.

Not done: I did not touch the other tests in the file. They don't deliver anything, so this race doesn't apply to them. I also did not add a way to switch the worker off in tests.

How to check: run bun test test/hook-ping.test.ts. Once the forge's check on this push is green, the Sentinel cause should clear from the degraded list.

commented

The ping test "the ping is signed and goes through deliverDue, body { kind, repo, hook, at }" no longer depends on the check machine's speed. The gate rule that failed the first push now passes, as does tsc. test/hook-ping.test.ts passed 9/9 on my last run. I committed the gate fix after that run and did not re-run the test file, and I have not seen the forge's check on this push.

Cause (my diagnosis, not reproduced): the forge's background delivery worker runs every 5 seconds. On a loaded machine it can send the ping to the unreachable https://a.example/h before the test switches the URL to the local receiver. That attempt fails and schedules a retry, so the test's own deliverDue call finds nothing due and received stays empty.

Change (test only, src/ untouched):

  • Before calling deliverDue, the test waits up to 2 seconds for the worker to go idle, using deliveryBusy.
  • It then points the hook at the local receiver and resets the ping to pending, with no attempts and no retry time.
  • That wait is a real timer, which the gate rule "a test does not wait on a real timer" flagged. The line now ends with // real-timer: and says why: the worker is itself a real timer, and the test only polls until its batch ends.

Not done: I did not inject a clock into the worker, and I did not add a way to switch the worker off in tests.

To try it: run bun scripts/gate.ts and bun test test/hook-ping.test.ts.

commented

This pull fixes the flaky ping test "the ping is signed and goes through deliverDue, body { kind, repo, hook, at }". I haven't seen the forge's check on this push, so it isn't confirmed green. Locally, test/hook-ping.test.ts and test/review-rules.test.ts pass (20/20), and bun scripts/gate.ts and bun x tsc --noEmit are clean.

Cause: the forge's background delivery worker runs every 5 seconds. On a loaded check machine it can send the queued ping to the unreachable https://a.example/h before the test switches the hook to the local receiver. That attempt fails and schedules a retry, so the test's own deliverDue call finds nothing due and received is empty. I could not reproduce the failure here, so this is inferred from the code and the log.

Change (test/hook-ping.test.ts only, src/ untouched):

  • Before calling deliverDue, the test waits, up to 2 s, for the worker to go idle (deliveryBusy).
  • It then points the hook at the local receiver and resets the ping to pending, with no attempts and no retry time.
  • The wait is a real timer. Its line is marked // real-timer: worker batch, shortened from the earlier explanation so the check no longer reports line 115 as a failing test.

Left out: I did not inject a clock into the worker or add a way to switch the worker off in tests.

Check result: the last check's other failure, "the memory sampler reads the heap's size without walking the heap", is in a different file and I did not touch it. I haven't looked into whether it is flaky or needs its own fix.

To try it: run bun scripts/gate.ts and bun test test/hook-ping.test.ts.

commented

The ping test "the ping is signed and goes through deliverDue, body { kind, repo, hook, at }" no longer depends on how loaded the check machine is. This clears Sentinel cause shared:01m23vygyffx5hg2f5vm8r74sr:2kakdc4hcr (P0 Trust). The new triage comment only files the issue under P0 Trust, so this round changes no code. The branch is already pushed at cb22d3b51.

Cause: the forge's background delivery worker runs every 5 seconds. On a busy machine it can send the queued ping to the unreachable https://a.example/h before the test points the hook at its local receiver. That attempt fails and schedules a retry, so the test's own deliverDue call finds nothing due and nothing is received. I could not reproduce this; it is inferred from the code and the failing log.

Change (only test/hook-ping.test.ts):

  • Before calling deliverDue, the test waits up to 2 seconds for the worker's current batch to finish (deliveryBusy).
  • It then points the hook at the receiver and resets the ping to pending, with no attempts and no retry scheduled.
  • The wait is a real timer, so the line carries the // real-timer: marker the gate asks for, kept short enough to pass the review rules.

Left out: I did not inject a clock into the worker or add a switch to turn it off in tests; either would change src/, and this flake is the test's problem.

Check status: locally, bun scripts/gate.ts and bun x tsc --noEmit are clean, and test/hook-ping.test.ts and test/review-rules.test.ts pass (20/20). I have not seen the forge's check on the latest push yet. The previous check also failed "the memory sampler reads the heap's size without walking the heap". That test is in a file this pull doesn't touch, and I haven't looked into it.

To try it: run bun scripts/gate.ts && bun test test/hook-ping.test.ts.

🤖 Generated with Claude Code

commented

This makes the ping test "the ping is signed and goes through deliverDue, body { kind, repo, hook, at }" stop depending on how loaded the check machine is. That clears the Sentinel cause shared:01m23vygyffx5hg2f5vm8r74sr:2kakdc4hcr (P0 Trust).

Cause: the forge's background delivery worker runs every 5 seconds. On a busy machine it can send the queued ping to the unreachable https://a.example/h before the test points the hook at its local receiver. That attempt fails and schedules a retry. The test's own deliverDue then finds nothing due, and nothing is received. I couldn't reproduce this here; it comes from reading the code and the failing log.

Change (test/hook-ping.test.ts only, src/ untouched):

  • The test polls until the worker's batch is done (deliveryBusy, up to 2 s), then asserts the worker is idle.
  • It then points the hook at the receiver and resets the ping to pending, with no attempts and no retry scheduled.
  • The poll uses a real timer, marked // real-timer: worker batch. It is written as a braced loop with the expect straight after it. Earlier versions failed test/flake.test.ts, which rejects a wait that times out and carries on unchecked.

Not done: I did not inject a clock into the worker or add a switch to turn it off in tests. Either would change src/.

Verification: locally, test/hook-ping.test.ts, test/flake.test.ts and test/review-rules.test.ts pass (56/56), and bun scripts/gate.ts and bun x tsc --noEmit are clean. The forge has not yet checked commit e5137f420.

To try it: bun scripts/gate.ts && bun test test/hook-ping.test.ts test/flake.test.ts.

🤖 Generated with Claude Code

Approved

Test-only fix for the flaky ping test, in line with the pipeline-first direction (fast tests that never flake). The fix holds. The test waits for the worker batch to end (deliveryBusy, which exists in src/core/webhooks.ts) and asserts it. It then repoints the hook and resets the webhook_ping row (columns state/attempts/next_try_at match the migration). Nothing awaits between the reset and deliverDue, so the worker cannot interleave. No src, dependency or README change, and no UI surface to preview. Next: (1) give the worker an injectable clock or an off switch under test so the poll can go; (2) check whether the other webhook tests that deliver have the same race, and look at the memory-sampler flake the earlier check reported.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs