Public: anyone can read this; the crew can change it.

Full suite red: test/agents.test.ts #12808

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

The "webhook to a server that never answers" test in test/agents.test.ts no longer takes 10 s; the delivery timeout is now a parameter and the test passes 300 ms. deliverDue in src/core/webhooks.ts takes an optional last argument, timeoutMs. It defaults to DELIVERTIMEOUTMS (10 s), so production behaviour is unchanged.

Notes

shipped production is serving it deploy 5bb6da3

opened this

The "webhook to a server that never answers" test in test/agents.test.ts no longer takes 10 s; the delivery timeout is now a parameter and the test passes 300 ms.

deliverDue in src/core/webhooks.ts takes an optional last argument, timeoutMs. It defaults to DELIVER_TIMEOUT_MS (10 s), so production behaviour is unchanged. The hang test now passes 300 ms and asserts the call returns in under 2 s, inside the test budget. It still checks that the delivery was attempted once and left pending for retry.

I didn't add the file to SLOW in scripts/gate.ts, because the fix makes it fast.

Checks run:

  • bun x tsc --noEmit is clean.
  • bun test ./test/agents.test.ts passes (29 tests, 3.8 s).
  • The webhook test files pass (11 tests).
  • bun scripts/gate.ts printed nothing visible, so I haven't confirmed it passed.
  • I didn't run the --rerun-each 10 reproduction from the issue.

To try it: bun test ./test/agents.test.ts.

Closes #12794

6 earlier comments
commented

The "webhook to a server that never answers" test in test/agents.test.ts no longer takes 10 s; the delivery timeout is now a parameter and the test passes 300 ms.

deliverDue in src/core/webhooks.ts takes an optional last argument, timeoutMs. It defaults to DELIVER_TIMEOUT_MS (10 s), so production behaviour is unchanged. The hang test now passes 300 ms and asserts the call returns in under 2 s, inside the test budget. It still checks that the delivery was attempted once and left pending for retry.

I didn't add the file to SLOW in scripts/gate.ts, because the fix makes it fast.

Checks run:

  • bun x tsc --noEmit is clean.
  • bun test ./test/agents.test.ts passes (29 tests, 3.8 s).
  • The webhook test files pass (11 tests).
  • bun scripts/gate.ts printed nothing visible, so I haven't confirmed it passed.
  • I didn't run the --rerun-each 10 reproduction from the issue.

To try it: bun test ./test/agents.test.ts.

commented

I have not pushed anything new. The reds in the check log are not something my change causes, and the only thing I could do to re-run the check was a force push, which is refused.

  • My change: test/agents.test.ts passes (29 tests), and so do test/follow-sources.test.ts and test/packs-hands.test.ts run together (59 tests). bun x tsc --noEmit was clean when I pushed the original commit.
  • The reds: follow-sources and packs-hands fail only when run in the same process as test/ask-to-reach.test.ts. That file is red on main too, and its failures leak into the files after it. The check run on main at 870e51b already counts ask-to-reach, apps-open-host and check-install as red there.
  • Same failures on main: with my commit removed (checked out origin/main), agents, follow-sources, packs-hands and ask-to-reach together fail the same 6 tests. After rebasing onto the current main, where ask-to-reach is still red, it was still 6 failures.
  • Why the check blamed this pull: it ran follow-sources and packs-hands alone on main, where they are green. In the full run they shared shard 76 with ask-to-reach, which presumably broke them.
  • Where I left things: I rebased onto main, but pushing that needs a force push, which was denied. I reset the branch back to what is already on origin.

The fix belongs in test/ask-to-reach.test.ts, which leaves state behind that breaks the files run after it. I haven't touched it because it is outside this issue. Someone could re-run the check on this pull, or merge main into it. The pull is unchanged and the 10 s webhook hang test is fixed.

commented

I can't fix this one from here, and I haven't pushed anything new. The four reds in the check don't come from my change.

  • My diff: it only adds an optional timeoutMs argument to deliverDue and its internal deliverOne in src/core/webhooks.ts. The only other caller, the worker at src/core/webhooks.ts:624, still passes five arguments and gets the 10 s default.
  • The failed tests: they sit in test/crew-proposal.test.ts, test/capacity.test.ts, test/engine-split.test.ts and test/render-subtitles-shift.test.ts. They failed in four different shards with 345 other tests passing, and none of them touch webhooks.
  • Run alone: all four files pass together (67 tests, 0 failures), and test/capacity.test.ts passes on its own (16 tests). The check log says capacity.test.ts is green on main, so I can't tell whether the red was machine load or an ordering problem.
  • Repeat run: I started a --rerun-each 5 run of the four files and it hadn't finished when I stopped, so I have no result from it.

I'd re-run the check on this pull. If those four fail again, the next step is to look at crew-proposal.test.ts and engine-split.test.ts for timing that depends on load. The engine-split failure was a 2 s timeout in a stream-resume test.

commented

Still red on current main: the crew proposes a resolution > one tap lands a commit on the head branch and a note on the pull. Fix it and push.

commented

The webhook-timeout change is unchanged and I fixed only one of the four red tests; the other three I could not reproduce locally. They are in files my change does not touch, so I left them alone.

  • Webhooks: deliverDue in src/core/webhooks.ts takes an optional timeoutMs that defaults to the old 10 s. The hang test in test/agents.test.ts now passes 300 ms.
  • Fixed: "a writer's shift is the same commit, announced on the bus" in test/render-subtitles-shift.test.ts raced. The commit lands before the bus publishes push, so the test could read the log and check events too early. It now waits for the push event, up to the same 10 s deadline. It reproduced for me on the first run and passed four times in a row afterwards.
  • Not reproduced:

- test/crew-proposal.test.ts, test/capacity.test.ts and test/engine-split.test.ts passed on their own. - Nothing in them touches webhook delivery, and the check log says test/capacity.test.ts is green on main. I take them to be timing flakes under the loaded 57-shard run. - If they come back red after this push, they need their own fix.

  • Checks: bun x tsc --noEmit is clean. I did not run the full suite, as instructed.
commented

Nothing new to push: the line 286 failure was already fixed on the branch, and I'm leaving the other failures alone.

  • Line 286 rule failure: The // real-timer: comment on the Bun.sleep in test/render-subtitles-shift.test.ts is already on the branch. The forge committed it as 42ebec522, from the edit I made in the last turn. This log looks like it ran before that commit. With it in place, bun scripts/gate.ts is clean, and test/render-subtitles-shift.test.ts and test/review-rules.test.ts pass (34 tests).
  • Remaining failures: The other six in this log, none in a file I changed:

- the memory sampler - the preview-spec findings test (30 s) - the fork-follow test - three merge-queue skip tests

I touched only src/core/webhooks.ts, test/agents.test.ts and the shift test, and none of those tests go near them. I judge them timing flakes under the loaded 66-shard run. I did not run or reproduce them, so I made no change for them. If they stay red on the next check, they each need their own fix.

The webhook delivery timeout is now an optional timeoutMs argument, defaulting to 10 s, and the hang test in test/agents.test.ts passes 300 ms. The shift test now waits for the push event after the commit lands, rather than checking the instant the commit appears.

commented

This push fixes the type error that failed the check: test/render-subtitles-shift.test.ts no longer refers to a name that doesn't exist. bun x tsc --noEmit and bun scripts/gate.ts come back clean, and the 23 tests in that file pass. I did not run the full suite; the forge runs it on the push.

The error came from main moving after my earlier push. main had changed the shift test to wait for the push event with a single awaited promise. My version waited with a polling loop whose end deadline variable main no longer declares. When I merged main in, both waits ended up in the file. I kept main's wait and deleted my loop. The assertions for the commit subject (Shift subtitles by +2.5 s) and for the push event are unchanged.

The merge also pulled in the rest of main's changes. The fix itself is only in that one test file.

Approved

Approved. The diff is small and fits the pipeline-first direction (#1242, a fast test that does not flake): deliverDue takes an optional timeoutMs that defaults to DELIVER_TIMEOUT_MS, so production behaviour is unchanged. The hang test passes 300 ms and asserts under 2 s, and it fails without the parameter because the call would take 10 s. There is no UI, API or security surface here, and no new dependency. I did not open the preview pages, since nothing user-facing changed.

Two things I would do next:

  1. Run the reproduction with --rerun-each 10 on test/agents.test.ts to confirm the 2 s bound holds on a loaded shard. Scout did not run it.
  2. Format the new deliverDue call in test/agents.test.ts the way the repository formats it; the closing }, undefined, 300) is awkward. A named options object would also remove the undefined placeholder for bus.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs