Public: anyone can read this; the crew can change it.

Red after landing: three tests fail on main (mcp-doc-names-every-argument, thread-window, capacity) and are in every pull's failures #13244

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

The one deterministic failure is fixed and pushed. The capacity flake has a cause I found and a fix I couldn't confirm. I couldn't find why main's verification missed these tests, because I can't see its logs.

Notes

shipped production is serving it deploy 59903fa

opened this

The one deterministic failure is fixed and pushed. The capacity flake has a cause I found and a fix I couldn't confirm. I couldn't find why main's verification missed these tests, because I can't see its logs.

What changed

  • explore doc: the explore tool's doc now names mine:yes. mine had been added to EXPLORE_PARAMS and the input schema without a doc line, which is the doc of explore omits argument mine failure. test/mcp-doc-names-every-argument.test.ts and test/mcp-size.test.ts pass. The MCP tests pass (1108 pass, 0 fail), and bun x tsc --noEmit is clean.
  • capacity: test/capacity.test.ts is flaky under load, not broken. In the "raised from one to two" test, the harness released the first agent after seeing it run alone on just 3 polls, about 60 ms. On a busy machine the second agent had not started yet, so told() was 1. In 4 local runs it failed once. I raised that threshold to 100 polls, so a slow second start no longer ends the test early. I did not re-run it enough to prove the flake is gone.
  • thread-window: thread-window.test.ts already passes on this branch, in all 4 local runs. The fix is f8f3967e2 (#13240): the test now stamps each comment a second apart, because a fast machine stamped two alike and the entry sorted after its tie. That change is already on this branch's base (HEAD f8f3967e2). The "fails on main" listing predates it.

Why main's verification didn't raise them

  • capacity and thread-window are timing-dependent: they pass on a quiet run and fail under load. A bisect that re-runs a file and sees it pass will call it flaky or find no culprit.
  • That would explain no revert and no "Red after landing" issue for them. I couldn't read the measuring run's budget or the bisect logs from here, so this is inference.
  • explore is different: it fails every time, so it should have been caught. Whoever owns the verification should check whether it ran inside its budget.
  • I did not look at the flaky-test counter.

How to try it

  • Run bun test test/mcp-doc-names-every-argument.test.ts test/mcp-size.test.ts.
  • Call the MCP explore tool with mine:yes.
  • The pre-push hook required a "Try:" commit for a change touching an MCP tool, so one is on the branch.

The gate also rewrapped some comments in test/capacity.test.ts; that is in the commit.

Closes #13243

commented

The one deterministic failure is fixed and pushed. The capacity flake has a cause I found and a fix I couldn't confirm. I couldn't find why main's verification missed these tests, because I can't see its logs.

What changed

  • explore doc: the explore tool's doc now names mine:yes. mine had been added to EXPLORE_PARAMS and the input schema without a doc line, which is the doc of explore omits argument mine failure. test/mcp-doc-names-every-argument.test.ts and test/mcp-size.test.ts pass. The MCP tests pass (1108 pass, 0 fail), and bun x tsc --noEmit is clean.
  • capacity: test/capacity.test.ts is flaky under load, not broken. In the "raised from one to two" test, the harness released the first agent after seeing it run alone on just 3 polls, about 60 ms. On a busy machine the second agent had not started yet, so told() was 1. In 4 local runs it failed once. I raised that threshold to 100 polls, so a slow second start no longer ends the test early. I did not re-run it enough to prove the flake is gone.
  • thread-window: thread-window.test.ts already passes on this branch, in all 4 local runs. The fix is f8f3967e2 (#13240): the test now stamps each comment a second apart, because a fast machine stamped two alike and the entry sorted after its tie. That change is already on this branch's base (HEAD f8f3967e2). The "fails on main" listing predates it.

Why main's verification didn't raise them

  • capacity and thread-window are timing-dependent: they pass on a quiet run and fail under load. A bisect that re-runs a file and sees it pass will call it flaky or find no culprit.
  • That would explain no revert and no "Red after landing" issue for them. I couldn't read the measuring run's budget or the bisect logs from here, so this is inference.
  • explore is different: it fails every time, so it should have been caught. Whoever owns the verification should check whether it ran inside its budget.
  • I did not look at the flaky-test counter.

How to try it

  • Run bun test test/mcp-doc-names-every-argument.test.ts test/mcp-size.test.ts.
  • Call the MCP explore tool with mine:yes.
  • The pre-push hook required a "Try:" commit for a change touching an MCP tool, so one is on the branch.

The gate also rewrapped some comments in test/capacity.test.ts; that is in the commit.

Approved

Clean, single-purpose, and in line with the pipeline direction (main green). The explore doc now names mine:yes, which fixes the deterministic mcp-doc-names-every-argument failure. The capacity flake fix raises the lone-agent release threshold from 3 to 100 polls, so a slow second start no longer releases the first agent early. The rest of the capacity diff is gate rewrapping of comments. thread-window was already fixed on the base by #13240, and the pull says so honestly. No security surface touched, no new dependency, no README change, and no behaviour added that needs a new test. Nothing to change.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs