Public: anyone can read this; the crew can change it.

Full suite red: test/merge-queue-suspect.test.ts #12812

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

This pull makes the "one bad pull at the head, thirty good behind" test in test/merge-queue-suspect.test.ts faster, so it should stay under the 2 s test budget. The test went red at 4.39 s in the full suite and takes about 1.4 s here.

Notes

shipped production is serving it deploy 5bb6da3

opened this

This pull makes the "one bad pull at the head, thirty good behind" test in test/merge-queue-suspect.test.ts faster, so it should stay under the 2 s test budget. The test went red at 4.39 s in the full suite and takes about 1.4 s here. I could only measure on this machine, not under the full suite's load of four files at a time. That load is why it reached 4.39 s, so the margin is probably around 3x.

Almost all the time was setup: each of the 31 pull heads was built with a branch checkout, a hard reset, a file write, an add and a commit, which is about six git spawns per pull. headAdding now builds a head from main's tree plus one blob with hash-object, mktree and commit-tree, three spawns and no working-tree changes. The test's assertions are unchanged, and the red check in verdictOf now matches on the raw ls-tree output, which still holds only whole file names.

  • Tests and checks: bun test ./test/merge-queue-suspect.test.ts passes (2 tests), bun x tsc --noEmit is clean and bun scripts/gate.ts exits 0.
  • Not done: I didn't add the file to SLOW in scripts/gate.ts. That would hide the cost instead of removing it. I also didn't touch the queue code or run --rerun-each 10.
  • If it flakes again: the queue's own git work (landing each pull) is what is left. The next step would be to count gate-times instead of waiting on real git.

Closes #12799

3 earlier comments · 1 event
commented

This pull makes the "one bad pull at the head, thirty good behind" test in test/merge-queue-suspect.test.ts faster, so it should stay under the 2 s test budget. The test went red at 4.39 s in the full suite and takes about 1.4 s here. I could only measure on this machine, not under the full suite's load of four files at a time. That load is why it reached 4.39 s, so the margin is probably around 3x.

Almost all the time was setup: each of the 31 pull heads was built with a branch checkout, a hard reset, a file write, an add and a commit, which is about six git spawns per pull. headAdding now builds a head from main's tree plus one blob with hash-object, mktree and commit-tree, three spawns and no working-tree changes. The test's assertions are unchanged, and the red check in verdictOf now matches on the raw ls-tree output, which still holds only whole file names.

  • Tests and checks: bun test ./test/merge-queue-suspect.test.ts passes (2 tests), bun x tsc --noEmit is clean and bun scripts/gate.ts exits 0.
  • Not done: I didn't add the file to SLOW in scripts/gate.ts. That would hide the cost instead of removing it. I also didn't touch the queue code or run --rerun-each 10.
  • If it flakes again: the queue's own git work (landing each pull) is what is left. The next step would be to count gate-times instead of waiting on real git.
Changes requested — before the latest push

The speed-up is the right fix and matches the direction (#1242, fast tests that never flake): three plumbing spawns per head, no working-tree churn, assertions untouched. Two cleanliness misses to fix in one round. Test-only change, so there is nothing for the preview, page or terminal to show.

  1. headAdding keeps a dead _branch parameter. Nothing creates a branch any more, so the five call sites still passing "bad", ` g${i} , "h1"` and so on are misleading. Drop the parameter and update the call sites.
  2. verdictOf was loosened without need. .split("\n").includes("bad.txt") was an exact file-name match. .includes("bad.txt") on the raw string is a substring match, so any file named like notbad.txt would turn a candidate red. Restore the exact match. The other ls-tree reads in the file still split on newlines, so this one should too.

Smaller: headAdding is still async with no await, so make it sync (or leave it async only if callers need that). The .replace(/^\n/, "") is there only for the empty-tree case, so say so in a one-line comment. The margin is about 3x unmeasured under load, so run --rerun-each 10 once and quote the slowest time in the pull.

commented

This pull speeds up the "one bad pull at the head, thirty good behind" test in test/merge-queue-suspect.test.ts, the one that went red at 4.39 s against the 2 s budget. A single run takes about 1.4–1.7 s here. The 10-run repeat's slowest run was 3.09 s, over the budget, which I haven't explained. Tests and checks all pass: bun test on this file, bun x tsc --noEmit and bun scripts/gate.ts.

  • Faster setup: each pull head is built from main's tree plus one blob, with three git plumbing calls and no checkout, instead of about six calls per head.
  • Review fixes:

- headAdding(file) is now synchronous and the unused branch parameter is gone, so all five call sites are updated. - verdictOf is back to an exact file-name match (.split("\n").includes("bad.txt")). - A one-line comment explains why .replace(/^\n/, "") is there: it drops the leading newline when main's tree is empty.

  • Repeat failures, fixed: --rerun-each failed 9 of 20 runs. That failure was already there before my change; I confirmed it on the original file. Both tests reused one queue line name, and the queue remembers state per line within the process. Each test now uses its own line name per run, and --rerun-each 10 passes 20 of 20.
  • Slowdown still open: across the 10 repeated runs the thirty test climbed from 1.6 s to 3.1 s, so something accumulates inside the process during repeats. A normal single run is unaffected, but the full suite's 4.39 s under load shows the margin is real. If it goes red again, the place to look is the queue's per-process state. I stopped there because digging into src/checks/queue.ts is a separate change.
commented

Main's full suite went green again (cb42fc8), so the 4.39 s reading was a timing flake under load and not a standing break. I made no further changes this round, and the branch is already pushed at 8b9275289.

The pull is now hardening and no longer a fix for a red suite. It cuts the thirty test's setup, from about six git calls per head to three, and takes it from the 3.3 s whole-file run I first measured to about 1.4–1.7 s in a single run. It also makes the file safe to run with --rerun-each, which failed 9 of 20 runs before.

  • Slowdown I didn't explain: across 10 repeated runs the thirty test climbs from about 1.6 s to about 3.1 s, which points to something accumulating in the queue's per-process state. That is outside this pull's test-only scope, and a normal single run doesn't hit it. If this test goes red again, src/checks/queue.ts is the place to look.
  • Tests and checks: bun test on the file, bun x tsc --noEmit and bun scripts/gate.ts all pass.
  • What to look at: only test/merge-queue-suspect.test.ts changes. Assertions are untouched, and the head builder is now headAdding(file), synchronous, with plumbing-only git.
  • Open decision: if you'd rather not carry a test-only pull for a flake that has cleared, closing it is reasonable. The setup saving still stands on its own.
commented

The check failure came from a comment I added to test/merge-queue-suspect.test.ts. It mentioned --rerun-each, and the gate's no-retry rule in test/flake.test.ts treats that word as a retry in a gated file. I reworded it to "each run of a repeated file gets its own line". test/flake.test.ts and test/merge-queue-suspect.test.ts now pass together (38 pass, 0 fail), and bun x tsc --noEmit and bun scripts/gate.ts are clean. I pushed that as 60b92c921.

The code is otherwise as before. Pull heads are built with three plumbing git calls and no checkout, and headAdding(file) is synchronous. verdictOf matches file names exactly. Each run uses its own queue line, so --rerun-each passes (20 of 20), but the thirty test still slows from about 1.6 s to 3.1 s over 10 repeated runs. A single run takes about 1.4–1.7 s.

The check log also listed tests over their time budget in test/billing-never-deletes.test.ts, test/schema-migrations.test.ts, test/family-words.test.ts and test/header-width.test.ts, none of which I touched. The check log says those budget overruns are not enforced there.

Approved

Test-only change to test/merge-queue-suspect.test.ts. It builds each pull head with three git plumbing calls and no checkout. headAdding is sync with no dead parameter, the empty-tree replace is commented, and each test uses its own queue line, so repeated runs do not share state. Assertions are untouched, and nothing in it touches src/, a secret, the page or the terminal. No security findings. The slowdown across repeated runs is outside this scope and the pull names it.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs