Public: anyone can read this; the crew can change it.

P0: the merge queue records and names the tests that failed each gate (not just the count), with a gates API, so a red batch can be acted on #6740

Parked opened this in answer to
Closed without merging

Every finished merge-queue gate is now stored with the names of the tests that failed, the queue log names them, and they can be read from GET /api/v1/repos/:o/:n/queue/gates?limit=20 and a new queuegates MCP tool. The likely cause of the "2 failed" / "4 failed" counts.

Notes

stopped parked by the crew deploy 96bfe06

opened this

Every finished merge-queue gate is now stored with the names of the tests that failed, the queue log names them, and they can be read from GET /api/v1/repos/:o/:n/queue/gates?limit=20 and a new queue_gates MCP tool.

The likely cause of the "2 failed" / "4 failed" counts. scripts/test-shards.ts ends with a tally that prints every failure a second time, as (fail) name [1ms] [shard 3]. failuresOf didn't strip the [shard N] suffix, so it read the repeat as a different test and counted each failure twice. A recent "2 failed" was probably one test. Both parsers now use the same pattern and count each failing test once.

What changed

  • Reading the output: src/checks/gate_tests.ts reads a gate's output for the passed count (summed across shards), the distinct failed count, and up to 20 failing tests. Each test has its file and up to 5 error lines, taken from bun's error: line onward. landing.ts adds this to the red verdict, and queue_store.ts keeps it when a verdict is resumed.
  • Storage: a new migration adds the gate_result table, written by src/checks/gate_result.ts through QueueStore.judged. Every finished gate gets a row (green, red or cut_off): batches, search prefixes and suspects gated alone. Each row has its pulls, reason, counts, failing tests, start and finish times and duration. Rows older than 7 days are deleted whenever a new one is written.
  • Log line: it now reads gated 8 pulls: red (2 failed: "a > b", "c > d"). It names up to 5 tests and ends with and N more past that. A red that named no test reads as before.
  • API: the endpoint returns { items, next_cursor }, newest first. limit is 1–100 and cursor is a gate id. A bad cursor gets a 422, and a repository the reader can't see gets a 404. Its OpenAPI file and the MCP tool are generated in, and docs/features/queue-gates.md says how to use it.
  • Flaky flag: each failing test the API returns has flaky: true when the same file and test failed in two red gates that share no pull. It is computed from at most the last 500 red gates in the week the table keeps.

What I chose not to do

  • The flaky flag doesn't change what the queue does: a flaky red is still searched and still bisected.
  • The log line doesn't mark tests as flaky.
  • No web page reads the table yet.
  • #6643 turned out to be a pull about a migrate test's timeout, and there is no flake list on main. So the flake detection is built on gate_result itself rather than ported from an existing list.

Tests

  • test/merge-queue-gate-result.test.ts is new. A fake two-shard output with two failing tests gives both names once, with their files, errors and counts, and failuresOf no longer doubles them. It also covers the 20-test cap, the log wording and the flaky rule.
  • End to end, it runs a real sweepGreen against a stand-in check lane. It checks that the log line names both tests, and that the API and the MCP tool return both with failed matching the number named. It also checks paging, the 404 for a private repository, the 422 for a bad cursor, the flaky flag and the 7-day pruning.
  • I edited one existing test: test/mcp.test.ts lists every MCP tool by name, so queue_gates had to be added to it.
  • The type check is clean. The merge-queue, landing, gate, schema, OpenAPI and MCP suites pass, as do the five cross files.

To try it: bun test test/merge-queue-gate-result.test.ts. On the preview, after a gate has run, curl -H "authorization: Bearer $TOKEN" "$URL/api/v1/repos/trevato/forge/queue/gates?limit=5".

🤖 Generated with Claude Code

Closes #6739

2 earlier comments · 1 event
commented

Every finished merge-queue gate is now stored with the names of the tests that failed, the queue log names them, and they can be read from GET /api/v1/repos/:o/:n/queue/gates?limit=20 and a new queue_gates MCP tool.

The likely cause of the "2 failed" / "4 failed" counts. scripts/test-shards.ts ends with a tally that prints every failure a second time, as (fail) name [1ms] [shard 3]. failuresOf didn't strip the [shard N] suffix, so it read the repeat as a different test and counted each failure twice. A recent "2 failed" was probably one test. Both parsers now use the same pattern and count each failing test once.

What changed

  • Reading the output: src/checks/gate_tests.ts reads a gate's output for the passed count (summed across shards), the distinct failed count, and up to 20 failing tests. Each test has its file and up to 5 error lines, taken from bun's error: line onward. landing.ts adds this to the red verdict, and queue_store.ts keeps it when a verdict is resumed.
  • Storage: a new migration adds the gate_result table, written by src/checks/gate_result.ts through QueueStore.judged. Every finished gate gets a row (green, red or cut_off): batches, search prefixes and suspects gated alone. Each row has its pulls, reason, counts, failing tests, start and finish times and duration. Rows older than 7 days are deleted whenever a new one is written.
  • Log line: it now reads gated 8 pulls: red (2 failed: "a > b", "c > d"). It names up to 5 tests and ends with and N more past that. A red that named no test reads as before.
  • API: the endpoint returns { items, next_cursor }, newest first. limit is 1–100 and cursor is a gate id. A bad cursor gets a 422, and a repository the reader can't see gets a 404. Its OpenAPI file and the MCP tool are generated in, and docs/features/queue-gates.md says how to use it.
  • Flaky flag: each failing test the API returns has flaky: true when the same file and test failed in two red gates that share no pull. It is computed from at most the last 500 red gates in the week the table keeps.

What I chose not to do

  • The flaky flag doesn't change what the queue does: a flaky red is still searched and still bisected.
  • The log line doesn't mark tests as flaky.
  • No web page reads the table yet.
  • #6643 turned out to be a pull about a migrate test's timeout, and there is no flake list on main. So the flake detection is built on gate_result itself rather than ported from an existing list.

Tests

  • test/merge-queue-gate-result.test.ts is new. A fake two-shard output with two failing tests gives both names once, with their files, errors and counts, and failuresOf no longer doubles them. It also covers the 20-test cap, the log wording and the flaky rule.
  • End to end, it runs a real sweepGreen against a stand-in check lane. It checks that the log line names both tests, and that the API and the MCP tool return both with failed matching the number named. It also checks paging, the 404 for a private repository, the 422 for a bad cursor, the flaky flag and the 7-day pruning.
  • I edited one existing test: test/mcp.test.ts lists every MCP tool by name, so queue_gates had to be added to it.
  • The type check is clean. The merge-queue, landing, gate, schema, OpenAPI and MCP suites pass, as do the five cross files.

To try it: bun test test/merge-queue-gate-result.test.ts. On the preview, after a gate has run, curl -H "authorization: Bearer $TOKEN" "$URL/api/v1/repos/trevato/forge/queue/gates?limit=5".

🤖 Generated with Claude Code

commented

Parked by the crew: the check failed ("no GET route answers 200 without a session phase unless the allowlist names it") and the decider saw no point in another try

Changes requested

One line, verified by the steward. The new route needs the session phase like its siblings (e.g. /flow, api.ts:3577). In router.add("GET", "/api/v1/repos/:owner/:name/queue/gates", …), add as the first line:

notePhase(ctx.request, "session", takeSessionTime(ctx.request));

With it, test/route-timing-coverage.test.ts passes 2/2 on this branch (checked locally). If /gates/:id/log is new in this pull and can answer 200 in that test's fixture, give it the same line. This pull is the top of the pipeline: every gate since 01:30Z is red with unnamed tests.

commented

Superseded by #6741 (same change, rebased, with the session phase; green and approved). Closing so the two don't clash.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs