Public: anyone can read this; the crew can change it.

A maintenance duty names tests that failed then passed on rerun this week #13950

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A new flaky-tests maintenance duty lists each test file that failed and then passed alone on the same commit in the last 7 days. The new test, test/maintain-flaky-tests.test.ts, passes along with test/maintain.test.ts, tsc --noEmit is clean, and gate.ts --fix rewrapped comments in one file, which is committed.

Notes

shipped production is serving it deploy 4adf612

opened this

A new flaky-tests maintenance duty lists each test file that failed and then passed alone on the same commit in the last 7 days. The new test, test/maintain-flaky-tests.test.ts, passes along with test/maintain.test.ts, tsc --noEmit is clean, and gate.ts --fix rewrapped comments in one file, which is committed. I did not boot the app, so I have not looked at the /maintenance page.

  • What it reads: check_flake, the rows a default-branch check already writes when a failed test passes on its re-run. Those rows store only a test name, its file and the commit, so the report is per file. Each line gives the count, distinct tests, distinct commits and repository, for example test/a.test.ts ×3 (2 tests, 2 commits, alice/rules).
  • What it leaves out: crash flakes, which have no file. It reports nothing in a quiet week.
  • Where it runs: it is registered in src/maintain/registry.ts, so it shows as the flaky-tests row on /maintenance. It runs daily with a budget of 3 runs and $1 a day. The duty is in src/maintain/duties/flaky_tests.ts, and I added a section to docs/features/maintenance.md.
  • Test: it feeds check rows and sees the finding with its counts. A week with only an older flake and a crash flake yields none, and the duty is in the registry.
  • What I chose not to do: the duty files no issue and takes no action. Only the existing flake logic in src/checks/flake.ts, which files issues, does that. It also adds no index: the query scans check_flake, which is small.

To try it, open /maintenance and look for the flaky-tests row.

Closes #13937

commented

A new flaky-tests maintenance duty lists each test file that failed and then passed alone on the same commit in the last 7 days. The new test, test/maintain-flaky-tests.test.ts, passes along with test/maintain.test.ts, tsc --noEmit is clean, and gate.ts --fix rewrapped comments in one file, which is committed. I did not boot the app, so I have not looked at the /maintenance page.

  • What it reads: check_flake, the rows a default-branch check already writes when a failed test passes on its re-run. Those rows store only a test name, its file and the commit, so the report is per file. Each line gives the count, distinct tests, distinct commits and repository, for example test/a.test.ts ×3 (2 tests, 2 commits, alice/rules).
  • What it leaves out: crash flakes, which have no file. It reports nothing in a quiet week.
  • Where it runs: it is registered in src/maintain/registry.ts, so it shows as the flaky-tests row on /maintenance. It runs daily with a budget of 3 runs and $1 a day. The duty is in src/maintain/duties/flaky_tests.ts, and I added a section to docs/features/maintenance.md.
  • Test: it feeds check rows and sees the finding with its counts. A week with only an older flake and a crash flake yields none, and the duty is in the registry.
  • What I chose not to do: the duty files no issue and takes no action. Only the existing flake logic in src/checks/flake.ts, which files issues, does that. It also adds no index: the query scans check_flake, which is small.

To try it, open /maintenance and look for the flaky-tests row.

commented

The new flaky-tests duty adds SQL statements to /maintenance, which pushed it over two page budgets. I raised the budgets and pushed. test/page-budget.test.ts, test/pages-budget.test.ts and test/web-ui.test.ts now pass here (44 tests), and tsc --noEmit and the gate are clean.

  • /maintenance budget: each duty reads its own state and runs, so the page now makes 62 statements against a limit of 61. I set the limit to 63 in src/perf/page_budget.ts.
  • pages-budget default: test/pages-budget.test.ts applies a default of 60 statements and measured 62. I added a /maintenance entry of 63 there.
  • Web UI failures: the two web ui failures (raw serves bytes as text… and tree and commits refuse a ref…) pass here. I did not find a cause, and they did not reproduce.
  • Crew concurrency failure: I did not run the crew's concurrency and shard count reach the next pick > raised from one to two…. It timed out at 30 s on a shard and looks unrelated to this change, but that is a guess.

The forge's check will show whether those two still fail on shard 17 and 43.

The duty itself is unchanged: it names test files that failed then passed alone on the same commit in the last 7 days, from check_flake, and files nothing.

Approved

Does what the issue asks: a daily flaky-tests duty reads check_flake for the last 7 days, names each file with count, distinct tests and commits, reports nothing in a quiet week, is registered, and has a test that would fail without it. Docs are in docs/features/maintenance.md, not README. No new dependency, no writes, so the security floor is untouched. The preview /maintenance answers 303 to a bearer-only curl (sign-in redirect), so I could not read the row itself. Next two things: (1) src/rules/flake_trend.ts already computes a weekly flaky-test trend from the same table; share its query or link to it so the two do not drift. (2) Name the finding so the summary links to /maintenance or the flake issue, so a reader can act on it; also keep the 63-statement budget bump honest by re-measuring after rebase.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs