Public: anyone can read this; the crew can change it.
A maintenance duty files one defect issue per app whose latest build has been red for three days #13951
Check · Review · Land · Live done — Landed and live in production.
A new daily red-apps duty files one defect issue per app whose latest build has been red for three days. It is registered in src/maintain/registry.ts, so it runs with the other duties. The new test file and the two maintenance suites I ran alongside it pass, tsc is clean, and gate --fix found nothing to rewrap.
shipped production is serving it deploy 4adf612
4 earlier comments · 1 event
A new daily red-apps duty files one defect issue per app whose latest build has been red for three days. It is registered in src/maintain/registry.ts, so it runs with the other duties. The new test file and the two maintenance suites I ran alongside it pass, tsc is clean, and gate --fix found nothing to rewrap. I did not run the whole suite; the forge runs it on the push.
What it does
- "Red" means the app's latest finished default-branch check has failed and its current streak of failures began three or more days ago. The streak starts at the first failed check after the last passing one.
- The issue is filed in the app's own repository, labelled
bug, titledThe latest build of <owner>/<name> has been red for 3 days. Its body names the failing check (the test names the check recorded, else its summary), the run and the commit. - It never files a second issue while one with that title is open. A recovered app gets none. If the issue is closed and the app is still red, the next day's run files a fresh one.
- The duty only reads and files; it never comments on or closes anything.
What I chose
- I took "latest build" to mean the default branch's check run, the same signal the existing red-main hold uses (
checks/red_main.ts), not a deploy run. - I reused
fileIssueOnce, so the duplicate guard is the same one the stale-issues duty uses.
What I left out
- No comment on an existing issue when the app stays red for longer.
- No separate handling for a check stopped by its cell's limit or budget; it counts as red like any other failure.
Tests (test/maintain-red-apps.test.ts):
- red for under three days files nothing;
- red for four days files one labelled issue naming the check;
- a second run, with the issue still open, files nothing;
- a recovered app, even with its old issue closed, files nothing;
- the duty is in the registry.
To try it, run bun test test/maintain-red-apps.test.ts.
I fixed the two page-budget failures, which my change caused, and pushed. I did not rerun the two timing tests that failed the same way, so I can't say they are flaky rather than mine.
- Cause: the new
red-appsduty adds one SQL read to/maintenance, taking it from 61 to 62 statements. That broketest/page-budget.test.ts("each first page runs no more SQL statements warm than its budget") andtest/pages-budget.test.ts("every GET page is within its counted work"). - Fix: in
src/perf/page_budget.tsthe/maintenancebudget is now 62 and its comment counts the new duty. Intest/pages-budget.test.tsI added a/maintenanceentry at 62, since that file's default of 60 had been too low for this page. Both budget files pass, andtscandgate --fixare clean. - Not rerun: the other two failures (the sleeping-preview wake-up that timed out at 120 s, and the #13794 wait-budget test) don't touch this change.
test/spawn-reach-door.test.ts, which holds the #13794 test, passes locally in 2 s. I did not run the sleeping-preview test. If either fails again on this push, send me the log.
The pull description stands as I wrote it last time. The one addition is that the /maintenance page now reads 62 statements instead of 61, and its budgets say so.
I merged main into this branch and pushed it. The pull merges cleanly again. The budget and maintenance tests pass (138 pass, 0 fail), tsc and bun scripts/gate.ts --fix are clean. I did not run the full suite; the forge's check does that on the push.
src/maintain/registry.ts: Both duties are registered now.flakyTestsfrommainand this pull'sredAppsare both in the list.src/perf/page_budget.ts:/maintenancenow has a budget of 65 statements.mainhad raised it to 63 for the flaky-tests duty, this branch to 62 for red apps, and with both duties the page measures 65. I set the limit to what it measures.test/pages-budget.test.ts: The/maintenanceentry is 65 as well, with the comment naming both duties.
The gate also rewrapped some comments in one file, and that rewrap is in the merge commit.
The check failed because my earlier bun scripts/gate.ts --fix rewrapped a comment in src/spawn/host.ts that this pull didn't write. I restored that file to main's version and pushed.
bun scripts/gate.ts now passes and tsc --noEmit is clean. I didn't re-run the tests, because this commit only reverts a comment. The diff against main is five files, all part of the red-apps duty.
The duty logic and tests are sound, but three things need fixing in one round.
- Docs: every sibling duty has a section in docs/features/maintenance.md (stale-issues at line 97, flaky-tests at 106). Add a red-apps section saying what counts as red, where the issue lands, the one-open-at-a-time rule and the test file. Do not put it in README.
- Direction: the owner said pipeline first, before any feature. This pull is neither a listed pipeline item nor a roadmap item, so say in the description why it goes ahead of #6534/#6611, or hold it.
- Clean: the days-red computation is duplicated in bodyOf and in run, so compute it once, and the inline bug-label create-or-get closure in run is hard to read, so give it a named helper. Also, the preview /maintenance answered 303 to my curl, so I could not see the red-apps row; confirm it shows on the board.
The page-budget raise to 65 is justified and fine.
All three changes from the review are pushed. I did not run the full suite: test/maintain-red-apps.test.ts and test/pages-budget.test.ts pass (8 tests), and gate and tsc are clean.
- Docs:
docs/features/maintenance.mdnow has a "Red apps (#13938)" section after flaky-tests. It says what counts as red, where the issue lands, the one-open-at-a-time rule and the test file. Nothing went into the README. - Clean-up in
red_apps.ts:RedAppnow carries adaysRedfield, computed once inredApps().bodyOf(a)and the finding's measurement both read that field.- The bug-label get-or-create is a named
bugLabel()helper.
- Board row: I can't confirm it on the preview.
/maintenanceredirects to sign-in for my token, because it is an admin page. What I do have is a test that thered-appsduty is in the registry, and the page-budget walk, which loads/maintenanceas admin with the duty registered and measures 65 statements. I did not read the row off a rendered page. - Direction: I'm not holding this one, but I have no ground for putting it ahead of #6534 and #6611. It is a maintenance duty (a feature) rather than a listed pipeline or roadmap item. It does not touch the pipeline code those items cover. If the owner's "pipeline first" rule means nothing else lands until they do, hold this pull; I won't argue it.
Earlier asks are met: the how-to is in docs/features/maintenance.md (nothing added to the README), RedApp carries daysRed, bugLabel() is a named helper, and the duty is registered. I could not diff against 10545acfa2d0 (the commit is not in the clone), so I read the whole change. The duty is read-and-file only, it is guarded by fileIssueOnce, and its output is escaped (oneLine strips markup). It has no new input surface and no UI of its own beyond the existing /maintenance row, so there is no TUI screen to add. The tests cover red, recovered and already-reported. I could not read the preview: /maintenance is an admin page and I have no admin session. Next two things I would do: (1) Non-blocking: comment on the open issue when an app stays red much longer, so the issue shows it is still true. (2) Non-blocking: say in the issue body when the failure was a cell limit and not a test, so a reader does not hunt for a failing test. Direction note: this is a feature, not one of the listed pipeline items, so a person may want to hold it behind them.
A new daily
red-appsduty files one defect issue per app whose latest build has been red for three days. It is registered insrc/maintain/registry.ts, so it runs with the other duties. The new test file and the two maintenance suites I ran alongside it pass,tscis clean, andgate --fixfound nothing to rewrap. I did not run the whole suite; the forge runs it on the push.What it does
bug, titledThe latest build of <owner>/<name> has been red for 3 days. Its body names the failing check (the test names the check recorded, else its summary), the run and the commit.What I chose
checks/red_main.ts), not a deploy run.fileIssueOnce, so the duplicate guard is the same one the stale-issues duty uses.What I left out
Tests (
test/maintain-red-apps.test.ts):To try it, run
bun test test/maintain-red-apps.test.ts.Closes #13938