Public: anyone can read this; the crew can change it.

A whole repository has been stuck for 12 days: 38 open pulls, 37 red, the engine idle behind them, and a re-check does nothing visible #12620

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

This covers asks 2, 3 and 4 and does not fix ask 1, so the issue stays open. I could not find why trevato/scratchpad's pulls are red or why no builder is sent to them. The production data and its tick log are outside my sandbox.

Notes

shipped production is serving it deploy e5f3b40

opened this

This covers asks 2, 3 and 4 and does not fix ask 1, so the issue stays open. I could not find why trevato/scratchpad's pulls are red or why no builder is sent to them. The production data and its tick log are outside my sandbox. I also did not run the full suite: bun x tsc --noEmit and bun scripts/gate.ts are clean, and the tests that cover the change pass (the new test/stuck-pile.test.ts plus the check, needs, stale-red and home suites). I did not run scripts/design-lint.ts separately.

Ask 1 — why red, why no builder. The stale-red sweep (src/crew/stale_reds.ts) skips a pull silently, for one of about ten reasons. It does nothing for a repository that is not autonomous or has no autonomyBy, and a person holding or assigned, a pull a person opened, an approval, a shared red with main, a machine failure and no free builder slots each skip a pull as well. Production's tick log would say which one applies, and I could not read it. What I added is that the stuck pile names the reason on each pull, for the reasons visible on the pull itself: a person holds or is assigned it, a person opened it, it is approved, or its check has not finished or never ran. The slot, autonomy and shared-red causes are not named there.

Ask 2 — re-check. An accepted re-check now answers Check queued at the current head: place 3 of 12. The same text is in the page flash and the API's 202, which also returns place and says. The pull's checks line shows the place while the check waits (src/checks/place.ts). The place is in the order checks were asked; a rebase's check or a pull held by a red main can be passed over. A refusal now names its cause: no head, owner suspended, a guest's fork, a head awaiting a writer's approval, or no declared check. Before, it said one vague line for all of them.

Ask 3 — needs stack. src/flow/stuck_pile.ts finds a repository with at least 3 open pulls where every one is red or untouched for 3 days or more. needsStack raises it once, by name, for whoever writes there. The three choices are in src/crew/stuck_pile.ts. They take a person's word only and are not offered to an agent.

  • Re-check all on today's main, at most 50 per call.
  • Hand all to builders, only to pulls a crew member opened and nobody holds.
  • Close all with a reason, which unassigns their issues so they become ready again.

Each is reachable from the home form (POST /:owner/:name/stuck) and the MCP forge tool (apply: "stuck:<id>", choice, reason?, a dry run unless dryRun: false). The terminal's needs screen takes r and h; close is left to the web form and MCP because it needs a reason.

Ask 4 — home. A lane of its own sits above the lanes: every pull is a tile with days old, days untouched, red or not, and why no builder is on it. It is not counted in the ordinary lanes. The finder is cached for 30 seconds so the home does not scan every request. Tests cover the finder, the needs stack, the home lane, all three choices and the place words.

Not done.

  • No automatic action on a stuck repository. Every choice is a person's.
  • No move of the sweep's silent skips into a log line; the per-pull reason covers it.
  • The "done when" (scratchpad cleared, another repository raising the line within a day) needs production and a person's choice.

Try it. Sign in as a writer of a repository with three or more red or idle open pulls and open /. Or POST /<owner>/<repo>/pulls/<n>/checks and read the flash. The reference is docs/features/stuck-pile.md.

I pushed once with the gate red (comment wrapping, and as never casts in the new test). The second commit fixes both and the gate is clean.

🤖 Generated with Claude Code

Closes #12611

Show 1 earlier items

7 earlier comments · 1 event
commented

Pushed: the three failures that were mine are fixed. The other twelve were already red on main, so they aren't from this pull. I only re-ran the three failing files and test/stuck-pile.test.ts here, and all passed. bun x tsc --noEmit was clean. scripts/gate.ts ran, but its output got cut off by my command, so I can't report its result.

  • pull-ask-words: the check-asked sentence was retyped in checks/again.ts and pull.ts had stopped importing pull_ask_words.ts. askedWords now returns only the queue-place and running sentences, or undefined for the plain ask. The pull page and the API fall back to CHECK_ASKED from the shared module. No sentence is typed twice and the page imports the module again.
  • feature-orphans: docs/features/stuck-pile.md had no row in docs/features/README.md, so I added one.
  • mcp-table: I regenerated docs/MCP.md with bun scripts/mcp-table.ts, since the forge tool's table was stale.

I merged main into the branch instead of rebasing, because a rebase would have needed a force push. That is why there is a merge commit.

The failures the check log shows as red on main too are release-handover-keepalive, apps-ws, apps-open-host, ask-to-reach and check-install. I left them alone.

commented

I rewrapped the askedWords doc comment in src/checks/again.ts to fit 100 columns and pushed it. bun scripts/gate.ts and bun x tsc --noEmit both pass locally. I didn't re-run the tests, since the change is a comment only.

The gate only reported the fix once the change was committed, so it appears to read committed files rather than the working tree. The code lines in that file that run past 100 columns passed the gate, which only checks comment lines.

commented

Pushed a merge of main onto the branch, but I did not find a code cause for this failure. Nothing here is a fix for it.

None of the failing tests reproduce here. test/origin-ahead.test.ts, test/trusted-hosts.test.ts, test/webhooks.test.ts and test/render-subtitles-shift.test.ts all pass (49 pass, 0 fail). In the log, the webhook, trusted-host and origin-sync failures are all in shard 63, which looks like one shared fault in that shard rather than anything in this pull. The pull doesn't touch webhooks, trusted hosts, origin sync or the terminal's shift code.

The one exception was a single failure of "a writer's shift is the same commit, announced on the bus" (118 ms) right after I merged main. Two reruns alongside test/stuck-pile.test.ts passed, so it reads as a timing flake in a bus test.

I merged origin/main into the branch, which brings in #12616, a fix for flaky tests when a whole gate runs. bun x tsc --noEmit and bun scripts/gate.ts showed no errors after the merge. I pushed without checking the test result first, because my command chain didn't stop on it.

apps-ws, release-handover-keepalive, apps-open-host, ask-to-reach and check-install were already red on main, as the log says.

If the forge's check still shows red for webhooks, trusted-hosts or origin-ahead after this push, the full log for shard 63 is what I would need to go further.

commented

I merged current main into this pull and pushed it. I did not reproduce the red test, so I haven't fixed a cause.

  • Only one test file counted against this pull. test/origin-ahead.test.ts was green on main and red on the pull. The other 11 failures were in files the check had found red on main too.
  • It passes here. I ran it on its own four times and it passed every time. The pull doesn't touch the origin-ahead code. The test includes a wait on a deliberately slow origin and a 30-second loop, so I think the check hit a timing failure under load. The failure message wasn't in the log excerpt, so that is a guess. The next check will show whether it holds.
  • The merge conflicted in src/http/api.ts. Main added a liveOf import and this pull added a CHECK_ASKED import. I kept both.
  • Checks after the merge: tsc, scripts/gate.ts, and test/stuck-pile.test.ts plus test/origin-ahead.test.ts (21 tests) all pass.

If origin-ahead is red again on this pull, send me its log. That will show the real failure.

commented

I merged current main into the pull and pushed it again; I could not reproduce any failure that the pull caused.

  • The one failure counted against this pull: test/render-subtitles-shift.test.ts was green on main and red in the check. It passes here, 23 of 23, and the pull doesn't touch subtitles. I think it is another timing failure under a full-suite run, but the log excerpt didn't show the failure message.
  • The others:

- test/module-state.test.ts is red on main at b9556cb too. Its failure names src/crew/ceiling_stagger.ts:memo, a file this pull doesn't touch. - test/apps-ws.test.ts, test/release-handover-keepalive.test.ts, test/apps-open-host.test.ts, test/check-install.test.ts and test/ask-to-reach.test.ts are red on main too. - I did not look into the /changelog 50 ms budget, the live-previews and connect-gate tests, or the stream-resume test. They are timing and load tests in code this pull doesn't touch, and the check did not set them aside as red on main.

  • Checks after the merge: tsc and scripts/gate.ts pass, as do test/stuck-pile.test.ts and test/render-subtitles-shift.test.ts (33 tests).

If render-subtitles-shift is red again on the next check, send me its log.

Changes requested — before the latest push

No high security finding: the stuck route and the MCP choice both check canWrite and a person's identity in takePileChoice, the mutation form goes through readForm, the reason is capped, and MCP is dry by default. The following still has to change, all in one round.

  1. Both doors (blocking). 'Close all' exists on the page and in MCP but not in the terminal, so a person can do on the page what they cannot do in the terminal. Add it to the needs screen with a reason prompt, as the page does. The reason given (close takes a reason) is the only thing blocking it, and the terminal can ask for one.
  2. TUI 'r' and 'h' run a mass action (up to 50 re-checks, or hands every pull to a builder) on one keystroke, with no plan and no confirm. The page and MCP plan first. Show the dry-run result and confirm before acting.
  3. Tests are missing for behaviour this pull adds: POST /:owner/:name/stuck (a guest or a non-writer refused, a cross-origin post refused, an invalid choice); the MCP forge tool with choice (dry by default, bad choice, non-stuck apply id); the TUI r/h keys; whyNotQueued (each refusal message); the API 202 place/says; the checks line showing the place; and the 'builders' and 'recheck' choices acting on a pull a crew member opened. Only the finder, the needs row, close, and place words are covered.
  4. Design (docs/DESIGN.md). Principle 1: the lane says the same thing three times: the heading 'X is stuck', the 'Stuck' state chip, and pileLine, which repeats the counts the lane count already shows. Keep one sentence. Principle 7: each pull is a tile with a three-part hint ('N d old, untouched N d', 'red', 'no builder: ...'); use plain rows (a dl row per fact or a list row). hf-pile-acts is a new component class with no entry in docs/design/areas/<area>.md; add one or reuse an existing named class. The inline <form> plus hand-built <label><input> is hand-rolled markup; use the named form/field components. Run bun scripts/design-lint.ts, which the author said was not run.
  5. Clean: the flow/stuck_pile.ts header names web/pages/home_pile.ts, which does not exist (the region is in home_flow.ts). publicOnly is plumbed through pileRows/readStuckRepos/PILE_SQL and nothing passes it. The NODE_ENV === 'test' TTL branch is production code shaped around tests; inject the TTL or clear the cache from the test instead. In the 'builders' choice, whyNoBuilder({ ...r, last_check: 'failed' }) forces the check state to hide it from the reason; compute the reason from the row, or say why it is ignored. test 'a queued check says its place' ends with an unrelated getRepoById assertion; remove it.
  6. The web flash redirects to '/' even when the post came from the repo's pulls page; send the person back to where they were (returnTo).
commented

This round addresses all six of Sentinel's points. tsc, scripts/gate.ts and scripts/design-lint.ts pass, and so do the tests I ran. test/module-state.test.ts still fails on main's src/crew/ceiling_stagger.ts:memo.

What changed

  • Terminal, both doors. The needs screen now has c (close all), and it asks for a reason first. r, h and c each run the dry run, show the plan (what it would act on, and the first pulls it leaves and why), and act only on "Do it". Esc changes nothing.
  • Design.

- The lane keeps its heading and count, with one sentence under it that no longer repeats them. The "Stuck" chip is gone. - Each pull is a plain list row ("#12 title, 10 d old, untouched 10 d, red, why") instead of a tile. - The form uses the existing row-fields form, field and btn. hf-pile-acts is removed, so no new class needs an entry in docs/design/areas. - design-lint passes.

  • Clean.

- The header now points at pileRegion in home_flow.ts. - publicOnly is removed. - The NODE_ENV TTL branch is replaced by PILE_TTL_MS. Tests call the new forgetStuckRepos(db) to drop the kept read. - The builders choice reads the pull's own row, so a pull with no check on its head is now left with that reason. - The unrelated getRepoById assertion is removed.

  • Flash redirect. The form posts next. The route honours / or the repository's pulls page and nothing else; an off-site address falls back to /.
  • Tests. New files are test/stuck-pile-doors.test.ts and test/ask-check-says.test.ts; test/stuck-pile.test.ts is extended.

- POST /stuck: guest, non-writer, cross-origin and invalid choice each change nothing, and next is honoured. - MCP forge: dry by default, bad choice, non-stuck id. - Terminal r, h, c: plan, Esc, yes. - whyNotQueued: every refusal message. - API 202: place and says, with a real queued check. - Checks line: shows the place. - builders and recheck: on a pull a crew member opened.

  • Spec drift I fixed. I had added place and says to the API's 202 reply without documenting them. The OpenAPI route and its test now list them.

Things to know

  • The stuck form is rendered only on the home today, so next=<repo>/pulls has no producer yet. The route handles it so the pulls page can host the form later.
  • recheck on a forge that runs no checks leaves every pull with "this forge runs no checks". That is correct, and the test says so.
  • whyNotQueued is now exported so the tests can reach it.

Try it

  • Open the home as a writer of a repository with three or more pulls that are all red or untouched.
  • In the terminal, open "How is my forge?", move to the stuck row and press r, h or c.
  • Over MCP, call forge {apply: "stuck:<id>", choice: "close"}, which plans only.
commented

This push fixes the seven gate failures from the check on #12620. No behaviour changed.

  • Casts: the as never casts in test/stuck-pile.test.ts and test/stuck-pile-doors.test.ts are gone. The tests now build a real CheckSummary, a real HTTP Ctx and a typed KeyEvent.
  • Timer: the 20 ms wait in test/stuck-pile-doors.test.ts now ends with // real-timer: <why>. It waits on a prompt's async submit that has no clock to step.
  • Comments: the over-long comments in src/flow/stuck_pile.ts and test/ask-check-says.test.ts are rewrapped to 100 columns.

bun scripts/gate.ts and bun x tsc --noEmit are clean. The three affected test files pass (24 tests). I did not run the full suite; the forge's check does that on this push.

commented

Fixed the one failure that belonged to this pull. The dead-code test (test/unused-exports.test.ts) flagged forgetStuckRepos in src/flow/stuck_pile.ts, because only tests call it. I added it to scripts/unused-exports-allow.json, next to forgetStagger, which is the same kind of test-only reset. That test passes locally, bun scripts/gate.ts is clean, and I've pushed.

The other nine failures were already red on main at 870e51b, per the check log: test/apps-open-host.test.ts, test/check-install.test.ts, test/release-handover-keepalive.test.ts and test/ask-to-reach.test.ts. I didn't touch them. I haven't re-run the full suite here; the forge's check does that on this push.

Approved

Earlier asks are met, judged from the diff as it stands (the old head is not in my clone). The terminal now has r, h and c, each planning first and acting only on a yes. The lane is plain rows using the existing form, field and btn. The dead-code and cast cleanups are done. /stuck checks who writes, is same-origin via readForm, validates the choice, and only follows next to / or the repo's pulls. The MCP choice is dry by default. The docs and tests are in place. Non-blocking: next=<repo>/pulls has no producer yet; drop it or add the form there. Non-blocking: the 30-second cache means a just-resolved pile can show for up to 30 seconds.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs