Public: anyone can read this; the crew can change it.

Fail fast, fail loud for the crew: every queue and run has a budget, past it is a failure named with its cause and handed to the crew at once #1270

shipped production is serving it opened this has it
opened this

The owner: "these failures that lock up the entire system need to be figured out. we need to fail faster and louder."

Today, 2026-10-13, the forge locked up five times and said nothing each time until a person read tables:

  • the review clock started at the preview's green, every pull lapsed, no reviewer was ever sent (#1216): hours, no line in the log;
  • six checks for already-merged pulls sat in the lane for two hours (#1239);
  • stale .git/config.lock files failed every follow-up on six pulls with a one-line error nobody saw (#1235);
  • a reviewer run took a builder's comment as its task and declined (#1237);
  • a check ran 45 minutes beside another; the OOM and the liveness restart re-queued everything (#1221).

The only loud thing was the home card "The check queue has waited over 30 minutes", and it said so after thirty minutes, with no cause and no action.

Done looks like:

  • [ ] Budgets, not patience. Every queue and every run has a budget written in one place (src/kernel/budgets.ts): a fast-tier check 5 min, a full gate 15 min, a review run 10 min, a check waiting in the lane 10 min, a run with no heartbeat 3 min. Past budget is a failure: the run is killed with budget: <what> took <n> min, limit <m>, re-queued once, and on the second time it is a card.
  • [ ] One sentence, one cause. The home line and the Today card name the stuck thing and its cause from the forge's own evidence, not the symptom: "22 checks wait behind one lane; the oldest since 04:13; the lane runs #1212 for 31 min (limit 15)". src/kernel/vitals.ts gathers it; /api/v1/health carries it as degraded: [...].
  • [ ] Loud for the crew, not the owner. The owner's words: "not necessarily loud for the user but for the crew." A budget breach becomes crew work at once: the forge files an issue with the evidence (what, since when, the cause sentence, the log lines), labels it P0 Trust, assigns the sentinel to diagnose and hand the mechanical fix to a builder, and the run starts within a tick. The owner sees one line on the home while it is open and nothing else unless the crew hands it over.
  • [ ] Sentinel hunts it (#1131): the nightly fault hunt reads the vitals history and files the mechanical fix for anything that was degraded more than ten minutes in the day.
  • [ ] Tests: a run past its heartbeat budget is killed and said; a lane older than its budget turns the health degraded with the sentence and files the issue once (not once per tick); the sentinel's run is queued on it.

Evidence: the issues above, all from one day.

commented

Parked by the crew: the check failed and the decider saw no point in another try (pull #1273 closed). Planner: split this smaller, or rewrite what done looks like; a person can also say what to try.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs