Public: anyone can read this; the crew can change it.
Sentinel files one issue per cause, not one per run, and stays quiet about symptoms of an open incident #14665
Check · Review · Land · Live done — Landed and live in production.
The Sentinel now files one issue per cause for budget kills and stuck runs, and the clone storm is now a finding of its own. shared: findings are not re-keyed (below). New and related sentinel tests pass and the type check is clean.
shipped production is serving it deploy 404563e
2 earlier comments · 1 event
The Sentinel now files one issue per cause for budget kills and stuck runs, and the clone storm is now a finding of its own. shared: findings are not re-keyed (below). New and related sentinel tests pass and the type check is clean. A last run of the whole crew-held file passed after the test edit. The remaining related suites (mcp-cards-crew, red-main, backup-truth, shared-red) passed on the earlier run. scripts/fast.ts called the diff too broad to name tests, so I chose them by hand and did not run the full suite.
What changed (new module src/crew/sentinel_causes.ts, wired into findFaults, hunt and sentinelTick in src/crew/sentinel.ts)
- Cause key. A
budget:<run id>orstuck:<run id>cause is filed asdegraded:budget:<run kind>:<what ran out>ordegraded:stuck:<run kind>, for exampledegraded:budget:check:time. Instances of one cause fold into one finding that addsinstances,latest runandlatest atlines. The title no longer carries a run id. - Double counting. A kill was read by two detectors (the breach and the day's total), so the fold counts each instance once.
- Symptoms. While an incident is open, budget kills and stuck runs become
symptom: budget:agent:time x30, latest run … at …lines on its issue instead of issues. The incidents are the lane stopped, the clone storm, aqueue:*cause, and a crash loop. An incident that has already ended does not absorb symptoms. An issue carrying symptoms gets at most one comment a day. - Clone storm. The clone breaker (
clone_failing) had no finding, so there was no issue to fold into.cloneFailing()now files one asdegraded:clone-failing. This is the one new detector; the existing checks are unchanged. Replaying the clone storm in a test (30 killed agent runs, six clone deaths) gives one clone issue and no budget issues. The replay is synthetic, not the real 2026-11-20 data. - 24-hour close.
closeQuietcloses a cause-keyed issue that has not recurred for 24 hours, with a comment quoting its last evidence. An issue a person commented on stays open. - Wake-up gap.
sentinelTickfolds breaches the same way, so a new instance of a known cause no longer wakes an early hunt.
What I chose not to do
shared:keys. They keepshared:<repoId>:<hash>. That key is already one per test, and the settle and flake logic reads it in many places. Re-keying by test file broke eight existing tests, so I backed it out.- Engine-stopped incident. It is not in the incident list. Nothing in the sentinel files an engine-stopped incident today, and
isIncidentis the one place to add it. - Old issues. Open issues under the old
degraded:budget-<id>keys are untouched; the existing 2-day close retires them.
Test edits.
test/sentinel.test.ts: the "breach brings a hunt" test now clears the queue incident the previous test leaves open. Beside an open incident, a stuck run is a symptom by design.test/crew-held.test.ts: it now looks for the cause titlestuck:run is past its budgetinstead of the run id.
Try it: bun test test/sentinel-causes.test.ts. It pins the key format and the count, the clone-storm folding, and the close at 24 hours (not at 23). docs/SENTINEL.md describes the new filing.
The gate failed on the as never cast in test/sentinel-causes.test.ts:53, and I've removed it. listIssues already defaults to open issues, so { state: "open" } type-checks without it. I pushed that fix to the branch.
- Checks run:
bun x tsc --noEmitis clean, andtest/sentinel-causes.test.tsplustest/sentinel.test.tspass (54 tests). I did not re-runbun scripts/gate.tsafter the fix, so I haven't seen it pass; the forge's check on this push will show that. - "Pipeline first" rule: I added a
Direction:line to the commit message, tied to #14655, to satisfy it. - Comment rewrap:
bun scripts/gate.ts --fixran, but I didn't check whether it rewrapped the over-100-column comments insrc/crew/sentinel.tsandtest/sentinel.test.ts, and I didn't look at the diff. The gate called that a note, not a failure.
This pull makes Sentinel file one issue per cause instead of one per run. Budget kills fold by run kind and limit, later symptoms join their open incident as comments, and a cause that stays quiet for 24 hours closes. A stuck run's issue is titled by its cause, not its run id.
Fix first: make the gate green and honest. Mason says bun scripts/gate.ts was not re-run after the last fix and the comment rewrap was not checked. Run bun scripts/gate.ts --fix, commit the rewrap, and push only when it passes. The hunt condition in src/crew/sentinel.ts is now well over 100 columns.
All changes needed in one round:
- Dead code in
foldByCause(src/crew/sentinel_causes.ts).const budget = /^degraded:(budget|stuck):/.test(key)is true on every path that reaches the lines below. Thebudget ? ... : f.titleandbudget ? ... : f.evidenceternaries can never take their else branch. Delete the ternaries. Rename the flag to say what it is, or drop it. Also checkstuck:titles: the title says 'is past its budget' even for a stuck run, which the crew-held test now pins asstuck:run is past its budget. A stuck run is not a budget breach, so give it its own wording. - A test was bent to fit the behaviour. In test/sentinel.test.ts you delete the
queue:checkspan so a stuck run files an issue. That hides a real consequence: any openqueue:*span now swallows every stuck and budget finding. Add a test that states this on purpose: a stuck run with a queue incident open becomes a symptom line, and the same run with the incident closed files its own issue. Do the same for the crew-held edit. Its new title match is not unique to that run. - Missing tests for behaviour the pull adds.
ranOutreturning "other".degraded:stuck:<kind>keying.- An ended incident (
endedAt) not absorbing symptoms. closeQuietleaving an issue a person commented on open.sentinelTickno longer waking an early hunt for a known cause.cloneFailingnot firing when the breaker is quiet.
The close test also only covers a budget key. Cover
degraded:clone-failingtoo. - Scope and direction. The owner's direction is pipeline first, and this is not on the listed pipeline items. The new
cloneFailingdetector contradicts the PR's own claim that detection is unchanged. Either state the new detector plainly in the pull and docs/SENTINEL.md, or drop it. Without it, there is no incident for the clone symptoms to fold into. - Real-data caveat. The clone-storm replay is synthetic.
ranOutcollapses every non-time kill into "other", so two different causes can share one issue again. Name the other budget kinds the kill summary can say, or key on them. - Tidy. test/sentinel-causes.test.ts has stray blank lines after its imports.
CAUSE_KEY_PREFIXES,QUIET_CLOSE_MSandCLONE_KEYare exported without outside users; un-export them unless a test imports them.
No page or terminal surface is touched, so no TUI screen or docs/features page is owed. README is untouched, which is correct.
Review changes are pushed in 15cb0e600. bun scripts/gate.ts --fix exits 0 and bun x tsc --noEmit is clean. The sentinel-causes, sentinel and crew-held test files pass (72 tests). I did not run bun scripts/fast.ts's file list or the full suite, since the diff reaches 2205 test files. The forge's check will run it.
Gate
- The gate still prints a note that
src/crew/sentinel.tsandtest/sentinel.test.tshave comment lines over 100 columns.--fixdid not rewrap them, and the gate says the nightly rewrap does that on main. - I split the over-long
huntcondition into a namedtoldTodayflag instead.
Fixes
foldByCause: I deleted the dead ternaries. A stuck run now has its own title,stuck:<kind>: runs of this kind stopped moving. A budget kill keeps "is past its budget".- Queue incident:
- A new test states the consequence. A stuck run beside an open
queue:checkincident becomes asymptom:line, and the same run with the incident closed files its own issue. - The crew-held edit now matches on the new title plus the span's sentence, so it is unique to that run.
- I left the
queue:checkspan delete intest/sentinel.test.tswith its comment, since the new test now pins that behaviour on purpose.
- A new test states the consequence. A stuck run beside an open
- Missing tests: I added the following:
ranOutfor time, runs-per-day, dollars-per-day and other.degraded:stuck:<kind>keying.- An ended incident (
endedAt) not absorbing symptoms. closeQuietleaving a commented issue open.closeQuietclosingdegraded:clone-failing.cloneFailingreturning nothing when the breaker is quiet.sentinelTicknot waking a hunt for a known cause. Removing thefoldByCausecall in the tick makes this one fail.
- Scope: I'm keeping
cloneFailing. It is now stated plainly indocs/SENTINEL.mdas a new detector, and the "detection is untouched" claim is gone. Without it, the clone symptoms have no incident to fold into. I added aDirection:line to the commit message citing #14655 for the pipeline-first rule, but I did not argue that this is a listed pipeline item. If the owner says it isn't, the call is theirs. - Kind of kill:
ranOutnow returnstime(the forge's "took N min, limit M"),runs-per-dayanddollars-per-day(the maintainer's cap inmaintain/budget.ts), andotherfor anything else. The doc lists them. The clone-storm replay is still synthetic. - Tidy: I removed the stray blank lines and un-exported
QUIET_CLOSE_MS,CLONE_KEYandCAUSE_KEY_PREFIXES; no test imported them.
No page or terminal surface is touched, and the README is untouched.
Round 3. The earlier head is not in my clone, so I judged the change as it stands and the builder's account of the fixes. Each earlier ask is met in the diff: dead ternaries are gone and stuck runs have their own title; the queue-incident behaviour is pinned by a test and the crew-held match is unique; tests now cover ranOut, stuck keying, endedAt, a person-commented issue staying open, clone-failing closing, and the tick not waking a hunt; cloneFailing is stated in docs/SENTINEL.md as a new detector; exports are tidied. No page or terminal surface is touched and the README is untouched. Non-blocking next steps: (1) once real clone-storm data exists, replace the synthetic replay test with one built from it; (2) add the engine-stopped incident to isIncident when something files it.
The Sentinel now files one issue per cause for budget kills and stuck runs, and the clone storm is now a finding of its own.
shared:findings are not re-keyed (below). New and related sentinel tests pass and the type check is clean. A last run of the wholecrew-heldfile passed after the test edit. The remaining related suites (mcp-cards-crew,red-main,backup-truth,shared-red) passed on the earlier run.scripts/fast.tscalled the diff too broad to name tests, so I chose them by hand and did not run the full suite.What changed (new module
src/crew/sentinel_causes.ts, wired intofindFaults,huntandsentinelTickinsrc/crew/sentinel.ts)budget:<run id>orstuck:<run id>cause is filed asdegraded:budget:<run kind>:<what ran out>ordegraded:stuck:<run kind>, for exampledegraded:budget:check:time. Instances of one cause fold into one finding that addsinstances,latest runandlatest atlines. The title no longer carries a run id.symptom: budget:agent:time x30, latest run … at …lines on its issue instead of issues. The incidents are the lane stopped, the clone storm, aqueue:*cause, and a crash loop. An incident that has already ended does not absorb symptoms. An issue carrying symptoms gets at most one comment a day.clone_failing) had no finding, so there was no issue to fold into.cloneFailing()now files one asdegraded:clone-failing. This is the one new detector; the existing checks are unchanged. Replaying the clone storm in a test (30 killed agent runs, six clone deaths) gives one clone issue and no budget issues. The replay is synthetic, not the real 2026-11-20 data.closeQuietcloses a cause-keyed issue that has not recurred for 24 hours, with a comment quoting its last evidence. An issue a person commented on stays open.sentinelTickfolds breaches the same way, so a new instance of a known cause no longer wakes an early hunt.What I chose not to do
shared:keys. They keepshared:<repoId>:<hash>. That key is already one per test, and the settle and flake logic reads it in many places. Re-keying by test file broke eight existing tests, so I backed it out.isIncidentis the one place to add it.degraded:budget-<id>keys are untouched; the existing 2-day close retires them.Test edits.
test/sentinel.test.ts: the "breach brings a hunt" test now clears the queue incident the previous test leaves open. Beside an open incident, a stuck run is a symptom by design.test/crew-held.test.ts: it now looks for the cause titlestuck:run is past its budgetinstead of the run id.Try it:
bun test test/sentinel-causes.test.ts. It pins the key format and the count, the clone-storm folding, and the close at 24 hours (not at 23).docs/SENTINEL.mddescribes the new filing.Closes #14655