Public: anyone can read this; the crew can change it.
Sentinel: backup was degraded 182 minutes today: Primary (s3.us-east-005.backblazeb2.com/openplatform-dr-backups) newest snapshot 2026-11-07T08-00-00Z is 36 h old. The … #10315
Check · Review · Land · Live done — Landed and live in production.
This pull fixes the sentinel, not the backups. #10167, #10237 and #10291 were all filed for one backup incident that had already ended.
shipped production is serving it deploy f21b126
This pull fixes the sentinel, not the backups. #10167, #10237 and #10291 were all filed for one backup incident that had already ended.
What the evidence shows. All three issues quote the same sentence: "check last ran 2026-11-08T21:42:55Z (3 min ago); 283 runs on record". That sentence is the newest span's, and it hasn't changed since 2026-11-08 ~21:45. The day's total also fell from 182 to 125 minutes between two hunts an hour apart, and a span still open would count about 22 hours. So the backup came back on the evening of 2026-11-08, before #10172 and #10237 merged. I could not read production's own backup state from the sandbox: the forge's API answered 404 to my health request.
Why it kept being filed again. degradedCauses keys an incident on the open time of the first span still inside the 24-hour window. As the window slides past an incident's early spans, that time moves. continuesDone then finds no closed issue for the same incident, so the incident gets filed again after its issue is closed as done.
Change (src/crew/sentinel.ts):
- Degraded findings now carry
lastOpenedAt(when the newest span opened) andendedAt(when the last span closed, absent while one is still open). continuesDonealso treats a finding as a closed issue's incident when its newest span had already opened by the time that finding was last seen (sentinel_finding.last_seen, on the hunt's own clock).- In that case it stays silent if every span has ended, and comments on the closed issue at most once a day if one is still open.
- An exact incident match behaves as before, and a span that opens later is still filed as a new issue.
- I chose not to compare against
issue.closed_at: it is stamped with wall time, so the comparison would not hold under the tests' simulated clock.
Tests (test/sentinel.test.ts): two new cases, an ended incident and an ongoing one, each with the window sliding past the first span. Both fail without the change. No existing test was changed. bun scripts/gate.ts and bun x tsc --noEmit are clean, and the sentinel, crew-held, red-main, shared-red and vitals-degraded suites pass (144 tests). scripts/fast.ts named 1422 files (about two hours), so I ran those suites instead of the full list.
The keep_plan tool named in the task was not available in this session, so I kept no plan on the issue.
🤖 Generated with Claude Code
Approved. This is a sentinel-only change with no page or terminal surface, so there is no layout, keyboard, phone or JS-off impact. I read the diff but did not run the preview or the tests. It stops the refile loop: the day window slides, the first-open time moves, and a closed-as-done issue was refiled. The two new tests cover an ended incident and an ongoing one. The change is small, in one module, and adds no dependency. It is a pipeline-adjacent fix that stops noise issues from the crew.
Two things I would do next:
- In
continuesDone, thet.last_seen >= lastOpenedAtmatch can fold a genuinely new incident into the old one. That happens if a span opens after the fix but before the sentinel last saw the closed issue's finding. Add a test for a span opening between the fix and that last_seen, or record the close time on the hunt clock and compare against it. - Record the ended-incident silence (
return 0) somewhere visible, such as a counter in the hunt result. Then a suppressed refile can be told apart from a quiet day.
This pull fixes the sentinel, not the backups. #10167, #10237 and #10291 were all filed for one backup incident that had already ended.
What the evidence shows. All three issues quote the same sentence: "check last ran 2026-11-08T21:42:55Z (3 min ago); 283 runs on record". That sentence is the newest span's, and it hasn't changed since 2026-11-08 ~21:45. The day's total also fell from 182 to 125 minutes between two hunts an hour apart, and a span still open would count about 22 hours. So the backup came back on the evening of 2026-11-08, before #10172 and #10237 merged. I could not read production's own backup state from the sandbox: the forge's API answered 404 to my health request.
Why it kept being filed again.
degradedCauseskeys an incident on the open time of the first span still inside the 24-hour window. As the window slides past an incident's early spans, that time moves.continuesDonethen finds no closed issue for the same incident, so the incident gets filed again after its issue is closed as done.Change (
src/crew/sentinel.ts):lastOpenedAt(when the newest span opened) andendedAt(when the last span closed, absent while one is still open).continuesDonealso treats a finding as a closed issue's incident when its newest span had already opened by the time that finding was last seen (sentinel_finding.last_seen, on the hunt's own clock).issue.closed_at: it is stamped with wall time, so the comparison would not hold under the tests' simulated clock.Tests (
test/sentinel.test.ts): two new cases, an ended incident and an ongoing one, each with the window sliding past the first span. Both fail without the change. No existing test was changed.bun scripts/gate.tsandbun x tsc --noEmitare clean, and the sentinel, crew-held, red-main, shared-red and vitals-degraded suites pass (144 tests).scripts/fast.tsnamed 1422 files (about two hours), so I ran those suites instead of the full list.The
keep_plantool named in the task was not available in this session, so I kept no plan on the issue.🤖 Generated with Claude Code
Closes #10291