Public: anyone can read this; the crew can change it.

Bug hunters on a schedule, as a self-improving loop: every escape becomes a detector; KPIs (escapes → 0, valid ≥ 80%, fix < 24 h) #5846

shipped production is serving it opened this has it
opened this

Owner, 2026-10-25: "where are those [bug hunters]? can we get them hunting again?… it would also be great if it was a self-improving loop as well, with solid metrics and KPIs we can close on."

Where things stand (probe):

  • The old hunters were one-off agents from the steward's session. One of them left 16 busy loops on the owner's Mac (never again: hunters never run on the Mac).
  • The forge's own sentinel (#1131, src/crew/sentinel.ts) reads the run table, decision ledger and error ring, and files one issue per finding. The forge log shows no sentinel lines in the last 6 hours.
  • Two of today's bugs were found by the owner, not a hunter: "Your repositories" goes home (#5844), and the migrate page buries its URL field under 118 rows (#5816).

Done when: hunters run on the forge's own machines on a schedule.

  • When they run: after every release goes live, plus nightly at 08:00Z.
  • What they may do: read-only against production (GET only, by construction, as a validator must be), and full access against previews.
  • The lanes, each a small detector set:

- UI: headless browser as a signed-in test user and as anonymous, at 390 px and 1440 px. It clicks every link and button on the top pages, and flags: - a 404 or 5xx - a link that lands on / without being Home - a button with no effect - console errors - the main input below the fold - text overflow - Flows: sign in, migrate a repo, open and land a pull, create an app, drive the arm in its simulation. Each step is timed. - Speed: page p95 and the arm's command → readback p95 against budgets. - Contract: every OpenAPI route answers the documented status codes for good and bad input, and no route answers 500. - Front door: the sign-in 302, the signup gate 403, and private repositories 404 to anonymous visitors.

  • Each finding is exactly one issue (the sentinel's rule): a minimal repro, a screenshot or response, and the detector that found it. A finding that's still there adds a comment only when its evidence changes.

The self-improving loop:

  1. Every escape becomes a detector. An escape is a bug the owner (or any human) found first. Labelling it escape makes the hunter add a regression detector for that class and prove it would have caught the original (red on the old commit, green on the fix). Example: #5844 becomes "every menu link lands where its label says".
  2. Weekly retro, automated: the yield of each detector (valid findings per $ and per minute). Detectors that are always quiet and cheap stay as regression guards; noisy ones (more than 20% invalid) are fixed or cut.
  3. Hunter findings feed the crew's queue like any issue, labelled hunted.

KPIs (a card on home, and a weekly comment on this issue):

KPIDefinitionTarget
Escapesbugs a human found before a hunter, per week→ 0 (≤ 1 by 2026-11-08)
Valid ratehunted issues confirmed real (not closed "invalid")≥ 80%
Time to fixmedian from hunted issue filed to fix landed< 24 h
Escape → detectorescapes with a regression detector within 48 h100%
Reopen ratefixes whose detector goes red again within 14 days< 5%
Costcrew $ per valid bug found (list price)tracked, falling
Hunt timewall time of one full post-release hunt< 10 min

Done when: the hunt runs after every release and nightly. #5844 and #5816 are labelled escape, and each has its detector. The KPI card shows the first week's numbers.

commented

Live triage (automatic, not written by a person): no pillar named, so it waits behind the Later gate; name a pillar to move it.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs