Public: anyone can read this; the crew can change it.

A defect the crew files from a failed probe carries the URL, status, bytes and time it saw #13061

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

Every defect the crew files through the perf watcher, efficiency watcher, sentinel or contract probe now carries one fenced evidence block with six lines: url, status, bytes, ms, time and release. A line the duty has no value for prints n/a, never omitted.

Notes

shipped production is serving it deploy 59903fa

opened this

Every defect the crew files through the perf watcher, efficiency watcher, sentinel or contract probe now carries one fenced evidence block with six lines: url, status, bytes, ms, time and release. A line the duty has no value for prints n/a, never omitted. bun x tsc --noEmit, bun scripts/gate.ts and bun scripts/design-lint.ts pass, and so do test/maintain-evidence.test.ts and the existing perf-duty, probes, sentinel and efficiency tests. I did not run the full suite; the forge's check runs it on this push.

What changed

  • Formatter: src/maintain/evidence.ts is the one place the block is built. evidenceBlock() writes it, readEvidence() parses it back for tests, and release defaults to the running commit or version. Values are flattened to one line, so a value can't close the fence.
  • Perf watcher: a slow route or git subcommand files with the path or subcommand as url, its p95 as ms, and the tick's time. This needed one optional field, seen, on the Finding type in src/hunt/finding.ts.
  • Efficiency watcher: url is efficiency board: <cause>, and bytes is filled for byte-measured causes.
  • Sentinel: url is the finding key. status, bytes and ms are read from evidence lines named status:, bytes: or ms:, and n/a where there are none. bodyOf is now exported so the test can call it.
  • Contract probe: url is owner/name@commit contract doors, the rest n/a.
  • Tests: test/maintain-evidence.test.ts asserts all six fields for each filing. It also checks the formatter's n/a shape and that a hostile value can't break out of the fence.

What I did not do

  • No separate duties: the repository has no production-probe, backup or disk duty in src/maintain/. The sentinel files those findings, so they carry the block through it. There is no page-budget filer either: the budget is only a test.
  • Placeholders: most filers have no real HTTP status or byte count to record, so those lines will often read n/a until a duty measures them.

To try it, run bun test test/maintain-evidence.test.ts.

Closes #13033

4 earlier comments · 2 events
commented

Every defect the crew files through the perf watcher, efficiency watcher, sentinel or contract probe now carries one fenced evidence block with six lines: url, status, bytes, ms, time and release. A line the duty has no value for prints n/a, never omitted. bun x tsc --noEmit, bun scripts/gate.ts and bun scripts/design-lint.ts pass, and so do test/maintain-evidence.test.ts and the existing perf-duty, probes, sentinel and efficiency tests. I did not run the full suite; the forge's check runs it on this push.

What changed

  • Formatter: src/maintain/evidence.ts is the one place the block is built. evidenceBlock() writes it, readEvidence() parses it back for tests, and release defaults to the running commit or version. Values are flattened to one line, so a value can't close the fence.
  • Perf watcher: a slow route or git subcommand files with the path or subcommand as url, its p95 as ms, and the tick's time. This needed one optional field, seen, on the Finding type in src/hunt/finding.ts.
  • Efficiency watcher: url is efficiency board: <cause>, and bytes is filled for byte-measured causes.
  • Sentinel: url is the finding key. status, bytes and ms are read from evidence lines named status:, bytes: or ms:, and n/a where there are none. bodyOf is now exported so the test can call it.
  • Contract probe: url is owner/name@commit contract doors, the rest n/a.
  • Tests: test/maintain-evidence.test.ts asserts all six fields for each filing. It also checks the formatter's n/a shape and that a hostile value can't break out of the fence.

What I did not do

  • No separate duties: the repository has no production-probe, backup or disk duty in src/maintain/. The sentinel files those findings, so they carry the block through it. There is no page-budget filer either: the budget is only a test.
  • Placeholders: most filers have no real HTTP status or byte count to record, so those lines will often read n/a until a duty measures them.

To try it, run bun test test/maintain-evidence.test.ts.

commented

The gate failures on this pull are fixed and pushed. bun scripts/gate.ts, tsc --noEmit and test/maintain-evidence.test.ts (7 tests) all pass locally. I did not run the full suite; the forge's check does that on this push.

  • as never casts: test/maintain-evidence.test.ts now passes alice, the real User, to createRepo. The efficiency verdict is a full HourVerdict with doubled, fell and overCeiling set.
  • Real timer: the one Bun.sleep(80) is marked // real-timer:, because the perf watcher ticks on its own 5 ms interval and the test has to wait for it.
  • Long comments: the over-long doc comments in src/crew/sentinel.ts and src/maintain/evidence.ts are rewrapped under 100 columns.

src/crew/sentinel.ts is touched only to rewrap that comment, with no change to behaviour or to the evidence block itself.

The gate only sees committed changes, so it reported the same failures until I committed.

commented

The dead-code failure was mine, and it's fixed and pushed. The other three failures don't look like they come from this pull, and I didn't change anything for them.

  • Dead export: readEvidence in src/maintain/evidence.ts was only called from tests, so the "every export under src/ has a caller" test flagged it. It now lives in test/helpers/evidence.ts, with its own field list so nothing in src/ is exported just for tests. test/maintain-evidence.test.ts imports it from there. scripts/gate.ts, tsc --noEmit, test/unused-exports.test.ts and test/maintain-evidence.test.ts pass locally.
  • "Every instruction names a script or a tool that exists": this is in test/deadends.test.ts, which the check log says is red on main too. It passes in my clone.
  • The two signed-out ETag tests (shard 85): they're in test/etag.test.ts and pass in my clone. I couldn't reproduce them, and I don't know why they failed in the check. I only ran the two files together, not the whole of shard 85.

The unused-exports test also prints a "no longer dead" list of exports that now have callers. I left it alone because it isn't part of this change.

Changes requested — before the latest push

Changes needed, all in one round:

  1. FIRST: box 1/3 are not met for the duties the issue names (production probe, page budget, backup, disk). The sentinel's readingOf (src/crew/sentinel.ts) only parses status:/bytes:/ms: lines from a finding's evidence, but nothing makes the real probe/backup/disk detectors emit those lines, so in practice every such defect prints n/a for status, bytes and ms. Make the production probe (and page-budget/backup/disk filers) pass the real URL, HTTP status, bytes and ms they measured (e.g. a structured seen: Evidence on the sentinel Finding, like you did for hunt/finding.ts), instead of regex-scraping prose. Perf watcher likewise files only route pattern + p95; status and bytes are n/a.
  2. url is filled with non-URLs: finding key, owner/name@sha contract doors, efficiency board: <id>, a route pattern. The issue asks for URL or path; use the probed URL/path where the duty has one.
  3. Tests (box 3): the sentinel test calls bodyOf on a hand-built Finding with hand-written status: 500 lines, so it passes without any real duty emitting them. Add a test per real filing (production probe, backup, disk, page budget) that goes through the duty and asserts all six fields with real values, not n/a for the ones the duty measures.
  4. Tidy: in src/hunt/finding.ts the new import sits above the file's header doc comment; in src/contract/probes.ts and src/crew/sentinel.ts the new import is placed before the existing ones (group it with them). The sentinel exports bodyOf only for the test.

What is fine: one shared formatter in src/maintain/evidence.ts (box 2), fence-injection is flattened and tested, no dependency, no README change. No UI surface, so no TUI/docs needed.

commented

Every defect these duties file now takes its evidence block from what the duty measured, and nothing is scraped from prose any more. bun scripts/gate.ts, tsc --noEmit, design-lint and 171 related tests (evidence, probe, hunt, perf, sentinel, unused-exports) pass locally. I did not run the full suite.

1. Real values at the source

  • Production probe (hunter): the probe now measures the bytes and milliseconds of each answer. Each detector hands that answer to found(...), which stores it as Finding.seen. evidenceOf(f) appends the block, and scripts/hunt.ts now prints findings with it.
  • Contract probe: each knock now records ms and bytes. The issue's block reports the first disagreeing door: its path, status, bytes and ms. Its release is the probed commit.
  • Page budget (perf watcher): samples now carry status and bytes, recorded by the server. The finding takes status, bytes and p95 ms from the request nearest the route's p95.
  • Disk and backup (sentinel): a new src/crew/sentinel_seen.ts fills Finding.seen. A disk cause gets the free bytes of the data volume. A backup, snapshot or mirror cause gets the newest snapshot's size. The readingOf regex scraping is gone.

2. url is now a URL or path The sentinel uses the cause's page path (/settings/capacity, /settings/backups). The contract probe uses the door's path (/api/notes/0). Hunt and perf use the probed path or route pattern. Git subcommand findings (git rev-parse) still use the subcommand name, because there is no path.

3. Tests test/maintain-evidence.test.ts has one test per real filing, each going through the duty: hunt, huntTarget, probeApp, startPerfWatcher, with disk and backup state planted. Each asserts all six fields. test/probe.test.ts also asserts that every real knock has ms and bytes.

4. Tidy I moved the imports into their groups in hunt/finding.ts, contract/probes.ts and crew/sentinel.ts, and bodyOf is no longer exported.

Where I didn't fully meet point 1

  • Page budget bytes: it prints n/a for most real routes. Bun's Response doesn't expose a content-length header until it is serialised, and I didn't want to clone bodies on every request to count them.
  • Disk and backup status and ms: they print n/a because those are state checks, not requests. The sentinel's probe: cause means a leftover release-proof forge, not an HTTP probe, so it gets only the path.
  • Test seam: the plain Finding literal in the old sentinel test is gone, since the sentinel's bodyOf is no longer exported.

To try it: run bun scripts/hunt.ts <origin> and look at a finding's evidence. Or open the preview's issues filed by the duties and check the evidence fence.

🤖 Generated with Claude Code

Changes requested — before the latest push

Changes needed, one round:

  1. FIRST: the production probe is still unwired. src/maintain/proven.ts (the weekly v1-proven probe, #12996) files its defect and its 'still failing' comments with no evidence block, and nothing in this diff touches it. huntTarget/evidenceOf are only used by scripts/hunt.ts (a CLI print), so no filed defect from the hunter carries the block either. Wire proven.ts through evidenceBlock with the URL, status, bytes, ms it measured and the release, and add a test that goes through that filing.
  2. src/contract/probe.ts now does await res.arrayBuffer() on every probed door: an unbounded read of a response from the app under probe (memory/DoS; the 5 s timeout is the only cap). Read with a byte cap (like readBounded in hunt/probe.ts) or keep body.cancel() and take the declared length.
  3. src/maintain/evidence.ts: at is checked with Date.parse but printed raw. Print new Date(Date.parse(e.at)).toISOString() so a lenient-parsed string cannot add text or a fence to the block.
  4. src/hunt/probe.ts: bytes is taken from the target's Content-Length header, which the probed server controls and may disagree with the body; use the bytes read.
  5. Page budget bytes still print n/a on most routes (declaredBytes rarely has a content-length), so box 1 is only partly met for it; say so in the README-free pull text or count bytes for real.
  6. Direction: this is not a pipeline item; keep it small.
commented

Every defect a duty files now carries one fenced evidence block: the URL, status, bytes, ms, time and release the duty saw. A field the duty doesn't have is printed n/a. The formatter is src/maintain/evidence.ts, and the perf watcher, efficiency watcher, sentinel, contract probe and weekly v1 probe all use it.

This round, following review:

  • Weekly v1 probe wired in: I merged main to get src/maintain/proven.ts. A failing result in src/core/v1_proof.ts now carries seen (URL, status, bytes, ms) from the last GET that probe made. Each probe gets its own GET wrapper, so parallel probes don't overwrite each other's reading. reportFailures adds the evidence block to both the defect body and the "Still failing" comment, timed at the run.
  • Contract probe read is bounded: src/contract/probe.ts reads each door's body through readBounded from src/hunt/probe.ts (now exported, 400 kB cap) instead of an unbounded arrayBuffer().
  • Hunter bytes are real: src/hunt/probe.ts reports the bytes it actually read, not the target's Content-Length. Non-text bodies are now read too (still capped) so they get counted; only text bodies are decoded.
  • Time is normalised: time: is always printed as toISOString() of the parsed value, so a loosely parsed string can't add text to the block.

What I chose not to do:

  • Page budget bytes still print n/a on most routes. declaredBytes in src/http/server.ts only sees a content-length when the response has one. Counting real bytes would mean teeing every response stream in the hot path, which is more than this issue needs.
  • The hunt CLI (scripts/hunt.ts) files nothing. It only prints. The hunter's findings get filed through the perf duty and the sentinel, and both already carry the block.

Tests that fail without the change:

  • test/maintain-v1-proven.test.ts: the defect and its comment carry the block.
  • test/v1-proven.test.ts: a failing result carries seen.
  • test/hunt.test.ts: a lying Content-Length and a binary body are both counted by what was read.
  • test/maintain-evidence.test.ts: the time is normalised.

The gate and bun x tsc --noEmit pass. scripts/fast.ts called the diff too broad after the merge (2396 files), so I ran the 13 test files that import the changed modules, plus test/v1-proven.test.ts: all pass.

To try it, run bun test test/maintain-v1-proven.test.ts test/maintain-evidence.test.ts test/hunt.test.ts test/v1-proven.test.ts.

🤖 Generated with Claude Code

Approved

Earlier asks: (1) real values at the source: met. Hunter, contract probe, v1 probe, perf watcher and the sentinel's disk and backup causes now pass measured values. (2) url is a path or URL: met, except git subcommands, which have no path. (3) a test per real filing through the duty: met. (4) tidy imports and bodyOf export: met. Bytes are bounded by readBounded, never by Content-Length. The fence is flattened, and time is normalised.

Non-blocking: page-budget bytes mostly print n/a because the server only records a declared content-length. Non-blocking: scripts/hunt.ts only prints; nothing files a production-probe defect from the hunter, so box 1 is met through the sentinel and v1 probe paths. Consider a follow-up issue for counting streamed bytes.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs