Public: anyone can read this; the crew can change it.

A failing probe keeps one open defect, updated, not a new defect each week #13155

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A failing v1 probe now keeps one open defect. A later failure adds a comment to it, and a later pass says so once and leaves the defect open for a person to close.

Notes

shipped production is serving it deploy 59903fa

opened this

A failing v1 probe now keeps one open defect. A later failure adds a comment to it, and a later pass says so once and leaves the defect open for a person to close. Type-check, the gate and test/maintain-v1-proven.test.ts plus test/maintain.test.ts pass (28 tests); I did not run the full suite — the forge's check does that on the push.

What changed

  • Matching (src/maintain/proven.ts): the open defect is now found by a stable line in its body, Probe key: v1-proven/<probe issue>, instead of the probe_defect event note. A person renaming the title no longer makes the probe open a copy. probe_defect is no longer written, so the src/core/event_audience.ts entry for it is now unused.
  • Existing defects: defects filed before this change have no key in their body. The probe will open one new defect for each of them the next time it fails. After that each keeps its own.
  • Failing: with an open defect the probe adds "Still failing on …" with the new evidence and files nothing. With none open it files a defect that carries the key.
  • Passing: when a probe passes and its defect is open, it adds "Passing again on …, close this defect when you are satisfied". It does this only if the latest comment isn't already that note, so repeated passes say it once. It never closes the defect.
  • Duty (src/maintain/duties/v1_proven.ts): the report now runs on every run, not only when something fails, so a pass can be noted. Unproven probes are still ignored.

Test: one new test in test/maintain-v1-proven.test.ts. It renames a defect's title and checks a second failure files nothing and adds one "Still" comment. It then checks two passing runs leave exactly one "Passing again" note and the defect still open.

Not done: nothing closes a defect automatically, as the issue asks. The docs/features/v1-proven.md file is unchanged. The "Releasing trevato/forge" hold note on the desk did not touch this change.

Try it: bun test test/maintain-v1-proven.test.ts.

🤖 Generated with Claude Code

Closes #13134

4 earlier comments · 1 event
commented

A failing v1 probe now keeps one open defect. A later failure adds a comment to it, and a later pass says so once and leaves the defect open for a person to close. Type-check, the gate and test/maintain-v1-proven.test.ts plus test/maintain.test.ts pass (28 tests); I did not run the full suite — the forge's check does that on the push.

What changed

  • Matching (src/maintain/proven.ts): the open defect is now found by a stable line in its body, Probe key: v1-proven/<probe issue>, instead of the probe_defect event note. A person renaming the title no longer makes the probe open a copy. probe_defect is no longer written, so the src/core/event_audience.ts entry for it is now unused.
  • Existing defects: defects filed before this change have no key in their body. The probe will open one new defect for each of them the next time it fails. After that each keeps its own.
  • Failing: with an open defect the probe adds "Still failing on …" with the new evidence and files nothing. With none open it files a defect that carries the key.
  • Passing: when a probe passes and its defect is open, it adds "Passing again on …, close this defect when you are satisfied". It does this only if the latest comment isn't already that note, so repeated passes say it once. It never closes the defect.
  • Duty (src/maintain/duties/v1_proven.ts): the report now runs on every run, not only when something fails, so a pass can be noted. Unproven probes are still ignored.

Test: one new test in test/maintain-v1-proven.test.ts. It renames a defect's title and checks a second failure files nothing and adds one "Still" comment. It then checks two passing runs leave exactly one "Passing again" note and the defect still open.

Not done: nothing closes a defect automatically, as the issue asks. The docs/features/v1-proven.md file is unchanged. The "Releasing trevato/forge" hold note on the desk did not touch this change.

Try it: bun test test/maintain-v1-proven.test.ts.

🤖 Generated with Claude Code

commented

A failing v1 probe now keeps one open defect. A later failure comments on it with the new evidence, and a later pass says so once and leaves the defect open for a person to close.

The first push failed the gate's 100-column comment rule on two comment lines in src/maintain/proven.ts. I rewrapped them. After that fix bun scripts/gate.ts, bun x tsc --noEmit and test/maintain-v1-proven.test.ts pass locally; I ran no other tests, so the full check on this push still has to confirm.

What changed

  • Matching (src/maintain/proven.ts): the open defect is found by a body line, Probe key: v1-proven/<probe issue>, not by title or the old probe_defect event note. Renaming the title no longer makes the probe file a copy. The probe_defect event is no longer written, and src/core/event_audience.ts still lists it.
  • Failing: if a defect is open, the probe comments "Still failing on …" with the new line and files nothing. If none is open, it files one that carries the key.
  • Passing: if a defect is open, the probe comments "Passing again on …, close this defect when you are satisfied". It skips this when its own latest comment is already that note, so it says it once. It never closes the defect.
  • Duty (src/maintain/duties/v1_proven.ts): the report now runs on every run, not only on failures, so a pass can be noted. Unproven probes are still ignored.

Existing defects: any defect filed before this change has no key in its body. The probe will open one new defect for it on its next failure; after that each probe keeps its own.

Test: a new test in test/maintain-v1-proven.test.ts renames a defect's title, checks a second failure files nothing and adds one "Still failing" comment, then runs two passes and checks exactly one "Passing again" note with the defect still open.

Try it: bun test test/maintain-v1-proven.test.ts.

🤖 Generated with Claude Code

commented

I merged main into this branch, resolved the two conflicts, and pushed. bun scripts/gate.ts and bun x tsc --noEmit ran clean, and test/maintain-v1-proven.test.ts passes (9 tests). I did not run the rest of the suite; the forge's check runs it on the push.

What the two sides meant, and how I kept both

  • src/maintain/proven.ts:

- From this pull, a probe's defect is found by its Probe key: line in the issue body. A passing probe leaves a single "Passing again" note on an open defect and does not close it. An unproven probe is skipped. - From main (#13033), a filed defect and its "Still failing" comment carry the evidence block of what the probe read. - The merged code applies both: a failing probe keeps one open defect and quotes the evidence block in both the new defect and the comment.

  • test/maintain-v1-proven.test.ts: I kept both tests, the one for the body key and the "Passing again" note, and the one for the evidence block.
Changes requested — before the latest push

The behaviour is right and the test covers it, but three things need fixing in one round.

  1. Dead code: probe_defect is no longer written, but src/core/event_audience.ts still lists it. Remove the entry, and check that nothing else reads that kind.
  2. Docs: the how-to docs/features/v1-proven.md is unchanged, yet the behaviour changed. It now matches by a Probe key: body line, notes "Passing again" once, and never closes the defect. Document that there.
  3. Existing open defects have no key, so the next failure files a duplicate for each. That is the exact thing the issue forbids, and the pull description admits it. Also match the legacy Defect of #N title, or fall back to the old probe_defect event lookup for defects that have no key. Add a test for it.

Smaller: the test does not cover a pass when no defect is open, which must comment nothing. It also does not cover fail, pass, fail again, where the "Passing again" note must be repeatable after a new failure comment.

Boxes 1 and 2 are met in the happy path, but I leave them open until item 3 is fixed, since legacy defects get duplicated. Box 3 holds once item 3 has its test.

commented

All three of Juniper's items are fixed and the smaller test gaps are closed, in one commit on the same branch (8c55222be). bun x tsc --noEmit and bun scripts/gate.ts are clean. test/maintain-v1-proven.test.ts passes (12 tests), and test/event-audience.test.ts and test/event-audience-coverage.test.ts pass too. I did not run the full suite; the forge's check does.

  1. Dead code. I removed the probe_defect entry from src/core/event_audience.ts. A grep across src, test, docs and scripts found no other reader or writer of that kind.
  2. Docs. docs/features/v1-proven.md now describes:

- the Probe key: v1-proven/<probe issue> body line; - how a repeat failure is matched, including the legacy title; - the one-time "Passing again on <date>" note, which leaves the defect open for a person to close.

  1. Legacy defects.

- Change: openDefect in src/maintain/proven.ts now also matches an open issue titled Defect of #N: , so a defect filed before the key existed gets a "Still failing" comment instead of a duplicate. - Why the title and not the event lookup: I chose the title match over the old probe_defect event lookup because that event is no longer written and would have kept dead code alive. The trailing colon and space stop #12 from matching #123. - Test: a new test files a keyless defect, fails its probe, and checks that no issue is added and one "Still failing" comment lands.

I also added the two smaller tests Juniper asked for:

  • Pass with nothing open: a pass with no open defect adds no issue and no comment.
  • Pass, fail, pass again: fail, pass, fail again, pass, pass leaves the comments as "Passing", "Still", "Passing". The note repeats after a new failure and not otherwise.

I did not retitle or backfill keys onto existing defects. The title match covers them until they are closed.

commented

This push fixes the failed gate check: bun scripts/gate.ts --fix rewrapped the over-long comment lines in src/core/event_audience.ts, and I committed that rewrap. It only changes comment layout, so behaviour is the same.

After the rewrap, bun scripts/gate.ts and bun x tsc --noEmit both pass locally. I did not rerun the tests, since no code changed.

The earlier changes in this pull are as before:

  • Legacy open defects are found by title.
  • The dead probe_defect audience is removed.
  • The key and the passing note are documented.
Approved

Earlier asks are met: the dead probe_defect audience entry is removed, docs/features/v1-proven.md describes the key, the legacy-title match and the one-time pass note, and the legacy-defect, pass-with-nothing-open and fail/pass/fail/pass cases each have a test. The latest push only rewraps comments in event_audience.ts for the gate, with no behaviour change. Two things I would do next: (1) Non-blocking: backfill the Probe key onto legacy defects when one is commented on, so the title fallback can be dropped later. (2) Non-blocking: have a pass comment carry the evidence block, as a failure does, so a person closing the defect sees what was read.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs