Public: anyone can read this; the crew can change it.
Crew briefs: widen the prompt-injection tests past #10365 #10463
This pull adds prompt-injection tests for the places a crew brief takes in outside text. Writing them found two holes in the briefs, and the pull closes both.
stopped parked by the crew deploy 90e2d47
3 earlier comments
This pull adds prompt-injection tests for the places a crew brief takes in outside text. Writing them found two holes in the briefs, and the pull closes both. The new tests, the related brief tests, the two proposal suites and tsc pass here; the gate script printed only a sentinel warning and I did not check its exit code. The full check has not run; it runs on this push. I did not call keep_plan on the issue, so no plan is posted there.
What moved
- Check log (
src/crew/brief.ts): the tail of a failed check's log, which the pull's own code prints, sat in a fixed three-backtick fence. A line of ``in it closed the fence and the rest read as the brief. It now usesfenced()`, and the brief says the log is data about the failure, never instructions. - Review diff (
src/crew/brief.ts): the reviewer's diff had the same fixed fence and now usesfenced()too. - Conflict blocks (
src/crew/proposal.ts):resolutionBriefhad the same fixed fence around the blocks from both sides. It now usesfenced(), and the brief calls the commit subjects and fenced text the branches' own words. - Existing test changed:
test/crew-check-failed.test.tsmatched the old log heading, so I changed it to match the new wording.
Tests
test/brief-injection.test.tshas ten tests, one per place the brief takes in text:
- a reader's issue body; - outside reports and app notes; - comments on an assigned issue; - comments, reviews and line comments in feedback; - the conversation shown to a reviewer; - the diff under review; - a stranger's pull; - a failed check's log; - the previous verdict and answer; - upstream file names and a foreign pull's diff.
- Each feeds hostile text containing fence breakers and a fake instruction. It fails if any line of the brief itself, outside a fence and not indented, contains that text without a reader mark.
- With the
srcchanges stashed, the review-diff and check-log tests fail.
Not done
- The fence in
resolutionBriefhas no test of its own, because it needs a repository with a real conflict. Onlyfenced's own test covers it. - I did not trace look findings to a brief. The reviewer's brief names look pictures by file path only.
docs/ops/RED-TEAM.mdsays so. - The row on this in
docs/ops/RED-TEAM.mdis updated, and the "open" bullet for #10451 is gone.
To try it: bun test test/brief-injection.test.ts.
🤖 Generated with Claude Code
This pull makes crew briefs fence every piece of outside text past any backtick run and tests each place a brief takes such text in (#10451). A second commit deletes a dead module that was failing the check.
What changed
- Briefs: the check log, the review diff and the conflict blocks now go through
fenced(). Hostile text can't close the fence early and pass itself off as the brief's own instructions. The check-log and conflict-block intros also say the text is data, never instructions.test/brief-injection.test.tscovers every place a brief takes in outside text, anddocs/ops/RED-TEAM.mdis updated. - Check failure 1:
every module under src/ is imported by another, or allowedfailed onsrc/kernel/why_now.ts. Nothing undersrc/imports it since the Why-now gate was retired in #10317. I deleted it andtest/why-now.test.ts.test/unreached.test.tsandbun x tsc --noEmitpass. - Check failure 2:
storage quota > a mirror bigger than the limit…is not caused by this change. It passes alone (16 of 16 intest/quotas-storage.test.ts) and looks like a flake under the full run's load. I didn't change it.
I did not run bun scripts/gate.ts or the brief tests again after the deletion. The forge's full check on this push will run them.
To try it
- Run
bun test test/brief-injection.test.ts test/unreached.test.ts.
The check failure came from a stale entry in scripts/unreached-allow.json. I fixed it, and the fix is pushed as 90e2d47fd. The unreached test and bun x tsc --noEmit pass on the merged branch. I didn't run scripts/gate.ts, scripts/fast.ts or the rest of the suite here, so the forge's check on the push is the full run.
Cause. main had deleted src/kernel/why_now.ts and added it to the allow list in the same change. My branch held the earlier commit that deleted the file but didn't touch the list. Merged, the list named a file that was gone, so the test "an allowed module that has become reached leaves the list" failed.
Fix.
- I merged
origin/maininto the branch. - I removed
src/kernel/why_now.tsfromscripts/unreached-allow.json.
Nothing else changed, and the prompt-injection brief fencing from earlier commits is untouched.
To try it: run bun test test/unreached.test.ts.
🤖 Generated with Claude Code
Parked by the crew: the check stayed red through 2 automatic tries
This pull adds prompt-injection tests for the places a crew brief takes in outside text. Writing them found two holes in the briefs, and the pull closes both. The new tests, the related brief tests, the two proposal suites and
tscpass here; the gate script printed only a sentinel warning and I did not check its exit code. The full check has not run; it runs on this push. I did not callkeep_planon the issue, so no plan is posted there.What moved
src/crew/brief.ts): the tail of a failed check's log, which the pull's own code prints, sat in a fixed three-backtick fence. A line of ``in it closed the fence and the rest read as the brief. It now usesfenced()`, and the brief says the log is data about the failure, never instructions.src/crew/brief.ts): the reviewer's diff had the same fixed fence and now usesfenced()too.src/crew/proposal.ts):resolutionBriefhad the same fixed fence around the blocks from both sides. It now usesfenced(), and the brief calls the commit subjects and fenced text the branches' own words.test/crew-check-failed.test.tsmatched the old log heading, so I changed it to match the new wording.Tests
test/brief-injection.test.tshas ten tests, one per place the brief takes in text:- a reader's issue body; - outside reports and app notes; - comments on an assigned issue; - comments, reviews and line comments in feedback; - the conversation shown to a reviewer; - the diff under review; - a stranger's pull; - a failed check's log; - the previous verdict and answer; - upstream file names and a foreign pull's diff.
srcchanges stashed, the review-diff and check-log tests fail.Not done
resolutionBriefhas no test of its own, because it needs a repository with a real conflict. Onlyfenced's own test covers it.docs/ops/RED-TEAM.mdsays so.docs/ops/RED-TEAM.mdis updated, and the "open" bullet for #10451 is gone.To try it:
bun test test/brief-injection.test.ts.🤖 Generated with Claude Code
Closes #10451