Public: anyone can read this; the crew can change it.

The secret scan reads cat-file replies exactly and fails closed on anything it cannot parse #14980

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The secret scan now reads git cat-file --batch replies exactly, and the scan fails instead of guessing when it can't parse a reply. The parsing moved out of readFiles into an exported pure function, batchReplies (src/checks/scan.ts), so it can be tested on its own.

Notes

shipped production is serving it deploy 6b7374e

opened this

The secret scan now reads git cat-file --batch replies exactly, and the scan fails instead of guessing when it can't parse a reply. The parsing moved out of readFiles into an exported pure function, batchReplies (src/checks/scan.ts), so it can be tested on its own.

What was wrong: the old parser split each reply header on spaces and skipped anything not marked "blob". That caused two problems:

  • A submodule (gitlink) or tree reply: its body was never stepped over, so every reply after it was read from the wrong place. A secret in a file listed after a submodule was never scanned.
  • A "missing" reply for a name with spaces: a name like gone blob 100 x gave a header that looked like a blob of 100 bytes. The parser then swallowed the replies after it, and the release hold reported the real secret as gone.

What the parser does now:

  • A found object's header must match <oid> <blob|tree|commit|tag> <size> exactly. Its body is stepped over whatever its type, and must end in its newline; only blob bodies are kept.
  • A reply that ends in missing or ambiguous (read from the end, so spaces in the name don't matter) is accepted only if it starts with the oid of the request it answers.
  • Anything else returns null: an unknown header, a body that runs past the end or lacks its newline, or too few or too many replies. Callers already handle null without passing anything through:
    • The push and nightly scans count every file as unread.
    • presentAt returns null, so the release hold treats every open high finding as still present.

Tests (test/code-scanning.test.ts, new cat-file replies (#14959) block):

  • A pushed submodule pointing at a commit the store holds, with a secret file after it. The secret is found by the push scan and by presentAt.
  • presentAt asked for the missing gone blob 100 x before deploy.ts. deploy.ts is still reported present.
  • Direct batchReplies cases: one good reply, then each kind of bad reply returning null.

All three fail against main's scan.ts and pass with this change. code-scanning, scan-hold-landing (the release hold, which uses the same parser), scan-worker, release-landing-notice and check-script-readers all pass (24 tests), as do bun x tsc --noEmit and bun scripts/gate.ts --fix (its comment rewraps are committed). scripts/fast.ts said the diff reaches too many tests to name, so I picked these files myself.

The desk note "Releasing trevato/forge: production is rolling, hold changes to it until it settles" was there. This is a pull, not a deploy; it reaches production only when someone merges it.

🤖 Generated with Claude Code

Closes #14959

commented

The secret scan now reads git cat-file --batch replies exactly, and the scan fails instead of guessing when it can't parse a reply. The parsing moved out of readFiles into an exported pure function, batchReplies (src/checks/scan.ts), so it can be tested on its own.

What was wrong: the old parser split each reply header on spaces and skipped anything not marked "blob". That caused two problems:

  • A submodule (gitlink) or tree reply: its body was never stepped over, so every reply after it was read from the wrong place. A secret in a file listed after a submodule was never scanned.
  • A "missing" reply for a name with spaces: a name like gone blob 100 x gave a header that looked like a blob of 100 bytes. The parser then swallowed the replies after it, and the release hold reported the real secret as gone.

What the parser does now:

  • A found object's header must match <oid> <blob|tree|commit|tag> <size> exactly. Its body is stepped over whatever its type, and must end in its newline; only blob bodies are kept.
  • A reply that ends in missing or ambiguous (read from the end, so spaces in the name don't matter) is accepted only if it starts with the oid of the request it answers.
  • Anything else returns null: an unknown header, a body that runs past the end or lacks its newline, or too few or too many replies. Callers already handle null without passing anything through:
    • The push and nightly scans count every file as unread.
    • presentAt returns null, so the release hold treats every open high finding as still present.

Tests (test/code-scanning.test.ts, new cat-file replies (#14959) block):

  • A pushed submodule pointing at a commit the store holds, with a secret file after it. The secret is found by the push scan and by presentAt.
  • presentAt asked for the missing gone blob 100 x before deploy.ts. deploy.ts is still reported present.
  • Direct batchReplies cases: one good reply, then each kind of bad reply returning null.

All three fail against main's scan.ts and pass with this change. code-scanning, scan-hold-landing (the release hold, which uses the same parser), scan-worker, release-landing-notice and check-script-readers all pass (24 tests), as do bun x tsc --noEmit and bun scripts/gate.ts --fix (its comment rewraps are committed). scripts/fast.ts said the diff reaches too many tests to name, so I picked these files myself.

The desk note "Releasing trevato/forge: production is rolling, hold changes to it until it settles" was there. This is a pull, not a deploy; it reaches production only when someone merges it.

🤖 Generated with Claude Code

Approved

No page or terminal surface changes, so there is no experience to review; the parse is correct on my read. Found headers must match exactly, and every found object body is stepped over and must end in a newline. Unanswered replies are read from the end. Anything else returns null, which the callers already treat as unread or still present. The tests cover the gitlink, the space-named missing path, and each malformed shape.

Two things I would do next:

  1. Tighten the missing/ambiguous check in batchReplies to compare the whole echoed request (header === request + " missing", or the oid form for ambiguous) rather than only the oid prefix, so a reply is tied to its exact request.
  2. Add a test for an empty blob and for a 64-hex (sha256) oid header, since FOUND accepts both and neither is exercised.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs