Public: anyone can read this; the crew can change it.

No token-shaped string is tracked: a test scans the repository's files for keys, PEM blocks and bearer tokens #14399

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

This adds test/no-secrets-tracked.test.ts, which reads every file git ls-files names, plus test/secrets-allow.json, the allow-list for test fixtures. The new test passes (4 tests, about 4 s), as do scripts/gate.ts --fix and tsc --noEmit.

Notes

shipped production is serving it deploy 4ec2c16

opened this

This adds test/no-secrets-tracked.test.ts, which reads every file git ls-files names, plus test/secrets-allow.json, the allow-list for test fixtures. The new test passes (4 tests, about 4 s), as do scripts/gate.ts --fix and tsc --noEmit. I pushed before running the lint tests that scan the whole tree; I ran ten of them afterwards and all 101 of their tests passed. I did not run the full set of 105 that scripts/fast.ts named, because its estimate was 8 minutes against a 5-minute budget.

What the scan fails on. It names file, line and rule, never the value:

  • PEM private key headers.
  • sk- keys.
  • gh[pousr]_ GitHub tokens (36 or more characters).
  • xox?- Slack tokens.
  • AWS AKIA/ASIA key ids.
  • Bearer literals of 20 or more characters.

One choice to check. About 60 test fixtures across roughly 50 files look like sk-ant-oat01-the-owners-work-account: words joined by hyphens. The sk- and Bearer rules therefore only fire when the value contains a run of 20 or more letters and digits that includes both, which is what a random key looks like. Without that rule the allow-list would have about 50 entries instead of a short one. The fixed vendor shapes (GitHub, Slack, AWS, PEM) match as they are, with no extra condition. The rule is written down in a comment in the test and has its own test both ways: word-built fixtures and ` Bearer ${token} ` pass, and a realistic random key fails.

The allow-list. It has six entries, one per file and rule, each with a reason. All six are stand-ins in tests that check redaction or refusal: apps-proxy-unavailable, desk, mcp-preview, notify-rollback, origins-oauth and sentinel. If an entry stops matching anything, the test fails, so the list can only shrink. A new false positive should be added to the list; the rules themselves should not be loosened.

Self-test. It builds a sample for every rule in memory from pieces, so the test file holds no key-shaped text. It checks that each sample is caught at the right line, that the finding never contains the value, and that every rule has a sample.

Not done. I did not change any existing fixture, and I did not add a scanner under scripts/; the issue scopes this to the two test files.

To try it: bun test test/no-secrets-tracked.test.ts. Then add a line like "gh"+"p_" followed by 36 random letters and digits, written as one literal, to any tracked file and commit it; the test names that file and line.

🤖 Generated with Claude Code

Closes #14395

commented

This adds test/no-secrets-tracked.test.ts, which reads every file git ls-files names, plus test/secrets-allow.json, the allow-list for test fixtures. The new test passes (4 tests, about 4 s), as do scripts/gate.ts --fix and tsc --noEmit. I pushed before running the lint tests that scan the whole tree; I ran ten of them afterwards and all 101 of their tests passed. I did not run the full set of 105 that scripts/fast.ts named, because its estimate was 8 minutes against a 5-minute budget.

What the scan fails on. It names file, line and rule, never the value:

  • PEM private key headers.
  • sk- keys.
  • gh[pousr]_ GitHub tokens (36 or more characters).
  • xox?- Slack tokens.
  • AWS AKIA/ASIA key ids.
  • Bearer literals of 20 or more characters.

One choice to check. About 60 test fixtures across roughly 50 files look like sk-ant-oat01-the-owners-work-account: words joined by hyphens. The sk- and Bearer rules therefore only fire when the value contains a run of 20 or more letters and digits that includes both, which is what a random key looks like. Without that rule the allow-list would have about 50 entries instead of a short one. The fixed vendor shapes (GitHub, Slack, AWS, PEM) match as they are, with no extra condition. The rule is written down in a comment in the test and has its own test both ways: word-built fixtures and ` Bearer ${token} ` pass, and a realistic random key fails.

The allow-list. It has six entries, one per file and rule, each with a reason. All six are stand-ins in tests that check redaction or refusal: apps-proxy-unavailable, desk, mcp-preview, notify-rollback, origins-oauth and sentinel. If an entry stops matching anything, the test fails, so the list can only shrink. A new false positive should be added to the list; the rules themselves should not be loosened.

Self-test. It builds a sample for every rule in memory from pieces, so the test file holds no key-shaped text. It checks that each sample is caught at the right line, that the finding never contains the value, and that every rule has a sample.

Not done. I did not change any existing fixture, and I did not add a scanner under scripts/; the issue scopes this to the two test files.

To try it: bun test test/no-secrets-tracked.test.ts. Then add a line like "gh"+"p_" followed by 36 random letters and digits, written as one literal, to any tracked file and commit it; the test names that file and line.

🤖 Generated with Claude Code

commented

This adds test/no-secrets-tracked.test.ts, which reads every tracked file and fails on a key-, token- or PEM-shaped string, naming the file, line and rule but never the value. Fixtures that are deliberately key-shaped go in test/secrets-allow.json, one entry per file and rule with a reason each.

What the scan fails on:

  • PEM private key headers.
  • sk- keys.
  • gh[pousr]_ GitHub tokens (36 or more characters).
  • xox?- Slack tokens.
  • AWS AKIA/ASIA key ids.
  • Bearer literals of 20 or more characters.

What to look at: the sk- and Bearer rules. These two fire only when the value contains a run of 20 or more letters and digits that includes both. About 60 existing fixtures look like sk-ant-oat01-the-owners-work-account: words joined by hyphens. Without this rule each of those files would need an allow-list entry. A test checks it both ways: word-built fixtures and ` Bearer ${token} ` pass, and a realistic random key fails. The GitHub, Slack, AWS and PEM rules match their exact shapes with no extra condition.

The allow-list has six entries. All are stand-ins in tests that check redaction or refusal: apps-proxy-unavailable, desk, mcp-preview, notify-rollback, origins-oauth and sentinel. An entry that no longer matches anything fails the test, so the list can only shrink. A new false positive should be added to the list; the rules should not be loosened.

Self-test. A sample for every rule is built in memory from pieces, so the test file itself holds no key-shaped text. The test checks that each sample is caught on the right line, that the finding never contains the value, and that every rule has a sample.

Change to scripts/tree.ts. The first check run failed because the check's directory is written by checkout-index and can sit inside another repository. There, git ls-files exits 0 with an empty list, so the scan read nothing and every allow-list entry looked stale. trackedFiles already walked the directory when git ls-files failed; it now does the same when the list is empty, and the new test lists files through it. test/tree.test.ts uses the same function and gets the fix too. I reproduced the failure by exporting the tree with checkout-index into a directory nested in a fresh repository; both tests pass there now, and in the clone.

Checks run: scripts/gate.ts --fix (it notes two lines over 100 columns in scripts/tree.ts that were already there), tsc --noEmit, and the test files tree, no-secrets-tracked, tree-scans, unused-exports and test-honesty, with no failures. I did not run the full list of 105 tests that scripts/fast.ts named, because its estimate was over the 5-minute budget.

Not done: no existing fixture was changed.

To try it: bun test test/no-secrets-tracked.test.ts. Then commit any tracked file containing ghp_ plus 36 random letters and digits, written as one literal; the test names that file and line.

🤖 Generated with Claude Code

Approved

Clean, test-only change. The scanner reads tracked files, reports file:line:rule without the value, and has six reasoned allow-list entries that fail when stale. The self-test builds samples in memory for every rule. The trackedFiles fallback to the walk on an empty list is justified and small. Not blocking: the allow-list is per file and rule, not per line, so a real token added later in an allowed file would pass.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs