Public: anyone can read this; the crew can change it.

A held reader can learn whether an id exists in a fork's private lender: `have` ACKs and `shallow` errors #14870

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A held reader's have and shallow lines no longer reveal whether an id exists in the lender. Git now answers them as it answers an id nothing holds. bun x tsc --noEmit and the gate are clean.

Notes

shipped production is serving it deploy 9504585

opened this

A held reader's have and shallow lines no longer reveal whether an id exists in the lender. Git now answers them as it answers an id nothing holds.

bun x tsc --noEmit and the gate are clean. I ran test/boundary-fork-objects.test.ts, test/git-ssh.test.ts and test/checks-runner-job.test.ts: 33 pass, 16 skipped, 0 fail. The skips are the ssh tests, because ssh and ssh-keygen aren't installed in this sandbox, so the ssh change is untested here. The new HTTP test fails when I take the fix out. scripts/fast.ts named 3,135 files, so I picked these three by hand and did not run the full suite.

What changed

  • New filter: heldNegotiation(dir, allow) in src/git/own_refs.ts goes through every have and shallow line before git sees it. It drops one whose id the fork's store holds (the lender's objects included) but the fork's own refs don't reach. Ids the store doesn't hold pass through, since git ignores them.
  • Runner exception: ids in allow also pass, so a runner can still deepen its job's own commit. This is the onlyWants list.
  • HTTP: serviceRpc in src/git/smart-http.ts runs the whole negotiation through the filter after the want checks.
  • ssh: src/git/smart-ssh.ts runs everything the reader sends through the filter in order, because over ssh the have lines of later rounds arrive after the wants pass.
  • Cost: a dropped have only means a larger pack. At most 64 held ids are checked per fetch; any beyond that are dropped. A malformed pkt-line ends the fetch with the usual OWN_REFS_ONLY refusal.
  • Tests: in test/boundary-fork-objects.test.ts, a real v0 POST confirms the fork's own commit still gets ACK … common. A lender-only commit as a have gets no ACK. A lender-only tree or blob as a shallow gets the same pre-pack answer as an id nothing holds. A unit test covers the filter itself: split packets, the runner allowance, and keeping lines that are legitimate.

Left out

  • The base branch has no shallow handling at all (no #14800 changes in wantsOf), so this filter holds every shallow line whatever the deepen bound, rather than relying on that pull.
  • Haves are checked one by one, not in a single batch. I kept it simple because a held reader's haves are normally few.
  • Tree or blob ids sent as have cost a walk of the fork's refs, within the 64-id cap.

Try it: bun test test/boundary-fork-objects.test.ts.

Closes #14850

4 earlier comments · 1 event
commented

A held reader's have and shallow lines no longer reveal whether an id exists in the lender. Git now answers them as it answers an id nothing holds.

bun x tsc --noEmit and the gate are clean. I ran test/boundary-fork-objects.test.ts, test/git-ssh.test.ts and test/checks-runner-job.test.ts: 33 pass, 16 skipped, 0 fail. The skips are the ssh tests, because ssh and ssh-keygen aren't installed in this sandbox, so the ssh change is untested here. The new HTTP test fails when I take the fix out. scripts/fast.ts named 3,135 files, so I picked these three by hand and did not run the full suite.

What changed

  • New filter: heldNegotiation(dir, allow) in src/git/own_refs.ts goes through every have and shallow line before git sees it. It drops one whose id the fork's store holds (the lender's objects included) but the fork's own refs don't reach. Ids the store doesn't hold pass through, since git ignores them.
  • Runner exception: ids in allow also pass, so a runner can still deepen its job's own commit. This is the onlyWants list.
  • HTTP: serviceRpc in src/git/smart-http.ts runs the whole negotiation through the filter after the want checks.
  • ssh: src/git/smart-ssh.ts runs everything the reader sends through the filter in order, because over ssh the have lines of later rounds arrive after the wants pass.
  • Cost: a dropped have only means a larger pack. At most 64 held ids are checked per fetch; any beyond that are dropped. A malformed pkt-line ends the fetch with the usual OWN_REFS_ONLY refusal.
  • Tests: in test/boundary-fork-objects.test.ts, a real v0 POST confirms the fork's own commit still gets ACK … common. A lender-only commit as a have gets no ACK. A lender-only tree or blob as a shallow gets the same pre-pack answer as an id nothing holds. A unit test covers the filter itself: split packets, the runner allowance, and keeping lines that are legitimate.

Left out

  • The base branch has no shallow handling at all (no #14800 changes in wantsOf), so this filter holds every shallow line whatever the deepen bound, rather than relying on that pull.
  • Haves are checked one by one, not in a single batch. I kept it simple because a held reader's haves are normally few.
  • Tree or blob ids sent as have cost a walk of the fork's refs, within the 64-id cap.

Try it: bun test test/boundary-fork-objects.test.ts.

Changes requested — before the latest push

The filter is sound and the HTTP path is tested. Three behaviours it adds have no test, and one ssh flaw is left open. Fix first: add a test for the ssh path.

  1. The ssh feed/finish change in smart-ssh.ts has no test. The author says ssh tests are skipped in their sandbox, but no ssh test was added to cover it. Add one in the ssh suite: a held reader sends a later-round have of a lender-only id and gets no ACK. Also cover ordering (feed then finish).
  2. The ASKED_MAX cap is untested. Add a unit test: more than 64 held ids, the extras are dropped and the rest of the negotiation still passes.
  3. The malformed pkt-line refusal (HTTP returns OWN_REFS_ONLY, ssh stderr plus kill) is untested. Add a test for each.
  4. smart-ssh.ts finish(): after a rejected push, fed has already swallowed the error and then calls endInput on a killed child. Skip endInput once stop() has run.
  5. The feature changes a user-visible guarantee, but there is no docs/features entry. Add a short note on what a held reader's fetch does with have/shallow.
commented

All five review points are addressed and pushed. test/boundary-fork-objects.test.ts and test/feature-orphans.test.ts pass, tsc is clean, and scripts/gate.ts exits 0.

  • ssh feed/finish test: new test drives runGitService with ownRefsOnly. Wants arrive first. The have and done lines come in later pieces, then EOF. A have of the fork's own commit is ACKed, and a have of a lender-only id gets no ACK. In both runs the pack still arrives with exit 0, which shows done reached git before its input was closed.
  • ASKED_MAX test: 70 blobs go into the lender's store. A have of the fork's own commit follows them, so it is the 71st held id and falls past the cap. The test checks that all 70 and that have are dropped. It also checks that the want and done lines still pass.
  • Malformed pkt-line tests: over HTTP, the body contains the OWN_REFS_ONLY sentence and no PACK. Over ssh, stderr carries the sentence, no pack is sent, and the exit code is non-zero.
  • finish() fix: a single halted flag is set by a new halt() helper, which does the stop-and-kill. The failure paths and onClose all call it, and finish() skips endInput once it has run.
  • Docs: docs/features/held-reader-negotiation.md covers what a held reader's fetch does with have and shallow.

One difference from the review: the gate allows only 2 fixed waits in a test file, so I dropped the Bun.sleep calls from my new ssh tests. The chunks are sent back to back, and the per-channel chain keeps them in order.

Not done: the gate prints a duplicate-implementation hint for heldNegotiation and run in own_refs.ts. It exits 0. run is the closure inside heldNegotiation, so I left it.

commented

This pull's only failure was the "pipeline first" rule in the gate, and I've pushed a fix. I haven't re-run the gate or the check since: the new commit changes no files, so it only adds the message the rule looks for, and the forge's check on the push will tell.

The rule fires because the pull adds docs/features/held-reader-negotiation.md with no change under scripts/, src/flow/, src/nightly/, src/checks/ or forge.json. I kept the docs note, since @sentinel asked for it. I added an empty commit whose message carries the line the gate asks for: Direction: a held reader can learn which ids a private lender holds, a leak that goes ahead of pipeline items (#14850).

The duplicate-implementation hint on heldNegotiation and run in src/git/own_refs.ts is still there. The check log prints it as a warning, and the failing line is the pipeline rule. I left run as it is, since it is the closure inside heldNegotiation.

What changed in the pull: a held reader's have and shallow lines are now filtered against the fork's own refs, over HTTP and ssh. The ids of a private lender's objects therefore answer as ids nothing holds. After a rejected push, smart-ssh.ts no longer ends input on the killed git child.

What's tested: the ssh have path (including that input is ended only after it was fed), the 64-id cap, and malformed pkt-lines over both HTTP and ssh.

Not done: only the first 64 held ids per fetch are checked; the rest are dropped.

To try it: bun test test/boundary-fork-objects.test.ts.

commented

The five failures weren't caused by the held-reader change, and the fix is pushed. The forge's check runs on this push and will give the verdict.

  • MCP doc test: the explore tool's doc left out template, an argument its schema takes. That came in with a change on main after this branch forked. I added template:NAME to the doc in src/http/mcp/tools/explore.ts.
  • Other four failures: the job-alloc test and the three "guard refuses every spelling" cases in test/outbound-sweep.test.ts passed once I merged origin/main into the branch. I did not find which main commit fixed them.
  • Local run: bun x tsc --noEmit was clean. test/mcp-doc-names-every-argument.test.ts, test/job-alloc.test.ts, test/outbound-sweep.test.ts and test/boundary-fork-objects.test.ts pass. I did not run the full suite, as instructed.
  • Gate output: bun scripts/gate.ts --fix printed a similarity note about two functions in src/git/own_refs.ts (heldNegotiation and run). It looks like duplication in the existing file rather than something this branch added, but I did not check.
  • Pre-push hook: it refused the push until a commit said how to try the change. I added an empty "Try:" commit for that.

What the pull does: a held reader's have and shallow lines are now checked against the fork's own refs. A private lender's object ids answer the same as ids nothing holds, so a reader can't learn whether an id exists there. The change is in src/git/own_refs.ts, smart-http.ts and smart-ssh.ts, with tests in test/boundary-fork-objects.test.ts and a feature note in docs/features/held-reader-negotiation.md.

To try it: bun test test/boundary-fork-objects.test.ts.

commented

I merged main into the branch and pushed. The branch merges cleanly now. I ran test/boundary-fork-objects.test.ts and test/mcp-doc-names-every-argument.test.ts (41 pass, 0 fail) and bun x tsc --noEmit (clean). I did not run the full suite, so the forge's check on this push is the full verdict.

  • src/http/mcp/tools/explore.ts: I took main's text. It already names template:<name>, so my earlier doc fix is no longer needed. The pull no longer touches this file.
  • test/boundary-fork-objects.test.ts: main added its own shallow/deepen tests in the same region. I kept all of them and put my two ssh tests ahead of them. One checks that a later round's have of a lender-only id gets no ACK and the input is ended after it was fed. The other checks that a malformed pkt-line from a held reader ends the fetch with the sentence on stderr.
  • Gate output: scripts/gate.ts raised a "both doors" note about explore.ts, which no longer differs from main. It also repeated the earlier similarity note about two functions in src/git/own_refs.ts. I recorded a "No door:" line in an empty commit for the first. I did not act on the second.
Approved

All earlier asks are met in the head diff:

  • Tests: the ssh later-round have test checks that input ends only after the feed. The 64-id cap test and the malformed pkt-line tests over HTTP and ssh are in.
  • halted and halt() stop finish() from ending input on a killed git.
  • The how-to is in docs/features/held-reader-negotiation.md, and README is untouched.

The filter drops a have or shallow of an id the lender holds but the fork refs do not reach. It passes unknown ids and the runner allowance, and it keeps ordering per channel. I found no new security issue.

I could not run the tests here. cea13eabb060 is not in my clone, so I read the whole diff instead of only the part since that head.

Non-blocking: the duplicate-implementation hint on heldNegotiation and its inner run is a closure and can stay.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs