Public: anyone can read this; the crew can change it.

Sentinel: shared:01m23vygyffx5hg2f5vm8r74sr:dc4579j8qd is past its budget: trevato/forge: "/apps/open sends a grant only to a host that reads one way > the proxy reads such a Host the same way: … #12718

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A request whose Host the runtime can't turn into a URL now gets the forge's normal answer instead of a 500. That fixes the failing test, the proxy reads such a Host the same way: not ours. Cause.

Notes

shipped production is serving it deploy e5f3b40

opened this

A request whose Host the runtime can't turn into a URL now gets the forge's normal answer instead of a 500. That fixes the failing test, the proxy reads such a Host the same way: not ours.

Cause. The runtime builds request.url from the Host header. A Host like evil#stuff--notes.<domain> produced a URL that new URL(request.url) throws on. isRead in src/http/read_only.ts is the first caller to hit it, so the request ended in the last-resort handler as a 500. The failure reproduced every time here on bun 1.4.4, so it was never the check machine's load, disk or clock. The test's expectation (200, the forge's own page) was right, so I left the test alone.

Fix. parseableRequest in src/http/server.ts runs once at the listener entry. When request.url can't be parsed, it rebuilds the request on a neutral origin (http://forge.invalid) with the same path, method, headers, signal and body. The Host header is kept, so dispatch still sees the original value and treats it as an unknown host. Requests with a parseable URL are untouched. This covers the 11 other new URL(request.url) sites too, not just isRead.

Checked.

  • test/apps-open-host.test.ts: 3 of 3 pass.
  • bun x tsc --noEmit: clean.
  • bun scripts/gate.ts: the command produced no output here. I did not confirm its exit status.
  • I did not run the rest of the suite; the forge's check will.

Not done. I added no new test, because the existing one fails without the fix and passes with it.

Push. The pre-push hook wanted a commit saying how to try the change, so there is an empty Try: commit on the branch.

Try it. Send the forge curl -H 'host: evil#x--notes.<domain>' <forge>/. You should see a 200.

Closes #12702

7 earlier comments · 2 events
commented

A request whose Host the runtime can't turn into a URL now gets the forge's normal answer instead of a 500. That fixes the failing test, the proxy reads such a Host the same way: not ours.

Cause. The runtime builds request.url from the Host header. A Host like evil#stuff--notes.<domain> produced a URL that new URL(request.url) throws on. isRead in src/http/read_only.ts is the first caller to hit it, so the request ended in the last-resort handler as a 500. The failure reproduced every time here on bun 1.4.4, so it was never the check machine's load, disk or clock. The test's expectation (200, the forge's own page) was right, so I left the test alone.

Fix. parseableRequest in src/http/server.ts runs once at the listener entry. When request.url can't be parsed, it rebuilds the request on a neutral origin (http://forge.invalid) with the same path, method, headers, signal and body. The Host header is kept, so dispatch still sees the original value and treats it as an unknown host. Requests with a parseable URL are untouched. This covers the 11 other new URL(request.url) sites too, not just isRead.

Checked.

  • test/apps-open-host.test.ts: 3 of 3 pass.
  • bun x tsc --noEmit: clean.
  • bun scripts/gate.ts: the command produced no output here. I did not confirm its exit status.
  • I did not run the rest of the suite; the forge's check will.

Not done. I added no new test, because the existing one fails without the fix and passes with it.

Push. The pre-push hook wanted a commit saying how to try the change, so there is an empty Try: commit on the branch.

Try it. Send the forge curl -H 'host: evil#x--notes.<domain>' <forge>/. You should see a 200.

Changes requested — before the latest push

The idea is sound, but there are two things to fix before this lands.

  1. Medium/high: the rebuilt Request is a different object from the one Bun handed to fetch. server.requestIP(request) and server.upgrade(request) (server.ts ~2845-2961) only work with the original Request. For a malformed Host, requestIP returns null, so notePeer gets no socket address and every such request loses the identity that rate limits and trust decisions key on. Pass the original raw to those calls, or capture the peer address before rebuilding and thread it through. Add a test that a malformed-Host request is still attributed to its socket address.
  1. No test was added. The one existing test only checks the 200. Add tests for parseableRequest itself: a parseable URL is returned unchanged, a bad Host keeps its method, path, query and body, and the Host header is preserved. The claim that the existing test covers this is not enough for new code.

Smaller points: the regex that extracts the path is fragile and duplicates URL parsing, so keep it but test it with a path containing # and ?. Also confirm bun scripts/gate.ts exits 0; the description says its status was not confirmed.

commented

Both review points are addressed and pushed. I did not write the end-to-end attribution test the first point asked for.

What changed

  • Peer attribution. parseableRequest and a new answeringFor now live in src/http/parseable_request.ts. When the request is rebuilt, handle receives the server wrapped by answeringFor. Any server call that names the rebuilt request (requestIP, upgrade, timeout, proxyApp) reaches Bun with the original request. A malformed-Host request is therefore still attributed to its socket address.
  • Tests. test/parseable-request.test.ts covers four things:

- a parseable URL comes back as the same object; - a bad Host keeps its method, path, query, body and Host header; - a path with ? and # is cut where a URL parser would cut it, and a URL with no path becomes /; - answeringFor makes requestIP(rebuilt) return the original request's address.

Checks

  • The new tests and test/apps-open-host.test.ts pass (7 of 7).
  • bun x tsc --noEmit is clean.
  • bun scripts/gate.ts exits 0, which I confirmed this time.

Where I deviated or left something out

  • The attribution test is a unit test on answeringFor, not a full server test. The runtime's requestIP isn't easy to observe from outside, so I tested the mapping directly. The server.ts wiring in fetch has no test of its own.
  • Bun ignores an overridden url on a real Request, so the tests use a small stub for a request with an unparseable URL, such as http://evil[x/…. The # Host from the issue still goes through the integration test.
  • I kept the path regex, as you suggested, and covered it with the tests above.
commented

I couldn't find a failure in this change: the only test the check counted against the pull passes here. I merged the latest main into the branch and pushed, so the forge reruns the check on that.

What the check log showed. Of the five suites it asked main about, four are red on main too (release-handover-keepalive, module-state, check-install, ask-to-reach), and the forge did not count them against this pull. The fifth, test/mirror-outbound.test.ts, is green on main and was counted.

Why I think it was load, not this change. The two failures I looked at in mirror-outbound ("a remote that resolves to 127.0.0.1 is refused before git starts" and fetchRefFrom) are about git remotes and DNS. They share shard 61 with the ask-to-reach failures, and they don't touch request parsing. I did not look at the log for the other failures.

What I ran, after merging the 6 new commits from main.

  • test/mirror-outbound.test.ts, test/apps-open-host.test.ts and test/parseable-request.test.ts: 11 of 11 pass.
  • bun x tsc --noEmit: clean.
  • bun scripts/gate.ts: exits 0.
  • test/module-state.test.ts fails here because src/crew/ceiling_stagger.ts:memo is missing from its baseline. That is the failure main already has, and I did not touch it.

If mirror-outbound fails again on the new check, its log is the next thing to read.

commented

A request whose Host the runtime can't turn into a URL now gets the forge's normal answer instead of a 500. This fixes /apps/open sends a grant only to a host that reads one way > the proxy reads such a Host the same way: not ours, which fails on every pull until it lands.

Cause. The runtime builds request.url from the Host header. A Host like evil#stuff--notes.<domain> gives a URL that new URL(request.url) throws on. isRead in src/http/read_only.ts is the first to hit it, and the error reaches the last-resort handler as a 500. The failure happens on every run, not just under load, so the test's expectation (200, the forge's own page) was right and the test is unchanged. Main does not fix this on its own; the sentinel's later reports show it still red on other pulls.

Change.

  • src/http/parseable_request.ts holds two functions, both used once in the public listener's fetch in src/http/server.ts:

- parseableRequest: a request whose URL can't be parsed is rebuilt on a neutral origin (http://forge.invalid). It keeps the method, path, query, headers (Host included, so dispatch still treats it as an unknown host), signal and body. A request that parses is returned as the same object. - answeringFor: wraps the server for a rebuilt request. Every server call that names it (requestIP, upgrade, timeout, the proxy) gets Bun's original request instead, so the peer address that rate limits and trust use is not lost.

  • test/parseable-request.test.ts covers four cases:

- a parseable URL comes back unchanged; - a bad Host keeps its method, path, query, body and Host header; - a path containing ? and # is cut where a URL parser would cut it, and a missing path becomes /; - the wrapped server answers requestIP with the original request's address.

Left out.

  • The peer-address check tests answeringFor directly with a stand-in server, not through a running listener.
  • The tests stand in a fake request object for a Bun request with a malformed URL, because Bun ignores an overridden url.
  • The path is extracted with a small regex rather than a URL parser, because no URL parser accepts these URLs.

Checks. On the branch merged with current main:

  • test/apps-open-host.test.ts, test/parseable-request.test.ts and test/mirror-outbound.test.ts pass (11 of 11).
  • bun x tsc --noEmit is clean and bun scripts/gate.ts exits 0.
  • The earlier check run counted only test/mirror-outbound.test.ts against this pull. It passes here, so I read that failure as load on its shard. Its other failing suites were red on main too.

Try it. curl -H 'host: evil#x--notes.<domain>' <forge>/ answers 200 with the forge's page, not 500.

🤖 Generated with Claude Code

commented

A Host header the runtime can't turn into a URL (for example evil#x--notes.<domain>) now gets the same answer as any unknown host, not a 500. This push also fixes an intermittent failure in test/crew-proposal.test.ts ("one tap lands a commit…") that blocked this pull. It is the only failure the check counted against this pull; the other three red files fail the same way on main.

Host header. src/http/parseable_request.ts rebuilds a request whose URL can't be parsed on a neutral origin. It keeps the path, method, headers (Host too, so dispatch still works) and body. answeringFor makes sure the server's own calls (requestIP, upgrade, timeout) still get the original request. src/http/server.ts puts both in front of handle. They are tested in test/parseable-request.test.ts.

Flaky test fix (src/crew/runner.ts). A resolve run records the pull's head when it starts and again when it finishes; if the head moved, that's the crew's commit. For a pull from a fork, another refresh of the head can write the snapshot ref before it updates the pull's row. The run's own refresh then sees nothing moved and reads a row still on the old head. The push looks like no push and commitOid stays empty. Here the page showed no crew card in about one run in six.

The fix reads the head from the fork's branch at both the start and the end of the run (forkTipOf). Any other pull still reads the row as before. Measuring both ends the same way keeps "a resolve run that ends ok without pushing records no commit" passing. That test changes the row by hand, so a one-sided fix broke it.

Not done: there's no new test that forces the race on purpose; the existing flaky test is what covers it. Before the fix it failed 1 run in 6. After it, it passed 12 runs of the single test and 10 runs of the whole file.

Checks run: bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 2227 files, too many to run, so I picked the 7 that cover this change: crew-proposal, proposal-card, proposal-run, fork-pull-approval, parseable-request, agent-runner-seam, efficiency-runner. All 52 tests pass.

To try the Host change: curl -H 'host: evil#x--notes.<domain>' <forge>/ answers like any unknown host, not 500.

🤖 Generated with Claude Code

commented

When the Host header is something the runtime cannot turn into a URL (for example evil#x--notes.<domain>), the server now answers like any unknown host instead of returning a 500. src/http/parseable_request.ts builds a new request on a neutral origin, keeping the path, method, headers (Host included) and body. Calls that pass that request back to the runtime (requestIP, upgrade, timeout) get the original request, so websockets and client-IP lookup still work. src/crew/runner.ts also reads a fork's pull head from the fork's own branch at both ends of a resolve run, so a pull row a concurrent refresh hasn't written yet doesn't look like "no push".

This push fixes the earlier red check. test/upstream_follow.test.ts and test/contribute-no-forge.test.ts had failed only because of what an earlier file in the same shard left behind. After its run, test/ask-to-reach.test.ts set FORGE_PARENT_LOCAL back to "" instead of removing it. On the check machine HTTPS_PROXY is a plain loopback proxy, so with that variable present cellProxy treated every later file in the shard as running in a cell. Outbound calls then went to the gate, which refuses 127.0.0.1 ("the remote's host is a private address"). That test now removes the variable when it was unset, the same way test/boundary-outbound.test.ts already does. Adding a test file in this pull changed which files share a shard, which is why the problem showed up here and not on main.

Verified:

  • Running ask-to-reach and upstream_follow in one process with HTTPS_PROXY=http://127.0.0.1:3128 gave 14 upstream_follow failures before the fix and none after.
  • The contribute-no-forge failure (a 429 where 200 was expected) never failed here, fix or no fix. My guess is the same leak, but that isn't shown; the check will tell.
  • The branch has main merged in. bun scripts/gate.ts and bun x tsc --noEmit pass.
  • test/upstream_follow.test.ts, test/contribute-no-forge.test.ts and test/parseable-request.test.ts pass.
  • Six tests in test/ask-to-reach.test.ts still fail. The check had already found them failing on main too, and in this sandbox their connections to maria--forge.test and otto--forge.test are blocked anyway.

To try the Host fix: curl -H 'host: evil#x--notes.<domain>' against the forge's / should get a 200, not a 500.

🤖 Generated with Claude Code

commented

This merges main into the branch. The conflict in src/http/server.ts came from both sides fixing the same bug: a request whose Host the runtime can't turn into a URL (for example evil#stuff--notes…).

  • This branch handles it with parseableRequest and answeringFor in src/http/parseable_request.ts. The rebuilt request keeps its Host header, method, body and abort signal. Calls to the server that name the rebuilt request (requestIP, upgrade, timeout) are passed the original request, the only one the runtime knows.
  • main (#12861) added an inline readable() in server.ts. It rewrites the Host to forge.invalid and doesn't pass server calls through to the original request.

I kept this branch's version in the fetch handler and removed main's readable(), which nothing calls any more. Main's new test, test/apps-open-host.test.ts, passes with this version: such a Host gets a 404 with no redirect, and the proxy answers 200 as the forge itself.

Checks run: bun scripts/gate.ts and bun x tsc --noEmit pass, and bun test on test/apps-open-host.test.ts, test/parseable-request.test.ts and test/ask-to-reach.test.ts gives 25 passed, 0 failed. During the tests the sandbox blocked outbound connections to maria--forge.test and otto--forge.test; no test failed.

🤖 Generated with Claude Code

the crew's run was interrupted: restart, resumes on the next pass — cancelled
commented

A request whose Host header the runtime can't turn into a URL now gets the forge's normal answer instead of a 500. This fixes /apps/open sends a grant only to a host that reads one way > the proxy reads such a Host the same way: not ours.

Cause. The runtime builds request.url from the Host header. A Host like evil#stuff--notes.<domain> gives a URL that new URL(request.url) throws on. isRead in src/http/read_only.ts is the first place to hit it, so the request fell through to the last-resort handler as a 500. It failed every time on bun 1.4.4, so it was never about the check machine's load, disk or clock. The test expected a 200 with the forge's own page, which is correct, so the test is unchanged.

Fix. parseableRequest in src/http/parseable_request.ts runs once where serve receives each request.

  • When request.url can't be parsed, it rebuilds the request on a neutral origin (http://forge.invalid). The path, method, headers, signal and body stay the same.
  • The Host header is kept, so dispatch sees the original value and treats it as an unknown host.
  • answeringFor hands the runtime the original request whenever server methods are called with the rebuilt one (requestIP, upgrade, timeout).
  • Requests with a parseable URL are untouched.

This also protects every other new URL(request.url) call, not just isRead. It replaces main's similar readable() helper, which differed in that it overwrote the Host header.

Test fix in test/check-land-settle.test.ts. The test was wrong, not the code. It merged pull two right after one and demanded a check that was green on the newest main. If two's check went green before one landed, the merge was refused check_stale within milliseconds; that is the 282 ms failure in the last check. The merge queue lands each step without that fresh-green demand (#3316), so the test's second landing now passes { pastGate: true } the same way. What the test asserts is unchanged: the first step's main check is cancelled, never started, and marked superseded by the second.

Checked.

  • test/check-land-settle.test.ts passed 3 runs in a row.
  • test/check-land-settle.test.ts, test/apps-open-host.test.ts, test/parseable-request.test.ts and test/ask-to-reach.test.ts together: 26 pass, 0 fail.
  • bun scripts/gate.ts and bun x tsc --noEmit: both exit 0.
  • scripts/fast.ts named 2,238 test files, because it compares the whole branch with main, so I didn't run that list and left the full suite to the forge's check.
  • During the test run the sandbox refused connections to maria--forge.test and otto--forge.test; those are names the tests themselves use and nothing failed.

Try it. curl -H 'host: evil#x--notes.<domain>' <forge>/ should return 200 with the forge's own page.

Closes #12702

🤖 Generated with Claude Code

Approved

The earlier peer-attribution ask is met: answeringFor maps the rebuilt request back to the original for requestIP, upgrade and timeout, so rate limits still key on the socket address. It has a unit test. The Host header is kept and the body goes through unchanged, so nothing new is trusted from the client. The test fixes in ask-to-reach (unsetting FORGE_PARENT_LOCAL) and check-land-settle (pastGate) are justified in the pull. I could not diff against 28bd762b442d because that commit is not in the clone, so I read the whole change instead. Non-blocking: the runner.ts forkTipOf race fix has no test that forces the race. Non-blocking: the server.ts wiring in fetch is covered only through apps-open-host.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs