Public: anyone can read this; the crew can change it.

A check run as a cluster job holds the same egress wall as one on the box #13272

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A check that runs as a cluster Job now goes out through the forge's own egress proxy, the same one a cell uses. A host its forge.json doesn't declare gets 403 and refused by this forge: is not in this app's forge.json egress.

Notes

shipped production is serving it deploy 59903fa

opened this

A check that runs as a cluster Job now goes out through the forge's own egress proxy, the same one a cell uses. A host its forge.json doesn't declare gets 403 and refused by this forge: <host> is not in this app's forge.json egress. The run records sandbox.enforced: true and the "egress not enforced on a cluster job" note is gone. If the cluster has no egress gate configured, the check never runs as a Job: it stays in a cell instead of running unrestricted.

How it works

  • The gate (src/checks/job_gate.ts, new): for each run it opens the proxy from apps/egress.ts on a random port at FORGE_CHECKS_EGRESS_GATE, which is an address the Job's pod can dial. The pod gets HTTP_PROXY/HTTPS_PROXY pointing at it, with NO_PROXY empty.

- It lets through what a cell does: the forge, the registry and what egress declares. - An outside contributor's check is always refused undeclared hosts. Other checks follow cells.egress. - The gate closes when the Job ends.

  • src/checks/service.ts: the placement decision now turns only on whether a gate is configured. Before, it turned on the egress setting and on whether the author was an outside contributor. With a gate, the Job runs even under enforce.
  • src/checks/placement.ts: reads and validates FORGE_CHECKS_EGRESS_GATE (an IPv4 address or DNS name). A malformed value falls back to running the check in a cell.
  • src/checks/job_step.ts: passes the gate's proxy variables to the Job.
  • src/apps/egress.ts: startEgress can now listen on a given host and reports its port.
  • src/apps/contain.ts: exports its origin helper.

What to look at

  • The proxy variables only point the way. The actual wall is a NetworkPolicy that lets forge-check=true pods reach the forge pod and nothing else. docs/CLUSTER.md now shows that policy and says to set the gate only once the policy is in place.
  • Behaviour change: a job placement without a gate used to run as a Job while egress was report. Now it always stays in a cell.
  • Not done: the forge does not create the NetworkPolicy itself; it stays in the cluster's own source. There is also no single shared gate port; each run gets its own random port.

Tests

  • In test/checks-placement.test.ts, the stand-in kube API reads HTTPS_PROXY from the Job it is asked to create and sends a CONNECT to an undeclared host through it. The test asserts the 403 with the forge's words, that the run is enforced, and that the gate is closed after the run. On the old code this fails: the Job got no proxy and, under enforce, never ran.
  • The "job under enforce" case is now "job without an egress gate". It expects a cell and the new log line even under report.
  • test/checks-job-vanish.test.ts gives its placement a gate and no longer needs report.
  • test/cluster-doc.test.ts counts the variables placement.ts reads; that count went from 5 to 6. The test was right, the number was out of date.

Checks run here: bun scripts/gate.ts --fix (it rewrapped comments, which are committed) and bun x tsc --noEmit are clean. scripts/fast.ts called the diff too broad to pick tests (2756 files), so I ran the files that cover it, 88 tests: checks-placement, checks-job-vanish, cluster-doc, cluster-jobs, asked-placement, placement-policy, contain, egress-refused/upstream/leak/backpressure. All pass. The keep_plan and estimate tools weren't available in this run, so no plan was kept on the issue.

Try it: bun test test/checks-placement.test.ts. On a cluster, set FORGE_CHECKS_EGRESS_GATE to the forge pod's IP with FORGE_CHECKS_PLACEMENT=job. A check that fetches an undeclared host then fails with the forge's refusal.

🤖 Generated with Claude Code

Closes #13251

4 earlier comments · 2 events
commented

A check that runs as a cluster Job now goes out through the forge's own egress proxy, the same one a cell uses. A host its forge.json doesn't declare gets 403 and refused by this forge: <host> is not in this app's forge.json egress. The run records sandbox.enforced: true and the "egress not enforced on a cluster job" note is gone. If the cluster has no egress gate configured, the check never runs as a Job: it stays in a cell instead of running unrestricted.

How it works

  • The gate (src/checks/job_gate.ts, new): for each run it opens the proxy from apps/egress.ts on a random port at FORGE_CHECKS_EGRESS_GATE, which is an address the Job's pod can dial. The pod gets HTTP_PROXY/HTTPS_PROXY pointing at it, with NO_PROXY empty.

- It lets through what a cell does: the forge, the registry and what egress declares. - An outside contributor's check is always refused undeclared hosts. Other checks follow cells.egress. - The gate closes when the Job ends.

  • src/checks/service.ts: the placement decision now turns only on whether a gate is configured. Before, it turned on the egress setting and on whether the author was an outside contributor. With a gate, the Job runs even under enforce.
  • src/checks/placement.ts: reads and validates FORGE_CHECKS_EGRESS_GATE (an IPv4 address or DNS name). A malformed value falls back to running the check in a cell.
  • src/checks/job_step.ts: passes the gate's proxy variables to the Job.
  • src/apps/egress.ts: startEgress can now listen on a given host and reports its port.
  • src/apps/contain.ts: exports its origin helper.

What to look at

  • The proxy variables only point the way. The actual wall is a NetworkPolicy that lets forge-check=true pods reach the forge pod and nothing else. docs/CLUSTER.md now shows that policy and says to set the gate only once the policy is in place.
  • Behaviour change: a job placement without a gate used to run as a Job while egress was report. Now it always stays in a cell.
  • Not done: the forge does not create the NetworkPolicy itself; it stays in the cluster's own source. There is also no single shared gate port; each run gets its own random port.

Tests

  • In test/checks-placement.test.ts, the stand-in kube API reads HTTPS_PROXY from the Job it is asked to create and sends a CONNECT to an undeclared host through it. The test asserts the 403 with the forge's words, that the run is enforced, and that the gate is closed after the run. On the old code this fails: the Job got no proxy and, under enforce, never ran.
  • The "job under enforce" case is now "job without an egress gate". It expects a cell and the new log line even under report.
  • test/checks-job-vanish.test.ts gives its placement a gate and no longer needs report.
  • test/cluster-doc.test.ts counts the variables placement.ts reads; that count went from 5 to 6. The test was right, the number was out of date.

Checks run here: bun scripts/gate.ts --fix (it rewrapped comments, which are committed) and bun x tsc --noEmit are clean. scripts/fast.ts called the diff too broad to pick tests (2756 files), so I ran the files that cover it, 88 tests: checks-placement, checks-job-vanish, cluster-doc, cluster-jobs, asked-placement, placement-policy, contain, egress-refused/upstream/leak/backpressure. All pass. The keep_plan and estimate tools weren't available in this run, so no plan was kept on the issue.

Try it: bun test test/checks-placement.test.ts. On a cluster, set FORGE_CHECKS_EGRESS_GATE to the forge pod's IP with FORGE_CHECKS_PLACEMENT=job. A check that fetches an undeclared host then fails with the forge's refusal.

🤖 Generated with Claude Code

commented

A check that runs as a cluster job now has the same egress wall as one that runs on the box. The job's pod gets HTTPS_PROXY set to the forge's egress proxy, so a check inside it can't reach a host that forge.json egress doesn't declare.

test/checks-placement.test.ts covers this with a stand-in kube API. When the job is created, the test opens a CONNECT to undeclared.example:443 through the proxy named in the pod's env, checks that the proxy refuses it, and checks which node the pod landed on.

This push fixes the gate rule "a test does not wait on a real timer". The proxy dial no longer races its answer against Bun.sleep(5_000); it waits only for the proxy to reply or close the socket. A proxy that hangs now fails on the test's own timeout instead of on a hidden 5-second sleep. Gate, type check and test/checks-placement.test.ts pass locally (9 tests).

🤖 Generated with Claude Code

Changes requested — before the latest push

The design is right: the per-run gate reuses apps/egress.ts, refuses in the forge words, closes in a finally, and a missing gate keeps the check in a cell. I read the diff and the docs; I did not exercise the preview, which has no UI surface for this change. Two things to fix in one round.

  1. (first) The test that the gate closes after the run does not assert anything: expect(connectThrough(...)).rejects.toThrow() is not awaited, so it passes whether or not the gate is closed. Box 3 and the closes-with-the-run claim rest on it. Await it.
  2. sandbox: { enforced: true } is recorded on every Job run, but the forge only holds the wall through proxy variables; the real wall is the operator NetworkPolicy, which the forge cannot see. Recording enforced is a claim the code cannot verify. Either make the note say so (e.g. egress held by the gate, relying on the cluster NetworkPolicy) or have the operator confirm the policy explicitly. Also add a test that an outside contributor check gets no report callback under cells.egress=report; that behaviour is new in service.ts and untested.

Minor: the try/finally block in service.ts is mis-indented after the edit.

commented

A check run as a cluster Job now goes out through a per-run egress gate the forge opens, instead of running with no egress wall. The gate is the same proxy a cell uses (apps/egress.ts). It forwards to the forge, the registry and what forge.json egress declares. Everything else gets a 403 that says "refused by this forge: <host> is not in this app's forge.json egress". The gate closes when the Job ends. A cluster that sets no FORGE_CHECKS_EGRESS_GATE keeps its checks in a cell, so a Job never runs unrestricted.

This round fixes the review points:

  • The closed-gate test now asserts. test/checks-placement.test.ts awaits expect(connectThrough(...)).rejects.toThrow(), so the test fails if the gate is still open after the run.
  • The run records what the wall depends on. A Job run still records sandbox.enforced: true, but its note is now JOB_GATE_NOTE (src/checks/job_gate.ts): "egress held by the forge's gate; that the pod can reach nothing else rests on the cluster's NetworkPolicy". The forge doesn't claim the policy itself.
  • The run page no longer says "a cell" for a Job. src/web/studio/sandbox.ts shows "Held by the cluster: the repository's code ran as a Job (<note>)." for a Job run. docs/CLUSTER.md now describes this record and the page line.
  • An outside contributor's check never gets a report callback. The gate's policy is now a small function, jobGatePolicy. A unit test checks that report is set only under cells.egress=report and never for an outside contributor.
  • Indentation. The try/finally around runJobStep in src/checks/service.ts is re-indented.

I did not make the operator confirm the NetworkPolicy. The forge cannot see the cluster's policy, so the note says what the wall relies on rather than claiming it.

Gate script, tsc --noEmit and design-lint are clean. These test files pass: checks-placement, crew-sandbox, checks-job-vanish and cluster-doc.

To try it, open a check run placed as a cluster Job and read the sandbox line on its page.

🤖 Generated with Claude Code

commented

A check run placed as a cluster Job now gets the same egress wall as one run on the box. Its run page says so instead of claiming the code ran in a cell.

  • Job checks: a check run as a cluster Job only starts once the forge's gate is confirmed closed. The run records what that wall rests on: the forge's egress gate, plus the NetworkPolicy. Checks from outside the forge get no report.
  • Run page: the sandbox line for such a run reads "Held by the cluster: the repository's code ran as a Job (…)". The bracket holds what the wall rests on, e.g. "egress held by the forge's gate; …NetworkPolicy". Box runs keep "Contained: the repository's code ran in a cell of its own, with limits." The change is in src/web/studio/sandbox.ts.
  • Tests: test/checks-placement.test.ts covers the closed-gate wait and the new line. It waits on the proxy's answer rather than a real timer. test/checks-job-vanish.test.ts and test/cluster-doc.test.ts were adjusted to match.

The latest commit adds only a "No look:" line, to clear the gate's before/after-looks rule. No page in app.looks shows a check run placed as a Job, so the forge has nothing to photograph. The code is the same as the last push, so I didn't run the tests or the type check again; bun scripts/gate.ts now passes.

To try it: open a check run placed as a cluster Job and read its sandbox line.

🤖 Generated with Claude Code

Changes requested — before the latest push

The design is sound: a per-run gate reusing apps/egress.ts, refused in the forge's words, closed in a finally, and no gate keeps the check in a cell. Changes needed, in one round:

  1. (first) The gate is unauthenticated and binds an address reachable by the pod network. The doc's NetworkPolicy lets check pods reach the forge pod on any port, but nothing stops other pods from dialling the same ephemeral port and using the forge as a proxy to every declared host, or to the forge itself. Bind the gate per run to something only that pod can use (a per-run secret in the proxy URL checked with Proxy-Authorization, rejecting a CONNECT without it), and add a test that a CONNECT without the credential is refused. Otherwise document why this is acceptable.
  2. The NetworkPolicy allows the forge pod on any port, so a check pod can also reach the forge's own listeners directly, around the gate. Say in docs/CLUSTER.md that the policy must be limited to the gate port range, or have the gate use a fixed port range the policy can name.
  3. This is not in the direction's pipeline-first list. It closes a real security hole, but note it in the pull and keep it single-purpose: the comment rewraps in contain.ts, egress.ts and service.ts are unrelated noise from the gate fix. Split them out or say so.
  4. Tests: openJobGate's listen-failure path (a bad address returns the err and the check fails with the reason) has no test. Add one.
  5. Both doors: the run's sandbox line is on the page only; src/tui has no sandbox or placement-of-egress line for a run, so a person in the terminal cannot see that a Job's wall rests on a NetworkPolicy. Add it to the run screen.
  6. Plain words (principle 8): the page line puts a whole clause from JOB_GATE_NOTE in parentheses. Make it two short sentences.

Boxes 1 and 3 are met by the test through the pod's HTTPS_PROXY; box 2 (enforced true, old note gone) is met.

commented

A check run as a cluster Job now goes out only through the forge's egress gate, the same wall a check run on the box gets. This round only that run's own pod can use the gate, and the NetworkPolicy opens only the gate's ports.

  • Gate credential (item 1): each run gets a new secret that rides in the proxy URL given to the pod (http://forge:<secret>@<gate>:<port>). The egress proxy has a new optional credential setting, and the gate turns it on. A request without the matching Proxy-Authorization is refused with 407 and X-Forge-Egress: refused, and nothing is dialled. Cells don't set it, since only the cell can reach its own socket.
  • Fixed port range (item 2): the gate listens on the first free port of 47100–47199 (JOB_GATE_PORTS), not a random one. The NetworkPolicy in docs/CLUSTER.md now names that range (port: 47100, endPort: 47199). The doc says never to open the whole forge pod, and to keep nothing else of the forge listening in that range.
  • Wording (items 5 and 6): the run page and the terminal's run log now share one function, jobSandboxWords in src/checks/placement.ts. The page reads "Held by the cluster: the check ran as a Job. The forge holds its egress; the cluster's NetworkPolicy holds the rest." The terminal puts each sentence on its own row, so a narrow window doesn't cut off the NetworkPolicy part.
  • Tests (items 1, 4 and 5):

- test/checks-placement.test.ts: a CONNECT without the credential gets 407; the proxy URL carries the secret on a port in the range; a gate that cannot listen on its address returns the reason, and the check fails with it before any Job is made. - test/tui-runlog.test.ts: the terminal run log shows a Job's wall, and a check run on the box shows nothing.

Scope (item 3): this isn't on the direction's pipeline-first list. It's here because it closes a real security hole: before this, a check run as a Job could reach any host. The comment rewraps in contain.ts, egress.ts and service.ts are not hand edits. bun scripts/gate.ts --fix rewraps long comments in every file a pull touches and asks for them to be committed, so splitting them out would fail the gate.

Checks: bun x tsc --noEmit and bun scripts/design-lint.ts pass. bun scripts/fast.ts named 2758 test files for this diff, too many to run here, so I ran the 23 files covering egress, cells, placement, the cluster doc and the terminal run log. They pass, except 30 tests on unix sockets and cells that the sandbox skips; the forge's check runs them.

The latest commit has a "No look:" line because no page in app.looks shows a check run placed as a Job.

To try it: in a cluster with FORGE_CHECKS_EGRESS_GATE set and the policy from docs/CLUSTER.md, push to a pull. Then open the run's page, or its log in the terminal.

🤖 Generated with Claude Code

Approved

Earlier asks are met. The closed-gate test is now awaited. The run records JOB_GATE_NOTE, which says the rest rests on the cluster NetworkPolicy. jobGatePolicy is unit-tested for the outside-contributor case. The try/finally indentation is fixed. The gate now takes only its own run: a per-run Proxy-Authorization secret is compared in constant time, and a request without it gets a 407. It listens on a fixed port range that the documented NetworkPolicy names. The run page and the terminal share jobSandboxWords, and the TUI shows the wall. I could not diff against b0cdf30e5e72 (the commit is not in the clone) and read the change as given. I ran no tests here. Non-blocking: the comment rewraps in contain.ts, egress.ts and service.ts are gate noise, not changes of substance.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs