Turning the check lane on in a cluster

By default a check runs in a cell on the forge's own machine. With FORGE_CHECKS_PLACEMENT=job it runs as a Kubernetes Job instead (src/cluster/jobs.ts). This page is exactly what an operator applies; the chart is the owner's and is not described here. A test (test/cluster-doc.test.ts) fails if this page drifts from the code.

Role

The forge talks to the API server with its pod's service-account token, in that pod's own namespace. It needs one Role in that namespace, bound to that service account:

apiGroupresourceverbswhy
batchjobscreatePOST /apis/batch/v1/namespaces/<ns>/jobs starts a check
batchjobsgetpolled until succeeded or failed; a 404 means it vanished
batchjobsdeletethe Job is removed when the run ends (propagationPolicy=Background)
""podslistfinds the Job's pod by job-name=<job>, for its exit code and node
""pods/loggetreads the pod's log into the run's log
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: forge-checks }
rules:
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["create", "get", "delete"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["list"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get"]

Nothing else is used: no watch, no update, no cluster-wide verbs.

Environment

Read once at boot by src/checks/placement.ts. A malformed value leaves process in place; nothing is guessed.

variableexamplemeaning
FORGE_CHECKS_PLACEMENTjobprocess (default) or job
FORGE_CHECKS_IMAGEregistry.example/forge-check:1the Job's image; required, no whitespace
FORGE_CHECKS_CLAIMforge-checksa ReadWriteMany claim holding the checks root; required
FORGE_CHECKS_NODE_SELECTORpool=checks,disk=ssdkey=value,key=value; optional
FORGE_CHECKS_TOLERATIONS[{"key":"dedicated","operator":"Equal","value":"checks","effect":"NoSchedule"}]JSON list, at most 20; optional
FORGE_CHECKS_EGRESS_GATE10.42.0.17 (the forge pod's IP)the address the forge's per-run egress gate listens on and a Job's pod dials; without it no check runs as a Job

Egress through the forge's gate

A Job holds the same egress wall as a cell (#13251). For each run the forge opens the proxy a cell's only way out is (apps/egress.ts, through src/checks/job_gate.ts) on FORGE_CHECKS_EGRESS_GATE, at the first free port of 47100-47199 (JOB_GATE_PORTS: one per Job running at once), and gives the pod HTTP_PROXY/HTTPS_PROXY pointing at it (NO_PROXY empty). The proxy URL carries a secret made for that run; a request to the gate without it as Proxy-Authorization is answered 407 and dials nothing, so another pod that reaches the port cannot use the forge as its way out. It forwards to the forge, the registry and what forge.json egress declares, and refuses anything else with 403 and refused by this forge: <host> is not in this app's forge.json egress; an outside contributor's check is enforced whatever cells.egress says. The run records the gate with a note (JOB_GATE_NOTE) that the rest rests on the cluster's NetworkPolicy, which the forge cannot see, and its page says so. The gate closes when the Job ends.

The variables only point the way: the wall is a NetworkPolicy that lets a check pod reach the forge pod on the gate's ports and nothing else. Name the range, never the whole pod: on any port a check pod could dial the forge's own listeners around the gate. Keep nothing else of the forge pod listening in 47100-47199. Every check pod carries the label forge-check: "true" (JOB_LABELS in src/checks/job_step.ts):

podSelector:
  matchLabels: { forge-check: "true" }
policyTypes: [Egress]
egress:
  - to: [{ podSelector: { matchLabels: { app: forge } } }]
    ports: [{ protocol: TCP, port: 47100, endPort: 47199 }]

Set FORGE_CHECKS_EGRESS_GATE only once that policy is in place. Without it the cluster cannot honour egress, so no check runs as a Job: each stays in a cell, records placement: process, and its log says placement: job skipped, this cluster names no egress gate for a Job (FORGE_CHECKS_EGRESS_GATE).

Confirming the lane is on

  1. Push to a pull, or re-run its check.
  2. The run's page says "ran as a cluster job on <node>"; a process run says

nothing.

  1. In the cluster, kubectl get jobs,pods -l forge-check=true shows the Job

while it runs (it is deleted when the run ends).

  1. If every run still says nothing, a variable above is missing or malformed:

the forge fell back to process.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs