Turning the check lane on in a cluster
By default a check runs in a cell on the forge's own machine. With
FORGE_CHECKS_PLACEMENT=job it runs as a Kubernetes Job instead
(src/cluster/jobs.ts). This page is exactly what an operator applies; the
chart is the owner's and is not described here. A test
(test/cluster-doc.test.ts) fails if this page drifts from the code.
Role
The forge talks to the API server with its pod's service-account token, in that pod's own namespace. It needs one Role in that namespace, bound to that service account:
| apiGroup | resource | verbs | why |
|---|---|---|---|
batch | jobs | create | POST /apis/batch/v1/namespaces/<ns>/jobs starts a check |
batch | jobs | get | polled until succeeded or failed; a 404 means it vanished |
batch | jobs | delete | the Job is removed when the run ends (propagationPolicy=Background) |
"" | pods | list | finds the Job's pod by job-name=<job>, for its exit code and node |
"" | pods/log | get | reads the pod's log into the run's log |
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: forge-checks }
rules:
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["create", "get", "delete"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["list"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
Nothing else is used: no watch, no update, no cluster-wide verbs.
Environment
Read once at boot by src/checks/placement.ts. A malformed value leaves
process in place; nothing is guessed.
| variable | example | meaning |
|---|---|---|
FORGE_CHECKS_PLACEMENT | job | process (default) or job |
FORGE_CHECKS_IMAGE | registry.example/forge-check:1 | the Job's image; required, no whitespace |
FORGE_CHECKS_CLAIM | forge-checks | a ReadWriteMany claim holding the checks root; required |
FORGE_CHECKS_NODE_SELECTOR | pool=checks,disk=ssd | key=value,key=value; optional |
FORGE_CHECKS_TOLERATIONS | [{"key":"dedicated","operator":"Equal","value":"checks","effect":"NoSchedule"}] | JSON list, at most 20; optional |
FORGE_CHECKS_EGRESS_GATE | 10.42.0.17 (the forge pod's IP) | the address the forge's per-run egress gate listens on and a Job's pod dials; without it no check runs as a Job |
Egress through the forge's gate
A Job holds the same egress wall as a cell (#13251). For each run the forge
opens the proxy a cell's only way out is (apps/egress.ts, through
src/checks/job_gate.ts) on FORGE_CHECKS_EGRESS_GATE, at the first free
port of 47100-47199 (JOB_GATE_PORTS: one per Job running at once), and
gives the pod HTTP_PROXY/HTTPS_PROXY pointing at it (NO_PROXY empty).
The proxy URL carries a secret made for that run; a request to the gate
without it as Proxy-Authorization is answered 407 and dials nothing, so
another pod that reaches the port cannot use the forge as its way out. It forwards to the forge, the registry and what forge.json
egress declares, and refuses anything else with 403 and
refused by this forge: <host> is not in this app's forge.json egress; an
outside contributor's check is enforced whatever cells.egress says. The
run records the gate with a note (JOB_GATE_NOTE) that the rest rests on
the cluster's NetworkPolicy, which the forge cannot see, and its page says
so. The gate closes when the Job ends.
The variables only point the way: the wall is a NetworkPolicy that lets a
check pod reach the forge pod on the gate's ports and nothing else. Name the
range, never the whole pod: on any port a check pod could dial the forge's
own listeners around the gate. Keep nothing else of the forge pod listening
in 47100-47199. Every check pod carries the label forge-check: "true"
(JOB_LABELS in src/checks/job_step.ts):
podSelector:
matchLabels: { forge-check: "true" }
policyTypes: [Egress]
egress:
- to: [{ podSelector: { matchLabels: { app: forge } } }]
ports: [{ protocol: TCP, port: 47100, endPort: 47199 }]
Set FORGE_CHECKS_EGRESS_GATE only once that policy is in place. Without
it the cluster cannot honour egress, so no check runs as a Job: each stays
in a cell, records placement: process, and its log says
placement: job skipped, this cluster names no egress gate for a Job (FORGE_CHECKS_EGRESS_GATE).
Confirming the lane is on
- Push to a pull, or re-run its check.
- The run's page says "ran as a cluster job on <node>"; a
processrun says
nothing.
- In the cluster,
kubectl get jobs,pods -l forge-check=trueshows the Job
while it runs (it is deleted when the run ends).
- If every run still says nothing, a variable above is missing or malformed:
the forge fell back to process.