Public: anyone can read this; nobody can change it.
ci: parallel per-ref gates + nightly app-E2E variant (#67 second half, #21) #57
Closed without the crew
Root-caused live on a fresh daughter (plat.local, agents 0.7.5): the ~10-min stall is NOT sha-locking — probePreview curls --resolve <host>:443:127.0.0.1, and from the agents POD 127.0.0.1 is the pod itself, so the liveness probe gets connection-refused on every poll and every review burns the full 10-min fallback. Plain in-cluster DNS (the CoreDNS rewrite germinate writes) returns 200 with /version == head sha. Fix: plat/agents#11 — DNS-first vantage with loopback fallback (dev mode preserved). Validated from both vantages on the live daughter.
Fixed in agents v0.7.6 (plat/agents#11 → release #13, on prod + in the genesis seed). Root cause was the loopback-only probe, not sha-locking. Validated live twice: fresh daughter reviewer engaged in seconds after a 21-min build, zero liveness-unconfirmed lines (val076 run 3, plat/mitosis#66).
What
Second half of #67 + the CI-seam follow-ups from #21: the cold-start gate now runs in parallel across refs and grows a nightly app-E2E variant that ships a real canary app unattended — default-SKIP until an operator seals a credential. Originally based on
fix/gate-a4-detection(#74); rebased onto main after #74 merged.1. Parallel gates
~/act_runner/config.yml, user unit restarted, re-registered cleanly, stayed registered and serving).coldstart-${{ github.ref }},cancel-in-progress: true): different PRs gate in parallel; a push to the same ref supersedes its in-flight run. Total load is capped by runner capacity, not the group.^k3d-ci[0-9]+-containers older than 150 min (a live run fits the 110-min job cap). Match is anchored — prod (k3d-platgold2) and live dev clusters are never touched.FJ_PORTinbin/germinate(one surgical knob, default:3000unchanged): the first parallel attempt killed run 11 withadmin auth failing after handoff— germinate's forgejo port-forward binds a FIXED127.0.0.1:3000, so the second germination's curls silently reached the FIRST cluster's forgejo (different admin password). The workflow now exports a per-run port (30000 + run_number % 10000). CAVEAT: PR branches forked before this fix still bind:3000— at most one such gate at a time until rebased.2. Nightly app-E2E (#67 second half)
On the nightly cron or an
e2e=truedispatch, after the standard gate,scripts/coldstart-assert.sh --e2eruns against the same CI daughter:plat-agents-secrets(fail fast)_app-template(Forgejo API, 201)Host <app>-plat.ci<N>.localvia the traefik svc)preview liveness unconfirmed(#57 fallback)plat.sh/app+ ResourceQuota + NetworkPoliciesCredential: the germination step reads repo Actions secret
E2E_CLAUDE_TOKENand seals it asCLAUDE_CODE_OAUTH_TOKEN(values never echoed). The secret is NOT created by this PR (the staged host token is compromised-pending-rotation and was deliberately not wired in). When absent, e2e-flagged runs print one loudE2E SKIPline (host-persisted) and exit 0 — nightly stays green-by-default. Sealing recipe:docs/ops/coldstart-ci.md+ comment on #67.Phase is hard-capped (
timeout 35mstep;E2E_TIMEOUTknob, default 1800 s). Humans run the identical thing:scripts/coldstart-assert.sh --e2e <ctx> <domain>.3. Gate-probe hardening (found live during validation)
A4's redirect probe used
kubectl run -i --rm, whose attach can MISS output printed before it connects — observed live as ano-XFP=?false-FAIL under 3-way host load. A4 and the new E5 probe now sharepod_probe(): run the pod detached, wait for a terminal phase, read pod logs (durable), delete. Re-ran the full standard gate on the same platform after the fix: PASS.Evidence
Parallel proof (runs 12 + 13, dispatched simultaneously)
(First attempt, runs 10+11: run 10 green, run 11 died on the
:3000cross-talk described above — that failure is what theFJ_PORTcommit fixes; runs 12+13 are the post-fix proof.)E2E default-SKIP proof (runs 10 + 12,
e2e=true, no secret — both runs green)Manual validation of the full
--e2epath (marked manual-validation)Scratch germination
ci9999on the vxrail host (PORTS=none, isolated KUBECONFIG,FJ_PORT=31999), credential taken from the host shell for this one run and never persisted into Forgejo/CI. All eight assertions passed; the canary shipped unattended in 793 s:Files
.forgejo/workflows/coldstart.yml— per-ref concurrency, orphan sweep, per-runFJ_PORT,e2einput + phase, step-level timeouts (40m germinate / 20m gate / 35m e2e; job cap 110m)scripts/coldstart-assert.sh—--e2emode (E1–E8),pod_probe()logs-based probes (A4 + E5), same PASS/FAIL/SKIP evidence-line contractbin/germinate—FJ_PORTknob (default 3000, behavior unchanged for single-host use)docs/ops/coldstart-ci.md— parallelism + superseding,FJ_PORT, nightly app-E2E, secret recipe, log locations🤖 Generated with Claude Code