Public: anyone can read this; nobody can change it.

workbenchthis machine

ci: parallel per-ref gates + nightly app-E2E variant (#67 second half, #21) #57

Closed opened this view on origin

Closed without the crew

opened this view on origin

What

Second half of #67 + the CI-seam follow-ups from #21: the cold-start gate now runs in parallel across refs and grows a nightly app-E2E variant that ships a real canary app unattended — default-SKIP until an operator seals a credential. Originally based on fix/gate-a4-detection (#74); rebased onto main after #74 merged.

1. Parallel gates

  • Runner capacity on vxrail raised 1 → 2 (~/act_runner/config.yml, user unit restarted, re-registered cleanly, stayed registered and serving).
  • Workflow concurrency is now per-ref (coldstart-${{ github.ref }}, cancel-in-progress: true): different PRs gate in parallel; a push to the same ref supersedes its in-flight run. Total load is capped by runner capacity, not the group.
  • Orphan sweep at run start: a superseded run can be cancelled before its teardown, so each run removes ^k3d-ci[0-9]+- containers older than 150 min (a live run fits the 110-min job cap). Match is anchored — prod (k3d-platgold2) and live dev clusters are never touched.
  • FJ_PORT in bin/germinate (one surgical knob, default :3000 unchanged): the first parallel attempt killed run 11 with admin auth failing after handoff — germinate's forgejo port-forward binds a FIXED 127.0.0.1:3000, so the second germination's curls silently reached the FIRST cluster's forgejo (different admin password). The workflow now exports a per-run port (30000 + run_number % 10000). CAVEAT: PR branches forked before this fix still bind :3000 — at most one such gate at a time until rebased.

2. Nightly app-E2E (#67 second half)

On the nightly cron or an e2e=true dispatch, after the standard gate, scripts/coldstart-assert.sh --e2e runs against the same CI daughter:

  • E1 claude credential sealed in plat-agents-secrets (fail fast)
  • E2 canary repo generated from _app-template (Forgejo API, 201)
  • E3 dispatcher bootstraps it: prime PR opens
  • E4 prime PR auto-merges (builder → preview → reviewer verdict → merge)
  • E5 prod serves 200 in-cluster (Host <app>-plat.ci<N>.local via the traefik svc)
  • E6 dispatcher log contains no preview liveness unconfirmed (#57 fallback)
  • E7 builder-finish → reviewer-queued gap < 120 s
  • E8 app prod namespace carries plat.sh/app + ResourceQuota + NetworkPolicies

Credential: the germination step reads repo Actions secret E2E_CLAUDE_TOKEN and seals it as CLAUDE_CODE_OAUTH_TOKEN (values never echoed). The secret is NOT created by this PR (the staged host token is compromised-pending-rotation and was deliberately not wired in). When absent, e2e-flagged runs print one loud E2E SKIP line (host-persisted) and exit 0 — nightly stays green-by-default. Sealing recipe: docs/ops/coldstart-ci.md + comment on #67.

Phase is hard-capped (timeout 35m step; E2E_TIMEOUT knob, default 1800 s). Humans run the identical thing: scripts/coldstart-assert.sh --e2e <ctx> <domain>.

3. Gate-probe hardening (found live during validation)

A4's redirect probe used kubectl run -i --rm, whose attach can MISS output printed before it connects — observed live as a no-XFP=? false-FAIL under 3-way host load. A4 and the new E5 probe now share pod_probe(): run the pod detached, wait for a terminal phase, read pod logs (durable), delete. Re-ran the full standard gate on the same platform after the fix: PASS.

Evidence

Parallel proof (runs 12 + 13, dispatched simultaneously)

# 16:07:15 UTC, ~25s after the two dispatches — both clusters germinating side by side:
k3d-ci12-serverlb    24 seconds ago
k3d-ci12-server-0    24 seconds ago
k3d-ci13-serverlb    25 seconds ago
k3d-ci13-server-0    25 seconds ago

# both green, distinct clusters, overlapping ~/ci-gate-logs timestamps:
run 12 (feat/gate-parallel-nightly): success — "COLDSTART GATE PASS (93s) ... context=k3d-ci12" (run12-gate.log mtime 16:25:24)
run 13 (main):                       success — "COLDSTART GATE PASS (94s) ... context=k3d-ci13" (run13-gate.log mtime 16:24:37)

# zero leaks after both teardowns:
docker ps -a --format '{{.Names}}' | grep -E 'k3d-ci1[23]-'   -> (empty)

(First attempt, runs 10+11: run 10 green, run 11 died on the :3000 cross-talk described above — that failure is what the FJ_PORT commit fixes; runs 12+13 are the post-fix proof.)

E2E default-SKIP proof (runs 10 + 12, e2e=true, no secret — both runs green)

$ cat ~/ci-gate-logs/run12-e2e.log
#### E2E SKIP: repo Actions secret E2E_CLAUDE_TOKEN is not sealed — app-E2E did NOT run (the standard gate above is this run's result); seal it per docs/ops/coldstart-ci.md (Nightly app E2E) to enable ####

Manual validation of the full --e2e path (marked manual-validation)

Scratch germination ci9999 on the vxrail host (PORTS=none, isolated KUBECONFIG, FJ_PORT=31999), credential taken from the host shell for this one run and never persisted into Forgejo/CI. All eight assertions passed; the canary shipped unattended in 793 s:

== coldstart E2E gate: context=k3d-ci9999 domain=ci9999.local app=plat/canary07021611 budget=1800s ==
PASS [E1-credential] (   1s) claude credential sealed in plat-agents-secrets (ns agents; value never printed)
PASS [E2-canary-created] (   3s) POST /repos/plat/_app-template/generate -> 201 (plat/canary07021611)
PASS [E3-prime-pr] (  96s) dispatcher bootstrapped plat/canary07021611: prime PR #1 open
PASS [E4-auto-merge] ( 610s) prime PR #1 merged ('v1 auto-merged after validation' log lines: 1)
PASS [E5-prod-200] ( 793s) https://canary07021611-plat.ci9999.local/ -> 200 from an in-cluster pod
PASS [E6-no-liveness-fallback] ( 793s) zero 'preview liveness unconfirmed' lines in the dispatcher log
PASS [E7-review-latency] ( 793s) builder-finish 2026-07-02T16:19:23.933Z -> reviewer-queued 2026-07-02T16:19:57.238Z: 34s (< 120s)
PASS [E8-ns-governance] ( 793s) ns plat--canary07021611--prod: plat.sh/app=canary07021611, 2 ResourceQuota, 6 NetworkPolicies (allow-egress allow-egress-postgres allow-ingress-from-workflows allow-ingress-postgres allow-ingress-traefik default-deny)
COLDSTART E2E GATE PASS ( 793s): app plat/canary07021611 shipped builder->reviewer->prod unattended

# dispatcher log lines behind E6/E7:
2026-07-02T16:19:23.933Z info run finished runId=builder-plat-canary07021611-... role=builder exitCode=0 durationMs=364063
2026-07-02T16:19:57.238Z info run queued  runId=reviewer-plat-canary07021611-... role=reviewer item=#1        (gap: 34s)
2026-07-02T16:21:52.412Z info v1 auto-merged after validation pr=plat/canary07021611#1

Files

  • .forgejo/workflows/coldstart.yml — per-ref concurrency, orphan sweep, per-run FJ_PORT, e2e input + phase, step-level timeouts (40m germinate / 20m gate / 35m e2e; job cap 110m)
  • scripts/coldstart-assert.sh — --e2e mode (E1–E8), pod_probe() logs-based probes (A4 + E5), same PASS/FAIL/SKIP evidence-line contract
  • bin/germinate — FJ_PORT knob (default 3000, behavior unchanged for single-host use)
  • docs/ops/coldstart-ci.md — parallelism + superseding, FJ_PORT, nightly app-E2E, secret recipe, log locations

🤖 Generated with Claude Code

commented

Root-caused live on a fresh daughter (plat.local, agents 0.7.5): the ~10-min stall is NOT sha-locking — probePreview curls --resolve <host>:443:127.0.0.1, and from the agents POD 127.0.0.1 is the pod itself, so the liveness probe gets connection-refused on every poll and every review burns the full 10-min fallback. Plain in-cluster DNS (the CoreDNS rewrite germinate writes) returns 200 with /version == head sha. Fix: plat/agents#11 — DNS-first vantage with loopback fallback (dev mode preserved). Validated from both vantages on the live daughter.

commented

Fixed in agents v0.7.6 (plat/agents#11 → release #13, on prod + in the genesis seed). Root cause was the loopback-only probe, not sha-locking. Validated live twice: fresh daughter reviewer engaged in seconds after a 21-min build, zero liveness-unconfirmed lines (val076 run 3, plat/mitosis#66).

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs