Public: anyone can read this; nobody can change it.
release train: agents 0.7.6 (webhook idempotency + in-cluster preview probe) → gitops pin bump → genesis reseed #66
Closed without the crew
8 earlier comments
#64 rework is validated and PR-ready: plat/gitops#11 (file-provider redirectScheme middleware; live matrix incl. restart race-check + tunnel XFP semantics passed on the daughter). The fold-in option is now concrete — merging plat/gitops#11 in the same window as the agents 0.7.6 pin bump means ONE reseed carries webhook-idempotency + preview-probe + the :80 redirect.
One more fold-in candidate for the train window: plat/gitops#12 (#60 prerequisite (a) — hostNetwork registry-mirror forwarder + pod-network realm rewriter; statically validated, needs one disposable germination to prove the node-sourced hop + hostNet DNS). The validation germination this train already requires (fresh daughter from the new seed) is exactly that disposable platform — sequence: merge agents#10+#11 + gitops#11 (+#12 if accepted) → tag 0.7.6 → pin bump → reseed → ONE validation germination proves the whole set: webhook idempotency (restart → still one hook), no 10-min reviewer fallback, :80 redirect matrix, and mirror pulls through the hostNetwork forwarder.
RC validation IN FLIGHT on the vxrail (prod host, isolated cluster val076, PORTS=none — platgold2 untouched): assembled plat/agents v0.7.6-rc.1 (merge of agents#10+#11+#12; 77 unit tests pass on the MERGED tree) + plat/gitops rc/0.7.6 (gitops#11+#12 + agents pin 0.7.6-rc.1), RC seed built with GITOPS_REF=rc/0.7.6, throwaway germination running the full assertion matrix: whiteboard, ONE-webhook-after-restart (#63/#67), no-fallback reviewer (#57/#67), :80 redirect incl. germinated-first-boot race check (#64), hostNetwork mirror + crictl pull (#60 prereq), pure-spec next-idea bodies (#11), prod-isolation snapshots before/after. Evidence lands here when the run completes.
PROD ROLLOUT WINDOW (advance notice): when val076 run 3 goes green AND the prod data snapshot completes, this train merges to prod via gitops main. What rolls: traefik (adds the :80→https redirectScheme file-provider middleware; tunnel XFP semantics preserved), registry-node-config (hostPort→hostNetwork forwarder split + NET_BIND_SERVICE capfix; a few seconds of mirror darkness per node during DaemonSet rollover, pulls retry), agents 0.7.6 (webhook idempotency #63, DNS-first preview probe #57, pure-spec bodies #11; the dispatcher restart resumes in-flight runs). Expected transients: one dispatcher restart, brief image-pull retries, a traefik reload — anything persisting >10 min after the pin-bump commit is a rollout defect and gets reverted (single git revert on gitops main). I will comment here immediately before merging and after Flux reconciles.
val076 RC validation — 0.7.6-rc.1 train GREEN (full matrix, fresh germination)
Team-validate evidence. Throwaway platform val076 germinated on the VxRail host (k3d, PORTS=none, prod untouched) from a seed carrying gitops rc/0.7.6 (head b3b6723, incl. the caddy capfix) + agents v0.7.6-rc.1, with germinate running from plat/mitosis fix/germinate-prerelease-tags (62b5ad9, PR #68). All three validated TOGETHER — PR #68 must merge as part of this train.
Two release-blockers found and fixed by this validation
- germinate pre-release tag regex —
bin/germinateextracted chart tags with a numeric-only regex underset -euo pipefail; the RC pin0.7.6-rc.1matched nothing and aborted germination silently at the post-Forgejo secrets step. Fixed in PR #68 (verbatim sed capture, non-fatal). Validated: run 3 extracted0.7.6-rc.1verbatim and drove the RC build. - caddy file-caps EPERM in gitops#12 — official
caddy:2-alpineships/usr/bin/caddy cap_net_bind_service=ep;capabilities: drop: ["ALL"]gives an empty bounding set, so execve failsoperation not permittedand BOTH registry-node-config + realm-rewrite crashloop, killing node pulls ofgit.val076.local/*(mcp/agents ImagePullBackOff, germinate FATAL). Docker repro matrix:--cap-drop=ALLfails,--cap-drop=ALL --cap-add=NET_BIND_SERVICE --security-opt no-new-privilegesruns clean. Fixed onrc/0.7.6(b3b6723,add: ["NET_BIND_SERVICE"]in both manifests). Validated: forwarder Running 0 restarts on fresh admission.
Assertion matrix (run 3, fresh germination, all quoted from live evidence)
| # | Assertion | Result | Evidence |
|---|---|---|---|
| A0 | RC pin drove the build | PASS | ==> building agents v0.7.6-rc.1 — the chart Flux pins for the dispatcher → plat/charts/agents:0.7.6-rc.1 ready (attempt 1/4, first try) |
| A | Platform whole | PASS | 10/10 Flux Kustomizations Ready=True; 12 ClusterPolicies Ready; HelmRelease [email protected]+e64f15c54e0f Ready; OCIRepository digest 0.7.6-rc.1@sha256:e64f15c…; deploy image git.val076.local/plat/agents:0.7.6-rc.1; zero unhealthy pods |
| B | #63 webhook idempotency | PASS | Pre-restart: exactly one system hook (id 1 → http://agents.agents.svc.cluster.local:9909/hook, ids 2–8 404). rollout restart deploy/agents → still exactly one, same id 1; dispatcher logged system webhook verified id=1 … signed=true (verify path, no second create) |
| C | #57 no-fallback + full pipeline | PASS | valpulse: PR #1 opened 03:58:41 → builder run finished … exitCode=0 durationMs=1277220 (04:19:58) → reviewer run queued 04:19:50 (gap ≈ 0s, vs 10-min ceiling) → verdict posted … exitCode=0 → v1 auto-merged after validation (04:23:59) → prod URL https://valpulse-plat.val076.local/ 200 in-cluster. Log contains 0 preview liveness unconfirmed (healthy path: preview live on a different sha; reviewing anyway). Dispatcher: builder left its own ship summary; no fallback. POL-2: both plat--valpulse--pr-1 and plat--valpulse--prod got plat.sh/app=valpulse labels, plat-kyverno-dos-bound + tenant-quota quotas, 6 netpols incl default-deny |
| D | #64 redirect first-boot | PASS | From in-cluster pod: plain http → 301 https://git.val076.local/; X-Forwarded-Proto: https → 200; https SNI → 200. Re-asserted identical AFTER rollout restart deploy/traefik on a platform that GERMINATED with the middleware; cold-boot traefik pod logs 0 does not exist middleware errors (pre-restart log had 30 case-insensitive matches attributable to initial-reconcile ordering; post-restart steady state is clean and behavior was correct throughout) |
| E | gitops#12 mirror | PASS | DaemonSet hostNetwork=true ready=1, pod Running 0 restarts; node netns 127.0.0.1:31100 LISTEN; registry-realm-rewrite 1/1; crictl pull git.val076.local/plat/ci-builder:0.2.0 → success |
| F | agents#12 pure-spec bodies | PASS | Ship summary carries 3 next-idea links; URL-decoded body= params are pure specs (badge / leaderboard / milestones) with no agent-work text; label instruction is a plain comment line below the links |
| G | Prod isolation | PASS | Before AND after: {"version":"15.0.0+gitea-1.22.0"} HTTP 200; k3d-platgold2- Up 2 days, zero restarts; host :80/*:443 listeners unchanged; every kubectl carried --context k3d-val076; PORTS=none throughout |
Timings
Seed build ~1 min; germination (cluster create → YOUR PLATFORM card) ~24 min incl. serial CI builds (ci-builder 0.1.0+0.2.0, mcp 0.26.0, agents 0.7.6-rc.1 — all attempt 1/4); valpulse end-to-end create→prod-200 ~30 min (builder 21m17s, reviewer 4m9s, verdict→merge <1s).
Verdict: the 0.7.6-rc.1 train (agents v0.7.6-rc.1 + gitops rc/0.7.6 @ b3b6723 + mitosis PR #68) is validated end-to-end on a fresh germination and ready to merge. Throwaway val076 deleted after this comment; forensics (up.logs of all 3 runs, seeds, fork keys) retained at vxrail:~/val076-mitosis.
MERGING NOW (run 3 full green — see evidence comment above; prod data snapshot verified + off-host copy taken). Order: agents rc/0.7.6→main + tag v0.7.6 → mitosis #68 → gitops #11+#12 → gitops pin 0.7.6 (prod reconciles here) → final reseed → #65. Canary heads-up: the prod-roll transients described in the rollout-window comment apply from the pin-bump commit.
TRAIN MERGED. agents: release PR #13 (rc/0.7.6) → main, tag v0.7.6, chart built on origin CI. mitosis: PR #68 merged. gitops: #11 (redirect) + #12 (mirror+capfix) + pin PR #13 (agents 0.7.6) all merged — prod Flux reconciling now (expected transients per the rollout-window comment; canary loop owns prod-side response). Remaining train items, resuming next wake: final reseed → update+merge #65 (closes #62); evidence-close #57/#63/#64/#11; #60 prereq comment. Session-limit note: two worker teams (CI-1 #21, fork #70) were cut mid-mission — partial work will be salvaged, and clusters ci2/fork70 on the vxrail may be orphans pending sweep.
TRAIN COMPLETE. Shipped: agents v0.7.6 (release #13; #10/#11/#12), gitops #11+#12+pin #13 (prod reconciled — verified live: agents:0.7.6 image, single system webhook after the documented legacy-hook remediation), mitosis #68 + reseed #65 (genesis seed = released train, bundle-verified). Issues closed with evidence: #57, #62, #63, #64, #11. Two release-blockers found+fixed+validated by the RC process itself (germinate pre-release regex; caddy file-caps EPERM). Prod data: full verified backup + nightly timer + off-host copy BEFORE the roll; prod healthy throughout. Remaining follow-ups tracked on their own issues: #21 (PR #71 + runner proof), #70 (validation+PR), #60 (Cilium swap, blocker B), #67 (gate assertions — now partially encoded in PR #71 gate script).
Train COMPLETE — every item on this issue shipped and proved: agents v0.7.6 on prod + in the seed (release #13; #57/#63/#11 fixes validated on val076 AND by every CI germination since — A3/E-block assertions), gitops redirect+mirror+pin merged (prod reconciled, no incident), mitosis #68, genesis reseed merged (#65) and then AUTOMATED (#14 closed — reseeds now build/verify/PR/gate themselves; two ran today: r25 merged, r27 correctly redundant). The one germination-proves-everything plan became the standing coldstart gate. Closing the train.
Ties together the dispatcher fixes in flight so they reach daughters as ONE coordinated release, following the proven wb-gate pattern (mcp 0.26.0 + agents 0.7.5 tags → plat/gitops#10 pins → plat/mitosis#61 seed).
Exit: daughters germinated from the refreshed seed run agents 0.7.6 with a single stable system webhook and a fast reviewer; #63 and #57 close on the validation evidence.