Public: anyone can read this; nobody can change it.

workbenchthis machine

release train: agents 0.7.6 (webhook idempotency + in-cluster preview probe) → gitops pin bump → genesis reseed #66

Closed opened this view on origin

Closed without the crew

opened this view on origin

Ties together the dispatcher fixes in flight so they reach daughters as ONE coordinated release, following the proven wb-gate pattern (mcp 0.26.0 + agents 0.7.5 tags → plat/gitops#10 pins → plat/mitosis#61 seed).

Exit: daughters germinated from the refreshed seed run agents 0.7.6 with a single stable system webhook and a fast reviewer; #63 and #57 close on the validation evidence.

8 earlier comments
commented

#64 rework is validated and PR-ready: plat/gitops#11 (file-provider redirectScheme middleware; live matrix incl. restart race-check + tunnel XFP semantics passed on the daughter). The fold-in option is now concrete — merging plat/gitops#11 in the same window as the agents 0.7.6 pin bump means ONE reseed carries webhook-idempotency + preview-probe + the :80 redirect.

commented

One more fold-in candidate for the train window: plat/gitops#12 (#60 prerequisite (a) — hostNetwork registry-mirror forwarder + pod-network realm rewriter; statically validated, needs one disposable germination to prove the node-sourced hop + hostNet DNS). The validation germination this train already requires (fresh daughter from the new seed) is exactly that disposable platform — sequence: merge agents#10+#11 + gitops#11 (+#12 if accepted) → tag 0.7.6 → pin bump → reseed → ONE validation germination proves the whole set: webhook idempotency (restart → still one hook), no 10-min reviewer fallback, :80 redirect matrix, and mirror pulls through the hostNetwork forwarder.

commented

RC validation IN FLIGHT on the vxrail (prod host, isolated cluster val076, PORTS=none — platgold2 untouched): assembled plat/agents v0.7.6-rc.1 (merge of agents#10+#11+#12; 77 unit tests pass on the MERGED tree) + plat/gitops rc/0.7.6 (gitops#11+#12 + agents pin 0.7.6-rc.1), RC seed built with GITOPS_REF=rc/0.7.6, throwaway germination running the full assertion matrix: whiteboard, ONE-webhook-after-restart (#63/#67), no-fallback reviewer (#57/#67), :80 redirect incl. germinated-first-boot race check (#64), hostNetwork mirror + crictl pull (#60 prereq), pure-spec next-idea bodies (#11), prod-isolation snapshots before/after. Evidence lands here when the run completes.

commented

PROD ROLLOUT WINDOW (advance notice): when val076 run 3 goes green AND the prod data snapshot completes, this train merges to prod via gitops main. What rolls: traefik (adds the :80→https redirectScheme file-provider middleware; tunnel XFP semantics preserved), registry-node-config (hostPort→hostNetwork forwarder split + NET_BIND_SERVICE capfix; a few seconds of mirror darkness per node during DaemonSet rollover, pulls retry), agents 0.7.6 (webhook idempotency #63, DNS-first preview probe #57, pure-spec bodies #11; the dispatcher restart resumes in-flight runs). Expected transients: one dispatcher restart, brief image-pull retries, a traefik reload — anything persisting >10 min after the pin-bump commit is a rollout defect and gets reverted (single git revert on gitops main). I will comment here immediately before merging and after Flux reconciles.

commented

val076 RC validation — 0.7.6-rc.1 train GREEN (full matrix, fresh germination)

Team-validate evidence. Throwaway platform val076 germinated on the VxRail host (k3d, PORTS=none, prod untouched) from a seed carrying gitops rc/0.7.6 (head b3b6723, incl. the caddy capfix) + agents v0.7.6-rc.1, with germinate running from plat/mitosis fix/germinate-prerelease-tags (62b5ad9, PR #68). All three validated TOGETHER — PR #68 must merge as part of this train.

Two release-blockers found and fixed by this validation

  1. germinate pre-release tag regex — bin/germinate extracted chart tags with a numeric-only regex under set -euo pipefail; the RC pin 0.7.6-rc.1 matched nothing and aborted germination silently at the post-Forgejo secrets step. Fixed in PR #68 (verbatim sed capture, non-fatal). Validated: run 3 extracted 0.7.6-rc.1 verbatim and drove the RC build.
  2. caddy file-caps EPERM in gitops#12 — official caddy:2-alpine ships /usr/bin/caddy cap_net_bind_service=ep; capabilities: drop: ["ALL"] gives an empty bounding set, so execve fails operation not permitted and BOTH registry-node-config + realm-rewrite crashloop, killing node pulls of git.val076.local/* (mcp/agents ImagePullBackOff, germinate FATAL). Docker repro matrix: --cap-drop=ALL fails, --cap-drop=ALL --cap-add=NET_BIND_SERVICE --security-opt no-new-privileges runs clean. Fixed on rc/0.7.6 (b3b6723, add: ["NET_BIND_SERVICE"] in both manifests). Validated: forwarder Running 0 restarts on fresh admission.

Assertion matrix (run 3, fresh germination, all quoted from live evidence)

#AssertionResultEvidence
A0RC pin drove the buildPASS==> building agents v0.7.6-rc.1 — the chart Flux pins for the dispatcher → plat/charts/agents:0.7.6-rc.1 ready (attempt 1/4, first try)
APlatform wholePASS10/10 Flux Kustomizations Ready=True; 12 ClusterPolicies Ready; HelmRelease [email protected]+e64f15c54e0f Ready; OCIRepository digest 0.7.6-rc.1@sha256:e64f15c…; deploy image git.val076.local/plat/agents:0.7.6-rc.1; zero unhealthy pods
B#63 webhook idempotencyPASSPre-restart: exactly one system hook (id 1 → http://agents.agents.svc.cluster.local:9909/hook, ids 2–8 404). rollout restart deploy/agents → still exactly one, same id 1; dispatcher logged system webhook verified id=1 … signed=true (verify path, no second create)
C#57 no-fallback + full pipelinePASSvalpulse: PR #1 opened 03:58:41 → builder run finished … exitCode=0 durationMs=1277220 (04:19:58) → reviewer run queued 04:19:50 (gap ≈ 0s, vs 10-min ceiling) → verdict posted … exitCode=0 → v1 auto-merged after validation (04:23:59) → prod URL https://valpulse-plat.val076.local/ 200 in-cluster. Log contains 0 preview liveness unconfirmed (healthy path: preview live on a different sha; reviewing anyway). Dispatcher: builder left its own ship summary; no fallback. POL-2: both plat--valpulse--pr-1 and plat--valpulse--prod got plat.sh/app=valpulse labels, plat-kyverno-dos-bound + tenant-quota quotas, 6 netpols incl default-deny
D#64 redirect first-bootPASSFrom in-cluster pod: plain http → 301 https://git.val076.local/; X-Forwarded-Proto: https → 200; https SNI → 200. Re-asserted identical AFTER rollout restart deploy/traefik on a platform that GERMINATED with the middleware; cold-boot traefik pod logs 0 does not exist middleware errors (pre-restart log had 30 case-insensitive matches attributable to initial-reconcile ordering; post-restart steady state is clean and behavior was correct throughout)
Egitops#12 mirrorPASSDaemonSet hostNetwork=true ready=1, pod Running 0 restarts; node netns 127.0.0.1:31100 LISTEN; registry-realm-rewrite 1/1; crictl pull git.val076.local/plat/ci-builder:0.2.0 → success
Fagents#12 pure-spec bodiesPASSShip summary carries 3 next-idea links; URL-decoded body= params are pure specs (badge / leaderboard / milestones) with no agent-work text; label instruction is a plain comment line below the links
GProd isolationPASSBefore AND after: {"version":"15.0.0+gitea-1.22.0"} HTTP 200; k3d-platgold2- Up 2 days, zero restarts; host :80/*:443 listeners unchanged; every kubectl carried --context k3d-val076; PORTS=none throughout

Timings

Seed build ~1 min; germination (cluster create → YOUR PLATFORM card) ~24 min incl. serial CI builds (ci-builder 0.1.0+0.2.0, mcp 0.26.0, agents 0.7.6-rc.1 — all attempt 1/4); valpulse end-to-end create→prod-200 ~30 min (builder 21m17s, reviewer 4m9s, verdict→merge <1s).

Verdict: the 0.7.6-rc.1 train (agents v0.7.6-rc.1 + gitops rc/0.7.6 @ b3b6723 + mitosis PR #68) is validated end-to-end on a fresh germination and ready to merge. Throwaway val076 deleted after this comment; forensics (up.logs of all 3 runs, seeds, fork keys) retained at vxrail:~/val076-mitosis.

commented

MERGING NOW (run 3 full green — see evidence comment above; prod data snapshot verified + off-host copy taken). Order: agents rc/0.7.6→main + tag v0.7.6 → mitosis #68 → gitops #11+#12 → gitops pin 0.7.6 (prod reconciles here) → final reseed → #65. Canary heads-up: the prod-roll transients described in the rollout-window comment apply from the pin-bump commit.

commented

TRAIN MERGED. agents: release PR #13 (rc/0.7.6) → main, tag v0.7.6, chart built on origin CI. mitosis: PR #68 merged. gitops: #11 (redirect) + #12 (mirror+capfix) + pin PR #13 (agents 0.7.6) all merged — prod Flux reconciling now (expected transients per the rollout-window comment; canary loop owns prod-side response). Remaining train items, resuming next wake: final reseed → update+merge #65 (closes #62); evidence-close #57/#63/#64/#11; #60 prereq comment. Session-limit note: two worker teams (CI-1 #21, fork #70) were cut mid-mission — partial work will be salvaged, and clusters ci2/fork70 on the vxrail may be orphans pending sweep.

commented

TRAIN COMPLETE. Shipped: agents v0.7.6 (release #13; #10/#11/#12), gitops #11+#12+pin #13 (prod reconciled — verified live: agents:0.7.6 image, single system webhook after the documented legacy-hook remediation), mitosis #68 + reseed #65 (genesis seed = released train, bundle-verified). Issues closed with evidence: #57, #62, #63, #64, #11. Two release-blockers found+fixed+validated by the RC process itself (germinate pre-release regex; caddy file-caps EPERM). Prod data: full verified backup + nightly timer + off-host copy BEFORE the roll; prod healthy throughout. Remaining follow-ups tracked on their own issues: #21 (PR #71 + runner proof), #70 (validation+PR), #60 (Cilium swap, blocker B), #67 (gate assertions — now partially encoded in PR #71 gate script).

commented

Train COMPLETE — every item on this issue shipped and proved: agents v0.7.6 on prod + in the seed (release #13; #57/#63/#11 fixes validated on val076 AND by every CI germination since — A3/E-block assertions), gitops redirect+mirror+pin merged (prod reconciled, no incident), mitosis #68, genesis reseed merged (#65) and then AUTOMATED (#14 closed — reseeds now build/verify/PR/gate themselves; two ran today: r25 merged, r27 correctly redundant). The one germination-proves-everything plan became the standing coldstart gate. Closing the train.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs