/goal /ultracode-workflows [GATE] validated grandchild acceptance gate — secrets+domains+rbac+policy on a VPS depth-2/3 platform #46

Track/Wave: GATE · Wave G6 · Size L Repos: mitosis, mcp, agents, gitops, _app-template Depends on: SPINE-4, SEC-1, DOM-1a, DOM-1b, DOM-2, RBAC-2, RBAC-3, RBAC-4, POL-2, POL-3, POL-4, POL-5, HYB-2

Why

The final deliverable: an unattended acceptance run proving all five primitives at once on a real-infra granddaughter — the client-deployable validation report.

Scope

Extend the CI-1 harness into a full acceptance gate on REAL VPS infra: germinate genesis→daughter→granddaughter, then on the granddaughter assert ALL five at once — (secrets) age-key rotation preserves a live app secret + prod/dev scoping holds; (domains) a real custom domain verifies via DNS-01 and serves a browser-trusted PRODUCTION Let's Encrypt cert on the public edge; (rbac) apps are private-by-Forgejo-permission through forwardAuth and agent runs use a scoped kube token, not the god-PAT; (policy) Kyverno rejects a plaintext Secret, an unregistered Ingress host, a blanket-egress app, and a god-PAT-scoped SA; (app) a real agent-built app with DB+bucket serves 200s at its public URL.

Acceptance

One unattended gate run passes EVERY assertion and emits a client-deployable validation report; any single failure fails the gate.

Notes

The capstone.

--- Part of the First Client Deploy → Governance roadmap. The owning agent may recurse into its own subagent team (ultracode workflow) if the task warrants. Honor the seam-serialization: land on a shared seam by rebase, and don't start a seam position until the prior one has merged.

GATE PASSED — mission complete. Coordinated release (mcp v0.26.0 + agents v0.7.5 tags, gitops#10 pins, mitosis#61 seed) + germinate-integration fix (bin/lib.sh dynamic trim so fresh platforms deploy Kyverno/policies/letsencrypt/cosign-signing + never drift; a set-u latent bug in the trim was caught+fixed; bin/germinate builds ci-builder 0.1.0+0.2.0; _app-template#6 makes cosign signing conditional). PROVEN LIVE on a fresh germination (wb-gate2): (1) secrets rotate_age_key preserves a live secret / old key decrypts nothing; (2) forwardAuth four-tier gate; (3) agent-ctx scoped token can-i own-ns yes / cross+secrets+cluster 403; (4) Kyverno deployed BY germinate — 12 ClusterPolicies Ready, rejects privileged/non-sovereign pods; (5) webhook HMAC tampered→401/signed→200; (6) a create_app app ships FIRST-TRY to a running prod pod (HTTP 200) in a Kyverno-GOVERNED namespace (generated quota + netpols). All 5 whiteboard dimensions live end-to-end on a fresh platform. DOM-1a public-TLS separately proven on the real product-garden.com VPS deploy.

Two gate assertions discovered during today's wb-gate validation, filed as #67: both failure modes SELF-HEAL (the app still ships and serves 200), so a naive gate pass would miss them.

1. Exactly one system webhook survives a dispatcher restart — one rollout restart produced two active admin hooks on the same URL → every event delivered twice (#63; fix in flight plat/agents#10). Assert by GET /api/v1/admin/hooks/{id} — the LIST endpoint returns [] while hooks exist.
2. Reviewer engages without the ~10-min fallback — the liveness probe used loopback from inside the pod while the preview was already 200 with the correct sha (#57; fix in flight plat/agents#11). Assert reviewer engagement < 3 min after preview-200/sha-match, no "preview liveness unconfirmed" log line.

Both should land in the unattended VPS gate run (and the #21 harness) so agents 0.7.6 (#66) stays fixed.

Depth-2 chain evidence for the GATE: genesis seed -> daughter (d2a) -> granddaughter (d2b)

Proving the released capabilities hold AT DEPTH: germinated d2a from the committed genesis/seed.tar.gz (fresh clone of main@8a7dc0e, the #70/#72 release), minted seed-d2.tar.gz FROM the running d2a, germinated d2b from that. Both clusters PORTS=none on the vxrail host; prod untouched (snapshots at bottom). #70's fork chain exercised for real at both hops — including a genuine tier fallback at hop 2.

Evidence matrix

Hop 1 — fork chain link (quoted)

mitosis fork tier-1: migrating https://git.open-platform.sh/plat/mitosis (one-time, mirror=false)…
mitosis fork RESULT: tier-1 src=https://git.open-platform.sh/plat/mitosis

ORIGIN on d2a (full chain present — tier-1 carries the parent's ORIGIN forward):

root: https://git.open-platform.sh/plat/mitosis
open-platform.sh germinated-from open-platform.sh 2026-06-10T03:33:22Z seed=seed.tar.gz
d2a.local germinated-from open-platform.sh 2026-07-02T15:38:31Z seed=seed.tar.gz

Hop 2 — seed FROM d2a, and which tier won (quoted)

bin/seed against d2a over the only route a PORTS=none parent offers (kubectl port-forward, FORGEJO_URL=http://127.0.0.1:3100). The manifest:

createdFrom: http://127.0.0.1:3100    <- invocation endpoint, recorded verbatim
domain: d2a.local                     <- the durable identity, fetched by the same script

d2b germination — parent alive, port-forward up, so the HOST-side tier-1 probe passed; the migrate runs in the daughter's Forgejo POD, where 127.0.0.1 is the pod itself:

mitosis fork tier-1: migrating http://127.0.0.1:3100/plat/mitosis (one-time, mirror=false)…
mitosis fork tier-1: migrate of http://127.0.0.1:3100/plat/mitosis FAILED (Forgejo-side) — falling through
mitosis fork tier-2: migrating https://git.open-platform.sh/plat/mitosis (one-time, mirror=false)…
mitosis fork RESULT: tier-2 src=https://git.open-platform.sh/plat/mitosis

The fallback design worked: a mid-chain tier failure produced a fully working platform (13/13 gate), full history, forge-recorded source, half-created repo cleaned before retry. This is exactly the resilience #70 promised.

The genealogy cost is real: ORIGIN on d2b (tier-2 shape — root's ORIGIN, not d2a's):

root: https://git.open-platform.sh/plat/mitosis
open-platform.sh germinated-from open-platform.sh 2026-06-10T03:33:22Z seed=seed.tar.gz
d2b.local germinated-from d2a.local 2026-07-02T16:57:55Z seed=seed-d2.tar.gz

The d2b line correctly names d2a as parent (manifest domain: is durable), so the chain stays inferable — but d2a's OWN line (d2a.local germinated-from open-platform.sh …) is absent, and original_url points at the root instead of the actual parent. Until createdFrom is durable, depth >= 2 genealogy degrades to the root on every port-forward-minted seed.

Findings

1. createdFrom fidelity gap (filed as #76, with fix direction): bin/seed records createdFrom: $FORGEJO_URL verbatim; a port-forward URL (the standard way to reach a PORTS=none parent) silently costs daughters the tier-1 fork. Fix: record https://git.<SEED_DOMAIN> (bareDomain from reserved-names — already fetched by the same script), falling back to $FORGEJO_URL only when unknown.
2. fj_pf fixed host port 3000 collides with concurrent germinations: my first d2a attempt died with FATAL: bootstrap Forgejo admin 'plat' never authenticated — root cause was a concurrent CI coldstart (ci9) holding 127.0.0.1:3000; germinate's auth gate silently authenticated against the WRONG cluster's Forgejo (bin/germinate fj_pf hardcodes 3000:3000; anonymous /api/v1/version on the wrong instance answers 200, so the port-forward "looks up"). CI serializes with itself via the concurrency group, but manual runs are unprotected. Re-ran with the port-forward on a distinct local port and everything passed — the ONLY local modification to the released code was that host-side port (bin/germinate lines 225/227, 3000->3007); no platform behavior touched. Worth a tiny fix (random free port, like coldstart-assert.sh already does). Not filed — staying within the one-issue budget for this run; happy to file/PR on request.
3. Gate A4 detection (already fixed on main while this ran): gate@8a7dc0e SKIPped A4 ("no redirect-to-https middleware present") because it greps Middleware CRs while the platform ships a file-provider middleware. #73/#74 (310081a) fixed detection via HelmRelease values; both gate runs above used the fixed script. Bonus depth observation: both daughters' mitosis forks carry 310081a — a merge that landed on root main ~30 min before hop 1 — because repo-migrate takes live HEAD, not the seed's snapshot.

Prod isolation (before / after)

BEFORE: api/v1/version HTTP 200 {"version":"15.0.0+gitea-1.22.0"}
        k3d-platgold2-serverlb Up 2 days / k3d-platgold2-server-0 Up 2 days
AFTER:  api/v1/version HTTP 200 {"version":"15.0.0+gitea-1.22.0"}
        k3d-platgold2-serverlb Up 2 days / k3d-platgold2-server-0 Up 2 days

Teardown verified: docker ps -a | grep k3d-d2 -> ZERO containers; d2 networks gone; throwaway PAT/fork-keys/kubeconfig removed. All runs used a dedicated KUBECONFIG — the host default context was never switched.

GATE evidence on a real public VPS granddaughter (2026-07-06) — 4 of 5 primitives live

On product-garden.com (depth-2, DigitalOcean bare-k3s + Cloudflare tunnel), a crew-built app running end-to-end:

• secrets ✅ per-fork sops-sealed (10 forged secrets sovereign at germination); Enforce denies plaintext Secrets.
• domains/TLS ✅ real public browser-trusted TLS (ssl_verify_result: 0) on git/mcp/app; Enforce denies unregistered Ingress hosts.
• policy ✅ Kyverno Enforce live (3-replica HA), rejects plaintext-secret / unregistered-host / non-sovereign image; the app deployed + serves in a Kyverno-governed ns with generated quota/limits/netpols + a cosign-signed image (verified).
• app ✅ qa-smoke link-shortener built by the crew, DB-backed (CNPG), serving 200 publicly.
• rbac/forwardAuth ❌ could not be enabled — traefik cannot resolve the per-app kubernetescrd forwardAuth Middleware, which 404s every app ingress (filed #93). Disabled so apps serve. agent-ctx scoped SA IS rendered on prod; forwardAuth's own service is healthy, but the traefik integration is broken on this stack.

So the multi-primitive acceptance holds for secrets+domains+policy+app+signing on a real VPS granddaughter, but the literal single unattended all-five gate is NOT met (forwardAuth #93 blocks it, and it wasn't one unattended run). Keeping open pending #93 + a re-run. Also surfaced the gate's structural blind spot: it only germinates k3d, so bare-VPS bugs (#89/#91/#93) are invisible to it.