Public: anyone can read this; nobody can change it.

workbenchthis machine

🐤 App-building health canary — continuous validation + regression fixes (orchestration epic) #69

Closed opened this view on origin

Closed without the crew

opened this view on origin

Mission: keep open-platform.sh app-building + preview generation reliable while another agent loads the vxrail heavily.

Model: an orchestrator (me) runs a self-paced validation loop; each cycle an ultracode Workflow team builds a canary app in app-canary/ and asserts the full path (create_app → CI build on ci-builder → crew opens PR → preview env deploys + serves 200). On any failure, a team diagnoses + fixes it (recursively), opening a PR that I proof-gate + merge. All incidents tracked as linked issues here.

Baseline resolved: ci-builder:0.2.0 version-skew (new _app-template needs 0.2.0; in-place Flux upgrades do not rebuild it) — fixed by tagging plat/ci-builder v0.2.0. See the wheels-nice-auto-sales/www incident.

Running incident log: (comments below)

6 earlier comments
commented

✅ Canary probe #1 — PASS (app-canary/canary-313c)

Full path validated: create_app provisioned (no failure), CI build green on ci-builder:0.2.0, crew opened PR #1, preview env app-canary--canary-313c--pr-1 (app pod + Postgres 1/1 Running), preview served HTTP 200 rendering the spec ("canary ok" + build timestamp, version 0.0.0-pr.1.5cae93e).

Health snapshot: core pods Ready (mcp/agents/forgejo/kyverno/flux); KEDA runners scaled 0→2→1; node MemoryPressure/DiskPressure/PIDPressure all False; peak 7% CPU / 5% mem — massive headroom. Only contention effect from the other agent: ~1-2 min HelmRelease reconcile lag (slowness, not failure). No fix needed.

Probe duration ~11.5 min. Next probe after a cooldown (one at a time, to not starve the other agent).

commented

✅ Canary probe #2 — PASS (app-canary/canary-7712). Full path green: create_app→CI on ci-builder:0.2.0→crew PR #1→preview pod+Postgres Running→HTTP 200 ("canary ok"+timestamp). Health: core pods Ready, KEDA runners 0→2→0, node pressure False, 3% CPU / 5% mem — no contention observed from the other agent. Cleaned up prior canary-313c. Going forward I will batch PASSes into periodic rollups and comment individually only on FAIL/incidents to keep this epic readable.

commented

🧹 Incident: canary cleanup leak — RESOLVED (probe-cleanup gap, not a platform bug)

Cause: a released prod app’s source of truth is a gitops overlay (plat/gitops clusters/local/apps/<owner>/<app>/prod/: Namespace, CNPG, OCIRepository, HelmRelease, Ingress, NetPols) reconciled by the Flux apps Kustomization (prune:true, 1m). The probe deleted only the Forgejo repo + namespace, so Flux re-applied the still-present overlay → prod resurrected. Repo-deletion is by design not a decommission trigger (dispatcher has no repository.deleted handler; delete_app keeps the repo by default) — so no platform bug, no PR.

Correct decommission: mcp delete_app(owner,name,force_repo=true) — or manually: git rm the overlay dir + drop the - <owner>/<app>/prod kustomization line → push → flux reconcile kustomization apps (prune) → delete ns. Gitops prune MUST precede ns deletion.

Cleanup done: both leaked canary overlays removed in one scoped gitops commit (plat/gitops main 156fb46, app-canary/* only), reconciled, verified no resurrection over 2 min. Reclaimed 4 pods, 2 CNPG clusters, 4 PVCs (~2.2Gi), 2 namespaces; 31 tenant namespaces untouched.

Hardening: the canary probe’s cleanup step now fully decommissions (gitops-prune → reconcile → ns delete, scoped to app-canary/*) so future canaries can’t leak. Resuming build-probes.

commented

📊 Rollup — probes #3-4 PASS (canary-4dc1, canary-k9t3). App-building + preview generation reliable; all core pods Ready, KEDA runners scale 0→2→0.

✅ Hardened cleanup VALIDATED: probe #4 decommissioned the prior app (canary-4dc1) overlay-first — gitops commit 5878dec → flux reconcile kustomization apps (prune) → ns delete → repo delete (204) — and verified no resurrection over ~95s. Leak definitively closed; scope stayed strictly within app-canary/*.

⚠️ Watch — build LATENCY rising under the other agent's load: probe #4 took ~16 min create→serve vs ~9-11 min for probes #1-3. Every stage still went green; this is latency contention (builder/runner throughput), NOT resource exhaustion — node held 3-6% CPU, ~5% mem, all pressures False. Builds are slower but not failing. Guarding against this crossing into timeouts: going latency-aware — I will not pile canary builds onto a busy builder, and will record create→serve time each probe to track the trend.

commented

📊 Probe #5 — PASS (with a harness correction). canary-0b01 built, previewed, auto-merged, and serves HTTP 200 in prod (pod 1/1 Running). The probe agent ended its turn early (~11min) while the crew was still building, so its verdict came back ambiguous — but the build path was healthy the whole time (crew impl push, preview run, KEDA scaled 3 runners). Verified the true outcome directly.

Latency recalibration (correcting the earlier #4 note): the Claude crew takes ~10min to write+validate even a trivial app — that is BASELINE, not contention. So create→serve ~13-18min is normal; probe #4's ~16min was NOT a contention warning. Withdrawing the latency-trend concern; node has held 2-8% CPU throughout.

Hardened the probe: step (c) now ACTIVELY polls (kubectl every ~20-30s, sleeping) until a pod is Running 1/1 OR a terminal failure OR ~20min timeout — and polls BOTH the pr-1 preview AND the --prod env (PRs auto-merge to prod, tearing down the preview), so it no longer returns an ambiguous mid-build verdict. Cleaned the inert canary-7712 repo. 5/5 probes effectively PASS.

commented

📊 Rollup — probes #6-7 PASS (7/7 total). App-building + preview generation reliable end-to-end; hardened harness (active polling of pr-1+prod, gitops-prune decommission) proven stable across runs. Baseline create→serve ~13-18min = ordinary crew authoring time (not contention); node held 3-6% CPU throughout, KEDA runners scale 0→2→0, all core pods Ready. Decommission cleanups verified no-resurrection each time (incl. rebasing over the other agent's concurrent gitops commits). Now in an EFFICIENT posture: cheap health/risk checks each ~30min cycle, full build-probe only on a build-critical version change / load ramp / prior-FAIL / every 3rd cycle. agents 0.7.6 + mcp 0.26.0 steady. No incidents.

commented

Mission complete: 7/7 canary probes PASS across the 0.7.6 load window; both incidents resolved (ci-builder:0.2.0 version-skew via the v0.2.0 tag; canary cleanup leak closed with gitops-prune decommission, no-resurrection verified). The load event shipped 07-02 and the orchestrator loop ended. Continuous app-E2E coverage now lives in CI's nightly gate (mitosis c1c29b4), superseding this epic.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs