Public: anyone can read this; nobody can change it.

workbenchthis machine

/goal /ultracode-workflows [SPINE-2] fix ci-builder cold-start KEDA/runner stall (#19) + validate seed refresh (#14 / PR #20) #22

Closed opened this view on origin

Closed without the crew

opened this view on origin

Track/Wave: SPINE-2 · Wave M1 · Size M Repos: mitosis, gitops Depends on: none

Why

The ci-builder build is the serial gate every other artifact (mcp, agents, every app image) depends on in germinate's build_and_verify chain. An unfixed intermittent stall (#19) means germinate is not reliably one-shot for a client. PR #20's seed refresh is blocked behind the same bug.

Scope

Diagnose why KEDA won't rescale a runner for an orphaned ci-builder run on a cold cluster (scale-to-zero race in the runner ScaledObject, or a webhook-delivery gap during the sqlite→postgres Forgejo handoff). Fix in gitops (runner HelmRelease/KEDA ScaledObject) or in bin/germinate's build_and_verify retry logic. Then land PR #20's refreshed seed and validate it as the bundled genesis/seed.tar.gz.

Acceptance

10 consecutive cold-start runs clear build_and_verify on attempt 1–2 with no manual intervention; PR #20's refreshed seed passes and becomes main's bundled seed.

Notes

Seam A (germination scripts) — chain head. Closes #19, advances #14.

--- Part of the First Client Deploy → Governance roadmap. The owning agent may recurse into its own subagent team (ultracode workflow) if the task warrants. Honor the seam-serialization: land on a shared seam by rebase, and don't start a seam position until the prior one has merged.

2 earlier comments
commented

Partial: seed-refresh half DONE (#20 merged); #19-fix half open as a watch. The refreshed seed is merged to main and validated (germinates the current stack, healthy). Bug #19 (KEDA runner stall) did NOT reproduce across 4 clean germinations on the 80-core vxrail — every ci-builder/mcp/agents build landed on attempt 1. It is intermittent/timing-sensitive, so it may still bite on smaller client hardware (a Hetzner 4-8 core box, slower serial builds). Keeping this + the standalone #19 open to reproduce on a constrained VM before the client deploy. No longer blocks SPINE-3 (dependency dropped; #18 merged).

commented

Audit (team-board): both halves delivered in substance; the formal acceptance was never recorded — PROPOSING closure, operator's call.

  • Seed half: PR #20 merged + validated (clean-host germination healthy), since superseded twice (#56, #61) — main's bundled seed is current.
  • #19 half: closed; did not reproduce across 4 clean vxrail germinations (every build attempt-1), and today's fresh wb-gate daughter again landed ALL CI builds (ci-builder 0.1.0+0.2.0, mcp, agents, app) attempt-1, runner registered via KEDA ScaledJob.

Unmet formality: the "10 consecutive cold-start runs" series was never counted, and the constrained-hardware (4–8 core VPS) reproduction this stayed open for hasn't happened — that residual belongs more naturally to #21 (CI-1 harness) and #35 (real-VPS run). Close if you agree the residual moves to #21/#35.

commented

Acceptance now demonstrably EXCEEDED: the earlier close-proposal noted the 10-consecutive-runs formality was never counted — the coldstart CI has since produced well over ten consecutive successful cold-start germinations on this hardware class (gate runs 4/6/8/9/10/12/13/14/15 + two reseed-PR gates + the depth-2 d2a/d2b pair, every CI build attempt-1, zero KEDA/runner stalls — logs in vxrail:~/ci-gate-logs). #19 has not reproduced since the 0.2.0-skew fix era; the seed-refresh half was #14, closed today with the automated pipeline. Closing SPINE-2.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs