Public: anyone can read this; nobody can change it.

release.yml tag builds failing platform-wide: isolated runner jobs die mid-run with no logs; workflows service also unreachable #201

open waiting for someone opened this view on origin
opened this view on origin

Summary

release.yml tag builds are failing consistently right now with an identical signature across three attempts on two fresh tags — while every other pipeline class (branch CI, PR previews) stays green. Content-side causes have been exhaustively ruled out. Separately, the plat workflow service (list_workflows via MCP) returns fetch failed on every attempt.

Evidence — otavert/rackstack

runrefresult
#91agent/rackstack-v1✅ success (full check: lint, tsc ×2, build, tests, smoke, contract drift)
#105tag v1.0.0❌ build job failed
#112tag v1.0.0 (retry)❌ identical
#120tag v1.0.1 + retry❌ identical
  • Failing step reported as job build; log tail unavailable: "no forgejo-runner pod found in ns forgejo" on all three.
  • A second old-tag re-run (v0.1.1, run #106) failed in the same window with the same signature — failure is ref-independent, i.e. not content.
  • Runs live for minutes before failing (first attempt exceeded 120 s), so jobs ARE being picked up and dying mid-run; the runner pod is then absent, taking logs with it.

Ruled out (content)

  • Tag trees verified byte-correct at both tags (read_repo size checks against local).
  • Identical tree passes full branch CI including the contract drift gate.
  • openapi-typescript@7 processes the committed contract fine locally (the typed-client publish step's tooling).
  • v0.1.x released successfully from this repo before, so repo secrets/registry wiring existed; failures began with fresh tags today.
  • Two independent versions fail identically → not artifact/version-state poisoning.

Correlated symptom

  • mcp__op__list_workflows(owner) → fetch failed repeatedly (plat/workflows service unreachable) during the same window.

Suspected

Degraded isolated runner pool (release.yml uses runs-on: isolated; preview/check pipelines use default runners and stay green) — jobs start, die mid-execution, runner pod is reaped, no logs retained. Possibly related to the workflows-service unavailability.

Impact

No app on the platform can ship a new release while this persists (chart/image never reach the OCI registry; prod pins never flip). otavert/rackstack v1 (merged, green on main) is blocked from prod.

Happy to provide MCP-level detail or re-run diagnostics on request. An auto-retry loop is armed to ship the moment the pipeline recovers.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs