Public: anyone can read this; nobody can change it.

🧫 gauntlet: rehome failing #271

Closed opened this view on origin

Closed without the crew

opened this view on origin

The gauntlet scenario rehome failed in https://git.open-platform.sh/open-platform/mitosis/actions/runs/22118 (2026-09-09T06:24:17Z).

Log tail:

    app-template flagged as a template repo
    wiki: published 22 pages as the native repo wiki (open-platform/mitosis/wiki)
==> registering runner token with forgejo
    runner token registered
==> bootstrapping git.gt947-rehome.local/open-platform/ci-runner:0.1.0 on the host (breaks the no-runner/no-image cycle)
    imported git.gt947-rehome.local/open-platform/ci-runner:0.1.0 into k3d cluster 'gt947-rehome'
    isolated-vintage seed: no workflow pins a ci-builder container — skipping ci-builder builds (CI runs host-mode on ci-runner)
==> building mcp v0.52.1 — the chart Flux pins for the platform MCP
==> building mcp v0.52.1 (attempt 1/4) — verifying open-platform/charts/mcp:0.52.1 lands (≤35m)
    …waiting for open-platform/charts/mcp:0.52.1 (35m left, CI=running)
    open-platform/charts/mcp:0.52.1 ready
==> building agents v0.43.0 — the chart Flux pins for the dispatcher
==> building agents v0.43.0 (attempt 1/4) — verifying open-platform/charts/agents:0.43.0 lands (≤35m)
    …waiting for open-platform/charts/agents:0.43.0 (35m left, CI=running)
    …waiting for open-platform/charts/agents:0.43.0 (33m left, CI=running)
    …waiting for open-platform/charts/agents:0.43.0 (32m left, CI=running)
    …waiting for open-platform/charts/agents:0.43.0 (31m left, CI=running)
    …waiting for open-platform/charts/agents:0.43.0 (30m left, CI=running)
    …waiting for open-platform/charts/agents:0.43.0 (29m left, CI=running)
    open-platform/charts/agents:0.43.0 ready
==> sealing the daughter CA into plat-mcp-secrets
    daughter CA sealed + gitops re-pushed (verified, ns=cert-manager)
==> finalizing: reconcile charts + secrets, then verify mcp + agents are up
FATAL: mcp never became Available — inspect: kubectl -n mcp describe pod (CA seal / chart pull)
── cluster state at failure (best-effort) ──
agents         agents-548f5df78f-6hwqc                                    0/1   ContainerCreating   0               10s
forgejo        forgejo-runner-isolated-nclcs-5kfng                        0/3   Completed           0               3m44s
mcp            mcp-757f7d9d68-bj7vp                                       0/1   Terminating         5               5m
mcp            mcp-799989fb6b-7cddk                                       0/1   Terminating         0               0s
mcp            plat-auth-7d6b744d8b-56d27                                 1/1   Terminating         0               5m
mcp            plat-auth-7d6b744d8b-l2rsf                                 1/1   Terminating         0               5m
monitoring     grafana-folder-acl-29815580-fffwg                          0/1   Error               0               3m58s
monitoring     grafana-folder-acl-29815580-h4jr4                          0/1   Error               0               3m47s
monitoring     grafana-folder-acl-29815582-4wvwt                          0/1   Error               0               2m5s
monitoring     grafana-folder-acl-29815582-8wqzj                          0/1   Error               0               115s
monitoring     grafana-folder-acl-29815584-5hg54                          0/1   Error               0               5s
plat-storage   cnpg-backup-minio-setup-sxz6s                              0/1   Completed           2               11m
flux-system    7m31s       Warning   NotificationDispatchFailed   alert/plat-gitops                                     failed to dispatch notification for Kustomization/flux-system/flux-system: failed to initialize notifier for provider 'plat-gitops': failed to initialize notifier: failed creating Gitea client: access token does not exist [sha: PLACEHOLDER]
flux-system    7m28s       Warning   NotificationDispatchFailed   alert/plat-gitops                                     failed to dispatch notification for Kustomization/flux-system/tenants: failed to initialize notifier for provider 'plat-gitops': failed to initialize notifier: failed creating Gitea client: access token does not exist [sha: PLACEHOLDER]
monitoring     6m26s       Warning   Unhealthy                    pod/kube-prometheus-stack-grafana-f677475f6-h27mr     Readiness probe failed: Get "http://10.42.0.57:3000/api/health": dial tcp 10.42.0.57:3000: connect: connection refused
monitoring     6m7s        Warning   Unhealthy                    pod/promgate-f556b6668-vsvph                          Readiness probe failed: Get "http://10.42.0.21:9091/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
monitoring     5m57s       Warning   Unhealthy                    pod/promgate-f556b6668-cjw6k                          Readiness probe failed: Get "http://10.42.0.20:9091/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
monitoring     5m32s       Warning   Unhealthy                    pod/prometheus-kube-prometheus-stack-prometheus-0     Liveness probe failed: Get "http://10.42.0.43:9090/-/healthy": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
monitoring     5m26s       Warning   Unhealthy                    pod/prometheus-kube-prometheus-stack-prometheus-0     Readiness probe failed: Get "http://10.42.0.43:9090/-/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
flux-system    5m          Warning   OCIArtifactPullFailed        ocirepository/agents                                  failed to determine artifact digest: GET http://forgejo-http.forgejo.svc.cluster.local:3000/v2/open-platform/charts/agents/manifests/0.43.0: MANIFEST_UNKNOWN:
monitoring     4m44s       Warning   Unhealthy                    pod/kube-prometheus-stack-grafana-7965ff4bd4-wt2cp    Readiness probe failed: Get "http://10.42.0.35:3000/api/health": dial tcp 10.42.0.35:3000: connect: connection refused
kube-system    4m10s       Warning   Unhealthy                    pod/metrics-server-5985cbc9d7-7ltxg                   Readiness probe failed: HTTP probe failed with statuscode: 500
monitoring     3m59s       Warning   BackoffLimitExceeded         job/grafana-folder-acl-29815578                       Job has reached the specified backoff limit
monitoring     3m45s       Warning   BackoffLimitExceeded         job/grafana-folder-acl-29815580                       Job has reached the specified backoff limit
mcp            2m45s       Warning   BackOff                      pod/mcp-757f7d9d68-bj7vp                              Back-off restarting failed container write-ca in pod mcp-757f7d9d68-bj7vp_mcp(854e58ea-d315-428b-9551-719fc69e0fdb)
monitoring     113s        Warning   BackoffLimitExceeded         job/grafana-folder-acl-29815582                       Job has reached the specified backoff limit
mcp            1s          Warning   InstallFailed                helmrelease/mcp                                       Helm install failed for release mcp/mcp with chart [email protected]+9faa51f2b5ae: timeout waiting for: [Deployment/mcp/mcp status: 'InProgress']...
NAMESPACE      NAME                                                       AGE   READY     STATUS
agents         helmrelease.helm.toolkit.fluxcd.io/agents                  11m   Unknown   Running 'install' action with timeout of 30m0s
mcp            helmrelease.helm.toolkit.fluxcd.io/mcp                     11m   False     Helm install failed for release mcp/mcp with chart [email protected]+9faa51f2b5ae: timeout waiting for: [Deployment/mcp/mcp status: 'InProgress']

NAMESPACE     NAME                                                           AGE   READY   STATUS
flux-system   kustomization.kustomize.toolkit.fluxcd.io/apps                 12m   False   dependency 'flux-system/platform' revision is not up to date
flux-system   kustomization.kustomize.toolkit.fluxcd.io/kyverno-policies     12m   False   dependency 'flux-system/kyverno' is not ready
flux-system   kustomization.kustomize.toolkit.fluxcd.io/monitoring-scrapes   12m   False   dependency 'flux-system/monitoring' is not ready

Re-run just this scenario: dispatch the gauntlet workflow with scenarios=rehome. The scenario is code: scripts/gauntlet/scenarios/rehome.sh. This issue is edited in place on every failure and closes itself on the next pass.

commented

Passing again in https://git.open-platform.sh/open-platform/mitosis/actions/runs/22134 — closing.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs