Public: anyone can read this; nobody can change it.

monitoring: grafana-folder-acl CronJob fails every 2m post-germinate — cannot reach grafana service (ECONNREFUSED) #233

Closed opened this view on origin

Closed without the crew

opened this view on origin

Symptom

On a fresh single-node germination, the grafana-folder-acl CronJob (/2 * * *, namespace monitoring) fails every run and leaves a steady stream of Error pods:

NAMESPACE    NAME                                STATUS
monitoring   grafana-folder-acl-29796244-...     Error
monitoring   grafana-folder-acl-29796246-...     Error
monitoring   grafana-folder-acl-29796248-...     Error

Pod log (Bun script):

TypeError: Unable to connect. Is the computer able to access the url?
  path: "http://kube-prometheus-stack-grafana.monitoring/api/folders?limit=100",
  errno: 0,
  code: "ConnectionRefused"
Bun v1.4.0 (Linux x64)

No grafana-folder-acl job has ever succeeded on this cluster.

Not a startup race

Grafana itself is healthy — kube-prometheus-stack-grafana pod is 3/3 Running, service 10.43.89.145:80/TCP, endpoints 10.42.0.80:3000. Yet the failures persist for many minutes and across every job cycle after Grafana became Ready.

To rule out the CronJob image, I ran an independent probe pod in the same namespace:

kubectl -n monitoring run netcheck --rm -i --restart=Never \
  --image=curlimages/curl -- curl http://kube-prometheus-stack-grafana.monitoring/api/health
# -> HTTP 000, pod terminated (Error)

So a plain curl pod in monitoring also cannot reach the Grafana service — the folder-acl job is just the visible victim.

Likely area: NetworkPolicies

monitoring has a default-deny-ingress plus allow-lists:

NAME                            POD-SELECTOR
allow-ingress-grafana           app.kubernetes.io/name=grafana
allow-ingress-intra-namespace   <none>
allow-ingress-prometheus-flow   app.kubernetes.io/name=prometheus
allow-ingress-prometheus-www    app.kubernetes.io/name=prometheus
default-deny-ingress            <none>

allow-ingress-intra-namespace (empty pod-selector) is presumably meant to let in-namespace pods (the folder-acl job, the sidecars) reach Grafana, but ingress to Grafana:3000 is being refused/blocked in practice. Worth checking whether allow-ingress-grafana actually admits the folder-acl Job's pod labels / the intra-namespace policy covers port 3000, or whether the folder-acl Job needs a label the allow-list selects on.

Environment

  • plat/mitosis origin/main @ b0ea45d, deploy/do, single droplet (ha=false), s-4vcpu-8gb, nyc3, k3s v1.31.5+k3s1.
  • Grafana 13.1.1; folder-acl image is a Bun script.
  • Observed ~25 min after germination completed; still failing.

Impact

Low urgency (Grafana serves fine; only the folder-ACL provisioning CronJob is broken), but it produces a continuous stream of Error pods every 2 minutes forever, which clutters the namespace and will trip any "failed pods" alerting.

commented

Fix on gitops main (already live on the parent; daughters inherit at next reseed): the allow-ingress-intra-namespace policy now also matches by namespace membership — your evidence showed the namespaceSelector path (traefik → grafana) working while the podSelector-only intra-namespace allow refused every short-lived job pod, which is a known-fragile match on k3s's embedded netpol controller. If your platform still shows Error pods after pulling the netpol change, re-open with a fresh kubectl -n monitoring run netcheck … probe.

commented

Closing platform-side: the namespace-membership match is live on the parent and ships in seed r779, so every new deploy is clean. For the existing platform from this report, pull the one-hunk netpol change into your gitops (clusters/local/platform/monitoring/netpol.yaml, gitops 617b0554) — or redeploy from the current seed — and the folder-acl job goes green. Re-open with a fresh netcheck probe if it doesn't.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs