Metrics

GET /metrics answers Prometheus text (text/plain; version=0.0.4), one format for the whole body. It names repositories and route patterns, so it is for a site admin alone: 404 without credentials and for anyone else who is not an admin, 200 for an admin's ro token. It is never cached (cache-control: no-store).

FamilyTypeLabelsMeaning
forge_egress_connectionsgaugerepo, destinationOpen egress connections.
forge_egress_bytes_ingaugerepo, destinationBytes received from the destination over open connections.
forge_egress_bytes_outgaugerepo, destinationBytes sent to the destination over open connections.
forge_egress_received_bytes_totalcounterdestinationBytes received from the destination since the process started, every app and check together.
forge_egress_sent_bytes_totalcounterdestinationBytes sent to the destination since the process started.
forge_http_request_duration_secondshistogrampattern (plus le)Seconds taken to answer a request, by the route pattern matched (never the raw path), since the process started. Buckets are route_timing's edges, 1 ms to 2.5 s, in seconds.
forge_check_duration_secondshistogram(plus le)Seconds a repository check took from start to finish, for checks that ended since the process started. Buckets 5 s to 30 min.
forge_check_queue_depthgaugenoneChecks queued and not yet started.
forge_crew_runs_totalcountermember, outcomeCrew runs that ended, by the member's username and ok, failed or cancelled.
forge_crew_spend_usd_totalcountermemberUS dollars the member's runs spent, from the usage ledger (list-price equivalent on a subscription).
forge_pulls_merged_totalcounterrepoPulls merged, from pull.merged_at.
forge_releases_totalcounternoneReleases that went live, from the release table's ok rows.
forge_ask_to_live_secondshistogramnone (plus le)Seconds from an issue opening to the first successful release of a pull that closes it. Buckets: 1 m, 5 m, 15 m, 1 h, 4 h, 12 h, 1 d, 3 d, 7 d.
forge_main_red_seconds_totalcounterrepoSeconds the default branch's check spent failing: from a failed check to the next green one, or to now while still red.
forge_telemetry_reads_totalcounterroute, readerReads of /api/v1/telemetry routes since the process started, by route and reader; the rate is the slope. The owner's /system page names each reader's pace over the last 10 minutes.
forge_loop_stall_milliseconds_totalcounterrouteMilliseconds the main thread stalled past 200 ms since the process started, charged in full to every route pattern running across the stall (none when no request was).
forge_loop_stalls_totalcounterrouteThose stalls, counted the same way.
forge_loop_stall_hour_millisecondsgaugerouteStall milliseconds in the last hour: the top 10 routes, most first, plus /api/v1/telemetry/flow always (0 when it stalled nothing).
forge_loop_stall_hour_countgaugerouteStalls in the last hour, for the routes the gauge above lists.
forge_app_upgaugeapp, env1 when the app environment is serving, 0 when it is asleep, stopped, deploying or failed.
forge_cell_cpu_seconds (samples forge_cell_cpu_seconds_total)counterapp, envCPU seconds (user plus system) used by the processes of a running app cell.
forge_cell_memory_bytesgaugeapp, envResident memory of the processes of a running app cell.

app is the app's slug, or its id for a private app: a private app's name is never in the body. env is production or pull-<n>. The cell families are what the cell watch last read (every few seconds); a cell that is not running has no series (a cell keeps the slug it started under, so after a rename its figures return when it restarts), and the forge starts nothing and reads no process to answer.

The duration histograms count every request the process has answered, and every check it has seen finish, since it started; their _bucket, _count and _sum only grow, so rate() and histogram_quantile() read them as Prometheus expects. A restart starts them at zero again, which Prometheus reads as a counter reset. A request counts the moment it is answered; a check counts at the first scrape after it finishes. The hourly route_timing table, which keeps 30 days, is the place to look further back.

The delivery series are read from the pull, release and run tables on every scrape, so the scrape cost grows with those tables. The check histogram reads only the checks finished since the last scrape, through the run_kind_finished index; the queue and crew series are small grouped reads.

The stall families (#9016) are the event loop blocked log lines (src/perf/loop.ts) as figures, so a fix's before and after reads from the forge itself. A stall overlapping two requests counts against both, as the log line names both, so summing over routes can exceed the wall time. To compare the telemetry flow read with #8978 (69.6 s of stalls in 6 minutes, about 700 s an hour at that rate), read forge_loop_stall_hour_milliseconds{route="/api/v1/telemetry/flow"}; for any window, increase(forge_loop_stall_milliseconds_total{route="/api/v1/telemetry/flow"}[6m]).

A public repository's repo label is owner/name; a private one appears under its id only, never its name.

Prometheus

Make an admin ro token in your account settings, then:

scrape_configs:
  - job_name: forge
    scheme: https
    metrics_path: /metrics
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/forge-token
    static_configs:
      - targets: ["forge.example.com"]

GET /metrics (site admin only; repository names are in the labels) serves forge_egress_connections, forge_egress_bytes_in and forge_egress_bytes_out per repo and destination as Prometheus text, so a leaked upstream socket shows as a gauge that never falls (#3719).

Beside them, forge_egress_received_bytes_total and forge_egress_sent_bytes_total count every byte since the process started, keyed by destination alone (never by run, so the series stay one per host). They show the bun cache working (#7546): on a warm cache, rate(forge_egress_received_bytes_total{destination="registry.npmjs.org:443"}[1h]) stays near zero, and a cold one is hundreds of megabytes an hour (#7541). Without Prometheus, the same hour is read four ways:

egress.npmTopRepoLastHour (the repository whose contained installs the registry sent most this hour, attributed by the cell's own meter, never by install output; null when none) and the ten busiest egress.destinations with their totals and last hour.

connections card.

same line, with the top repository, above the open connections; all three read egressHour.

the registry on a warm cache: what the cell's egress proxy received. A check outside a cell logs none: its repository's bun cache is shared, so no figure there would be its own. A contained check, gate or app install that received more than 1 MiB from registry.npmjs.org ends its log with install fetched N MiB from the npm registry; the shared cache missed` (#8382), failed or aborted installs included.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs