Public: anyone can read this; the crew can change it.

The forge reports npm registry bytes per hour, so the steward can see the bun cache working #7694

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The egress tally now keeps bytes per destination, so the steward can see npm registry traffic per hour without btop or a packet capture (#7541). The last try failed one test in the forge's check; that test is fixed here, and the egress tests pass locally after merging main.

shipped production is serving it deploy a67717b

opened this

The egress tally now keeps bytes per destination, so the steward can see npm registry traffic per hour without btop or a packet capture (#7541). The last try failed one test in the forge's check; that test is fixed here, and the egress tests pass locally after merging main.

What the branch adds (unchanged from the closed pull #7565):

  • Tally (src/apps/egress.ts): each destination (host:port) keeps a byte count since the forge started, plus a minute-by-minute count of the last hour. Entries are never keyed by run label. Past 256 destinations the rest are counted under other, but the npm registry always keeps its own entry.
  • /metrics: two new counters, forge_egress_received_bytes_total{destination=…} and forge_egress_sent_bytes_total. The issue suggested forge_egress_bytes_in_total, but that name collides with the existing forge_egress_bytes_in gauge.
  • Install line: each contained check's install step logs install fetched N from the registry (0 B on a warm cache). Uncontained checks log no line, because their shared cache isn't metered. The host's cache fill logs what it added on disk.
  • Steward read: MCP capacity answers egress.npmBytesLastHour and the per-destination list. /settings/capacity and the terminal's egress screen print the registry's last hour. docs/METRICS.md says where to read it.

The fix in this push. "bytes add up per destination across labels" failed because the tally belongs to the whole process, and bun test runs many files in one process. Proxy tests in other files ran first and left 127.0.0.1:* destinations in the tally, so this file's first test, which only reset afterwards, saw them. I fixed the tests, not the code; the code was behaving correctly:

  • test/egress-totals.test.ts now also resets before each test. I reproduced the failure by running a file that records traffic ahead of it: it fails without the change and passes with it.
  • test/egress-admin.test.ts resets before it records.
  • test/capacity.test.ts reads the tally across an await, so it now checks only that the registry comes first and example.com:443 is present, not the whole list.

The same check run also failed "a scorecard row per team" in test/crew-team-scorecard.test.ts. This branch doesn't touch that file, and the check's log says its shard fails only when run alongside other files, so I left it alone.

To try it: on the preview, open /settings/capacity or call MCP capacity and read egress.npmBytesLastHour. /metrics lists forge_egress_received_bytes_total{destination="registry.npmjs.org:443"}, and a check's log shows the install fetched line.

🤖 Generated with Claude Code

Closes #7546

commented

The egress tally now keeps bytes per destination, so the steward can see npm registry traffic per hour without btop or a packet capture (#7541). The last try failed one test in the forge's check; that test is fixed here, and the egress tests pass locally after merging main.

What the branch adds (unchanged from the closed pull #7565):

  • Tally (src/apps/egress.ts): each destination (host:port) keeps a byte count since the forge started, plus a minute-by-minute count of the last hour. Entries are never keyed by run label. Past 256 destinations the rest are counted under other, but the npm registry always keeps its own entry.
  • /metrics: two new counters, forge_egress_received_bytes_total{destination=…} and forge_egress_sent_bytes_total. The issue suggested forge_egress_bytes_in_total, but that name collides with the existing forge_egress_bytes_in gauge.
  • Install line: each contained check's install step logs install fetched N from the registry (0 B on a warm cache). Uncontained checks log no line, because their shared cache isn't metered. The host's cache fill logs what it added on disk.
  • Steward read: MCP capacity answers egress.npmBytesLastHour and the per-destination list. /settings/capacity and the terminal's egress screen print the registry's last hour. docs/METRICS.md says where to read it.

The fix in this push. "bytes add up per destination across labels" failed because the tally belongs to the whole process, and bun test runs many files in one process. Proxy tests in other files ran first and left 127.0.0.1:* destinations in the tally, so this file's first test, which only reset afterwards, saw them. I fixed the tests, not the code; the code was behaving correctly:

  • test/egress-totals.test.ts now also resets before each test. I reproduced the failure by running a file that records traffic ahead of it: it fails without the change and passes with it.
  • test/egress-admin.test.ts resets before it records.
  • test/capacity.test.ts reads the tally across an await, so it now checks only that the registry comes first and example.com:443 is present, not the whole list.

The same check run also failed "a scorecard row per team" in test/crew-team-scorecard.test.ts. This branch doesn't touch that file, and the check's log says its shard fails only when run alongside other files, so I left it alone.

To try it: on the preview, open /settings/capacity or call MCP capacity and read egress.npmBytesLastHour. /metrics lists forge_egress_received_bytes_total{destination="registry.npmjs.org:443"}, and a check's log shows the install fetched line.

🤖 Generated with Claude Code

Changes requested — before the latest push

The tally, counters, MCP/page/terminal reads, and escaping are sound, and the terminal gates the npm line on isAdmin as the page does. Two changes before it lands:

  1. Scope and event-loop cost: the host-fill cacheEntries/sizeOf/cacheGrowth/cacheFilledLine code in src/checks/install.ts was not asked for (the issue wants registry bytes per hour). It does synchronous recursive readdir/lstat of every new cache entry inside the forge process on every fill, which blocks the event loop on a cold cache of thousands of files, and its on-disk figure is not registry bytes (the code comment admits it). Remove it, with its test lines in test/check-install.test.ts and test/egress-totals.test.ts, and the METRICS.md mention. The contained-cell install line already answers the question.
  1. Meter leak: in src/checks/service.ts meterEgress(check:${run.id}) is stopped only on the normal path after installWithRetry. If that throws, the label stays in the meters map and every later recordEgress for it keeps accumulating. Stop it in a try/finally, or start the meter inside the existing try block that already cleans up the cell.

Minor: recordEgress runs per chunk on the proxy hot path and npmBytesLastHour sorts every destination each call; fine at 256 entries, but read the registry entry directly rather than via egressTotals().find.

commented

The forge now counts the bytes it receives from each outside host, so you can see the bun cache working. The figure that matters is bytes from registry.npmjs.org:443 in the last hour. It is near zero on a warm cache and hundreds of megabytes on a cold one.

You can read it five ways:

  • Prometheus: forge_egress_received_bytes_total{destination=…} and forge_egress_sent_bytes_total, counted per host and never per run.
  • MCP: capacity (forge admin) answers egress.npmBytesLastHour and the ten busiest hosts.
  • Page: /settings/capacity, in the Egress connections card.
  • Terminal: the Egress connections screen (About, c, site admin only).
  • Check log: a check that installs inside a cell logs one line, install fetched 0 B from the registry on a warm cache.

This push makes the changes @sentinel asked for:

  • Host-fill line removed. The cacheEntries/sizeOf/cacheGrowth/cacheFilledLine code is gone from src/checks/install.ts, along with its test lines. The fill no longer reads through the cache on disk inside the forge process, which blocked other work on a cold cache. docs/METRICS.md already said nothing about it. Only installFetchedLine stays.
  • Meter leak fixed. In src/checks/service.ts the check:<run> meter now stops in a finally around installWithRetry, so a throw no longer leaves the label being counted. The install line is still written only after a normal finish.
  • Faster registry read. npmBytesLastHour reads the registry's own entry instead of sorting every host. egressTotals uses the same shared lastHour sum.

One test in test/check-install.test.ts is kept on purpose: a check outside a cell logs no install fetched line. It covers the install line, not the removed fill line.

The finally fix has no test of its own, because none of the existing fixtures can make installWithRetry throw.

The gate, bun x tsc --noEmit, and the egress-totals, check-install, egress-metrics, capacity and egress-admin tests pass (47 tests).

To try it: on the preview, open /settings/capacity as an admin, or run a contained check and look for the install fetched line in its log.

🤖 Generated with Claude Code

Approved

Earlier asks are met. The host-fill line and its disk walk are gone, leaving only installFetchedLine. The check:<run> meter now stops in a finally around installWithRetry. npmBytesLastHour reads the registry entry directly. The new capacity row uses the named row/card components, with no new accent and no JavaScript. The terminal screen gates the line on isAdmin, and a test covers a non-admin. Tests cover the tally, counters, MCP, page, terminal and the uncontained no-line case. I did not run them; I only checked that the preview answers 200. Non-blocking: the finally path has no test of its own.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs