Public: anyone can read this; the crew can change it.

The check's full gate prints its wall time, shard count, peak memory and ten slowest files, and names any file over 45 s #2067

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

I added the summary block, but I couldn't paste it from this pull's own check log as the issue asks. The pull opens when this run ends and only then does its check run, so the block will be at the end of the first check log after this.

shipped production is serving it deploy eb43263

opened this

I added the summary block, but I couldn't paste it from this pull's own check log as the issue asks. The pull opens when this run ends and only then does its check run, so the block will be at the end of the first check log after this.

The full timed gate (--gate on the full tier, unsliced, which is check.run) now ends its log with one block: how many shards ran and why, the wall time, the cell's peak memory, the ten slowest files, and a line for each file over 45 s. The new test file passes, and so does test/shards.test.ts (49 tests across the two), along with tsc --noEmit and scripts/gate.ts. I did not run the full gate here, so this printing step only gets exercised by the check on the cell.

── gate summary
test-shards: shards: 8 = least of 16 CPUs, SHARDS_MAX 8, 412 files; 16 CPUs, cgroup memory 8.00 GiB (memory.max), not capped by memory
test-shards: wall: 312.4s
test-shards: peak memory: 6.50 GiB of 8.00 GiB (memory.peak)
test-shards: slowest 10 of 412 files:
   197.0s test/studio.test.ts
   …
test-shards: over 45 s: test/studio.test.ts (197.0s)

These numbers are made up to show the layout.

  • scripts/gate-summary.ts (new):

- summaryLines() takes plain data and returns lines. - cgroupMemory(read) reads the limit (memory.max, else memory.limit_in_bytes) and the peak (memory.peak, else memory.max_usage_in_bytes) through a reader function you pass in. - A missing file, max, or cgroup v1's "unlimited" value counts as no number. The line then reads peak memory: unknown here.

  • scripts/test-shards.ts:

- Prints the block last, only on the timed path, red or green. - The wall time is the same figure the tally line uses. - The exit code doesn't change: a file over 45 s is only a warning. - The CPU count and FORGE_CHECK_SHARDS are now read before that variable is cleared, so the block can name them.

  • test/gate-summary.test.ts (new, added to GATE) covers no cgroup file, cgroup v2 and v1 files, a file over 45 s (exactly 45 s is not flagged), fewer than ten files, and the FORGE_CHECK_SHARDS reason. One expectation of mine was wrong while writing it: 1.25 s prints as 1.3s, so the test now uses 1.24 s.

What to look at: WORKER_MEMORY_BYTES is not in scripts/shards.ts on main; it comes with the open #2064. On main the shard count is the smallest of the CPU count, SHARDS_MAX (8) and the number of files, so memory never lowers it. The block reports the cgroup limit next to the count and says "not capped by memory". When #2064 lands, why() in gate-summary.ts should name its memory cap. I left scripts/shards.ts unchanged.

To try it: run bun test test/gate-summary.test.ts, then read the end of this pull's check log. If the cell runs fewer shards than it has CPUs, that is SHARDS_MAX, not memory: the check cell is set to 16 CPUs, so expect 8 shards.

🤖 Generated with Claude Code

Closes #2065

Approved — before the latest push

Does what #2065 asks: the timed full gate ends its log with shard count and why, wall time, cgroup peak (or "unknown here"), the ten slowest files, and one warning line per file over 45 s. It never changes the exit code. It is CI tooling with no page, input or write surface, so the security, terminal and both-doors items do not apply. The new module is small, takes plain data and a reader function, and has tests for each branch. It is wired into the gate list. Two things I would do next: (1) in scripts/gate-summary.ts, bytesIn and first() both check V1_UNLIMITED, so drop one of the two. (2) When fail-fast skips shards, say so in the "slowest N of M files" line, because M then counts files that ran rather than files named.

commented

The full gate on land was red and named no test; the pull is open again.

commented

I made both of juniper's changes and pushed again. I could not find out why the land gate went red: the forge keeps that gate's lines only under its data directory, and nothing I can reach shows them.

Since the timed full gate (--gate on the full tier, unsliced, which is check.run) ends its log with one block:

  • the shard count and the limits that set it,
  • the wall time,
  • the cell's peak memory from the cgroup (memory.peak, else memory.max_usage_in_bytes, else "peak memory: unknown here"),
  • the ten slowest files,
  • one line per file over 45 s, each starting test-shards: over 45 s:.

The block prints whether the gate is red or green and never changes the exit code. The summary is built in scripts/gate-summary.ts from plain data; test/gate-summary.test.ts is in GATE.

This push (from juniper's review):

  • bytesIn alone now rejects cgroup v1's "unlimited" value; the second check in first() is gone.
  • When fail-fast skips files, the heading says so, e.g. slowest 1 of 1 files (39 of 40 skipped after a red file):. A new test covers it.

The red land gate:

  • What I checked: this pull's own check was green, but it ran the fast tier in 4 shards, not the timed path, so it never printed the block. I then ran the timed path locally, with a temporary switch over three files that I have since removed. It ran to the end, exited 0 and printed the block.
  • What I can't tell: "named no test" means the gate went red without any (fail) line. That could be a shard killed partway (for example out of memory), a timeout, or the gate stopped by a forge roll. From here I can't tell which.
  • What to look at: the next full gate on land now prints the block whether it is red or green, so a red run will show its shard count, peak memory and slowest files.

Not done:

  • WORKER_MEMORY_BYTES: it is not on main; it arrives with #2064. On main the shard count is the smallest of the CPU count, SHARDS_MAX (8) and the number of files, so memory never lowers it. The block reports the cgroup limit and says "not capped by memory".
  • The vxrail block itself: I still can't paste it here, because the full gate only runs on land and its log isn't readable through the API. It will be at the end of the next land gate's log.

To try it: bun test test/gate-summary.test.ts.

🤖 Generated with Claude Code

commented

The timed full gate (--gate on the full tier, unsliced, which is check.run) now ends its log with one block: how many shards ran and why, the wall time, the cell's peak memory, the ten slowest files, and one test-shards: over 45 s: line for each file over 45 s. The block prints whether the gate is red or green and never changes the exit code.

The last check failed because #2064 landed on main and changed the same shardCount call in scripts/test-shards.ts. This push merges main and keeps #2064's memory bound unchanged: the CPUs, the cgroup memory divided by WORKER_MEMORY_BYTES, and SHARDS_MAX (now 32). The cgroup is read once, and both shardCount and the summary use that reading. Merging also let the shard line name the memory bound, which #2065 asked for, and say so when memory held the gate under one shard per CPU:

test-shards: shards: 8 = least of 16 CPUs; 8 by memory, 4.00 GiB (cgroup memory.max) at 512 MiB each; SHARDS_MAX 32; 12 files — fewer than the CPUs because of memory

On the check cell's 16 CPUs and 8 GiB, memory allows 16 workers, so that clause should not appear there.

  • scripts/gate-summary.ts: summaryLines() turns plain data into the lines. cgroupMemory(read) reads the limit and the peak (memory.peak, else memory.max_usage_in_bytes). When neither file is there, the line reads peak memory: unknown here. Following juniper's review, cgroup v1's "unlimited" value is rejected in one place, and when fail-fast skips files the heading says so (slowest 1 of 1 files (39 of 40 skipped after a red file):).
  • test/gate-summary.test.ts (in GATE): the expected lines are worked out by hand. It covers no cgroup file, cgroup v2 and v1, junk values, a file over 45 s (45 s itself is not flagged), fewer than ten files, skipped files, the memory-held case and FORGE_CHECK_SHARDS.

Earlier, the full gate on land went red without naming a test. I could not read that gate's lines, so its cause is unknown. Running the timed path locally over a few files, with a switch I removed afterwards, exits 0 and prints the block. From now on a red land gate also prints the block.

test/gate-summary.test.ts, test/shards.test.ts (52 tests), tsc --noEmit and scripts/gate.ts pass. The cell's own numbers will be at the end of this pull's land-gate log; nothing I can reach shows that log, so it is not pasted here.

To try it, run bun test test/gate-summary.test.ts.

🤖 Generated with Claude Code

Approved

Build-tooling only: no page, no request path, no new dependency, and nothing here touches identity, bodies or the terminal. gate-summary.ts is a small single-concept module (plain data in, lines out, cgroup read through an injected reader). Missing, "max" and v1-unlimited values come out as "unknown here" rather than an invented number. The summary never changes the exit code. Tests are hand-computed and cover the edge cases (45 s exactly is not flagged, fewer than ten files, fail-fast skips, v1/v2, junk). Two things I would do next: (1) test the wiring in test-shards.ts, because the if (timed) call site is covered only by a manual run and times relies on each shard holding exactly one file (result.files[0]); assert that or add a one-file-per-shard test. (2) Once the first land gate prints the block, paste it in the pull or an issue and tune SLOW_SECONDS from real numbers.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs