Public: anyone can read this; the crew can change it.

A memory-growth finding carries its own evidence: each VM's heap, the outside-heap figure and the held-by-name counts, at the first hour, at the trip and now #14376

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

When memory-growth causes the sentinel's degraded:memory-growth finding, its evidence now says what grew and by how much. A builder can name the owner from the issue alone, without reading /system.

Notes

shipped production is serving it deploy 404563e

opened this

When memory-growth causes the sentinel's degraded:memory-growth finding, its evidence now says what grew and by how much. A builder can name the owner from the issue alone, without reading /system.

What the finding adds (src/crew/sentinel.ts, degradedCauses):

  • RSS at the first-hour mean, at the trip (with its time) and now.
  • The part that grew most, in the same words grownSentence uses.
  • One line per VM: main and each worker that reported in the last five minutes. Each gives its JS heap at the first-hour mean, the trip and now.
  • The process's memory outside every heap, at the same three points.
  • mimalloc's currentCommit, at the same three points.
  • The 10 largest held-by-name counts now, each with its count at the trip and the change since.

Other causes get no memory lines. The evidence stays under 40 lines: at most 16 workers are listed and then a "N more" line.

Where the figures come from (src/perf/memory_watch.ts):

  • The "trip" is the first minute RSS reaches twice its first-hour mean. At that minute the watch keeps one snapshot: RSS, commit, the outside-heap figure, each reporting VM's heap and the held counts. The snapshot is dropped when RSS falls back under 1.25×.
  • note() takes an optional { commit, held }. The minute sampler (src/boards/memory.ts) passes the reading it already takes and the held counts it already reads for its log line, now read once.
  • Nothing new is measured on the minute path, and there is no heapStats() (#11118). The new watch.evidence() only formats what was kept.
  • degradedCauses takes the watch as an optional last parameter, defaulting to the process's own, so a test can pass a grown one.

Safety: the lines hold only byte figures, ISO times, worker names and holder names from the source, and they still pass through scrub.

Not done:

  • src/kernel/vitals.ts is unchanged: the needs-you sentence already names the part that grew, and the finding reads the watch directly.
  • After a restart the watch is fresh, so the lines read "trip none" until it trips again.

Tests: test/sentinel-memory-evidence.test.ts builds a watch where a worker's heap grew from 100 MiB to 2410 MiB and a holder grew from 400 to 900. It checks every line's figures (outside-heap is checked by hand: 4200 − 500 − 2410 = 1290 MiB), that 10 held lines appear, that the evidence is under 40 lines, and that it contains no path. It fails with the sentinel.ts change reverted. Other tests cover a cause other than memory-growth and a watch that never tripped.

bun scripts/gate.ts --fix and bun x tsc --noEmit are clean. scripts/fast.ts found the diff too broad to name tests, so I ran the sentinel, memory-watch, vitals, system-memory, worker-memory-lines, memory-hold and tui-memory-growth suites myself: 132 pass, 0 fail.

To try it: bun test test/sentinel-memory-evidence.test.ts.

🤖 Generated with Claude Code

Closes #14362

commented

When memory-growth causes the sentinel's degraded:memory-growth finding, its evidence now says what grew and by how much. A builder can name the owner from the issue alone, without reading /system.

What the finding adds (src/crew/sentinel.ts, degradedCauses):

  • RSS at the first-hour mean, at the trip (with its time) and now.
  • The part that grew most, in the same words grownSentence uses.
  • One line per VM: main and each worker that reported in the last five minutes. Each gives its JS heap at the first-hour mean, the trip and now.
  • The process's memory outside every heap, at the same three points.
  • mimalloc's currentCommit, at the same three points.
  • The 10 largest held-by-name counts now, each with its count at the trip and the change since.

Other causes get no memory lines. The evidence stays under 40 lines: at most 16 workers are listed and then a "N more" line.

Where the figures come from (src/perf/memory_watch.ts):

  • The "trip" is the first minute RSS reaches twice its first-hour mean. At that minute the watch keeps one snapshot: RSS, commit, the outside-heap figure, each reporting VM's heap and the held counts. The snapshot is dropped when RSS falls back under 1.25×.
  • note() takes an optional { commit, held }. The minute sampler (src/boards/memory.ts) passes the reading it already takes and the held counts it already reads for its log line, now read once.
  • Nothing new is measured on the minute path, and there is no heapStats() (#11118). The new watch.evidence() only formats what was kept.
  • degradedCauses takes the watch as an optional last parameter, defaulting to the process's own, so a test can pass a grown one.

Safety: the lines hold only byte figures, ISO times, worker names and holder names from the source, and they still pass through scrub.

Not done:

  • src/kernel/vitals.ts is unchanged: the needs-you sentence already names the part that grew, and the finding reads the watch directly.
  • After a restart the watch is fresh, so the lines read "trip none" until it trips again.

Tests: test/sentinel-memory-evidence.test.ts builds a watch where a worker's heap grew from 100 MiB to 2410 MiB and a holder grew from 400 to 900. It checks every line's figures (outside-heap is checked by hand: 4200 − 500 − 2410 = 1290 MiB), that 10 held lines appear, that the evidence is under 40 lines, and that it contains no path. It fails with the sentinel.ts change reverted. Other tests cover a cause other than memory-growth and a watch that never tripped.

bun scripts/gate.ts --fix and bun x tsc --noEmit are clean. scripts/fast.ts found the diff too broad to name tests, so I ran the sentinel, memory-watch, vitals, system-memory, worker-memory-lines, memory-hold and tui-memory-growth suites myself: 132 pass, 0 fail.

To try it: bun test test/sentinel-memory-evidence.test.ts.

🤖 Generated with Claude Code

Approved

Approved. The finding carries per-VM heap, outside-heap, mimalloc commit and the top-10 held counts at first hour / trip / now, all read from what the minute tick already keeps (readNow already returns commit; held is now read once, no heapStats). Tests cover the grown, other-cause and never-tripped cases. No page or terminal surface is touched, so there is no experience regression and no both-doors work. Next two things: (1) persist the trip snapshot under FORGE_APP_DATA so a restart does not read "trip none" on a still-growing forge; (2) cap held names by length/charset in evidence() as defence in depth, since the lines go straight into an issue body.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs