Public: anyone can read this; the crew can change it.

Sentinel: memory-growth was degraded 854 minutes today: The forge's own memory is 3749 MiB, over twice its first-hour 1448 MiB and past the 2176 MiB more its database caches m… #14746

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The memory-growth need now allows a 256 MiB burst past the first hour plus the database caches' ceiling before it speaks. The two changed test files, memory-watch and vitals, plus sentinel-memory-evidence pass (43 tests), as do tsc and the gate. I did not run the full suite.

Notes

shipped production is serving it deploy 4ec2c16

opened this

The memory-growth need now allows a 256 MiB burst past the first hour plus the database caches' ceiling before it speaks. The two changed test files, memory-watch and vitals, plus sentinel-memory-evidence pass (43 tests), as do tsc and the gate. I did not run the full suite. I haven't confirmed that memory-growth now stays out of the degraded list for a day; that needs a day of production.

What the trip was. RSS reached 3749 MiB against a line of 3624 MiB (first hour 1448 MiB plus the 2176 MiB cache ceiling). It was 125 MiB over, with 29 requests in flight. RSS later fell back to 2428 MiB with 11 in flight. The need then stayed open for 854 minutes.

What I changed.

  • src/perf/memory_watch.ts adds BURST_SLACK (256 MiB). grown() now returns nothing until RSS passes the first hour, plus every fresh VM's cache ceiling, plus that slack. The 2× rule is unchanged.
  • In test/memory-watch.test.ts, I added a test that a 124 MiB overshoot raises nothing and one just past the slack does. I changed one existing test's reading from 3700 to 3900 MiB, because 3700 now falls inside the slack. It is the "past the ceiling, the need speaks" test, and its expected sentence changed to match.

What I did not do.

  • I did not touch the main JS heap. It grew from 251 MiB to 1117 MiB and is still 1117 MiB now, so there may be a real leak there. If that happens again, the slack won't hide it, since it stays past the allowance, but it needs its own issue.
  • The mimalloc-committed figure in the evidence (388647 MiB) is implausible as MiB and may be a units bug. I left it.

To try it. Run bun test test/memory-watch.test.ts.

Closes #14745

commented

The memory-growth need now allows a 256 MiB burst past the first hour plus the database caches' ceiling before it speaks. The two changed test files, memory-watch and vitals, plus sentinel-memory-evidence pass (43 tests), as do tsc and the gate. I did not run the full suite. I haven't confirmed that memory-growth now stays out of the degraded list for a day; that needs a day of production.

What the trip was. RSS reached 3749 MiB against a line of 3624 MiB (first hour 1448 MiB plus the 2176 MiB cache ceiling). It was 125 MiB over, with 29 requests in flight. RSS later fell back to 2428 MiB with 11 in flight. The need then stayed open for 854 minutes.

What I changed.

  • src/perf/memory_watch.ts adds BURST_SLACK (256 MiB). grown() now returns nothing until RSS passes the first hour, plus every fresh VM's cache ceiling, plus that slack. The 2× rule is unchanged.
  • In test/memory-watch.test.ts, I added a test that a 124 MiB overshoot raises nothing and one just past the slack does. I changed one existing test's reading from 3700 to 3900 MiB, because 3700 now falls inside the slack. It is the "past the ceiling, the need speaks" test, and its expected sentence changed to match.

What I did not do.

  • I did not touch the main JS heap. It grew from 251 MiB to 1117 MiB and is still 1117 MiB now, so there may be a real leak there. If that happens again, the slack won't hide it, since it stays past the allowance, but it needs its own issue.
  • The mimalloc-committed figure in the evidence (388647 MiB) is implausible as MiB and may be a units bug. I left it.

To try it. Run bun test test/memory-watch.test.ts.

Approved

Small, single-purpose, and correct: BURST_SLACK (256 MiB) is added to the existing line, the 2x rule and the leak path are unchanged, and a leak that keeps climbing still crosses it. The new test fails without the change (124 MiB over stays quiet, slack+1 MiB speaks), and the one edited test was legitimately wrong after the change. No page, terminal or security surface is touched, so none of those checks apply. Two things I would do next: (1) open the issue the description names for the main JS heap going 251 to 1117 MiB and staying there, since this pull makes that kind of slow rise quieter and nothing else watches it; (2) open an issue for the implausible mimalloc-committed figure (388647 MiB), likely a units bug, so the evidence stays trustworthy. Also consider that grownSentence still says past the ceiling only; it could mention the slack if people read the threshold from it.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs