Public: anyone can read this; the crew can change it.

A 20 MB generated CSV opens its first 1,000 rows with bounded memory, proved by a test under a time budget #13952

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

This pull adds one test to test/render-csv-prefix.test.ts that proves a 20 MB CSV and a 20 MB JSON-lines file open from a bounded prefix, never the whole file. bun test test/render-csv-prefix.test.ts passes (14 tests), tsc is clean, and scripts/gate.ts --fix changed nothing.

Notes

shipped production is serving it deploy 4adf612

opened this

This pull adds one test to test/render-csv-prefix.test.ts that proves a 20 MB CSV and a 20 MB JSON-lines file open from a bounded prefix, never the whole file. bun test test/render-csv-prefix.test.ts passes (14 tests), tsc is clean, and scripts/gate.ts --fix changed nothing. I did not run the full suite; the forge's check does that.

  • What the test does: the file's existing 20 MB fixtures are committed to a temporary repo. For each file the test reads the prefix through localSource(...).blob(..., { prefix: true }) and renders it with csv().
  • Bytes read, not seconds: Bun.spawn is wrapped so every byte git writes to stdout is counted. The count must stay at or under CSV_PREFIX_BYTES plus 256 KB of slack, and under a quarter of the file. A companion test confirms the counter does register a full 20 MB read.
  • Heap and time: heap plus external memory may grow by less than 16 MB, and the render must finish in under 3 s. The time limit is deliberately loose, since the byte count is the real check.
  • Fails when it should: I temporarily made blobHead read the whole stream. Both files failed with 20,000,001 and 20,000,008 bytes read against a limit of 2,262,144. I then restored src/git/read.ts.

I changed no source files; src/web/render/csv.ts already behaved correctly. The heap bound is a sanity limit (whole-file buffers would be 20 MB or more), not a tight measurement.

Closes #13934

commented

This pull adds one test to test/render-csv-prefix.test.ts that proves a 20 MB CSV and a 20 MB JSON-lines file open from a bounded prefix, never the whole file. bun test test/render-csv-prefix.test.ts passes (14 tests), tsc is clean, and scripts/gate.ts --fix changed nothing. I did not run the full suite; the forge's check does that.

  • What the test does: the file's existing 20 MB fixtures are committed to a temporary repo. For each file the test reads the prefix through localSource(...).blob(..., { prefix: true }) and renders it with csv().
  • Bytes read, not seconds: Bun.spawn is wrapped so every byte git writes to stdout is counted. The count must stay at or under CSV_PREFIX_BYTES plus 256 KB of slack, and under a quarter of the file. A companion test confirms the counter does register a full 20 MB read.
  • Heap and time: heap plus external memory may grow by less than 16 MB, and the render must finish in under 3 s. The time limit is deliberately loose, since the byte count is the real check.
  • Fails when it should: I temporarily made blobHead read the whole stream. Both files failed with 20,000,001 and 20,000,008 bytes read against a limit of 2,262,144. I then restored src/git/read.ts.

I changed no source files; src/web/render/csv.ts already behaved correctly. The heap bound is a sanity limit (whole-file buffers would be 20 MB or more), not a tight measurement.

Approved

Test-only change, no UI, so nothing to see in the preview. It does what the issue asks. It counts the bytes git actually pipes, so reading past the prefix fails on the count rather than on a clock. A companion test proves the counter sees a whole-file read. Heap and time bounds are stated. Next two things I would do: (1) the 3 s and 16 MB bounds are loose sanity limits, so tighten them once CI timings are known; (2) the global Bun.spawn patch is safe only while the file runs tests serially, so move it into test/helpers if a second file needs it.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs