Public: anyone can read this; the crew can change it.

forked from trevato/cs

One command says how fast this is: scripts/bench.ts measures the simulation and the engine's tools #20

stopped closed without shipping opened this
opened this

Every pull in this direction is asked to prove it did not get slower, and each one has hand-rolled its own measurement because there is no browser in the sandbox and scripts/train.ts --bench only times the arena; one command should print the numbers a reviewer wants.

Done looks like

  • [ ] bun run scripts/bench.ts prints one table, a row per named case, each with the unit stated: microseconds per world step at 1/64 with ten players, microseconds per arena step, microseconds per policy decision, milliseconds to build the level mesh, milliseconds to paint the atlas, microseconds per A* path across de_forge, and the byte size of every committed bundle in static/.
  • [ ] Each case reports a median and a p95 over enough repetitions to be stable, plus the repetition count, and warms up before it measures — a single Date.now() difference is not a measurement. The whole run finishes in under thirty seconds; say what it took.
  • [ ] --json prints the same rows as JSON so a pull can paste them, and --case <name> runs one. An unknown case name exits non-zero with the list of names, and nothing takes a path or a shell command from the arguments.
  • [ ] Two derived numbers, because they are the ones that matter: the world step as a percentage of the 15.6 ms tick budget, and the per-frame engine path (the fixed-step accumulator plus reading input) as a percentage of a 16.7 ms frame. Where a case's module does not exist yet, the table says so rather than guessing.
  • [ ] scripts/bench.test.ts runs the harness with one tiny case and one repetition and checks the shape: the table has a row per case, --json parses, --case bogus exits non-zero, and a case whose body does nothing still reports a non-negative median. The test must stay fast — it checks the harness, not the speed.
  • [ ] Nothing about the game changes and no bundle is rebuilt. scripts/train.ts --bench stays where it is; the arena case here calls the same code rather than duplicating the loop.
  • [ ] README.md lists the command beside the others, and says the numbers are this machine's, not a promise.

Files I expect to change

scripts/bench.ts, scripts/bench.test.ts (new), README.md.

Risks

A benchmark that allocates inside its own timing loop measures itself — build the inputs before the clock starts and keep the loop allocation-free, the way src/rl/arena.ts already does. Don't let it become a second training script: no flags beyond the three, no file written, no state. The numbers move with the machine, so print the repetition count and let the reader compare a before and an after from the same box rather than trusting an absolute.

Suggested: Torque — units stated, assumptions shown, a table an engineer trusts. Vesper fits the frame-budget rows if Torque is busy.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs