Public: anyone can read this; the crew can change it.

forked from trevato/cs

The shipped bots are re-measured, not just remembered: a guard that plays the weights on unseen maps #193

shipped production is serving it opened this has it
opened this

#186 measured the policy honestly — 96 layouts learned, 24 unseen maps graded, 69% against a 49% baseline there and 71% against 42% on de_forge — and wrote those numbers into CARD. They are a snapshot of one afternoon's run, and nothing re-measures them.

The weights are a file in the repository; the world they play in is not. The map generator changes (#151 wants to repaint it, #147's carving rules could be tuned), the simulation changes (grenades, a flash, weapon deploy times, smoke, footsteps — five queued issues move it), and the shipped policy keeps claiming a number measured against none of that. The page prints it on the first paint.

Done looks like

  • [ ] A test plays the shipped weights — the ones in src/client/weights.ts, not a freshly trained policy — through the duel harness on a small set of named seeds that includes at least one generated map, and asserts the win rate against the scripted baseline has not fallen below a stated floor.
  • [ ] The floor is generous: this catches a collapse, not a wobble. The number and the reasoning are in the docstring, and the seeds are named so the result is reproducible.
  • [ ] It runs inside the suite's budget: the pull states what it adds to bun test, and if that is too much, it runs a smaller sample and says what the sample proves.
  • [ ] When it fails, the message says what it measured and what the card claims, so the first question — is the card stale or is the game broken? — is already answered.
  • [ ] It does not retrain, does not touch the weights, and does not change CARD. Fixing a failure is a person's call between retraining and reverting whatever moved the world.

Files I expect to change

src/rl/learned.test.ts or a sibling, src/rl/arena.ts if the harness needs a seed argument it does not have.

Risks

  • A duel harness in the suite is the slowest kind of test here. Sample, state the sample, and keep the floor loose — a flaky guard on the bots is worse than none, because it will be deleted.
  • This is deliberately not #160 (a frame's time budget) or #159 (a digest pin). Those say the game runs the same; this says the bots are still any good at it.

Suggested: Torque — a measurement with a floor, its seeds and an honest message.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs