The shipped bots are re-measured, not just remembered: a guard that plays the weights on unseen maps #226

src/rl/guard.ts is a standing measurement of the weights the repository actually ships: it plays src/client/weights.ts greedily, through the trainer's own harness, on a named sample — de_forge plus two maps from the held-out stream, forty duels each, seeds derived from the word guard and printed in the report — against the scripted bots the policy replaced on exactly the same duels, and fails if the share won falls under a 45% floor. Measured on those seeds today: 68% shipped (81/27/12) against 46% scripted; five other base seeds put the policy between 63% and 74%, so one standard error is 4.3 points and the floor sits five of them below the measurement and about where the scripted bots sit unaided — falling through it means the shipped policy is no better than the bots it replaced, which is a collapse and not a wobble. The reasoning is in the FLOOR docstring. It costs about 0.6s (bun test src/rl goes 1.89s → 2.5s; the whole suite ~17.5s → ~18.5s), and it retrains nothing, writes nothing and does not touch CARD.

Look at verdict in src/rl/guard.ts and the two tests that exercise its message with synthetic tallies: on failure it prints what it measured per map, what CARD claims beside it, and whether the scripted baseline fell with it — a baseline that held means the weights stopped working and the card describes a game that moved on; a baseline that fell means the duels got harder for everyone and the game is where to look. It proposes nothing, because retraining and reverting are both a person's call. src/rl/evaluate.ts is the one change outside the guard: the duel harness existed in scripts/train.ts and again in learned.test.ts and was about to exist a third time, so it moved out of the trainer unchanged and all three import it — same seeds, same numbers, and "won 68%" now means one thing. What I did not do: nothing pins the layouts (that is #159's digest), nothing re-measures CARD itself, and there is no page or terminal view of the guard — it is a test, run on every push.

🤖 Generated with Claude Code

Closes #193

main is merged and pushed; bun run test is green (999 pass, ~16s).

src/rl/guard.ts is a standing measurement of the weights the repository ships: it plays them greedily, through the trainer's own evaluate, on a named sample — de_forge plus two maps from the held-out stream, forty duels each, seeds derived from the word guard and printed in the report — against the scripted bots the policy replaced on exactly the same duels, and fails if the share won falls under a 45% floor. Measured on those seeds: 68% shipped (81/27/12) against 46% scripted, unchanged by the merge. Five other base seeds put the policy between 63% and 74%, so one standard error is 4.3 points and the floor sits five of them below the measurement and about where the scripted bots sit unaided — a collapse, not a wobble. The reasoning is in the FLOOR docstring. It costs about 0.6s, retrains nothing, writes nothing and does not touch CARD.

The merge conflict was scripts/train.ts, where both sides edited the imports and the block around evaluate; the result keeps both meanings — main's --policy flag, cardModule and SHAPE from packed.ts, and this branch's move of evaluate/greedyDriver into src/rl/evaluate.ts (one harness for the trainer, learned.test.ts and the guard, so "won 68%" means one thing). The other thing #223 changed is that policy() now throws until a policy is handed in, so the guard and the tests beside it take theirs from useShipped() — the same weights.json a browser downloads, read off disk — rather than unpacking a module constant; prose in the guard and the README follows the file that moved.

Worth looking at: verdict in src/rl/guard.ts and the two tests that exercise its message with synthetic tallies. On failure it prints the per-map measurement, CARD's claim beside it, and whether the scripted baseline fell with it — a baseline that held means the weights stopped working, a baseline that fell means the duels got harder for everyone — and it proposes nothing, because retraining and reverting are both a person's call. One thing I did not fix because it is main's and pre-existing: since #223, src/client/packed.ts evaluates SHAPE at module level inside the observe → world → bots → packed cycle, so entering the graph at src/rl/ (bun test src/rl/, on main as well as here) throws Cannot access 'OBS_SIZE' before initialization; the full bun test is unaffected, and nothing in this pull tells anyone to run a single file that way.

🤖 Generated with Claude Code