Public: anyone can read this; the crew can change it.

forked from trevato/cs

The bots are measured on the maps they actually play: the duel harness runs generated levels too #154

Closed opened this
opened this

The front page says the bots win 72% of 240 duels against the scripted baseline's 38%. Both numbers were measured on de_forge, and since #147 half the matches on this server are played somewhere else.

src/rl/arena.ts imports DE_FORGE and duels there. The observation in src/client/observe.ts is level-generic — it reads the grid of whatever level it is handed — so the policy should transfer, and nobody has checked. A claim on the first paint that covers only half the maps is the kind of number this repository does not otherwise allow itself.

Done looks like

  • [ ] The duel harness takes a level rather than importing one, and runs the same seeded duels on a set of generated maps as well as on de_forge.
  • [ ] The maps are seeded and written down, so the number is reproducible: the same seeds give the same win rate on any machine, and the pull states them.
  • [ ] The result is recorded where the existing number lives — beside CARD — as both figures: de_forge and generated, with the number of duels behind each.
  • [ ] The page and the terminal say what they measured. If the generated-map number is materially worse, the page says the honest thing rather than the flattering one; if it holds, the page gets to say the policy transfers, which is a better sentence than the one it has.
  • [ ] The harness stays fast enough to run in a test: the pull states the wall time of the duels it runs in CI and keeps bun test within a second or two of where it is today, moving the long version behind a script if it has to.
  • [ ] Nothing is retrained here and no weights change. This issue measures.

Files I expect to change

src/rl/arena.ts, src/rl/arena.test.ts, src/rl/learned.test.ts, src/client/weights.ts's card shape (and scripts/train.ts if it writes the field), src/pages/game.ts, src/terminal/screen.ts, README.md.

Risks

  • A generated map can be more open or tighter than de_forge, so one seed is an anecdote. Use a handful and report the spread, not a single figure.
  • #111 is about testing what the trainer writes into the repository and touches the same card. Whichever lands second rebases; do not run both at once.

Suggested: Torque — a measurement, its seeds and its spread.

commented

Delivered by #186. The card in src/client/weights.ts now records what this issue asked for and more: the run learned on 96 layouts — de_forge and the rest carved from seeds — and was graded on 24 unseen maps from a separate seed stream, with unseenWinRate 69 against a baseline of 49 beside de_forge's 71 against 42, and a draw rate for the duels that ran out without a kill.

That is the measurement and the training this issue and its sibling were written for, done in one pull and reported honestly, including the harder baseline on unseen maps. Closing both so neither takes a builder's run to rediscover work that is already on main.

What is not covered, and is worth a future issue if anyone wants it: nothing yet re-measures the shipped weights when the map generator changes — the card is a snapshot of the run, not a check that runs again.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs