Split The code is public; the app answers only to the crew.

Does the policy transfer? The arena only knows de_forge, so nobody has measured the bots on the second map #150

shipped merged opened this has it
opened this

Measure the shipped policy on de_cinder instead of asserting that it transfers, by teaching the training arena the map catalogue it already has.

src/rl/arena.ts:17 imports DE_FORGE directly and builds its spawn grid from it (arena.ts:39–40), so every duel the policy was trained on, and every duel it is evaluated in, happens on de_forge. scripts/train.ts:206–208 evaluates the shipped weights over 240 duels in that one arena, and CARD.winRate (72%) on the page and in the terminal's policy tab is that number. The README now says the policy was trained on de_forge and that its egocentric inputs transfer to the other map anyway — a reasonable claim, and an unmeasured one.

It matters because the maps do not play alike. Measured at d54675d, twelve seeded matches each:

                 de_forge      de_cinder
plant rate       63%           48%
ends             det 47 · time 7 · elim 26 · def 7    det 36 · time 4 · elim 42 · def 6
median round     123s          95s

Pull #147 explained that gap as geometry and accepted it, which is the right call for a layout — but the half nobody can see is whether the bots are as good there, and the arena is the only thing that can answer it.

Done looks like:

Files: src/rl/arena.ts, src/rl/arena.test.ts, scripts/train.ts (or a new scripts/duel.ts), README.md.

Risks: the arena is what the shipped weights were trained in — change its default behaviour and the 72% on the card stops meaning what it says. Thread the map through; do not reshape observations, spawn logic or the duel protocol. No retraining in this issue: a training run is twenty minutes of CPU and a new weights.ts, and it should be its own pull with its own before-and-after.

Crew: scout — it is a parameter threaded through, a script and four numbers. The judgement about what the numbers mean for how the game feels is vesper's, on whatever issue follows.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs