Split The code is public; the app answers only to the crew.

Does the policy transfer? The arena only knows de_forge, so nobody has measured the bots on the second map #165

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/cs:scout/150-does-the-policy-transfer-the-arena-only → main 2 commits fork 2 threads · 2 unresolved

  • Merged. scout/150-does-the-policy-transfer-the-arena-only was merged into main as 62fc119.

Review

Sign in to review this change.
8 files changed +190 −28
README.md +34 −9
@@ -523,15 +523,40 @@ the room. The fight is a reflex, and that is a policy learned with PPO.
523523 plan takes a `MapDef` and asks it for the sites, the spawns and a path; there
524524 is no per-map branch anywhere in `plan.ts`, `bots.ts` or `findPath`, so
525525 `de_cinder` is played by exactly the code that plays `de_forge`. The policy is
526 the half that was **trained on `de_forge`** — `src/rl/arena.ts` duels on that
527 grid and nowhere else. What it learned is not a map, though: its 32 inputs are
528 distances, angles and clocks relative to the bot itself (see *What it sees*),
529 so nothing it holds is in map coordinates and it transfers rather than breaks.
530 What does change is the *shape* of the fights those inputs describe — a
531 courtyard with four mouths hands the policy more duels at once than a corridor
532 does, and a site the terrorists reach first is a site they hold rather than
533 take. That shows up in the numbers rather than in the code, and the two tables
534 in *Measuring the round* are what it looks like.
526 the half that was **trained on `de_forge`** — `scripts/train.ts` still duels
527 there and nowhere else, and that is what `CARD` on the page and in the
528 terminal's policy tab measures. What it learned is not a map, though: its 32
529 inputs are distances, angles and clocks relative to the bot itself (see *What
530 it sees*), so nothing it holds is in map coordinates — and rather than take
531 that on faith, `src/rl/arena.ts` takes any map from the catalogue (defaulting
532 to `de_forge`, so `CARD`'s number did not move) and `scripts/duel.ts` plays
533 the shipped weights, and the scripted baseline they are measured against, over
534 the same 240 fixed duels on whichever one you name:
535
536 ```sh
537 bun run scripts/duel.ts # de_forge, what CARD was measured on
538 bun run scripts/duel.ts --map de_cinder # the other map, same seeds, same duels
539 ```
540
541 Measured at `013620e`:
542
543 | | `de_forge` | `de_cinder` |
544 | --- | --- | --- |
545 | shipped policy won | **73%** | **75%** |
546 | scripted baseline won | 39% | 43% |
547
548 The policy is not worse on `de_cinder` — if anything a couple of points
549 better, and the gap over its own baseline (34 points on `de_forge`, 32 on
550 `de_cinder`) holds either way, so the egocentric inputs are transferring the
551 way the design intended. (`de_forge`'s numbers here are close to but not
552 exactly `CARD`'s 72%/42% — the weights have not changed since they were
553 written, but the arena they are dropped into has, in the commits since; see
554 `scripts/duel.ts` for a rerun against today's code.) What *does* change with
555 the map is the shape of the fights those inputs describe — a courtyard with
556 four mouths hands the policy more duels at once than a corridor does, and a
557 site the terrorists reach first is a site they hold rather than take — and
558 that is a fact about the level, not the bot, which is what the two tables in
559 *Measuring the round* are about.
535560
536561 The plan is one site a side a round, rolled from the match's seed and the round
537562 number rather than by each bot for itself, so the five terrorists arrive
scripts/args.ts added +5
@@ -0,0 +1,5 @@
1 /** `--name value` off argv, or null if the flag is missing or has nothing after it. */
2 export const flagValue = (argv: string[], name: string): string | null => {
3 const at = argv.indexOf(`--${name}`);
4 return at < 0 ? null : (argv[at + 1] ?? null);
5 };
scripts/duel.test.ts added +37
@@ -0,0 +1,37 @@
1 import { expect, test } from "bun:test";
2 import { parseArgs } from "./duel.ts";
3
4 const CARD_SEED = 20260914 + 555;
5
6 test("no flags fall back to de_forge, 240 duels, and train.ts's own evaluation seed", () => {
7 expect(parseArgs([])).toEqual({ map: "de_forge", duels: 240, seed: CARD_SEED });
8 });
9
10 test("map, duels and seed are read off the command line", () => {
11 expect(parseArgs(["--map", "de_cinder", "--duels", "500", "--seed", "7"])).toEqual({
12 map: "de_cinder", duels: 500, seed: 7,
13 });
14 });
15
16 // The whole point of the flag is comparing two maps on one seed; a typo that
17 // silently measured the default map again would read as the new map's numbers.
18 test("a map that is not in the catalogue falls back to the default rather than a name nothing holds", () => {
19 expect(parseArgs(["--map", "de_nothing"]).map).toBe("de_forge");
20 expect(parseArgs(["--map"]).map).toBe("de_forge");
21 expect(parseArgs(["--map", "DE_CINDER"]).map).toBe("de_forge");
22 expect(parseArgs(["--map", "../../etc/passwd"]).map).toBe("de_forge");
23 });
24
25 test("a malformed or missing value falls back to the default rather than NaN", () => {
26 expect(parseArgs(["--duels", "nope"]).duels).toBe(240);
27 expect(parseArgs(["--duels", "-5"]).duels).toBe(240);
28 expect(parseArgs(["--duels"]).duels).toBe(240);
29 expect(parseArgs(["--seed", "abc"]).seed).toBe(CARD_SEED);
30 expect(parseArgs(["--seed"]).seed).toBe(CARD_SEED);
31 });
32
33 test("a negative or non-integer seed falls back the same way a non-positive duel count does", () => {
34 expect(parseArgs(["--seed", "-3"]).seed).toBe(CARD_SEED);
35 expect(parseArgs(["--seed", "1.5"]).seed).toBe(CARD_SEED);
36 expect(parseArgs(["--seed", "0"]).seed).toBe(0); // zero is a fine seed, not a falsy one
37 });
scripts/duel.ts added +52
@@ -0,0 +1,52 @@
1 /**
2 * Measures the shipped policy — and the scripted baseline it is measured
3 * against — over the same fixed duels `scripts/train.ts` plays to write
4 * `CARD`, on a map of your choosing. It does not retrain: the weights come
5 * from `weights.ts` exactly as the game ships them, and `--map` only changes
6 * which arena they are dropped into.
7 *
8 * bun run scripts/duel.ts # de_forge, what CARD was measured on
9 * bun run scripts/duel.ts --map de_cinder # the other map, same seeds, same duels
10 * bun run scripts/duel.ts --duels 500 --seed 7
11 */
12
13 import { policy } from "../src/client/bots.ts";
14 import { DEFAULT_MAP, MAP_NAMES, mapFor, type MapName } from "../src/client/map.ts";
15 import { flagValue } from "./args.ts";
16 import { evaluate, greedyDriver, type Outcome } from "./train.ts";
17
18 // train.ts's own default seed (20260914) plus the offset its evaluations are
19 // always called with, so a duel here lines up with the number on the card.
20 const CARD_SEED = 20260914 + 555;
21
22 type Options = { map: MapName; duels: number; seed: number };
23
24 const isMapName = (value: string | null): value is MapName =>
25 (MAP_NAMES as readonly string[]).includes(value as string);
26
27 /** A missing or malformed value keeps the default rather than guessing at it. */
28 export function parseArgs(argv: string[]): Options {
29 const value = (name: string): string | null => flagValue(argv, name);
30 const mapArg = value("map");
31 const duels = value("duels") === null ? NaN : Number(value("duels"));
32 const seed = value("seed") === null ? NaN : Number(value("seed"));
33 return {
34 map: isMapName(mapArg) ? mapArg : DEFAULT_MAP,
35 duels: Number.isFinite(duels) && duels > 0 ? Math.round(duels) : 240,
36 seed: Number.isInteger(seed) && seed >= 0 ? seed : CARD_SEED,
37 };
38 }
39
40 const rate = (o: Outcome): number => Math.round((o.won / (o.won + o.lost + o.draw)) * 100);
41
42 function main(): void {
43 const opts = parseArgs(process.argv.slice(2));
44 const map = mapFor(opts.map);
45 console.log(`${opts.duels} seeded duels on ${opts.map}, seed ${opts.seed}`);
46 const shipped = evaluate(greedyDriver(policy()), opts.duels, opts.seed, map);
47 const baseline = evaluate("scripted", opts.duels, opts.seed, map);
48 console.log(`shipped ${shipped.won}/${shipped.lost}/${shipped.draw} (${rate(shipped)}% won)`);
49 console.log(`scripted ${baseline.won}/${baseline.lost}/${baseline.draw} (${rate(baseline)}% won)`);
50 }
51
52 if (import.meta.main) main();
scripts/measure.ts +2 −4
@@ -22,6 +22,7 @@ import {
2222 } from "../src/client/world.ts";
2323 import { DEFAULT_MAP, MAP_NAMES, mapFor, type MapName } from "../src/client/map.ts";
2424 import { reportLines, summarize, type RoundRecord } from "../src/client/rounds.ts";
25 import { flagValue } from "./args.ts";
2526
2627 export type RulesName = "short" | "long";
2728 export type Options = {
@@ -37,10 +38,7 @@ const DEFAULTS: Options = {
3738 * missing or malformed value keeps the default rather than guessing at it.
3839 */
3940 export function parseArgs(argv: string[]): Options {
40 const value = (name: string): string | null => {
41 const at = argv.indexOf(`--${name}`);
42 return at < 0 ? null : (argv[at + 1] ?? null);
43 };
41 const value = (name: string): string | null => flagValue(argv, name);
4442 const matches = value("matches") === null ? NaN : Number(value("matches"));
4543 const seed = value("seed") === null ? NaN : Number(value("seed"));
4644 const rulesArg = value("rules");
scripts/train.ts +6 −1
@@ -17,6 +17,7 @@ import { join } from "node:path";
1717 import { HEADS, decode } from "../src/client/action.ts";
1818 import { botThink, makeBrain } from "../src/client/bots.ts";
1919 import { rng } from "../src/client/math.ts";
20 import { DE_FORGE, type MapDef } from "../src/client/map.ts";
2021 import { OBS_SIZE } from "../src/client/observe.ts";
2122 import {
2223 createPolicy, forward, greedy, newActivations, pack, policyBytes, policyOver, sample,
@@ -58,15 +59,19 @@ export type Outcome = { won: number; lost: number; draw: number; damage: number
5859 * Play `duels` fixed duels against the scripted bots. Both drivers get the
5960 * same seeds, so the same spawns, the same distances and the same opponent
6061 * skills: the difference in the result is the difference in the driver.
62 * `map` defaults to de_forge — the arena the shipped weights were trained
63 * and are still measured in — and takes any other map to measure the same
64 * fixed duels on it instead, without retraining anything.
6165 */
6266 export function evaluate(
6367 drive: ((arena: Arena) => void) | "scripted",
6468 duels: number,
6569 seed: number,
70 map: MapDef = DE_FORGE,
6671 ): Outcome {
6772 const out: Outcome = { won: 0, lost: 0, draw: 0, damage: 0 };
6873 for (let i = 0; i < duels; i++) {
69 const arena = createArena(seed + i * 7919, scriptedFoe);
74 const arena = createArena(seed + i * 7919, scriptedFoe, map);
7075 if (drive === "scripted") arena.learner.brain = makeBrain(0.65, seed + i, false);
7176 // Both drivers get the same six seconds; the scripted brain turns its own
7277 // view every tick, the policy decides once every SKIP ticks.
src/rl/arena.test.ts +24
@@ -1,6 +1,7 @@
11 import { expect, test } from "bun:test";
22 import { decode, HEADS, STILL, toInput } from "../client/action.ts";
33 import { policy } from "../client/bots.ts";
4 import { colOf, DE_CINDER, DE_FORGE, rowOf, walkable } from "../client/map.ts";
45 import { OBS_SIZE } from "../client/observe.ts";
56 import { forward, greedy, newActivations } from "../client/policy.ts";
67 import { aimAt, bodyCentre, canSee } from "../client/world.ts";
@@ -26,6 +27,29 @@ test("a duel starts with two live players in range of each other", () => {
2627 }
2728 });
2829
30 test("an arena built on de_cinder spawns its duellists on that map's walkable cells, and none of de_forge's", () => {
31 let sawACellDeForgeWouldReject = false;
32 for (let seed = 0; seed < 40; seed++) {
33 const arena = createArena(9000 + seed * 131, stillFoe, DE_CINDER);
34 expect(arena.map).toBe(DE_CINDER);
35 // The spawn coordinates below come from `openCellsOf(arena.map)`, not from
36 // the world — this is the assertion that catches a `map` that reaches
37 // `reset` but never reaches `createWorld`, which would still spawn on
38 // de_cinder's open cells inside de_forge's walls, bullets and sightlines.
39 expect(arena.world.map).toBe(DE_CINDER);

Approved. Both gaps are closed, and I checked them the same way I found them.

The anchored line is the fix. I deleted the map argument from createWorld in createArena again, leaving everything else as written, and this time the suite went red — arena.test.ts:39, one fail out of twelve, on exactly the mutation that silently spawned de_cinder's cells inside de_forge's walls last round. The 75% in the README now has a test standing behind it rather than a coincidence. bun run check is green at ac4e67d, 699 pass, up the five scripts/duel.test.ts adds, and the four measured numbers reproduce exactly on this head: de_forge 175/29/36 shipped (73%) against 93/74/73 scripted (39%), de_cinder 181/31/28 (75%) against 103/70/67 (43%).

scripts/duel.test.ts is real too — I replaced the isMapName guard with a cast and it failed on the catalogue case, so the fallback that keeps a typo'd --map from reading as the other map's numbers is held down. Pulling the flag reader out into scripts/args.ts and having measure.ts share it was the nicer end of the nit; one three-line module named for one thing, both callers tested, and measure.test.ts is still green over it. (train.ts keeps its own arg() — a different helper with typed defaults, and leaving it alone is the right call for a change this size.)

The rest of the list has nothing to answer for. src/rl/arena.ts is reached only by scripts/train.ts and src/rl/{ppo,pool,shard}.ts, never by the server or the bundle, so no request, identity, body-cap or escaping surface moves; the only inputs are a script's own argv, validated against a closed catalogue; no shell, no SQL, no new dependency, no secret, no dead code. Nothing lands on the page or in src/tty.ts, so the two-doors rule is not in play, and CARD is untouched because the default path is untouched — which is what the unchanged arena tests prove.

The preview URL still answers 404 at / for me, and so does production, so I am reading that as the platform and not this branch; the forge's screenshots at this head render the menu, the map select and the leaderboard on phone and desktop, and the branch served locally answers / in ~11 ms with CSP, nosniff and Referrer-Policy intact.

Honest numbers, a commit cited instead of a hand-edited weights.ts, and a clear statement that the policy transfers rather than degrades. Ship it.

40 for (const pos of [arena.learner.pos, arena.foe.pos]) {
41 const col = colOf(DE_CINDER, pos[0]);
42 const row = rowOf(DE_CINDER, pos[1]);
43 expect(walkable(DE_CINDER, col, row)).toBe(true);
44 if (!walkable(DE_FORGE, col, row)) sawACellDeForgeWouldReject = true;
45 }
46 }
47 // Proof the arena is actually reading de_cinder's grid and not silently
48 // falling back to the default: a spawn that de_forge's own geometry would
49 // never produce, because it is solid rock there.
50 expect(sawACellDeForgeWouldReject).toBe(true);
51 });
52
2953 test("a decision is held for SKIP ticks, so the clock moves four at a time", () => {
3054 const arena = createArena(4242, stillFoe);
3155 expect(arena.ticks).toBe(0);
src/rl/arena.ts +30 −14
@@ -1,5 +1,8 @@
11 /**
2 * The arena: one duel on de_forge, stepped as fast as the machine will go.
2 * The arena: one duel, stepped as fast as the machine will go. `createArena`
3 * defaults to de_forge — the map the shipped weights were trained and are
4 * still evaluated on — and takes any other `MapDef` from the catalogue for a
5 * duel on it instead; `scripts/duel.ts` is how to ask for one.
36 *
47 * It is the real game — the real map, the real movement, the real bullets and
58 * the real damage — with the round, the economy and the bomb taken out, because
@@ -14,11 +17,11 @@
1417
1518 import { decode, STILL, toInput, type Intent } from "../client/action.ts";
1619 import { makeBrain, botThink } from "../client/bots.ts";
17 import { DE_FORGE, walkable, worldOf } from "../client/map.ts";
20 import { DE_FORGE, walkable, worldOf, type MapDef } from "../client/map.ts";
1821 import { angleDelta, clamp, dist, rng, type Vec3 } from "../client/math.ts";
1922 import { observe, OBS_SIZE, type Memory } from "../client/observe.ts";
2023 import type { Input, Player, World } from "../client/world.ts";
21 import { aimAt, applyInput, bodyCentre, canSee, createWorld, NO_INPUT } from "../client/world.ts";
24 import { aimAt, applyInput, bodyCentre, canSee, createWorld, DEFAULT_DIFFICULTY, NO_INPUT } from "../client/world.ts";
2225 import { SHORT } from "../client/match.ts";
2326
2427 export const TICK = 1 / 64;
@@ -33,16 +36,28 @@ export const EPISODE_TICKS = 384; // six seconds, which is a long fight
3336 */
3437 export const SKIP = 4;
3538
36 /** Every cell a player can stand in, once, so a reset is two array lookups. */
37 const OPEN: Vec3[] = (() => {
38 const out: Vec3[] = [];
39 for (let r = 0; r < DE_FORGE.gridH; r++)
40 for (let c = 0; c < DE_FORGE.gridW; c++) if (walkable(DE_FORGE, c, r)) out.push(worldOf(DE_FORGE, c, r, 2));
41 return out;
42 })();
39 /**
40 * Every cell a player can stand in, once a map, so a reset is two array
41 * lookups rather than a walk of the grid. Built lazily and cached rather than
42 * at module load: a map nothing ever plays on should not pay for its own
43 * grid walk, and de_forge's arena — the one the shipped weights were trained
44 * and are still evaluated in — still gets exactly the list it always did.
45 */
46 const OPEN_CACHE = new Map<MapDef, Vec3[]>();
47 function openCellsOf(map: MapDef): Vec3[] {
48 let cells = OPEN_CACHE.get(map);
49 if (cells === undefined) {
50 cells = [];
51 for (let r = 0; r < map.gridH; r++)
52 for (let c = 0; c < map.gridW; c++) if (walkable(map, c, r)) cells.push(worldOf(map, c, r, 2));
53 OPEN_CACHE.set(map, cells);
54 }
55 return cells;
56 }
4357
4458 export type Arena = {
4559 world: World;
60 map: MapDef;
4661 learner: Player;
4762 foe: Player;
4863 memory: Memory;
@@ -86,14 +101,14 @@ export function policyFoe(think: (arena: Arena) => Intent): (arena: Arena, dt: n
86101 };
87102 }
88103
89 export function createArena(seed: number, opponent = scriptedFoe): Arena {
104 export function createArena(seed: number, opponent = scriptedFoe, map: MapDef = DE_FORGE): Arena {
90105 // Two players, one a side: createWorld with a team size of one gives exactly
91106 // that — the person's slot and one bot — and we drive both ourselves.
92 const world = createWorld(SHORT, "learner", "ct", 1, seed);
107 const world = createWorld(SHORT, "learner", "ct", 1, seed, DEFAULT_DIFFICULTY, map);

Close, and the measurement itself is sound — I reran it at 489c3d3 and got the README's four numbers exactly: de_forge 175/29/36 shipped (73%) against 93/74/73 scripted (39%), de_cinder 181/31/28 (75%) against 103/70/67 (43%). bun run check is green here too, 694 pass. No runtime surface is touched — src/rl/arena.ts is reached only by scripts/train.ts and src/rl/{ppo,pool,shard}.ts, never by the server or the bundle — so there is nothing to say about identity, request bodies, escaping or headers on this one. No new dependency, no secret, no dead code, and it is honest about CARD's 72%/42% rather than hand-editing weights.ts. One miss, and it is the line the whole pull rests on.

Fix first: the new test does not cover the change that produces the number. This anchored line is the only place the named map reaches the world, and nothing asserts it. src/rl/arena.test.ts checks where the duellists stand, and those coordinates come from openCellsOf(arena.map) in reset — not from the world. I deleted the map argument from this createWorld call, leaving everything else as written, and the whole arena suite still passed, 12 of 12. The arena was then spawning players on de_cinder's open cells inside de_forge's geometry — wrong walls, wrong line of sight, wrong bullets — and bun run scripts/duel.ts --map de_cinder returned 67%/36% instead of 75%/43%. That is an eight-point swing in the README's headline claim with a green check over it. World already carries map, so expect(arena.world.map).toBe(DE_CINDER) inside the loop you already wrote closes it; a canSee assertion across a pair of cells the two maps disagree about would be stronger still.

Second: scripts/duel.ts's parseArgs is exported and has no test, while its twin in scripts/measure.ts has scripts/measure.test.ts beside it reading the same --map/--seed flags — including a case whose comment is exactly this failure: “a typo that silently measured the default map again would read as the new map's numbers.” duel.ts falls back to de_forge on any unknown --map, which is the right behaviour and safe — I checked that --map ../../etc/passwd and --duels -5 --seed abc all print de_forge's numbers under a de_forge heading — but it is silent, and a scripts/duel.test.ts in the shape of measure.test.ts is the cheap way to keep it. Optional while you are there: this parseArgs re-implements measure.ts's flag reader almost verbatim, and one shared reader would suit the repo's one-concept-per-module shape. Nit, not a blocker.

The preview URL answers 404 at / for me right now, but production answers 404 too and the forge's screenshots at this head render the menu, the map select and the leaderboard on both phone and desktop, so I am reading that as the platform rather than this branch; serving the branch locally, / answers 200 in 11 ms with CSP, nosniff and Referrer-Policy intact. Nothing on the page or in src/tty.ts changes here, so the two-doors rule has nothing to answer for.

Add the arena.world.map assertion and the duel.ts argument test and this is a straight approve — the work behind it is right, and the honesty about the CARD drift is the best part of it.

93108 const learner = world.me;
94109 const foe = world.players[1] as Player;
95110 const arena: Arena = {
96 world, learner, foe,
111 world, map, learner, foe,
97112 memory: { sinceSeen: 9 },
98113 foeMemory: { sinceSeen: 9 },
99114 ticks: 0,
@@ -180,7 +195,8 @@ export function reset(arena: Arena): void {
180195 // Two spots the right distance apart, preferring a pair that can or cannot
181196 // see each other as the coin said. The distance is not negotiable and the
182197 // line of sight is: a duel at the wrong range teaches the wrong weapon.
183 const cell = (): Vec3 => OPEN[Math.floor(random() * OPEN.length)] as Vec3;
198 const open = openCellsOf(arena.map);
199 const cell = (): Vec3 => open[Math.floor(random() * open.length)] as Vec3;
184200 let a = cell(), b = cell();
185201 for (let tries = 0; tries < 400; tries++) {
186202 const d = dist(a, b);

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs