trevato/cs-engine

Public: anyone can read this; the crew can change it.

forked from trevato/cs · proposes

More
Forks3g fMapg aRunsg rEventsFlows

bot retraining. bots get confused on procedural maps. need to train them to b… #186

Merged vespercrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

vesper/cs-engine:vesper/177-bot-retraining-bots-get-confused-on-proc → main 4 commits fork

  • Merged. vesper/177-bot-retraining-bots-get-confused-on-proc was merged into main as 51ea7ef.

Review

Sign in to review this change.
19 files changed +488 −117
README.md +21 −6
@@ -19,7 +19,8 @@ and the example is the price tag.
1919 ```sh
2020 PORT=8080 FORGE_APP_DATA=./data bun run src/server.ts # run it here
2121 bun run scripts/build.ts # rebuild every static/<name>.<hash>.js, with its size
22 bun run scripts/train.ts --iters 1000 # retrain the bots' policy
22 bun run scripts/train.ts --iters 1200 # retrain the bots' policy, on 96 maps
23 bun run scripts/train.ts --iters 1200 --maps 1 # the way it used to train: de_forge only
2324 bun run scripts/train.ts --bench # how fast the arena steps
2425 bun run scripts/pack.ts [dir] # write the engine out as a package (default dist/engine)
2526 bun test # the check every pull runs
@@ -586,7 +587,10 @@ renderer, no atlas, no mesh and no sound ship in that bundle, and
586587 it (`webgl2`, `AudioContext`, `Math.tanh`): a door is only worth having if
587588 importing it costs what you use.
588589 - `src/rl/` — training, which no game module imports: `arena` is the environment,
589 a real duel on de_forge with the round and the economy taken out. The algorithm
590 a real duel with the round and the economy taken out, on whichever level it is
591 handed, and `maps` is the two streams of levels it is handed — the ones a run
592 learns on and the ones it is graded on, disjoint by construction and checked
593 to be. The algorithm
590594 is not in here — PPO is `src/engine/learn.ts`, behind the door, because nothing
591595 in it is about shooting; what is here is this game's arena.
592596 `scripts/train.ts` runs the two together and writes `src/client/weights.ts`. It is the
@@ -792,10 +796,21 @@ now a policy learned with PPO.
792796 vector cut into the heads above, a value the trainer reads and the page throws
793797 away, and int8 packing for the weights. The game brings the observation, the
794798 heads and the trained bytes; the engine brings the arithmetic.
795 - **Where it learned** — `src/rl/arena.ts`, a duel on de_forge with the real
796 movement, the real bullets and the real damage, and no round, economy or bomb.
797 It steps about 13,000 ticks a second on one core, allocation-free; the policy
798 decides every fourth tick and the decision is held in between.
799 - **Where it learned** — `src/rl/arena.ts`, a duel with the real movement, the
800 real bullets and the real damage, and no round, economy or bomb. It steps
801 about 13,000 ticks a second on one core, allocation-free; the policy decides
802 every fourth tick and the decision is held in between.
803 - **On which maps** — all of them, which is what #177 changed. The game carves a
804 layout out of the match id, so most of the maps a bot will ever be played on
805 did not exist when it was trained; a policy trained on de_forge alone learned
806 de_forge, and on a generated map it drew fights it should have finished.
807 `src/rl/maps.ts` is the list: de_forge and ninety-five layouts from one stream
808 of seeds to learn on, twenty-four from another stream to be graded on and
809 never trained on. Every arena rolls onto another map each time its duel ends, so a
810 run fights on all of them many times over and none of them long enough to
811 learn its corners. What the policy does on the held-out maps is on the card,
812 on the page and in the terminal, and `src/rl/learned.test.ts` holds the
813 shipped weights to it.
799814 - **How it learned** — `src/engine/learn.ts`, about three hundred lines behind
800815 the engine's door: GAE, a clipped surrogate, an entropy bonus, Adam, and a
801816 backward pass written out by hand and checked against finite differences in
scripts/train.ts +71 −12
@@ -5,6 +5,7 @@
55 * bun run scripts/train.ts --bench # how fast the arena steps
66 * bun run scripts/train.ts --iters 300 # train and write the weights
77 * bun run scripts/train.ts --iters 300 --dry # train without writing
8 * bun run scripts/train.ts --maps 1 # one map, the way it used to train
89 *
910 * Nothing here is imported by the server or the page: this file runs on a
1011 * machine with a few minutes to spare, and its only output is a table of
@@ -26,9 +27,12 @@ import {
2627 } from "../src/engine/index.ts";
2728 import { OBS_SIZE } from "../src/client/observe.ts";
2829 import {
29 createArena, EPISODE_TICKS, observeFoe, observeLearner, policyFoe, reset, scriptedFoe, SKIP,
30 createArena, EPISODE_TICKS, observeFoe, observeLearner, policyFoe, relevel, reset, scriptedFoe, SKIP,
3031 step, stepInput, stillFoe, type Arena,
3132 } from "../src/rl/arena.ts";
33 import { holdoutLevels, trainingLevels } from "../src/rl/maps.ts";
34 import { DE_FORGE } from "../src/client/map.ts";
35 import type { Level } from "../src/client/level.ts";
3236
3337 export const SHAPE: Shape = { obs: OBS_SIZE, hidden: 64, heads: HEADS };
3438
@@ -44,24 +48,32 @@ const ENVS = arg("envs", 64);
4448 const STEPS = arg("steps", 32);
4549 const ITERS = arg("iters", 240);
4650 const SEED = arg("seed", 20260914);
51 /** How many layouts a run learns on: de_forge and the rest carved from seeds. */
52 const MAPS = arg("maps", 96);
53 /** How many layouts it is graded on, none of which it trained on. */
54 const HOLDOUT = arg("holdout", 24);
4755
4856 // --- evaluation -----------------------------------------------------------
4957
5058 export type Outcome = { won: number; lost: number; draw: number; damage: number };
5159
5260 /**
53 * Play `duels` fixed duels against the scripted bots. Both drivers get the
54 * same seeds, so the same spawns, the same distances and the same opponent
55 * skills: the difference in the result is the difference in the driver.
61 * Play `duels` fixed duels against the scripted bots, dealt round-robin over
62 * `levels`. Both drivers get the same seeds and the same maps, so the same
63 * spawns, the same distances and the same opponent skills: the difference in
64 * the result is the difference in the driver. A duel that ends in neither a
65 * win nor a loss is a draw, and a draw is the number to watch across maps — it
66 * is six seconds in which the bot never finished the fight.
5667 */
5768 export function evaluate(
5869 drive: ((arena: Arena) => void) | "scripted",
5970 duels: number,
6071 seed: number,
72 levels: Level[] = [DE_FORGE],
6173 ): Outcome {
6274 const out: Outcome = { won: 0, lost: 0, draw: 0, damage: 0 };
6375 for (let i = 0; i < duels; i++) {
64 const arena = createArena(seed + i * 7919, scriptedFoe);
76 const arena = createArena(seed + i * 7919, scriptedFoe, levels[i % levels.length] as Level);
6577 if (drive === "scripted") arena.learner.brain = makeBrain(0.65, seed + i, false);
6678 // Both drivers get the same six seconds; the scripted brain turns its own
6779 // view every tick, the policy decides once every SKIP ticks.
@@ -93,7 +105,9 @@ export function greedyDriver(policy: Policy): (arena: Arena) => void {
93105 // --- the loop -------------------------------------------------------------
94106
95107 function benchmark(): void {
96 const arenas = Array.from({ length: 64 }, (_, i) => createArena(SEED + i));
108 const levels = trainingLevels(MAPS);
109 const arenas = Array.from({ length: 64 }, (_, i) =>
110 createArena(SEED + i, scriptedFoe, levels[i % levels.length] as Level));
97111 const obs = new Float32Array(OBS_SIZE);
98112 const policy = createPolicy(SHAPE, rng(1));
99113 const act = newActivations(SHAPE);
@@ -149,11 +163,26 @@ async function train(): Promise<void> {
149163 return u < still ? stillFoe : u < still + scripted ? scriptedFoe : selfPlay;
150164 };
151165
152 const arenas = Array.from({ length: ENVS }, (_, e) => createArena(SEED + e * 104729, opponentFor(e, 0)));
166 // The maps. Every arena starts on one of them and rolls onto another every
167 // time its duel ends, so a run of this length fights on all of them many
168 // times over and none of them long enough to learn its corners. There is no
169 // curriculum on the map on purpose: what the policy sees is egocentric, so a
170 // corridor is a corridor, and mixing the layouts from the first step is what
171 // stops "the enemy went left" from meaning a particular left.
172 const levels = trainingLevels(MAPS);
173 const holdout = holdoutLevels(HOLDOUT);
174 let rolled = 0;
175 const nextLevel = (): Level => levels[rolled++ % levels.length] as Level;
176 const arenas = Array.from({ length: ENVS }, (_, e) =>
177 createArena(SEED + e * 104729, opponentFor(e, 0), nextLevel()));
153178
154179 const baseline = evaluate("scripted", 120, SEED + 555);
180 const baselineUnseen = evaluate("scripted", 120, SEED + 777, holdout);
181 console.log(`${levels.length} maps to learn on (de_forge and ${levels.length - 1} carved from seeds), ` +
182 `${holdout.length} held back to be graded on`);
155183 console.log(
156 `baseline (scripted brain in the same duels): ${baseline.won} won / ${baseline.lost} lost / ${baseline.draw} drawn`,
184 `baseline (scripted brain in the same duels): ${baseline.won} won / ${baseline.lost} lost / ${baseline.draw} drawn` +
185 `, and on maps it has never seen ${baselineUnseen.won}/${baselineUnseen.lost}/${baselineUnseen.draw}`,
157186 );
158187
159188 const episodeReturn = new Float64Array(ENVS);
@@ -199,7 +228,7 @@ async function train(): Promise<void> {
199228 finished += 1;
200229 if (result.won) wins += 1;
201230 if (result.lost) losses += 1;
202 reset(arena);
231 relevel(arena, nextLevel());
203232 arena.opponent = opponentFor(e, progress);
204233 }
205234 }
@@ -236,10 +265,21 @@ async function train(): Promise<void> {
236265 const trained = evaluate(greedyDriver(policy), 240, SEED + 555);
237266 const shipped = evaluate(greedyDriver(quantised), 240, SEED + 555);
238267 const baselineAgain = evaluate("scripted", 240, SEED + 555);
268 // The same duels again on the maps no arena in this run was ever put on:
269 // what the policy does on a layout it has never seen is the number the whole
270 // map curriculum is for, and a draw there is the bot that got confused.
271 const unseen = evaluate(greedyDriver(quantised), 240, SEED + 777, holdout);
272 const unseenBaseline = evaluate("scripted", 240, SEED + 777, holdout);
239273 const rate = (o: typeof trained): number => Math.round((o.won / (o.won + o.lost + o.draw)) * 100);
240 console.log(`\ntrained ${trained.won}/${trained.lost}/${trained.draw} (${rate(trained)}% won)`);
241 console.log(`shipped ${shipped.won}/${shipped.lost}/${shipped.draw} (${rate(shipped)}% won) after int8 packing`);
242 console.log(`scripted ${baselineAgain.won}/${baselineAgain.lost}/${baselineAgain.draw} (${rate(baselineAgain)}% won)`);
274 const draws = (o: typeof trained): number => Math.round((o.draw / (o.won + o.lost + o.draw)) * 100);
275 console.log(`\nde_forge, the map it has always been measured on:`);
276 console.log(` trained ${trained.won}/${trained.lost}/${trained.draw} (${rate(trained)}% won)`);
277 console.log(` shipped ${shipped.won}/${shipped.lost}/${shipped.draw} (${rate(shipped)}% won) after int8 packing`);
278 console.log(` scripted ${baselineAgain.won}/${baselineAgain.lost}/${baselineAgain.draw} (${rate(baselineAgain)}% won)`);
279 console.log(`${holdout.length} maps it has never seen:`);
280 console.log(` shipped ${unseen.won}/${unseen.lost}/${unseen.draw} (${rate(unseen)}% won, ${draws(unseen)}% drawn)`);
281 console.log(` scripted ${unseenBaseline.won}/${unseenBaseline.lost}/${unseenBaseline.draw} ` +
282 `(${rate(unseenBaseline)}% won, ${draws(unseenBaseline)}% drawn)`);
243283
244284 if (flag("dry")) return;
245285 const out = join(import.meta.dir, "..", "src", "client", "weights.ts");
@@ -253,6 +293,11 @@ async function train(): Promise<void> {
253293 duels: 240,
254294 winRate: rate(shipped),
255295 baselineWinRate: rate(baselineAgain),
296 maps: levels.length,
297 unseenMaps: holdout.length,
298 unseenWinRate: rate(unseen),
299 unseenBaselineWinRate: rate(unseenBaseline),
300 unseenDrawRate: draws(unseen),
256301 }));
257302 console.log(`\nwrote ${out} (${packed.bytes.length} base64 chars)`);
258303 }
@@ -267,6 +312,11 @@ type Card = {
267312 duels: number;
268313 winRate: number;
269314 baselineWinRate: number;
315 maps: number;
316 unseenMaps: number;
317 unseenWinRate: number;
318 unseenBaselineWinRate: number;
319 unseenDrawRate: number;
270320 };
271321
272322 function weightsModule(packed: ReturnType<typeof pack>, card: Card): string {
@@ -287,9 +337,18 @@ export type Card = {
287337 updates: number;
288338 stepsPerSecond: number;
289339 minutes: number;
340 /** Duels evaluated on de_forge, for \`winRate\` and \`baselineWinRate\`. */
290341 duels: number;
291342 winRate: number;
292343 baselineWinRate: number;
344 /** Layouts the run learned on: de_forge and the rest carved from seeds. */
345 maps: number;
346 /** Layouts it was graded on and never trained on, from the other seed stream. */
347 unseenMaps: number;
348 unseenWinRate: number;
349 unseenBaselineWinRate: number;
350 /** Duels on those maps that ran the six seconds out without a kill. */
351 unseenDrawRate: number;
293352 };
294353
295354 export const CARD: Card = ${JSON.stringify(card, null, 2)};
src/client/bots.test.ts +38 −2
@@ -1,10 +1,14 @@
11 import { expect, test } from "bun:test";
2 import { applyIntent, blindness, botThink, CARD, flashed, makeBrain, policy, think } from "./bots.ts";
2 import {
3 applyIntent, blindness, botThink, CARD, flashed, makeBrain, pathTo, policy, think,
4 } from "./bots.ts";
5 import { colOf, findPath, rowOf, walkable, worldOf } from "./level.ts";
6 import { dist, type Vec3 } from "../engine/index.ts";
37 import { OBS_SIZE } from "./observe.ts";
48 import { FLASH } from "./weapons.ts";
59 import { DE_FORGE } from "./map.ts";
610 import { COMPETITIVE } from "./match.ts";
7 import { aimAt, bodyCentre, createWorld, type Player } from "./world.ts";
11 import { aimAt, applyInput, bodyCentre, createWorld, type Player } from "./world.ts";
812
913 const duel = (seed = 2024): { world: ReturnType<typeof createWorld>; me: Player; bot: Player } => {
1014 const world = createWorld(DE_FORGE, COMPETITIVE, "you", "ct", 1, seed);
@@ -83,6 +87,38 @@ test("a bot that has only just seen you does not shoot yet", () => {
8387 expect(immediate.fire).toBe(false);
8488 });
8589
90 test("a goal nothing can walk to is nudged to a cell something can", () => {
91 // A crate is solid to the search and standing on it is not: a bomb dropped
92 // up there is a point A* answers nothing about.
93 const crate = worldOf(DE_FORGE, 21, 18, 0); // one of de_forge's jumpable crates
94 expect(walkable(DE_FORGE, colOf(DE_FORGE, crate[0]), rowOf(DE_FORGE, crate[1]))).toBe(false);
95 const from = DE_FORGE.spawns.t[0] as Vec3;
96 expect(findPath(DE_FORGE, from, crate)).toHaveLength(0);
97
98 const path = pathTo(DE_FORGE, from, crate);
99 expect(path.length).toBeGreaterThan(0);
100 const end = path[path.length - 1] as Vec3;
101 expect(Math.hypot(end[0] - crate[0], end[1] - crate[1])).toBeLessThan(3 * 128 * 1.5);
102 });
103
104 test("a bot whose objective sits on a crate walks to it rather than standing still", () => {
105 const world = createWorld(DE_FORGE, COMPETITIVE, "you", "ct", 1, 4242);
106 world.match.phase = "live";
107 const bot = world.players[1] as Player;
108 bot.team = "t";
109 bot.hasBomb = false;
110 world.me.alive = false; // nothing to fight: this is about the walking
111 // The bomb, dropped on top of a crate in mid.
112 world.bomb.state = "dropped";
113 world.bomb.pos = worldOf(DE_FORGE, 21, 18, 0);
114 const before = dist(bot.pos, world.bomb.pos);
115 for (let tick = 0; tick < 64 * 3; tick++) {
116 world.time += 1 / 64;
117 applyInput(world, bot, botThink(world, bot, 1 / 64), 1 / 64);
118 }
119 expect(dist(bot.pos, world.bomb.pos)).toBeLessThan(before - 200);
120 });
121
86122 test("the card describes a real training run", () => {
87123 expect(CARD.envSteps).toBeGreaterThan(1_000_000);
88124 expect(CARD.updates).toBeGreaterThan(0);
src/client/bots.ts +38 −2
@@ -16,7 +16,7 @@ import {
1616 angleDelta, clamp, dist, forward, newActivations, rng, sample, unpack,
1717 type Activations, type Policy, type Vec3,
1818 } from "../engine/index.ts";
19 import { colOf, findPath, rowOf, siteAt, walkable, type Level } from "./level.ts";
19 import { colOf, findPath, rowOf, siteAt, walkable, worldOf, type Level } from "./level.ts";
2020 import { observe, OBS_SIZE, type Memory } from "./observe.ts";
2121 import { CARD, PACKED } from "./weights.ts";
2222 import { FLASH, WEAPONS, type WeaponId } from "./weapons.ts";
@@ -178,6 +178,42 @@ const nearby = (level: Level, brain: BotBrain, at: Vec3, spread: number): Vec3 =
178178 return at;
179179 };
180180
181 /** How many cells out a goal nothing walks to may be nudged, and how many tries. */
182 const NUDGE_RINGS = 3;
183 const NUDGE_TRIES = 8;
184
185 /**
186 * A path to `goal`, or to the nearest cell something can walk to.
187 *
188 * A* aims at a cell, and not every point a round hands a bot is one: a bomb
189 * dropped on top of a jumpable crate stands on a cell the search calls solid,
190 * and so does a body that fell into a corner of a map carved from a seed. The
191 * search then comes back with nothing, and a bot with no path stands exactly
192 * where it is until it asks again three seconds later — the standing about
193 * that looks, from the other end of a corridor, like a bot that has forgotten
194 * what it was doing. So a goal nothing walks to is nudged outwards in rings
195 * until a cell answers, at most `NUDGE_TRIES` searches deep, and only a goal
196 * walled off entirely leaves the bot without a plan.
197 */
198 export function pathTo(level: Level, from: Vec3, goal: Vec3): Vec3[] {
199 const direct = findPath(level, from, goal);
200 if (direct.length > 0) return direct;
201 const col = colOf(level, goal[0]);
202 const row = rowOf(level, goal[1]);
203 let tries = 0;
204 for (let ring = 1; ring <= NUDGE_RINGS; ring++) {
205 for (let dr = -ring; dr <= ring; dr++)
206 for (let dc = -ring; dc <= ring; dc++) {
207 if (Math.max(Math.abs(dc), Math.abs(dr)) !== ring) continue;
208 if (!walkable(level, col + dc, row + dr)) continue;
209 if (tries++ >= NUDGE_TRIES) return [];
210 const path = findPath(level, from, worldOf(level, col + dc, row + dr, goal[2]));
211 if (path.length > 0) return path;
212 }
213 }
214 return [];
215 }
216
181217 function pickTarget(world: World, bot: Player, brain: BotBrain): Player | null {
182218 let best: Player | null = null;
183219 let bestScore = Infinity;
@@ -360,7 +396,7 @@ export function botThink(world: World, bot: Player, dt: number): Input {
360396 world.time > brain.repathAt || objective !== brain.plannedFor;
361397 if (stale) {
362398 brain.goal = chooseGoal(world, bot, brain);
363 brain.path = findPath(world.level, bot.pos, brain.goal);
399 brain.path = pathTo(world.level, bot.pos, brain.goal);
364400 brain.step = 0;
365401 brain.plannedFor = objective;
366402 brain.repathAt = world.time + 3 + brain.random() * 3;
src/client/replay.test.ts +17 −10
@@ -34,7 +34,14 @@ const SECONDS = 40;
3434 const TICKS = SECONDS * TICKS_PER_SECOND;
3535
3636 /**
37 * Forty seconds of a five-a-side round on de_forge, seed 7. Re-pinned for #164,
37 * Forty seconds of a five-a-side round on de_forge, seed 7. Re-pinned for
38 * #177, which retrained the policy the ten bots in it fight with: every second
39 * from the tenth on — the first at which a bot has an enemy to decide about —
40 * hashes differently, and the nine before it are the numbers #164 left to the
41 * character, which is the freeze and the walk out of spawn. Nobody's footing
42 * moved; the decisions did.
43 *
44 * Before that it was re-pinned for #164,
3845 * which put a second grenade in the game, and for exactly the reason #142 was
3946 * re-pinned before it: what moved is the *hash*, not the play. A pocket is two
4047 * counts now rather than one, every body carries the seconds of a flash it
@@ -47,18 +54,18 @@ const TICKS = SECONDS * TICKS_PER_SECOND;
4754 * anything: the bots buy no grenades and the scripted person never presses
4855 * either key, so no flash goes off and no `blind` is ever non-zero here.
4956 */
50 const ROUND = "ca91ea4cb43109a961fa2b63d44bc8d27de4e426b79016341ee3e5e0e6a7043e";
57 const ROUND = "3dd837d703034e7c0ba08c2633300e1b28f8e0b4e160193077dba84438538c3e";
5158 const ROUND_SECONDS = [
5259 "5978706f2efd", "aa6aa9a5a3c9", "f822e864db93", "6f1cf4de1926",
5360 "25923b9cb132", "83b06bc2430e", "b97f8d621af5", "c1b94afedda0",
54 "700c15a0f6c1", "8c0ad0d13658", "d7aad60e3ff3", "d4e5187b49b4",
55 "60730fe44cc8", "a4c501cdf06e", "e23f99650039", "8f0fd8311650",
56 "3e2a50a35374", "0eaec9fa5595", "a4da64d03f4f", "a741df777b49",
57 "a9e376a00a92", "97551fd34e2a", "1342599e6fe1", "b3805e3f7b45",
58 "54fedea960e5", "867ccb00fbfd", "853c878b3121", "f723b6926421",
59 "adca551d5dcd", "dc6e57a96c4b", "5612a3feff88", "302de76251c8",
60 "237a92008fdd", "108e5db94381", "ab4ecb1715fb", "7cbeb300896f",
61 "2afdbc98a409", "0db10ea3eb14", "069dfe802cf2", "fca970b88b16",
61 "700c15a0f6c1", "c38f50dab1ca", "7a99f7ac430d", "704874f88a78",
62 "362cdfb7dc07", "dc096b96ac6f", "c7f5998a8d2a", "0d20ab5e1686",
63 "8ca62062fe8a", "87d4d63b8ff1", "37390d1539de", "eaeef81da1d6",
64 "a33856d0ce01", "6dde9a94751a", "4c929dfc9607", "ee774c17027d",
65 "9135e1bb46df", "643a85bf9c93", "3b62f2485ccd", "f6cee0354c77",
66 "94578e2ff523", "7efd91c9c90d", "769e3d6d5c1f", "861c0e487740",
67 "471399956ceb", "4391147a2f38", "12a4eed48969", "fcde6163b6d4",
68 "ec71e47d09bb", "2b5f5f764b2f", "ed40a891d704", "8d43dd52fa3e",
6269 ];
6370
6471 /**
src/client/weights.test.ts +1 −1
@@ -18,7 +18,7 @@ import { OBS_SIZE } from "./observe.ts";
1818 import { PACKED } from "./weights.ts";
1919
2020 /** What `unpack(PACKED)` has dequantised to since the weights were trained. */
21 const DIGEST = "4d908fcfd1198e8326d2711c95761f08770e681ee817b42af95f38f2c3b502d1";
21 const DIGEST = "5582db35be828690f086d23bb6ee92c7807696188634bac6c4c8b09730068aa5";
2222
2323 test("the shipped weights unpack to the shape the game asks the policy for", () => {
2424 const p = unpack(PACKED);
src/client/weights.ts +24 −10

Diff not shown — a line 10,799 characters wide (limit 2,000). +24 −10

View file
src/games.ts +11
@@ -259,6 +259,17 @@ export type Game = {
259259 * `src/client/replay.test.ts` are unchanged to the byte, which is the real
260260 * measurement here. Neither other bundle moved: nothing outside `src/client/`
261261 * changed. Measured 2026-09-20 after #113.
262 * #177 retrained the policy on ninety-six maps instead of one, and the shooter
263 * grew 364 raw / 176 gzipped bytes for it, measured on top of #113: 90,509 →
264 * 90,873 raw and 40,933 → 41,109 gzipped. Almost none of that is code — the
265 * weights are the same eight kilobytes of int8 they always were, and the only
266 * new lines in the bundle are `pathTo` in `src/client/bots.ts` and five more
267 * numbers on the card. What grew is the base64: int8 that quantises to fewer
268 * runs of the same byte compresses less well, which is a retrained policy and
269 * not a feature. `src/rl/maps.ts`, the arena and the trainer ship in no bundle
270 * at all. It fits the 92,000 / 42,000 #164 raised with 1,127 raw and 891
271 * gzipped to spare, so no ceiling moved; neither other bundle changed a byte.
272 * Measured 2026-09-20, with #113 and #189 merged in.
262273 */
263274 export const GAMES: readonly Game[] = [
264275 {
src/pages/game.ts +4 −1
@@ -165,7 +165,10 @@ export function gamePage(db: Database, { who: visitor, ...options }: PageOptions
165165 ${CARD.envSteps.toLocaleString("en")} ticks of this same simulation, in
166166 ${escape(String(CARD.minutes))} minutes on one core. In ${CARD.duels} fixed duels it wins
167167 ${CARD.winRate}% against the scripted bots this game used to have, which win
168 ${CARD.baselineWinRate}% of the same duels. Its weights are about eight kilobytes
168 ${CARD.baselineWinRate}% of the same duels. It learned on ${CARD.maps} layouts — de_forge and
169 ${CARD.maps - 1} carved from seeds — and on ${CARD.unseenMaps} more it was never trained on it
170 still wins ${CARD.unseenWinRate}% against the same bots' ${CARD.unseenBaselineWinRate}%, which is
171 what a map made the moment you press play asks of it. Its weights are about eight kilobytes
169172 of int8 inside the bundle — there is no inference server and nothing is sent anywhere. Each
170173 bot's own skill still decides how fast it can turn, how steady its hands are and how long it
171174 takes to react, so there is still a ladder from fodder to uncomfortable.</p>
src/rl/arena.test.ts +27 −1
@@ -1,10 +1,14 @@
11 import { expect, test } from "bun:test";
22 import { STILL, toInput } from "../client/action.ts";
33 import { OBS_SIZE } from "../client/observe.ts";
4 import { colOf, openCells, rowOf, walkable, type Level } from "../client/level.ts";
5 import { DE_FORGE } from "../client/map.ts";
46 import { aimAt, bodyCentre, canSee } from "../client/world.ts";
57 import {
6 createArena, EPISODE_TICKS, observeFoe, observeLearner, reset, SKIP, step, stepInput, stillFoe,
8 createArena, EPISODE_TICKS, observeFoe, observeLearner, relevel, reset, SKIP, step, stepInput,
9 stillFoe,
710 } from "./arena.ts";
11 import { holdoutLevels } from "./maps.ts";
812
913 test("a duel starts with two live players in range of each other", () => {
1014 for (let seed = 0; seed < 25; seed++) {
@@ -123,6 +127,28 @@ test("both sides can be observed, and a reset puts the duel back", () => {
123127 expect(arena.world.time).toBe(0);
124128 });
125129
130 test("an arena rolls onto another map, and the duel starts there", () => {
131 const arena = createArena(1234, stillFoe);
132 expect(arena.world.level).toBe(DE_FORGE);
133 const elsewhere = holdoutLevels(1)[0] as Level;
134 for (let i = 0; i < 10; i++) step(arena, { ...STILL, forward: 1 });
135
136 relevel(arena, elsewhere);
137 expect(arena.world.level).toBe(elsewhere);
138 expect(arena.ticks).toBe(0);
139 expect(arena.learner.health).toBe(100);
140 // Both of them stand on a cell of the new map, not on the old map's floor.
141 for (const player of [arena.learner, arena.foe]) {
142 const col = colOf(elsewhere, player.pos[0]);
143 const row = rowOf(elsewhere, player.pos[1]);
144 expect(walkable(elsewhere, col, row)).toBe(true);
145 }
146 // And the cells a fresh duel is dealt from are the new map's, not the old
147 // map's: every spawn from here on is somewhere that exists.
148 expect(arena.open.length).toBe(openCells(elsewhere, 2).length);
149 expect(arena.open.length).not.toBe(openCells(DE_FORGE, 2).length);
150 });
151
126152 test("the arena leaves no render effects behind to grow without bound", () => {
127153 const arena = createArena(2468, stillFoe);
128154 for (let i = 0; i < 50; i++) step(arena, { ...STILL, fire: true });
src/rl/arena.ts +16
@@ -216,6 +216,22 @@ export function reset(arena: Arena): void {
216216 arena.potential = potentialOf(arena, canSee(world, learner, foe));
217217 }
218218
219 /**
220 * Move an arena onto another map and start a fresh duel there. Everything that
221 * is about the level and not about the two players is replaced — the world's
222 * level, which is what the bullets, the view and the bots read, and the list of
223 * cells a spawn is drawn from — and nothing is allocated that was not already
224 * cached: `openOf` keeps one list per level and the levels themselves are made
225 * once in `maps.ts`. This is how one arena covers a hundred layouts in a run
226 * instead of one, which is the difference between a policy that learned to
227 * fight and a policy that learned de_forge.
228 */
229 export function relevel(arena: Arena, level: Level): void {
230 arena.world.level = level;
231 arena.open = openOf(level);
232 reset(arena);
233 }
234
219235 /** Write what the learner sees into `out`. */
220236 export function observeLearner(arena: Arena, out: Float32Array, at = 0): Float32Array {
221237 const seen = arena.learner.alive && arena.foe.alive && canSee(arena.world, arena.learner, arena.foe);
src/rl/learned.test.ts +43 −9
@@ -3,7 +3,10 @@ import { decode, HEADS } from "../client/action.ts";
33 import { botThink, makeBrain, policy } from "../client/bots.ts";
44 import { forward, greedy, newActivations } from "../engine/index.ts";
55 import { OBS_SIZE } from "../client/observe.ts";
6 import { DE_FORGE } from "../client/map.ts";
7 import type { Level } from "../client/level.ts";
68 import { createArena, EPISODE_TICKS, observeLearner, scriptedFoe, step, stepInput } from "./arena.ts";
9 import { holdoutLevels } from "./maps.ts";
710
811 /**
912 * The claim this whole change rests on: the trained policy wins fights the
@@ -14,14 +17,17 @@ import { createArena, EPISODE_TICKS, observeLearner, scriptedFoe, step, stepInpu
1417 const DUELS = 40;
1518 const SEED = 4242;
1619
17 function duels(drive: "policy" | "scripted"): number {
20 type Tally = { won: number; lost: number; draw: number };
21
22 /** `count` duels, dealt round-robin over `levels`, driven one way or the other. */
23 function duels(drive: "policy" | "scripted", count = DUELS, levels: Level[] = [DE_FORGE]): Tally {
1824 const net = policy();
1925 const obs = new Float32Array(OBS_SIZE);
2026 const act = newActivations(net.shape);
2127 const actions = new Int32Array(HEADS.length);
22 let won = 0;
23 for (let i = 0; i < DUELS; i++) {
24 const arena = createArena(SEED + i * 7919, scriptedFoe);
28 const tally: Tally = { won: 0, lost: 0, draw: 0 };
29 for (let i = 0; i < count; i++) {
30 const arena = createArena(SEED + i * 7919, scriptedFoe, levels[i % levels.length] as Level);
2531 if (drive === "scripted") arena.learner.brain = makeBrain(0.65, SEED + i, false);
2632 while (arena.ticks < EPISODE_TICKS && arena.learner.alive && arena.foe.alive) {
2733 if (drive === "scripted") {
@@ -33,17 +39,45 @@ function duels(drive: "policy" | "scripted"): number {
3339 step(arena, decode(actions));
3440 }
3541 }
36 if (!arena.foe.alive) won += 1;
42 if (!arena.foe.alive) tally.won += 1;
43 else if (!arena.learner.alive) tally.lost += 1;
44 else tally.draw += 1;
3745 }
38 return won;
46 return tally;
3947 }
4048
4149 test("the trained policy wins more of the same duels than the bots it replaced", () => {
4250 const learned = duels("policy");
4351 const scripted = duels("scripted");
44 expect(scripted).toBeGreaterThan(0); // the baseline is not a straw man
45 expect(learned).toBeGreaterThan(scripted);
52 expect(scripted.won).toBeGreaterThan(0); // the baseline is not a straw man
53 expect(learned.won).toBeGreaterThan(scripted.won);
4654 // The measured margin is 72% against 38% over 240 duels; this asks for far
4755 // less, so it fails on a policy that regressed and not on ordinary variance.
48 expect(learned / DUELS).toBeGreaterThan(0.5);
56 expect(learned.won / DUELS).toBeGreaterThan(0.5);
57 });
58
59 /**
60 * And the claim this pull rests on: the same policy, on maps it has never
61 * seen. The game carves a layout out of the match id, so most of the maps it
62 * will ever be played on did not exist when it was trained; `maps.ts` holds a
63 * stream of seeds back from the training pool for exactly this measurement.
64 *
65 * What a confused bot does here is not lose — it draws. Six seconds of a duel
66 * go by, nobody dies, and from the other end of a corridor that is a bot that
67 * has forgotten what it was doing. On exactly these duels the weights before
68 * this pull won 68% and drew 14%, against 71% and 8% on de_forge; the ones
69 * this pull ships win 72% and draw 7%, which is the same fight on a map nobody
70 * wrote. The threshold below sits between the two, so a policy trained on one
71 * map again fails here.
72 */
73 const UNSEEN_DUELS = 192;
74
75 test("the trained policy fights on maps nobody wrote as well as on the one it knows", () => {
76 const maps = holdoutLevels(24);
77 const learned = duels("policy", UNSEEN_DUELS, maps);
78 const scripted = duels("scripted", UNSEEN_DUELS, maps);
79 expect(learned.won).toBeGreaterThan(scripted.won);
80 expect(learned.won / UNSEEN_DUELS).toBeGreaterThan(0.6);
81 // The fights it does not win are fights it lost, not fights it stood through.
82 expect(learned.draw / UNSEEN_DUELS).toBeLessThan(0.11);
4983 });
src/rl/maps.test.ts added +30
@@ -0,0 +1,30 @@
1 import { expect, test } from "bun:test";
2 import { flaws } from "../client/generate.ts";
3 import { DE_FORGE } from "../client/map.ts";
4 import { generatedLevels, holdoutLevels, HOLDOUT_SEED, trainingLevels, TRAINING_SEED } from "./maps.ts";
5
6 test("a run learns on de_forge and on many maps nobody wrote", () => {
7 const levels = trainingLevels(16);
8 expect(levels).toHaveLength(16);
9 expect(levels[0]).toBe(DE_FORGE);
10 const names = new Set(levels.map((l) => l.name));
11 expect(names.size).toBe(16); // sixteen different maps, not one map sixteen times
12 });
13
14 test("the maps a policy is graded on are maps it never trained on", () => {
15 const trained = new Set(trainingLevels(256).map((l) => l.name));
16 for (const level of holdoutLevels(16)) expect(trained.has(level.name)).toBe(false);
17 expect(TRAINING_SEED).not.toBe(HOLDOUT_SEED);
18 });
19
20 test("a seed is a map: the same call twice is the same level object", () => {
21 const first = generatedLevels(4, TRAINING_SEED);
22 const again = generatedLevels(4, TRAINING_SEED);
23 expect(again.map((l) => l.name)).toEqual(first.map((l) => l.name));
24 // Carved once and kept: an arena's per-level caches hang off this identity.
25 for (let i = 0; i < first.length; i++) expect(again[i]).toBe(first[i]);
26 });
27
28 test("every map in both streams is one a round can actually be played on", () => {
29 for (const level of [...trainingLevels(12), ...holdoutLevels(8)]) expect(flaws(level)).toEqual([]);
30 });
src/rl/maps.ts added +71
@@ -0,0 +1,71 @@
1 /**
2 * The maps a policy trains on, and the maps it is graded on.
3 *
4 * The game ships two kinds of level: de_forge, written by hand, and whatever
5 * `generate.ts` carves out of the match id — a layout nobody has ever seen,
6 * made the moment the round starts. A policy trained on one map learns that
7 * map: it wins its duels on de_forge and stalls on a generated one, because
8 * "the enemy went left" means something different in a room it has stood in a
9 * million times than in a corridor it has never seen. The fix is the ordinary
10 * one — train on many maps and grade on maps that were not among them — and
11 * this module is the list, so the trainer and the test that checks the shipped
12 * weights cannot disagree about which maps are which.
13 *
14 * Two streams of seeds, from two words, mixed by the same LCG the generator
15 * uses for its own retries. They are deterministic, so a run is repeatable and
16 * a test can ask for the eighth held-out map by name; and they are disjoint,
17 * which `maps.test.ts` checks by name rather than by trust — a held-out map
18 * that was in the training pool grades nothing.
19 */
20
21 import { generateLevel, seedFrom } from "../client/generate.ts";
22 import { DE_FORGE } from "../client/map.ts";
23 import type { Level } from "../client/level.ts";
24
25 /** Where each stream starts. Two words, hashed the way a match id is. */
26 export const TRAINING_SEED = seedFrom("train");
27 export const HOLDOUT_SEED = seedFrom("hold out");
28
29 /**
30 * Carving a map is 3.4 ms of flood fills and A*, and a training run asks for
31 * the same hundred maps every time an arena rolls onto one. They are made once
32 * and kept: a level is read-only once built, and the arena's own per-level
33 * caches hang off the identity of this object.
34 */
35 const carved = new Map<number, Level>();
36 const levelAt = (seed: number): Level => {
37 const known = carved.get(seed);
38 if (known !== undefined) return known;
39 const made = generateLevel(seed);
40 carved.set(seed, made);
41 return made;
42 };
43
44 /** The next seed in a stream: the LCG `generate.ts` retries a bad seed with. */
45 export const nextSeed = (seed: number): number => (Math.imul(seed, 1664525) + 1013904223) >>> 0;
46
47 /** `count` generated maps from the stream that starts at `seed`, in order. */
48 export function generatedLevels(count: number, seed: number): Level[] {
49 const out: Level[] = [];
50 let at = seed >>> 0;
51 for (let i = 0; i < count; i++) {
52 out.push(levelAt(at));
53 at = nextSeed(at);
54 }
55 return out;
56 }
57
58 /**
59 * What a run learns on: de_forge first, because it is the map the game is
60 * mostly played on and the one every number in the card has always been
61 * measured on, then `count - 1` maps from the training stream.
62 */
63 export const trainingLevels = (count = 64): Level[] =>
64 [DE_FORGE, ...generatedLevels(Math.max(0, count - 1), TRAINING_SEED)];
65
66 /**
67 * What a run is graded on: maps from the other stream, which no arena in a
68 * training run is ever put on. A win here is a win on a layout the policy has
69 * never seen, which is the whole claim this module exists to make checkable.
70 */
71 export const holdoutLevels = (count = 8): Level[] => generatedLevels(count, HOLDOUT_SEED);
src/terminal/keys.test.ts +1 −1
@@ -11,7 +11,7 @@ import type { MatchRow } from "../db.ts";
1111 import type { Data, View } from "./screen.ts";
1212 import { press, removed } from "./keys.ts";
1313
14 const CARD = { trainedAt: "2026-09-14", decisions: 1, envSteps: 1, updates: 1, stepsPerSecond: 1, minutes: 1, duels: 1, winRate: 1, baselineWinRate: 1 } satisfies Card;
14 const CARD = { trainedAt: "2026-09-14", decisions: 1, envSteps: 1, updates: 1, stepsPerSecond: 1, minutes: 1, duels: 1, winRate: 1, baselineWinRate: 1, maps: 1, unseenMaps: 1, unseenWinRate: 1, unseenBaselineWinRate: 1, unseenDrawRate: 1 } satisfies Card;
1515
1616 const match = (id: string, name: string): MatchRow => ({
1717 id,
src/terminal/screen.test.ts +11 −2
@@ -20,6 +20,11 @@ const CARD: Card = {
2020 duels: 240,
2121 winRate: 72,
2222 baselineWinRate: 38,
23 maps: 96,
24 unseenMaps: 8,
25 unseenWinRate: 69,
26 unseenBaselineWinRate: 41,
27 unseenDrawRate: 7,
2328 };
2429
2530 // A fixture is a signed-in player unless a test asks for a guest: `user_id`
@@ -124,10 +129,14 @@ test("the policy tab prints the card the bots were trained with", () => {
124129 expect(lines).toContain(" simulation ticks 8,192,000 in 12.3 min");
125130 expect(lines).toContain(" throughput 11,134 ticks a second");
126131 expect(lines).toContain(" PPO updates 1000");
127 expect(lines).toContain(" duels evaluated 240");
132 expect(lines).toContain(" maps learned on 96 (de_forge and 95 carved from seeds)");
133 expect(lines).toContain(" duels on de_forge 240");
128134 expect(lines).toContain(" won by the policy 72%");
129135 expect(lines).toContain(" won by the old bots 38%");
130 expect(lines).toContain(" retrain with: bun run scripts/train.ts --iters 1000");
136 expect(lines).toContain(" on maps it never saw 8 maps, 240 duels");
137 expect(lines).toContain(" won by the policy 69%, drawn 7%");
138 expect(lines).toContain(" won by the old bots 41%");
139 expect(lines).toContain(" retrain with: bun run scripts/train.ts --iters 1200");
131140 });
132141
133142 test("the last line is FORGE_APP_SSH verbatim, or says the terminal is off", () => {
src/terminal/screen.ts +6 −2
@@ -87,11 +87,15 @@ function policyLines(card: Card): string[] {
8787 line("simulation ticks", `${card.envSteps.toLocaleString("en")} in ${card.minutes} min`),
8888 line("throughput", `${card.stepsPerSecond.toLocaleString("en")} ticks a second`),
8989 line("PPO updates", String(card.updates)),
90 line("duels evaluated", String(card.duels)),
90 line("maps learned on", `${card.maps} (de_forge and ${card.maps - 1} carved from seeds)`),
91 line("duels on de_forge", String(card.duels)),
9192 line("won by the policy", `${card.winRate}%`),
9293 line("won by the old bots", `${card.baselineWinRate}%`),
94 line("on maps it never saw", `${card.unseenMaps} maps, ${card.duels} duels`),
95 line("won by the policy", `${card.unseenWinRate}%, drawn ${card.unseenDrawRate}%`),
96 line("won by the old bots", `${card.unseenBaselineWinRate}%`),
9397 "",
94 ` ${esc}2mretrain with: bun run scripts/train.ts --iters 1000${esc}0m`,
98 ` ${esc}2mretrain with: bun run scripts/train.ts --iters 1200${esc}0m`,
9599 ];
96100 }
97101
static/app.537d8dffb3.js deleted −58

Diff not shown — a line 71,613 characters wide (limit 2,000). −58

View file
static/app.c6a7c68916.js added +58

Diff not shown — a line 71,983 characters wide (limit 2,000). +58

View file

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs