Split The code is public; the app answers only to the crew.

rl #4

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

Pushed, working tree clean, 106 tests passing, / answers in 6 ms with the trained weights in the bundle. --- The bots now learn to fight instead of being told how.

Notes

shipped merged

opened this

Pushed, working tree clean, 106 tests passing, / answers in ~6 ms with the trained weights in the bundle.

---

The bots now learn to fight instead of being told how. The round's plan stays scripted — pathing, buying, planting, defusing, all of which A\* already solved — and the fight becomes a policy: 32 egocentric inputs (src/client/observe.ts), six small categorical heads (src/client/action.ts), a two-layer network (src/client/policy.ts) trained with PPO in src/rl/ against a duel arena built on the real map, movement, bullets and damage with the round and economy removed.

The run behind the shipped weights: 8,192,000 simulation ticks in 12 minutes on one core, 1,000 updates, about 11k ticks a second. On 240 fixed duels — same seeds, same spawns, same opponent skills for both drivers — the int8 policy wins 72% where the scripted bots it replaced win 38% of the same duels. Those numbers live in src/client/weights.ts and are what the page and the terminal's new policy tab print; src/rl/learned.test.ts re-runs a 40-duel version so a regression fails the check rather than the browser.

Worth looking at first: the reward in src/rl/arena.ts, which took four wrong turns before it worked — discounted shaping paid bots to hide, zeroing the potential at a kill clawed back the kill reward, a per-shot ammunition penalty taught the fire head never to fire, and a strong opponent from step one teaches hiding permanently (hence the curriculum on range and on opponent skill). Then src/client/bots.ts, where skill became a handicap on the hands rather than on the policy — turn rate, a wobble the bot cannot see, the reaction beat — so the ladder from fodder to uncomfortable survives, and where the pre-existing full-round test caught the one thing a duel arena could not: a policy trained on rifles holds the trigger down, which on a semi-automatic pistol is one shot and then silence.

Not done, deliberately: reinforcement learning on the round itself. Objectives are a 115-second sparse-reward problem needing far more compute than a preview deploy should cost, and the scripted plan beats a badly-trained one. scriptedFight stays in bots.ts unused by the game, because it is both the arena's opponent and the baseline — a baseline that moves with the policy measures nothing. No dependencies were added. <https://puffer.ai/> is not reachable from this sandbox (the network here is the forge and the package registry), so the influence is from what PufferLib is known for rather than its documentation — flat observation and action spaces, environment throughput as the thing that decides whether a run is possible, plain PPO over a clever algorithm — and the README says so rather than implying I read it.

🤖 Generated with Claude Code

Budget reached at $10.24; this is where it stopped.

Closes #3

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs