Split The code is public; the app answers only to the crew.

rl #7

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

Clean tree, 122 tests passing, the app boots, / answers in 7ms with the security floor intact, and the branch is pushed. Training was one thread and one opponent — itself, ten updates ago — and both were the ceiling on how good the bots could get in the minutes a run has.

Notes

shipped merged

opened this

Clean tree, 122 tests passing, the app boots, / answers in 7ms with the security floor intact, and the branch is pushed.

Training was one thread and one opponent — itself, ten updates ago — and both were the ceiling on how good the bots could get in the minutes a run has. This change splits a PPO update into shards: src/rl/shard.ts owns a range of environments and writes its own rows of one shared rollout, then differentiates its own slice of each minibatch; src/rl/worker.ts is one shard on a thread, and src/rl/pool.ts runs them while keeping the minibatch plan, the advantage normalisation and the Adam step on the main thread. Every buffer that crosses a thread is a view onto a SharedArrayBuffer and every random draw is keyed to its environment, so the experience collected is identical bit for bit whether it is one shard or sixteen — src/rl/pool.test.ts holds it to that against both one shard and real workers, and checks that the summed gradient is the same sum. src/rl/cores.ts reads the cgroup quota rather than trusting hardwareConcurrency, because this container reports 80 cores and is allowed two, and oversubscribing a quota does not fail to help, it hurts: measured on whole updates, 9k ticks a second on one thread, 17k across two, 5k across eight. A league of four frozen selves replaced the single snapshot opponent.

The shipped weights are retrained on that: 1500 updates, 12.3M ticks in 12.2 minutes, 1.5× the experience of the previous run in the same wall clock. On the 240 fixed duels the two measure the same, 72% won against the scripted bots' 42% — that benchmark looks near its ceiling, a third of its losses being six-second draws — so the evidence for the swap is head to head: over 600 duels, 300 from each side of the m4a4/ak47 asymmetry, the new policy takes 332 and the old one 231.

Two things to look at in the preview. The terminal's policy tab was drawn and named in the tab bar but the tab key only cycled standings and matches, so nothing could reach it; src/tabs.ts is now the order the key walks, with a test that every tab is reachable — press <kbd>Tab</kbd> twice to see the card, which now also reports the shard count. And world.test.ts's plant test stood a motionless player on a site for three seconds with five live CT bots: the better policy kills them a second in, so the test had been measuring the firefight rather than the plant, and it now keeps the planter on their feet and checks the plant. That is a test I changed because it was wrong, not code I bent to it.

What I did not do: the game still plays the policy greedily against scripted-skill handicaps rather than league opponents, the arena is still a two-player duel with no round or economy, and --workers past the quota is left as a knob rather than refused — it is slower, not broken, and the README says so. <https://puffer.ai/> is still unreachable from this sandbox, so the influence remains PufferLib's shape rather than its docs: flat observation and action spaces, throughput as the thing that decides whether a run is possible, plain PPO.

🤖 Generated with Claude Code

Closes #3

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs