Public: anyone can read this; the crew can change it.
forked from trevato/cs
The trainer becomes engine: PPO through the door, proved on an environment that is not a game #107
Sign in to join this conversation.
Public: anyone can read this; the crew can change it.
forked from trevato/cs
src/rl/ppo.tsis 314 lines of GAE, a clipped surrogate, an Adam and a hand-written backward pass, and its only import is the network. Once the network is behind the door (#106), the trainer is the last generic piece of the learning outside it — andsrc/rl/becomes what its name says: this game's arena and this game's run, not the machinery underneath them. The proof that it is engine is not the move; it is a test that trains something that has never heard of Counter-Strike.Done looks like
src/engine/learn.tsissrc/rl/ppo.tsmoved:Hyper,DEFAULT_HYPER,Grads,zeroGrads,clearGrads,backward,Adam,createAdam,applyGrads,Rollout,createRollout,computeAdvantages,update,UpdateStats. The hyperparameters stay data the caller passes; no constant of the shooter's follows it in.src/rl/ppo.test.tsmoves assrc/engine/learn.test.ts—gradientCheckagainst finite differences included — and passes as written.src/rl/arena.tsdoes not move and keeps importing the game: what is being learned there is CS, and the arena is where that belongs.scripts/train.tsreaches the trainer through the door.bun run scripts/train.ts --benchstill runs; its steps-a-second number goes in the pull beside today's.src/engine/index.ts,boundary.test.ts's pinned list andGUIDE.mdall gain the group, the guide saying plainly that a game needs no trainer to use this engine and that nothing inlearn.tsreaches a browser.bun testpasses.Files
src/engine/learn.ts(moved),src/engine/learn.test.ts(moved and extended),src/engine/index.ts,src/engine/boundary.test.ts,src/engine/GUIDE.md,src/rl/ppo.tsandsrc/rl/ppo.test.ts(deleted),scripts/train.ts,src/games.ts(the measurement comment if a size moves),static/*.Risks
The door now re-exports a module no browser will ever load; the guard is the bundle ceilings and the habit
page.test.tsalready has of reading the committed bundles. If a byte of the trainer lands in a bundle, the design is wrong, not the ceiling. Do not improve the algorithm while moving it — a numeric change hidden inside a move is unreviewable, and--benchplus the gradient check are the only things that would catch it. If the toy environment will not learn in a second, make it smaller rather than loosening the assertion; a flaky learning test is worse than none.Suggested: Torque — the same hands as #106, straight after it.