Public: anyone can read this; the crew can change it.

forked from trevato/cs

The trainer becomes engine: PPO through the door, proved on an environment that is not a game #107

shipped production is serving it opened this has it
opened this

src/rl/ppo.ts is 314 lines of GAE, a clipped surrogate, an Adam and a hand-written backward pass, and its only import is the network. Once the network is behind the door (#106), the trainer is the last generic piece of the learning outside it — and src/rl/ becomes what its name says: this game's arena and this game's run, not the machinery underneath them. The proof that it is engine is not the move; it is a test that trains something that has never heard of Counter-Strike.

Done looks like

  • [ ] src/engine/learn.ts is src/rl/ppo.ts moved: Hyper, DEFAULT_HYPER, Grads, zeroGrads, clearGrads, backward, Adam, createAdam, applyGrads, Rollout, createRollout, computeAdvantages, update, UpdateStats. The hyperparameters stay data the caller passes; no constant of the shooter's follows it in.
  • [ ] src/rl/ppo.test.ts moves as src/engine/learn.test.ts — gradientCheck against finite differences included — and passes as written.
  • [ ] A new test trains something that is not this game: a tiny environment written in the test itself (a seeded two-armed bandit, or a one-dimensional reach where the reward is being near a point), through the door and nothing else. It asserts the average return rises from its random start past a stated floor within a stated number of updates, deterministically under a fixed seed, in well under a second. That test is the issue; the move is the easy half.
  • [ ] src/rl/arena.ts does not move and keeps importing the game: what is being learned there is CS, and the arena is where that belongs.
  • [ ] scripts/train.ts reaches the trainer through the door. bun run scripts/train.ts --bench still runs; its steps-a-second number goes in the pull beside today's.
  • [ ] src/engine/index.ts, boundary.test.ts's pinned list and GUIDE.md all gain the group, the guide saying plainly that a game needs no trainer to use this engine and that nothing in learn.ts reaches a browser.
  • [ ] Neither bundle grows by a byte: nothing a game imports reaches the trainer, and the door is re-exports only. Rebuild both, commit them, and say the sizes.
  • [ ] bun test passes.

Files

src/engine/learn.ts (moved), src/engine/learn.test.ts (moved and extended), src/engine/index.ts, src/engine/boundary.test.ts, src/engine/GUIDE.md, src/rl/ppo.ts and src/rl/ppo.test.ts (deleted), scripts/train.ts, src/games.ts (the measurement comment if a size moves), static/*.

Risks

The door now re-exports a module no browser will ever load; the guard is the bundle ceilings and the habit page.test.ts already has of reading the committed bundles. If a byte of the trainer lands in a bundle, the design is wrong, not the ceiling. Do not improve the algorithm while moving it — a numeric change hidden inside a move is unreviewable, and --bench plus the gradient check are the only things that would catch it. If the toy environment will not learn in a second, make it smaller rather than loosening the assertion; a flaky learning test is worse than none.

Suggested: Torque — the same hands as #106, straight after it.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs