Split The code is public; the app answers only to the crew.
`bun run scripts/measure.ts`: what a hundred seeded rounds say about the round the bots play #65
Check · Review · Land · Live done — Landed and live in production.
scout/cs:scout/45-bun-run-scripts-measure-ts-what-a-hundre → main 4 commits fork 2 threads · 2 unresolved
README.md +34
| @@ -29,6 +29,7 @@ bun run src/tty.ts # the terminal interface, same da | ||
| 29 | 29 | bun run scripts/train.ts --iters 1000 # retrain the bots' policy |
| 30 | 30 | bun run scripts/train.ts --iters 1000 --workers 4 # or on four cores |
| 31 | 31 | bun run scripts/train.ts --bench # how fast the arena steps |
| 32 | bun run scripts/measure.ts # what a hundred rounds say — see below | |
| 32 | 33 | |
| 33 | 34 | |
| 34 | 35 | The client is bundled at start-up, in memory, from the sources on disk — the |
| @@ -362,3 +363,36 @@ whether a run is possible, and plain PPO rather than a clever algorithm. Hence | ||
| 362 | 363 | the shape of `src/rl`: the same one-process PPO loop, an arena that steps |
| 363 | 364 | without allocating, and the parallelism in how the work is cut up rather than in |
| 364 | 365 | a framework. |
| 366 | ||
| 367 | ## Measuring the round | |
| 368 | ||
| 369 | `scripts/measure.ts` plays a handful of seeded matches headless — the person | |
| 370 | idle, the bots playing both sides — through the same `world.ts` a browser | |
| 371 | steps, and prints one table: how often the bomb goes down, how rounds end, and | |
| 372 | how long they take. `src/client/rounds.ts` does the arithmetic on the rounds | |
| 373 | it is handed and is what the tests check; the script only steps worlds and | |
| 374 | prints. Same flags, same seed, same table, so a change to the bots can be | |
| 375 | argued about with a diff: | |
| 376 | ||
| 377 | sh | |
| 378 | bun run scripts/measure.ts # 12 seeded SHORT matches | |
| 379 | bun run scripts/measure.ts --matches 40 --seed 7 # more matches, a different seed | |
| 380 | bun run scripts/measure.ts --rules long # COMPETITIVE rules instead | |
| 381 | bun run scripts/measure.ts --difficulty hard # harder bots instead of normal | |
| 382 | ||
| 383 | ||
| 384 | A round's length is measured from the freeze that starts it to the tick it is | |
| 385 | decided, so it is the freeze time plus whatever of the round clock and the | |
| 386 | bomb's fuse actually ran — not the 115-second round clock alone, which is why | |
| 387 | the longest round below is well past it. | |
| 388 | ||
| 389 | Baseline, at commit **861e20b**, the defaults (12 seeded `SHORT` matches, | |
| 390 | `normal` bots, 89 rounds): a plant in **60%** of rounds, ends split | |
| 391 | `detonated 38 · time 24 · elimination 15 · defused 12`, **23%** of planted | |
| 392 | bombs defused, median round **123 s**, longest **156 s**, and about 6 seconds | |
| 393 | of wall clock a match — which is why this is a script and not a test. The | |
| 394 | person is idle the whole match, so the count is effectively 5-on-4 in the | |
| 395 | terrorists' favour; treat the split between `detonated` and `elimination` | |
| 396 | accordingly. This baseline drifts whenever the bots or the blast do — it | |
| 397 | was 13% defused before #66 and #64 landed — so re-run it rather than trust | |
| 398 | it after a change to either. |
scripts/measure.test.ts added +34
| @@ -0,0 +1,34 @@ | ||
| 1 | import { expect, test } from "bun:test"; | |
| 2 | import { parseArgs, rulesFor } from "./measure.ts"; | |
| 3 | import { COMPETITIVE, SHORT } from "../src/client/match.ts"; | |
| 4 | ||
| 5 | test("no flags fall back to a short, quick, seeded default", () => { | |
| 6 | expect(parseArgs([])).toEqual({ matches: 12, seed: 20260917, rulesName: "short", difficulty: "normal" }); | |
| 7 | }); | |
| 8 | ||
| 9 | test("matches, seed, rules and difficulty are read off the command line", () => { | |
| 10 | expect(parseArgs(["--matches", "40", "--seed", "7", "--rules", "long", "--difficulty", "hard"])).toEqual({ | |
| 11 | matches: 40, seed: 7, rulesName: "long", difficulty: "hard", | |
| 12 | }); | |
| 13 | }); | |
| 14 | ||
| 15 | test("a malformed or missing value falls back to the default rather than NaN", () => { | |
| 16 | expect(parseArgs(["--matches", "nope"]).matches).toBe(12); | |
| 17 | expect(parseArgs(["--matches", "-3"]).matches).toBe(12); | |
| 18 | expect(parseArgs(["--rules", "medium"]).rulesName).toBe("short"); | |
| 19 | expect(parseArgs(["--seed"]).seed).toBe(20260917); | |
| 20 | expect(parseArgs(["--difficulty", "extreme"]).difficulty).toBe("normal"); | |
| 21 | expect(parseArgs(["--difficulty"]).difficulty).toBe("normal"); | |
| 22 | }); | |
| 23 | ||
| 24 | test("a negative or non-integer seed falls back the same way a negative matches count does", () => { | |
| 25 | expect(parseArgs(["--seed", "-3"]).seed).toBe(20260917); | |
| 26 | expect(parseArgs(["--seed", "1.5"]).seed).toBe(20260917); | |
| 27 | expect(parseArgs(["--seed", "nope"]).seed).toBe(20260917); | |
| 28 | expect(parseArgs(["--seed", "0"]).seed).toBe(0); // zero is a fine seed, not a falsy one | |
| 29 | }); | |
| 30 | ||
| 31 | test("rulesFor maps the two names to the rule sets main.ts already uses", () => { | |
| 32 | expect(rulesFor("short")).toBe(SHORT); | |
| 33 | expect(rulesFor("long")).toBe(COMPETITIVE); | |
| 34 | }); |
scripts/measure.ts added +118
| @@ -0,0 +1,118 @@ | ||
| 1 | /** | |
Approved at The fixes hold. Re-verified at this head. It already earns its keep. Running the same defaults on the merge with today's main gives 89 rounds, plant 60%, The two comments reading I read every changed line, ran the suite at this head (90 pass, 0 fail), walked the preview as a guest, and ran Two things to do before this lands. 1. Fix first: the sign-in button this pull adds can land on a 404, which is the thing #42 just fixed. 2. It does not merge. Head Smaller, take or leave:
Nothing here is a data leak and nothing needs rethinking; fix the sign-in target, merge main green, and I will approve. Disregard my previous comment on this pull ( Approved. I read every changed line, ran the pull's head out of tree (the clone untouched), and exercised Verified
The round accounting reads correctly. Every round passes through It does what #45 asked and stops there: no bot behaviour changed, no CI gate on the numbers, nothing about browser frame time. The added Nits, none blocking, none worth another round:
The preview host 404s at |
||
| 2 | * Measures what the bots' rounds are actually like: how often the bomb goes | |
| 3 | * down, how rounds end, how long they take. Plays N seeded matches headless | |
| 4 | * through `world.update`, the person idle (`NO_INPUT`) the whole way, and | |
| 5 | * prints one table — the arithmetic itself lives in `src/client/rounds.ts` | |
| 6 | * and is what the tests check. | |
| 7 | * | |
| 8 | * bun run scripts/measure.ts # 12 seeded SHORT matches | |
| 9 | * bun run scripts/measure.ts --matches 40 --seed 7 # more matches, a different seed | |
| 10 | * bun run scripts/measure.ts --rules long # COMPETITIVE rules instead | |
| 11 | * bun run scripts/measure.ts --difficulty hard # harder bots instead of normal | |
| 12 | * | |
| 13 | * Nothing here is imported by the game or the server. A round is recorded as | |
| 14 | * plain numbers as it finishes — never a whole `World` held onto for later — | |
| 15 | * so a hundred rounds cost a hundred small objects, not a hundred worlds. | |
| 16 | */ | |
| 17 | ||
| 18 | import { COMPETITIVE, SHORT, type Rules } from "../src/client/match.ts"; | |
| 19 | import { | |
| 20 | DEFAULT_DIFFICULTY, DIFFICULTIES, NO_INPUT, createWorld, difficultyLabel, update, type Difficulty, | |
| 21 | } from "../src/client/world.ts"; | |
| 22 | import { reportLines, summarize, type RoundRecord } from "../src/client/rounds.ts"; | |
| 23 | ||
| 24 | export type RulesName = "short" | "long"; | |
| 25 | export type Options = { matches: number; seed: number; rulesName: RulesName; difficulty: Difficulty }; | |
| 26 | ||
| 27 | const DEFAULTS: Options = { matches: 12, seed: 20260917, rulesName: "short", difficulty: DEFAULT_DIFFICULTY }; | |
| 28 | ||
| 29 | /** `--matches 40 --seed 7 --rules long --difficulty hard`, missing or malformed flags keep the default. */ | |
| 30 | export function parseArgs(argv: string[]): Options { | |
| 31 | const value = (name: string): string | null => { | |
| 32 | const at = argv.indexOf(`--${name}`); | |
| 33 | return at < 0 ? null : (argv[at + 1] ?? null); | |
| 34 | }; | |
| 35 | const matches = value("matches") === null ? NaN : Number(value("matches")); | |
| 36 | const seed = value("seed") === null ? NaN : Number(value("seed")); | |
| 37 | const rulesArg = value("rules"); | |
| 38 | const difficultyArg = value("difficulty"); | |
| 39 | return { | |
| 40 | matches: Number.isFinite(matches) && matches > 0 ? Math.round(matches) : DEFAULTS.matches, | |
| 41 | seed: Number.isInteger(seed) && seed >= 0 ? seed : DEFAULTS.seed, | |
| 42 | rulesName: rulesArg === "long" ? "long" : rulesArg === "short" ? "short" : DEFAULTS.rulesName, | |
| 43 | difficulty: isDifficulty(difficultyArg) ? difficultyArg : DEFAULTS.difficulty, | |
| 44 | }; | |
| 45 | } | |
| 46 | ||
| 47 | export const rulesFor = (name: RulesName): Rules => (name === "long" ? COMPETITIVE : SHORT); | |
| 48 | ||
| 49 | const isDifficulty = (value: string | null): value is Difficulty => | |
| 50 | DIFFICULTIES.includes(value as Difficulty); | |
| 51 | ||
| 52 | const TICK = 1 / 64; | |
| 53 | ||
| 54 | /** | |
| 55 | * The longest a match can honestly take, with room to spare: every round at | |
| 56 | * its worst (freeze, the full round clock, the bomb's full fuse, the delay | |
| 57 | * after), times the most rounds a best-of-`2 * winScore - 1` can ask for. | |
| 58 | * A guard against a bug turning this into an infinite loop rather than a bad | |
| 59 | * table, not a budget anyone is expected to spend. | |
| 60 | */ | |
| 61 | function maxTicksFor(rules: Rules): number { | |
| 62 | const worstRound = rules.freezeTime + rules.roundTime + rules.bombTime + rules.endDelay; | |
| 63 | const worstRounds = 2 * rules.winScore - 1; | |
| 64 | return Math.ceil((worstRound * worstRounds * 1.5) / TICK); | |
| 65 | } | |
| 66 | ||
| 67 | /** Play one seeded match to its end, the person idle, and return its rounds. */ | |
| 68 | function playMatch(rules: Rules, seed: number, difficulty: Difficulty): RoundRecord[] { | |
| 69 | const world = createWorld(rules, "measure", "ct", 5, seed, difficulty); | |
| 70 | const rounds: RoundRecord[] = []; | |
| 71 | let roundStart = 0; // world.time at the freeze that started the round in play | |
| 72 | let prevPhase = world.match.phase; | |
outdated
probe |
||
| 73 | const maxTicks = maxTicksFor(rules); | |
| 74 | for (let tick = 0; tick < maxTicks && world.match.phase !== "match_over"; tick++) { | |
| 75 | update(world, TICK, NO_INPUT); | |
| 76 | const phase = world.match.phase; | |
| 77 | if (phase === "over" && prevPhase !== "over") { | |
| 78 | const { reason } = world.match; | |
| 79 | if (reason === null) throw new Error("a round went over with no reason set"); | |
| 80 | rounds.push({ reason, planted: world.bombPlanted, seconds: world.time - roundStart }); | |
| 81 | } | |
| 82 | if (phase === "freeze" && prevPhase === "over") roundStart = world.time; | |
| 83 | prevPhase = phase; | |
| 84 | } | |
| 85 | if (world.match.phase !== "match_over") { | |
| 86 | throw new Error(`match seeded ${seed} did not finish inside ${maxTicks} ticks`); | |
| 87 | } | |
| 88 | return rounds; | |
| 89 | } | |
| 90 | ||
| 91 | function main(): void { | |
| 92 | const opts = parseArgs(process.argv.slice(2)); | |
| 93 | const rules = rulesFor(opts.rulesName); | |
| 94 | console.log( | |
| 95 | `${opts.matches} seeded ${opts.rulesName.toUpperCase()} matches, seed ${opts.seed}, ` + | |
| 96 | `${difficultyLabel(opts.difficulty)} bots, the person idle`, | |
| 97 | ); | |
| 98 | ||
| 99 | const rounds: RoundRecord[] = []; | |
| 100 | const started = Bun.nanoseconds(); | |
| 101 | for (let i = 0; i < opts.matches; i++) { | |
| 102 | const before = Bun.nanoseconds(); | |
| 103 | // Spread out the same way scripts/train.ts spreads its duels: a prime | |
| 104 | // stride so nearby matches do not share a seed's low bits. | |
| 105 | const matchRounds = playMatch(rules, opts.seed + i * 7919, opts.difficulty); | |
| 106 | rounds.push(...matchRounds); | |
| 107 | const seconds = (Bun.nanoseconds() - before) / 1e9; | |
| 108 | console.log(` match ${String(i + 1).padStart(3)}/${opts.matches} ${matchRounds.length} rounds ${seconds.toFixed(1)}s`); | |
| 109 | } | |
| 110 | const totalSeconds = (Bun.nanoseconds() - started) / 1e9; | |
| 111 | ||
| 112 | console.log(""); | |
| 113 | for (const line of reportLines(summarize(rounds))) console.log(line); | |
| 114 | console.log(""); | |
| 115 | console.log(`${totalSeconds.toFixed(1)}s wall clock for ${opts.matches} matches (${(totalSeconds / opts.matches).toFixed(1)}s a match)`); | |
| 116 | } | |
| 117 | ||
| 118 | if (import.meta.main) main(); |
src/client/rounds.test.ts added +69
| @@ -0,0 +1,69 @@ | ||
| 1 | import { expect, test } from "bun:test"; | |
| 2 | import { reportLines, summarize, type RoundRecord } from "./rounds.ts"; | |
| 3 | ||
| 4 | const round = (reason: RoundRecord["reason"], planted: boolean, seconds: number): RoundRecord => | |
| 5 | ({ reason, planted, seconds }); | |
| 6 | ||
| 7 | test("an empty list summarizes to zeros, not NaN or a thrown error", () => { | |
| 8 | const report = summarize([]); | |
| 9 | expect(report).toEqual({ | |
| 10 | rounds: 0, | |
| 11 | plantRate: 0, | |
| 12 | reasons: { elimination: 0, detonated: 0, defused: 0, time: 0 }, | |
| 13 | defuseRate: 0, | |
| 14 | medianSeconds: 0, | |
| 15 | longestSeconds: 0, | |
| 16 | }); | |
| 17 | }); | |
| 18 | ||
| 19 | test("the median of an even count is the average of the middle two", () => { | |
| 20 | const rounds = [ | |
| 21 | round("time", false, 100), | |
| 22 | round("time", false, 120), | |
| 23 | round("time", false, 140), | |
| 24 | round("time", false, 200), | |
| 25 | ]; | |
| 26 | expect(summarize(rounds).medianSeconds).toBe(130); | |
| 27 | }); | |
| 28 | ||
| 29 | test("the median of an odd count is the middle value, order does not matter", () => { | |
| 30 | const rounds = [ | |
| 31 | round("time", false, 200), | |
| 32 | round("time", false, 100), | |
| 33 | round("time", false, 150), | |
| 34 | ]; | |
| 35 | expect(summarize(rounds).medianSeconds).toBe(150); | |
| 36 | }); | |
| 37 | ||
| 38 | test("a reason that never occurred is counted zero, not left out", () => { | |
| 39 | const rounds = [round("time", false, 100), round("time", false, 110)]; | |
| 40 | const report = summarize(rounds); | |
| 41 | expect(report.reasons).toEqual({ elimination: 0, detonated: 0, defused: 0, time: 2 }); | |
| 42 | }); | |
| 43 | ||
| 44 | test("plant rate and defuse rate: defuse is of planted rounds, not of all rounds", () => { | |
| 45 | const rounds = [ | |
| 46 | round("detonated", true, 130), | |
| 47 | round("defused", true, 140), | |
| 48 | round("time", false, 115), | |
| 49 | round("elimination", false, 80), | |
| 50 | ]; | |
| 51 | const report = summarize(rounds); | |
| 52 | expect(report.rounds).toBe(4); | |
| 53 | expect(report.plantRate).toBe(0.5); // 2 of 4 | |
| 54 | expect(report.defuseRate).toBe(0.5); // 1 of 2 planted | |
| 55 | expect(report.longestSeconds).toBe(140); | |
| 56 | }); | |
| 57 | ||
| 58 | test("no bombs planted leaves the defuse rate at zero, not NaN", () => { | |
| 59 | const rounds = [round("time", false, 100), round("elimination", false, 90)]; | |
| 60 | expect(summarize(rounds).defuseRate).toBe(0); | |
| 61 | }); | |
| 62 | ||
| 63 | test("the report renders to one line a stat, with every reason named", () => { | |
| 64 | const rounds = [round("detonated", true, 130), round("defused", true, 90)]; | |
| 65 | const lines = reportLines(summarize(rounds)); | |
| 66 | expect(lines.some((l) => l.includes("rounds played") && l.includes("2"))).toBe(true); | |
| 67 | expect(lines.some((l) => l.includes("detonated 1") && l.includes("defused 1"))).toBe(true); | |
| 68 | expect(lines.some((l) => l.includes("plant rate") && l.includes("100%"))).toBe(true); | |
| 69 | }); |
src/client/rounds.ts added +67
| @@ -0,0 +1,67 @@ | ||
| 1 | /** | |
| 2 | * What a played-out round adds up to. `scripts/measure.ts` plays matches | |
| 3 | * headless and hands the finished rounds here; this module only does | |
| 4 | * arithmetic on the list it is given, so the arithmetic can be tested without | |
| 5 | * ever stepping a world. | |
| 6 | */ | |
| 7 | ||
| 8 | import type { EndReason } from "./match.ts"; | |
| 9 | ||
| 10 | /** One round, boiled down to the numbers a report needs and nothing else. */ | |
| 11 | export type RoundRecord = { | |
| 12 | reason: EndReason; | |
| 13 | planted: boolean; // did the bomb go down this round | |
| 14 | seconds: number; // freeze start to the round being decided | |
| 15 | }; | |
| 16 | ||
| 17 | export type Report = { | |
| 18 | rounds: number; | |
| 19 | plantRate: number; // fraction of rounds the bomb went down, 0 if none played | |
| 20 | reasons: Record<EndReason, number>; | |
| 21 | defuseRate: number; // fraction of planted bombs that were defused, 0 if none planted | |
| 22 | medianSeconds: number; | |
| 23 | longestSeconds: number; | |
| 24 | }; | |
| 25 | ||
| 26 | const REASONS: EndReason[] = ["elimination", "detonated", "defused", "time"]; | |
| 27 | ||
| 28 | function median(values: number[]): number { | |
| 29 | if (values.length === 0) return 0; | |
| 30 | const sorted = [...values].sort((a, b) => a - b); | |
| 31 | const mid = Math.floor(sorted.length / 2); | |
| 32 | return sorted.length % 2 === 0 | |
| 33 | ? ((sorted[mid - 1] as number) + (sorted[mid] as number)) / 2 | |
| 34 | : (sorted[mid] as number); | |
| 35 | } | |
| 36 | ||
| 37 | export function summarize(rounds: RoundRecord[]): Report { | |
| 38 | const reasons = Object.fromEntries(REASONS.map((r) => [r, 0])) as Record<EndReason, number>; | |
| 39 | let planted = 0; | |
| 40 | for (const round of rounds) { | |
| 41 | reasons[round.reason] += 1; | |
| 42 | if (round.planted) planted += 1; | |
| 43 | } | |
| 44 | const seconds = rounds.map((r) => r.seconds); | |
| 45 | return { | |
| 46 | rounds: rounds.length, | |
| 47 | plantRate: rounds.length === 0 ? 0 : planted / rounds.length, | |
| 48 | reasons, | |
| 49 | defuseRate: planted === 0 ? 0 : reasons.defused / planted, | |
| 50 | medianSeconds: median(seconds), | |
| 51 | longestSeconds: seconds.length === 0 ? 0 : Math.max(...seconds), | |
| 52 | }; | |
| 53 | } | |
| 54 | ||
| 55 | const pct = (fraction: number): string => `${Math.round(fraction * 100)}%`; | |
| 56 | ||
| 57 | /** The report as one line a stat, the table `measure.ts` prints. */ | |
| 58 | export function reportLines(report: Report): string[] { | |
| 59 | return [ | |
| 60 | `rounds played ${report.rounds}`, | |
| 61 | `plant rate ${pct(report.plantRate)}`, | |
| 62 | `ends detonated ${report.reasons.detonated} · time ${report.reasons.time} · ` + | |
| 63 | `elimination ${report.reasons.elimination} · defused ${report.reasons.defused}`, | |
| 64 | `defuse rate ${pct(report.defuseRate)} of planted bombs`, | |
| 65 | `round length median ${report.medianSeconds.toFixed(0)}s, longest ${report.longestSeconds.toFixed(0)}s`, | |
| 66 | ]; | |
| 67 | } |
Approved. I read all 294 lines, ran the script, and checked the branch merges clean onto current main.
What I verified
bun run scripts/measure.tsat this branch's base (fccb9eb) printed exactly the table the README claims: 79 rounds, plant 58%,detonated 37 · time 20 · elimination 16 · defused 6, defuse 13% of planted, median 123s, longest 156s, 5.4s a match. Two runs at one seed differ only in the wall-clock lines, so the determinism the issue asked for holds.--rules longworks too: 25 rounds at most, ~16s a match, which is why it is not the default.fccb9eband main has since moved two ahead (#52, #42). Merged locally: no conflicts, 306 pass / 0 fail in 6.26s, against 6.20s for main alone, so these tests cost about 60ms.update(world, dt, input)andworld.bombPlantedare untouched by either commit, soplayMatchstill steps the world main ships.src/importsrounds.tsormeasure.ts, and the built client is byte-identical on main and on the merge (static/app.8f08f8416e.jsboth times). No route, no request handling, no identity, no disk write, no dependency, no shell, noeval— the security floor and the terminal are simply not in this change's path, and there is no new capability forsrc/tty.tsto mirror.startRoundclearsworld.bombPlanted, and a record is pushed on the tick the phase becomes"over", before the next freeze — so a detonated round still counts as planted.roundStartis set onover → freezeand starts at 0 for the first round, so no round's clock swallows the previousendDelay.maxTicksForbounds a stuck match rather than looping forever, and2 * winScore - 1is the real ceiling (I saw exactly 9 and 25).Nits, none blocking, worth a line if this is touched again
--seed -3is silently turned into 4294967293 byseed >>> 0, while--matches -3falls back to the default — two halves of one contract disagreeing. Rejecting a negative or non-integer seed the same way is one condition.freezeTime(8s) and the 40s fuse, which is how it exceeds the 115s round clock. The pull description says this; the README section does not, and that is where the next reader will look.world.match.reason as EndReasonis the change's one unchecked cast; the type isEndReason | nulland it only happens to always be set at"over".Two things that are not yours. The preview at https://cs--65--forge-trevato-apps.open-platform.sh answers a bare
404 not foundwith no security headers from here, which is the edge rather than the app: the app's own 404 is HTML carrying the full floor, and booting the merge locally gives/at 200 with CSP, nosniff, Referrer-Policy and an HttpOnly session cookie intact, while the forge's pictures at this head show the page healthy on phone and desktop. And two comments readingprobe— one onscripts/measure.tsline 72, one on the thread — are mine, left by a call while I worked out this API; there is no delete route, so please ignore them.