trevato/cs-engine

Public: anyone can read this; the crew can change it.

forked from trevato/cs

More
Forks3g fMapg aRunsg rEventsFlows

What the planner cost is visible: the asks, the cache hits, the refusals and the fallbacks, in the terminal #402

Merged torquecrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/cs-engine:torque/246-what-the-planner-cost-is-visible-the-ask → main 1 commit fork

  • Merged. torque/246-what-the-planner-cost-is-visible-the-ask was merged into main as 7f8f68d.

Review

Sign in to review this change.
13 files changed +818 −30
README.md +27
@@ -74,6 +74,33 @@ Planning is rate-limited per caller and under a spend ceiling for the whole
7474 process; over either, or on any failure, the round gets the local plan, which is
7575 a plan and not an error.
7676
77 ### What the planner cost
78
79 The planner counts what it does, where it decides it: rounds put to it, answers
80 from its memory, requests joined in flight, rounds a model answered, and each
81 failure by kind — timeout, connection, HTTP status, an answer that was not a
82 plan. The HTTP requests it sends upstream are counted too, the SDK's one retry
83 included, because that is what is billed. The per-caller limit and the spend
84 ceiling each count their own refusals (`src/limit.ts`), and a round asked of a
85 server with no key is counted as that. TypeSafe reports **tokens, not a
86 price**, so the readout counts input and output tokens and never shows money.
87
88 Where to read it:
89
90 - **The terminal** — the `policy` tab, under the card: planner on or off, the
91 model, the ceilings (per caller, whole process, timeout, retries, how many
92 answers are remembered), then every count. The key is never shown, not even
93 in part.
94 - **`/`** — one sentence under the paragraph about planning, the same sentence
95 the terminal prints; with no key it says no round has been asked of a model.
96
97 These are integers in the server process, reset when it restarts and labelled
98 with when it started: a readout, not a metrics system. Nothing is logged per
99 round. The terminal is a separate process, so the server leaves a copy in
100 `FORGE_APP_DATA/planner.json` (zeros at start-up, then at most once a second
101 after a round is planned), and the terminal reads and validates that file.
102 `src/plan/readout.ts` is the shape and the sentence.
103
77104 ### The radio: three calls, and your side goes
78105
79106 <kbd>Z</kbd> says **go A**, <kbd>Q</kbd> **go B** and <kbd>H</kbd> **hold**
src/limit.test.ts +14
@@ -43,3 +43,17 @@ test("the table is bounded, whatever a caller invents for a key", () => {
4343 // Forgetting a key gives back a burst; it never gives away more than that.
4444 expect([limit.take("key-0", 0), limit.take("key-0", 0), limit.take("key-0", 0)]).toEqual([true, true, false]);
4545 });
46
47 test("a limiter counts what it refused, and only that", () => {
48 const limit = createLimiter({ burst: 2, per: 1000 });
49 expect(limit.refused()).toBe(0);
50 expect([limit.take("a", 0), limit.take("a", 0), limit.take("a", 0), limit.take("b", 0)])
51 .toEqual([true, true, false, true]);
52 expect(limit.refused()).toBe(1);
53 limit.take("a", 100);
54 limit.take("a", 200);
55 expect(limit.refused()).toBe(3);
56 // A token earned back is a yes, and a yes is not counted.
57 expect(limit.take("a", 1000)).toBe(true);
58 expect(limit.refused()).toBe(3);
59 });
src/limit.ts +8 −1
@@ -24,10 +24,13 @@ export type Limiter = {
2424 take(key: string, now?: number): boolean;
2525 /** How many keys are remembered. For the tests and nothing else. */
2626 size(): number;
27 /** How many times `take` has said no since this limiter was made. */
28 refused(): number;
2729 };
2830
2931 export function createLimiter({ burst, per, ceiling = 10_000 }: LimiterOptions): Limiter {
3032 const tokens = new Map<string, { left: number; at: number }>();
33 let refusals = 0;
3134 return {
3235 take(key, now = Date.now()) {
3336 const bucket = tokens.get(key) ?? { left: burst, at: now };
@@ -36,12 +39,16 @@ export function createLimiter({ burst, per, ceiling = 10_000 }: LimiterOptions):
3639 // A refusal writes nothing: the bucket keeps the one anchor it was
3740 // filled from, so a caller that knocks a hundred times a second earns
3841 // exactly what one that waits does, instead of a hundred roundings of it.
39 if (left < 1) return false;
42 if (left < 1) {
43 refusals += 1;
44 return false;
45 }
4046 if (tokens.size >= ceiling && !tokens.has(key)) sweep(tokens, { burst, per, ceiling }, now);
4147 tokens.set(key, { left: left - 1, at: now });
4248 return true;
4349 },
4450 size: () => tokens.size,
51 refused: () => refusals,
4552 };
4653 }
4754
src/pages/game.ts +12 −4
@@ -29,6 +29,7 @@ import { mapFrom, mapLink, type MapChoice } from "../mapname.ts";
2929 import { LONGEST_PASTE } from "../matchlink.ts";
3030 import { roomLink } from "../roomlink.ts";
3131 import type { Who } from "../who.ts";
32 import { plannedSaid, type Readout } from "../plan/readout.ts";
3233 import { figure } from "./figure.ts";
3334 import { navFor, noscriptFor, othersHtml } from "./games.ts";
3435 import { WEAPONS_HREF, WEAPONS_SUMMARY } from "./weapons.ts";
@@ -163,6 +164,12 @@ export type PageOptions = Pick<ShellOptions, "env" | "pull" | "commit" | "script
163164 * every round in the page, which is what the paragraph then says.
164165 */
165166 planning?: boolean;
167 /**
168 * What the planner has done since the server started (`src/plan/readout.ts`),
169 * said under the paragraph in the same words the terminal uses. Absent says
170 * nothing, which is what a page rendered without a server behind it knows.
171 */
172 plans?: Readout;
166173 /**
167174 * The moment this page is being rendered at: which day the match of the day
168175 * is, and which matches in the feed still read as live. Absent is now, and a
@@ -469,7 +476,7 @@ export const policyHtml = (card: Card): string =>
469476 * given a key — the page never claims a model is planning anything when the
470477 * only planner running is the arithmetic in the page.
471478 */
472 export const planHtml = (planning: boolean): string =>
479 export const planHtml = (planning: boolean, plans?: Readout): string =>
473480 `<p>Above the fight the bots now plan a round rather than five separate ones. Each side gets a
474481 site, a shape — everyone through one site, a split across both, or a fake with a lurker on the
475482 other — a tempo and a decision about whether to save, and every bot gets a job cut from it:
@@ -485,7 +492,8 @@ export const planHtml = (planning: boolean): string =>
485492 : `This deployment has no TypeSafe key, so every round is planned in your browser by the same
486493 arithmetic the server would use as its fallback: nothing is sent anywhere, and the plan is
487494 still a plan.`
488 }</p>`;
495 }</p>${plans === undefined ? "" : `
496 <p class="hint" id="planned">${escape(plannedSaid(plans))}</p>`}`;
489497
490498 /**
491499 * The pause panel. Its line about Escape is rendered with the sentence that is
@@ -559,7 +567,7 @@ const SAVED_LINE = `<p class="hint" id="saved" role="status" aria-live="polite">
559567
560568 export function gamePage(
561569 db: Database,
562 { who: visitor, asked = NOTHING_ASKED, card = CARD, planning = false, now = Date.now(), ...options }: PageOptions,
570 { who: visitor, asked = NOTHING_ASKED, card = CARD, planning = false, plans, now = Date.now(), ...options }: PageOptions,
563571 ): string {
564572 const daily = dailyOn(now);
565573 return page({
@@ -625,7 +633,7 @@ export function gamePage(
625633 Everything you see is generated — the map, the textures, the models and the sounds — so the
626634 whole game is one small download and no asset files.</p>
627635 ${policyHtml(card)}
628 ${planHtml(planning)}
636 ${planHtml(planning, plans)}
629637 <p>The match runs entirely in your browser; the leaderboard here is what your browser
630638 reported about it, checked against what a match can produce. The name on a row is not
631639 the browser's word, though: signed in to the forge, your results are recorded under
src/plan/readout.test.ts added +77
@@ -0,0 +1,77 @@
1 /**
2 * The readout: the shape the planner's counts are shown in, the one sentence
3 * the page and the terminal share, and the file the terminal reads — parsed
4 * like any other input, because it is one.
5 */
6
7 import { expect, test } from "bun:test";
8 import { mkdtempSync, writeFileSync } from "node:fs";
9 import { tmpdir } from "node:os";
10 import { join } from "node:path";
11 import {
12 emptyTally, localPlans, parseReadout, plannedSaid, readoutFile, readReadout, type Readout, writeReadout,
13 } from "./readout.ts";
14
15 const readout = (over: Partial<Readout> = {}): Readout => ({
16 on: true,
17 model: "jev-1",
18 since: "2026-09-23T10:04:11.201Z",
19 at: "2026-09-23T11:00:00.000Z",
20 limits: { cache: 500, timeout: 2500, retries: 1, caller: { burst: 12, per: 3000 }, spend: { burst: 30, per: 2000 } },
21 tally: {
22 ...emptyTally(),
23 asks: 10, hits: 3, joined: 1, answered: 4, sent: 7,
24 failed: { timeout: 1, connection: 0, status: { "500": 1 }, invalid: 0, other: 0 },
25 tokens: { input: 1200, output: 80 },
26 },
27 refused: { caller: 2, spend: 1 },
28 unkeyed: 0,
29 ...over,
30 });
31
32 test("the sentence is the counts, added up by hand", () => {
33 const r = readout();
34 // 10 to the planner + 3 refused = 13 asked; 2 failed + 3 refused = 5 local.
35 expect(localPlans(r)).toBe(5);
36 expect(plannedSaid(r)).toBe(
37 "Since this server started at 2026-09-23 10:04 UTC, 13 rounds asked to be planned: 4 answered by jev-1, " +
38 "4 shared an answer already asked for, and 5 planned locally — 3 over a rate limit, 2 because the model failed. " +
39 "That was 7 requests upstream and 1,200 tokens in, 80 out; TypeSafe reports tokens, not a price. " +
40 "These counts are this process's and start again at zero when it restarts.",
41 );
42 });
43
44 test("a deployment with no key says so plainly", () => {
45 expect(plannedSaid(readout({ on: false, tally: emptyTally() })))
46 .toBe("No TypeSafe key: no round has been asked of a model since this server started at 2026-09-23 10:04 UTC, and none will be.");
47 });
48
49 test("what is written is what is read back, and nothing else is", () => {
50 const dir = mkdtempSync(join(tmpdir(), "readout-"));
51 expect(readReadout(dir)).toBeNull();
52 writeReadout(dir, readout());
53 expect(readReadout(dir)).toEqual(readout());
54 // A field the shape does not have does not survive the parse.
55 writeFileSync(readoutFile(dir), JSON.stringify({ ...readout(), apiKey: "sk-live" }));
56 expect(JSON.stringify(readReadout(dir))).not.toContain("sk-live");
57 });
58
59 test("a file that is not a readout is none", () => {
60 const dir = mkdtempSync(join(tmpdir(), "readout-"));
61 writeFileSync(readoutFile(dir), "{not json");
62 expect(readReadout(dir)).toBeNull();
63 const bad: unknown[] = [
64 null, [], "x",
65 { ...readout(), on: "yes" },
66 { ...readout(), model: "" },
67 { ...readout(), model: "m".repeat(65) },
68 { ...readout(), since: "yesterday" },
69 { ...readout(), unkeyed: -1 },
70 { ...readout(), unkeyed: 1.5 },
71 { ...readout(), refused: { caller: 1 } },
72 { ...readout(), tally: { ...readout().tally, failed: { ...readout().tally.failed, status: { "600": 1 } } } },
73 { ...readout(), tally: { ...readout().tally, failed: { ...readout().tally.failed, status: { __proto__x: 1 } } } },
74 { ...readout(), limits: { ...readout().limits, spend: { burst: 1 } } },
75 ];
76 for (const v of bad) expect(parseReadout(v)).toBeNull();
77 });
src/plan/readout.ts added +241
@@ -0,0 +1,241 @@
1 /**
2 * What the planner did, as a readout. The planner and the two limiters in front
3 * of it count in this process, where each decision is made — a cache hit is
4 * only visible inside `src/plan/typesafe.ts`, a refusal only inside
5 * `src/limit.ts` — and this file is the shape those counts are shown in, on the
6 * page and in the terminal.
7 *
8 * This is a readout, not a metrics system. Every figure is an integer since the
9 * server process started (`since`), none is kept anywhere after it stops, and
10 * each is labelled as such wherever it is shown. Nothing here logs.
11 *
12 * The terminal is a different process from the server, so the server leaves a
13 * copy of the readout in `FORGE_APP_DATA/planner.json` — overwritten whole, a
14 * second after the counts move and at start-up — and the terminal reads it.
15 * The file is a window onto the running server, not a history: a restart
16 * writes zeros over it. What the terminal reads is from disk, so it is parsed
17 * here like any other input, and a file that is not a readout is none.
18 *
19 * Money: TypeSafe answers with the tokens a request used and no price, so the
20 * readout counts tokens and requests and never invents a currency.
21 */
22
23 import { renameSync, readFileSync, writeFileSync } from "node:fs";
24 import { join } from "node:path";
25
26 /** The planner's own counts. Every round put to it ends as exactly one of
27 * `hits`, `joined`, `answered` or one of the `failed` kinds. */
28 export type Tally = {
29 /** Rounds put to the planner. */
30 asks: number;
31 /** Answered from the planner's memory: no request made. */
32 hits: number;
33 /** Joined a request already in flight for the same round: no request made. */
34 joined: number;
35 /** HTTP requests sent upstream, the SDK's one retry included. What is billed. */
36 sent: number;
37 /** Rounds a model's answer planned. */
38 answered: number;
39 /** Rounds that went upstream and came back as the local plan, by why. */
40 failed: {
41 /** An attempt ran past its timeout, retry included. */
42 timeout: number;
43 /** DNS, TLS, a dropped connection: no status came back. */
44 connection: number;
45 /** An HTTP status that was not success, by status: `{ "500": 2, "401": 1 }`. */
46 status: Record<string, number>;
47 /** An answer arrived and was not a plan this game can play. */
48 invalid: number;
49 /** Anything else the SDK threw. */
50 other: number;
51 };
52 /** Tokens the answers reported using. Tokens, not money: no price comes back. */
53 tokens: { input: number; output: number };
54 };
55
56 export const emptyTally = (): Tally => ({
57 asks: 0, hits: 0, joined: 0, sent: 0, answered: 0,
58 failed: { timeout: 0, connection: 0, status: {}, invalid: 0, other: 0 },
59 tokens: { input: 0, output: 0 },
60 });
61
62 /** How many rounds went upstream and came back as the local plan. */
63 export const failures = (tally: Tally): number =>
64 tally.failed.timeout + tally.failed.connection + tally.failed.invalid + tally.failed.other +
65 Object.values(tally.failed.status).reduce((sum, n) => sum + n, 0);
66
67 export type Rate = { burst: number; per: number };
68
69 /** Everything the page and the terminal say about the planner. Never the key. */
70 export type Readout = {
71 /** Whether this deployment was given a key. */
72 on: boolean;
73 /** The model asked for: `TYPESAFE_MODEL`, or the SDK's default. */
74 model: string;
75 /** When the server process that counted these started, ISO 8601 UTC. */
76 since: string;
77 /** When these counts were read, ISO 8601 UTC. */
78 at: string;
79 /** What the deployment is configured with. */
80 limits: {
81 /** Answers remembered before the memory is dropped. */
82 cache: number;
83 /** Milliseconds one attempt gets. */
84 timeout: number;
85 /** Retries behind a failed attempt. */
86 retries: number;
87 /** Per caller: `burst` rounds back to back, then one every `per` ms. */
88 caller: Rate;
89 /** The whole process, the same way: the spend ceiling. */
90 spend: Rate;
91 };
92 tally: Tally;
93 /** Rounds a limiter turned away before the planner saw them. */
94 refused: { caller: number; spend: number };
95 /** Rounds asked of this server with no key to ask with. */
96 unkeyed: number;
97 };
98
99 /** Every round this server answered with the local plan, for whatever reason. */
100 export const localPlans = (r: Readout): number => failures(r.tally) + r.refused.caller + r.refused.spend + r.unkeyed;
101
102 /** The file the server leaves for the terminal. */
103 export const readoutFile = (dir: string): string => join(dir, "planner.json");
104
105 /**
106 * Overwrite the copy the terminal reads. Written beside and renamed over, so a
107 * terminal reading at the same moment sees the old readout or the new one,
108 * never half of either.
109 */
110 export function writeReadout(dir: string, readout: Readout): void {
111 const file = readoutFile(dir);
112 writeFileSync(`${file}.tmp`, JSON.stringify(readout));
113 renameSync(`${file}.tmp`, file);
114 }
115
116 /** The copy on disk, or null when there is none or it is not a readout. */
117 export function readReadout(dir: string): Readout | null {
118 let text: string;
119 try {
120 text = readFileSync(readoutFile(dir), "utf8");
121 } catch {
122 return null;
123 }
124 if (text.length > 16_384) return null;
125 try {
126 return parseReadout(JSON.parse(text));
127 } catch {
128 return null;
129 }
130 }
131
132 // --- the boundary: a file on disk is input like any other -------------------
133
134 type Obj = Record<string, unknown>;
135
136 const isObj = (v: unknown): v is Obj => typeof v === "object" && v !== null && !Array.isArray(v);
137
138 /** A count: a non-negative whole number a double holds exactly. */
139 const count = (v: unknown): number | null => (Number.isSafeInteger(v) && (v as number) >= 0 ? (v as number) : null);
140
141 const text = (v: unknown, max: number): string | null =>
142 typeof v === "string" && v.length > 0 && v.length <= max ? v : null;
143
144 const when = (v: unknown): string | null => {
145 const s = text(v, 32);
146 return s !== null && /^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d(\.\d+)?Z$/.test(s) ? s : null;
147 };
148
149 const rate = (v: unknown): Rate | null => {
150 if (!isObj(v)) return null;
151 const burst = count(v.burst);
152 const per = count(v.per);
153 return burst === null || per === null ? null : { burst, per };
154 };
155
156 /** Parse throws on the first thing that is wrong; `parseReadout` turns that into null. */
157 class NotAReadout extends Error {}
158
159 const need = <T>(v: T | null): T => {
160 if (v === null) throw new NotAReadout();
161 return v;
162 };
163
164 function tallyFrom(v: unknown): Tally {
165 if (!isObj(v) || !isObj(v.failed) || !isObj(v.tokens) || !isObj(v.failed.status)) throw new NotAReadout();
166 const status: Record<string, number> = {};
167 const codes = Object.entries(v.failed.status);
168 if (codes.length > 500) throw new NotAReadout();
169 for (const [code, n] of codes) {
170 if (!/^[1-5]\d\d$/.test(code)) throw new NotAReadout();
171 status[code] = need(count(n));
172 }
173 return {
174 asks: need(count(v.asks)),
175 hits: need(count(v.hits)),
176 joined: need(count(v.joined)),
177 sent: need(count(v.sent)),
178 answered: need(count(v.answered)),
179 failed: {
180 timeout: need(count(v.failed.timeout)),
181 connection: need(count(v.failed.connection)),
182 status,
183 invalid: need(count(v.failed.invalid)),
184 other: need(count(v.failed.other)),
185 },
186 tokens: { input: need(count(v.tokens.input)), output: need(count(v.tokens.output)) },
187 };
188 }
189
190 /** A readout, or null. Only the fields above survive; nothing else is kept. */
191 export function parseReadout(v: unknown): Readout | null {
192 try {
193 if (!isObj(v) || typeof v.on !== "boolean" || !isObj(v.limits) || !isObj(v.refused)) return null;
194 const l = v.limits;
195 return {
196 on: v.on,
197 model: need(text(v.model, 64)),
198 since: need(when(v.since)),
199 at: need(when(v.at)),
200 limits: {
201 cache: need(count(l.cache)),
202 timeout: need(count(l.timeout)),
203 retries: need(count(l.retries)),
204 caller: need(rate(l.caller)),
205 spend: need(rate(l.spend)),
206 },
207 tally: tallyFrom(v.tally),
208 refused: { caller: need(count(v.refused.caller)), spend: need(count(v.refused.spend)) },
209 unkeyed: need(count(v.unkeyed)),
210 };
211 } catch (error) {
212 if (error instanceof NotAReadout) return null;
213 throw error;
214 }
215 }
216
217 // --- said: one sentence the page and the terminal both print ----------------
218
219 const n = (x: number): string => x.toLocaleString("en");
220 const rounds = (x: number): string => `${n(x)} round${x === 1 ? "" : "s"}`;
221
222 /** `2026-09-23T10:04:11.201Z` as `2026-09-23 10:04 UTC`. */
223 export const minuteOf = (iso: string): string => `${iso.slice(0, 10)} ${iso.slice(11, 16)} UTC`;
224
225 /**
226 * What the readout says in words, as plain text for the caller to escape. The
227 * page and the terminal print this same sentence, so they cannot disagree
228 * with each other or with the counts.
229 */
230 export function plannedSaid(r: Readout): string {
231 const since = `since this server started at ${minuteOf(r.since)}`;
232 if (!r.on) return `No TypeSafe key: no round has been asked of a model ${since}, and none will be.`;
233 const t = r.tally;
234 const asked = t.asks + r.refused.caller + r.refused.spend + r.unkeyed;
235 const refused = r.refused.caller + r.refused.spend;
236 return `${since[0]?.toUpperCase()}${since.slice(1)}, ${rounds(asked)} asked to be planned: ` +
237 `${n(t.answered)} answered by ${r.model}, ${n(t.hits + t.joined)} shared an answer already asked for, ` +
238 `and ${n(localPlans(r))} planned locally — ${n(refused)} over a rate limit, ${n(failures(t))} because the model failed. ` +
239 `That was ${n(t.sent)} request${t.sent === 1 ? "" : "s"} upstream and ${n(t.tokens.input)} tokens in, ` +
240 `${n(t.tokens.output)} out; TypeSafe reports tokens, not a price. These counts are this process's and start again at zero when it restarts.`;
241 }
src/plan/typesafe.test.ts +100
@@ -7,6 +7,8 @@
77
88 import { expect, test } from "bun:test";
99 import { localPlan } from "./plan.ts";
10 import { createLimiter } from "../limit.ts";
11 import { emptyTally, failures } from "./readout.ts";
1012 import { createPlanner } from "./typesafe.ts";
1113 import type { Situation } from "./situation.ts";
1214
@@ -164,3 +166,101 @@ test("the cache is bounded: past the ceiling it is dropped rather than grown", a
164166 expect(planner?.size()).toBe(1);
165167 });
166168
169
170 // --- what it did: the counts ------------------------------------------------
171
172 test("an answer, then an error, then a refusal: three rounds, three different counts", async () => {
173 // The route's order, in miniature: the limiter decides, then the planner.
174 const limit = createLimiter({ burst: 2, per: 60_000 });
175 let calls = 0;
176 const planner = createPlanner({
177 apiKey: "sk-test",
178 fetch: async () => {
179 calls += 1;
180 return calls === 1 ? reply() : new Response("nope", { status: 500 });
181 },
182 });
183 const round = async (n: number): Promise<string> =>
184 limit.take("caller", 0) ? ((await planner?.plan(situation({ round: n })))?.source ?? "none") : "refused";
185
186 expect(await round(1)).toBe("typesafe");
187 expect(await round(2)).toBe("local");
188 expect(await round(3)).toBe("refused");
189
190 const tally = planner?.tally();
191 expect(tally?.asks).toBe(2);
192 expect(tally?.answered).toBe(1);
193 expect(tally?.failed.status).toEqual({ "500": 1 });
194 expect(tally === undefined ? -1 : failures(tally)).toBe(1);
195 expect(limit.refused()).toBe(1);
196 // Three requests upstream for two rounds: the 500 was retried once.
197 expect(tally?.sent).toBe(3);
198 // The reply said one token in and one out, and that is all that is claimed.
199 expect(tally?.tokens).toEqual({ input: 1, output: 1 });
200 });
201
202 test("memory and a joined request are counted as the answers they reused, and cost no request", async () => {
203 let release = (): void => {};
204 const held = new Promise<void>((resolve) => {
205 release = resolve;
206 });
207 const planner = createPlanner({ apiKey: "sk-test", fetch: async () => (await held, reply()) });
208 const both = Promise.all([planner?.plan(situation()), planner?.plan(situation())]);
209 release();
210 await both;
211 await planner?.plan(situation());
212 expect(planner?.tally()).toEqual({ ...emptyTally(), asks: 3, answered: 1, joined: 1, hits: 1, sent: 1,
213 tokens: { input: 1, output: 1 } });
214 });
215
216 test("each failure is counted by its kind, and every ask ends as exactly one thing", async () => {
217 const answering = (body: () => Response | Promise<Response>) =>
218 createPlanner({ apiKey: "sk-test", timeout: 20, fetch: async () => body() });
219
220 const invalid = answering(() => reply({ tSite: { type: "choice", choice: "Middle" } }));
221 await invalid?.plan(situation());
222 expect(invalid?.tally().failed.invalid).toBe(1);
223
224 // A body that is not JSON is an answer that is not a plan, not a crash.
225 const junk = answering(() => new Response("<html>a proxy</html>", { headers: { "content-type": "text/html" } }));
226 await junk?.plan(situation());
227 expect(junk?.tally().failed.invalid).toBe(1);
228
229 const refused = answering(() => new Response("{}", { status: 401 }));
230 await refused?.plan(situation());
231 expect(refused?.tally().failed.status).toEqual({ "401": 1 });
232
233 const slow = createPlanner({
234 apiKey: "sk-test",
235 timeout: 20,
236 fetch: (_url, init) => new Promise((_resolve, reject) => {
237 init?.signal?.addEventListener("abort", () => reject(init.signal?.reason));
238 }),
239 });
240 await slow?.plan(situation());
241 expect(slow?.tally().failed.timeout).toBe(1);
242
243 const gone = answering(() => {
244 throw new TypeError("fetch failed");
245 });
246 await gone?.plan(situation());
247 expect(gone?.tally().failed.connection).toBe(1);
248
249 for (const p of [invalid, junk, refused, slow, gone]) {
250 const t = p?.tally() ?? emptyTally();
251 expect(t.asks).toBe(t.hits + t.joined + t.answered + failures(t));
252 expect(t.answered).toBe(0);
253 }
254 });
255
256 test("the tally handed out is a copy: a caller cannot move the planner's counts", async () => {
257 const planner = createPlanner({ apiKey: "sk-test", fetch: async () => reply() });
258 const tally = planner?.tally();
259 if (tally !== undefined) tally.asks = 99;
260 expect(planner?.tally().asks).toBe(0);
261 });
262
263 test("the model it reports is the one configured, or the SDK's default", () => {
264 expect(createPlanner({ apiKey: "sk-test", model: "jev-2" })?.model).toBe("jev-2");
265 expect(createPlanner({ apiKey: "sk-test" })?.model).toBe(process.env.TYPESAFE_DEFAULT_MODEL ?? "jev-latest");
266 });
src/plan/typesafe.ts +87 −18
@@ -36,10 +36,22 @@
3636 * `localPlan`: the deterministic plan in `src/plan/plan.ts`, which is what a
3737 * deployment with no key runs all the time. The game never waits on this and
3838 * never fails because of it.
39 *
40 * ## What it did
41 *
42 * Nothing here logs a line per round: a round a minute per player is noise in
43 * the run log. Instead the planner counts, as integers, at the place each thing
44 * is decided — a memory hit, a joined request, every HTTP attempt the SDK makes
45 * (counted in the `fetch` it is handed, so a retry is a request too), an answer,
46 * and each failure by its kind — and `tally()` hands them back as plain data.
47 * `src/plan/readout.ts` is how they are shown; they reset with the process.
3948 */
4049
41 import { choice, noul, score, TypeSafeClient, type Fetch } from "@typesafe-ai/sdk";
50 import {
51 APIConnectionError, APIError, APITimeoutError, choice, noul, score, TypeSafeClient, type Fetch,
52 } from "@typesafe-ai/sdk";
4253 import { localPlan, type RoundPlan, type Shape, type SiteName, type TeamPlan } from "./plan.ts";
54 import { emptyTally, type Tally } from "./readout.ts";
4355 import { keyOf, type Situation } from "./situation.ts";
4456
4557 export type Planner = {
@@ -47,6 +59,10 @@ export type Planner = {
4759 plan(situation: Situation): Promise<RoundPlan>;
4860 /** How many answers are being remembered. For the tests. */
4961 size(): number;
62 /** What it has done since it was made: a copy, so the caller cannot move it. */
63 tally(): Tally;
64 /** The model it asks for: the option, or the SDK's default. */
65 model: string;
5066 };
5167
5268 export type PlannerOptions = {
@@ -63,9 +79,10 @@ export type PlannerOptions = {
6379 };
6480
6581 /** One attempt gets this long, and there is one retry behind it. */
66 const TIMEOUT = 2500;
82 export const TIMEOUT = 2500;
83 export const RETRIES = 1;
6784
68 const CEILING = 500;
85 export const CEILING = 500;
6986
7087 /**
7188 * The planner, or null when this deployment has no key — which is the state
@@ -75,30 +92,52 @@ const CEILING = 500;
7592 export function createPlanner(options: PlannerOptions): Planner | null {
7693 if (typeof options.apiKey !== "string" || options.apiKey.trim() === "") return null;
7794 const { model, timeout = TIMEOUT, ceiling = CEILING, fetch } = options;
95 const counts = emptyTally();
96 const send = fetch ?? ((url, init) => globalThis.fetch(url, init));
7897 const client = new TypeSafeClient({
7998 apiKey: options.apiKey,
8099 ...(model === undefined ? {} : { defaultModel: model }),
81100 timeout,
82 retry: { maxRetries: 1 },
83 ...(fetch === undefined ? {} : { fetch }),
101 retry: { maxRetries: RETRIES },
102 // Every attempt passes through here, the retry included: this is the
103 // count of requests the deployment is billed for, not of rounds.
104 fetch: (url, init) => {
105 counts.sent += 1;
106 return send(url, init);
107 },
84108 });
85109 const answers = new Map<string, RoundPlan>();
86110 const flight = new Map<string, Promise<RoundPlan>>();
87111
88112 return {
89113 size: () => answers.size,
114 tally: () => structuredClone(counts),
115 model: client.defaultModel,
90116 async plan(situation) {
117 counts.asks += 1;
91118 const key = keyOf(situation);
92119 // The key is the situation, round included, so two browsers playing the
93120 // same round of the same match ask upstream once between them.
94121 const held = answers.get(key);
95 if (held !== undefined) return held;
122 if (held !== undefined) {
123 counts.hits += 1;
124 return held;
125 }
96126 const running = flight.get(key);
97 if (running !== undefined) return running;
98 const asking = ask(client, situation).catch((error: unknown) => {
99 console.warn(`plan: falling back to the local plan — ${describeError(error)}`);
100 return localPlan(situation);
101 });
127 if (running !== undefined) {
128 counts.joined += 1;
129 return running;
130 }
131 const asking = ask(client, situation, counts).then(
132 (plan) => {
133 counts.answered += 1;
134 return plan;
135 },
136 (error: unknown) => {
137 fail(counts, error);
138 return localPlan(situation);
139 },
140 );
102141 flight.set(key, asking);
103142 const plan = await asking;
104143 flight.delete(key);
@@ -113,8 +152,24 @@ export function createPlanner(options: PlannerOptions): Planner | null {
113152 };
114153 }
115154
116 const describeError = (error: unknown): string =>
117 error instanceof Error ? `${error.name}: ${error.message}` : "the planner threw something that is not an error";
155 /** An answer arrived and was not a plan: counted apart from the wire failing. */
156 class NotAPlan extends Error {}
157
158 /**
159 * One failure, counted by what it was. The timeout is asked before the
160 * connection because the SDK's timeout is a kind of connection error; a status
161 * is kept by its number, which is bounded by being one.
162 */
163 function fail(counts: Tally, error: unknown): void {
164 const failed = counts.failed;
165 if (error instanceof NotAPlan) failed.invalid += 1;
166 else if (error instanceof APITimeoutError) failed.timeout += 1;
167 else if (error instanceof APIConnectionError) failed.connection += 1;
168 else if (error instanceof APIError && Number.isInteger(error.status) && error.status >= 100 && error.status <= 599) {
169 const code = String(error.status);
170 failed.status[code] = (failed.status[code] ?? 0) + 1;
171 } else failed.other += 1;
172 }
118173
119174 /**
120175 * The questions, and the answers turned back into a plan. Both sides are asked
@@ -122,8 +177,8 @@ const describeError = (error: unknown): string =>
122177 * answer about the attack it expects, and two requests could not know that
123178 * about each other.
124179 */
125 async function ask(client: TypeSafeClient, situation: Situation): Promise<RoundPlan> {
126 const { answers, model } = await client.systemOne({
180 async function ask(client: TypeSafeClient, situation: Situation, counts: Tally): Promise<RoundPlan> {
181 const { answers: typed, model, usage } = await client.systemOne({
127182 state: {
128183 game: "a five-a-side bomb-defusal round, Counter-Strike rules, on a map with two bombsites called A and B",
129184 map: situation.map,
@@ -150,11 +205,19 @@ async function ask(client: TypeSafeClient, situation: Situation): Promise<RoundP
150205 ctSave: noul("Should the defenders save their money this round instead of buying?", SAVE_CRITERIA),
151206 },
152207 });
153 const t = teamFrom(answers.tSite.choice, answers.tShape.choice, answers.tTempo.score, answers.tSave.noul);
154 const ct = teamFrom(answers.ctSite.choice, answers.ctShape.choice, answers.ctTempo.score, answers.ctSave.noul);
208 // Tokens are what the vendor bills by, and the one figure it reports: they
209 // are counted whether or not the answer turns out to be a plan.
210 counts.tokens.input += tokensOf(usage, "input_tokens");
211 counts.tokens.output += tokensOf(usage, "output_tokens");
212 // Typed by the questions, and read as what arrived: a body that was not JSON
213 // comes back from the SDK as a string, not as eight answers.
214 const answers = typed as unknown as Record<string, Record<string, unknown> | undefined> | null;
215 if (typeof answers !== "object" || answers === null) throw new NotAPlan("the answers were not answers");
216 const t = teamFrom(answers.tSite?.choice, answers.tShape?.choice, answers.tTempo?.score, answers.tSave?.noul);
217 const ct = teamFrom(answers.ctSite?.choice, answers.ctShape?.choice, answers.ctTempo?.score, answers.ctSave?.noul);
155218 // An answer that is not a plan is not half-used: the round gets the local one
156219 // whole, so both sides are always playing the same kind of round.
157 if (t === null || ct === null) throw new Error("the answers were not a plan this game can play");
220 if (t === null || ct === null) throw new NotAPlan("the answers were not a plan this game can play");
158221 return { round: situation.round, source: "typesafe", model: modelName(model), t, ct };
159222 }
160223
@@ -183,6 +246,12 @@ const SAVE_CRITERIA = {
183246 false: "the side should buy what it can afford and play the round",
184247 } as const;
185248
249 /** A token count from the wire, or nothing: it is added to an integer. */
250 const tokensOf = (usage: unknown, field: string): number => {
251 const n = typeof usage === "object" && usage !== null ? (usage as Record<string, unknown>)[field] : undefined;
252 return Number.isSafeInteger(n) && (n as number) >= 0 && (n as number) < 10_000_000 ? (n as number) : 0;
253 };
254
186255 /** A model name is a string from somewhere else: it is shown, so it is bounded. */
187256 const modelName = (model: unknown): string =>
188257 typeof model === "string" && model.length > 0 && model.length <= 64 ? model : "typesafe";
src/server.test.ts +28 −1
@@ -21,6 +21,7 @@ import { WEAPONS } from "./client/weapons.ts";
2121 import { endpoints, escape } from "./pages/shell.ts";
2222 import { localPlan, parsePlan } from "./plan/plan.ts";
2323 import { parseSituation, type Situation } from "./plan/situation.ts";
24 import { plannedSaid, readReadout } from "./plan/readout.ts";
2425 import { sign } from "./who.ts";
2526 import { PAGE_ACTIONS, PAGE_IDS } from "./client/stage.test-helper.ts";
2627 import { EXAMPLE_ACTIONS, EXAMPLE_IDS } from "./example/stage.test-helper.ts";
@@ -43,7 +44,7 @@ const BUNDLES = bundlesDir();
4344 const KEY = "the-key-the-forge-signs-with";
4445 process.env.FORGE_APP_IDENTITY_KEY = KEY;
4546 const { server, MATCH_LIMIT, GUEST_ADDRESS_LIMIT, PLAN_ID, ROOM_LIMIT, SOCKET_PATH, hub, onError, findScripts,
46 holdStatic } = await import("./server.ts");
47 holdStatic, plannerReadout } = await import("./server.ts");
4748 const base = `http://${server.hostname}:${server.port}`;
4849
4950 /** The two headers the forge puts on a request it proxies for a signed-in person. */
@@ -1358,6 +1359,32 @@ test("a page with no planner behind it never tells a browser to ask for one", as
13581359 expect(body).toContain(escape("Above the fight the bots now plan a round"));
13591360 });
13601361
1362 test("a round asked of a server with no key is counted as that, and the terminal's copy follows", async () => {
1363 // Written at start-up, before any round: the terminal has a readout to read.
1364 const first = readReadout(process.env.FORGE_APP_DATA ?? "");
1365 expect(first?.on).toBe(false);
1366 expect(first?.since).toBe(plannerReadout().since);
1367
1368 const before = plannerReadout().unkeyed;
1369 expect((await askPlan(JSON.stringify(situation()))).status).toBe(200);
1370 const after = plannerReadout();
1371 expect(after.unkeyed).toBe(before + 1);
1372 // Nothing went upstream and nothing was refused: there was nothing to ask with.
1373 expect(after.tally.asks).toBe(0);
1374 expect(after.refused).toEqual({ caller: 0, spend: 0 });
1375 expect(JSON.stringify(after)).not.toContain("TYPESAFE_API_KEY");
1376
1377 // A second after the round, the file the terminal reads says the same.
1378 await Bun.sleep(1200);
1379 expect(readReadout(process.env.FORGE_APP_DATA ?? "")?.unkeyed).toBe(after.unkeyed);
1380 });
1381
1382 test("the page says in words what the counts say, and with no key it says so plainly", async () => {
1383 const body = await (await fetch(base + "/")).text();
1384 expect(body).toContain(`<p class="hint" id="planned">${escape(plannedSaid(plannerReadout()))}</p>`);
1385 expect(body).toContain("No TypeSafe key: no round has been asked of a model since this server started at");
1386 });
1387
13611388 test("the endpoint meta is how a page names where a plan comes from", () => {
13621389 expect(endpoints([{ id: PLAN_ID, href: "/api/plan" }]))
13631390 .toBe(`<meta name="endpoint" id="${PLAN_ID}" content="/api/plan">\n`);
src/server.ts +55 −2
@@ -67,7 +67,8 @@ import { mapFrom } from "./mapname.ts";
6767 import { seedFrom } from "./client/generate.ts";
6868 import { localPlan } from "./plan/plan.ts";
6969 import { parseSituation } from "./plan/situation.ts";
70 import { createPlanner, type Planner } from "./plan/typesafe.ts";
70 import { CEILING, createPlanner, type Planner, RETRIES, TIMEOUT } from "./plan/typesafe.ts";
71 import { emptyTally, type Readout, writeReadout } from "./plan/readout.ts";
7172 import { matchFrom } from "./matchid.ts";
7273 import { errorPage, notFoundPage, STYLE as MISS_STYLE } from "./pages/miss.ts";
7374 import { boardsHtml, rangePage } from "./pages/range.ts";
@@ -255,6 +256,54 @@ const planLimit = createLimiter(PLAN_LIMIT);
255256 export const PLAN_SPEND = { burst: 30, per: 2000 };
256257 const planSpend = createLimiter(PLAN_SPEND);
257258
259 /**
260 * What the planner has done since this process started, and what it is set up
261 * with: the planner and the two limiters count where they decide, and the one
262 * decision made here — there is no key to ask with — is counted here. Plain
263 * data, read in microseconds, never the key. `src/plan/readout.ts` says what
264 * each figure means.
265 */
266 const startedAt = new Date().toISOString();
267 let unkeyed = 0;
268 export const plannerReadout = (): Readout => ({
269 on: planner !== null,
270 model: planner?.model ?? process.env.TYPESAFE_MODEL ?? "jev-latest",
271 since: startedAt,
272 at: new Date().toISOString(),
273 limits: { cache: CEILING, timeout: TIMEOUT, retries: RETRIES, caller: PLAN_LIMIT, spend: PLAN_SPEND },
274 tally: planner?.tally() ?? emptyTally(),
275 refused: { caller: planLimit.refused(), spend: planSpend.refused() },
276 unkeyed,
277 });
278
279 /**
280 * The terminal is another process, so it reads the readout from a file the
281 * server leaves in FORGE_APP_DATA: written at start-up (zeros — a restart is a
282 * reset, and says so) and then at most once a second, a second after a round
283 * was planned. Work that can wait, done after the response and off its path;
284 * a disk that will not take it is said once in the run log and not again.
285 */
286 const readoutDir = process.env.FORGE_APP_DATA ?? "./data";
287 let readoutDue: ReturnType<typeof setTimeout> | null = null;
288 let readoutFailed = false;
289 const leaveReadout = (): void => {
290 try {
291 writeReadout(readoutDir, plannerReadout());
292 } catch (error) {
293 if (!readoutFailed) console.warn(`planner readout not written: ${error instanceof Error ? error.message : error}`);
294 readoutFailed = true;
295 }
296 };
297 const readoutMoved = (): void => {
298 if (readoutDue !== null) return;
299 readoutDue = setTimeout(() => {
300 readoutDue = null;
301 leaveReadout();
302 }, 1000);
303 readoutDue.unref?.();
304 };
305 leaveReadout();
306
258307 /**
259308 * How often one caller may open a room. A room is a world stepped 64 times a
260309 * second for as long as somebody is in it, so opening one is the most expensive
@@ -529,6 +578,7 @@ const RENDER: Record<string, Render> = {
529578 // the page makes no request at all.
530579 endpoints: planner === null ? [] : [{ id: PLAN_ID, href: "/api/plan" }],
531580 planning: planner !== null,
581 plans: plannerReadout(),
532582 who: who(request, identityKey), asked: asked(request),
533583 }),
534584 extra,
@@ -875,8 +925,11 @@ async function route(request: Request, listener: Bun.Server<Joined>): Promise<Re
875925 const situation = parseSituation(parsed);
876926 if (situation === null) return text("bad situation", 422);
877927 const caller = `plan:${listener.requestIP(request)?.address ?? "?"}|${readSession(request) ?? "-"}`;
928 if (planner === null) unkeyed += 1;
878929 const mine = planner !== null && planLimit.take(caller) && planSpend.take("all");
879 return json(mine ? await (planner as Planner).plan(situation) : localPlan(situation));
930 const plan = mine ? await (planner as Planner).plan(situation) : localPlan(situation);
931 readoutMoved();
932 return json(plan);
880933 }
881934
882935 /**
src/terminal/screen.ts +76 −1
@@ -22,6 +22,7 @@ import { type Daily, RANKED_BY } from "../daily.ts";
2222 import type { DayRun, Liveness, MatchRow, RunRow, Standing } from "../db.ts";
2323 import { displayName, headshotShare, hitShare, liveness, ratio } from "../db.ts";
2424 import { matchOf } from "../matchid.ts";
25 import { failures, localPlans, minuteOf, plannedSaid, type Rate, type Readout } from "../plan/readout.ts";
2526
2627 export type Tab = "standings" | "matches" | "policy" | "today" | "runs" | "weapons";
2728
@@ -61,6 +62,12 @@ export type Data = {
6162 /** Every range run ever recorded. */
6263 runTotal: number;
6364 card: Card;
65 /**
66 * What the bots' round planner has done since the server started, from the
67 * copy the server leaves in FORGE_APP_DATA (`src/plan/readout.ts`). Null
68 * when there is none to read — no server has run here yet. Absent is null.
69 */
70 plans?: Readout | null;
6471 /**
6572 * The moment those rows were read, which is the only clock this file has:
6673 * it tells a match still posting from one that stopped (`liveness` in
@@ -136,6 +143,74 @@ function policyLines(card: Card): string[] {
136143 ];
137144 }
138145
146 /** `{ burst: 12, per: 3000 }` as a person reads it. */
147 const rateSaid = ({ burst, per }: Rate): string => `${burst} back to back, then 1 every ${per / 1000} s`;
148
149 /**
150 * The round planner beside the policy card: what this deployment is set up
151 * with, then what it did, then the page's own sentence about it. Every count is
152 * since `since` and labelled so; the key is never in a readout to be shown.
153 */
154 function plannerLines(r: Readout | null): string[] {
155 const line = (k: string, v: string): string => ` ${esc}2m${pad(k, 22)}${esc}0m${printable(v)}`;
156 const head = ` ${esc}1mwho plans the bots' rounds${esc}0m`;
157 if (r === null) return [head, "", ` ${esc}2mno readout yet: the server has not run in this environment${esc}0m`];
158 const t = r.tally;
159 const n = (x: number): string => x.toLocaleString("en");
160 const statuses = Object.entries(t.failed.status).sort(([a], [b]) => a.localeCompare(b));
161 const lines = [
162 head,
163 "",
164 line("planner", r.on ? `TypeSafe AI, model ${r.model}` : "off — no TYPESAFE_API_KEY, every round is planned locally"),
165 ];
166 if (r.on)
167 lines.push(
168 line("key", "set (never shown)"),
169 line("per caller", rateSaid(r.limits.caller)),
170 line("spend ceiling", `${rateSaid(r.limits.spend)}, whole process`),
171 line("per attempt", `${r.limits.timeout} ms, ${r.limits.retries} retry`),
172 line("answers remembered", `up to ${n(r.limits.cache)}`),
173 );
174 lines.push(
175 "",
176 ` ${esc}2mcounts since the server started at ${printable(minuteOf(r.since))}; reset on restart${esc}0m`,
177 line("read at", minuteOf(r.at)),
178 line("rounds to the planner", n(t.asks)),
179 line(" answered by model", n(t.answered)),
180 line(" from memory", n(t.hits)),
181 line(" joined in flight", n(t.joined)),
182 line(" failed", n(failures(t))),
183 line(" timeout", n(t.failed.timeout)),
184 line(" connection", n(t.failed.connection)),
185 ...statuses.map(([code, count]) => line(` HTTP ${code}`, n(count))),
186 line(" not a plan", n(t.failed.invalid)),
187 line(" other", n(t.failed.other)),
188 line("refused, per caller", n(r.refused.caller)),
189 line("refused, spend limit", n(r.refused.spend)),
190 line("asked with no key", n(r.unkeyed)),
191 line("local plans served", n(localPlans(r))),
192 line("requests upstream", `${n(t.sent)} (retries included)`),
193 line("tokens", `${n(t.tokens.input)} in, ${n(t.tokens.output)} out — tokens, not money`),
194 "",
195 ...wrap(plannedSaid(r), 76).map((l) => ` ${esc}2m${printable(l)}${esc}0m`),
196 );
197 return lines;
198 }
199
200 /** Words into lines no wider than `width`, for a sentence under a table. */
201 function wrap(text: string, width: number): string[] {
202 const lines: string[] = [];
203 let line = "";
204 for (const word of text.split(" ")) {
205 if (line !== "" && line.length + 1 + word.length > width) {
206 lines.push(line);
207 line = word;
208 } else line = line === "" ? word : `${line} ${word}`;
209 }
210 if (line !== "") lines.push(line);
211 return lines;
212 }
213
139214 /**
140215 * The weapons, the same table the page renders and out of the same data: the
141216 * numbers the game is tuned by, and what one shot of each does to a person.
@@ -310,7 +385,7 @@ export function screen(view: View, data: Data): string[] {
310385 if (view.tab === "runs") lines.push(...runLines(data.runs, cursor));
311386 else if (view.tab === "weapons") lines.push(...weaponLines());
312387 else if (view.tab === "today") lines.push(...todayLines(data.daily, data.today));
313 else if (view.tab === "policy") lines.push(...policyLines(data.card));
388 else if (view.tab === "policy") lines.push(...policyLines(data.card), "", ...plannerLines(data.plans ?? null));
314389 else if (view.tab === "standings") lines.push(...standingsLines(data.standings, cursor));
315390 else lines.push(...matchLines(data.matches, cursor, data.now));
316391 lines.push("");
src/tty.test.ts +83 −2
@@ -20,6 +20,7 @@ import { tmpdir } from "node:os";
2020 import { join } from "node:path";
2121 import { matchCount, openDatabase, saveMatch, type Player } from "./db.ts";
2222 import type { Result } from "./result.ts";
23 import { emptyTally, type Readout } from "./plan/readout.ts";
2324 import { plain } from "./terminal/screen.ts";
2425 import { run, type Io } from "./tty.ts";
2526
@@ -71,7 +72,11 @@ type Session = {
7172 * test can decide what is on the screen when the person arrives.
7273 */
7374 function session(
74 options: { env?: Record<string, string | undefined>; seed?: (db: Database) => void } = {},
75 options: {
76 env?: Record<string, string | undefined>;
77 seed?: (db: Database) => void;
78 plans?: () => Readout | null;
79 } = {},
7580 ): Session {
7681 const db = openDatabase(mkdtempSync(join(tmpdir(), "tty-")));
7782 options.seed?.(db);
@@ -93,7 +98,10 @@ function session(
9398 // session, nothing reads it: `screen.ts` owns the columns.
9499 const output = { write: (text: string) => { writes.push(text); }, get columns() { return state.columns; } };
95100 const resize = { on: (_event: "resize", listener: () => void) => { resizes.push(listener); } };
96 const io: Io = { input, output, resize, exit: (code) => { exits.push(code); }, db, env: options.env ?? {} };
101 const io: Io = {
102 input, output, resize, exit: (code) => { exits.push(code); }, db, env: options.env ?? {},
103 ...(options.plans === undefined ? {} : { plans: options.plans }),
104 };
97105 run(io);
98106
99107 const frame = (): string[] => (writes.at(-1) ?? "").slice(CLEAR.length).split("\r\n").map(plain);
@@ -315,3 +323,76 @@ test("a name out of the database cannot move the cursor or clear the screen", ()
315323 expect(tty.frame().at(-1)).toBe(`removed ${attack.replace(/\x1b/g, "·")}'s match`);
316324 expect(matchCount(tty.db)).toBe(0);
317325 });
326
327 // --- the round planner, beside the policy card ------------------------------
328
329 /** A readout as the server leaves it: the same counts the page is given. */
330 const readout = (over: Partial<Readout> = {}): Readout => ({
331 on: true,
332 model: "jev-1",
333 since: "2026-09-23T10:04:11.201Z",
334 at: "2026-09-23T11:00:00.000Z",
335 limits: { cache: 500, timeout: 2500, retries: 1, caller: { burst: 12, per: 3000 }, spend: { burst: 30, per: 2000 } },
336 tally: {
337 ...emptyTally(),
338 asks: 3, answered: 1, sent: 3,
339 failed: { timeout: 0, connection: 0, status: { "500": 1 }, invalid: 1, other: 0 },
340 tokens: { input: 640, output: 16 },
341 },
342 refused: { caller: 1, spend: 0 },
343 unkeyed: 0,
344 ...over,
345 });
346
347 const policyTab = (tty: Session): string[] => {
348 tty.press("\t");
349 tty.press("\t");
350 return tty.frame();
351 };
352
353 test("the policy tab shows what the planner did: an answer, a failure and a refusal each counted", () => {
354 const key = "sk-live-do-not-print-me";
355 const tty = session({ env: { TYPESAFE_API_KEY: key }, plans: () => readout() });
356 const lines = policyTab(tty);
357 expect(lines[4]).toBe(" the policy the bots fight with");
358 expect(lines).toContain(" who plans the bots' rounds");
359 expect(lines).toContain(" planner TypeSafe AI, model jev-1");
360 expect(lines).toContain(" key set (never shown)");
361 expect(lines).toContain(" spend ceiling 30 back to back, then 1 every 2 s, whole process");
362 expect(lines).toContain(" counts since the server started at 2026-09-23 10:04 UTC; reset on restart");
363 expect(lines).toContain(" read at 2026-09-23 11:00 UTC");
364 // Every line fits the 80 columns the other tabs keep to.
365 for (const line of lines) expect(line.length).toBeLessThanOrEqual(80);
366 expect(lines).toContain(" answered by model 1");
367 expect(lines).toContain(" failed 2");
368 expect(lines).toContain(" HTTP 500 1");
369 expect(lines).toContain(" not a plan 1");
370 expect(lines).toContain(" refused, per caller 1");
371 expect(lines).toContain(" local plans served 3");
372 expect(lines).toContain(" tokens 640 in, 16 out — tokens, not money");
373 // The key is in this session's environment and nowhere on its screen.
374 expect(tty.writes.join("")).not.toContain(key);
375 expect(tty.writes.join("")).not.toContain("sk-");
376 });
377
378 test("a deployment with no key reads as off, with nothing asked of a model", () => {
379 const lines = policyTab(session({ plans: () => readout({ on: false, tally: emptyTally(), refused: { caller: 0, spend: 0 } }) }));
380 expect(lines).toContain(" planner off — no TYPESAFE_API_KEY, every round is planned locally");
381 expect(lines).not.toContain(" key set (never shown)");
382 expect(lines.join(" ")).toContain("No TypeSafe key: no round has been asked of a model");
383 });
384
385 test("with no readout to read, the terminal says the server has not run rather than showing zeros", () => {
386 const lines = policyTab(session());
387 expect(lines).toContain(" no readout yet: the server has not run in this environment");
388 expect(lines.join("\n")).not.toContain("rounds to the planner");
389 });
390
391 test("the readout is read again on every draw, so r shows the counts as they are now", () => {
392 let asks = 0;
393 const tty = session({ plans: () => readout({ tally: { ...emptyTally(), asks, answered: asks } }) });
394 policyTab(tty);
395 asks = 7;
396 tty.press("r");
397 expect(tty.frame()).toContain(" rounds to the planner 7");
398 });
src/tty.ts +10 −1
@@ -2,7 +2,7 @@
22 * cs, in a terminal. The forge runs this on a pty when someone `ssh`es to the
33 * app, with FORGE_APP_DATA pointing at the same database the web page uses:
44 * the same leaderboard, the same matches. j/k move, tab cycles the standings,
5 * the matches, the policy card, the match of the day — the id everybody is
5 * the matches, the policy card and what the round planner has done, the match of the day — the id everybody is
66 * playing today and the board of who has played it best — and the range's runs,
77 * x removes the selected row from whichever of the two tables it is in, r
88 * refreshes, q leaves. Plain ANSI, no dependencies.
@@ -27,6 +27,7 @@ import {
2727 } from "./db.ts";
2828 import { dailyOn } from "./daily.ts";
2929 import { CARD } from "./client/weights.ts";
30 import { readReadout, type Readout } from "./plan/readout.ts";
3031 import { printable, screen, type Data, type View } from "./terminal/screen.ts";
3132 import { press, removed } from "./terminal/keys.ts";
3233
@@ -57,6 +58,12 @@ export type Io = {
5758 db: Database;
5859 /** FORGE_APP_ENV, FORGE_APP_PULL and FORGE_APP_SSH are all that is read. */
5960 env: Record<string, string | undefined>;
61 /**
62 * What the round planner has done, as the server last left it: the real one
63 * reads FORGE_APP_DATA/planner.json, a test hands back a readout of its own.
64 * Absent reads as none.
65 */
66 plans?(): Readout | null;
6067 };
6168
6269 /** Wipe the screen and put the cursor home. Written on every paint. */
@@ -95,6 +102,7 @@ export function run(io: Io): void {
95102 total: matchCount(io.db),
96103 runTotal: runCount(io.db),
97104 card: CARD,
105 plans: io.plans?.() ?? null,
98106 // Read with the rows, and only ever here: the screen tells a live match
99107 // from an abandoned one against the moment it was handed.
100108 now,
@@ -159,5 +167,6 @@ if (import.meta.main) {
159167 exit: (code) => process.exit(code),
160168 db: openDatabase(),
161169 env: process.env,
170 plans: () => readReadout(process.env.FORGE_APP_DATA ?? "./data"),
162171 });
163172 }

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs