Public: anyone can read this; the crew can change it.

A run's cost line shows tokens in and out and the price per million used #13985

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

torque/forge:torque/13968-a-run-s-cost-line-shows-tokens-in-and-ou → main 8 commits fork 3 threads · 3 unresolved

  • Merged. torque/13968-a-run-s-cost-line-shows-tokens-in-and-ou was merged into main as 96e6403.

Review

Sign in to review this change.
8 files changed +287 −2
docs/features/README.md +1
@@ -266,6 +266,7 @@ Read with the guide page [Your apps](../guide-your-apps.md).
266266 - [Pause and resume a maintenance duty](maintenance-duties.md)
267267 - [Pinned toolchains (rust)](toolchains.md)
268268 - [A run's page folds its steps and its log](run-folds.md)
269 - [A run's cost line shows tokens and the price per million](run-price.md)
269270 - [Scheduled tools: the forge calls an absorbed app's tool on time](feature-schedule.md)
270271 - [Scrape evidence](scrapes.md)
271272 - [The box pass](box-pass.md)
docs/features/run-price.md added +19
@@ -0,0 +1,19 @@
1 # Run price
2
3 A finished agent run's page, and its log screen in the terminal, show the arithmetic behind its cost beside the ledger's figure (#13968):
4
5 200,000 in × $3/M + 10,000 out × $15/M = $0.7500
6
7 Tokens in and out come from the usage ledger (`core/usage.ts`), counted by the provider. The prices are dollars per 1,000,000 tokens. The product is computed in whole micro-dollars (`tokens × $/M` is already µ$), so no float drift enters. When the product and the ledger's `costUsd` differ at the cent, the line ends `· ledger $0.4000`: the ledger is the provider's own total and counts cache reads and writes at one rate, so the two can honestly differ.
8
9 A model with no declared price reads `price per million unknown`. It is never guessed. A run that spent no tokens shows no line. The line is shown to the people who may read the run's log (writers on the page, the same readers in the terminal).
10
11 ## Declaring prices: `FORGE_MODEL_PRICES`
12
13 `model=in/out`, comma separated, dollars per million tokens, e.g. `claude-opus-5-5=15/75,claude-sonnet-5-5=3/15`. The model name is the one the run recorded (`run.model`). An entry that is not exactly that shape (a model of letters, digits and `._:-`, two numbers of up to six digits and six decimals) is dropped; the rest still apply. The variable is parsed once and again only if its text changes.
14
15 It is a variable of the forge process itself, set by whoever runs it, like the other `FORGE_*` variables. forge.json cannot carry it: `env` there names secrets the app reads and refuses any name starting `FORGE_`, and holds no values. Until it is set every run says unknown.
16
17 ## Where it is
18
19 `src/core/run_price.ts` (`parsePrices`, `runPrice`, `priceWords`), the page in `runPriceLine` (`src/web/pages/runs.ts`), the terminal in `src/tui/screens/runlog.ts`. The crew-activity row on the issue and pull pages still shows the cost alone: the line is one run's, and a row has no room for it. Tests: `test/cost-tokens.test.ts`, `test/tui-runlog.test.ts`.
src/core/run_price.ts added +104
@@ -0,0 +1,104 @@
1 /**
2 * Run price — a run's cost line as arithmetic a reader can check:
3 * tokens in, tokens out, the price per million of each, and the product,
4 * set beside the ledger's own `costUsd` (`core/usage.ts`).
5 *
6 * Units: prices are US dollars per 1,000,000 tokens. The product is
7 * usd = tokensIn × inPerM / 1e6 + tokensOut × outPerM / 1e6
8 * computed in whole micro-dollars so no float drift enters the comparison.
9 *
10 * The ledger's `costUsd` is the provider's own total, and `tokensIn` counts
11 * cache reads and writes at one rate, so a product can honestly differ from
12 * it: when it does the page shows both. A model with no declared price is
13 * `unknown`, never a guess. Prices come from `FORGE_MODEL_PRICES`, e.g.
14 * `opus=15/75,sonnet=3/15` (model=in/out, dollars per million).
15 */
16
17 export interface Price {
18 readonly inPerM: number;
19 readonly outPerM: number;
20 }
21
22 export type Prices = ReadonlyMap<string, Price>;
23
24 export type RunPrice =
25 | { readonly known: false }
26 | {
27 readonly known: true;
28 readonly price: Price;
29 /** Dollars by the declared price, 6 places. */
30 readonly productUsd: number;
31 readonly ledgerUsd: number;
32 /** True when the product and the ledger differ at the ledger's cent. */
33 readonly differs: boolean;
34 };
35
36 const NUMBER = /^\d{1,6}(\.\d{1,6})?$/;
37 const MODEL = /^[A-Za-z0-9._:-]{1,100}$/;
38
39 /** Parse `model=in/out,…`; an entry that is not exactly that is dropped. */
40 export function parsePrices(text: string | undefined): Prices {
41 const out = new Map<string, Price>();
42 for (const part of (text ?? "").split(",")) {
43 const [model = "", rates = "", extra] = part.trim().split("=");
44 const [a = "", b = "", more] = rates.split("/");
45 if (extra !== undefined || more !== undefined) continue;
46 if (!MODEL.test(model) || !NUMBER.test(a) || !NUMBER.test(b)) continue;
47 out.set(model, { inPerM: Number(a), outPerM: Number(b) });
48 }
49 return out;
50 }
51
52 /**
53 * Tokens × dollars-per-million is already whole micro-dollars: tokens × $/M ÷ 1e6 dollars = tokens
54 * × $/M µ$.
55 */
56 const micro = (tokens: number, perM: number): number => Math.round(tokens * perM);
57
58 export function runPrice(
59 usage: { tokensIn: number; tokensOut: number; costUsd: number },
60 model: string,
61 prices: Prices,
62 ): RunPrice {
63 const price = prices.get(model);
64 if (price === undefined) return { known: false };
65 const productUsd = (micro(usage.tokensIn, price.inPerM) + micro(usage.tokensOut, price.outPerM)) / 1e6;
66 const cents = (usd: number): number => Math.round(usd * 100);
67 return {
68 known: true,
69 price,
70 productUsd,
71 ledgerUsd: usage.costUsd,
72 differs: cents(productUsd) !== cents(usage.costUsd),
73 };
74 }
75
76 let parsedFrom: string | undefined;
outdated

Env is re-parsed on every render, and FORGE_MODEL_PRICES is undocumented outside this comment and unset in forge.json. Parse once at boot and document in docs/features/.

77 let parsed: Prices | undefined;
78
79 /** `FORGE_MODEL_PRICES`, parsed once: re-read only when the variable's text changes. */
80 export function pricesFromEnv(): Prices {
81 const text = process.env.FORGE_MODEL_PRICES;
82 if (parsed === undefined || text !== parsedFrom) {
83 parsed = parsePrices(text);
84 parsedFrom = text;
85 }
86 return parsed;
87 }
88
89 /**
90 * The line as plain words, for the terminal:
91 * `1,200 in × $3/M + 300 out × $15/M = $0.0081 · ledger $0.01`, or
92 * `price per million unknown`.
93 */
94 export function priceWords(
95 usage: { tokensIn: number; tokensOut: number; costUsd: number },
96 model: string,
97 prices: Prices = pricesFromEnv(),
98 ): string {
99 const p = runPrice(usage, model, prices);
100 if (!p.known) return "price per million unknown";
101 const n = (t: number): string => t.toLocaleString("en-US");
102 const ledger = p.differs ? ` · ledger $${p.ledgerUsd.toFixed(4)}` : "";
103 return `${n(usage.tokensIn)} in × $${p.price.inPerM}/M + ${n(usage.tokensOut)} out × $${p.price.outPerM}/M = $${p.productUsd.toFixed(4)}${ledger}`;
104 }
src/core/runs.ts +4
@@ -104,6 +104,8 @@ export interface Run {
104104 readonly turns: number;
105105 /** For an agent run: the context its session held at the end, in tokens; 0 when unreported. */
106106 readonly contextTokens: number;
107 /** For an agent run: the model it ran on; empty when not recorded. */
108 readonly model: string;
107109 /**
108110 * For an agent run: tokens it read and wrote, all invocations summed;
109111 * null when unknown (never 0 for unknown).
@@ -155,6 +157,7 @@ interface RunRow {
155157 heartbeat_at: string | null;
156158 turns: number;
157159 context_tokens: number;
160 model: string | null;
158161 tokens_in: number | null;
159162 tokens_out: number | null;
160163 pushed: number | null;
@@ -192,6 +195,7 @@ const toRun = (r: RunRow): Run => ({
192195 heartbeatAt: r.heartbeat_at,
193196 turns: r.turns,
194197 contextTokens: r.context_tokens,
198 model: r.model ?? "",
195199 tokensIn: r.tokens_in ?? null,
196200 tokensOut: r.tokens_out ?? null,
197201 pushed: r.pushed === null ? null : r.pushed !== 0,
src/tui/screens/runlog.ts +22 −2
@@ -20,6 +20,8 @@ import { subscribeRunLog } from "../../runs/log.ts";
2020 import { stoppedWords } from "../../runs/cancelled.ts";
2121 import { Scroll, live, openOnWeb } from "./common.ts";
2222 import { costText, envLabel, runTone } from "./runwords.ts";
23 import { priceWords } from "../../core/run_price.ts";
24 import { getUsage } from "../../core/usage.ts";
2325 import { jobSandboxWords } from "../../checks/placement.ts";
2426
2527 const LOG_LINES = 200;
@@ -36,6 +38,8 @@ export function runLogScreen(
3638 let view: RunLogView | undefined;
3739 let lines: string[] = [];
3840 let loaded = false;
41 /** The run's tokens × price line (`core/run_price.ts`); empty until the ledger has the run. */
42 let price = "";
3943 let follow = true;
4044
4145 const read = (): void => {
@@ -44,6 +48,12 @@ export function runLogScreen(
4448 view = got.ok && !got.value.logHidden ? got.value : undefined;
4549 lines = // The tail window can hold a few more than asked for.
4650 lines = view === undefined ? [] : view.lines.slice(-LOG_LINES);
51 const used =
52 view !== undefined && view.run.kind === "agent" ? getUsage(f.app.db, runId) : undefined;
53 price =
54 view === undefined || used === undefined || used.tokensIn + used.tokensOut === 0
55 ? ""
56 : priceWords(used, view.run.model);
4757 loaded = true;
4858 if (follow) scroll.top = Number.MAX_SAFE_INTEGER;
4959 ctx.invalidate();
@@ -110,7 +120,12 @@ export function runLogScreen(
110120 // does. One sentence a row, so a narrow terminal cuts neither.
111121 const wall =
112122 run.placement === "job" && run.sandboxEnforced === true ? jobSandboxWords(run.sandboxNote).split(/(?<=\.) /) : [];
113 const rows = split(area, "v", [len(1), ...wall.map(() => len(1)), fill(1)]);
123 const rows = split(area, "v", [
124 len(1),
125 ...(price === "" ? [] : [len(1)]),
126 ...wall.map(() => len(1)),
127 fill(1),
128 ]);
114129 const head = rows[0];
115130 const body = rows.at(-1);
116131 const tone = runTone(run.state);
@@ -130,8 +145,13 @@ export function runLogScreen(
130145 style(tone === undefined ? c.mutedFg : c.fg, c.bg),
131146 head!.x + head!.w,
132147 );
148 const lead = price === "" ? 1 : 2;
149 if (price !== "") {
150 const row = rows[1]!;
151 s.text(row.x, row.y, truncate(price, row.w), style(c.mutedFg, c.bg), row.x + row.w);
152 }
133153 wall.forEach((words, i) => {
134 const row = rows[i + 1]!;
154 const row = rows[i + lead]!;
135155 s.text(row.x, row.y, truncate(words, row.w), style(c.mutedFg, c.bg), row.x + row.w);
136156 });
137157 if (lines.length === 0) {
src/web/pages/runs.ts +24
@@ -31,6 +31,7 @@ import {
3131 } from "../../core/repos.ts";
3232 import { getRun, type Run, type RunKind } from "../../core/runs.ts";
3333 import { getUsage, issueUsage } from "../../core/usage.ts";
34 import { pricesFromEnv, runPrice, type Prices } from "../../core/run_price.ts";
3435 import { costLineFor, type CostThing } from "../../core/ledger_rollup.ts";
3536 import { getUsersByIds, type User } from "../../core/users.ts";
3637 import { notePhase, takeSessionTime } from "../../core/route_timing.ts";
@@ -160,6 +161,29 @@ export function runTokens(db: Database, run: Run): Html {
160161 return html`<span class="num" id="run-tokens"
161162 >${tokenCount(used.tokensIn, "counted")} in ·
162163 ${tokenCount(used.tokensOut, "counted")} out</span
164 > ${runPriceLine(run.model, used)}`;
outdated

Both doors: the terminal run screen has no price line, and there is no docs/features how-to. Add both.

165 }
166
167 /**
168 * The arithmetic behind a run's cost (`core/run_price.ts`): tokens × price per
169 * million = product, and the ledger's own figure beside it when the two differ
170 * at the cent. A model with no declared price says "price per million unknown".
171 */
172 export function runPriceLine(
173 model: string,
174 used: { tokensIn: number; tokensOut: number; costUsd: number },
175 prices: Prices = pricesFromEnv(),
176 ): Html {
177 const priced = runPrice(used, model, prices);
outdated

Extra query per render for a model the caller's run already carries. Take the model as an argument.

178 if (!priced.known)
179 return html`<span class="num muted" id="run-price">price per million unknown</span>`;
180 const { price, productUsd, ledgerUsd, differs } = priced;
181 return html`<span class="num" id="run-price"
182 >${tokenCount(used.tokensIn, "counted")} × $${String(price.inPerM)}/M +
183 ${tokenCount(used.tokensOut, "counted")} × $${String(price.outPerM)}/M =
184 $${productUsd.toFixed(4)}${differs
185 ? html` · ledger $${ledgerUsd.toFixed(4)}`
186 : html``}</span
163187 >`;
164188 }
165189
test/cost-tokens.test.ts added +92
@@ -0,0 +1,92 @@
1 /** A run's cost line: tokens × price per million, by hand, against the ledger (#13968). */
2 import { expect, test } from "bun:test";
3 import { mkdtempSync } from "node:fs";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { createApp } from "../src/app.ts";
7 import { createRepo } from "../src/actions/repos.ts";
8 import { createUser } from "../src/core/users.ts";
9 import { claimRun, createRun, finishRun, noteRunProgress } from "../src/core/runs.ts";
10 import { parsePrices, pricesFromEnv, priceWords, runPrice } from "../src/core/run_price.ts";
11 import { getUsage, recordUsage } from "../src/core/usage.ts";
12 import { runPriceLine } from "../src/web/pages/runs.ts";
13
14 const prices = parsePrices("opus=15/75,sonnet=3/15,bad=x/1,also=1/2/3");
15
16 test("parsePrices keeps well-formed entries and drops the rest", () => {
17 expect([...prices.keys()]).toEqual(["opus", "sonnet"]);
18 expect(parsePrices(undefined).size).toBe(0);
19 });
20
21 test("by hand: 200,000 in × $3/M + 10,000 out × $15/M = $0.60 + $0.15 = $0.75, equal to the ledger", () => {
22 const r = runPrice({ tokensIn: 200_000, tokensOut: 10_000, costUsd: 0.75 }, "sonnet", prices);
23 expect(r).toMatchObject({ known: true, productUsd: 0.75, ledgerUsd: 0.75, differs: false });
24 });
25
26 test("by hand: 1,234,567 in × $15/M + 7,891 out × $75/M = $18.518505 + $0.591825 = $19.11033", () => {
27 const r = runPrice({ tokensIn: 1_234_567, tokensOut: 7_891, costUsd: 19.11 }, "opus", prices);
28 expect(r).toMatchObject({ known: true, productUsd: 19.11033, differs: false });
29 });
30
31 test("a ledger figure that differs at the cent is flagged", () => {
32 const r = runPrice({ tokensIn: 200_000, tokensOut: 10_000, costUsd: 0.4 }, "sonnet", prices);
33 expect(r).toMatchObject({ known: true, productUsd: 0.75, ledgerUsd: 0.4, differs: true });
34 });
35
36 test("a model with no declared price is unknown", () => {
37 expect(runPrice({ tokensIn: 1, tokensOut: 1, costUsd: 1 }, "mystery", prices)).toEqual({ known: false });
38 expect(runPrice({ tokensIn: 1, tokensOut: 1, costUsd: 1 }, "", prices)).toEqual({ known: false });
39 });
40
41 async function seed(model: string | undefined) {
42 const app = createApp({ dataDir: mkdtempSync(join(process.env.TMPDIR ?? tmpdir(), "price-")), port: 0 });
43 const u = await createUser(app.db, { origin: "seed", trusted: true, username: "alice", email: "[email protected]", password: "a-good-passphrase", adminIfFirst: true });
44 if (!u.ok) throw new Error(u.error);
45 const made = await createRepo(app, u.value, { name: "r" });
46 if (!made.ok) throw new Error(String(made.error));
47 const run = createRun(app.db, { repoId: made.value.id, kind: "agent" });
48 claimRun(app.db, run.id);
49 if (model !== undefined) noteRunProgress(app.db, run.id, { model });
50 finishRun(app.db, run.id, { state: "ok", summary: "done", costUsd: 0.75 });
51 recordUsage(app.db, {
52 runId: run.id, repoId: made.value.id, issueId: null, agentId: null, subscriptionId: null,
53 tokensIn: 200_000, tokensOut: 10_000, costUsd: 0.75, wallMs: 1, outcome: "ok", at: new Date().toISOString(),
54 });
55 return { db: app.db, id: run.id };
56 }
57
58 test("the run page line shows tokens, price per million and the product equal to the ledger cost", async () => {
59 const { db, id } = await seed("sonnet");
60 const used = getUsage(db, id)!;
61 const text = String(runPriceLine("sonnet", used, prices));
62 expect(text).toContain("$3/M");
63 expect(text).toContain("$15/M");
64 expect(text).toContain("$0.7500");
65 expect(text).not.toContain("ledger $");
66 expect(Number((/=\s*\$([0-9.]+)/.exec(text.replace(/<[^>]*>/g, ""))?.[1]))).toBe(used.costUsd);
67 });
68
69 test("the run page line says unknown for an unpriced model", async () => {
70 const { db, id } = await seed("mystery");
71 expect(String(runPriceLine("mystery", getUsage(db, id)!, prices))).toContain("price per million unknown");
72 });
73
74 test("priceWords is the terminal's line: the same arithmetic as words", () => {
75 const u = { tokensIn: 200_000, tokensOut: 10_000, costUsd: 0.4 };
76 expect(priceWords(u, "sonnet", prices)).toBe("200,000 in × $3/M + 10,000 out × $15/M = $0.7500 · ledger $0.4000");
77 expect(priceWords(u, "nope", prices)).toBe("price per million unknown");
78 });
79
80 test("pricesFromEnv parses once per text, not per call", () => {
81 const was = process.env.FORGE_MODEL_PRICES;
82 try {
83 process.env.FORGE_MODEL_PRICES = "a=1/2";
84 const first = pricesFromEnv();
85 expect(pricesFromEnv()).toBe(first);
86 process.env.FORGE_MODEL_PRICES = "a=3/4";
87 expect(pricesFromEnv().get("a")).toEqual({ inPerM: 3, outPerM: 4 });
88 } finally {
89 if (was === undefined) delete process.env.FORGE_MODEL_PRICES;
90 else process.env.FORGE_MODEL_PRICES = was;
91 }
92 });
test/tui-runlog.test.ts +21
@@ -7,6 +7,7 @@ import { appendFileSync } from "node:fs";
77 import { announceRun, cancelRepoRun } from "../src/actions/runs.ts";
88 import { createRepo } from "../src/actions/repos.ts";
99 import { createApp, type App as Forge } from "../src/app.ts";
10 import { recordUsage } from "../src/core/usage.ts";
1011 import { claimRun, createRun, finishRun, noteRunProgress } from "../src/core/runs.ts";
1112 import { JOB_GATE_NOTE } from "../src/checks/job_gate.ts";
1213 import type { Repo } from "../src/core/repos.ts";
@@ -199,3 +200,23 @@ test("r rereads the log, and o prints the web address", async () => {
199200 await key(app, "o");
200201 expect(app.screen.toText()).toContain(`/alice/lamp/runs/${run.id}`);
201202 });
203
204 test("an agent run's log screen shows tokens × price per million (#13968)", async () => {
205 const run = createRun(forge.db, { repoId: repo.id, kind: "agent" });
206 claimRun(forge.db, run.id);
207 noteRunProgress(forge.db, run.id, { model: "tui-priced" });
208 finishRun(forge.db, run.id, { state: "ok", summary: "done", costUsd: 0.75 });
209 recordUsage(forge.db, {
210 runId: run.id, repoId: repo.id, issueId: null, agentId: null, subscriptionId: null,
211 tokensIn: 200_000, tokensOut: 10_000, costUsd: 0.75, wallMs: 1, outcome: "ok", at: new Date().toISOString(),
212 });
213 const was = process.env.FORGE_MODEL_PRICES;
214 process.env.FORGE_MODEL_PRICES = "tui-priced=3/15";
215 try {
216 const app = await open(alice, run.id);
217 expect(app.screen.toText()).toContain("200,000 in × $3/M + 10,000 out × $15/M = $0.7500");
218 } finally {
219 if (was === undefined) delete process.env.FORGE_MODEL_PRICES;
220 else process.env.FORGE_MODEL_PRICES = was;
221 }
222 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs