trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns15g rEventsFlows

Tokens as a unit people can feel: files, runs, issues and pulls carry a token count the way storage carries MB and GB #554

Merged torquecrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/403-tokens-as-a-unit-people-can-feel-files-r → main 8 commits fork

  • Merged. torque/403-tokens-as-a-unit-people-can-feel-files-r was merged into main as 63af7f3.

Review

Sign in to review this change.
17 files changed +655 −18
docs/DESIGN.md +12
@@ -93,6 +93,16 @@ Rules:
9393 and descriptions.
9494 - **Truncation is honest**: one line, ellipsis, and a `title` with the whole
9595 string. Never a fade-out mask — masks lie about whether there is more.
96 - **A token count is shown where a byte size is shown, never as a badge on
97 its own.** Tokens are a unit, like KB: they sit beside a file's size, a
98 directory's total, a run's cost or a window's dollars and percentage, so
99 the context intuition grows next to the storage and money ones. One
100 formatter (`kernel/tokens-format.ts`, rendered by `tokenCount()` /
101 `sizeWithTokens()` in `web/ui.ts`) writes them all: `850 tok`, `1.2k tok`,
102 `48k tok`, `1.1M tok`. A count the provider reported is bare; an estimate
103 (bytes ÷ a stated bytes-per-token ratio for prose, code or data) carries a
104 `~`, and every count's `title` says which tokenizer and whether it is an
105 estimate. A binary file has no token count — no number beats a wrong one.
96106
97107 ---
98108
@@ -1123,3 +1133,5 @@ deletes a repository.
11231133 9. Nothing inside a linked row is itself a link.
11241134 10. Any control that only works with a script carries `js-only`, so a page
11251135 without JavaScript never shows a dead button.
1136 11. Every token count sits beside a byte size, a cost or a percentage, and
1137 goes through `tokenCount()` — never a hand-written `k`.
forge.json +1 −1
@@ -6,7 +6,7 @@
66 "ready": 90
77 },
88 "check": {
9 "run": "bun x tsc --noEmit && bun scripts/test-shards.ts --timeout 60000 -- crew review-route review-lines review-gate crew-wrapup crew-limits crew-parks crew-sandbox steer session-settle landing decide ledger checks dependencies map-closing autonomy studio studio-words forks fork-flow cross-repo-pull pulls apps apps-adversarial apps-busy apps-token surfaces-adversarial orgs web git-trust helpers shards migrate vitals guard asks feed feed-page offline share cloud run-secrets forge-cloudflare usage credentials standing board board-windows spawn-plan spawn web-apps fleet-ops meter told-by",
9 "run": "bun x tsc --noEmit && bun scripts/test-shards.ts --timeout 60000 -- crew review-route review-lines review-gate crew-wrapup crew-limits crew-parks crew-sandbox steer session-settle landing decide ledger checks dependencies map-closing autonomy studio studio-words forks fork-flow cross-repo-pull pulls apps apps-adversarial apps-busy apps-token surfaces-adversarial orgs web git-trust helpers shards migrate vitals guard asks feed feed-page offline share cloud run-secrets forge-cloudflare usage tokens credentials standing board board-windows spawn-plan spawn web-apps fleet-ops meter told-by",
1010 "timeout": 1800
1111 }
1212 }
src/core/usage.ts +28
@@ -126,6 +126,34 @@ export function getUsage(db: Database, runId: string): UsageEntry | undefined {
126126 return row ? toEntry(row) : undefined;
127127 }
128128
129 /**
130 * Everything the runs on one issue or pull spent, over all time, or
131 * `undefined` when none has landed in the ledger. One aggregate over the
132 * table: the ledger is a row per run, small beside the run table itself.
133 */
134 export function issueUsage(
135 db: Database,
136 issueId: string,
137 ): { runs: number; tokensIn: number; tokensOut: number; costUsd: number } | undefined {
138 const row = db
139 .query<
140 { runs: number; tokens_in: number; tokens_out: number; cost_usd: number },
141 [string]
142 >(
143 `SELECT COUNT(*) AS runs, SUM(tokens_in) AS tokens_in,
144 SUM(tokens_out) AS tokens_out, SUM(cost_usd) AS cost_usd
145 FROM usage WHERE issue_id = ?1`,
146 )
147 .get(issueId);
148 if (!row || row.runs === 0) return undefined;
149 return {
150 runs: row.runs,
151 tokensIn: row.tokens_in,
152 tokensOut: row.tokens_out,
153 costUsd: row.cost_usd,
154 };
155 }
156
129157 /** What the ledger can be summed by. Each maps to one fixed column. */
130158 export type UsageBy = "subscription" | "repo" | "agent" | "issue" | "kind";
131159 export const USAGE_BY: readonly UsageBy[] = [
src/kernel/tokens-format.ts added +175
@@ -0,0 +1,175 @@
1 /**
2 * Tokens as a unit — the one formatter for a token count, and the one
3 * estimate for text nobody has run through a model yet.
4 *
5 * A count is shown the way a byte size is (`web/ui.ts` `fileSize`): short,
6 * tabular, never on its own (docs/DESIGN.md, "Token counts"). Two bases:
7 *
8 * - `counted` — the provider reported it (the usage ledger, `core/usage.ts`:
9 * Claude's own tokenizer, whole tokens). Shown bare: `48k tok`.
10 * - `estimated` — bytes divided by a calibrated bytes-per-token ratio for
11 * the file's kind. Shown with a tilde: `~1.2k tok`.
12 *
13 * The ratios are an assumption, stated here and in every tooltip: Anthropic
14 * documents roughly 3.5 English characters per Claude token, and OpenAI's
15 * cl100k guidance is ~4 for prose. Code splits finer than prose (operators,
16 * indentation, identifiers broken at case changes); structured data finer
17 * still (quotes, braces, short keys). The ratios below are those rules of
18 * thumb, not a measurement: good enough to feel a size, never to bill one.
19 *
20 * Pure: plain numbers and strings in and out, no I/O.
21 */
22
23 export type TokenBasis = "counted" | "estimated";
24
25 /** What a file is, for the purpose of how finely it tokenizes. */
26 export type TextKind = "prose" | "code" | "data";
27
28 /** Bytes per token, by kind. Prose 4.0, code 3.5, data 3.0 (see above). */
29 export const BYTES_PER_TOKEN: Readonly<Record<TextKind, number>> = {
30 prose: 4.0,
31 code: 3.5,
32 data: 3.0,
33 };
34
35 const PROSE = new Set(["md", "markdown", "txt", "rst", "adoc", "org", "tex"]);
36 const DATA = new Set([
37 "json",
38 "jsonl",
39 "ndjson",
40 "yaml",
41 "yml",
42 "toml",
43 "csv",
44 "tsv",
45 "xml",
46 "svg",
47 "html",
48 "htm",
49 "lock",
50 "ini",
51 "cfg",
52 "sql",
53 ]);
54 // Files a model would not read as text at all: no estimate is better than
55 // a confident wrong one.
56 const BINARY = new Set([
57 "png",
58 "jpg",
59 "jpeg",
60 "gif",
61 "webp",
62 "ico",
63 "bmp",
64 "avif",
65 "pdf",
66 "zip",
67 "gz",
68 "tgz",
69 "bz2",
70 "xz",
71 "zst",
72 "7z",
73 "tar",
74 "jar",
75 "wasm",
76 "woff",
77 "woff2",
78 "ttf",
79 "otf",
80 "eot",
81 "mp3",
82 "mp4",
83 "mov",
84 "webm",
85 "ogg",
86 "wav",
87 "flac",
88 "db",
89 "sqlite",
90 "so",
91 "dylib",
92 "dll",
93 "exe",
94 "bin",
95 "o",
96 "a",
97 "class",
98 "pyc",
99 ]);
100
101 function extension(path: string): string {
102 const name = path.slice(path.lastIndexOf("/") + 1).toLowerCase();
103 const dot = name.lastIndexOf(".");
104 return dot <= 0 ? "" : name.slice(dot + 1);
105 }
106
107 /**
108 * How a path tokenizes, or `undefined` for a file that is not text. A name
109 * with no extension (`Makefile`, `LICENSE`) counts as code: the finer ratio
110 * errs toward more tokens, the safer side of an estimate about context.
111 */
112 export function textKind(path: string): TextKind | undefined {
113 const ext = extension(path);
114 if (BINARY.has(ext)) return undefined;
115 if (PROSE.has(ext)) return "prose";
116 if (DATA.has(ext)) return "data";
117 return "code";
118 }
119
120 /**
121 * The estimated token count of `bytes` of the file at `path`: bytes ÷ the
122 * kind's ratio, rounded up (one byte is one token, never zero). `undefined`
123 * for a binary file or a size that is not a byte count.
124 */
125 export function estimateTokens(bytes: number, path: string): number | undefined {
126 if (!Number.isSafeInteger(bytes) || bytes < 0) return undefined;
127 const kind = textKind(path);
128 if (kind === undefined) return undefined;
129 return Math.ceil(bytes / BYTES_PER_TOKEN[kind]);
130 }
131
132 /**
133 * A token count the way storage is written: `850 tok`, `1.2k tok`,
134 * `48k tok`, `1.1M tok`. One decimal below ten of a unit, whole above;
135 * rounding that reaches the next unit moves to it (999,999 is `1.0M`, not
136 * `1000k`). An estimate carries a leading `~`.
137 */
138 export function formatTokens(n: number, basis: TokenBasis = "counted"): string {
139 const mark = basis === "estimated" ? "~" : "";
140 const count = Number.isFinite(n) && n > 0 ? Math.round(n) : 0;
141 return `${mark}${scaled(count)} tok`;
142 }
143
144 function scaled(n: number): string {
145 if (n < 1000) return String(n);
146 for (const [unit, size] of [
147 ["k", 1e3],
148 ["M", 1e6],
149 ["B", 1e9],
150 ] as const) {
151 const v = n / size;
152 const text = v < 9.95 ? v.toFixed(1) : String(Math.round(v));
153 if (Number(text) < 1000 || unit === "B") return `${text}${unit}`;
154 }
155 return String(n);
156 }
157
158 /**
159 * The tooltip every count carries: which tokenizer, the exact number, and
160 * — for an estimate — the ratio it came from and that it is one.
161 */
162 export function tokenNote(
163 n: number,
164 basis: TokenBasis,
165 kind?: TextKind,
166 ): string {
167 const exact = Math.max(0, Math.round(Number.isFinite(n) ? n : 0));
168 const count = `${exact.toLocaleString("en-US")} token${exact === 1 ? "" : "s"}`;
169 if (basis === "counted") return `${count}, as counted by Claude's tokenizer`;
170 const ratio =
171 kind === undefined
172 ? "3–4 bytes per token"
173 : `${BYTES_PER_TOKEN[kind].toFixed(1)} bytes per token for ${kind}`;
174 return `About ${count} — an estimate at ${ratio}; Claude's tokenizer will differ`;
175 }
src/web/pages/board.ts +5 −2
@@ -50,7 +50,7 @@ import { repoBase, signInFirst } from "../chrome.ts";
5050 import { boardJs, liveRegionsJs } from "../client.ts";
5151 import { html, when, type Html } from "../html.ts";
5252 import { page, redirect } from "../layout.ts";
53 import { avatar, icon, time } from "../ui.ts";
53 import { avatar, icon, time, tokenCount } from "../ui.ts";
5454 import { voiceFor } from "../studio/voice.ts";
5555 import {
5656 perShippedWords,
@@ -384,7 +384,10 @@ function windowsStrip(ctx: Ctx): Html | null {
384384 ></span>`
385385 }
386386 <span class="board-window-nums"
387 >${figures([words.used, words.remaining, words.resets])}</span
387 >${figures([words.used, words.remaining, words.resets])}${when(
388 w.tokensIn + w.tokensOut > 0,
389 () => html` · ${tokenCount(w.tokensIn + w.tokensOut, "counted")}`,
390 )}</span
388391 ></span
389392 >`;
390393 })}
src/web/pages/edit.ts +14 −1
@@ -37,7 +37,7 @@ import {
3737 type ProposeTarget,
3838 type RepoScope,
3939 } from "../scope.ts";
40 import { btn, card, field, icon } from "../ui.ts";
40 import { btn, card, field, icon, sizeWithTokens } from "../ui.ts";
4141 import { pathCrumbs, refSetFrom } from "./refs.ts";
4242
4343 /** What the form is holding, so a refusal can hand every word back. */
@@ -190,6 +190,19 @@ function editorForm(
190190 pathCrumbs(base, scope.facts.name, draft.ref, draft.path, true),
191191 )}
192192 <span class="spacer"></span>
193 ${when(
194 !draft.creating,
195 () =>
196 html`<span
197 class="muted small num"
198 id="editor-size"
199 title="The file as opened, before your changes"
200 >${sizeWithTokens(
201 new TextEncoder().encode(draft.content).byteLength,
202 draft.path,
203 )}</span
204 >`,
205 )}
193206 ${when(
194207 isMarkdownPath(draft.path) || draft.creating,
195208 () =>
src/web/pages/issues.ts +2 −1
@@ -61,7 +61,7 @@ import {
6161 threadContext,
6262 threadFragment,
6363 } from "./thread.ts";
64 import { crewActivityCard } from "./runs.ts";
64 import { crewActivityCard, workSpent } from "./runs.ts";
6565 import { studioTells } from "../studio/lists.ts";
6666 import {
6767 PAGE_TITLES,
@@ -540,6 +540,7 @@ export async function issuePage(ctx: Ctx): Promise<Response> {
540540 >${icon("external")}view on origin</a
541541 >`,
542542 )}
543 ${workSpent(ctx, repo, issue.id)}
543544 </div>
544545 </div>
545546 <div class="split">
src/web/pages/pull.ts +2 −1
@@ -96,7 +96,7 @@ import {
9696 import { previewCard } from "./apps.ts";
9797 import { lookCard } from "./looks.ts";
9898 import { pullState } from "./pulls.ts";
99 import { crewActivityCard } from "./runs.ts";
99 import { crewActivityCard, workSpent } from "./runs.ts";
100100 import { studioPull } from "../studio/change.ts";
101101 import { studioRecord } from "../studio/nouns.ts";
102102
@@ -834,6 +834,7 @@ export async function pullPage(ctx: Ctx): Promise<Response> {
834834 >${icon("external")}view on origin</a
835835 >`,
836836 )}
837 ${workSpent(ctx, repo, pull.id)}
837838 <span class="mono">${pull.headBranch} → ${pull.baseBranch}</span>
838839 ${badge(`${status.ahead} commit${status.ahead === 1 ? "" : "s"}`, {
839840 variant: "secondary",
src/web/pages/repo.ts +34 −3
@@ -28,6 +28,7 @@ import {
2828 import { isSafeRef, isSafeTreePath } from "../../git/guard.ts";
2929 import { blobForRender, type TreeEntry } from "../../git/read.ts";
3030 import { repoDir } from "../../git/repos.ts";
31 import { estimateTokens } from "../../kernel/tokens-format.ts";
3132 import { safeBack, type Ctx } from "../../http/context.ts";
3233 import { json, notFound } from "../../http/respond.ts";
3334 import type { Blob, RepoSource } from "../../view/source.ts";
@@ -57,6 +58,8 @@ import {
5758 card,
5859 empty,
5960 fileSize,
61 sizeWithTokens,
62 tokenCount,
6063 icon,
6164 iconBtn,
6265 listCard,
@@ -328,14 +331,41 @@ export function treeListing(
328331 entry.size >= 0,
329332 () =>
330333 html`<span class="row-trail muted small num"
331 >${fileSize(entry.size)}</span
334 >${sizeWithTokens(entry.size, target)}</span
332335 >`,
333336 )}
334337 </a>
335338 </li>`,
336339 );
337340 }
338 return listCard(rows);
341 return html`${listCard(rows)}${directoryTotal(entries)}`;
342 }
343
344 /**
345 * What the files at this level weigh together: bytes and estimated tokens,
346 * the two summed over the same files. Subdirectories carry no size in a
347 * tree listing (`git ls-tree -l` gives `-`), so they are not in it — the
348 * line says "here" and the tooltip says why. Binary files add bytes but no
349 * tokens. Nothing when there are no files.
350 */
351 export function directoryTotal(entries: readonly TreeEntry[]): Html {
352 let files = 0;
353 let bytes = 0;
354 let tokens = 0;
355 for (const entry of entries) {
356 if (entry.type !== "blob" || entry.size < 0) continue;
357 files += 1;
358 bytes += entry.size;
359 tokens += estimateTokens(entry.size, entry.name) ?? 0;
360 }
361 if (files === 0) return html``;
362 return html`<p
363 class="muted small num tree-total"
364 title="Files at this level only; subdirectories are not counted"
365 >
366 ${files.toLocaleString("en-US")} file${files === 1 ? "" : "s"} here ·
367 ${fileSize(bytes)} · ${tokenCount(tokens, "estimated")}
368 </p>`;
339369 }
340370
341371 /**
@@ -763,7 +793,8 @@ export function renderBlob(
763793 view === "rendered" && o.blob.kind === "text"
764794 ? renderedMarkdown(o.blob)
765795 : blobBody(base, o.ref, o.path, o.blob);
766 const sizeText = o.blob.kind === "too_large" ? "" : fileSize(o.blob.size);
796 const sizeText =
797 o.blob.kind === "too_large" ? "" : sizeWithTokens(o.blob.size, o.path);
767798 const fileHref = `${base}/blob/${encoded}/${o.path}`;
768799 return html`<div class="code-pane" id="code-pane" data-path="${o.path}">
769800 <div class="path-head">
src/web/pages/runs.ts +45 −1
@@ -11,6 +11,7 @@
1111 * id, so the redirect itself says nothing about what exists.
1212 */
1313
14 import type { Database } from "bun:sqlite";
1415 import {
1516 cancelRepoRun,
1617 listRepoRuns,
@@ -18,9 +19,16 @@ import {
1819 steerRepoRun,
1920 } from "../../actions/runs.ts";
2021 import { getIssueById, getIssuesByIds, type Issue } from "../../core/issues.ts";
21 import { canRead, canWrite, getRepoById, type Repo } from "../../core/repos.ts";
22 import {
23 canRead,
24 canSeeOwnerFacts,
25 canWrite,
26 getRepoById,
27 type Repo,
28 } from "../../core/repos.ts";
2229 import { listHeldSecrets } from "../../core/run_secret.ts";
2330 import { getRun, isTerminal, type Run, type RunKind } from "../../core/runs.ts";
31 import { getUsage, issueUsage } from "../../core/usage.ts";
2432 import { getUserById, getUsersByIds, type User } from "../../core/users.ts";
2533 import type { Ctx } from "../../http/context.ts";
2634 import {
@@ -52,6 +60,7 @@ import {
5260 seg,
5361 sha,
5462 time,
63 tokenCount,
5564 } from "../ui.ts";
5665
5766 // ---- words ----
@@ -105,6 +114,38 @@ export const reasonWords = (reason: string): string =>
105114 /** Dollars, two places — a run's spend is a small number read at a glance. */
106115 export const costText = (usd: number): string => `$${usd.toFixed(2)}`;
107116
117 /**
118 * What the crew's runs on one issue or pull spent altogether: the cost and,
119 * beside it, the tokens read and written (`core/usage.ts` `issueUsage`).
120 * The owner's facts, like a run's cost — nothing for a reader who cannot
121 * write here, and nothing before a run has landed in the ledger.
122 */
123 export function workSpent(ctx: Ctx, repo: Repo, issueId: string): Html {
124 if (!canSeeOwnerFacts(ctx.app.db, repo, ctx.auth.user)) return html``;
125 const used = issueUsage(ctx.app.db, issueId);
126 if (used === undefined) return html``;
127 return html`<span class="num" id="work-spent"
128 >${used.runs} run${used.runs === 1 ? "" : "s"} ·
129 ${costText(used.costUsd)} · ${tokenCount(used.tokensIn, "counted")} in ·
130 ${tokenCount(used.tokensOut, "counted")} out</span
131 >`;
132 }
133
134 /**
135 * What a finished agent run read and wrote, in tokens, from the usage ledger
136 * (`core/usage.ts`) — counted by the provider, so no tilde. Placed beside the
137 * cost, never alone; nothing until the ledger has the run, or when it spent
138 * nothing.
139 */
140 export function runTokens(db: Database, run: Run): Html {
141 const used = run.kind === "agent" ? getUsage(db, run.id) : undefined;
142 if (used === undefined || used.tokensIn + used.tokensOut === 0) return html``;
143 return html`<span class="num" id="run-tokens"
144 >${tokenCount(used.tokensIn, "counted")} in ·
145 ${tokenCount(used.tokensOut, "counted")} out</span
146 >`;
147 }
148
108149 /**
109150 * What kind of line a run wrote: a tool the agent reached for (`→ Bash …`),
110151 * a word from the forge itself (cloning, pushing, what the model cost), or
@@ -408,6 +449,9 @@ export async function runPage(ctx: Ctx): Promise<Response> {
408449 <span class="num" id="run-cost" ${run.costUsd > 0 ? null : html`hidden`}
409450 >${costText(run.costUsd)} spent</span
410451 >
452 ${when(canSeeOwnerFacts(ctx.app.db, repo, ctx.auth.user), () =>
453 runTokens(ctx.app.db, run),
454 )}
411455 <span class="num" id="run-turns" ${run.turns > 0 ? null : html`hidden`}
412456 >${run.turns} turn${run.turns === 1 ? "" : "s"}</span
413457 >
src/web/pages/subscriptions.ts +27 −6
@@ -46,6 +46,7 @@ import {
4646 row,
4747 time,
4848 timeAgo,
49 tokenCount,
4950 userLink,
5051 } from "../ui.ts";
5152 import { whenText } from "../../crew/limits.ts";
@@ -72,18 +73,34 @@ const kindText = (s: Subscription): string =>
7273 const usd = (n: number): string => `$${n.toFixed(2)}`;
7374 const mark = (f: Figure<unknown>): string => (f.source === "read" ? "" : "~");
7475
75 /** `5-hour: ~$3.20 used of ~$10.00 · resets 12:04 UTC` — one window, plainly. */
76 function windowText(w: WindowStanding): string {
76 /**
77 * `5-hour: ~$3.20 used of ~$10.00 (32%) · 1.2M tok · resets 12:04 UTC` — one
78 * window, plainly: the dollars, the share of the cap when it is known, and
79 * beside them the tokens the crew's runs read and wrote in it, so the money
80 * and the context intuitions meet in one line.
81 */
82 function windowText(w: WindowStanding): Html {
7783 const name = w.window === "5h" ? "5-hour" : "weekly";
7884 const used = `${mark(w.used)}${usd(w.used.value)} used`;
79 const of = w.cap === null ? "" : ` of ${mark(w.cap)}${usd(w.cap.value)}`;
85 const of =
86 w.cap === null
87 ? ""
88 : ` of ${mark(w.cap)}${usd(w.cap.value)}${
89 w.cap.value > 0
90 ? ` (${Math.min(100, Math.round((w.used.value / w.cap.value) * 100))}%)`
91 : ""
92 }`;
8093 const full =
8194 w.remaining?.source === "read" && w.remaining.value === 0 ? " · full" : "";
8295 const resets =
8396 w.resetsAt === null
8497 ? " · opens on next use"
8598 : ` · resets ${w.resetsAt.source === "read" ? "" : "~"}${whenText(new Date(w.resetsAt.value))}`;
86 return `${name}: ${used}${of}${full}${resets}`;
99 const tokens = w.tokensIn + w.tokensOut;
100 return html`${name}: ${used}${of}${when(
101 tokens > 0,
102 () => html` · ${tokenCount(tokens, "counted")}`,
103 )}${full}${resets}`;
87104 }
88105
89106 const resting = (s: Subscription, now: Date): boolean =>
@@ -170,7 +187,9 @@ export function subscriptionsPage(ctx: Ctx): Response {
170187 )}${when(
171188 st !== undefined,
172189 () =>
173 html`<br />${st!.windows.map(windowText).join(" · ")}`,
190 html`<br />${st!.windows.map((w, i) =>
191 i === 0 ? windowText(w) : html` · ${windowText(w)}`,
192 )}`,
174193 )}`,
175194 trail: html`${when(
176195 st !== undefined,
@@ -298,7 +317,9 @@ export function subscriptionsPage(ctx: Ctx): Response {
298317 Windows in US dollars of list price. A figure marked ~ is inferred
299318 from what the crew's runs spent; the rest the provider said. The
300319 cap is learned the first time the provider calls a window full,
301 and the share kept for people is never spent by the crew.
320 and the share kept for people is never spent by the crew. Tokens
321 are what the crew's runs read and wrote in the window, counted by
322 Claude's tokenizer.
302323 </p>`,
303324 )}
304325 <div class="mt" id="spend" data-live="run.*,usage.*">
src/web/studio/change.ts +2
@@ -31,6 +31,7 @@ import {
3131 import { previewCard } from "../pages/apps.ts";
3232 import { lookCard } from "../pages/looks.ts";
3333 import { pullState } from "../pages/pulls.ts";
34 import { workSpent } from "../pages/runs.ts";
3435 import { composer, conversation, originGlyph } from "../pages/thread.ts";
3536 import {
3637 decisionActions,
@@ -165,6 +166,7 @@ export async function studioPull(
165166 >${icon("external")}view on origin</a
166167 >`,
167168 )}
169 ${workSpent(ctx, repo, pull.id)}
168170 </div>
169171 </div>`;
170172
src/web/studio/run.ts +7 −1
@@ -27,7 +27,12 @@ import { html, when, type Html } from "../html.ts";
2727 import { page } from "../layout.ts";
2828 import { btn, icon } from "../ui.ts";
2929 import { previewCard } from "../pages/apps.ts";
30 import { costText, durationText, runTitle } from "../pages/runs.ts";
30 import {
31 costText,
32 durationText,
33 runTitle,
34 runTokens,
35 } from "../pages/runs.ts";
3136 import { sandboxLine } from "./sandbox.ts";
3237 import { logBox, runVoice, RUN_TRANSCRIPT_CAP } from "./log.ts";
3338 import { changeHref, tellHref } from "./nouns.ts";
@@ -136,6 +141,7 @@ export async function studioRun(
136141 ${run.costUsd > 0 ? null : html`hidden`}
137142 >${costText(run.costUsd)} spent</span
138143 >
144 ${runTokens(db, run)}
139145 <span
140146 class="num"
141147 id="run-turns"
src/web/studio/tell.ts +7 −1
@@ -35,7 +35,12 @@ import {
3535 userLink,
3636 } from "../ui.ts";
3737 import { previewCard } from "../pages/apps.ts";
38 import { costText, durationText, reasonWords } from "../pages/runs.ts";
38 import {
39 costText,
40 durationText,
41 reasonWords,
42 workSpent,
43 } from "../pages/runs.ts";
3944 import { composer, conversation, originGlyph } from "../pages/thread.ts";
4045 import { changeHref, tellHref } from "./nouns.ts";
4146 import {
@@ -267,6 +272,7 @@ export async function studioTell(
267272 >${icon("external")}view on origin</a
268273 >`,
269274 )}
275 ${workSpent(ctx, repo, issue.id)}
270276 ${issue.assigneeName !== null
271277 ? html`<span
272278 >${when(issue.assigneeIsAgent, () => icon("bot"))}
src/web/style/content.ts +2
@@ -213,6 +213,8 @@ a.origin-chip:hover { border-color: var(--border-strong); color: var(--fg); }
213213 .tree-row { padding: 8px 14px; min-height: 36px; }
214214 .tree-row .ico { color: var(--faint); }
215215 .tree-row[data-kind="tree"] .ico { color: var(--accent); }
216 .tree-total { margin: 6px 2px 0; text-align: right; }
217 .tok { white-space: nowrap; }
216218
217219 /* The file's location stays on screen while its contents scroll past. Opaque
218220 on purpose: code sliding under a blur is noise, not depth. */
src/web/ui.ts +30
@@ -13,6 +13,13 @@
1313 import type { EnvState, EnvStatus } from "../apps/runtime.ts";
1414 import type { PreviewData } from "../apps/data.ts";
1515 import type { RunKind, RunState } from "../core/runs.ts";
16 import {
17 estimateTokens,
18 formatTokens,
19 textKind,
20 tokenNote,
21 type TokenBasis,
22 } from "../kernel/tokens-format.ts";
1623 import { icon, type IconName } from "./icons.ts";
1724 import { Html, html, raw, when } from "./html.ts";
1825
@@ -114,6 +121,29 @@ export function fileSize(bytes: number): string {
114121 return `${(bytes / (1024 * 1024)).toFixed(1)} MB`;
115122 }
116123
124 /**
125 * A token count as a size: `~1.2k tok`, with the tokenizer and whether it is
126 * an estimate in the tooltip (`kernel/tokens-format.ts`). Only ever placed
127 * beside a byte size, a cost or a percentage — never as a badge of its own.
128 */
129 export function tokenCount(
130 n: number,
131 basis: TokenBasis,
132 kind?: "prose" | "code" | "data",
133 ): Html {
134 return html`<span class="tok" title="${tokenNote(n, basis, kind)}"
135 >${formatTokens(n, basis)}</span
136 >`;
137 }
138
139 /** A file's byte size and, when it is text, its estimated tokens beside it. */
140 export function sizeWithTokens(bytes: number, path: string): Html {
141 const tokens = estimateTokens(bytes, path);
142 return tokens === undefined
143 ? html`${fileSize(bytes)}`
144 : html`${fileSize(bytes)} · ${tokenCount(tokens, "estimated", textKind(path))}`;
145 }
146
117147 // ---- buttons ----
118148
119149 export interface BtnOptions {
test/tokens.test.ts added +262
@@ -0,0 +1,262 @@
1 /**
2 * Tokens as a unit: one formatter, checked against a hand calculation, and
3 * the quiet places a count appears — beside a file's bytes, a directory's
4 * total, a run's cost, an issue's spend and a subscription's window — and
5 * the places it must not, for a reader who may not see spend.
6 */
7
8 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
9 import { mkdtemp, rm } from "node:fs/promises";
10 import { tmpdir } from "node:os";
11 import { join } from "node:path";
12 import { openIssue } from "../src/actions/issues.ts";
13 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
14 import { createApp, type App } from "../src/app.ts";
15 import type { Repo } from "../src/core/repos.ts";
16 import { createRun } from "../src/core/runs.ts";
17 import { addSubscription } from "../src/core/subscriptions.ts";
18 import { issueUsage, recordUsage } from "../src/core/usage.ts";
19 import { createUser, type User } from "../src/core/users.ts";
20 import { serve, type Servers } from "../src/http/server.ts";
21 import {
22 estimateTokens,
23 formatTokens,
24 textKind,
25 tokenNote,
26 } from "../src/kernel/tokens-format.ts";
27
28 describe("the formatter", () => {
29 test("writes counts the way storage is written", () => {
30 const cases: [number, string][] = [
31 [0, "0 tok"],
32 [850, "850 tok"],
33 [999, "999 tok"],
34 [1000, "1.0k tok"],
35 [1234, "1.2k tok"],
36 [9949, "9.9k tok"],
37 [9950, "10k tok"], // 9.95k rounds to the whole unit, not "10.0k"
38 [48_000, "48k tok"],
39 [999_499, "999k tok"],
40 [999_500, "1.0M tok"], // never "1000k"
41 [1_100_000, "1.1M tok"],
42 [12_345_678, "12M tok"],
43 [2_500_000_000, "2.5B tok"],
44 ];
45 for (const [n, text] of cases) expect(formatTokens(n)).toBe(text);
46 });
47
48 test("an estimate carries a tilde; nonsense is zero, never NaN", () => {
49 expect(formatTokens(1234, "estimated")).toBe("~1.2k tok");
50 expect(formatTokens(Number.NaN)).toBe("0 tok");
51 expect(formatTokens(-5)).toBe("0 tok");
52 });
53
54 test("the tooltip names the tokenizer and says when it is an estimate", () => {
55 expect(tokenNote(46_500, "counted")).toBe(
56 "46,500 tokens, as counted by Claude's tokenizer",
57 );
58 const est = tokenNote(200, "estimated", "code");
59 expect(est).toContain("About 200 tokens");
60 expect(est).toContain("an estimate at 3.5 bytes per token for code");
61 expect(tokenNote(1, "counted")).toStartWith("1 token,");
62 });
63 });
64
65 describe("the estimate", () => {
66 test("is bytes ÷ the kind's ratio, rounded up", () => {
67 // Prose 4.0, code 3.5, data 3.0 bytes per token.
68 expect(estimateTokens(4000, "README.md")).toBe(1000);
69 expect(estimateTokens(700, "src/a.ts")).toBe(200);
70 expect(estimateTokens(701, "src/a.ts")).toBe(201); // 200.29 → 201
71 expect(estimateTokens(3000, "data.json")).toBe(1000);
72 expect(estimateTokens(1, "Makefile")).toBe(1); // no extension: code
73 expect(estimateTokens(0, "empty.txt")).toBe(0);
74 });
75
76 test("a binary file or a size that is not a byte count has none", () => {
77 expect(estimateTokens(5000, "logo.PNG")).toBeUndefined();
78 expect(estimateTokens(5000, "dist/app.wasm")).toBeUndefined();
79 expect(estimateTokens(-1, "src")).toBeUndefined();
80 expect(estimateTokens(1.5, "a.ts")).toBeUndefined();
81 expect(textKind(".bashrc")).toBe("code"); // a leading dot is a name
82 expect(textKind("notes.MD")).toBe("prose");
83 });
84 });
85
86 describe("token counts on the pages", () => {
87 let dataDir: string;
88 let app: App;
89 let servers: Servers;
90 let alice: User;
91 let repo: Repo;
92 let cookie = "";
93 let bobCookie = "";
94 let runId = "";
95 let issueNumber = 0;
96
97 const signIn = async (username: string) =>
98 (
99 (
100 await fetch(`${servers.url}/signin`, {
101 method: "POST",
102 body: new URLSearchParams({ username, password: "a-good-passphrase" }),
103 redirect: "manual",
104 })
105 ).headers.get("set-cookie") ?? ""
106 ).split(";")[0]!;
107 const get = async (path: string, c: string) =>
108 (await fetch(`${servers.url}${path}`, { headers: { cookie: c } })).text();
109
110 beforeAll(async () => {
111 dataDir = await mkdtemp(join(tmpdir(), "forge-tokens-"));
112 app = createApp({ dataDir, port: 0 });
113 servers = await serve(app);
114 for (const [username, adminIfFirst] of [
115 ["alice", true],
116 ["bob", false],
117 ] as const) {
118 const u = await createUser(app.db, {
119 username,
120 email: `${username}@forge.local`,
121 password: "a-good-passphrase",
122 adminIfFirst,
123 });
124 if (!u.ok) throw new Error(u.error);
125 if (username === "alice") alice = u.value;
126 }
127 const made = await createRepo(app, alice, { name: "one", visibility: "public" });
128 if (!made.ok) throw new Error("repo");
129 repo = made.value;
130 const committed = await commitToBranch(app, alice, repo, {
131 files: [
132 { path: "README.md", content: "a".repeat(4000) },
133 { path: "data.json", content: "1".repeat(3000) },
134 { path: "logo.png", content: new Uint8Array(8) },
135 { path: "src/a.ts", content: "x".repeat(700) },
136 ],
137 });
138 if (!committed.ok) throw new Error(`commit: ${String(committed.error)}`);
139
140 const issue = openIssue(app, alice, repo, { title: "make it so" });
141 if (!issue.ok) throw new Error("issue");
142 issueNumber = issue.value.number;
143 const line = {
144 repoId: repo.id,
145 issueId: issue.value.id,
146 agentId: null,
147 subscriptionId: null,
148 wallMs: 60_000,
149 outcome: "ok" as const,
150 at: new Date().toISOString(),
151 };
152 runId = createRun(app.db, {
153 repoId: repo.id,
154 kind: "agent",
155 issueId: issue.value.id,
156 }).id;
157 recordUsage(app.db, {
158 ...line,
159 runId,
160 tokensIn: 46_500,
161 tokensOut: 800,
162 costUsd: 0.42,
163 });
164 const second = createRun(app.db, {
165 repoId: repo.id,
166 kind: "agent",
167 issueId: issue.value.id,
168 }).id;
169 recordUsage(app.db, {
170 ...line,
171 runId: second,
172 tokensIn: 3500,
173 tokensOut: 200,
174 costUsd: 0.08,
175 });
176
177 cookie = await signIn("alice");
178 bobCookie = await signIn("bob");
179 });
180 afterAll(async () => {
181 servers.stop();
182 app.close();
183 await rm(dataDir, { recursive: true, force: true });
184 });
185
186 test("a directory lists each file's tokens beside its bytes, and a total", async () => {
187 const page = await get("/alice/one/tree/main", cookie);
188 // README.md: 4000 B = 3.9 KB; 4000 ÷ 4.0 = 1000 tokens.
189 expect(page).toContain("3.9 KB · <span");
190 expect(page).toContain(">~1.0k tok</span");
191 // The binary file has bytes and no tokens.
192 expect(page).toContain(">8 B</span");
193 // Files here: 4000 + 3000 + 8 = 7008 B = 6.8 KB; 1000 + 1000 + 0 tokens.
194 expect(page).toMatch(/3 files here ·\s+6\.8 KB ·\s+<span[^>]*>~2\.0k tok</);
195 expect(page).toContain("subdirectories are not counted");
196 });
197
198 test("a file shows its tokens beside its size, with the estimate said", async () => {
199 const page = await get("/alice/one/blob/main/src/a.ts", cookie);
200 // 700 B; 700 ÷ 3.5 = 200 tokens.
201 expect(page).toMatch(
202 /1 lines · 700 B · <span class="tok" title="About 200 tokens — an estimate at 3\.5 bytes per token for code/,
203 );
204 expect(page).toContain(">~200 tok</span");
205 });
206
207 test("the editor shows the file's size and tokens as opened", async () => {
208 const page = await get("/alice/one/edit/main/data.json", cookie);
209 expect(page).toContain('id="editor-size"');
210 // 3000 B = 2.9 KB; 3000 ÷ 3.0 = 1000 tokens.
211 expect(page).toMatch(/2\.9 KB · <span class="tok"[^>]*>~1\.0k tok</);
212 });
213
214 test("a run shows tokens in and out beside its cost, to the owner only", async () => {
215 const page = await get(`/alice/one/runs/${runId}`, cookie);
216 // 46,500 in → 47k; 800 out. Counted, so no tilde.
217 expect(page).toMatch(
218 /id="run-tokens"\s*><span class="tok" title="46,500 tokens, as counted by Claude&#39;s tokenizer"\s*>47k tok<\/span\s*>\s*in ·/,
219 );
220 expect(page).toContain(">800 tok</span");
221 const read = await get(`/alice/one/runs/${runId}`, bobCookie);
222 expect(read).not.toContain('id="run-tokens"');
223 });
224
225 test("an issue shows what the work on it consumed, summed by hand", async () => {
226 // 46,500 + 3,500 = 50,000 in; 800 + 200 = 1,000 out; $0.42 + $0.08.
227 expect(issueUsage(app.db, repo.id)).toBeUndefined();
228 const page = await get(`/alice/one/issues/${issueNumber}`, cookie);
229 expect(page).toMatch(/id="work-spent"\s*>2 runs ·\s+\$0\.50 ·/);
230 expect(page).toContain(">50k tok</span");
231 expect(page).toContain(">1.0k tok</span");
232 const read = await get(`/alice/one/issues/${issueNumber}`, bobCookie);
233 expect(read).not.toContain('id="work-spent"');
234 });
235
236 test("a subscription's windows carry tokens beside the dollars", async () => {
237 const added = addSubscription(app.db, dataDir, {
238 name: "account one",
239 token: "sk-ant-oat01-tokens-test-account",
240 });
241 if (!added.ok) throw new Error(added.error);
242 recordUsage(app.db, {
243 runId: createRun(app.db, { repoId: repo.id, kind: "agent" }).id,
244 repoId: repo.id,
245 issueId: null,
246 agentId: null,
247 subscriptionId: added.value.id,
248 tokensIn: 1_200_000,
249 tokensOut: 34_000,
250 costUsd: 2,
251 wallMs: 1000,
252 outcome: "ok",
253 at: new Date(Date.now() - 3_600_000).toISOString(),
254 });
255 // 1,200,000 + 34,000 = 1,234,000 → 1.2M, in both windows.
256 const settings = await get("/settings/subscriptions", cookie);
257 expect(settings.match(/>1\.2M tok</g)?.length).toBe(2);
258 expect(settings).toContain("counted by\n Claude's tokenizer");
259 const board = await get("/board", cookie);
260 expect(board.match(/>1\.2M tok</g)?.length).toBe(2);
261 });
262 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs