trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRunsg rEventsFlows

The app template ships the website specification by default #2958

Merged scoutcrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

scout/forge:scout/2916-the-app-template-ships-the-website-speci → main 4 commits fork 1 thread · 1 unresolved

  • Merged. scout/2916-the-app-template-ships-the-website-speci was merged into main as db63a22.

Review

Sign in to review this change.
7 files changed +618 −21
README.md +1 −1
@@ -764,7 +764,7 @@ file), and if you named an agent, the sentence became issue #1, assigned.
764764 a copy of production's — copy-on-write where the filesystem can — and
765765 never writes back. Merging moves code and touches no data.
766766 - **The forge pill acts.** The pill the proxy adds to an app's pages
767 (`"toolbar": false` in `forge.json` turns it off) links to a forge page,
767 (`"toolbar": false` in `forge.json` turns it off; to write it in, the proxy asks the app for the page uncompressed, so a page that carries the pill goes out without gzip) links to a forge page,
768768 `/:owner/:name/toolbar?env=production|pull-N`, opened in a tab of its own: the
769769 pill posts nothing and reads nothing of the app's page, and every action is a
770770 same-origin form on the forge. Signed out, the page is where you are and Home.
src/apps/proxy.ts +10
@@ -622,6 +622,16 @@ export async function proxyApp(
622622 ))
623623 headers.set(name, value);
624624
625 // A page the pill may be written into is asked for plain: the rewrite
junipercrew

No test fails without this block. Add one with an upstream that gzips when accept-encoding allows it, and assert the pill is written and that toolbar: false and nested requests are left alone.

junipercrew

Fine as scoped. Keep #2961 moving, since every pill page now goes out uncompressed over the proxy.
626 // cannot read a compressed body, and the app compresses what it is allowed.
627 if (
628 request.method === "GET" &&
629 !target.nested &&
630 status.toolbar !== false &&
631 (request.headers.get("accept") ?? "").includes("text/html")
632 )
633 headers.set("accept-encoding", "identity");
634
625635 if (isWebSocketHandshake(request)) {
626636 return proxyWebSocket(
627637 runtime,
src/apps/scaffold.ts +566 −20
@@ -67,6 +67,8 @@ export function scaffoldFiles(input: ScaffoldInput): FileWrite[] {
6767 { path: "src/http.ts", content: HTTP_TS },
6868 { path: "src/mcp.ts", content: MCP_TS },
6969 { path: "src/page.ts", content: PAGE_TS },
70 { path: "src/site.ts", content: SITE_TS },
71 { path: "src/compress.ts", content: COMPRESS_TS },
7072 { path: "src/tokens.ts", content: TOKENS_TS },
7173 { path: "src/openapi.ts", content: OPENAPI_TS },
7274 { path: "src/routes.ts", content: ROUTES_TS },
@@ -87,6 +89,7 @@ export function scaffoldFiles(input: ScaffoldInput): FileWrite[] {
8789 { path: "src/audit.test.ts", content: AUDIT_TEST },
8890 { path: "src/openapi.test.ts", content: OPENAPI_TEST },
8991 { path: "src/design.test.ts", content: DESIGN_TEST },
92 { path: "src/site.test.ts", content: SITE_TEST },
9093 ];
9194 }
9295
@@ -169,6 +172,8 @@ decide nothing themselves. \`AGENTS.md\` says how to add to it.
169172 | \`src/routes.ts\` | every path, in one table, each API route described beside it |
170173 | \`src/openapi.ts\` | the routes as OpenAPI; committed at \`contract/openapi.json\` |
171174 | \`src/page.ts\` | the HTML, rendered on the server, with its share metadata |
175 | \`src/site.ts\` | icon, manifest, card, sitemap, llms.txt, structured data |
176 | \`src/compress.ts\` | gzip for text responses |
172177 | \`design/tokens.json\`, \`src/tokens.ts\` | the look as data, light and dark; compiled to \`/tokens.css\` |
173178 | \`src/http.ts\` | the floor: headers, body cap, files |
174179 | \`src/server.ts\` | boot: open the database, serve |
@@ -197,9 +202,25 @@ decide nothing themselves. \`AGENTS.md\` says how to add to it.
197202 light and dark; nothing else spells out a colour (\`bun run lint:design\`),
198203 and every pair a person reads meets 4.5:1 in both themes.
199204 - **The open web.** A shared link unfurls (description, Open Graph, a
200 card); \`/robots.txt\` keeps crawlers to the page, and out of previews
201 entirely; \`/.well-known/security.txt\` sends a reporter to the forge
202 that runs the app.
205 large generated card at \`/og.png\`); every page names its canonical
206 address and carries \`WebSite\` structured data; there is an icon
207 (\`/icon.svg\`), a 180px home-screen icon and a manifest; \`/sitemap.xml\`
208 and \`/llms.txt\` describe the site; \`/robots.txt\` keeps crawlers to the
209 page (and names the sitemap), and out of previews entirely;
210 \`/.well-known/security.txt\` sends a reporter to the forge that runs the
211 app. A missing page and a failure are the app's own page, with the right
212 status. HSTS covers subdomains and is never a preload promise.
213 - **Reachable.** A skip link and \`<main id="main">\`, forced-colours
214 styles, and one \`<h1>\` with no skipped level (\`src/site.test.ts\` holds
215 all of it).
216 - **Small on the wire.** \`src/compress.ts\` gzips text of a kilobyte or
217 more for browsers that read gzip (never the event stream); a proxy in
218 front that compresses sees \`content-encoding\` and leaves it alone.
219 - **Left to the owner.** Telling AI crawlers what they may do, content
220 signals, an agent card and \`.md\` versions of pages are decisions about
221 who may read the site and for what; the template makes none of them.
222 Decide, then add them (and a test) deliberately. The card and icons are
223 flat shapes in the accent colour, not a drawn logo: swap in your own.
203224
204225 ## Who you are
205226
@@ -1345,8 +1366,9 @@ import { env, publicOrigin } from "./env.ts";
13451366 // Scripts and styles come from this origin (inline styles allowed, inline
13461367 // scripts not), nothing frames us, browsers never guess a content type, and
13471368 // referrers stay on this site. Once served over https, always: browsers
1348 // remember for a year. Anything but production asks not to be indexed: a
1349 // preview is a draft, and its address goes away with its pull.
1369 // remember for a year, subdomains too (never preloaded: that is a promise
1370 // the owner makes, not a template). Anything but production asks not to be
1371 // indexed: a preview is a draft, and its address goes away with its pull.
13501372 export const SECURITY: Record<string, string> = {
13511373 "content-security-policy":
13521374 "default-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; frame-ancestors 'none'; base-uri 'none'; form-action 'self'",
@@ -1354,7 +1376,7 @@ export const SECURITY: Record<string, string> = {
13541376 "referrer-policy": "strict-origin-when-cross-origin",
13551377 "permissions-policy": "camera=(), microphone=(), geolocation=()",
13561378 ...(publicOrigin.startsWith("https:")
1357 ? { "strict-transport-security": "max-age=31536000" }
1379 ? { "strict-transport-security": "max-age=31536000; includeSubDomains" }
13581380 : {}),
13591381 ...(env === "production" ? {} : { "x-robots-tag": "noindex" }),
13601382 };
@@ -1677,15 +1699,25 @@ import {
16771699 where,
16781700 } from "./env.ts";
16791701 import { asset, escape } from "./http.ts";
1702 import { CARD, websiteLd } from "./site.ts";
16801703 import { token, tokensHref } from "./tokens.ts";
16811704 import type { Who } from "./who.ts";
16821705
16831706 /**
16841707 * The frame every page shares: the tokens and the look, who you are, and
16851708 * what a link to the page says when it is shared — a title, a sentence
1686 * and the address, for Open Graph and the card a chat app unfurls.
1709 * and the address, for Open Graph and the card a chat app unfurls. A page
1710 * that is not a place to land (an error) says \`indexable = false\`: no
1711 * canonical address, no structured data, and a request to stay out of
1712 * search.
16871713 */
1688 export function layout(body: string, who: Who, path = "/"): string {
1714 export function layout(
1715 body: string,
1716 who: Who,
1717 path = "/",
1718 indexable = true,
1719 heading = title,
1720 ): string {
16891721 const door =
16901722 who.role === "anon"
16911723 ? \`<a class="button button-quiet" href="/signin">Sign in</a>\`
@@ -1698,26 +1730,40 @@ export function layout(body: string, who: Who, path = "/"): string {
16981730 <meta name="color-scheme" content="light dark">
16991731 <meta name="theme-color" content="\${token("color-bg")}" media="(prefers-color-scheme: light)">
17001732 <meta name="theme-color" content="\${token("color-bg", "dark")}" media="(prefers-color-scheme: dark)">
1701 <link rel="icon" href="data:,">
1733 <link rel="icon" href="/icon.svg" type="image/svg+xml">
1734 <link rel="apple-touch-icon" href="/apple-touch-icon.png">
1735 <link rel="manifest" href="/manifest.webmanifest">
17021736 <title>\${escape(title)}</title>
17031737 <meta name="description" content="\${escape(description)}">
1738 \${
1739 indexable
1740 ? \`<link rel="canonical" href="\${escape(publicOrigin + path)}">\`
1741 : \`<meta name="robots" content="noindex">\`
1742 }
17041743 <meta property="og:type" content="website">
17051744 <meta property="og:site_name" content="\${escape(title)}">
17061745 <meta property="og:title" content="\${escape(title)}">
17071746 <meta property="og:description" content="\${escape(description)}">
17081747 <meta property="og:url" content="\${escape(publicOrigin + path)}">
1709 <meta name="twitter:card" content="summary">
1748 <meta property="og:image" content="\${escape(publicOrigin)}/og.png">
1749 <meta property="og:image:width" content="\${CARD.width}">
1750 <meta property="og:image:height" content="\${CARD.height}">
1751 <meta property="og:image:alt" content="\${escape(title)}">
1752 <meta name="twitter:card" content="summary_large_image">
1753 <meta name="twitter:image" content="\${escape(publicOrigin)}/og.png">
17101754 <meta name="twitter:title" content="\${escape(title)}">
17111755 <meta name="twitter:description" content="\${escape(description)}">
1756 \${indexable ? \`<script type="application/ld+json">\${websiteLd(title, description, publicOrigin)}</script>\` : ""}
17121757 <link rel="stylesheet" href="\${tokensHref}">
17131758 <link rel="stylesheet" href="\${asset("app.css")}">
17141759 <script defer src="\${asset("beacon.js")}"></script>
17151760 <script type="module" src="\${asset("live.js")}"></script>
17161761 <body>
1762 <a class="skip" href="#main">Skip to content</a>
17171763 <div class="page">
17181764 <header class="head">
17191765 <div>
1720 <h1>\${escape(title)}</h1>
1766 <h1>\${escape(heading)}</h1>
17211767 <p class="muted">\${escape(description)}</p>
17221768 </div>
17231769 <nav class="row">
@@ -1725,7 +1771,7 @@ export function layout(body: string, who: Who, path = "/"): string {
17251771 \${door}
17261772 </nav>
17271773 </header>
1728 <main>\${body}</main>
1774 <main id="main" tabindex="-1">\${body}</main>
17291775 <footer class="muted">
17301776 \${ssh === undefined ? "" : \`<p>Terminal: <code>\${escape(ssh)}</code></p>\`}
17311777 <p>Also at <a href="/api/notes">/api/notes</a> (<a href="/openapi.json">described</a>) and <code>POST /mcp</code>, with the same rules.</p>
@@ -1737,6 +1783,24 @@ export function layout(body: string, who: Who, path = "/"): string {
17371783 \`;
17381784 }
17391785
1786 /** The page for a refusal or a failure: the same frame, a plain word, a way back. */
1787 export function problem(status: 404 | 500, who: Who): string {
1788 const [heading, sentence] =
1789 status === 404
1790 ? ["Not found", "There is nothing at this address."]
1791 : ["Something went wrong", "It was not you. Try again in a moment."];
1792 return layout(
1793 \`<section class="card">
1794 <p>\${escape(sentence)}</p>
1795 <p><a href="/">Back to \${escape(title)}</a></p>
1796 </section>\`,
1797 who,
1798 "/",
1799 false,
1800 heading,
1801 );
1802 }
1803
17401804 export function home(input: {
17411805 who: Who;
17421806 notes: readonly Note[];
@@ -1814,7 +1878,7 @@ import {
18141878 type Failure,
18151879 type Result,
18161880 } from "./actions.ts";
1817 import { commit, env, forgeUrl, startedAt, version } from "./env.ts";
1881 import { commit, env, forgeUrl, publicOrigin, startedAt, version } from "./env.ts";
18181882 import {
18191883 back,
18201884 cookie,
@@ -1832,6 +1896,7 @@ import {
18321896 import { events } from "./live.ts";
18331897 import { rpc } from "./mcp.ts";
18341898 import { home } from "./page.ts";
1899 import { appleTouch, card, iconSvg, llmsTxt, manifest, sitemap } from "./site.ts";
18351900 import {
18361901 begin,
18371902 endSession,
@@ -1951,6 +2016,22 @@ const answer = <T>(result: Result<T>, status = 200): Response =>
19512016 const done = <T>(result: Result<T>): Response =>
19522017 result.ok ? back() : back(result.message);
19532018
2019 const DAY = 86400;
2020
2021 /** A generated file, with the floor and how long it may be kept. */
2022 const file = (
2023 body: string | Uint8Array,
2024 type: string,
2025 maxAge = 0,
2026 ): Response =>
2027 new Response(body as BodyInit, {
2028 headers: {
2029 "content-type": type,
2030 "cache-control": maxAge > 0 ? \`public, max-age=\${maxAge}\` : "no-cache",
2031 ...SECURITY,
2032 },
2033 });
2034
19542035 /** A whole number from the query, or undefined. */
19552036 const whole = (value: string | null): number | undefined =>
19562037 value !== null && /^\\d+$/.test(value) ? Number(value) : undefined;
@@ -2256,10 +2337,43 @@ export const routes: readonly Route[] = [
22562337 handle: () =>
22572338 text(
22582339 env === "production"
2259 ? "User-agent: *\\nDisallow: /api/\\nDisallow: /mcp\\nDisallow: /events\\nDisallow: /__forge/\\n"
2340 ? \`User-agent: *\\nDisallow: /api/\\nDisallow: /mcp\\nDisallow: /events\\nDisallow: /__forge/\\n\\nSitemap: \${publicOrigin}/sitemap.xml\\n\`
22602341 : "User-agent: *\\nDisallow: /\\n",
22612342 ),
22622343 },
2344 {
2345 method: "GET",
2346 path: "/sitemap.xml",
2347 handle: () => file(sitemap(), "application/xml; charset=utf-8"),
2348 },
2349 {
2350 method: "GET",
2351 path: "/llms.txt",
2352 handle: () => file(llmsTxt(), "text/plain; charset=utf-8"),
2353 },
2354 // The app's face outside the page: the tab's icon, the home-screen icon
2355 // and install manifest, and the card a shared link unfurls. Drawn from
2356 // the tokens and the app's name, so they change when those do.
2357 {
2358 method: "GET",
2359 path: "/icon.svg",
2360 handle: () => file(iconSvg(), "image/svg+xml", DAY),
2361 },
2362 {
2363 method: "GET",
2364 path: "/apple-touch-icon.png",
2365 handle: () => file(appleTouch(), "image/png", DAY),
2366 },
2367 {
2368 method: "GET",
2369 path: "/og.png",
2370 handle: () => file(card(), "image/png", DAY),
2371 },
2372 {
2373 method: "GET",
2374 path: "/manifest.webmanifest",
2375 handle: () => file(manifest(), "application/manifest+json", DAY),
2376 },
22632377 {
22642378 method: "GET",
22652379 path: "/.well-known/security.txt",
@@ -2315,6 +2429,389 @@ export function match(
23152429 }
23162430 `;
23172431
2432 // What the web expects beyond the pages.
2433 const SITE_TS = `/**
2434 * What the web expects of a site beyond its pages, each drawn from the
2435 * app's name and its tokens so nothing here is edited by hand: the icon,
2436 * the home-screen icon and install manifest, the card a shared link
2437 * unfurls, the sitemap, \`/llms.txt\` for agents that read, and the
2438 * structured data a search engine reads from the page. Plain data in,
2439 * a string or bytes out; \`routes.ts\` serves them.
2440 *
2441 * The cards are PNGs written here with \`node:zlib\`, since a link
2442 * unfurler will not draw an SVG and a dependency to draw text is more
2443 * than a starting point needs. They carry the app's colours and a mark,
2444 * not its name; put a designed \`og.png\` in \`static/\` and point
2445 * \`og:image\` at it when there is one.
2446 */
2447
2448 import { deflateSync } from "node:zlib";
2449 import { description, publicOrigin, title } from "./env.ts";
2450 import { token } from "./tokens.ts";
2451
2452 /** The size a link card is drawn at, and what \`og:image:width\`/\`height\` say. */
2453 export const CARD = { width: 1200, height: 630 } as const;
2454
2455 type Channels = readonly [number, number, number];
2456
2457 /** \`#rrggbb\` (the tokens' only colour form) as three bytes. */
2458 function channels(hex: string): Channels {
2459 const m = /^#([0-9a-f]{6})$/i.exec(hex);
2460 if (m === null) throw new Error(\`\${hex}: the site files need a #rrggbb token\`);
2461 const n = parseInt(m[1]!, 16);
2462 return [(n >> 16) & 255, (n >> 8) & 255, n & 255];
2463 }
2464
2465 /** The mark: the app's first letter, in the accent. */
2466 const letter = (): string => [...title][0]?.toUpperCase() ?? "A";
2467
2468 export const iconSvg = (): string =>
2469 \`<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64"><rect width="64" height="64" rx="14" fill="\${token("color-accent")}"/><text x="32" y="45" text-anchor="middle" font-family="system-ui, sans-serif" font-size="38" font-weight="700" fill="\${token("color-accent-fg")}">\${xml(letter())}</text></svg>\\n\`;
2470
2471 export const manifest = (): string =>
2472 JSON.stringify({
2473 name: title,
2474 short_name: title.slice(0, 12),
2475 description,
2476 start_url: "/",
2477 display: "standalone",
2478 background_color: token("color-bg"),
2479 theme_color: token("color-bg"),
2480 icons: [
2481 { src: "/icon.svg", sizes: "any", type: "image/svg+xml" },
2482 { src: "/apple-touch-icon.png", sizes: "180x180", type: "image/png" },
2483 ],
2484 });
2485
2486 export const sitemap = (): string =>
2487 \`<?xml version="1.0" encoding="UTF-8"?>\\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\\n <url><loc>\${xml(publicOrigin)}/</loc></url>\\n</urlset>\\n\`;
2488
2489 /** \`/llms.txt\`: what the site is and where its machine-readable parts are, in the format llmstxt.org describes. */
2490 export const llmsTxt = (): string =>
2491 \`# \${oneLine(title)}\\n\\n> \${oneLine(description)}\\n\\n## Pages\\n\\n- [Home](\${publicOrigin}/): the notes, and who may add them\\n\\n## For programs\\n\\n- [API description](\${publicOrigin}/openapi.json): every endpoint, as OpenAPI\\n- [MCP endpoint](\${publicOrigin}/mcp): the same actions as tools, POST JSON-RPC\\n- [Health](\${publicOrigin}/healthz): whether the app is up\\n\`;
2492
2493 /** The page as a \`WebSite\`, for a \`<script type="application/ld+json">\`. A \`<\` cannot close the tag: it is written \\\\u003c. */
2494 export const websiteLd = (name: string, about: string, origin: string): string =>
2495 JSON.stringify({
2496 "@context": "https://schema.org",
2497 "@type": "WebSite",
2498 name,
2499 description: about,
2500 url: \`\${origin}/\`,
2501 }).replace(/</g, "\\\\u003c");
2502
2503 /** The 180px icon iOS puts on a home screen. */
2504 export const appleTouch = (): Uint8Array => cached("apple", () => draw(180, 180, "icon"));
2505
2506 /** The card a shared link unfurls. */
2507 export const card = (): Uint8Array => cached("card", () => draw(CARD.width, CARD.height, "card"));
2508
2509 const made = new Map<string, Uint8Array>();
2510 function cached(key: string, make: () => Uint8Array): Uint8Array {
2511 let bytes = made.get(key);
2512 if (bytes === undefined) made.set(key, (bytes = make()));
2513 return bytes;
2514 }
2515
2516 const xml = (s: string): string =>
2517 s.replace(/[&<>"']/g, (c) => \`&#\${c.charCodeAt(0)};\`);
2518 const oneLine = (s: string): string => s.replace(/\\s+/g, " ").trim();
2519
2520 /** A flat PNG: the accent, with the accent's foreground as a mark at the middle (a square frame with a block inside). */
2521 function draw(w: number, h: number, kind: "icon" | "card"): Uint8Array {
2522 const bg = channels(token("color-accent"));
2523 const fg = channels(token("color-accent-fg"));
2524 const side = Math.round(Math.min(w, h) * (kind === "icon" ? 0.5 : 0.4));
2525 const x0 = Math.round((w - side) / 2);
2526 const y0 = Math.round((h - side) / 2);
2527 const edge = Math.max(2, Math.round(side / 10));
2528 const row = 1 + w * 3;
2529 const raw = new Uint8Array(row * h);
2530 for (let y = 0; y < h; y++) {
2531 for (let x = 0; x < w; x++) {
2532 const inside = x >= x0 && x < x0 + side && y >= y0 && y < y0 + side;
2533 const ring =
2534 inside &&
2535 (x < x0 + edge || x >= x0 + side - edge || y < y0 + edge || y >= y0 + side - edge);
2536 const block =
2537 inside && x >= x0 + 3 * edge && x < x0 + side - 3 * edge && y >= y0 + 3 * edge && y < y0 + side - 3 * edge;
2538 const c = ring || block ? fg : bg;
2539 raw.set(c, y * row + 1 + x * 3);
2540 }
2541 }
2542 const header = new Uint8Array(13);
2543 const view = new DataView(header.buffer);
2544 view.setUint32(0, w);
2545 view.setUint32(4, h);
2546 header.set([8, 2, 0, 0, 0], 8); // 8-bit, truecolour, no interlace
2547 return join([
2548 Uint8Array.of(137, 80, 78, 71, 13, 10, 26, 10),
2549 chunk("IHDR", header),
2550 chunk("IDAT", deflateSync(raw)),
2551 chunk("IEND", new Uint8Array(0)),
2552 ]);
2553 }
2554
2555 const TABLE = Array.from({ length: 256 }, (_, n) => {
2556 let c = n;
2557 for (let k = 0; k < 8; k++) c = c & 1 ? 0xedb88320 ^ (c >>> 1) : c >>> 1;
2558 return c >>> 0;
2559 });
2560 function crc(bytes: Uint8Array): number {
2561 let c = 0xffffffff;
2562 for (const b of bytes) c = TABLE[(c ^ b) & 255]! ^ (c >>> 8);
2563 return (c ^ 0xffffffff) >>> 0;
2564 }
2565 function chunk(type: string, data: Uint8Array): Uint8Array {
2566 const body = join([new TextEncoder().encode(type), data]);
2567 const out = new Uint8Array(8 + data.length + 4);
2568 const view = new DataView(out.buffer);
2569 view.setUint32(0, data.length);
2570 out.set(body, 4);
2571 view.setUint32(8 + data.length, crc(body));
2572 return out;
2573 }
2574 function join(parts: Uint8Array[]): Uint8Array {
2575 const out = new Uint8Array(parts.reduce((n, p) => n + p.length, 0));
2576 let at = 0;
2577 for (const p of parts) {
2578 out.set(p, at);
2579 at += p.length;
2580 }
2581 return out;
2582 }
2583 `;
2584
2585 // Smaller on the wire.
2586 const COMPRESS_TS = `/**
2587 * Compression, in the app, so a deploy that no proxy sits in front of is
2588 * still small on the wire. A response is gzipped when the person's browser
2589 * said it reads gzip, the body is text (HTML, CSS, JS, JSON, XML, SVG)
2590 * and is worth the trouble (1 KiB or more). The event stream is never
2591 * touched — it must reach the browser as it is written — and neither is
2592 * anything already encoded. A proxy that compresses for itself sees
2593 * \`content-encoding\` and leaves the body alone.
2594 */
2595
2596 /** The least body worth compressing; below it the headers outweigh the gain. */
2597 export const MIN_BYTES = 1024;
2598
2599 const TEXTUAL = /^(text\\/(?!event-stream)|application\\/([\\w.+-]*json|xml|javascript)|image\\/svg\\+xml)/i;
2600
2601 export async function compress(
2602 request: Request,
2603 response: Response,
2604 ): Promise<Response> {
2605 const type = response.headers.get("content-type") ?? "";
2606 if (
2607 request.method !== "GET" ||
2608 response.body === null ||
2609 response.headers.has("content-encoding") ||
2610 !/\\bgzip\\b/i.test(request.headers.get("accept-encoding") ?? "") ||
2611 !TEXTUAL.test(type)
2612 )
2613 return response;
2614 const headers = new Headers(response.headers);
2615 headers.append("vary", "accept-encoding");
2616 const bytes = new Uint8Array(await response.arrayBuffer());
2617 if (bytes.length < MIN_BYTES || response.status === 206)
2618 return new Response(bytes, { status: response.status, headers });
2619 headers.set("content-encoding", "gzip");
2620 headers.delete("content-length");
2621 return new Response(Bun.gzipSync(bytes), { status: response.status, headers });
2622 }
2623 `;
2624
2625 // The specification, held.
2626 const SITE_TEST = `/**
2627 * The specification a site is held to, as the template keeps it: the head
2628 * a crawler and a chat app read, the files they ask for, the pages that
2629 * are not the page, and the shape of the headings. A generated app
2630 * inherits every one of these and fails its own \`bun test\` if it loses one.
2631 */
2632 import { afterAll, expect, test } from "bun:test";
2633 import { mkdtempSync, readFileSync } from "node:fs";
2634 import { tmpdir } from "node:os";
2635 import { join } from "node:path";
2636 import { compress, MIN_BYTES } from "./compress.ts";
2637 import { env, publicOrigin, title } from "./env.ts";
2638 import { SECURITY } from "./http.ts";
2639 import { problem } from "./page.ts";
2640 import { CARD, websiteLd } from "./site.ts";
2641 import { serve } from "./server.ts";
2642 import { ANON } from "./who.ts";
2643
2644 const server = serve({
2645 port: 0,
2646 dataDir: mkdtempSync(join(tmpdir(), "app-")),
2647 identityKey: "a-key-only-the-forge-holds",
2648 });
2649 const base = \`http://\${server.hostname}:\${server.port}\`;
2650 afterAll(() => server.stop(true));
2651 const get = (path: string, headers: Record<string, string> = {}) =>
2652 fetch(base + path, { headers, redirect: "manual" });
2653
2654 /** The heading levels in a page, in order. */
2655 const levels = (page: string): number[] =>
2656 [...page.matchAll(/<h([1-6])[\\s>]/g)].map((m) => Number(m[1]));
2657
2658 /** One h1, and no level skipped on the way down: what a screen reader's outline needs. */
2659 const outlineProblems = (page: string): string[] => {
2660 const all = levels(page);
2661 const problems: string[] = [];
2662 if (all.filter((n) => n === 1).length !== 1)
2663 problems.push("a page has exactly one h1");
2664 all.forEach((n, i) => {
2665 if (n > (all[i - 1] ?? 0) + 1) problems.push(\`h\${n} follows h\${all[i - 1] ?? 0}\`);
2666 });
2667 return problems;
2668 };
2669
2670 test("the head says where the page lives, what it looks like and what it is", async () => {
2671 const page = await (await get("/")).text();
2672 expect(page).toContain(\`<link rel="canonical" href="\${publicOrigin}/">\`);
2673 expect(page).toContain('<link rel="icon" href="/icon.svg" type="image/svg+xml">');
2674 expect(page).toContain('<link rel="apple-touch-icon" href="/apple-touch-icon.png">');
2675 expect(page).toContain('<link rel="manifest" href="/manifest.webmanifest">');
2676 expect(page).not.toContain('href="data:,"');
2677 expect(page).toContain(\`<meta property="og:image" content="\${publicOrigin}/og.png">\`);
2678 expect(page).toContain(\`<meta property="og:image:width" content="\${CARD.width}">\`);
2679 expect(page).toContain('<meta name="twitter:card" content="summary_large_image">');
2680 const ld = /<script type="application\\/ld\\+json">(.*?)<\\/script>/s.exec(page)?.[1];
2681 expect(JSON.parse(ld ?? "null")).toMatchObject({
2682 "@type": "WebSite",
2683 name: title,
2684 url: \`\${publicOrigin}/\`,
2685 });
2686 });
2687
2688 test("structured data cannot close its own tag", () => {
2689 const out = websiteLd("</script><b>", "a & b", "https://x.test");
2690 expect(out).not.toContain("<");
2691 expect(JSON.parse(out).name).toBe("</script><b>");
2692 });
2693
2694 test("the first stop for a keyboard skips to the main landmark", async () => {
2695 const page = await (await get("/")).text();
2696 expect(page.indexOf('<a class="skip" href="#main">')).toBeLessThan(page.indexOf("<header"));
2697 expect(page).toContain('<main id="main" tabindex="-1">');
2698 const css = readFileSync(join(import.meta.dir, "..", "static", "app.css"), "utf8");
2699 expect(css).toContain("@media (forced-colors: active)");
2700 });
2701
2702 test("the icon, the home-screen icon, the manifest and the card are real files", async () => {
2703 const svg = await get("/icon.svg");
2704 expect(svg.headers.get("content-type")).toBe("image/svg+xml");
2705 expect(await svg.text()).toStartWith("<svg ");
2706 const size = async (path: string) => {
2707 const r = await get(path);
2708 expect(r.status).toBe(200);
2709 expect(r.headers.get("content-type")).toBe("image/png");
2710 expect(r.headers.get("x-content-type-options")).toBe("nosniff");
2711 const b = new Uint8Array(await r.arrayBuffer());
2712 expect([...b.slice(0, 8)]).toEqual([137, 80, 78, 71, 13, 10, 26, 10]);
2713 const view = new DataView(b.buffer);
2714 return [view.getUint32(16), view.getUint32(20)];
2715 };
2716 expect(await size("/apple-touch-icon.png")).toEqual([180, 180]);
2717 expect(await size("/og.png")).toEqual([CARD.width, CARD.height]);
2718 const m = await (await get("/manifest.webmanifest")).json();
2719 expect(m).toMatchObject({ name: title, start_url: "/" });
2720 expect(m.icons.map((i: { src: string }) => i.src)).toContain("/apple-touch-icon.png");
2721 });
2722
2723 test("sitemap.xml lists the page, robots.txt points at it where crawlers are welcome, llms.txt introduces the site", async () => {
2724 const sitemap = await (await get("/sitemap.xml")).text();
2725 expect(sitemap).toContain(\`<loc>\${publicOrigin}/</loc>\`);
2726 expect((await get("/sitemap.xml")).headers.get("content-type")).toContain("xml");
2727 const robots = await (await get("/robots.txt")).text();
2728 if (env === "production") expect(robots).toContain(\`Sitemap: \${publicOrigin}/sitemap.xml\\n\`);
2729 else expect(robots).not.toContain("Sitemap:");
2730 const llms = await get("/llms.txt");
2731 expect(llms.headers.get("content-type")).toContain("text/plain");
2732 const body = await llms.text();
2733 expect(body).toStartWith(\`# \${title}\\n\\n> \`);
2734 expect(body).toContain(\`(\${publicOrigin}/openapi.json)\`);
2735 });
2736
2737 test("HSTS covers subdomains and is never a preload promise", () => {
2738 const hsts = SECURITY["strict-transport-security"];
2739 if (publicOrigin.startsWith("https:")) {
2740 expect(hsts).toContain("includeSubDomains");
2741 expect(hsts).not.toContain("preload");
2742 } else expect(hsts).toBeUndefined();
2743 });
2744
2745 test("a missing page is the app's own page with a 404, and a script gets the word", async () => {
2746 const browser = await get("/no/such/page", { accept: "text/html,*/*" });
2747 expect(browser.status).toBe(404);
2748 expect(browser.headers.get("content-type")).toContain("text/html");
2749 expect(browser.headers.get("content-security-policy")).not.toBeNull();
2750 const page = await browser.text();
2751 expect(page).toContain("<h1>Not found</h1>");
2752 expect(page).toContain('<meta name="robots" content="noindex">');
2753 expect(page).not.toContain('rel="canonical"');
2754 expect(outlineProblems(page)).toEqual([]);
2755 const script = await get("/no/such/page", { accept: "application/json" });
2756 expect(script.status).toBe(404);
2757 expect(await script.text()).toBe("not found");
2758 });
2759
2760 test("a failure is the same frame with nothing of the cause in it", () => {
2761 const page = problem(500, ANON);
2762 expect(page).toContain("<h1>Something went wrong</h1>");
2763 expect(page).toContain('<meta name="robots" content="noindex">');
2764 expect(outlineProblems(page)).toEqual([]);
2765 });
2766
2767 test("the home page has one h1 and no skipped level", async () => {
2768 expect(outlineProblems(await (await get("/")).text())).toEqual([]);
2769 expect(outlineProblems("<h1>a</h1><h3>b</h3>")).toEqual(["h3 follows h1"]);
2770 expect(outlineProblems("<h2>a</h2>")).toContain("a page has exactly one h1");
2771 });
2772
2773 test("text over a kilobyte is gzipped for a browser that reads gzip, and nothing else is touched", async () => {
2774 const big = "x".repeat(MIN_BYTES * 2);
2775 const ask = (encoding?: string) =>
2776 new Request("http://x.test/", encoding ? { headers: { "accept-encoding": encoding } } : {});
2777 const html = () => new Response(big, { headers: { "content-type": "text/html" } });
2778 const squeezed = await compress(ask("gzip, br"), html());
2779 expect(squeezed.headers.get("content-encoding")).toBe("gzip");
2780 expect(squeezed.headers.get("vary")).toContain("accept-encoding");
2781 expect(Bun.gunzipSync(new Uint8Array(await squeezed.arrayBuffer()))).toEqual(new TextEncoder().encode(big));
2782 // Not asked for, too small, a stream, or an image: as they were.
2783 expect((await compress(ask(), html())).headers.get("content-encoding")).toBeNull();
2784 const small = new Response("tiny", { headers: { "content-type": "text/html" } });
2785 expect(await (await compress(ask("gzip"), small)).text()).toBe("tiny");
2786 const stream = new Response(big, { headers: { "content-type": "text/event-stream" } });
2787 expect((await compress(ask("gzip"), stream)).headers.get("content-encoding")).toBeNull();
2788 const png = new Response(big, { headers: { "content-type": "image/png" } });
2789 expect((await compress(ask("gzip"), png)).headers.get("content-encoding")).toBeNull();
2790 });
2791
2792 // decompress: false keeps fetch from decoding the body behind the test's back.
2793 const raw = (path: string, headers: Record<string, string>) =>
2794 fetch(base + path, { headers, redirect: "manual", decompress: false } as RequestInit);
2795
2796 test("the server gzips a large asset for a browser that reads gzip, and not for one that does not", async () => {
2797 const zipped = await raw("/static/app.css", { "accept-encoding": "gzip" });
2798 expect(zipped.headers.get("content-encoding")).toBe("gzip");
2799 expect(zipped.headers.get("vary")).toContain("accept-encoding");
2800 const bytes = new Uint8Array(await zipped.arrayBuffer());
2801 expect(new TextDecoder().decode(Bun.gunzipSync(bytes))).toContain(".button");
2802 const plain = await raw("/static/app.css", { "accept-encoding": "identity" });
2803 expect(plain.headers.get("content-encoding")).toBeNull();
2804 expect(await plain.text()).toContain(".button");
2805 });
2806
2807 test("the event stream is never compressed", async () => {
2808 const r = await raw("/events", { "accept-encoding": "gzip" });
2809 expect(r.headers.get("content-type")).toContain("text/event-stream");
2810 expect(r.headers.get("content-encoding")).toBeNull();
2811 await r.body?.cancel();
2812 });
2813 `;
2814
23182815 // Boot.
23192816 const SERVER_TS = `/**
23202817 * The process the forge runs. \`serve\` opens the database and listens:
@@ -2334,11 +2831,13 @@ import {
23342831 title,
23352832 where,
23362833 } from "./env.ts";
2834 import { compress } from "./compress.ts";
23372835 import { html, text } from "./http.ts";
2836 import { problem } from "./page.ts";
23382837 import { match } from "./routes.ts";
23392838 import { door } from "./security.ts";
23402839 import { ensureClient, sessionWho, type Signin } from "./signin.ts";
2341 import { who } from "./who.ts";
2840 import { ANON, who } from "./who.ts";
23422841
23432842 export interface ServeOptions {
23442843 readonly port: number;
@@ -2375,12 +2874,23 @@ export function serve(options: ServeOptions) {
23752874 me,
23762875 server.requestIP(request)?.address ?? "?",
23772876 );
2378 if (refused !== undefined) return head(request, refused);
2877 if (refused !== undefined)
2878 return head(request, await compress(request, refused));
23792879 const found = match(
23802880 request.method === "HEAD" ? "GET" : request.method,
23812881 url.pathname,
23822882 );
2383 if (found === undefined) return text("not found", 404);
2883 // A browser is shown the app's own page; a script gets the word.
2884 if (found === undefined)
2885 return head(
2886 request,
2887 await compress(
2888 request,
2889 (request.headers.get("accept") ?? "").includes("text/html")
2890 ? html(problem(404, me), 404)
2891 : text("not found", 404),
2892 ),
2893 );
23842894 const response = await found.route.handle({
23852895 request,
23862896 url,
@@ -2389,7 +2899,7 @@ export function serve(options: ServeOptions) {
23892899 params: found.params,
23902900 signin: options.signin,
23912901 });
2392 return head(request, response);
2902 return head(request, await compress(request, response));
23932903 },
23942904 // A bug answers with the floor and nothing else: never the runtime's
23952905 // own error page, which carries source and paths, and never without
@@ -2397,7 +2907,11 @@ export function serve(options: ServeOptions) {
23972907 error(error) {
23982908 // Inside the generated app, which has no forge log module.
23992909 console.error(error);
2400 return html("<h1>Something went wrong</h1>", 500);
2910 try {
2911 return html(problem(500, ANON), 500);
2912 } catch {
2913 return html("<h1>Something went wrong</h1>", 500);
2914 }
24012915 },
24022916 });
24032917 }
@@ -2600,6 +3114,23 @@ code,
26003114 .muted {
26013115 color: var(--color-muted);
26023116 }
3117 /* The first thing a keyboard reaches: a way past the header, seen only when focused. */
3118 .skip {
3119 position: absolute;
3120 left: var(--space-3);
3121 top: var(--space-3);
3122 transform: translateY(-300%);
3123 padding: var(--space-2) var(--space-3);
3124 background: var(--color-accent);
3125 color: var(--color-accent-fg);
3126 border-radius: var(--radius-md);
3127 }
3128 .skip:focus {
3129 transform: none;
3130 }
3131 main:focus {
3132 outline: none;
3133 }
26033134 .you {
26043135 color: var(--color-muted);
26053136 white-space: nowrap;
@@ -2742,6 +3273,21 @@ footer code {
27423273 visibility: hidden;
27433274 }
27443275 }
3276 /* Forced colours replace ours with the person's own; what only a background
3277 or a shadow drew gets a border, so it is still there to see. */
3278 @media (forced-colors: active) {
3279 .card,
3280 .input,
3281 .button,
3282 .pill,
3283 .toast,
3284 .skip {
3285 border: 1px solid CanvasText;
3286 }
3287 :focus-visible {
3288 outline-color: Highlight;
3289 }
3290 }
27453291 /* Motion is a courtesy: whoever asked for less gets none. */
27463292 @media (prefers-reduced-motion: reduce) {
27473293 *,
@@ -3571,7 +4117,7 @@ test("a link to the page unfurls: description, Open Graph, a card", async () =>
35714117 'property="og:description"',
35724118 'property="og:url"',
35734119 'property="og:type" content="website"',
3574 'name="twitter:card" content="summary"',
4120 'name="twitter:card" content="summary_large_image"',
35754121 'name="theme-color"',
35764122 ])
35774123 expect([meta, page.includes(meta)]).toEqual([meta, true]);
test/apps-busy.test.ts +1
@@ -167,6 +167,7 @@ describe("apps that are busy", () => {
167167 { path: "src/audit.test.ts", content: null },
168168 { path: "src/openapi.test.ts", content: null },
169169 { path: "src/security.test.ts", content: null },
170 { path: "src/site.test.ts", content: null },
170171 ],
171172 });
172173 if (!committed.ok) throw new Error(String(committed.error));
test/apps-identity.test.ts +1
@@ -25,6 +25,7 @@ const SERVER_SUITES = [
2525 "src/audit.test.ts",
2626 "src/openapi.test.ts",
2727 "src/security.test.ts",
28 "src/site.test.ts",
2829 ];
2930 import {
3031 APPS_DOMAIN,
test/apps-toolbar.test.ts +38
@@ -43,6 +43,16 @@ describe("the forge pill", () => {
4343 fetch(request) {
4444 const path = new URL(request.url).pathname;
4545 const html = { "content-type": "text/html; charset=utf-8" };
46 if (path === "/negotiate") {
47 // Gzips whenever it is allowed to, and says what it was asked for.
48 const saw = request.headers.get("accept-encoding") ?? "";
49 const headers = { ...html, "x-saw": saw };
50 return /\bgzip\b/.test(saw)
51 ? new Response(gzipSync(HTML), {
52 headers: { ...headers, "content-encoding": "gzip" },
53 })
54 : new Response(HTML, { headers });
55 }
4656 if (path === "/json") return Response.json({ a: 1 });
4757 if (path === "/img") return new Response(PNG, { headers: { "content-type": "image/png" } });
4858 if (path === "/bare") return new Response("<p>no body tag</p>", { headers: html });
@@ -179,6 +189,34 @@ describe("the forge pill", () => {
179189 toolbar = undefined;
180190 });
181191
192 test("an app that gzips whenever it may still gets the pill, because the page is asked for plain", async () => {
193 toolbar = undefined;
194 const res = await fetch(`${servers.url}/negotiate`, {
195 headers: { host: `notes.${DOMAIN}`, accept: "text/html", "accept-encoding": "gzip" },
196 redirect: "manual",
197 });
198 expect(res.headers.get("x-saw")).toBe("identity");
199 expect(await res.text()).toBe(HTML.replace("</body>", `${tag()}</body>`));
200 });
201
202 test("the client's own accept-encoding is left alone where no pill goes in", async () => {
203 const ask = (host: string, accept: string) =>
204 fetch(`${servers.url}/negotiate`, {
205 headers: { host, accept, "accept-encoding": "gzip" },
206 redirect: "manual",
207 });
208 // Not a page: an API client or an asset fetch.
209 expect((await ask(`notes.${DOMAIN}`, "application/json")).headers.get("x-saw")).toBe("gzip");
210 // Nested: a guest's world, never given the pill.
211 expect((await ask(`guest--notes.${DOMAIN}`, "text/html")).headers.get("x-saw")).toBe("gzip");
212 // Opted out.
213 toolbar = false;
214 const off = await ask(`notes.${DOMAIN}`, "text/html");
215 toolbar = undefined;
216 expect(off.headers.get("x-saw")).toBe("gzip");
217 expect(await off.text()).toBe(HTML);
218 });
219
182220 test("the script is served from the forge, immutable, small, and plain", async () => {
183221 const res = await fetch(`${servers.url}${TOOLBAR_PATH}`);
184222 expect(res.status).toBe(200);
test/apps-ws.test.ts +1
@@ -246,6 +246,7 @@ describe("apps: websockets through the proxy", () => {
246246 { path: "src/audit.test.ts", content: null },
247247 { path: "src/openapi.test.ts", content: null },
248248 { path: "src/security.test.ts", content: null },
249 { path: "src/site.test.ts", content: null },
249250 ],
250251 });
251252 if (!committed.ok) throw new Error(String(committed.error));

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs