sitemap.xml and llms.txt are built at most once a minute, never once per stranger's request #3149

/sitemap.xml and /llms.txt now each read their repository list from the database at most once a minute per forge. A repository that stops being public drops out of both at the next request. The last attempt failed because suspending an account left that account's repositories in the cached lists. That is fixed. test/standards.test.ts and test/suspended-hidden.test.ts pass (20 tests). I have not run the type check because bun can't write to its temp dir here, and I have not run the full suite.

• Caching: the sitemap already had the 60-second WeakMap<App, Fresh<…>> cache. /llms.txt now has the same. Concurrent requests share one build. Both keep only repository names, so a caller varying Host cannot force a rebuild.
• Clearing: forgetListings(app) empties both caches. These now call it:

- making a repository private (updateRepoSettings), - an origin sync closing a mirror that went private upstream, - suspending, restoring or removing an account, - deleting a repository.

• Seam: listingBuilds counts database reads per listing, so tests count builds rather than waiting on a clock.
• Tests: in test/standards.test.ts, 50 requests to each file build the body once. A repository turned private leaves both files at once. A suspended account's repository leaves both files at once and returns when the account is restored.

What I left out: a new public repository appears within a minute rather than instantly (the cache window). Other things that change who can see a repository, such as org retirement or transfer, are not cleared by hand and age out within a minute.

To try it, fetch /sitemap.xml and /llms.txt a few times. Then make a repository private, or suspend its owner, and fetch them again.

🤖 Generated with Claude Code

Closes #3142

/sitemap.xml and /llms.txt now each read their repository list from the database at most once a minute per forge. A repository that stops being public drops out of both at the next request. The last attempt failed because suspending an account left that account's repositories in the cached lists. That is fixed. test/standards.test.ts and test/suspended-hidden.test.ts pass (20 tests). I have not run the type check because bun can't write to its temp dir here, and I have not run the full suite.

• Caching: the sitemap already had the 60-second WeakMap<App, Fresh<…>> cache. /llms.txt now has the same. Concurrent requests share one build. Both keep only repository names, so a caller varying Host cannot force a rebuild.
• Clearing: forgetListings(app) empties both caches. These now call it:

- making a repository private (updateRepoSettings), - an origin sync closing a mirror that went private upstream, - suspending, restoring or removing an account, - deleting a repository.

• Seam: listingBuilds counts database reads per listing, so tests count builds rather than waiting on a clock.
• Tests: in test/standards.test.ts, 50 requests to each file build the body once. A repository turned private leaves both files at once. A suspended account's repository leaves both files at once and returns when the account is restored.

What I left out: a new public repository appears within a minute rather than instantly (the cache window). Other things that change who can see a repository, such as org retirement or transfer, are not cleared by hand and age out within a minute.

To try it, fetch /sitemap.xml and /llms.txt a few times. Then make a repository private, or suspend its owner, and fetch them again.

🤖 Generated with Claude Code

/sitemap.xml and /llms.txt now read their repository lists from the database at most once a minute. Anything that changes who is listed now clears that list itself, retired organizations included. I couldn't run bun x tsc --noEmit here because bun can't write to its temp directory in this sandbox, so the type check is unverified. The tests for test/standards.test.ts, accounts.test.ts and accounts-retire.test.ts pass.

• Where the cache lives: the cache is now in src/core/listings.ts, so nothing in src/actions/* or src/origins/sync.ts imports from the web layer any more. It holds the cached lists, forgetListings(db), and sitemapRepos and llmsRepos, which web/standards.ts calls. Both lists are still stored without an origin, so a caller who varies Host can't force a rebuild.
• What clears it: the cache is cleared at the write itself rather than at each caller, in these places:

- setSuspended, scrubAccount and retireOrg in core/accounts.ts and core/orgs.ts - updateRepoSettings and the repository delete in core/repos.ts - the mirror visibility update in origins/sync.ts

This means the HTTP, MCP and TTY doors are covered without any per-door calls, and I removed the earlier calls in actions/*.

• Tests: a new test retires an organization with a public repository and checks it drops out of both lists at once. The existing tests for making a repository private and for suspending an account still pass.
• Tidying: the imports sit in order, and listingBuilds is exported with a comment naming it a test seam. I hand-formatted the two flagged test spots because no formatter is installed here.

I chose not to clear the lists when a repository is created, so a new public repository shows up within a minute. Clearing on every creation would let a burst of creations force a rebuild each time, which is what this pull exists to prevent. The new test calls forgetListings itself after creating its repository for that reason.

To try it: curl localhost:$PORT/sitemap.xml repeatedly, and check that the output changes only after a minute or after a repository is made private, deleted, or its owner suspended or removed.

🤖 Generated with Claude Code

/sitemap.xml and /llms.txt now read their repository lists from the database at most once a minute, and whatever changes who is listed clears them. The type check and full suite were not run here, because bun can't write its temp directory in this sandbox, so the forge's check on this push will be the first run of them. The standards and account-retire tests (21) pass.

• Cache: it lives in src/core/listings.ts, so actions/* and origins/sync.ts no longer import from the web layer. web/standards.ts only reads it.
• Clearing: the cache is cleared at the core write that changes who is listed:

- suspending or removing an account - retiring an organization - changing a repository's visibility or deleting it - a mirror sync that changes a repository's visibility

• This round:

- scrubAccount now clears the listings after the account row is written, not before. - I deleted the orphaned LLMS_REPOS comment above llmsTxt. - I removed the doubled blank line left where sitemapPaths was deleted.

• Left out: creating a public repository does not clear the cache, so a new one appears within a minute. Clearing on every creation would let a burst of creations force a rebuild each time.

To try it: request /sitemap.xml repeatedly; the list changes only after a minute, or when a repository goes private or is deleted, or its owner is suspended or removed, or its organization is retired.

🤖 Generated with Claude Code