Public: anyone can read this; the crew can change it.
The sitemap lists /explore and every public repository's page #5679
/sitemap.xml now lists /, /explore and every public, unarchived repository. Each repository entry carries a taken from its last push, and a repository never pushed to has none. I merged main again to fix the conflict in src/web/standards.ts and pushed.
stopped parked by the crew deploy de81220
4 earlier comments
/sitemap.xml now lists /, /explore and every public, unarchived repository. Each repository entry carries a <lastmod> taken from its last push, and a repository never pushed to has none. I merged main again to fix the conflict in src/web/standards.ts and pushed. tsc --noEmit is clean and test/standards.test.ts passes (18 tests). I did not run the full suite or the forge's check.
- List size: up to 50,000 URLs, the sitemap protocol's limit (it was 5,000 repositories).
- Caching: the built document is kept for a minute per origin, up to 8 origins, and is dropped together with the repository rows. The 50,000 escaped entries are not rebuilt on every request.
- Merge conflict: main had since split the handler into
reposandrendertiming phases. I kept this branch's per-origin cache and wrapped its work in those same two phases, soServer-Timingstill names them. - Not done: no sitemap index for forges over 50,000 public repositories; the list is cut at the newest ones.
To try it, fetch /sitemap.xml on the preview at https://lab--5679--forge-trevato-apps.open-platform.sh.
/sitemap.xml now lists /, /explore and every public, unarchived repository, each with a <lastmod> from its last push. The list is capped at 50,000 URLs. bun x tsc --noEmit is clean and test/standards.test.ts passes (18 tests). I did not run the rest of the suite. The branch is pushed and has main merged in, so the forge's check will run on it.
- What the sitemap lists: a repository that was never pushed to is listed without a
<lastmod>. Private and archived repositories are left out. - The merge: main had meanwhile changed the sitemap to cache its rows without an origin and to time each phase. The merge conflicted in
src/web/standards.ts, and I kept main's structure with/exploreand the<lastmod>added on top. I removed the extra per-origin cache of the whole document that my branch had added, because main already avoids building perHost. - The tests:
test/standards.test.tscovers a public repository listed with its lastmod, private and archived ones left out, and the cap. - Not done:
llms.txt, which the issue says another issue changes in the same file.
To try it, fetch /sitemap.xml on the preview.
One thing was off during the merge. Something committed and pushed for me while I was sorting out a transient index.lock, so I did not make that commit myself. The tree is clean and nothing is missing.
/sitemap.xml now lists /, /explore and every public, unarchived repository, each with a <lastmod> from its last push. This run changed nothing: the branch already has current main merged in and is pushed. I did not rerun the tests this time. bun x tsc --noEmit and test/standards.test.ts (18 tests) passed on the previous run.
- What the sitemap lists: a repository that was never pushed to is listed without a
<lastmod>. Private and archived repositories are left out, and the list is capped at 50,000 URLs. - How it is built: the sitemap's rows are read once a minute and the origin is added per request, as main already does. The last-push lookup is an index seek.
- Tests: they cover a public repository listed with its lastmod, private and archived ones left out, the cap, and the query plan.
- Docs:
docs/STANDARDS.mdnow says the sitemap carries/exploreandlastmod. - Not done:
llms.txt, which the issue says another issue changes in the same file.
To try it, fetch /sitemap.xml on the preview.
I found nothing in this pull to fix, so I pushed an empty commit (de81220d) to make the forge re-run the check.
The failure log you sent ends at the command line and shows no error. Here is what I ran:
- Gate:
bun scripts/gate.tspasses. - Type check:
tsc --noEmitis clean. - Sitemap tests:
test/standards.test.tspasses. - Branch state: it already contains
origin/mainatd28c8f3.
The fast-tier shards run here did show failures, in the asks and look tests. test/asks.test.ts fails the same way on a clean checkout of origin/main, so my change does not cause it. I did not check the look tests (the Chromium captures) against main.
If the re-run fails again, send me the full log.
Parked by the crew: the check stayed red through 2 automatic tries
/sitemap.xmlnow lists/,/exploreand every public, unarchived repository. Each repository entry carries a<lastmod>taken from its last push, and a repository never pushed to has none. I merged main again to fix the conflict insrc/web/standards.tsand pushed.tsc --noEmitis clean andtest/standards.test.tspasses (18 tests). I did not run the full suite or the forge's check.reposandrendertiming phases. I kept this branch's per-origin cache and wrapped its work in those same two phases, soServer-Timingstill names them.To try it, fetch
/sitemap.xmlon the preview at https://lab--5679--forge-trevato-apps.open-platform.sh.Continues #5679