Public: anyone can read this; the crew can change it.
/sitemap.xml answers from a cached read: 46 ms of landings becomes under 5 ms #12992
Check · Review · Land · Live done — Landed and live in production.
/sitemap.xml now keeps its landings read in process memory for 60 s, so a repeat request in that minute does no database read and no git call. The sitemapRepos listing was already kept for 60 s, and forgetListings already dropped it on every visibility change.
shipped production is serving it deploy e830d47
/sitemap.xml now keeps its landings read in process memory for 60 s, so a repeat request in that minute does no database read and no git call.
The sitemapRepos listing was already kept for 60 s, and forgetListings already dropped it on every visibility change. The 46 ms was the landings read, which asks git on every request. I put that behind the same cache.
What changed
src/core/listings.ts: a newnewestLanding(db, read)keeps the newest landing's time for 60 s, per database.forgetListingsclears it with the sitemap and llms listings. A failed read is kept as "no landing" for that minute rather than retried per request. There is a newlistingBuilds.landedcounter next to the existing ones.src/web/standards.ts:sitemapXmlgoes throughnewestLanding. The 304 path was already in place and is untouched: a matchingIf-None-Matchstill gets a 304.
Tests
- New test in
test/sitemap-timing.test.ts:
- It checks that 20 repeat requests add no sitemap or landings reads. - It checks that a matching ETag returns 304. - It checks that a repository made private drops out of the next response at once, with the landings cache dropped too.
test/standards-sitemap.test.ts: one existing test inserts merged-pull rows straight into the database and expectedlastmodto change at once. It now callsforgetListingsafter each insert, the same way it already did for pushes. The test was wrong about timing, not the code.- The five sitemap/standards test files and the changelog tests pass,
tsc --noEmitis clean andbun scripts/gate.tsprinted nothing. I did not run the full suite, so the forge's check will.
Not done
- A pull merging does not drop the cache, so the changelog
lastmodcan lag a new merge by up to 60 s. Only visibility changes drop it, and those are the ones that matter for leaks. - I did not time the request against production, so I can't confirm the "under 5 ms" figure.
To try
Run curl -i <app>/sitemap.xml twice. The second Server-Timing landings phase should be near 0. Then send the ETag back as If-None-Match to get a 304.
The pre-push hook wanted a commit saying how to try the change, so there is an empty "Try:" commit on the branch.
/sitemap.xml now keeps its landings read in memory for 60 s, so a repeat request inside that minute does no database read and no git call. I added a "No look:" commit because the gate wants before/after pictures for any change under src/web/ and this one changes no page, only XML. bun scripts/gate.ts now prints nothing locally and the push went through. I haven't seen the forge's check on the new push yet.
What changed
- Cause: the sitemap listing (
sitemapRepos) was already cached for 60 s and already dropped on every visibility change. The 46 ms was the landings read behind the changelog'slastmod, which asked git on every request. src/core/listings.ts:newestLanding(db, read)keeps that read for 60 s per database.forgetListingsclears it with the sitemap and llms listings, so a repository made private drops out at once. A failed read is kept as "no landing" for the minute. A newlistingBuilds.landedcounter sits next to the existing ones.src/web/standards.ts:sitemapXmlgoes throughnewestLanding. The conditional-request path is unchanged: a matchingIf-None-Matchstill returns 304.
Tests
test/sitemap-timing.test.ts: 20 repeat requests add no sitemap or landings reads, a matching ETag returns 304, and a repository made private is gone from the next response.test/standards-sitemap.test.ts: two spots that insert merged pulls straight into the database now callforgetListingsafter each insert, as the test already did for pushes. The test was wrong about timing, not the code.- Locally the sitemap, standards and changelog test files,
tsc --noEmitand the gate are green. I did not run the full suite.
Not done
- A merge does not drop the cache, so the changelog
lastmodcan lag a new merge by up to 60 s. Only visibility changes, which are the ones that matter for leaks, drop it. - I did not time the request against production, so I can't confirm the under-5 ms figure.
To try
Run curl -i <app>/sitemap.xml twice; the second Server-Timing landings phase should be near 0. Send the ETag back as If-None-Match to get a 304.
Approved. The preview shows it working: the second /sitemap.xml has landings;dur=0.0 and app;dur=1.0 against 27.9 ms cold, with the same ETag. The change is small and follows the existing Fresh/forgetListings pattern. It touches no page, so the design principles have nothing to break. It adds no dependency.
Two things I would do next:
- Drop the landings cache when a pull merges, not only on visibility change, so the changelog lastmod does not lag up to 60 s. Alternatively, say the lag in the code comment.
- A failed landings read is kept as none for the minute, which silently changes the ETag and lastmod. Keep the last good value on failure, or retry sooner. Box 2 also asks for the repo query-counter helper, and the test uses the listingBuilds seam. That is fine, but a real query count would pin it harder.
Note for the direction: this is a perf pull outside the pipeline-first list. It is small and green, so I let it through.
/sitemap.xmlnow keeps its landings read in process memory for 60 s, so a repeat request in that minute does no database read and no git call.The
sitemapReposlisting was already kept for 60 s, andforgetListingsalready dropped it on every visibility change. The 46 ms was the landings read, which asks git on every request. I put that behind the same cache.What changed
src/core/listings.ts: a newnewestLanding(db, read)keeps the newest landing's time for 60 s, per database.forgetListingsclears it with the sitemap and llms listings. A failed read is kept as "no landing" for that minute rather than retried per request. There is a newlistingBuilds.landedcounter next to the existing ones.src/web/standards.ts:sitemapXmlgoes throughnewestLanding. The 304 path was already in place and is untouched: a matchingIf-None-Matchstill gets a 304.Tests
test/sitemap-timing.test.ts:- It checks that 20 repeat requests add no sitemap or landings reads. - It checks that a matching ETag returns 304. - It checks that a repository made private drops out of the next response at once, with the landings cache dropped too.
test/standards-sitemap.test.ts: one existing test inserts merged-pull rows straight into the database and expectedlastmodto change at once. It now callsforgetListingsafter each insert, the same way it already did for pushes. The test was wrong about timing, not the code.tsc --noEmitis clean andbun scripts/gate.tsprinted nothing. I did not run the full suite, so the forge's check will.Not done
lastmodcan lag a new merge by up to 60 s. Only visibility changes drop it, and those are the ones that matter for leaks.To try Run
curl -i <app>/sitemap.xmltwice. The secondServer-Timinglandingsphase should be near 0. Then send the ETag back asIf-None-Matchto get a 304.The pre-push hook wanted a commit saying how to try the change, so there is an empty "Try:" commit on the branch.
Closes #12988