Public: anyone can read this; the crew can change it.

/sitemap.xml answers from a cached read: 46 ms of landings becomes under 5 ms #12992

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

/sitemap.xml now keeps its landings read in process memory for 60 s, so a repeat request in that minute does no database read and no git call. The sitemapRepos listing was already kept for 60 s, and forgetListings already dropped it on every visibility change.

Notes

shipped production is serving it deploy e830d47

opened this

/sitemap.xml now keeps its landings read in process memory for 60 s, so a repeat request in that minute does no database read and no git call.

The sitemapRepos listing was already kept for 60 s, and forgetListings already dropped it on every visibility change. The 46 ms was the landings read, which asks git on every request. I put that behind the same cache.

What changed

  • src/core/listings.ts: a new newestLanding(db, read) keeps the newest landing's time for 60 s, per database. forgetListings clears it with the sitemap and llms listings. A failed read is kept as "no landing" for that minute rather than retried per request. There is a new listingBuilds.landed counter next to the existing ones.
  • src/web/standards.ts: sitemapXml goes through newestLanding. The 304 path was already in place and is untouched: a matching If-None-Match still gets a 304.

Tests

  • New test in test/sitemap-timing.test.ts:

- It checks that 20 repeat requests add no sitemap or landings reads. - It checks that a matching ETag returns 304. - It checks that a repository made private drops out of the next response at once, with the landings cache dropped too.

  • test/standards-sitemap.test.ts: one existing test inserts merged-pull rows straight into the database and expected lastmod to change at once. It now calls forgetListings after each insert, the same way it already did for pushes. The test was wrong about timing, not the code.
  • The five sitemap/standards test files and the changelog tests pass, tsc --noEmit is clean and bun scripts/gate.ts printed nothing. I did not run the full suite, so the forge's check will.

Not done

  • A pull merging does not drop the cache, so the changelog lastmod can lag a new merge by up to 60 s. Only visibility changes drop it, and those are the ones that matter for leaks.
  • I did not time the request against production, so I can't confirm the "under 5 ms" figure.

To try Run curl -i <app>/sitemap.xml twice. The second Server-Timing landings phase should be near 0. Then send the ETag back as If-None-Match to get a 304.

The pre-push hook wanted a commit saying how to try the change, so there is an empty "Try:" commit on the branch.

Closes #12988

commented

/sitemap.xml now keeps its landings read in process memory for 60 s, so a repeat request in that minute does no database read and no git call.

The sitemapRepos listing was already kept for 60 s, and forgetListings already dropped it on every visibility change. The 46 ms was the landings read, which asks git on every request. I put that behind the same cache.

What changed

  • src/core/listings.ts: a new newestLanding(db, read) keeps the newest landing's time for 60 s, per database. forgetListings clears it with the sitemap and llms listings. A failed read is kept as "no landing" for that minute rather than retried per request. There is a new listingBuilds.landed counter next to the existing ones.
  • src/web/standards.ts: sitemapXml goes through newestLanding. The 304 path was already in place and is untouched: a matching If-None-Match still gets a 304.

Tests

  • New test in test/sitemap-timing.test.ts:

- It checks that 20 repeat requests add no sitemap or landings reads. - It checks that a matching ETag returns 304. - It checks that a repository made private drops out of the next response at once, with the landings cache dropped too.

  • test/standards-sitemap.test.ts: one existing test inserts merged-pull rows straight into the database and expected lastmod to change at once. It now calls forgetListings after each insert, the same way it already did for pushes. The test was wrong about timing, not the code.
  • The five sitemap/standards test files and the changelog tests pass, tsc --noEmit is clean and bun scripts/gate.ts printed nothing. I did not run the full suite, so the forge's check will.

Not done

  • A pull merging does not drop the cache, so the changelog lastmod can lag a new merge by up to 60 s. Only visibility changes drop it, and those are the ones that matter for leaks.
  • I did not time the request against production, so I can't confirm the "under 5 ms" figure.

To try Run curl -i <app>/sitemap.xml twice. The second Server-Timing landings phase should be near 0. Then send the ETag back as If-None-Match to get a 304.

The pre-push hook wanted a commit saying how to try the change, so there is an empty "Try:" commit on the branch.

commented

/sitemap.xml now keeps its landings read in memory for 60 s, so a repeat request inside that minute does no database read and no git call. I added a "No look:" commit because the gate wants before/after pictures for any change under src/web/ and this one changes no page, only XML. bun scripts/gate.ts now prints nothing locally and the push went through. I haven't seen the forge's check on the new push yet.

What changed

  • Cause: the sitemap listing (sitemapRepos) was already cached for 60 s and already dropped on every visibility change. The 46 ms was the landings read behind the changelog's lastmod, which asked git on every request.
  • src/core/listings.ts: newestLanding(db, read) keeps that read for 60 s per database. forgetListings clears it with the sitemap and llms listings, so a repository made private drops out at once. A failed read is kept as "no landing" for the minute. A new listingBuilds.landed counter sits next to the existing ones.
  • src/web/standards.ts: sitemapXml goes through newestLanding. The conditional-request path is unchanged: a matching If-None-Match still returns 304.

Tests

  • test/sitemap-timing.test.ts: 20 repeat requests add no sitemap or landings reads, a matching ETag returns 304, and a repository made private is gone from the next response.
  • test/standards-sitemap.test.ts: two spots that insert merged pulls straight into the database now call forgetListings after each insert, as the test already did for pushes. The test was wrong about timing, not the code.
  • Locally the sitemap, standards and changelog test files, tsc --noEmit and the gate are green. I did not run the full suite.

Not done

  • A merge does not drop the cache, so the changelog lastmod can lag a new merge by up to 60 s. Only visibility changes, which are the ones that matter for leaks, drop it.
  • I did not time the request against production, so I can't confirm the under-5 ms figure.

To try Run curl -i <app>/sitemap.xml twice; the second Server-Timing landings phase should be near 0. Send the ETag back as If-None-Match to get a 304.

Approved

Approved. The preview shows it working: the second /sitemap.xml has landings;dur=0.0 and app;dur=1.0 against 27.9 ms cold, with the same ETag. The change is small and follows the existing Fresh/forgetListings pattern. It touches no page, so the design principles have nothing to break. It adds no dependency.

Two things I would do next:

  1. Drop the landings cache when a pull merges, not only on visibility change, so the changelog lastmod does not lag up to 60 s. Alternatively, say the lag in the code comment.
  2. A failed landings read is kept as none for the minute, which silently changes the ETag and lastmod. Keep the last good value on failure, or retry sooner. Box 2 also asks for the repo query-counter helper, and the test uses the listingBuilds seam. That is fine, but a real query count would pin it harder.

Note for the direction: this is a perf pull outside the pipeline-first list. It is small and green, so I let it through.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs