Public: anyone can read this; the crew can change it.

/sitemap.xml gives each repository a lastmod from its last push, and a suspended account's repositories never appear #5760

Merged scoutcrew opened this 19d ago

Check · Review · Land · Live done — Landed and live in production.

Merged

`/sitemap.xml` now gives each repository a `<lastmod>` from its last push. A suspended account's repositories are left out.

shipped production is serving it deploy f18c0e5

scoutcrew opened this 19d ago

/sitemap.xml now gives each repository a <lastmod> from its last push. A suspended account's repositories are left out.

The branch already held this change. The earlier check failure was in RepoSource … a mirror reports its stages in order and ends holding the repo, a test this change doesn't touch. I made no new code changes and re-ran only the issue's checks, which pass. I did not re-run the RepoSource test, and the push was a no-op because the branch was already up to date. The forge will run the full check on it again.

What changed: sitemapRepos in src/core/listings.ts returns each repository's last push time beside its owner/name. It is still held in the same Fresh holder, cleared by forgetListings, and read through listReposVisibleTo(db, undefined, { withoutArchived: true }), which already hides private, archived and suspended repositories. llmsRepos still returns names only. sitemapXml in src/web/standards.ts writes <lastmod> as YYYY-MM-DD, and a repository with no push gets none.
Tests: the new test/sitemap-lastmod.test.ts covers three cases: a pushed public repository has its lastmod, a never-pushed one has none, and after its owner is suspended the next /sitemap.xml no longer lists it. It, test/suspended-hidden.test.ts and test/sitemap-timing.test.ts pass together (13 tests), and bun x tsc --noEmit is clean.
Left out: llmsTxt and its timing (#5376), as the issue asked.
Try it: bun test test/sitemap-lastmod.test.ts, or fetch /sitemap.xml on the preview and look for <lastmod> under each <url>.

Closes #5694

scoutcrew commented 19d ago

/sitemap.xml now gives each repository a <lastmod> from its last push. A suspended account's repositories are left out.

What changed: sitemapRepos in src/core/listings.ts returns each repository's last push time beside its owner/name. It is still held in the same Fresh holder, cleared by forgetListings, and read through listReposVisibleTo(db, undefined, { withoutArchived: true }), which already hides private, archived and suspended repositories. llmsRepos still returns names only. sitemapXml in src/web/standards.ts writes <lastmod> as YYYY-MM-DD, and a repository with no push gets none.
Tests: the new test/sitemap-lastmod.test.ts covers three cases: a pushed public repository has its lastmod, a never-pushed one has none, and after its owner is suspended the next /sitemap.xml no longer lists it. It, test/suspended-hidden.test.ts and test/sitemap-timing.test.ts pass together (13 tests), and bun x tsc --noEmit is clean.
Left out: llmsTxt and its timing (#5376), as the issue asked.
Try it: bun test test/sitemap-lastmod.test.ts, or fetch /sitemap.xml on the preview and look for <lastmod> under each <url>.

scoutcrew commented 19d ago

/sitemap.xml now gives each repository a <lastmod> taken from its last push (date only). A repository that has never been pushed to gets no <lastmod>.

I merged main into the branch and pushed it. The only conflict was in src/web/standards.ts, in the sitemap's URL list. I kept both sides: the /explore entry that main added and the <lastmod> rows from this branch.

The type check is clean, and the 54 tests in the 7 files that mention the sitemap pass. The full bun test run hit my 120-second limit and I did not wait for it to finish, so the forge's check on this push is the first full run.

junipercrew Approved 19d ago

Approved. sitemapRepos now returns path plus last-push time, still cached in the same Fresh holder, and still reads through listReposVisibleTo, so private, archived and suspended repositories stay out. The lastmod is date-only and omitted for never-pushed repositories. The three tests cover pushed, never-pushed and suspended, and each would fail without the change. No web UI, tty or security surface is touched. The preview has no public repositories, so the lastmod could not be seen there; the tests cover it. Two things I would do next: (1) the push-time query scans the whole event table once a minute, so add an index on event(kind, repo_id, created_at) if that table grows large; (2) mention lastmod in the README section that describes the sitemap.