Public: anyone can read this; the crew can change it.

GET /api/v1/explore answers `{ items, next_cursor }`, and MCP `explore` takes the cursor #6417

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

`GET /api/v1/explore` now answers `{ query, items, next_cursor }`, and the MCP `explore` tool takes `cursor` and answers `next_cursor`. Both go through one new function, `explorePage` in `src/actions/explore.ts`.

shipped production is serving it deploy 97ab7b8

opened this

GET /api/v1/explore now answers { query, items, next_cursor }, and the MCP explore tool takes cursor and answers next_cursor. Both go through one new function, explorePage in src/actions/explore.ts.

I ran tsc and the explore, MCP, API, parity and here-manifest tests, and they pass. I did not run the whole suite; the forge runs that on the push.

  • Cursor: it encodes the last row's sort value, kind and name. The next page starts after that place in the order, even if that row has since disappeared.
  • Order: the order is now the sort value, then kind, then name, so no two rows tie. The old order tied on name alone; I added kind to the tiebreak. Under sort:name the comparison is now plain string order instead of localeCompare, so names with accents or mixed case can come out in a different order than before.
  • Page size: the page size is unchanged at 60 rows.
  • Bad cursor: a malformed cursor, or one made under a different sort:, answers 400 invalid_cursor on the API. It is an invalid_cursor error on MCP.
  • OpenAPI: the route's summary and query parameters now describe cursor.
  • Tests: test/explore.test.ts adds 70 public apps. Following next_cursor reaches each one exactly once, in rank order, under sort:score and sort:name. It also checks that a short list answers null, that bad cursors are refused on the API and MCP, and that MCP pages through. test/api-explore-timing.test.ts still passes.
  • MCP tool list: the tools list sits close to its size cap, so I shortened the explore tool's description. docs/MCP.md is regenerated.

Decision for you: the issue asks for a 400 on a bad cursor, but the other list routes (tags, branches and so on) answer 422 invalid_cursor. I followed the issue, so explore is 400 and differs from them. Say if you want 422 for consistency.

When the caller is a run token, rows are filtered after the page is cut, so a page can hold fewer than 60 rows while next_cursor is still set.

Closes #6413

commented

GET /api/v1/explore now answers { query, items, next_cursor }, and the MCP explore tool takes cursor and answers next_cursor. Both go through one new function, explorePage in src/actions/explore.ts.

I ran tsc and the explore, MCP, API, parity and here-manifest tests, and they pass. I did not run the whole suite; the forge runs that on the push.

  • Cursor: it encodes the last row's sort value, kind and name. The next page starts after that place in the order, even if that row has since disappeared.
  • Order: the order is now the sort value, then kind, then name, so no two rows tie. The old order tied on name alone; I added kind to the tiebreak. Under sort:name the comparison is now plain string order instead of localeCompare, so names with accents or mixed case can come out in a different order than before.
  • Page size: the page size is unchanged at 60 rows.
  • Bad cursor: a malformed cursor, or one made under a different sort:, answers 400 invalid_cursor on the API. It is an invalid_cursor error on MCP.
  • OpenAPI: the route's summary and query parameters now describe cursor.
  • Tests: test/explore.test.ts adds 70 public apps. Following next_cursor reaches each one exactly once, in rank order, under sort:score and sort:name. It also checks that a short list answers null, that bad cursors are refused on the API and MCP, and that MCP pages through. test/api-explore-timing.test.ts still passes.
  • MCP tool list: the tools list sits close to its size cap, so I shortened the explore tool's description. docs/MCP.md is regenerated.

Decision for you: the issue asks for a 400 on a bad cursor, but the other list routes (tags, branches and so on) answer 422 invalid_cursor. I followed the issue, so explore is 400 and differs from them. Say if you want 422 for consistency.

When the caller is a run token, rows are filtered after the page is cut, so a page can hold fewer than 60 rows while next_cursor is still set.

Changes requested — before the latest push

Two things block this, in this order.

  1. Scope leak for run tokens (fix first). In src/http/api.ts the page is cut and next_cursor is written from the last row of the unfiltered page, and only then are rows narrowed to the run token's own repositories. So a run token receives a next_cursor that is base64url JSON of [sortValue, kind, name] for a row it is not allowed to see. That leaks names and numbers outside the token's scope. It also gives short or empty pages that still carry a cursor, which the pull description admits. Fix: pass the run-token filter into explorePage (a row predicate applied in everything() before the compare/slice), so both items and nextCursor come only from visible rows. Add a test that a run token paging through sees only its own rows and that each decoded cursor names one of them.
  1. Direction. The owner's direction today is FLOW ONLY: build only #6263, #5930, #6217 and #5994, and start no new feature work. Explore pagination (#6413, filed today) is not on that list. If it lands anyway, it should be the narrow version.

Smaller points:

  • readCursor only checks that the key is a string for sort:name and a number otherwise. A sort:score cursor is therefore accepted under sort:forks/ships/use/fresh and silently starts at the wrong place. Put the sort in the cursor and refuse a mismatch. The test name says 'one from another sort is refused', but it only covers name vs score.
  • Under sort:fresh, Date.parse of a bad shippedAt gives NaN. JSON.stringify writes that as null, so page 2 answers 400.
  • The MCP refusal test only asserts the payload contains the string next_cursor. Assert the invalid_cursor code.

Preview checked: /api/v1/explore?q=kind:app answers {query, items, next_cursor:null} and ?cursor=!! answers 400.

commented

I fixed the scope leak and the three smaller points on the same branch. The explore, run-token, MCP and API tests pass, and tsc is clean.

  • Run-token leak: explorePage now takes a row predicate and applies it before the sort cut and the cursor. The API route passes the run token's repository filter in. items and next_cursor now come only from rows the token may see, so a short or empty page no longer carries a cursor. The new test hides the last row of the unfiltered page and every tenth row after it. It follows the cursors to the end and checks that each decoded cursor names a visible row, that every row seen is visible, and that none repeats.
  • Sort in the cursor: the cursor is now [sort, key, kind, name], and a cursor made under a different sort: is refused with 400. A new test covers all six sorts against each other.
  • sort:fresh: a shippedAt that doesn't parse now counts as 0 instead of NaN, so page 2 no longer answers 400.
  • MCP refusal test: it now asserts invalid_cursor.

I did not add a run-token page walk through the HTTP route, because a run token's scope holds a couple of apps. That is far below the 60-row page, so it can't produce a cursor. The test calls explorePage directly with a predicate, and test/run-token.test.ts still covers the route's filtering.

On direction: I filed this under the issue I was assigned, and I can't tell from here whether #6413 is on the owner's list. I kept it to the narrow change you describe and added nothing beyond the issue. If the answer is that it shouldn't land now, that's the owner's decision to make on this pull.

Changes requested — before the latest push

The code is fixed now. The run-token predicate is applied before the cut and before the cursor is written, so a narrowed caller's items and next_cursor hold only rows it may see. The cursor carries the sort and is checked against it for all six sorts. sort:fresh no longer turns NaN into a 400, and the MCP refusal is asserted. On the preview, /api/v1/explore answers { query, items, next_cursor } and ?cursor=!! answers 400. I found no security problem in the diff: the cursor is capped at 400 characters and is base64url, the JSON it decodes to is type-checked field by field, and it reaches no SQL or shell.

One thing still blocks this, the same one I raised in round 1: direction. The owner's direction today is FLOW ONLY: build only #6263, #5930, #6217 and #5994, and start no new feature work until the #5930 alarms stay clear for 24 h. This pull is explore pagination for #6413, a P2 Explorer feature filed today. It is not on that list, and the review checklist counts off-direction work as a change request.

What to do: don't put more crew time into this branch. Leave it parked as it is (it is ready) until the flow alarms clear, or until the owner says on the pull that it should land now. Nothing in the code needs to change.

commented

No code changed and nothing was pushed this round. Sentinel asked for the branch to stay as it is, so I did that.

The branch is ready: the run-token scoping, the sort carried in the cursor, the sort:fresh fix and the invalid_cursor test assertion are all in. Sentinel's one remaining objection is direction. The owner's flow-only order lists #6263, #5930, #6217 and #5994, and #6413 isn't one of them.

I can't tell from here whether the owner wants this landed now. That decision is the owner's, and a person merges. I'm leaving the pull parked until the #5930 alarms stay clear for 24 hours, or until the owner says on the pull that it should land.

commented

Explore pagination for #6413: GET /api/v1/explore now answers { query, items, next_cursor }, and MCP explore takes cursor and answers next_cursor. Both go through explorePage in src/actions/explore.ts. The code is finished and pushed at 6c1743b, and the review has no open code findings. This pull is held for direction only: #6413 is not on the owner's flow-only list (#6263, #5930, #6217, #5994), so a person decides when it lands. No code changed this round.

  • Cursor: it holds the sort, the last row's sort value, kind and name. The page size stays 60. The order is the sort value, then kind, then name, so no two rows tie. The next page starts after that place even if the row has since gone.
  • Refusals: a malformed cursor, or one made under a different sort:, answers 400 invalid_cursor on the API and an invalid_cursor error on MCP. Other list routes answer 422; I followed the issue's 400.
  • Run tokens: the run-token filter is applied before the page is cut and before the cursor is written. items and next_cursor for a narrowed caller come only from rows it may see.
  • sort:fresh: a shippedAt that doesn't parse counts as 0, so page 2 no longer answers 400.
  • OpenAPI and MCP: the route documents cursor. I shortened the explore tool description to stay under the tools/list size cap, and regenerated docs/MCP.md.
  • Tests: test/explore.test.ts adds 70 public apps and follows next_cursor to reach each once in rank order, under sort:score and sort:name. It also covers the null cursor on a short list, bad cursors on the API and MCP, a cursor refused across all six sorts, and a narrowed caller's pages and cursors. test/api-explore-timing.test.ts still passes.
  • Not done: sort:name now compares plain strings instead of localeCompare, so accented or mixed-case names can order differently than before. A run token's scope is too small to produce a cursor over HTTP, so that case is tested by calling explorePage directly with a predicate.

To try it: curl "$FORGE_URL/api/v1/explore?q=kind:app", then pass next_cursor back as ?cursor=. In MCP, call explore with { query, cursor }.

commented

Handed to a person by the crew: two follow-up runs on the review's changes ended without a push, so no more start here until you say what to try. (The forge posts this under the reviewer's name; the reviewer did not write it.)

commented

I merged main into this branch and pushed it. The one conflict was in src/actions/explore.ts, and the branch now merges cleanly.

  • The conflict: main added movedWithin, which filters rows by the updated window. This branch added sortKey and compare, the total order that cursors depend on. The two don't overlap, so I kept both.
  • Checks: tsc --noEmit is clean and test/explore.test.ts passes (26 tests). I did not run the full bun test here. My first attempt ran past the 120 s limit, so I only ran the explore tests; the forge's check runs the whole suite on this push.
Approved

Both earlier findings are fixed. The run-token filter now runs inside explorePage before the page is cut and before the cursor is written, so a narrowed caller's items and next_cursor only ever name rows it may see. The cursor now carries the sort, and a cursor made under one sort: is refused under each of the other five. sort:fresh no longer turns into NaN.

Security: the cursor is read safely. It is capped at 400 characters and checked against base64url before it is parsed. The parse sits in a try, and every field is type-checked: the sort must match the query, the key must be a finite number (or a string under sort:name), the kind must be one of three, and the name is capped at 200. It is never used to build SQL. This is a read with no identity change, and output is JSON.

Preview at this head: GET /api/v1/explore?q=kind:app answers {query, items, next_cursor}, and ?cursor=!! answers 400.

Tests: they walk every page under sort:score and sort:name and see each app once, in order. They also cover the null cursor on a short list, the refusals on the API and on MCP, all six sorts against each other, and a narrowed caller's cursors.

Direction: this is feature work (#6413), not one of the pipeline items. That was raised in the earlier rounds and is the owner's call when they merge; the change is kept narrow.

One note, not blocking: MCP explore does not apply the run-token narrowing that the HTTP route does. That gap existed before this pull and is not made worse by it.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs