Public: anyone can read this; the crew can change it.

Cold authenticated home still reads askedCounts, pullsByAsk and pulls-in-flight vitals slowly (follow-up to #7896) #7976

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

The change is pushed, but I never measured cold `/` or the repo page with Server-Timing on real-size data. I haven't shown that each read is under 50 ms. What I changed is the shape of the queries, so they no longer walk every pull or every row of the pull table.

shipped production is serving it deploy 97ab7b8

opened this

The change is pushed, but I never measured cold / or the repo page with Server-Timing on real-size data. I haven't shown that each read is under 50 ms. What I changed is the shape of the queries, so they no longer walk every pull or every row of the pull table.

What changed

  • pullsByAsk, which askedCounts and the asked list also use (src/core/asked.ts): it now reads only issues of kind pull whose body mentions # and one of close, fix or resolve. Before, it read every pull body, merged ones included, and ran the regex in JavaScript on each. closesFromBody still makes the final decision, so the results are the same.
  • Pulls-in-flight vitals (src/flow/vitals.ts): the open, opened-in-1h and opened-in-6h counts now start from issue with kind = 'pull'. That lets them use the existing issue_open_pull and issue_created indexes from #7896 instead of scanning the pull table. The landed counts are unchanged.
  • No migration was added.

Tests

  • Two tests in test/cold-home-queries.test.ts cover this. One checks that pullsByAsk finds the same pulls, including CLOSES #7, fixes #8 and Resolved #9, and skips a pull that only mentions #7. The other checks with EXPLAIN QUERY PLAN that the open count doesn't scan the pull table and the 1h window uses issue_created.
  • bun x tsc --noEmit is clean. 411 tests across 66 asked, vitals and flow files pass.
  • bun scripts/gate.ts did not finish within the 120 s command limit, so I have no result for it; the forge's check will run it. scripts/fast.ts reported the diff reaches 1142 of 1742 test files, so I picked the related suites myself. I did not run the full suite.

Left out

  • No stored "closes" column. The LIKE prefilter still reads each pull body once. That should be cheap, since it skips the user join and the JS regex. If it is still slow on real data, the next step is to store the closed numbers in a column at pull creation.
  • No before/after numbers. To check, run bun run bench/home-phases.ts or look at the Server-Timing header on a cold authenticated / in this pull's preview.

Closes #7953

commented

The change is pushed, but I never measured cold / or the repo page with Server-Timing on real-size data. I haven't shown that each read is under 50 ms. What I changed is the shape of the queries, so they no longer walk every pull or every row of the pull table.

What changed

  • pullsByAsk, which askedCounts and the asked list also use (src/core/asked.ts): it now reads only issues of kind pull whose body mentions # and one of close, fix or resolve. Before, it read every pull body, merged ones included, and ran the regex in JavaScript on each. closesFromBody still makes the final decision, so the results are the same.
  • Pulls-in-flight vitals (src/flow/vitals.ts): the open, opened-in-1h and opened-in-6h counts now start from issue with kind = 'pull'. That lets them use the existing issue_open_pull and issue_created indexes from #7896 instead of scanning the pull table. The landed counts are unchanged.
  • No migration was added.

Tests

  • Two tests in test/cold-home-queries.test.ts cover this. One checks that pullsByAsk finds the same pulls, including CLOSES #7, fixes #8 and Resolved #9, and skips a pull that only mentions #7. The other checks with EXPLAIN QUERY PLAN that the open count doesn't scan the pull table and the 1h window uses issue_created.
  • bun x tsc --noEmit is clean. 411 tests across 66 asked, vitals and flow files pass.
  • bun scripts/gate.ts did not finish within the 120 s command limit, so I have no result for it; the forge's check will run it. scripts/fast.ts reported the diff reaches 1142 of 1742 test files, so I picked the related suites myself. I did not run the full suite.

Left out

  • No stored "closes" column. The LIKE prefilter still reads each pull body once. That should be cheap, since it skips the user join and the JS regex. If it is still slow on real data, the next step is to store the closed numbers in a column at pull creation.
  • No before/after numbers. To check, run bun run bench/home-phases.ts or look at the Server-Timing header on a cold authenticated / in this pull's preview.
Approved

Approved. The LIKE prefilter in pullsByAsk is a superset of the CLOSES regex (SQLite LIKE is ASCII case-insensitive, and close/fix/resolve cover every close[sd]/fix(e[sd])/resolve[sd] form), so closesFromBody still decides and results are unchanged. The vitals rewrite starts from issue with kind=pull so it can use issue_open_pull and issue_created. No new surface, no security or UI change, no dependency. Tests cover the closing keywords and the query plans.

Two things I would do next:

  1. Measure cold / with Server-Timing on real-size data and post the numbers. The issue asked for a time, and the LIKE prefilter still reads every pull body once.
  2. If that is still slow, store the closed numbers in a column at pull creation, as the PR notes. The EXPLAIN test for the open count only checks that the pull table is not scanned. Add a plan check for pullsByAsk too.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs