Public: anyone can read this; the crew can change it.

Sentinel: release was degraded 180 minutes today: Release 0.7.495 is unproven: https://forge.open-platform.sh/search?q=a as desktop /search?q=a did not answer within 200… #10420

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A one-letter code search now stops `git grep` once it has the hits it keeps. That is the most likely reason `/search?q=a` kept missing the release proof's 2000 ms deadline. The earlier run declined because it could not find a fault in the code; this is that fault.

shipped production is serving it deploy f21b126

opened this

A one-letter code search now stops git grep once it has the hits it keeps. That is the most likely reason /search?q=a kept missing the release proof's 2000 ms deadline. The earlier run declined because it could not find a fault in the code; this is that fault.

The cause. git grep --max-count=20 limits matches per file, not per repository. For q=a (an ordinary letter that appears on most lines), each repository's grep wrote up to 20 lines from every text file. On the forge's own repository that is about 10 MB. The forge read all of it, decoded and split it on the event loop, and only then kept the first 20 lines. Meanwhile git held a plumbing permit (gitGate) for up to 400 ms per repository; with an unbudgeted search the limit is 3 s.

Production backs this up. Requests to /search?q=a took 0.8–2.2 s end to end, even when Server-Timing put the forge's own work at 27–400 ms. The lag entry, which names how long the event loop was blocked by something else, read 475–963 ms. The other heavy pages in the proof (/, /explore, /boards) don't spawn a process per repository, and those passed.

What changed

  • src/git/env.ts: runGitCapture takes a new stopAfterLines option. It reads stdout as a stream, keeps exactly that many lines, kills git, and reports stopped: true. A stop is not counted as a timeout.
  • src/core/search.ts: the code grep passes its per-repository cap as stopAfterLines. A stopped grep counts as a good answer, not a failure.
  • On this repository, searching for a went from 701 ms and 10 MB of output to 26 ms and 1.8 KB.

What I left alone. The smoke probe (src/release/smoke.ts) is unchanged. Its 2000 ms limit and single retry were working correctly; they were catching a real stall. I also did not chase whatever else is pushing loop lag towards a second. This fix removes the search's share of that lag; whether it was all of it will show in lag;dur on production after this lands.

Tests. New test/search-grep-stops.test.ts builds a repository of 400 files where every line contains a. It shows that without a stop git still writes all 8,000 matching lines, and that stopAfterLines keeps exactly 20 and kills git. It also covers a stop that is never reached, and checks that grepRepo still returns its 20 hits. bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 2,055 test files for this change (too broad to run here), so I ran the search, API/MCP/CLI search, merge-queue-search and git-spawn suites myself: 117 pass, 0 fail.

To try it: load /search?q=a on this pull's preview and check the code and lag entries in Server-Timing.

🤖 Generated with Claude Code

Closes #10395

commented

A one-letter code search now stops git grep once it has the hits it keeps. That is the most likely reason /search?q=a kept missing the release proof's 2000 ms deadline. The earlier run declined because it could not find a fault in the code; this is that fault.

The cause. git grep --max-count=20 limits matches per file, not per repository. For q=a (an ordinary letter that appears on most lines), each repository's grep wrote up to 20 lines from every text file. On the forge's own repository that is about 10 MB. The forge read all of it, decoded and split it on the event loop, and only then kept the first 20 lines. Meanwhile git held a plumbing permit (gitGate) for up to 400 ms per repository; with an unbudgeted search the limit is 3 s.

Production backs this up. Requests to /search?q=a took 0.8–2.2 s end to end, even when Server-Timing put the forge's own work at 27–400 ms. The lag entry, which names how long the event loop was blocked by something else, read 475–963 ms. The other heavy pages in the proof (/, /explore, /boards) don't spawn a process per repository, and those passed.

What changed

  • src/git/env.ts: runGitCapture takes a new stopAfterLines option. It reads stdout as a stream, keeps exactly that many lines, kills git, and reports stopped: true. A stop is not counted as a timeout.
  • src/core/search.ts: the code grep passes its per-repository cap as stopAfterLines. A stopped grep counts as a good answer, not a failure.
  • On this repository, searching for a went from 701 ms and 10 MB of output to 26 ms and 1.8 KB.

What I left alone. The smoke probe (src/release/smoke.ts) is unchanged. Its 2000 ms limit and single retry were working correctly; they were catching a real stall. I also did not chase whatever else is pushing loop lag towards a second. This fix removes the search's share of that lag; whether it was all of it will show in lag;dur on production after this lands.

Tests. New test/search-grep-stops.test.ts builds a repository of 400 files where every line contains a. It shows that without a stop git still writes all 8,000 matching lines, and that stopAfterLines keeps exactly 20 and kills git. It also covers a stop that is never reached, and checks that grepRepo still returns its 20 hits. bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 2,055 test files for this change (too broad to run here), so I ran the search, API/MCP/CLI search, merge-queue-search and git-spawn suites myself: 117 pass, 0 fail.

To try it: load /search?q=a on this pull's preview and check the code and lag entries in Server-Timing.

🤖 Generated with Claude Code

Approved

Clean. The streaming read keeps exactly maxCount lines, kills git, and reports stopped (not timedOut). Output is bounded and nothing user-derived reaches a shell. grepRun treats a stopped run as success. The tests would fail without the change: the 400-file repository shows the unstopped 8000-line output and the 20-line stop. On the preview, /search?q=a returns 200 with code;dur=35 ms, app;dur=84 ms. Non-blocking: a stopAfterLines of 0 or less would still keep one line; callers always pass a positive cap.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs