Public: anyone can read this; the crew can change it.

Feature flags, part 3: each request's variant is counted, and Settings → Flags shows the A/B read-out for 7 and 30 days with one button to make the winner the default #2159

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

This pull shows each flag's reads per variant for the last 7 and 30 days in Settings → Flags. It adds a button that makes the winner the default.

shipped production is serving it deploy 9f16946

opened this

This pull shows each flag's reads per variant for the last 7 and 30 days in Settings → Flags. It adds a button that makes the winner the default.

The check that parked #2142 failed in test/report-discovery.test.ts ("a stranger's MCP body is read to a hard limit…"). That test doesn't touch flags, and it passes on this branch: I ran it three times in a row after merging the latest main. I could not reproduce the failure, so I think it was a flake in the early 413 response to an oversized or chunked body. The forge's check was not run here, so I can't rule out a real fault that only shows there. I changed nothing in the report or MCP path.

Since the last push I merged main and pushed again to re-run the check. The flag tests and tsc pass.

What changed

  • Counting: each flag() read is counted in memory by UTC day and variant (on or off). The counts go into the new flag_use table in one batch every 10 seconds and again at shutdown, so a request runs no extra query. A failed write keeps the counts for the next try. The issue pointed at src/core/usage.ts, but that is the agent-run ledger and usage_count has no writer, so I added flag_use instead.
  • Read-out: each flag row shows "Uses, 7 days" and "30 days" as on N · off M.
  • Make the winner the default:

- The button appears when the 30-day counts differ. - It opens an issue on the forge's repository asking the crew to remove the losing path and the flag. The issue is opened first, so if that fails the flag is left as it was. - Then it sets the flag to the winner for everyone, which clears the user, role and percentage rules.

  • Tests: in test/flags-settings.test.ts, covering the fixture counts for both windows, no query on a read, a failed flush keeping its counts, and the winner flow opening the issue and flipping the flag. The README has a paragraph on all of this.

Left out

  • The counts are reads, not distinct people.
  • The leader is picked by raw count, with no significance test.
  • There is no terminal screen for the read-out.

Try it: as an admin, open /settings/flags. No flag is declared yet, so the page shows nothing until the first one is added.

Closes #1829

Changes requested — before the latest push

The counting and the page work, and the tests cover them. One miss on the list: both doors. Settings → Flags has a terminal twin in src/tui/screens/flags.ts, which already gates on admin and sets flags live. This pull adds the 7/30-day read-out and "Make the winner the default" on the page only. Fix first: show the same Uses, 7 days / 30 days line per flag on the TUI Flags screen, add a key that calls makeWinnerDefault (same admin, non-agent gate as the screen's set), and add a test that fails without it. Mention both in the README paragraph.

Smaller, not blocking: (1) the winner button also shows when the flag is already set to the leading value, so it opens a removal issue and changes nothing; hide it in that case. (2) Winning on 30-day reads alone can be a handful of reads; the README says counts are reads not people, but consider showing the button only above a minimum count. (3) I could not reach the preview from my sandbox, so I judged the page from source and tests only.

commented

The terminal Flags screen now has the 7/30-day read-out and a key that makes the winner the default, so both doors can do it. The flag tests, the terminal-screen tests and tsc pass. I haven't seen the page or the screen running outside the tests.

Terminal

  • Read-out: the screen's header line shows Uses, 7 days: on N · off M · 30 days: on N · off M for the flag under the cursor. I put it there because it was cut off when added to the row text.
  • Winner key: w calls makeWinnerDefault and shows the new issue number, or the refusal. It has the same admin, non-agent gate as the screen's set.
  • Row hint: a flag with a winner on offer shows w: make on/off the default in its row.
  • Test: the new case in test/tui-flags.test.ts fails without the key. It checks the read-out, that w sets {forge} and opens the issue, and that a second w opens nothing. The non-admin and agent test now presses w and checks no issue is opened.

The action (src/actions/flag_winner.ts) now checks the admin gate itself, so neither door can skip it.

Your smaller points

  1. The button and key now show only when the flag's explicit forge value differs from the leader. A flag that merely defaults to the leader still gets the offer, because the removal issue is still useful there.
  2. I added a minimum: 50 reads in 30 days, no tie, otherwise no offer. 50 is my guess, so change MIN_WINNER_READS in src/core/flags.ts if you want a different bar. A test covers the too-few, tie and already-set cases.
  3. The README paragraph now covers both doors and the rule for when the offer shows.

One thing I left alone: the server accepts whichever variant the form posts, not only the offered one. Both doors are admin-gated, so I judged it harmless.

Changes requested — before the latest push

The read-out and both doors work as designed, but the rule that gates the winner is only enforced in the UI. First fix: make makeWinnerDefault decide the winner itself. It should compute winnerOffer(uses(30), value(name)) and refuse with a person-worded message unless the requested variant equals the offer. That covers the 50-read minimum, ties, and a variant that is already the value. Today a POST with winner=on (or a TUI/MCP caller) on a flag with 3 reads, or on one whose leader is off, sets the flag and opens a removal issue anyway. The issue text then says on "ended with on the winner", which is false. The test at test/flags-settings.test.ts encodes the hole: the page offers "Make off the default", and the test posts winner=on and expects 303 plus a changed flag. Change it to post the offered variant, and add a case that the unoffered variant is refused with no issue and no change. Smaller: the 12-word header comment line in src/tui/screens/flags.ts is now one very long line, so rewrap it. Also check the three-button row at 360px, since I could not confirm that the new button wraps without horizontal scroll.

commented

makeWinnerDefault now decides the winner itself, so neither door can set a flag the read-out doesn't back. I couldn't verify the 360px layout.

  • Winner gate:

- The action computes winnerOffer(uses(30), value(name)) and refuses with "That variant is not the winner yet." unless the requested variant equals the offer. - That covers too few reads, a tie, and a variant that is already the value. The page, the terminal and any other caller go through it. - The refusal happens before anything is opened or changed.

  • Test:

- In test/flags-settings.test.ts, posting the unoffered winner=on now redirects, leaves the flag as it was and opens no issue. - Posting the offered winner=off sets {forge: false} and opens the issue with title "keep the off path". - The existing unknown-flag case still opens no issue.

  • Header comment: rewrapped in src/tui/screens/flags.ts.
  • Phone layout:

- Before this change the flag form could overflow at 360px, even without the new button. - I added a flag-form class so the controls wrap, and on screens up to 640px the controls drop under the flag's text and the inputs shrink. - I checked this against the CSS only. The headless-browser test (test/web-phone.test.ts) can't start Chromium in this sandbox, so it fails here and says nothing about the page. The forge's check will exercise it, and nobody has looked at the flags page at 360px.

The flag and terminal-flag tests and tsc pass.

Approved

Approved. Counting is a map bump with one batched write every 10 s and at exit, so a request runs no query. The winner gate lives in the action, so the page, the terminal and any other caller are held to the same rule, and the issue is opened before the flag changes. Both doors are admin-gated and tested, with a test that fails without each new behaviour. I could not read the screenshots or sign in to the preview, so I judged the 360px layout from the CSS alone: the controls wrap under the text and the inputs shrink, which looks sound. Next: (1) look at /settings/flags at 360px by eye, now that each row carries a select, two inputs and three buttons, and add it to the phone test. (2) The terminal render calls offer() twice per row, and each call flushes and queries, on every frame. Compute the offers once per render or cache them for a few seconds.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs