R&D proposal: object commands ("pick up the red block") and a chat agent, with model options and safety #569

Proposal (steward, 2026-10-25). Owner, 2026-10-22: "it would be cool if there was an agent in chat too… I could describe ideas and it goes… there are objects there. It's hard to grab honestly, but do you think you could turn them into commands?" Status: research and proposal only. Nothing is built until the owner picks the options below.

1. Object commands ("pick up the red block, put it in the cup")

How: the overhead camera ("Above") plus the table calibration (#439/#492) map each pixel to a point on the table in mm. The flow:

1. Snapshot: grab a frame from the overhead camera.
2. Detect: a vision model returns [{name, box}].
3. Pick: convert the box centre to table mm, then plan hover → descend → close the gripper → lift using the existing pose/move API and the inverse kinematics already in serve.
4. Place: the same hover and descend, then open the gripper.

Safety, all of it existing machinery:

• Every command runs inside a normal turn, never beside one.
• The safety zone and the governor (180°/s) apply.
• STOP wins.
• The gripper's settled counts as holding.
• A failed grasp re-snapshots once, then says so plainly.

Build steps, each one a pull with tests:

• (a) Pixel ↔ table mapping, with a click-to-point test: click a spot on the overhead view and the arm hovers there. That alone is a big precision win.
• (b) Primitives: pick(x,y) and place(x,y) as server verbs, tested on the simulated bus.
• (c) Detection plus naming the objects. The page offers the found objects as buttons ("red block", "cup") before any free text.
• (d) Free text, mapped onto those primitives.

2. The chat agent ("describe an idea and it goes")

Two roles, split on purpose:

• Arm agent (in the app): turns words into the primitives above, shows a plan ("I'll pick the red block at (120, −40) mm and drop it in the cup"), and runs it only after a tap on Go. It can't move anything the UI couldn't.
• Forge agent (the feedback door, forge #4758): "the slider is laggy on my phone" becomes a filed issue with a screenshot, so the crew builds it. It never edits code live.

3. Model options (owner's pick)

Recommendation: an Anthropic key with a hard monthly cap ($20), used only by the arm agent. The crew's subscription isn't touched. Run step (a) first, since it needs no model at all.

4. Check-ins

One line in the steward report each day while it's in flight. A demo GIF on this issue after (a) and after (c).

Owner decisions:

1. Go or no-go on step (a).
2. Which model.
3. Whether visitors (not just the owner) may use object commands.

Parked by the crew: the run ended without a pull: declined: issue #569 is a research proposal, and it says nothing is built until the owner picks the options. It has no comments and no owner decision. The three decisions it lists (go or no-go on step (a), whic

Parked by the crew: the run ended without a pull: declined: Issue #569 is a research proposal, and it says "Nothing is built until the owner picks the options below." Nobody has made those decisions yet. Its only comment is the crew's earlier parking note, whi

Parked twice since a person last spoke here, so it is the planner's now. Every reason:

1. the run ended without a pull: declined: issue #569 is a research proposal, and it says nothing is built until the owner picks the options. It has no comments and no owner decision. The three decisions it lists (go or no-go on step (a), whic
2. the run ended without a pull: declined: Issue #569 is a research proposal, and it says "Nothing is built until the owner picks the options below." Nobody has made those decisions yet. Its only comment is the crew's earlier parking note, whi

The planner splits it, replaces it with a fresh issue, or defers it; a person can also say what to try. Nothing starts here until then, and the board lists it.

Planner, 2026-10-25. State: waiting on the owner. I could not split, replace or defer it.

• Why not split or replace: every piece needs the owner's go first. Any issue I opened would be handed to a builder, who would decline it for the same reason, a third time.
• Why not defer: the forge refused (no_later_gate). This repository has no open "Later gate: …" issue, and only the owner or a writer can open one.
• Why not drop: it is the owner's ask, so it is not mine to drop.

What unsticks it, either one:

1. @trevato answers the three decisions here: go or no-go on step (a) (pixel ↔ table click-to-hover, no model needed), which model, and whether visitors may use it. Then I split the chosen steps into single-pull issues.
2. Or a person opens a "Later gate: …" issue. Then I defer this behind it and it leaves the builders' board.

Planner, again: still waiting on the owner, with nothing new since my comment above. No owner answer and no Later gate, so I cannot defer it. Any issue I split out would be declined by a builder for the third time. Either one unsticks it: the owner answers the three decisions, or a person opens a "Later gate: …" issue. Until then this comes back to the planner unchanged on every run.

Planner: unchanged, still waiting on the owner (see my comment above). It needs the owner's three decisions, or a "Later gate: …" issue to defer it behind.

Planner, 2026-10-25: left open, still waiting on the owner. None of the four states fits:

• Split or replace: every piece is new scope that the owner has not yet approved. #86 forbids new scope, and this issue says "nothing is built until the owner picks". Any issue I opened would be declined a third time.
• Defer: refused, because there is no open "Later gate: …" issue.
• Close as blocked, which #86 allows: the forge refuses it (invalid_reason). It only accepts split, dropped or done.
• Drop: it is the owner's ask, so it is not mine to drop.

To unstick it, @trevato does one of these: answer the three decisions (go/no-go on step (a), the model, visitors), open a "Later gate: …" issue, or close this one.

Planner: unchanged and left open, waiting on the owner. The reasons are in my comment above, and the forge refused a close as blocked (invalid_reason). It moves when @trevato answers the three decisions, opens a "Later gate: …" issue, or closes it.

Planner: unchanged and left open, waiting on the owner (see above).

Planner: unchanged and left open, waiting on the owner (see above).

Planner: unchanged and left open, waiting on the owner (see above).

Planner, 2026-10-26. State: left open, waiting on the owner. I could not put it in any of the four states.

• Split or replace: every step needs the owner's go first, and #86 allows no new scope. A builder would decline any piece for a third time. My token also gets forbidden on every close here today, so I could not close this as split either.
• Defer: there is still no open "Later gate: …" issue.
• Drop: it is the owner's ask.

Either of these unsticks it: @trevato answers the three decisions (go on step (a), which model, whether visitors may use it), or a person opens a "Later gate: …" issue, or closes this one.

Planner, 2026-10-26 (again): left open, waiting on the owner, unchanged from my comment above. There is still no owner answer and no "Later gate: …" issue. My token still gets forbidden on any close, so it cannot be split or replaced.

Planner, 2026-10-27. State: left open, waiting on the owner. I could not put it in any of the four states.

• Split or replace: every step needs the owner's go, and #86 allows no new scope. Also, every close with my token still comes back forbidden (I tried today on #582 and #192), so a split could not be finished anyway.
• Defer: there is still no open "Later gate: …" issue.
• Drop: it is the owner's ask.

Any one of these unsticks it: @trevato answers the three decisions (go on step (a), which model, whether visitors may use it), opens a "Later gate: …" issue, or closes this one.

Planner, 2026-10-27 (run 2): unchanged and left open, waiting on the owner. There is still no owner answer to the three decisions, no open "Later gate: …" issue to defer it behind, and every close with my token still comes back forbidden (I tried #582 again today). So it cannot be split or replaced. It moves when @trevato answers here, opens a Later gate, or closes it.

Planner, 2026-10-27 (run 3): unchanged and left open, waiting on the owner. There is still no answer to the three decisions and no "Later gate: …" issue, and my closes are refused with forbidden. So I cannot split, replace or defer it.

Planner, 2026-10-27 (run 4): unchanged and left open, waiting on the owner. Every piece of it needs the owner's go, and this direction forbids new scope, so I am not splitting it or replacing it with build issues a builder would decline a third time. There is no "Later gate: …" issue to defer it behind. It moves when @trevato answers the three decisions or closes it.

Planner, 2026-10-27 (run 5): left open, waiting on the owner. I tried each state again:

• Split or replace: every step needs the owner's go, and #86 allows no new scope.
• Defer: there is still no open "Later gate: …" issue.
• Drop: it is the owner's ask.
• Close: every close with my token is still refused with forbidden (tried just now on #365, #192, #508 and #582).

It moves when @trevato does one of these: answers the three decisions (go on step (a), which model, whether visitors may use it), opens a "Later gate: …" issue, or closes this one.

Planner, 2026-10-27 (run 6): left open, waiting on the owner. Nothing has changed since my last comment: there is no answer to the three decisions and no "Later gate: …" issue, and #86 forbids new scope. @trevato: answer here, open a Later gate, or close this.

Planner (run 7): left open, waiting on the owner. Unchanged; see my comment from run 5.

Planner (run 8): left open, waiting on the owner. Unchanged; see my comment from run 5.

Planner (run 9): left open, waiting on the owner. Nothing has changed since run 5. There is still no "Later gate: …" issue to defer it behind, and #86 forbids new scope, so I cannot split it or replace it. It is the owner's ask, so I cannot drop it either. @trevato: answer the three decisions, open a Later gate, or close this.

Planner (run 10): left open, waiting on the owner. Unchanged since run 9: no owner answer, no Later gate, and no new scope allowed under #86.

Planner (run 11): left open, waiting on the owner. Unchanged; see run 9.

Planner (run 12): left open, waiting on the owner. Unchanged; see run 9.

Planner (run 13): left open, waiting on the owner. Unchanged; see run 9.

Planner (run 14): left open, waiting on the owner. Unchanged; see run 9.

Planner (run 15): left open, waiting on the owner. Unchanged; see run 9.

Planner (run 16): left open, waiting on the owner. Nothing has changed since run 9. There is no answer to the three decisions, there is still no "Later gate: …" issue, and #86 allows no new scope. Every close with my token is still refused with forbidden (I tried again today on #508, #365 and #192), so a split or a replace could not be finished anyway. @trevato: answer the three decisions, open a Later gate, or close this.

Planner (run 17): left open, waiting on the owner. Unchanged since run 16: there is no answer to the three decisions, no "Later gate: …" issue to defer it behind, and #86 allows no new scope.

Planner (run 18): left open, waiting on the owner. Unchanged since run 16.

Planner (run 19): left open, waiting on the owner. Unchanged since run 16.

Planner, 2026-10-28: left open, waiting on the owner. Same as run 16. There is no answer to the three decisions and no "Later gate: …" issue to defer it behind. #86 allows no new scope, so I am not splitting it. I tried closing again just now and every close with my token still comes back forbidden, so a split or a replace could not be finished anyway. It is the owner's ask, so it is not mine to drop. @trevato: answer the three decisions, open a Later gate, or close this.