Skip to content

Native chat: inline visuals for Grok, OpenCode and OMP chats - #26530

Merged
brennanb2025 merged 3 commits into
mainfrom
brennanb2025/query-native-chat-visuals
Oct 8, 2026
Merged

brennanb2025 merged 3 commits into
mainfrom
brennanb2025/query-native-chat-visuals

Conversation

@brennanb2025

@brennanb2025 brennanb2025 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 4 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​245 $\color{#cf222e}{\Huge{\mathbf{−}}}$​3 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​242
Prod 6 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​254 $\color{#cf222e}{\Huge{\mathbf{−}}}$​19 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​235

ELI5

Native chat can show charts, diagrams and mockups inside an agent's reply, but only Claude and Codex chats knew how. Grok, OpenCode and OMP chats never learned the feature existed, so they never drew one. This teaches those three agents too.

What Changed

Before: an inline visual could only appear in a Claude or Codex chat. A Grok, OpenCode or OMP chat had no visuals folder and no instructions, so it answered in plain text even when a chart would help.

After: a new Grok, OpenCode (2.x) or OMP chat gets the same visuals instructions (the orca-chat-visuals skill) and its own visuals folder, and the chat view already draws what it writes. The existing Inline visuals switch in Chat settings turns it off for these agents too.

Mechanism. The protocol these agents speak (ACP) has no way to hand an agent a skill, so each agent's launch description now says how that agent loads one for a single launch, without touching the user's own agent settings:

  • Grok: grok agent --plugin-dir <Orca's skill plugin> stdio, the per-process flag Grok documents for this. Grok has no supported-version list in Orca, so when visuals are on, the launch asks for its --version first. Releases before 1.0.44 (the oldest I could check) start without visuals instead of failing on an unknown flag.
  • OpenCode 2.x: Orca's skill folder is added to skills.paths in the inline config variable, keeping any inline config the user already set. 2.x adds that list to the one in the user's config files.
  • OMP: omp acp --plugin-dir <Orca's skill plugin>, the same per-process flag and plugin layout as Grok. OMP adds the plugin beside the user's own installed plugins and skill folders and writes nothing to disk for it, so the user's global, profile, project and extra config files apply exactly as they set them. No extra version check: every OMP release Orca runs chats on (17.0.5 and later) has the flag.
  • Every step is best effort: if an agent can't load the skill, the chat starts without visuals, and an inherited ORCA_CHAT_VISUALS_DIR is stripped so the agent never writes into another chat's folder.

Code: src/main/acp/acp-visuals-skill.ts (per-agent loaders), src/main/acp/acp-structured-launch-visuals.ts and acp-structured-launch-resolution.ts (launch wiring), src/main/acp/acp-launch-specs.ts (each agent's loader).

Why

ACP has no channel for skills, so loading has to happen through each agent's own startup options. The alternative, installing the skill into the user's real agent folders (~/.grok/skills and the like), would change their setup for agents outside Orca too. Per-launch options leave their setup untouched.

Differences from Claude and Codex chats:

  • OpenCode 1.x gets no visuals (intended). On 1.x the inline list replaces the user's own skills.paths (checked on 1.18.31), so their own skills would vanish from Orca chats.
  • No pre-granted write access to the visuals folder (temporary). Codex chats get the folder as a writable root. Unless full access is on, these agents will probably ask permission the first time they write a page. Pre-allowing it means merging into each agent's permission rules, which could override the user's own; that needs its own change.
  • OMP lists the skill as a /skill:orca-chat-visuals command (intended for now). OMP ignores the field Claude uses to hide it.
  • OMP 18.1.5 to 18.1.8 get no visuals unless the user turned on OMP's Claude skill sources (intended). Those releases skip skills from --plugin-dir by default; 18.1.9 fixed it. They were current for about a day and a half, so there's no version-range check, only a code comment.
  • Grok in its shared "leader" mode gets no visuals (temporary). Grok ignores --plugin-dir there. A per-session way to hand Grok a plugin exists, but using it needs new plumbing in the session start/resume requests; follow-up.

Linked Issue

N/A (internal follow-up to #26099 / #26358).

Visual Proof

Hidden Orca window at this branch's head, chats answered by a local stand-in model (no sign-in needed) so each step is repeatable. The stand-in draws a chart only if the agent was actually given the orca-chat-visuals skill, and writes it with the agent's own shell command.

OpenCode, Inline visuals on: the chart renders inline in the reply.

OpenCode chat with the chart drawn inline

OpenCode, Inline visuals off: the agent isn't given the skill, so it answers in plain text.

OpenCode chat with Inline visuals off answering in plain text

OMP, default permissions: writing the page asks for approval like any other shell command (the card shows the command, not an Orca folder path); after "Allow once" the chart renders.

OMP permission card for writing the chart
OMP chat with the chart drawn inline

Grok: no signed-in account was used. The chat's Grok process was started as grok agent --always-approve --plugin-dir <Orca>/resources/native-chat-visuals stdio.

Testing

  • New unit tests: src/main/acp/acp-visuals-skill.test.ts, src/main/acp/acp-structured-launch-visuals.test.ts; updated Grok args and OMP environment tests.

  • src/main/acp, structured-agent runtime and visuals tests: 434 passing under Node. pnpm tc:node passes; orca-ci-checks passes 22/22.

  • Against the real CLIs, each in an isolated scratch home with no credentials:

    • OMP 17.0.5: with the flag, the skill is offered and in the model's system prompt on a new chat and on a resumed one (session/load), and the visuals folder reaches OMP's shell. Skill folders the user set in a project .omp/config.yml, an OMP profile, or their own extra config files are all still listed beside it. OMP writes nothing extra to its home.
    • OpenCode 2.0.14: lists orca-chat-visuals, and the user's own skills.paths skills still show.
    • OpenCode 1.18.31: loads it but drops the user's own paths, hence the 2.x-only gate.
    • Grok 1.0.46: accepts --plugin-dir and discovers the plugin (skills=1).
  • In the app (macOS, hidden window, stand-in model): OpenCode and OMP chats drew the chart inline; with Inline visuals off, OpenCode got no skill and answered in plain text; Grok's process carried --plugin-dir.

  • Not verified: a real model choosing to draw a visual; a Grok chat drawing one (needs a signed-in Grok); denying the write (OpenCode didn't ask by default, so there was nothing to deny); the chart still showing after an app restart; Windows, Linux and SSH (code reading only).

  • I manually tested these changes locally

  • Automated tests added/updated, or explained why not below

Review

Reviewed in two rounds plus a final readiness pass; summary in the PR comments.

Agent skill upstream boundary

  • Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

  • SSH: structured chats for these agents run on the host that owns the workspace (local or a remote Orca server), and the skill and folder are resolved on that host. WSL chats are refused for these agents today.
  • Windows: Grok gets an argv entry and OMP uses the platform path separator in PI_CONFIG_FILES; neither path goes through a shell.

Checklist

  • This PR is small and focused
  • I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
  • Before/after screenshots or videos attached for UI changes, or N/A with reason
  • Self-reviewed for correctness, security, and performance
  • Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
  • pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

Each ACP agent loads the chat-visuals skill its own way for one launch,
without touching the user's agent config: Grok through agent
--plugin-dir, OpenCode 2.x through inline skills.paths, OMP through a
PI_CONFIG_FILES overlay that keeps the user's own skill folders. A chat
whose agent can't load it starts without visuals.
OMP's settings layers replace lists, so the PI_CONFIG_FILES overlay hid
a repo's or profile's own skill folders in Orca's OMP chats, and its
files were never removed. omp acp takes --plugin-dir like Grok, which
adds the skill beside the user's own plugins and skill folders and
leaves their config untouched.
@brennanb2025

Copy link
Copy Markdown
Contributor Author

Review summary

The problem. Inline visuals (charts, diagrams and mockups drawn inside an agent's reply) worked only in Claude and Codex chats. Grok, OpenCode and OMP chats in Chat UI were never given the visuals instructions or a folder to write into, so they always answered in plain text.

What changes for users. New and resumed Grok, OpenCode (2.x) and OMP chats in Chat UI can now draw inline visuals, the same as Claude and Codex chats. The existing Inline visuals switch in Chat settings turns it off for them too. Terminal-based chats are unchanged; they're out of scope here.

Fixed during review

  • OMP chats lost the user's own skills (high severity, from the first version of this PR). OMP's config files replace a list rather than add to it. The first version wrote a config file listing the user's skill folders plus Orca's, but it rebuilt "the user's folders" from only some of the places OMP reads. A repo's own .omp/config.yml skills and an OMP profile's skills therefore disappeared from Orca's OMP chats. Two reviewers reproduced this with the real OMP CLI. The fix drops the config file: OMP now gets Orca's skill through its --plugin-dir flag, which adds a plugin without touching any setting. Rechecked with the real OMP: project, profile, user and installed-plugin skills all stay, Orca's skill appears beside them on new and resumed chats, and nothing is written to OMP's folders. This also removed a second problem: those config files were never deleted.
  • Code comment (low). OMP 18.1.5 to 18.1.8 accept --plugin-dir but skip its skills by default. The comment now says so instead of claiming every release loads it.

Deferred

  • Grok's opt-in shared "leader" mode ignores --plugin-dir, so those chats quietly get no visuals. Grok has a per-session way to pass a plugin, but using it needs new plumbing in the session start and resume requests. Follow-up.
  • OMP 18.1.5 to 18.1.8 get no visuals unless the user turned on OMP's Claude skill sources. Those releases were current for about a day and a half, so there's no version-range check.
  • No pre-granted write access to the visuals folder. In default permission mode, OMP asks before the write like any other shell command. OpenCode's defaults let it through.
  • With visuals on, each Grok launch runs grok --version first (about 20 ms). OpenCode's session start is about 0.7 s slower with the extra skills folder; measured on a heavily loaded machine.
  • Small cleanups: an OpenCode helper sits in src/main/acp/ rather than src/main/opencode/, and an OMP launch comment lost its explanation of how permissions are answered.

Found while testing, not caused by this PR

  • On a heavily loaded machine, a new OpenCode chat can fail with "Chat could not be started." OpenCode 2.0.14 gives up after 5 seconds while building its model list for a new session. This reproduces with main's exact launch settings. Retrying the first session request once, or logging the error details (Orca's log currently shows [Object]), would be a separate fix.
  • When an OpenCode or OMP chat can't reach its model provider, the chat can end with "Done" and no reply or error.

Verified

  • Two review rounds: an adversarial review of the whole change, an independent design check that arrived at the same overall approach, and a fresh review of the fix. Then a final readiness pass covering remote servers, crash risks, security, performance, compatibility and cross-platform code: nothing above low severity.
  • Real CLIs in throwaway homes with no sign-in: OMP 17.0.5, OpenCode 2.0.14 and 1.18.31, and Grok 1.0.46.
  • In the app on macOS (hidden window, local stand-in model): OpenCode and OMP chats drew the chart inline, Inline visuals off removed the skill, and Grok's process carried --plugin-dir. Screenshots are in the PR description.
  • CI on 658634f93b2: every check that ran passed (static analysis and typecheck, all test shards, packaging for macOS and Windows).

Not verified

  • A real model deciding on its own to draw a visual. The stand-in model checks only that the skill was offered, then follows its instructions.
  • A Grok chat drawing a visual, which needs a signed-in Grok.
  • What happens when the user denies the write.
  • The chart still showing after an app restart. This path is shared with Claude and Codex and unchanged here.
  • Windows, Linux and SSH remote servers, which were checked by reading code only.

@brennanb2025
brennanb2025 marked this pull request as ready for review October 8, 2026 11:36
@coderabbitai

coderabbitai Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1bdd5201-5e27-42af-b61b-9bec355f0929
📥 Commits

Reviewing files that changed from the base of the PR and between 8ae5264 and 658634f.

📒 Files selected for processing (10)
  • src/main/acp/acp-launch-specs.ts
  • src/main/acp/acp-structured-adapter.test-support.ts
  • src/main/acp/acp-structured-launch-resolution-omp.test.ts
  • src/main/acp/acp-structured-launch-resolution.ts
  • src/main/acp/acp-structured-launch-visuals.test.ts
  • src/main/acp/acp-structured-launch-visuals.ts
  • src/main/acp/acp-structured-session-adapter.test.ts
  • src/main/acp/acp-visuals-skill.test.ts
  • src/main/acp/acp-visuals-skill.ts
  • src/main/runtime/structured-agent-runtime-registrations.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 7 remain after this review.


📝 Walkthrough

Walkthrough

ACP launch specs now support agent-specific visuals-skill loaders and plugin-directory arguments. Launch resolution prepares visuals, probes versions when needed, loads skills, and applies the resulting environment settings. Runtime registration passes native chat visuals configuration to ACP launch resolution. Tests cover loader behavior, launch arguments, environment handling, and failure cases.

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 65863

No identified issue blocks merging. If visuals preparation fails, the chat can still launch without visuals.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 12 functions across 10 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: enabling native inline visuals for Grok, OpenCode, and OMP chats.
Description check ✅ Passed The description is complete and directly matches the pull request. It includes the user-facing behavior, implementation details, rationale, linked issue context, visual proof, testing results, limitat…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@brennanb2025
brennanb2025 merged commit 7bdb0b6 into main Oct 8, 2026
59 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant