Repository navigation
Native chat: add an inline visuals switch and simplify the skill #26358
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
5500830
e7bd0c7
17c5d8e
9d6a0be
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,58 +1,33 @@ | ||
| --- | ||
| name: orca-chat-visuals | ||
| description: Show a chart, diagram, timeline, comparison or UI mockup inline in this Orca chat by writing one self-contained HTML file and adding a visual line to your reply. Use when seeing something would explain the answer faster than prose or a small Markdown table, and only when the ORCA_CHAT_VISUALS_DIR environment variable is set. | ||
| description: Show a chart, diagram, table, report or mockup inline in this Orca chat as an HTML | ||
| page. Use when a visual would help the user understand your answer. Only available when the | ||
| ORCA_CHAT_VISUALS_DIR environment variable is set. | ||
| user-invocable: false | ||
| --- | ||
|
|
||
| # Inline visuals in an Orca chat | ||
|
|
||
| Orca can render an HTML page inside your reply. Reach for it when the reader gains from seeing the | ||
| answer: trends, many values side by side, flows, architecture, layouts. Answers that read fine as | ||
| text, a short list or a small table don't need one. | ||
|
|
||
| ## When you can't make one | ||
|
|
||
| - This chat's folder is the absolute path in the `ORCA_CHAT_VISUALS_DIR` environment variable. Read | ||
| it from your shell (`printenv ORCA_CHAT_VISUALS_DIR`; in PowerShell `$env:ORCA_CHAT_VISUALS_DIR`). | ||
| - If the variable is unset or empty, or writing there fails, don't make a visual. | ||
| - Stay inside what the user allowed. In plan or read-only mode don't make a visual. If a write | ||
| there is refused, don't retry it; never switch modes or ask for wider access for a visual. | ||
| - Never put visuals in the user's project and never edit `.gitignore` or other files for them. | ||
|
|
||
| Without a visual, use a Markdown table, a Mermaid code block or plain prose instead. | ||
|
|
||
| ## Writing the file | ||
|
|
||
| - One visual is one complete HTML document saved directly in that folder (no subfolders). | ||
| - Every visual gets a file name not used before in this chat: a few lowercase words and a short | ||
| random suffix, like `latency-by-region-7c1e.html`. Never overwrite or edit an earlier visual; | ||
| write a new file. Allowed: letters, digits, `.`, `_` and `-`; starts with a letter or digit; | ||
| ends in `.html`; at most 128 characters; no `..`; not a Windows device name like `con` or `nul`. | ||
| - Put everything in the file: inline `<style>` and `<script>`, data embedded. Keep it under 512 KB. | ||
| - Scripts, styles and fonts may load only from `https://cdn.jsdelivr.net`, `https://unpkg.com`, | ||
| `https://cdnjs.cloudflare.com`, `https://esm.sh`, `https://fonts.googleapis.com` and | ||
| `https://fonts.gstatic.com`. Pin library versions. | ||
| - The page can't make requests: no `fetch`, `XMLHttpRequest`, `WebSocket` or `EventSource`. | ||
| Images must be inline SVG or `data:` URLs. No forms, popups, nested frames or navigation. | ||
| - Fit the width you're given (fluid layout, `max-width: 100%`). Don't size anything to the viewport | ||
| height (`100vh`, `height: 100%` on the page): Orca sizes the frame to your content. Aim for well | ||
| under 1000px tall. | ||
| - Match the user's theme with Orca's CSS variables: `--background`, `--foreground`, `--muted`, | ||
| `--muted-foreground`, `--border`, `--primary`, `--accent`, and `--chart-1` to `--chart-5` for data | ||
| series. Leave the page background transparent. When the user switches theme the values change | ||
| and `<html>` gains or loses the `dark` class, with no reload and no event: use the variables from | ||
| CSS or SVG, and redraw a canvas chart when that class changes (a `MutationObserver` on `<html>`). | ||
| - Use real text for labels and give the page a heading so it reads without color alone. | ||
| Orca can show an HTML page inside your reply. Use one whenever a visual would make the answer | ||
| clearer. | ||
|
|
||
| ## Writing the page | ||
|
|
||
| - Save it in the folder named by `ORCA_CHAT_VISUALS_DIR` (`printenv ORCA_CHAT_VISUALS_DIR`; in | ||
| PowerShell `$env:ORCA_CHAT_VISUALS_DIR`). If it's unset or you can't write there, skip the visual. | ||
| - One complete HTML file per visual, directly in that folder, under 512 KB, with a new file name | ||
| each time (e.g. `latency-by-region-7c1e.html`): letters, digits, `.`, `_`, `-`, ending in `.html`. | ||
| - Inline your styles, scripts and data. Libraries and fonts may load only from cdn.jsdelivr.net, | ||
| unpkg.com, cdnjs.cloudflare.com, esm.sh, fonts.googleapis.com and fonts.gstatic.com. The page | ||
| can't make network requests; use inline SVG or `data:` URLs for images. | ||
| - Don't size anything to the viewport height (`100vh`); Orca fits the frame to your content. | ||
| - Optional: Orca's theme colors are available as CSS variables (`--background`, `--foreground`, | ||
| `--muted-foreground`, `--border`, `--primary`, `--chart-1` to `--chart-5`). | ||
|
|
||
| ## Showing it | ||
|
|
||
| Only after the file exists, add this line to your reply on its own line, at the top level (not in | ||
| a code block, list, table or quote): | ||
| After the file exists, put this on its own line in your reply, outside any code block: | ||
|
|
||
| ::orca-visual{file="latency-by-region-7c1e.html" title="Latency by region"} | ||
|
|
||
| - `file` is the bare file name, never a path. `title` is optional plain text, at most 120 | ||
| characters, without `"` or `\`. | ||
| - At most 8 visuals in one reply. | ||
| - Don't announce the visual or walk through it point by point. Do state the takeaway in a sentence, | ||
| so the reply still makes sense where the visual can't be shown. | ||
| `file` is the bare file name, never a path; `title` is optional. At most 8 per reply. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -39,21 +39,24 @@ export async function resolveClaudeLaunchFlags( | |
| deps.cliFlags?.supports(CLAUDE_THINKING_DISPLAY_FLAG, launch, budgetMs, startProbe) ?? false | ||
| // Unlike the thinking display, a missed answer costs the chat its skill for its whole life, so | ||
| // the plugin check waits for the probe up to its own kill time: bounded, and instant once known. | ||
| const preparedVisuals = deps.prepareVisuals?.(sessionId) ?? Promise.resolve(null) | ||
| const [thinksFirst, visuals, loadsPlugins] = await Promise.all([ | ||
| thinking(), | ||
| deps.prepareVisuals?.(sessionId) ?? null, | ||
| deps.prepareVisuals | ||
| ? (deps.cliFlags?.supports( | ||
| CLAUDE_PLUGIN_DIR_FLAG, | ||
| launch, | ||
| CLAUDE_CLI_FLAG_PROBE_KILL_AFTER_MS | ||
| ) ?? false) | ||
| : false | ||
| preparedVisuals, | ||
| preparedVisuals.then((visuals) => | ||
| visuals | ||
| ? (deps.cliFlags?.supports( | ||
| CLAUDE_PLUGIN_DIR_FLAG, | ||
| launch, | ||
| CLAUDE_CLI_FLAG_PROBE_KILL_AFTER_MS | ||
| ) ?? false) | ||
| : false | ||
| ) | ||
| ]) | ||
| // The plugin check may have outwaited the thinking budget and learned the version meanwhile; a | ||
| // probe that gave none is not asked again here. | ||
| const thinks = | ||
| thinksFirst || (deps.prepareVisuals ? await thinking(THINKING_RECHECK_BUDGET_MS, false) : false) | ||
| thinksFirst || (visuals ? await thinking(THINKING_RECHECK_BUDGET_MS, false) : false) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🔎 Supported by static analysis🏁 Script executed: #!/bin/bash
# Inspect the supported probe lifecycle and its timeout behavior.
ast-grep outline src/main/claude/claude-cli-flag-support.ts --items all --match 'supports|probe'
rg -n -C 8 'supports\(|startProbe|budgetMs|CLAUDE_CLI_FLAG_PROBE_KILL_AFTER_MS' src/main/claude/claude-cli-flag-support.tsRepository: stablyai/orca Length of output: 4814 Keep the thinking recheck independent of visuals. If the first thinking check expires while a cold CLI-version probe is still running, a launch with visuals disabled skips the bounded recheck. The launch can therefore omit readable thinking even when the probe finishes before the recheck budget ends. Keep the recheck available without starting a visuals plugin probe. |
||
| return { | ||
| thinkingDisplayArgs: thinks ? THINKING_DISPLAY_ARGS : {}, | ||
| visuals: visuals ? { visuals, pluginDir: loadsPlugins ? visuals.skill.pluginDir : null } : null | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,154 @@ | ||
| import { existsSync, mkdtempSync, rmSync } from 'node:fs' | ||
| import { tmpdir } from 'node:os' | ||
| import { join } from 'node:path' | ||
| import { afterEach, describe, expect, it, vi } from 'vitest' | ||
| import { LOCAL_EXECUTION_HOST_ID } from '../../shared/execution-host' | ||
| import { createClaudeStructuredLaunchResolver } from '../claude/claude-structured-launch-resolution' | ||
| import { CLAUDE_PLUGIN_DIR_FLAG, type ClaudeCliFlag } from '../claude/claude-cli-flag-support' | ||
| import { createCodexStructuredLaunchResolver } from '../codex/codex-structured-launch-resolution' | ||
| import { | ||
| adapterFor, | ||
| fakeCodex, | ||
| identityFor | ||
| } from '../codex/codex-structured-session-adapter-fixture' | ||
| import { agentSessionRecordFixture } from './agent-session-record-test-fixture' | ||
| import { | ||
| createNativeChatVisualsDelivery, | ||
| NATIVE_CHAT_VISUALS_DIR_ENV | ||
| } from './native-chat-visuals-delivery' | ||
| import { nativeChatVisualsFolderFor } from './native-chat-visuals-folder' | ||
|
|
||
| const SKILL = { pluginDir: '/app/plugin', skillsRoot: '/app/plugin/skills' } | ||
| const MANUAL = { approvalPolicy: 'on-request', sandbox: 'workspace-write' } as const | ||
| const scratch: string[] = [] | ||
| afterEach(() => { | ||
| for (const dir of scratch.splice(0)) { | ||
| rmSync(dir, { recursive: true, force: true }) | ||
| } | ||
| }) | ||
|
|
||
| function delivery(isEnabled: () => boolean) { | ||
| const stateDirectory = mkdtempSync(join(tmpdir(), 'orca-visuals-setting-')) | ||
| scratch.push(stateDirectory) | ||
| const prepareVisuals = createNativeChatVisualsDelivery({ | ||
| stateDirectory, | ||
| logger: { warn: vi.fn(), error: vi.fn() }, | ||
| isEnabled, | ||
| resolveSkill: async () => SKILL | ||
| }) | ||
| return { stateDirectory, prepareVisuals } | ||
| } | ||
|
|
||
| describe('inline visuals launch preference', () => { | ||
| it('starts Claude without visuals while preserving configured plugins, user folders and attachment access', async () => { | ||
| const sessionId = 'claude-disabled' | ||
| const { stateDirectory, prepareVisuals } = delivery(() => false) | ||
| const supports = vi.fn(async (_flag: ClaudeCliFlag) => true) | ||
| const record = agentSessionRecordFixture({ sessionId }) | ||
| const launch = await createClaudeStructuredLaunchResolver({ | ||
| store: { getRecord: () => record, pinLaunchDirectory: vi.fn() }, | ||
| resolveWorkspacePath: async () => '/repo', | ||
| resolveCommand: () => '/bin/claude', | ||
| resolveAuthPolicy: () => ({ stripAuthEnv: false }), | ||
| resolveLaunchArgs: () => ['--add-dir', '/user/notes', '--plugin-dir', '/user/plugin'], | ||
| resolveEnv: () => ({ [NATIVE_CHAT_VISUALS_DIR_ENV]: '/inherited', KEEP: 'yes' }), | ||
| resolveInheritedEnv: async () => ({ PATH: '/bin' }), | ||
| attachmentDirectory: '/state/agent-session-attachments', | ||
| hasTranscript: async () => false, | ||
| cliFlags: { supports }, | ||
| prepareVisuals | ||
| })({ identity: { ...identityFor(sessionId), agent: 'claude', providerHandle: null } }) | ||
| expect(launch.options).not.toHaveProperty('plugins') | ||
| expect(launch.options.extraArgs).toHaveProperty('plugin-dir', '/user/plugin') | ||
| expect(launch.options.additionalDirectories).toEqual([ | ||
| '/user/notes', | ||
| '/state/agent-session-attachments' | ||
| ]) | ||
| expect(launch.env).not.toHaveProperty(NATIVE_CHAT_VISUALS_DIR_ENV) | ||
| expect(launch.env?.KEEP).toBe('yes') | ||
| expect(supports.mock.calls.map(([flag]) => flag)).not.toContain(CLAUDE_PLUGIN_DIR_FLAG) | ||
| expect(existsSync(nativeChatVisualsFolderFor(stateDirectory, sessionId))).toBe(false) | ||
| }) | ||
|
|
||
| it('gives new Codex chats the current preference without changing another live chat', async () => { | ||
| let enabled = true | ||
| const { stateDirectory, prepareVisuals } = delivery(() => enabled) | ||
| const codex = fakeCodex({ | ||
| 'config/read': () => ({ | ||
| config: { sandbox_workspace_write: { writable_roots: ['/user/notes'] } } | ||
| }) | ||
| }) | ||
| const resolveLaunch = createCodexStructuredLaunchResolver({ | ||
| store: { | ||
| getRecord: (sessionId) => | ||
| agentSessionRecordFixture({ | ||
| sessionId, | ||
| provider: 'codex', | ||
| accountHome: { variable: 'CODEX_HOME', path: '/home/codex' } | ||
| }), | ||
| pinLaunchDirectory: vi.fn() | ||
| }, | ||
| resolveWorkspacePath: async () => '/repo', | ||
| resolveCommand: () => '/bin/codex', | ||
| resolveLaunchArgs: () => [], | ||
| resolvePermissionPolicy: () => MANUAL, | ||
| resolveEnvironment: async () => ({ | ||
| [NATIVE_CHAT_VISUALS_DIR_ENV]: '/inherited', | ||
| KEEP: 'yes' | ||
| }), | ||
| prepareVisuals | ||
| }) | ||
| const adapter = adapterFor(codex, {}, [], { resolveLaunch }) | ||
| const start = (sessionId: string) => | ||
| adapter.acquire({ | ||
| identity: { | ||
| ...identityFor(sessionId), | ||
| hostId: LOCAL_EXECUTION_HOST_ID, | ||
| providerHandle: null | ||
| }, | ||
| fence: 1, | ||
| spawnToken: sessionId | ||
| }) | ||
| try { | ||
| await start('first') | ||
| const first = codex.connections[0]! | ||
| const firstCalls = [...first.calls] | ||
| const firstFolder = nativeChatVisualsFolderFor(stateDirectory, 'first') | ||
| expect(first.launch.env?.[NATIVE_CHAT_VISUALS_DIR_ENV]).toBe(firstFolder) | ||
| expect(first.calls.find(({ method }) => method === 'thread/start')?.params).toMatchObject({ | ||
| config: { 'sandbox_workspace_write.writable_roots': ['/user/notes', firstFolder] } | ||
| }) | ||
|
|
||
| enabled = false | ||
| await start('disabled') | ||
| const disabled = codex.connections[1]! | ||
| expect(disabled.launch.env).not.toHaveProperty(NATIVE_CHAT_VISUALS_DIR_ENV) | ||
| expect(disabled.launch.env?.KEEP).toBe('yes') | ||
| expect(disabled.calls.map(({ method }) => method)).not.toContain('skills/extraRoots/set') | ||
| const openIndex = disabled.calls.findIndex(({ method }) => method === 'thread/start') | ||
| expect(disabled.calls.slice(0, openIndex).map(({ method }) => method)).not.toContain( | ||
| 'config/read' | ||
| ) | ||
| expect( | ||
| disabled.calls.find(({ method }) => method === 'thread/start')?.params | ||
| ).not.toHaveProperty('config') | ||
| expect(existsSync(nativeChatVisualsFolderFor(stateDirectory, 'disabled'))).toBe(false) | ||
| expect(first.closed).toBe(false) | ||
| expect(first.calls).toEqual(firstCalls) | ||
|
|
||
| enabled = true | ||
| await start('reenabled') | ||
| const reenabled = codex.connections[2]! | ||
| expect( | ||
| reenabled.calls.find(({ method }) => method === 'skills/extraRoots/set')?.params | ||
| ).toEqual({ extraRoots: [SKILL.skillsRoot] }) | ||
| expect(reenabled.launch.env?.[NATIVE_CHAT_VISUALS_DIR_ENV]).toBe( | ||
| nativeChatVisualsFolderFor(stateDirectory, 'reenabled') | ||
| ) | ||
| expect(first.closed).toBe(false) | ||
| expect(first.calls).toEqual(firstCalls) | ||
| } finally { | ||
| await adapter.closeAll() | ||
| } | ||
| }) | ||
| }) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🔎 Supported by static analysis
🏁 Script executed:
Repository: stablyai/orca
Length of output: 27215
🏁 Script executed:
Repository: stablyai/orca
Length of output: 41934
🏁 Script executed:
Repository: stablyai/orca
Length of output: 41817
🏁 Script executed:
Repository: stablyai/orca
Length of output: 42465
🏁 Script executed:
Repository: stablyai/orca
Length of output: 42461
Document the filename constraints enforced by the visual validator.
The current guidance permits filenames longer than 128 characters, containing
.., or using Windows device names.isNativeChatVisualFileNamerejects these names, soparseNativeChatVisualDirectiveLinereturnsnulland the visual is not mounted.Suggested fix
📝 Committable suggestion