From ca4a6a5c492f0b9abe65dc7383ce377ddbe812bb Mon Sep 17 00:00:00 2001 From: Ling Test Date: Mon, 26 Oct 2026 23:47:02 +0800 Subject: [PATCH 01/15] =?UTF-8?q?docs(design):=20add=20UX=20gap=20roadmap?= =?UTF-8?q?=20=E2=80=94=2013=20competitive=20gap=20design=20docs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Competitive comparison vs Codex Desktop / MiniMax / ZCode / WorkBuddy. Each doc covers compatibility analysis, upgrade paths, and a phased implementation plan with acceptance criteria. Signing/auto-update and hosted model access deliberately excluded at roadmap review. --- ...ux-gap-roadmap-01-document-deliverables.md | 110 +++++++++++++++++ .../ux-gap-roadmap-02-gen-ui-artifacts.md | 97 +++++++++++++++ ...roadmap-03-notifications-automations-p3.md | 102 +++++++++++++++ .../ux-gap-roadmap-04-im-integrations.md | 116 ++++++++++++++++++ .../ux-gap-roadmap-05-built-in-browser.md | 93 ++++++++++++++ ...x-gap-roadmap-06-computer-use-hardening.md | 93 ++++++++++++++ .../ux-gap-roadmap-07-memory-citations.md | 91 ++++++++++++++ .../ux-gap-roadmap-08-parallel-work-board.md | 84 +++++++++++++ .../ux-gap-roadmap-09-cloud-tasks-sync.md | 65 ++++++++++ .../ux-gap-roadmap-10-skill-marketplace.md | 88 +++++++++++++ docs/design/ux-gap-roadmap-11-voice-input.md | 90 ++++++++++++++ .../ux-gap-roadmap-12-spreadsheet-viewer.md | 89 ++++++++++++++ .../ux-gap-roadmap-13-startup-diagnostics.md | 90 ++++++++++++++ docs/design/ux-gap-roadmap.md | 47 +++++++ 14 files changed, 1255 insertions(+) create mode 100644 docs/design/ux-gap-roadmap-01-document-deliverables.md create mode 100644 docs/design/ux-gap-roadmap-02-gen-ui-artifacts.md create mode 100644 docs/design/ux-gap-roadmap-03-notifications-automations-p3.md create mode 100644 docs/design/ux-gap-roadmap-04-im-integrations.md create mode 100644 docs/design/ux-gap-roadmap-05-built-in-browser.md create mode 100644 docs/design/ux-gap-roadmap-06-computer-use-hardening.md create mode 100644 docs/design/ux-gap-roadmap-07-memory-citations.md create mode 100644 docs/design/ux-gap-roadmap-08-parallel-work-board.md create mode 100644 docs/design/ux-gap-roadmap-09-cloud-tasks-sync.md create mode 100644 docs/design/ux-gap-roadmap-10-skill-marketplace.md create mode 100644 docs/design/ux-gap-roadmap-11-voice-input.md create mode 100644 docs/design/ux-gap-roadmap-12-spreadsheet-viewer.md create mode 100644 docs/design/ux-gap-roadmap-13-startup-diagnostics.md create mode 100644 docs/design/ux-gap-roadmap.md diff --git a/docs/design/ux-gap-roadmap-01-document-deliverables.md b/docs/design/ux-gap-roadmap-01-document-deliverables.md new file mode 100644 index 000000000..564f5ee27 --- /dev/null +++ b/docs/design/ux-gap-roadmap-01-document-deliverables.md @@ -0,0 +1,110 @@ +# Document Deliverables — PPT / Word / Excel Generation Loop + +## Status + +Status: design approved for implementation +Owner: Desktop / Web / Daemon +Last verified: 2026-10-26 + +## Background and competitive reference + +MiniMax Code and WorkBuddy both win users on "give a task, receive a file": +presentations, cleaned spreadsheets, formatted reports. Ling today can operate +Excel/WPS through the desktop Office driver (Windows) and can preview docx/pdf +text in the desktop right dock, but it has no story for *producing* office +documents as deliverables, and no pptx/xlsx viewer at all. + +## Goal + +A user asks a teammate or channel to "make a deck / draft a report / build a +workbook", and receives a real `.pptx` / `.docx` / `.xlsx` artifact they can +open, preview in Ling, and hand off. + +## Non-goals + +- Pixel-perfect WYSIWYG editors inside Ling. Generation is agent-driven; + editing stays in the native app. +- Cloud rendering. Everything is produced locally by the daemon. + +## Design + +### 1. Generation path: sandboxed node toolrunner, not the agent's shell + +Add a daemon-side deliverable builder under +`packages/daemon/src/features/deliverables/`: + +- Input: a normalized JSON spec (`DeckSpec`, `DocSpec`, `WorkbookSpec`) or a + markdown outline the agent produced. +- Builders: `pptxgenjs` (pptx), `docx` (docx), `exceljs` (xlsx). All three are + pure-JS, license-compatible, and already mainstream in Electron apps. +- Output: written to the workspace artifacts directory and registered through + the existing artifact registry (`packages/daemon/src/features/automations/artifacts.ts` + pattern), producing a `RunArtifact` with `type: "file"`, + `contentType` `application/vnd.openxmlformats-*`. + +The agent invokes this through one new harness tool +`deliverable_create` (bounded input schema, approval-gated like other tools), +rather than free-form npm/script execution. This reuses the approval and +authority model instead of creating a parallel one. + +### 2. Spec schemas live in `packages/contracts/src/deliverables.ts` + +```ts +export interface DeckSpec { title: string; slides: DeckSlide[]; } +export interface DeckSlide { layout: "title"|"bullets"|"image"|"two_column"; title?: string; bullets?: string[]; imageRef?: string; notes?: string; } +export interface WorkbookSpec { sheets: { name: string; rows: (string|number|null)[][]; formulas?: string[] }[]; } +export interface DocSpec { heading: string; blocks: DocBlock[]; } // paragraphs, headings, tables, lists +``` + +Versioned (`specVersion: 1`) so future renderers can be added without breaking +stored specs. + +### 3. Preview: extend `PreviewContentKind` + +`packages/contracts/src/artifacts.ts` gains `"pptx" | "docx" | "xlsx"` kinds. +Rendering strategy, desktop-first: + +- pptx/docx: mammoth-style text+structure extraction already exists for docx + (`apps/desktop/main/local-files/preview.ts` uses `mammoth`); add pptx text + extraction (`pptx` unzip + slide XML parse — no new heavy dep, `fflate` is + already a web devDependency and works server-side). +- xlsx: covered by the spreadsheet viewer design + ([ux-gap-roadmap-12-spreadsheet-viewer.md](ux-gap-roadmap-12-spreadsheet-viewer.md)) — + this item only wires the content kind. +- Web (no desktop): text-structure fallback preview, same as today's docx + fallback, plus "open in system app" affordance on desktop. + +### 4. Compatibility analysis + +| Existing surface | Impact | Handling | +|---|---|---| +| `ArtifactType` union | additive | New `"file"` artifacts unchanged; only `contentType` distinguishes office docs. No union change needed. | +| `PreviewContentKind` union | additive | Add 3 kinds; all existing switch statements audited + exhaustive default kept (`unsupported`), so old renderers degrade safely. | +| Office driver (Windows Excel/WPS control) | orthogonal | Driver *operates* apps; deliverables *create files*. No shared code path; the driver can later open a generated file but that is not required for V1. | +| Skill library | integration | Ships as a built-in skill pack (`deliverables`) so the behavior is discoverable and evolvable without daemon changes. | +| Approvals | reuse | `deliverable_create` is a normal tool: `ask | smart | always` authorization applies unchanged. | + +### 5. Incompatible-case upgrade path + +If a future spec change breaks v1 (e.g. speaker notes become blocks), the +stored spec JSON keeps `specVersion`; builders ship versioned builders +(`buildDeckV1`), and the registry stores the *built file*, never the live spec. +Old artifacts therefore never re-render; nothing to migrate. + +## Implementation plan + +1. `packages/contracts`: `deliverables.ts` schemas + zod-style validation used by daemon. +2. `packages/daemon/src/features/deliverables/`: builders, route + `POST /deliverables/build` (daemon-auth scoped), artifact registration. +3. Harness tool registration + approval wiring; slash intent `/deliverable`. +4. Preview kinds in daemon `previews` feature + desktop renderer + web fallback. +5. Built-in skill pack with 2 recipes (report→docx, outline→deck). +6. Tests: unit (builders produce valid zip/OOXML, round-trip open via parser), + mock-daemon Playwright spec producing and previewing a deck, screenshot evidence. + +## Acceptance + +- Agent can create a 5-slide deck from a channel instruction; file appears in + artifacts and previews with slide titles. +- Existing artifact/preview suites stay green; new Playwright spec captures + before/after screenshots under `LING_CAPTURE_UI2_EVIDENCE=1`. diff --git a/docs/design/ux-gap-roadmap-02-gen-ui-artifacts.md b/docs/design/ux-gap-roadmap-02-gen-ui-artifacts.md new file mode 100644 index 000000000..f394eae74 --- /dev/null +++ b/docs/design/ux-gap-roadmap-02-gen-ui-artifacts.md @@ -0,0 +1,97 @@ +# Interactive HTML Artifacts (Gen UI) + +## Status + +Status: design approved for implementation +Owner: Web / Contracts / Daemon +Last verified: 2026-10-26 + +## Background and competitive reference + +ZCode renders assistant-produced HTML as inline interactive views (charts, +simulators, tools) that the user can click through next to the conversation. +Ling's artifact previews are static (`markdown`, `code`, `image`, `diff`, ...). +A large class of useful outputs — data visualizations, small calculators, +interactive mockups — currently arrive as code the user must save and open. + +## Goal + +A conversation or run can produce a self-contained HTML artifact that the user +opens in a sandboxed, interactive preview inside Ling (right dock / artifact +viewer), without it ever getting script access to Ling's APIs or the daemon. + +## Design + +### 1. Contract: `html_interactive` preview kind + +`packages/contracts/src/artifacts.ts`: + +```ts +export interface PreviewHtmlInteractive { + readonly html: string; // full document, self-contained + readonly sandbox: readonly string[]; // fixed allowlist, see below +} +``` + +`PreviewContentKind` gains `"html_interactive"`. `PreviewContentResponse` +gains `htmlInteractive?: PreviewHtmlInteractive`. + +### 2. Rendering: iframe sandbox, zero trust + +- Rendered in an `