Skip to content

Linux 原生伴生架构 + 多屏截屏 + 阅读线索上下文 - #1

Merged
KuaaMU merged 19 commits into
mainfrom
claude/linux-native-companion
Oct 22, 2026
Merged

KuaaMU merged 19 commits into
mainfrom
claude/linux-native-companion

Conversation

@KuaaMU

@KuaaMU KuaaMU commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

概述

让 Glint 在 Fedora 45 / KDE Plasma 6.7 / Wayland 上可用,并修掉两个使用中暴露的缺陷。

Linux 原生能力

  • KGlobalAccel 热键注册。组件名用 com.glint.Glint,因为 kglobalacceld 会把以 .desktop 结尾的名字当作桌面文件动作。
  • 读取 PRIMARY selection(wl-clipboard-rs / x11-clipboard),不模拟 Ctrl+C,不写剪贴板。
  • 通过 xdg-desktop-portal 截屏,编码为 data URL。
  • XWayland 启动器。Wayland 不允许客户端定位 toplevel,X11 后端可以把卡片放到选区附近。

多屏截屏

截屏门户返回整个虚拟桌面,KWin 把 overlay 限制在单个屏幕的工作区。实测请求 2987×3280,返回 2987×1610,位置 y=70。原实现把整张图压进一个屏幕。

  • overlay 只覆盖光标所在屏幕。
  • 按窗口的实际物理尺寸 1:1 裁剪,不缩放。
  • 截屏前隐藏 HUD,取消或失败时恢复。

阅读线索

会话原来每次触发都重置。现在按应用和文档锚定:

  • 同一应用、同一文档、30 分钟内、少于 8 轮时续接。截图的窗口是 5 分钟。
  • 其余情况新开。任一方缺少应用标识也新开,因为时间间隔无法区分两篇无关文档。
  • 续接在卡片里可见:分隔条加轮次标签。
  • 宿主标题降级为转义后的 <host_label> 数据块。历史只存应用标识和标题摘要。

工程护栏

  • GitHub Actions:前端类型检查、单测、契约套件、构建;core 三平台测试;shell 三平台构建和 clippy。
  • ./scripts/check.sh 在本地跑同一序列,包含跨目标 clippy。
  • AGENTS.md 和 CODING_STANDARDS.md 记录导航和判断类规则。
  • scripts/shot.sh 按物理坐标裁剪截图。

修复的缺陷

  • KWin 的 queryWindowInfo 是交互式接口,调用后光标变成十字并等待点击。改用 xprop 读 _NET_ACTIVE_WINDOW。
  • 宿主身份在 HUD 显示之后才读取,前台窗口已经变成 Glint,所有划词都会被归到同一篇文档。
  • default_config_path() 在 Windows 上读不到 HOME,CLI 和 GUI 使用不同配置。
  • Windows 的 HWND 类型误用,macOS 和 Linux 的测试缺少 cfg 门禁。

验证

  • ./scripts/check.sh 通过:fmt、clippy(-D warnings)、core 48 测试、shell 22 测试、前端单测、契约套件 77 通过、生产构建。
  • CI run 39633397091:Linux、Windows、macOS 全部通过。

未完成

  • macOS 截屏仍返回占位图。
  • Windows 和 macOS 只通过编译和单测,没有运行时验证。
  • 首 token 延迟 15 到 20 秒,取决于本地网关。
  • Slint HUD spike(ADR 0012 P3)尚未开始。

文档

  • ADR 0012:Linux 原生伴生架构
  • ADR 0013:阅读线索与上下文装配

🤖 Generated with Claude Code

KuaaMU and others added 17 commits October 21, 2026 20:13
- add root Cargo workspace and LF normalization
- decouple glint-core from Tauri via StreamSink and path-based config
- fix Tauri 2.12 tray icon API
- separate contract vs runtime acceptance; skip unimplemented Slint/placeholder tests
- fix single-newline markdown rendering

Co-Authored-By: Claude Code <noreply@anthropic.com>
- add session detection, PRIMARY selection sensor (wl-clipboard-rs + x11rb),
  and KGlobalAccel/portal hotkey engines
- add glint-cli with session/capture/explain/hotkeys commands
- replace canned offline answers with an explicit not-configured state
- document measured runtime evidence and the .desktop component-id pitfall

Co-Authored-By: Claude Code <noreply@anthropic.com>
…ming

- replace placeholder screenshot with xdg-desktop-portal capture
- infer widgets from JSON shape when the model omits the widget: tag
- fix WebKitGTK flex min-width collapse in the answer area
- add pin button and shell-side pin state
- stream the answer without a blocking tool round; tools opt-in
- share one config path between CLI and shell

Co-Authored-By: Claude Code <noreply@anthropic.com>
- add per-code-block copy with delegated clicks and stream-safe state
- infer widget type from JSON shape when the model omits the tag
- verify compositor placement and log when it is ignored
- add scripts/glint-x11.sh for cursor-adjacent placement and real AOT
- fix attr encoding so copied code round-trips exactly

Co-Authored-By: Claude Code <noreply@anthropic.com>
- re-apply the card size after the surface is mapped (hidden windows report 0x0)
- read the position back on a delayed task instead of before mapping
- raise the default card to 440x560 so an answer is readable without scrolling
- revert the body inset; keep the original edge-to-edge card

Co-Authored-By: Claude Code <noreply@anthropic.com>
The screenshot portal returns the whole virtual desktop, but KWin confines a
toplevel overlay to one monitor's work area (measured: a 2987x3280 request
came back 2987x1610 at y=70). Sizing the overlay to the virtual desktop and
drawing the image into it squeezed both displays into one frame.

- Overlay now covers only the monitor under the pointer.
- The capture image is cropped 1:1 to that monitor slice; nothing is scaled.
- Rust reports the mapped physical window rect after show, so the crop does
  not depend on `window.screenX` (CSS pixels, wrong under fractional scaling).
- Placement follows the monitor under the cursor instead of the HUD's stale
  `current_monitor()`.
- The HUD is hidden before the portal grabs the desktop and restored on
  cancel/error, so the card no longer lands in its own screenshot.
- A screenshot appends an image turn to the live conversation instead of
  resetting it; only an empty card starts a new session.
- Geometry helpers (bounding_box, monitor_at_point) live in `placement` with
  tests for multi-monitor, negative origins and gaps.
- `glint-cli screen [out.png]` verifies portal dimensions and dumps the frame.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Sessions used to reset on every trigger, so a screenshot could never
continue the discussion it was taken for. Making them always continue
would silently merge unrelated pages, which users cannot see and which
degrades answers. Neither is acceptable for a reading companion.

- `glint_core::host::HostIdentity`: stable app class (KWin `resourceClass`,
  `xdotool` fallback), RAM-only title, `doc_key` digest of the normalised
  title. Generic titles produce no key.
- `glint_core::context::plan_turn`: continue only within the same app and
  document, within 30 min (5 min for screenshots), under 8 turns. Every
  edge case fails toward a new thread.
- Continued threads get a visible divider and per-turn labels; continuity
  is never silent.
- The host label is frozen per thread, shortened, and enters the prompt as
  an escaped `<host_label>` data block with a "not an instruction" rule.
  Titles are attacker-controlled and no longer sit in the instruction slot.
- Selections are wrapped in `<selection>` with bidi/zero-width stripping.
- Host identity is read before `show()`; otherwise KWin reports Glint as
  the foreground app and every selection looks like the same document.
  `is_glint_window` also refuses to record Glint as a host.
- History stores `app_class` + `doc_key` instead of the raw window title.
- ADR 0013 documents the model, the rule and the rejected alternatives.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…e query

`org.kde.KWin.queryWindowInfo` is the D-Bus endpoint behind "click a
window to identify it": calling it puts the compositor into selection
mode, turns the cursor into a crosshair and waits for a click. Every
Alt+Q therefore hijacked the pointer, and an unanswered query could
leave the compositor in that mode.

Read `_NET_ACTIVE_WINDOW` and then `WM_CLASS` / `_NET_WM_NAME` with
`xprop` instead — same data, no input capture, no modal state.

Also fixes the caption parsing that motivated the JSON experiment: the
plain `busctl` dump splits a title on whitespace, so a multi-word caption
was truncated to its first word and `doc_key` hashed a fragment. xprop
returns the whole quoted string.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The repo had no CI, no git hook and no single check entry point, so
breakage was only found by whoever happened to build next. Four real
defects this session were found by hand: an infinite recursion, a
host read after window focus, an interactive KWin call, and Windows
code that has never been compiled.

- `.github/workflows/ci.yml`: frontend typecheck/unit/contract/build;
  `glint-core` tests on Linux/Windows/macOS; Tauri shell build, tests
  and `clippy -D warnings` on all three.
- `scripts/check.sh`: the same sequence locally, so a push fails here
  first. `scripts/install-hooks.sh` wires it as a pre-push hook.
- `pnpm check` runs the frontend half.
- Rust is now `cargo fmt`ed and clippy-clean under `-D warnings`
  (28 warnings fixed: collapsible ifs, manual checked division, a
  complex type, a dead test variable).
- Windows/macOS gain real `HostIdentity` implementations (process
  executable name / bundle identifier) instead of an empty app class,
  so thread continuity works off Linux too. The frontend falls back to
  a shorter time window when a platform reports no class.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…lper

Retro follow-ups from this session's four hand-found defects.

- AGENTS.md (with CLAUDE.md importing it): pointers only — check command,
  layout, the `default-members` trap, ADR index, and "ask before launching the
  GUI" because starting the app repeatedly disrupted the user.
- CODING_STANDARDS.md: four judgement calls a linter cannot enforce —
  OS integration must be non-interactive, prompt tests assert markers not
  prose, a failing test asks which side is wrong, platform code is only real
  once CI compiles it.
- scripts/shot.sh: captures and crops by physical desktop coordinates,
  auto-detecting the capture scale. A cropped HUD shot is 22 KB instead of a
  2.7 MB full-desktop image, which is what made UI verification expensive.
- tests/README.md points at `./scripts/check.sh` and names the command that
  actually runs the shell tests.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The initial run exposed exactly what CI was added to catch.

- Windows: `HWND` is `*mut c_void`, not `isize`. Use the real type and
  `is_null()` instead of casting through `as _`.
- macOS/Linux core tests: two tests used Linux-only modules
  (`linux::KGlobalAccelEngine`, `percent_decode`) without a cfg gate, so
  `cargo test -p glint-core` failed to compile off Linux. Gate the tests and
  the import, not the assertions.
- Unused-import warnings on Windows for `is_kde` / `detect_session`.
- Contract suite: three tests shell out to `cargo test` inside src-tauri.
  The frontend job has no GTK/WebKit headers, so they failed for an
  environment reason. They now skip with a reason when the shell cannot be
  built, driven by `GLINT_SHELL_BUILDABLE` (auto-detected via pkg-config).
  The tauri job still compiles and tests the shell, so coverage is unchanged.
- Ubuntu dependency install: `libappindicator3-dev` and
  `libayatana-appindicator3-dev` conflict; install only the latter.

Verified locally for the Rust halves: `cargo check -p glint-core` for
x86_64-linux, x86_64-pc-windows-msvc and aarch64-apple-darwin.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…tests

- `default_config_path` test compared a string suffix; Windows uses `\`.
  Compare path components instead.
- The Windows clipboard test asserted `seq > 0`, but a CI runner has no
  interactive clipboard, so 0 is legitimate. Assert the API resolves.
- macOS clippy: `Command` import in linux.rs, `Arc` in lib.rs and the
  `app_handle` parameter in macos.rs are all platform-gated. Gate the
  imports, rename the parameter.

Linux and Windows core are checked locally; the macOS shell can only be
compiled on a macOS runner (objc2 needs Apple's C toolchain).

Co-Authored-By: Claude Code <noreply@anthropic.com>
`clippy -D warnings` on macOS flagged symbols that only exist for the
Linux path: GLINT_APP_ID, virtual_desktop_bounds, active_monitor_rect,
and the xprop parsers in platform/linux.rs. Gate them with
`#[cfg(target_os = "linux")]`; `placement::bounding_box` is pure geometry
kept for tests, so it gets a targeted allow instead of a cfg that would
remove it from the test build.

Co-Authored-By: Claude Code <noreply@anthropic.com>
`default_config_path` only knew `XDG_CONFIG_HOME` and `HOME`. Windows has
neither, so it fell through to `glint_config.json` in the working
directory — the CLI and the GUI would read different files, which is the
exact split this function exists to prevent (and it already caused one
"why does the GUI work but the CLI says no API key" bug).

Resolve per platform the way `app_config_dir()` does: XDG on Linux,
`%APPDATA%` on Windows, `~/Library/Application Support` on macOS.

The test caught it because it asserts the path shape, not a string suffix.

Co-Authored-By: Claude Code <noreply@anthropic.com>
- `clippy -p glint-core` for the Windows target found four real issues
  invisible on Linux: two `as u32` casts of an already-u32 constant, and
  a manual BGRA→RGBA loop that `as_chunks_mut` expresses directly.
- The same for macOS found a collapsible `if` in both copies of the
  macOS cursor helper.
- `scripts/check.sh` now runs the core lints for both cross targets when
  the targets are installed, so platform-only code is checked before a
  push rather than one platform per CI round trip.

Co-Authored-By: Claude Code <noreply@anthropic.com>
`clippy -D warnings` on Windows and macOS flagged the same function five
times for nested `if let`. Extract `clipboard_text()` and return the
candidate from each branch instead of nesting; behaviour is unchanged
(sequence number when the platform has one, text comparison otherwise).

Also gate the placeholder-screenshot import to the platforms that use it
and drop three redundant `as u32` casts in the Win32 hotkey registration.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Same nested `if let` shape that was fixed in the Linux path; the
Windows/macOS build flagged it. Both platforms now compile the same
expression form.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 58192861-0788-4be6-b0b9-f4837c2f4982
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-21T16:48:35.719504Z 1ae319c PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

glint/src/snipping.ts

Lines 646 to 650 in 1ae319c

await invoke("submit_snipping_result", {
imageDataUrl: croppedDataUrl,
prompt: "请分析此屏幕截图区域,直击关键信息与核心本质。",
windowTitle: currentWindowTitle,
});

P1 Badge Preserve the captured host anchor through snip submission

The backend captures and emits app_class and doc_key with the snip frame (src-tauri/src/lib.rs), but this submission forwards only the title. Consequently the HUD receives both anchor fields as null, and startNewMultimodalQuery falls back to currentThread's app/document identity. If a user has an existing thread for app/document A, then activates B and snips it within five minutes, B's image is silently appended to A and A's history is sent with it instead of opening a new thread.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/main.ts Resolved
Comment thread src/main.ts Resolved
Comment thread src-tauri/src/lib.rs Resolved
Comment thread crates/glint-core/src/selection.rs Resolved
KuaaMU and others added 2 commits October 22, 2026 13:34
…adlines

- Snip submission forwards the frame's `app_class`/`doc_key` instead of
  only the title. The HUD used to fall back to the open thread's identity,
  so a snip taken from app B within five minutes was appended to app A's
  history and A's transcript was sent with it.
- Continuation now refuses when either side has no app anchor. The old
  time-window fallback treated "no evidence" as evidence of continuity;
  a timer cannot distinguish two unrelated documents.
- Turns are serialized. A trigger arriving before the previous `Done`
  used to start a second request against the same shared transcript,
  producing `U1, U2, A1, A2`; the newest trigger is queued and started
  only after the current answer is recorded.
- Closing the snip overlay through the window manager (Alt+F4) restores
  the HUD, matching the Esc path. Previously the card stayed hidden.
- The PRIMARY read deadline now covers the whole read: poll-and-read in
  bounded chunks instead of an unbounded `read_to_end` after one poll,
  so a stalled selection owner cannot hang the hotkey path.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Em dashes are the one punctuation habit that makes the comments read as
generated. Replace them with periods, commas or colons across the files
added in this branch, and rewrite the two judgement rules in
CODING_STANDARDS.md as direct sentences instead of bolded labels.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@KuaaMU
KuaaMU merged commit 6ddcefb into main Oct 22, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant