Skip to content

feat(agent): add remote helper design doc and NDJSON proto types - #1913

Closed
nitrobass24 wants to merge 6 commits into
developfrom
feat/remote-helper-design
Closed

feat(agent): add remote helper design doc and NDJSON proto types#1913
nitrobass24 wants to merge 6 commits into
developfrom
feat/remote-helper-design

Conversation

@nitrobass24

@nitrobass24 nitrobass24 commented May 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds condensed remote helper design doc covering architecture, Backend interface, wire protocol, path safety, SSH credentials, schema, and phased implementation plan
  • Adds pkg/agent/proto — shared NDJSON wire types for qui ↔ qui-helper communication

Proto package

  • Command, Result, HelloBanner envelopes
  • All op request/response payload types (stat, lstat, walk, statfs, readdir, samefs, mkdir, remove, tree ops, cancel)
  • Leaf package — zero internal/ imports (CI-enforceable)

Test plan

  • go build ./pkg/agent/proto/... — clean
  • go test -race -count=1 ./pkg/agent/proto/... — 38 tests pass
  • go list -deps ./pkg/agent/proto/... | grep -c 'qui/internal' — 0 (no internal imports)

Summary by CodeRabbit

  • Documentation

    • Added a comprehensive design spec for Remote Helper support: architecture, deployment, SSH credential model, security/path-safety rules, audit/logging, instance configuration, and phased rollout guidance.
  • New Features

    • Defined a stable NDJSON-over-SSH helper protocol and complete filesystem RPC contracts (stat/lstat/walk/readdir/statfs/samefs/mkdir/remove, tree hardlink/reflink flows, control and diagnostic ops) for remote filesystem operations.
  • Tests

    • Added JSON round-trip tests validating protocol message serialization, streaming semantics, error fields, and NDJSON framing.

Review Change Stack

Condensed design doc (428 lines) replacing the 3,500-line whitepaper
from PR #1820. Covers architecture, Backend interface, wire protocol,
path safety, SSH credentials, schema, helper binary, and phased
implementation plan (8 PRs).

pkg/agent/proto is a leaf package (zero internal/ imports) defining
the shared NDJSON wire types for qui <-> qui-helper communication:
Command/Result envelopes, HelloBanner, and all op request/response
payloads.
@coderabbitai

coderabbitai Bot commented May 19, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Adds a Remote Helper design doc, NDJSON-over-SSH protocol types/constants, filesystem operation request/response contracts, and comprehensive JSON round-trip tests for helper↔host communication.

Changes

Remote Helper Protocol and Design

Layer / File(s) Summary
Remote Helper Design Specification
documentation/design/remote-helper.md
Complete architecture specification covering system overview, filesystem RPC surface, NDJSON wire protocol (Command/Result/HelloBanner envelopes, streaming semantics, error codes), SSH credential and deployment flow with TOFU host-key capture and cross-compilation, database schema changes, SSH connection pooling, API endpoints, frontend UI changes for Local/Remote helper selection, and a phased implementation roadmap.
Wire Protocol Core Types and Constants
pkg/agent/proto/proto.go
Foundation structures for NDJSON messaging: Command (request correlation via RequestID), Result (responses with optional error/payload/streaming-frame state), HelloBanner (session handshake with versioning/capabilities/platform metadata), operation name constants (OpStat, OpWalk, OpTreeHardlink, etc.), IsStreamingOp discriminator function, and stable error code constants for version-independent error handling.
Filesystem Operation Contracts
pkg/agent/proto/ops.go
Request/response data types for all filesystem operations: metadata queries (Stat, Lstat, Walk with streaming, ReadDir, Statfs, SameFS), mutations (Mkdir, Remove/RemoveAll with ignore-list support), atomic tree operations (TreeHardlink, TreeReflink, TreeRemove) using hardlinktree.TreePlan with rollback semantics, plus control (Cancel) and diagnostic (Echo) messages.
Protocol Message Tests
pkg/agent/proto/proto_test.go
Test suite validating JSON serialization round-trips for all protocol types: generic roundTrip[T] helper, coverage for Command (omitempty deadline behavior), Result (error/streaming state preservation), HelloBanner, all filesystem operation types, tree operations with rollback/diagnostic fields, streaming truncation, per-entry errors, zero-value field omission, and NDJSON single-line format enforcement.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested labels

documentation, tests

Suggested reviewers

  • s0up4200
  • buroa

Poem

🐰 I hopped across bytes and SSH tide,

NDJSON lanes where helpers bide.
Contracts, frames, and banners bright,
Tests keep every path polite.
A rabbit cheers — protocol takes flight.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 5.71% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding a remote helper design document and NDJSON protocol types to the agent package.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/remote-helper-design

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@nitrobass24 nitrobass24 self-assigned this May 19, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@documentation/design/remote-helper.md`:
- Line 9: Add explicit language tags to the fenced code blocks shown (e.g. the
block containing "qui host                                seedbox" should be
```text, the block with the long ssh command beginning
`command="/home/user/.config/qui-helper/qui-helper serve --stdio ...` should be
```bash, and the block with "POST   /api/instances/{id}/ssh-test         -- test
SSH credentials, return host key fingerprint" should be ```text) so each fenced
block has a language label and avoids MD040 markdownlint warnings.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: e9a504da-75c7-4655-a3d8-3c19cf4e6672

📥 Commits

Reviewing files that changed from the base of the PR and between 5bcf648 and e3408d0.

📒 Files selected for processing (4)
  • documentation/design/remote-helper.md
  • pkg/agent/proto/ops.go
  • pkg/agent/proto/proto.go
  • pkg/agent/proto/proto_test.go

Comment thread documentation/design/remote-helper.md
@nitrobass24 nitrobass24 added enhancement New feature or request remote-agent labels May 19, 2026
@nitrobass24
nitrobass24 requested a review from s0up4200 May 19, 2026 16:38
Replace monolithic callsite refactor with incremental feature-by-feature
migration ordered by user-visible functionality. Each PR proves one
feature works through the Backend interface before moving to the next.
Phase 2 is just missing files detection (proves the pattern). Phase 3
builds the remote helper infrastructure. Phase 5 iteratively adds
remaining filesystem features after the helper is released, so each
new feature works for both local and remote instances.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
documentation/design/remote-helper.md (1)

208-210: ⚡ Quick win

Confirm and document the minimum Go toolchain requirement for os.Root.

The design relies on os.Root (Go 1.24+), but the implementation plan doesn’t explicitly gate rollout on repo toolchain compatibility. Please add an explicit “requires Go >= 1.24” note (or fallback plan) in the phase plan to avoid integration drift across PRs.

#!/bin/bash
set -euo pipefail

echo "== go.mod files and declared Go versions =="
fd '^go\.mod$' -a | while read -r mod; do
  echo "--- $mod"
  rg -n '^\s*go\s+[0-9]+\.[0-9]+' "$mod" || true
done

echo
echo "== Existing references to os.Root / toolchain assumptions =="
rg -n -C2 'os\.Root|Go 1\.24|toolchain|min(imum)? Go|openat2' documentation go.mod || true

Also applies to: 340-342

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@documentation/design/remote-helper.md` around lines 208 - 210, Update the
design doc section referencing os.Root to explicitly state "requires Go >= 1.24"
and either gate rollout on that toolchain or describe a clear fallback plan;
specifically edit the phase plan in documentation/design/remote-helper.md (the
os.Root mentions around the phase plan and the repeated note at lines ~208-210
and ~340-342) to: 1) add a one-line minimum toolchain requirement "Go >= 1.24"
next to the os.Root description, 2) describe the chosen rollout guard (e.g.,
bump repo go version in go.mod or CI check) OR a documented fallback behavior
when older toolchains are used, and 3) reference the os.Root symbol and openat2
dependency so reviewers can find the rationale.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@documentation/design/remote-helper.md`:
- Around line 208-210: Update the design doc section referencing os.Root to
explicitly state "requires Go >= 1.24" and either gate rollout on that toolchain
or describe a clear fallback plan; specifically edit the phase plan in
documentation/design/remote-helper.md (the os.Root mentions around the phase
plan and the repeated note at lines ~208-210 and ~340-342) to: 1) add a one-line
minimum toolchain requirement "Go >= 1.24" next to the os.Root description, 2)
describe the chosen rollout guard (e.g., bump repo go version in go.mod or CI
check) OR a documented fallback behavior when older toolchains are used, and 3)
reference the os.Root symbol and openat2 dependency so reviewers can find the
rationale.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 55260473-b5b1-423c-9181-9827e891d666

📥 Commits

Reviewing files that changed from the base of the PR and between 5849113 and f133b77.

📒 Files selected for processing (1)
  • documentation/design/remote-helper.md

@nitrobass24 nitrobass24 added this to the v1.19.0 milestone May 20, 2026
@s0up4200 s0up4200 removed this from the v1.19.0 milestone May 21, 2026
@nitrobass24 nitrobass24 closed this Aug 9, 2026
@nitrobass24 nitrobass24 reopened this Aug 11, 2026
@nitrobass24
nitrobass24 marked this pull request as draft August 11, 2026 20:44
@nitrobass24

Copy link
Copy Markdown
Contributor Author

Moving to draft as the backend FSOPS callsites dont depend on this.
We also need to make a decision if we are going to just use native remote SFTP/SSH calls first and then if performance reasons dictate actually move to a deployable agent. If we want to start with a deployable agent, we can just merge this and build the agent.

nitrobass24 added a commit that referenced this pull request Aug 13, 2026
Replaces the deployable-agent direction from the remote-helper design
(PR #1913): a single SSH connection per instance carrying the SFTP
subsystem plus opt-in exec channels, with capabilities probed at connect.
The key's own restrictions pick the tier — an internal-sftp key gets a
degraded-but-working mode (no file identity, so hardlink features switch
off visibly), an exec-capable key gets full functionality. Documents the
op mapping, security model, slimmed schema scope for #1917, and the open
questions (file identity wire form, SameFilesystem without fsids, BSD
remotes).
@nitrobass24

Copy link
Copy Markdown
Contributor Author

Closing in favor of the SFTP-native direction — no deployed agent. The new design is docs/remote-backend-design.md on #1914. Keeping this branch as the reference for the helper tier and its NDJSON protocol in case we ever hit a performance wall that native SFTP+exec can't clear.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request remote-agent

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants