Skip to content

Verify: local Qwen engine end-to-end on device (real download → real llama-server → quality, offline, fallback) #115

Description

@realproject7

Goal

The Local · Qwen 4B tier is code-complete and CI-tested, but ONLY against a fake llama-server (packages/engine/test/local-llm-engine.e2e.test.ts uses fake-llama-server.mjs). No real end-to-end run is on record: real 2.5 GB model download, real llama-server b9601 startup, real translation/summary/coaching quality, offline operation, and the mid-session CLI→local fallback. Verify it for real on this Mac and file follow-ups for quality gaps.

Where (reference, verified on main @ fa26d60)

  • packages/engine/src/pins.ts:24-31 — pinned Qwen3-4B-Q4_K_M.gguf (2.497 GB, SHA-256) + llama.cpp b9601
  • src/host/local-tier.ts:90-173 — binary + model acquisition, LazyLocalEngine
  • packages/engine/src/local-llm-engine.ts:101-167 — server spawn args, health probe (60 s startup / 2 s per-poll / 30 s per-request)
  • packages/engine/src/fallback-router.ts:35 — CLI→local mid-session switch
  • Settings engine selector: src/settings-sheet.ts:63-66

Verification checklist (report N/N on this ticket)

  1. Fresh download on this Mac: progress %, SHA-256 pass, .part resume after a forced interruption.
  2. llama-server starts within budget; --reasoning-budget 0 honored (no multi-second think stalls).
  3. Engine=local live session (network OFF after download): captions translated, summary + board generated, coaching rewrite works; subjective quality notes vs CLI on the same audio.
  4. Fallback: start with engine=cli, kill the CLI process mid-session → router switches to local without dropping in-flight batches; status surfaced.
  5. Resource notes: RSS + per-request latency during a 15-min session; zombie check on quit ([follow-up] SIGTERM leaves a zombie overlay window + orphaned llama-server/host after the Rust process exits #66 regression: no orphaned llama-server).
  6. Usage events report $0 cost; gauge unaffected.

Deliverables

Checklist results on this ticket + follow-up tickets for anything below bar (esp. translation quality — if Qwen3-4B output is unusable, a model-upgrade/parameter follow-up).

Routing

agent/claude (PO direct — device-bound verification, manual §9/§12).

Part of #107. Depends on #108 (fast dev loop) for iteration comfort, not strictly blocking.

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/claudeBuilt by local Claude agent (needs this device)enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions