Plan Apple Silicon local-first inference

[cuevaio]

Destination

An implementation-ready migration spec for Tab’s initial validated local-first target, the available M4 Pro Mac with 24 GB memory: automatic Suggestions run locally with a user-selected model from the Local Model Catalog, while double-Option explicitly requests remote inference. The spec settles product, architecture, privacy, personalization, billing, telemetry, validation, and rollout decisions without implementing the production migration or claiming support for untested Mac configurations.

Notes

• Domain: use the canonical language in CONTEXT.md; consult relevant records in docs/adr/.
• Starting research: docs/deep-research-report.md.
• Sessions should consult /grilling, /domain-modeling, and the ticket-specific skill; use /prototype for benchmark artifacts.
• Automatic local Suggestions are unlimited and never silently fall through to remote inference. If local inference is unavailable, automatic Suggestions are unavailable and durable status explains why.
• Double-Option is the explicit remote-inference action and preserves current backend Personal Memory retrieval, stronger-model generation, metadata telemetry, and quota semantics.
• Preserve the existing bounded, ephemeral Memory Extraction Window. The backend remains authoritative for Personal Memory and performs LLM-based Memory Extraction; raw window content is not durably stored.
• Add a read-only Personal Memory Replica for local personalization. It works offline while signed in, is purged at explicit sign-out, and is not used for local Memory Extraction. Signed-out local Suggestions may continue without personalization.
• Local personalization uses a bounded deterministic shortlist, not local embeddings. Backend Memory Extraction continues to use embeddings.
• The Local Model Catalog is curated and versioned. Downloads and model updates require user approval; security revocation may disable a compromised model. Arbitrary model import is not supported.
• Local telemetry remains metadata-only: inference source, catalog model/version, coarse hardware tier, latency, outcome, and length; never Typing Context, memory content, generated text, or exact hardware identifiers.
• Available benchmark hardware and initial validated target is an M4 Pro Mac with 24 GB memory. Results must not be generalized into support claims for untested Mac configurations.
• Planning is the default: tickets resolve decisions or produce disposable evidence needed for decisions; production implementation is outside this map.

Decisions so far

• Compare Apple Silicon local inference runtimes — Benchmark pinned llama.cpp/Metal, stateful Core ML, and MLX Swift behind one helper boundary; select only from measured results on the validated target.

• Define the Personal Memory Replica contract - The encrypted per-account/device read-only replica converges atomically to backend Personal Memory, supports bounded offline use and online propagation, purges on identity exit, and fails without blocking Suggestions.

• Define local Suggestion acceptance thresholds - Automatic local Suggestions must pass conjunctive M4 Pro/24 GB gates for warm latency, quality, memory pressure, battery, disk, startup, cancellation, and reliability before shipping.

• Prototype the local Suggestion benchmark - AC high-power screens advanced llama.cpp Qwen3.5/Qwen2.5/Gemma and MLX Qwen2.5 as inconclusive quality/resource candidates; the exact tested Core ML combination failed.

• Compare local Suggestion quality of AC screening survivors - All four AC survivors failed the 98% contract-compliance and 100% sensitive-withholding gates, so none advances to resource or reliability validation.

• Prototype a quality-safe local Suggestion configuration - Deterministic safety and contract boundaries pass, but every candidate remains conclusively below the 70% usefulness gate; calibrate the unexpectedly low evaluator before expanding models.

• Calibrate the local Suggestion quality evaluator - Independent-candidate agent scoring confirmed all tested paths remain below 70%, exposed order instability in paired judging, and left human acceptance calibration unresolved.

• Measure the attainable automated quality ceiling of the fixed Suggestion corpus — Two stronger remote references reached 95.8% automated usefulness, demonstrating the fixed corpus can clear the preserved 70% gate.

• Prototype a stronger local Suggestion quality candidate — Qwen3.5 9B Q4_K_M through pinned llama.cpp passed every preserved automated quality gate at 85.8% usefulness and advances as an exact configuration to resource and reliability validation.

Not yet specified

• The production inference process boundary, lifecycle, warm-residency, cancellation, crash recovery, and resource-pressure behavior after runtime evidence narrows the viable architecture.
• The catalog manifest, artifact hosting, provenance, signing, checksum, download-resume, storage, update, revocation, and cleanup design after runtime/model formats are known.
• The local Personal Memory ranking algorithm and prompt budget after model context and quality thresholds are known.
• The desktop routing/state-machine, API contract, status UI, onboarding, and failure-recovery changes after runtime and model lifecycle decisions settle.
• Release sequencing, migration from cloud-automatic behavior, rollback, and support policy after architecture and telemetry decisions settle.
• Which existing ADRs and PRD boundaries must be superseded once the target architecture is settled.

Out of scope

• Production implementation of the migration.
• Obtain an 8 GB Apple Silicon benchmark target and support claims for 8 GB or other untested Apple Silicon configurations; broader hardware validation is deferred.
• Intel Mac, Windows, and Linux support.
• Arbitrary user-provided or bring-your-own models.
• Local embeddings on the automatic Suggestion hot path.
• Local Memory Extraction or local authoritative Personal Memory storage.
• Durable raw typing logs or durable Memory Extraction Window content.
• Redesigning the existing remote model/provider path beyond explicit routing and source contracts.