Context
The partner runs in the browser with the reader's own API key. The truest free option, with no key and no account at all, is a model that runs on the reader's own device through WebGPU. This is the project's answer for someone who has no budget, and it keeps everything private. The owner's phone (an iPhone 17 Air) sets the floor: the smallest option must run there.
What we want
An on-device model option, added to the existing provider picker, that needs no key. Tiered models so the floor runs on a phone and stronger models run on a laptop.
How
- Use WebLLM (
@mlc-ai/web-llm), which loads a quantized model into the browser via WebGPU and runs inference locally. Lazy-load it so it never bloats the initial bundle.
- Add a
local provider in src/lib/providers.js and a local branch in the stream client (the multi-provider client, currently src/lib/anthropic.js). It must use the same system prompt and the same human-acts-first flow.
- A tiered, labeled model picker with download sizes:
- Floor (iPhone 17 Air): a small quantized model in the one-to-three-billion range.
- Mid (iPad, average laptop): three-to-four billion.
- Ceiling (MacBook M5): seven-to-eight billion.
- One-time download, cached on the device. Show progress.
Acceptance criteria
- A reader with no key can pick an on-device model and study, fully offline after the download.
- The floor model runs on the owner's iPhone 17 Air.
- The model never routes through any server or any other provider.
- Labeled free, private, and experimental, with one honest line: small models are the most likely to break the partner's one rule and invent Talmud text, so a stronger model is better for serious study.
Pointers
src/lib/providers.js, the stream client, src/pages/Settings.jsx. WebLLM: https://github.com/mlc-ai/web-llm . Test WebGPU support on iOS Safari early, since memory limits there are the real constraint.
Risk
WebGPU memory on phones is tight; the floor model may be weak. The fidelity caveat is real and must be surfaced to the reader.
Context
The partner runs in the browser with the reader's own API key. The truest free option, with no key and no account at all, is a model that runs on the reader's own device through WebGPU. This is the project's answer for someone who has no budget, and it keeps everything private. The owner's phone (an iPhone 17 Air) sets the floor: the smallest option must run there.
What we want
An on-device model option, added to the existing provider picker, that needs no key. Tiered models so the floor runs on a phone and stronger models run on a laptop.
How
@mlc-ai/web-llm), which loads a quantized model into the browser via WebGPU and runs inference locally. Lazy-load it so it never bloats the initial bundle.localprovider insrc/lib/providers.jsand alocalbranch in the stream client (the multi-provider client, currentlysrc/lib/anthropic.js). It must use the same system prompt and the same human-acts-first flow.Acceptance criteria
Pointers
src/lib/providers.js, the stream client,src/pages/Settings.jsx. WebLLM: https://github.com/mlc-ai/web-llm . Test WebGPU support on iOS Safari early, since memory limits there are the real constraint.Risk
WebGPU memory on phones is tight; the floor model may be weak. The fidelity caveat is real and must be surfaced to the reader.