Add local model catalog and optional bundled 0.5B GGUF

[bernoussama]

What

LazyShell can now recommend local Ollama/LM Studio models and optionally download a ~469 MB bundled GGUF for offline use. Weights are not in the npm package.

Local catalog

When the provider is Ollama or LM Studio, config setup offers a catalog plus Custom…:

• CPU: qwen2.5-coder:0.5b, qwen2.5-coder:1.5b (new default), hf.co/AryaYT/nl2shell-0.8b
• GPU: qwen2.5-coder:3b, qwen2.5-coder:7b, westenfelder/NL2SH

LM Studio default is qwen2.5-coder-1.5b-instruct.

Bundled model (opt-in / opt-out)

• First-run Yes/No downloads Qwen2.5-Coder-0.5B-Instruct Q4_K_M (~469 MB, Apache-2.0) to ~/.lazyshell/models/ with SHA-256 verification.
• Skip with --skip-bundled-model or LSH_SKIP_BUNDLED_MODEL=1 (persisted as skipped).
• Later: lazyshell model install / lazyshell model remove.
• If Ollama is selected but not running and the GGUF is installed, LazyShell starts a downloaded llama-server and uses the existing OpenAI-compatible path.

Small local models use a compact few-shot prompt and a command extractor (strip markdown, take the command line) so they do not echo hardware-prompt noise.

Groq evals

CI evals failed because Groq decommissioned llama-3.3-70b-versatile and qwen-qwq-32b. Defaults/judges now use openai/gpt-oss-120b and openai/gpt-oss-20b.

Bundled model eval

bun run eval:bundled — ExpectedToken 100% (6/6). Results: eval-results/eval-bundled-1788483267742.json

Prompt	Output
list files including hidden, long format	ls -la
current working directory	pwd
make folder test-project	mkdir test-project
find javascript files recursively	find . -type f -name '*.js'
show system information	uname -a
check disk usage	df -h

  

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[bernoussama]

Solid PR overall — checksum-pinned GGUF and llama.cpp runtime downloads, no shell interpolation in spawn, and the compact-prompt + extractCommand plumbing is a sensible way to tame a 0.5B model. A few things worth addressing:

Bugs / correctness

1. src/lib/ai.ts — hasCloudApiKey() checks MISTRAL_API_KEY, but envProvider()'s map has no ['MISTRAL_API_KEY', 'mistral'] entry. A Mistral-only user hits "No API key found…" from getDefaultModelAsync() with a valid key set, and resolveInitialProvider() disagrees with envProvider(). Add mistral to the map (or drop MISTRAL from hasCloudApiKey).
2. src/lib/bundled-model.ts ensureBundledServer() — the "already running" probe only checks that http://127.0.0.1:18765/health returns ok. Any unrelated local service with a /health route on that port makes LazyShell point its OpenAI-compatible client at it. Verify via GET /v1/models containing BUNDLED_MODEL.id before trusting the port.
3. src/lib/bundled-model.ts — if waitForServer() times out, the spawned llama-server is left running while serverReady is cleared; kill serverProcess on the failure path. Related: nothing stops the server when the CLI exits (stopBundledServer is only called from remove/eval), so llama-server silently outlives every run and holds RAM. Document it as a resident daemon, add an exit hook, or provide lazyshell model stop.
4. src/lib/bundled-model.ts downloadFile() — a mid-download failure leaves the ~469 MB .partial file behind (only removeBundledModel cleans it), and there's no resume, so a dropped connection restarts from byte 0. rm the temp file in a catch.

Code quality / nits

5. eslint.config.js turns no-floating-promises off repo-wide — that's masking, not fixing. Scope the override or fix the call sites.
6. isBundledModelInstalled() SHA-256s 469 MB and runs 3–4× per startup (resolveInitialProvider → initializeConfig → prepareLocalRuntime → ensureBundledServer). Memoize per-process or short-circuit on a size+mtime match.
7. src/commands/config.ts editProvider() starts a 469 MB download with no confirmation when switching to the bundled provider — confirm first, like the first-run prompt does.
8. The committed eval-results/*.json files are generated artifacts (and the first one documents a failing 0.67 run) — gitignore eval-results/ and drop them from the PR.
9. src/commands/model.ts formatMb() divides by 1,000,000, so the UI shows "491 MB" while the README/first-run copy say ~469 MB (MiB). Pick one unit.
10. generateCommandStruct() applies generationLimits().temperature to generateObject but drops maxTokens: 64 — harmless today since the schema constrains output, but the intent leaks.

Small README typo too: "offers a catalog plus Custom…:".