A creative way to see whether your local models are actually good enough: ask them to build visual experiences, then compare the results.
Live workbench • Source • eeshans.com
Requirements: Node.js 24+ and offgrid-ai for running benchmarks.
npm install
npm run devModel benchmarks are usually abstract: scores, tokens, leaderboards, and tiny deltas that do not always explain what a model can actually do.
This project takes a more practical route. It gives models small visual build tasks like a solar system, a sakura tree, a macro wildflower scene, or a sunset ocean study. The model has to produce a working HTML artifact. The app captures a screenshot and 20-second preview video, then lets you compare outputs by model or by prompt.
If a model understands layout, animation, visual hierarchy, browser APIs, and instruction-following, the result usually looks better. If it struggles, you can see exactly where it falls apart.
The point of view is intentionally narrow: this is an Apple Silicon daily-driver stack benchmark, not a universal local-inference leaderboard. The project is about finding which combination of model source (oMLX, LM Studio, Ollama, cloud), model artifact (GGUF quant, MLOX format, API), coding harness (Pi, OpenCode, manual chat), and prompt workflow is usable enough for real local work.
offgrid-ai is the runner — it manages local models, launches servers, and prepares benchmark runs.
npm install -g offgrid-ai
offgrid-ai models # select a model → run, benchmark, or inspect
offgrid-ai run <profile> # start the model server and open Pi
offgrid-ai benchmark # prepare a benchmark run (standalone)The model-first flow in offgrid-ai models lets you pick a model, then choose an action:
- Run — start the server and open Pi
- Benchmark — choose a prompt, create a run slot in this repo
- Reconfigure — change context window, MTP, or other settings
- Details — inspect ports, paths, and flags
For benchmark runs, offgrid-ai creates the run directory under runs/ with metadata.json and prompt.md, then you:
- Start the model with
offgrid-ai run <profile> - Open the gallery with
npm run dev - Find your run in the gallery and copy the prompt from the run details
- Paste into Pi and generate
- Open the app locally with
npm run dev. - Browse results in the workbench using By prompt, By model, or Compare views.
- Toggle cloud models on/off with the Include cloud models checkbox.
- In Compare, select rows to compare visual runs side by side.
The live site is the same workbench as a static export of captured results. The local app is where run preparation, capture, deletion, folder-opening, and config sync happen.
Cloud models (GPT, Claude, Gemini, DeepSeek, etc.) are benchmarked the same way: offgrid-ai benchmark → choose "Custom / cloud" → enter a model name. The prompt is copied into your cloud tool of choice. No local server needed.
| Backend | Type | Model source | Setup |
|---|---|---|---|
| llama.cpp | Local server | ~/.lmstudio/models/ GGUF |
offgrid-ai models → Set up |
| llama.cpp MTP | Local server (speculative decoding) | ~/.lmstudio/models/ GGUF |
offgrid-ai models → Set up → MTP variant |
| Ollama | Managed server | Ollama API (localhost:11434) |
offgrid-ai models → Set up |
| oMLX | Managed server | oMLX API (127.0.0.1:8000) |
offgrid-ai models → Set up |
| Cloud | No server | Any OpenAI-compatible API | offgrid-ai benchmark → Custom / cloud |
Install dependencies:
npm installStart the local gallery:
npm run devoMLX defaults to:
http://127.0.0.1:8000/v1
LM Studio defaults to:
http://localhost:1234/v1
Ollama defaults to:
http://localhost:11434/v1
After you capture some runs locally:
npm run publish
git add public/export
git commit -m "data: publish gallery update"
git pushThat single publish command refreshes the publish-safe gallery snapshot in public/export/, runs checks/tests, and creates a smoke-testable static site in dist-static/.
The GitHub Pages workflow deploys at https://localai.eeshans.com/ on every push to main.
The browser workbench is intentionally one site, not separate local and public apps. Local dev mode and the static GitHub Pages export share the same Astro page and viewer modules. Static mode only loads export/manifest.json and hides local operational controls.
The viewer bootstrap lives in public/js/app.js; feature behavior is split into small controller modules under public/js/ for data loading, model sources, capture, prepare-run, detail actions, workbench rendering, operational-control visibility, icons, and the machine-profile pill.
Captured videos default to 20 seconds at 1600×900. The defaults are defined in src/lib/capture-media.ts as DEFAULT_VIDEO_DURATION_MS and DEFAULT_VIEWPORT.
| Folder | Purpose |
|---|---|
benchmarks/ |
Source benchmark prompts |
runs/ |
Local run folders, generated prompts, HTML, screenshots, and videos |
public/export/ |
Tracked publish-safe gallery snapshot: manifest plus copied preview media |
dist-static/ |
Final static site artifact |
runs/ and dist-static/ are ignored by git so local experiments and build output do not get committed accidentally. public/export/ is tracked because it is the public gallery data source.
npm run dev # local app with operational controls
npm run build # server/API build
npm run build:static # static gallery build from existing export
npm run publish # refresh export, validate, and build static site
npm test # unit tests
npm run test:e2e # Playwright tests
npm run check # Astro + TypeScript checks- Runs are local by default.
- oMLX, LM Studio, and Ollama stay on your machine.
- The production site may use PostHog pageview analytics; local builds keep analytics disabled unless explicitly enabled with public env vars.
- The app only publishes captured media, the benchmark prompt text, publish-safe summary metadata, and a build-time machine profile when you build the static export.
- Raw generated HTML, prepared run prompts, raw responses, stream logs, launch commands, local service URLs, and local filesystem paths are kept local and are not included in the published workbench.
Personal project by Eeshan Srivastava.