Where I try out ideas about LLM inference that I find neat and that don't seem to have been tried much. The loose
theme: what would it take for calling a model to be as boring as calling fetch? You call it and stop thinking
about it. It fails in known ways, takes about as long as you expected, and when something does go wrong you can see
exactly what happened.
An idea gets tried in the cheapest place first, usually an agent with a prompt and a tool. If it still looks good it goes further down: into the server, the inference engine, and for a few, eventually training. Some only work if several of those change together, which is why this leans on forks of OpenWebUI, Ollama (heavily modified) and llama.cpp.
So this is a hypothesis-testing apparatus more than a codebase. Every row of the table below is a claim that might be wrong rather than a feature that is planned. What should outlast any of them is the finding: a short technical note on how the thing works and whether it held up, and a proper implementation wherever it actually belongs. That is what the third column is saying. None of them point back here.
Nearly all of it was generated by Claude Code, and I have not reviewed most of it. It is glue: whatever an idea needed in order to be tried at all, and that turns out to be a lot. The tests are not a quality claim either. They are there so the agent does not break the rest of the glue while it builds the next piece.
So expect it to be uneven, and expect interfaces to change whenever an experiment needs them to. It is not a framework or a product, and it will not become one. It is not a reference implementation either: if an idea here works, what is worth having is the write-up and a real implementation in the place the idea belongs, not this code borrowed. The glue is scaffolding, and scaffolding comes down.
It started as a Chrome extension that puts window.ml on web pages, and the agent still runs there, because a web
page is an easy source of real tasks with me watching. That part is incidental now.
So far this is mostly glue. Few of the ideas have actually been tested; the middle column says which.
| Idea | Tried so far | Where it would end up if it works |
|---|---|---|
| You can see exactly what the model saw on every run, and where the time went | the sidebar, the exports, the server's event stream | everywhere |
| A tool's output is referred to by a short pointer instead of being copied back into the context | a prompt and a dereference tool; one A/B pilot | post-training |
| The agent manages its own context: it collapses stale tool outputs to their pointers with a projection it picks, and expands them when it needs them again. Pointers name typed objects (a table, not its text) that JavaScript and Python compute over | typed table pointers both runtimes read; collapse itself not started (notes) | the agent, then post-training |
| Arithmetic in a thinking block is worked out exactly and the result spliced into the output | not started | the inference engine's decode loop, then post-training |
| The server decides where a model goes and how long it stays loaded, learned from how models actually get used, with no knobs for the caller | the Ollama fork, and the routing a mixture of experts actually does, recorded in the llama.cpp fork | the server |
| The client tells the server who is waiting on each request and which requests belong together | sent and recorded, as data for the placement and keep-alive predictor | the server |
| The server predicts each generation's decode speed and corrects itself from what it measures | the Ollama fork | the server |
| Read-only work runs without asking me; everything else asks. The permission is whether the script is written in a small JavaScript dialect that can only read, always halts, and leaves nothing behind when it refuses | the dialect, still growing | the agent |
| An agent uses my credentials without ever being able to read them: it holds a handle, and the secret stays in the service worker | not started (notes) | the extension's background worker |
| I drive a browser agent at home from my phone through a relay that cannot drive it itself | window-ml-hub relays sealed traffic and a connector republishes a box's telemetry through it, self-hosted. The extension's own connector is not built, so nothing drives an agent through it yet (notes) | a small hub server, and the extension |
Right now. Finishing the first row, because the rest depends on it: the benchmark tooling reads that data and the pointer A/B needs the benchmark, so both are paused until it is done. On the server side I am working on predicting placement and keep-alive from recorded use, which is a statistics problem rather than a coding one.
window.ml lives in the page's main world, so any page the extension is active on can call it, and a hostile page
can subvert it. Keep its site access on "On click". See the trust model.
Against stock Ollama and OpenWebUI the agent works; the parts that need a fork say "not reported" instead.
- Set it up: docs/SETUP.md for the extension, docs/FULL-SETUP.md for a backend from scratch.
- The page API (
ml.chat,ml.agent, …): docs/API.md. - What needs which fork: docs/FORKED-BACKENDS.md.
- Watching or driving a runtime from elsewhere: window-ml-hub, a separate Rust repository that relays traffic it cannot read.
- Notes on how the code works, mostly for the agent: AGENTS.md and docs/dev/. Building and testing: CONTRIBUTING.md.
MIT. Fork whatever is useful.