edgeLMs

Zero-training proof-of-concepts for small local models on endpoint data. One frozen embedder, four jobs, nothing trained — a hands-on, measured answer to how far Rung 0 (no training) goes before fine-tuning is worth the labels.

Status: exploration complete. All four POCs were run as falsifiable experiments on real public data with pre-registered fail conditions. The findings are below and in docs/PRD.md §4.

What we learned (in one line)

The frozen embedder is strong at similarity / recall jobs and weak at judgment jobs. It wins when the task is "find the thing that's close." It loses when the task is "make a decision the query's meaning doesn't contain."

Experiment	Job	Verdict	Why
Entity resolution — blocking	similarity / recall	✅ wins — 94% recall@20, 161× fewer comparisons	the true match is always close; a shortlist only needs it in the top-k
Command anomaly — catching	similarity / recall	✅ beats a regex — 90% vs 55% recall at equal false-alarm cost	a novel attack still resembles known attacks (catches keyword-free reverse-shells)
Log anomaly — structured	judgment (novelty)	❌ loses to a rule	an "unseen-template" regex already owns tidy logs — cheaper and exact
Entity resolution — matching	judgment (identity)	❌ weak	same category ≠ same instance; two different Sony TVs embed nearly identically
Command anomaly — blocking	judgment (intent)	❌ over-flags — 96% of legit installers	intent lives in the URL, not the command text
Router	judgment (difficulty)	❌ fails — ~coin flip (ROC-AUC 0.55→0.51)	difficulty is a prediction about a future model, not the query's meaning

The product read: the small local model's honest role is a recall-first narrower / router — it cheaply proposes ("these ~20 records might match", "this looks suspicious") and escalates the judgment to a precise layer (a rule, a bigger model, a person). Exactly the escalation ladder — now backed by measurement. The one POC that clearly earns a fine-tune is the router.

The idea in one picture

one pretrained embedder (all-MiniLM-L6-v2, frozen)
        │  embed(text) → vector
  ┌─────┼───────────┬────────────┐
  ▼     ▼           ▼            ▼
log   command     router      entity
anomaly anomaly               resolution

The model is shared and frozen; each job's "knowledge" is cheap data (a per-system baseline, a dozen labeled examples, a threshold) — not weights. command-anomaly and router share shared/classify.js verbatim — swapping the prototypes is the entire difference. That reuse is the point.

Reproduce the experiments (real data)

The falsifiable versions live in experiments/. Each has a get-data.sh that fetches the real public dataset (gitignored) and a run.js that prints threshold-free metrics, a cheap rival, and a pre-registered verdict.

npm install                                   # first run also downloads the ~90MB model, once

# each experiment: fetch real data, then run
experiments/01-log-anomaly/get-data.sh        && node experiments/01-log-anomaly/run.js
experiments/02-command-anomaly/get-data.sh    && node experiments/02-command-anomaly/run.js
experiments/03-router/get-data.sh             && node experiments/03-router/run.js
experiments/04-entity-resolution/get-data.sh  && node experiments/04-entity-resolution/run.js

Data sources: Loghub (BGL/Mac logs), NL2Bash + Cowrie honeypot + reverse-shell payloads, RouterBench (per-query model correctness), ER-Magellan / Leipzig (Amazon-Google, Abt-Buy).

pocs/ holds the original throwaway spikes on synthetic fixtures — kept for reference. experiments/ holds the real, falsifiable versions the findings above are based on. npm run log-anomaly (etc.) still runs the quick synthetic pocs/ demos.

Next rung

The router is the standout Rung-1 candidate — a measured Rung-0 failure where the signal is real but not in the surface (train it on outcome labels, à la RouteLLM). See docs/PRD.md §4.5 (synthesis) and §8 (candidate datasets).

License

Apache-2.0.