@localmode/wllama

wllama provider for LocalMode -- run any GGUF model in the browser via llama.cpp compiled to WebAssembly.

Features

Run any of the 160,000+ GGUF models from HuggingFace
Works in all modern browsers (no WebGPU required, only WASM)
30 curated models: 25 language (including 4 vision-language, 2 reasoning), 3 embedding, 2 reranker
True streaming via createChatCompletion({ stream: true }) for real token-by-token output
Structured output / JSON mode via response_format (text, json_object, json_schema)
Reranking via wllama.reranker() with cross-encoder GGUF models (Jina, BGE)
Embedding models via wllama.embedding() (nomic-embed, mxbai-embed, bge-small)
WebGPU acceleration with useWebGPU and nGpuLayers settings
Tool calling via providerOptions.wllama.tools and tool_choice
Vision/multimodal input with mmprojUrl for VLMs (Holo2 4B/8B, Gemma 4 E2B/E4B)
Reasoning mode for DeepSeek-R1 style chain-of-thought thinking
Performance tuning -- KV cache quantization, flash attention, speculative decoding
Grammar sampling via GBNF for constrained text output
LoRA adapters for fine-tuned model loading
Jinja chat templates enabled by default for accurate prompt formatting
Model management -- list cached models, clear cache
GGUF metadata inspection via HTTP Range requests (~4KB download)
Model discovery -- search the 160,000+ GGUF repos on HuggingFace and list a repo's .gguf files (searchGGUFModels / listGGUFFiles)
Browser compatibility checking before downloading multi-GB files
Auto-detects CORS isolation for multi-threaded inference
Chrome MV3 extension support (auto-resolves bundled WASM binaries)
Full AbortSignal cancellation support

Installation

pnpm install @localmode/wllama @localmode/core

Quick Start

import { generateText } from '@localmode/core';
import { wllama } from '@localmode/wllama';

const model = wllama.languageModel(
  'bartowski/Llama-3.2-1B-Instruct-GGUF:Llama-3.2-1B-Instruct-Q4_K_M.gguf'
);

const { text } = await generateText({
  model,
  prompt: 'Explain quantum computing in simple terms.',
});

console.log(text);

True Streaming

doStream() uses createChatCompletion({ stream: true }) for real token-by-token streaming. Each token is yielded as it arrives from the WASM backend -- no buffering. For prompt-only input (no messages or systemPrompt), it falls back to a single-chunk delivery via doGenerate().

import { streamText } from '@localmode/core';
import { wllama } from '@localmode/wllama';

const model = wllama.languageModel('Llama-3.2-1B-Instruct-Q4_K_M');

const result = await streamText({
  model,
  messages: [{ role: 'user', content: 'Write a haiku about the sea.' }],
});

for await (const chunk of result.stream) {
  process.stdout.write(chunk.text); // real token-by-token output
}

GGUF Metadata Inspection

Inspect any GGUF model before downloading:

import { parseGGUFMetadata } from '@localmode/wllama';

const metadata = await parseGGUFMetadata(
  'bartowski/Llama-3.2-1B-Instruct-GGUF:Llama-3.2-1B-Instruct-Q4_K_M.gguf'
);

console.log(metadata.architecture);   // 'llama'
console.log(metadata.quantization);   // 'Q4_K_M'
console.log(metadata.contextLength);  // 131072
console.log(metadata.parameterCount); // ~1.24B

GGUF Model Discovery

Search the 160,000+ GGUF repos on HuggingFace and list a repo's .gguf files -- an anonymous, cancellable browse surface with no model download:

import { searchGGUFModels, listGGUFFiles, HFApiError } from '@localmode/wllama';

// Search GGUF-tagged repos (an empty query browses the top models for the sort).
const { results, nextCursor } = await searchGGUFModels({
  query: 'llama 1b',
  sort: 'downloads', // 'downloads' | 'likes' | 'lastModified'
});

for (const repo of results) {
  console.log(repo.repoId, repo.downloads, repo.likes);
}

// Paginate: pass nextCursor back (with the same query + sort) for page two.
// nextCursor === null means the last page.

// List a repo's .gguf files with byte sizes and parsed quant labels.
const files = await listGGUFFiles('bartowski/Llama-3.2-1B-Instruct-GGUF');
// → [{ filename: 'Llama-3.2-1B-Instruct-Q4_K_M.gguf', sizeBytes: 807694464, quantLabel: 'Q4_K_M' }, …]

Both functions support AbortSignal ({ abortSignal }) so stale searches can be cancelled on keystroke or model switch. Abort rejections propagate untouched (the original AbortError, never wrapped). Failures throw a typed HFApiError:

try {
  await searchGGUFModels({ query, sort: 'downloads' });
} catch (err) {
  if (err instanceof HFApiError) {
    // err.kind is 'rate-limit' (HTTP 429), 'not-found' (404/401/403), or 'network'
    console.warn(err.kind, err.message);
  }
}

Pair with checkGGUFBrowserCompatFromURL (below) to verify a discovered file can run on the current device before downloading.

Embedding Models

Generate text embeddings from GGUF embedding models using wllama.embedding():

import { embed } from '@localmode/core';
import { wllama } from '@localmode/wllama';

const model = wllama.embedding('nomic-embed-text-v1.5-Q4_K_M');

const { embedding } = await embed({ model, value: 'Hello world' });
console.log(embedding.length); // 768

Three curated embedding models are included in the catalog:

Model	Size	Dimensions	Description
`nomic-embed-text-v1.5-Q4_K_M`	78MB	768	High-quality text embeddings for semantic search
`mxbai-embed-large-v1-Q4_K_M`	197MB	1024	Top-quality English embeddings
`bge-small-en-v1.5-Q8_0`	35MB	384	Lightweight English embeddings

Dimensions are auto-detected from GGUF metadata when not specified.

Reranking

Rerank search results using cross-encoder GGUF models via wllama.reranker():

import { rerank } from '@localmode/core';
import { wllama } from '@localmode/wllama';

const model = wllama.reranker('jina-reranker-v2-base-multilingual-Q4_K_M');

const { results } = await rerank({
  model,
  query: 'machine learning frameworks',
  documents: [
    'PyTorch is a deep learning framework',
    'A recipe for chocolate cake',
    'TensorFlow supports distributed training',
    'The weather forecast for tomorrow',
  ],
});

// Results sorted by relevance score
for (const r of results) {
  console.log(`[${r.score.toFixed(3)}] ${r.text}`);
}

Two curated reranker models are included in the catalog:

Model	Size	Context	Description
`jina-reranker-v2-base-multilingual-Q4_K_M`	163MB	1K	Jina Reranker v2, multilingual cross-encoder
`bge-reranker-v2-m3-Q4_K_M`	218MB	8K	BAAI BGE Reranker v2 M3, long-context multilingual

Tool Calling

Models that support tool calling (marked with supportsToolCalling in the catalog) can use tools via providerOptions:

import { generateText } from '@localmode/core';
import { wllama } from '@localmode/wllama';

const model = wllama.languageModel('Llama-3.2-1B-Instruct-Q4_K_M');

const result = await generateText({
  model,
  prompt: 'What is the weather in Tokyo?',
  providerOptions: {
    wllama: {
      tools: [{
        type: 'function',
        function: {
          name: 'get_weather',
          description: 'Get current weather for a city',
          parameters: {
            type: 'object',
            properties: { city: { type: 'string' } },
            required: ['city'],
          },
        },
      }],
      tool_choice: 'auto',
    },
  },
});

The result includes a toolCalls array when the model invokes tools.

Structured Output / JSON Mode

Constrain model output to valid JSON via providerOptions.wllama.response_format. Three modes are available:

import { generateText } from '@localmode/core';
import { wllama } from '@localmode/wllama';

const model = wllama.languageModel('Qwen2.5-3B-Instruct-Q4_K_M');

// Free-form JSON output
const { text } = await generateText({
  model,
  messages: [{ role: 'user', content: 'List 3 planets as JSON with name and diameter.' }],
  providerOptions: {
    wllama: {
      response_format: { type: 'json_object' },
    },
  },
});

// Schema-constrained JSON output
const { text: structured } = await generateText({
  model,
  messages: [{ role: 'user', content: 'Give me info about Mars.' }],
  providerOptions: {
    wllama: {
      response_format: {
        type: 'json_schema',
        json_schema: {
          name: 'planet',
          schema: {
            type: 'object',
            properties: {
              name: { type: 'string' },
              diameter_km: { type: 'number' },
              moons: { type: 'integer' },
            },
            required: ['name', 'diameter_km'],
          },
          strict: true,
        },
      },
    },
  },
});

The WllamaResponseFormat type is exported for TypeScript consumers.

Vision / Multimodal

Vision-language models (e.g., Holo2) accept image input alongside text. The model requires an mmprojUrl pointing to the vision projection GGUF. Catalog entries for vision models include mmprojUrl automatically:

import { generateText } from '@localmode/core';
import { wllama } from '@localmode/wllama';

// Catalog entries include mmprojUrl automatically
const model = wllama.languageModel('Holo2-4B-Q4_K_M');
console.log(model.supportsVision); // true

// Pass images as multimodal content parts (base64-encoded)
const { text } = await generateText({
  model,
  messages: [{
    role: 'user',
    content: [
      { type: 'text', text: 'Describe this screenshot.' },
      { type: 'image', data: base64ImageData, mimeType: 'image/png' },
    ],
  }],
});

For custom vision models, pass mmprojUrl in the model settings:

const model = wllama.languageModel('my-repo/my-vlm-GGUF:model.gguf', {
  mmprojUrl: 'https://huggingface.co/my-repo/my-vlm-GGUF/resolve/main/mmproj-f16.gguf',
});

Audio Input (Experimental)

Models that accept audio content parts can receive audio alongside text. Audio data is passed as base64-encoded strings and converted to ArrayBuffer internally:

import { generateText } from '@localmode/core';
import { wllama } from '@localmode/wllama';

const model = wllama.languageModel('my-audio-model');

const { text } = await generateText({
  model,
  messages: [{
    role: 'user',
    content: [
      { type: 'text', text: 'Transcribe this audio.' },
      { type: 'audio', data: base64AudioData, mimeType: 'audio/wav' },
    ],
  }],
});

Note: Audio support depends on the underlying GGUF model's capabilities. No curated catalog models currently support audio input.

WebGPU Acceleration

Enable WebGPU to offload transformer layers to the GPU for faster inference:

const model = wllama.languageModel('Llama-3.2-1B-Instruct-Q4_K_M', {
  useWebGPU: true, // enable GPU offload; falls back to WASM if unavailable
});

// Or auto-detect WebGPU availability
const model2 = wllama.languageModel('Llama-3.2-1B-Instruct-Q4_K_M', {
  useWebGPU: 'auto',
});

// Fine-grained control: offload specific number of layers
const model3 = wllama.languageModel('Llama-3.2-1B-Instruct-Q4_K_M', {
  nGpuLayers: 20, // offload 20 layers; use -1 for all layers
});

// gpuAccelerated is available on the concrete WllamaLanguageModel class
import { WllamaLanguageModel } from '@localmode/wllama';
const wllamaModel = new WllamaLanguageModel('Llama-3.2-1B-Instruct-Q4_K_M', { useWebGPU: true });
console.log(wllamaModel.gpuAccelerated); // true when WebGPU is active

WebGPU settings also work with embedding models:

const embedModel = wllama.embedding('nomic-embed-text-v1.5-Q4_K_M', {
  useWebGPU: 'auto',
});

Jinja Chat Templates

wllama v3 uses the model's built-in Jinja chat template for accurate prompt formatting. This is enabled by default (useJinja: true). If a model's template causes errors, wllama automatically falls back to raw completion mode.

const model = wllama.languageModel('Llama-3.2-1B-Instruct-Q4_K_M', {
  useJinja: false, // disable Jinja templates (use raw completion)
});

Reasoning Mode

Enable chain-of-thought reasoning for DeepSeek-R1 style thinking models:

import { generateText } from '@localmode/core';
import { wllama } from '@localmode/wllama';

const model = wllama.languageModel('DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M', {
  reasoning: true,
  reasoningFormat: 'deepseek',       // 'none' | 'deepseek-legacy' | 'deepseek'
  reasoningBudgetTokens: 1024,       // limit thinking tokens
});

const { text } = await generateText({
  model,
  messages: [{ role: 'user', content: 'Solve: if 3x + 7 = 22, what is x?' }],
});

When streaming, reasoning content is surfaced via reasoning_content delta chunks alongside regular content.

Performance Configuration

KV Cache Quantization

Reduce memory usage for long contexts by quantizing the KV cache:

const model = wllama.languageModel('Llama-3.1-8B-Instruct-Q4_K_M', {
  cacheTypeK: 'q4_0',  // quantize key cache (f32, f16, q8_0, q5_1, q5_0, q4_1, q4_0)
  cacheTypeV: 'q4_0',  // quantize value cache
});

Flash Attention

const model = wllama.languageModel('Llama-3.2-3B-Instruct-Q4_K_M', {
  flashAttention: true,
});

Speculative Decoding

Use a smaller draft model for 2-3x faster inference:

const model = wllama.languageModel('Llama-3.1-8B-Instruct-Q4_K_M', {
  specDraftModel: 'https://huggingface.co/bartowski/SmolLM2-135M-Instruct-GGUF/resolve/main/SmolLM2-135M-Instruct-Q4_K_M.gguf',
  specDraftNgl: -1,     // GPU layers for draft model
  specDraftNMin: 2,     // min draft tokens
  specDraftNMax: 8,     // max draft tokens
  specDraftPMin: 0.5,   // min probability threshold
});

Grammar Sampling

Constrain output to a GBNF grammar for structured text beyond JSON mode:

import { generateText } from '@localmode/core';
import { wllama } from '@localmode/wllama';

const model = wllama.languageModel('Qwen2.5-3B-Instruct-Q4_K_M');

const { text } = await generateText({
  model,
  prompt: 'Generate a valid email address',
  providerOptions: {
    wllama: {
      grammar: `root ::= [a-zA-Z0-9._%+-]+ "@" [a-zA-Z0-9.-]+ "." [a-zA-Z]{2,}`,
    },
  },
});

Model Management

List and clear cached GGUF models stored in the browser's OPFS:

import { listCachedModels, clearAllModelCache, deleteModelCache, isModelCached, refreshModel } from '@localmode/wllama';

// List all cached models
const cached = await listCachedModels();
for (const m of cached) {
  console.log(`${m.name}: ${(m.size / 1024 / 1024).toFixed(1)}MB`);
}

// Delete a specific model
await deleteModelCache('Llama-3.2-1B-Instruct-Q4_K_M');

// Re-download a model (replaces corrupted cache)
await refreshModel('SmolLM2-135M-Instruct-Q4_K_M', {
  onProgress: (p) => console.log(`${p.progress}%`),
});

// Clear everything
await clearAllModelCache();

LoRA Adapters

Load fine-tuned LoRA adapters alongside the base model:

const model = wllama.languageModel('Llama-3.2-1B-Instruct-Q4_K_M', {
  loraAdapters: [
    { path: 'https://example.com/my-lora-adapter.gguf', scale: 1.0 },
  ],
});

Set loraInitWithoutApply: true to initialize adapters without applying them (for manual control).

Extended Sampling Parameters

Fine-grained control over generation via providerOptions.wllama:

const { text } = await generateText({
  model,
  prompt: 'Write a creative story.',
  providerOptions: {
    wllama: {
      top_k: 40,                    // top-K sampling (limit token pool)
      repeat_penalty: 1.1,          // repetition penalty
      repeat_last_n: 64,            // window for repetition penalty
      min_p: 0.05,                  // minimum probability threshold
      seed: 42,                     // reproducible generation
      penalty_freq: 0.5,            // frequency penalty
      penalty_present: 0.5,         // presence penalty
      typ_p: 0.95,                  // locally typical sampling
      dynatemp_range: 0.2,          // dynamic temperature range
      dynatemp_exponent: 1.0,       // dynamic temperature exponent
      mirostat: 2,                  // Mirostat sampling (0=off, 1=v1, 2=v2)
      mirostat_tau: 5.0,            // Mirostat target entropy
      mirostat_eta: 0.1,            // Mirostat learning rate
      logit_bias: [[128, -100]],    // bias specific token logits
      samplers_sequence: 'tkpm',    // custom sampler order
      n_probs: 5,                   // return top-N token probabilities
    },
  },
});

Curated Model Catalog

The package ships 30 curated GGUF models optimized for browser use: 25 language models (including 4 vision-language and 2 reasoning), 3 embedding models, and 2 reranker models. Use catalog keys as shorthand model IDs:

import { wllama } from '@localmode/wllama';

// Use a catalog key directly
const model = wllama.languageModel('Llama-3.2-1B-Instruct-Q4_K_M');

Vision-Language Models

Model	Size	Context	Vision	Description
`Holo2-4B-Q4_K_M`	2.8GB	256K	Yes	Hcompany Holo2 4B UI-grounding VLM, best for browser-agent / GUI navigation
`Holo2-8B-Q4_K_M`	5.1GB	256K	Yes	Hcompany Holo2 8B premium UI-grounding VLM, highest-quality grounding
`Gemma-4-E2B-IT-Q4_K_M`	3.46GB	131K	Yes	Google Gemma 4 E2B IT, 5.1B params (2.3B effective PLE), vision + tool calling
`Gemma-4-E4B-IT-Q4_K_M`	5.41GB	131K	Yes	Google Gemma 4 E4B IT, 8B params (~4B effective PLE), vision + tool calling

Text Models (Selected)

Model	Size	Context	Description
`SmolLM2-135M-Instruct-Q4_K_M`	70MB	8K	Tiniest GGUF model, instant loading
`Llama-3.2-1B-Instruct-Q4_K_M`	750MB	128K	Llama 3.2 1B, great for simple tasks
`Qwen2.5-3B-Instruct-Q4_K_M`	1.94GB	32K	High quality multilingual generation
`Phi-4-mini-instruct-Q4_K_M`	2.3GB	4K	Microsoft Phi-4, strong reasoning
`Llama-3.1-8B-Instruct-Q4_K_M`	4.92GB	128K	Best quality for capable devices

Embedding Models

Model	Size	Dimensions	Description
`nomic-embed-text-v1.5-Q4_K_M`	78MB	768	Nomic Embed Text v1.5, high-quality semantic search
`mxbai-embed-large-v1-Q4_K_M`	197MB	1024	MxBai Embed Large v1, top-quality English embeddings
`bge-small-en-v1.5-Q8_0`	35MB	384	BAAI BGE Small, lightweight on-device embeddings

See WLLAMA_MODELS in the source for the full catalog of all 30 models.

Catalog Fields

Each WllamaModelEntry has optional capability fields:

vision?: boolean + mmprojUrl?: string — multimodal (text + image) models with vision projection
supportsToolCalling?: boolean — models verified for OAI-compatible tool calling
isEmbeddingModel?: boolean + dimensions?: number — embedding-only models
isRerankerModel?: boolean — cross-encoder reranker models (used with wllama.reranker())
supportsReasoning?: boolean — reasoning/thinking models (e.g., DeepSeek-R1)
nGpuLayers?: number — recommended GPU layer count for WebGPU acceleration

Browser Compatibility Check

Check if a model can run on the current device:

import { checkGGUFBrowserCompatFromURL } from '@localmode/wllama';

const result = await checkGGUFBrowserCompatFromURL(
  'bartowski/Llama-3.2-1B-Instruct-GGUF:Llama-3.2-1B-Instruct-Q4_K_M.gguf'
);

if (result.canRun) {
  console.log('Ready to run!', result.estimatedSpeed);
} else {
  console.log('Warnings:', result.warnings);
  console.log('Suggestions:', result.recommendations);
}

CORS Multi-Threading

For 2-4x faster inference, add these HTTP headers to your server:

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

Without these headers, wllama falls back to single-threaded mode automatically.

Safari / iOS Compatibility

wllama v3 requires the WebAssembly Memory64 proposal, which is not yet available in Safari or iOS browsers. To support Safari users, install the optional compatibility package:

pnpm add @wllama/wllama-compat

This package provides a fallback WASM build that works without Memory64. No code changes needed — wllama detects and uses the compat build automatically when available.

Browser	Status
Chrome 80+	Supported
Edge 80+	Supported
Firefox 141+	Supported (Windows), 147+ (macOS Apple Silicon)
Safari	Requires `@wllama/wllama-compat`
iOS Safari	Requires `@wllama/wllama-compat`

Chrome Extension Support (MV3)

The wllama WASM binaries can be bundled in Chrome Manifest V3 extensions. When chrome.runtime.getURL is available, the provider automatically resolves WASM paths via the extension runtime instead of fetching from a CDN (which would be blocked by MV3's "no remotely hosted code" rule).

Bundle the WASM file under wllama-wasm/ in your extension:

your-extension/
  wllama-wasm/
    wllama.wasm

No additional configuration is needed -- the provider detects the extension context automatically.

Output Token IDs

Note: As of wllama v3, per-token IDs are no longer exposed via the OAI-compatible API. The outputTokenIds extension is not available. If you need raw token access for cross-modal use cases (e.g., Orpheus TTS), consider using a provider that exposes token-level output.

Documentation

Full documentation at localmode.dev/docs/wllama.

Acknowledgments

This package is built on wllama by ngxson and llama.cpp by Georgi Gerganov — GGUF model inference via llama.cpp compiled to WebAssembly. GGUF metadata parsing uses @huggingface/gguf by HuggingFace.

License

MIT

Navigation Menu

Uh oh!

@localmode/wllama

Features

Installation

Quick Start

True Streaming

GGUF Metadata Inspection

GGUF Model Discovery

Embedding Models

Reranking

Tool Calling

Structured Output / JSON Mode

Vision / Multimodal

Audio Input (Experimental)

WebGPU Acceleration

Jinja Chat Templates

Reasoning Mode

Performance Configuration

KV Cache Quantization

Flash Attention

Speculative Decoding

Grammar Sampling

Model Management

LoRA Adapters

Extended Sampling Parameters

Curated Model Catalog

Vision-Language Models

Text Models (Selected)

Embedding Models

Catalog Fields

Browser Compatibility Check

CORS Multi-Threading

Safari / iOS Compatibility

Chrome Extension Support (MV3)

Output Token IDs

Documentation

Acknowledgments

License

Uh oh!

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

@localmode/wllama

Features

Installation

Quick Start

True Streaming

GGUF Metadata Inspection

GGUF Model Discovery

Embedding Models

Reranking

Tool Calling

Structured Output / JSON Mode

Vision / Multimodal

Audio Input (Experimental)

WebGPU Acceleration

Jinja Chat Templates

Reasoning Mode

Performance Configuration

KV Cache Quantization

Flash Attention

Speculative Decoding

Grammar Sampling

Model Management

LoRA Adapters

Extended Sampling Parameters

Curated Model Catalog

Vision-Language Models

Text Models (Selected)

Embedding Models

Catalog Fields

Browser Compatibility Check

CORS Multi-Threading

Safari / iOS Compatibility

Chrome Extension Support (MV3)

Output Token IDs

Documentation

Acknowledgments

License