Skip to content

Llama.md

Latest commit

 

History

History
912 lines (719 loc) · 44.6 KB

Llama.md

File metadata and controls

912 lines (719 loc) · 44.6 KB
Raw
title Llama Class
module_name llama_cpp.llama
source_file llama_cpp/llama.py
class_name Llama
last_updated 2026-09-03
version_target latest

Overview

The Llama class is the core, high-level Python wrapper for a llama.cpp model. It handles model loading, memory management (KV cache), tokenization, and generation (both base text completion and chat formatting). It includes advanced features like dynamic LoRA routing, dual-mode hybrid/recurrent checkpointing, speculative decoding, and context shifting.

Role in the Library

Llama is the main user-facing entry point for loading a GGUF model and creating a native llama.cpp context. It exposes completion, chat, tokenization, embedding, state, sampling, and runtime configuration APIs through one managed object.

Use Llama when one application needs a general-purpose model interface. For embedding-only applications, LlamaEmbedding provides embedding-oriented defaults and additional reranking helpers while inheriting the same model and context lifecycle.

Constructor (__init__)

Initialize the model and context. Normal inference construction allocates model and context resources according to the loading and offloading parameters. vocab_only=True avoids loading weight tensors, while no_alloc=True simulates model allocation for inspection and planning rather than inference.

Core Model & Hardware Parameters

Parameter Type Default Description
model_path str Required Model file path (GGUF format)
mmproj_path Optional[str] None Optional multimodal projection GGUF. When provided, Llama creates a generic MTMD chat handler; it replaces an explicitly supplied chat_handler. Prefer passing it by keyword.
n_gpu_layers Union[int, Literal["auto", "all"]] "auto" Maximum number of model layers stored in VRAM:
• auto/-1: auto-selected by llama.cpp
• all/-2: all possible layers
• positive integer N: offload up to N layers
• 0: disable layer offload
cpu_moe bool False Whether to keep all MoE weights on CPU
n_cpu_moe int 0 Number of first N MoE layers to keep on CPU (compatible with cpu_moe)
split_mode int LLAMA_SPLIT_MODE_LAYER Model GPU split mode:
• LLAMA_SPLIT_MODE_NONE: single GPU
• LLAMA_SPLIT_MODE_ROW: row-level split
• LLAMA_SPLIT_MODE_LAYER: layer-level split
load_mode int (llama_load_mode) LLAMA_LOAD_MODE_AUTO How model data is loaded. AUTO lets llama.cpp choose from device capabilities; the explicit LLAMA_LOAD_MODE_* values are described below.
lazy_mode int (llama_lazy_mode) LLAMA_LAZY_MODE_AUTO Controls on-demand reads for architecture-marked tensors when mmap is active.
main_gpu int 0 With LLAMA_SPLIT_MODE_NONE, selects the GPU for the whole model. With ROW, selects the GPU for small tensors and intermediate results. It is ignored with LAYER.
tensor_split Optional[List[float]] None Proportional split of tensors across GPUs (max LLAMA_MAX_DEVICES).
kv_overrides Optional[Dict[str, Union[bool, int, float, str]]] None Key-value overrides for the model metadata.
use_mmap, use_direct_io, use_mlock bool False Deprecated compatibility arguments. They no longer configure native loading; use load_mode.
vocab_only bool False Load model metadata and vocabulary without weight tensors.
check_tensors bool False Validate tensor data while loading the model. This increases load time.
use_extra_bufts bool True Allow extra backend buffer types, including supported weight-repacking paths.
no_host bool False Bypass the ordinary host buffer so compatible extra buffer types can be used. Advanced backend option.
no_alloc bool False Load metadata and simulate model allocation without allocating tensor data. Intended for inspection and planning rather than inference.
load_mtp bool False Load the target model's NextN/MTP tensors. This is enabled automatically for built-in MTP through speculative; normally it should not be set manually.
numa Union[bool, int] False NUMA strategy (e.g., GGML_NUMA_STRATEGY_DISTRIBUTE).

Model Load Modes

load_mode replaces the legacy use_mmap, use_direct_io, and use_mlock arguments. It accepts a member of llama_cpp.llama_load_mode:

Value Integer Description
LLAMA_LOAD_MODE_AUTO -1 Default. Auto-detect the loading behavior from device capabilities; currently resolves to mmap where supported and ordinary loading otherwise.
LLAMA_LOAD_MODE_NONE 0 Use no special model-loading mode.
LLAMA_LOAD_MODE_MMAP 1 Memory-map the model.
LLAMA_LOAD_MODE_MLOCK 2 Keep the loaded model in RAM rather than allowing it to be swapped or compressed.
LLAMA_LOAD_MODE_MMAP_MLOCK 3 Memory-map the model and keep its mapped pages in RAM.
LLAMA_LOAD_MODE_DIRECT_IO 4 Use direct I/O when it is available.

lazy_mode accepts a member of llama_cpp.llama_lazy_mode:

Value Integer Description
LLAMA_LAZY_MODE_OFF 0 Read complete tensors up front.
LLAMA_LAZY_MODE_AUTO 1 Default. Lazily read architecture-marked tensors only when they are larger than 4 GiB. Requires mmap.
LLAMA_LAZY_MODE_ON 2 Read rows of every architecture-marked tensor on demand. Requires mmap. 
import llama_cpp

llm = llama_cpp.Llama(
    model_path="models/model.gguf",
    load_mode=llama_cpp.llama_load_mode.LLAMA_LOAD_MODE_MMAP_MLOCK,
    lazy_mode=llama_cpp.llama_lazy_mode.LLAMA_LAZY_MODE_AUTO,
)

The legacy loading arguments are retained only for call compatibility. They no longer configure the underlying model parameters and may emit a deprecation warning; set load_mode explicitly instead. Use the following migration mapping:

Legacy configuration Replacement
use_mmap=False, use_mlock=False load_mode=LLAMA_LOAD_MODE_NONE
use_mmap=True, use_mlock=False load_mode=LLAMA_LOAD_MODE_MMAP
use_mmap=False, use_mlock=True load_mode=LLAMA_LOAD_MODE_MLOCK
use_mmap=True, use_mlock=True load_mode=LLAMA_LOAD_MODE_MMAP_MLOCK
use_direct_io=True load_mode=LLAMA_LOAD_MODE_DIRECT_IO

Context & Batch Parameters

Parameter Type Default Description
seed int LLAMA_DEFAULT_SEED RNG seed used by the model. LLAMA_DEFAULT_SEED requests a random seed.
n_ctx int 512 Text context size. Set to 0 to load from model metadata.
n_keep int 256 Preferred number of leading tokens to preserve during automatic context shifting.
n_batch int 2048 Maximum number of tokens in a logical prompt-processing batch. The effective value cannot exceed n_ctx.
n_ubatch int 512 Maximum number of tokens in a physical micro-batch processed by llama.cpp.
n_seq_max int 1 Maximum independent sequence states in one decode batch. Embedding calls split automatically at this limit; larger values enable more parallel sequences.
n_rs_seq int 0 Experimental recurrent-state snapshots retained per sequence for rollback. 0 disables rollback snapshots.
n_outputs_max int 0 Maximum outputs in a physical batch. 0 lets llama.cpp use the effective n_batch.
n_outputs_max_per_seq int 1 Maximum outputs per sequence. 0 lets llama.cpp use the effective n_outputs_max.
n_threads Optional[int] None Number of threads for generation (defaults to CPU count // 2).
n_threads_batch Optional[int] None Number of threads for batch processing (defaults to CPU count).
ctx_type int LLAMA_CONTEXT_TYPE_DEFAULT Context implementation selected by llama.cpp. Keep the default unless a model or backend requires another context type.

Embedding, Attention & KV Parameters

Parameter Type Default Description
embeddings bool False Enable embedding extraction alongside logits. Must be True before calling embed() or create_embedding().
pooling_type int LLAMA_POOLING_TYPE_UNSPECIFIED Pooling strategy for embedding output. UNSPECIFIED follows model metadata, NONE returns token-level vectors, and RANK returns classifier or reranking output.
attention_type Optional[int] LLAMA_ATTENTION_TYPE_UNSPECIFIED Attention mode used by the context. None is normalized to UNSPECIFIED, which lets llama.cpp select the model-compatible behavior.
logits_all bool False Retain logits for every evaluated token instead of only requested outputs. Completion log probabilities require this mode.
flash_attn_type Optional[int] LLAMA_FLASH_ATTN_TYPE_AUTO Controls when Flash Attention is enabled. None is normalized to AUTO.
rope_scaling_type Optional[int] LLAMA_ROPE_SCALING_TYPE_UNSPECIFIED RoPE scaling strategy. The unspecified value follows model metadata.
rope_freq_base float 0.0 RoPE base frequency override. 0.0 follows model metadata.
rope_freq_scale float 0.0 RoPE frequency scaling override. 0.0 follows model metadata.
yarn_ext_factor float -1.0 YaRN extrapolation mix factor. A negative value follows model metadata.
yarn_attn_factor float 1.0 YaRN attention magnitude scaling factor.
yarn_beta_fast float 32.0 YaRN low-correction dimension.
yarn_beta_slow float 1.0 YaRN high-correction dimension.
yarn_orig_ctx int 0 Original YaRN context size. 0 follows model metadata.
offload_kqv bool True Offload K, Q, and V tensor operations to the selected device when supported.
no_perf bool False Disable native performance timing collection when True.
op_offload Optional[bool] None Whether supported host tensor operations may be offloaded to a device. None keeps llama.cpp's default.
swa_full Optional[bool] None Use a full-size sliding-window-attention cache. None keeps llama.cpp's default.
kv_unified Optional[bool] None Use a unified KV buffer for all sequences. LlamaEmbedding enables this automatically.
type_k / type_v Optional[int] None KV cache data types for keys and values. None uses llama.cpp defaults.

Advanced, Chat & Sampling Parameters

Parameter Type Default Description
chat_format Optional[str] None String specifying the chat template (e.g., "llama-2", "chatml"). Guessed from GGUF if None.
chat_handler Optional[LlamaChatCompletionHandler] None Optional custom handler. See [[ChatHandlers]].
chat_template_name Optional[str] None Named chat template passed to the generic MTMD handler created by mmproj_path.
chat_handler_kwargs Dict[str, Any] {} Additional keyword arguments passed to the generic MTMD chat handler created by mmproj_path.
tokenizer Optional[BaseLlamaTokenizer] None Override the tokenizer used by the high-level API. By default, LlamaTokenizer wraps the loaded model vocabulary.
draft_model Optional[LlamaDraftModel] None Deprecated stateless draft callback kept for compatibility. New code should use speculative.
speculative Optional[Union[SpecConfig, LlamaSpecEngine]] None Stateful speculative configuration or engine. Supports the complete begin/draft/process/accept lifecycle; it cannot be combined with draft_model.
ctx_checkpoints int 16 Max hybrid/recurrent context checkpoints to keep. Set to 0 to disable checkpointing for single-turn fast paths.
checkpoint_interval int 4096 Token interval for saving periodic Hybrid/Recurrent checkpoints during long prompt evaluation.
checkpoint_on_device bool False Store Hybrid/Recurrent checkpoint tensor payloads in llama_context-owned device buffers via LLAMA_STATE_SEQ_FLAGS_ON_DEVICE. Reduces device-to-host copy overhead, but only one active checkpoint per seq_id is safe.
last_n_tokens_size int 64 Default recent-token history used by repetition-style samplers when their per-call history length is zero.
spm_infill bool False Use the Suffix/Prefix/Middle order for infill instead of Prefix/Suffix/Middle.

Runtime Logging Parameters

Parameter Type Default Description
verbose bool True Backward-compatible boolean native logging switch. False keeps only error-level llama.cpp / ggml logs; True enables debug-level native logs. If verbosity is provided, verbosity takes precedence over verbose.
verbosity Optional[Union[int, str, bool]] None Fine-grained llama.cpp-style native runtime log verbosity. Numeric levels: 0=output, 1=error, 2=warning, 3=info, 4=trace, 5=debug. Use verbosity=3 for llama.cpp-style default info logs. String aliases such as "silent", "quiet", "info", "trace", and "debug" are also accepted.
log_filters Optional[Sequence[str]] None Optional substring filters for native runtime logs. If any provided substring appears in a decoded backend log message, that message is suppressed. The default logger may include built-in filters for noisy low-level logs such as CUDA Graph id %d reuse messages. Pass an empty list [] to disable default substring filtering.
log_filters_case_sensitive bool True Whether log_filters should match case-sensitively. Defaults to True for predictable low-level backend log filtering.

Unknown extra constructor keywords are accepted through **kwargs for compatibility but are not applied to model or context parameters. Use explicit named parameters from the current signature.

Core Methods

create_chat_completion

Generates a chat response using the configured chat_format or chat_handler. It also accepts presence_penalty as a compatibility alias for the primary present_penalty parameter; an explicitly non-default present_penalty takes precedence.

import llama_cpp

model = llama_cpp.Llama(model_path="models/qwen2.5-7b-instruct.gguf", n_gpu_layers=-1)

response = model.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain KV caching."}
    ],
    temperature=0.7,
    max_tokens=2048
)
print(response["choices"][0]["message"]["content"])

Structured JSON Output

Pass response_format={"type": "json_object"} to constrain the response to valid JSON. Add a schema property when the output must follow a specific JSON Schema:

response = model.create_chat_completion(
    messages=[{"role": "user", "content": "Extract: Ada is 36 years old."}],
    response_format={
        "type": "json_object",
        "schema": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "age": {"type": "integer"},
            },
            "required": ["name", "age"],
        },
    },
    temperature=0.0,
)

The schema is converted to a llama.cpp grammar. Keep schemas small and explicit for better reliability with compact models.

Tool Calling

Chat handlers and model templates that support tool calling accept OpenAI-style tools and tool_choice arguments:

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"],
            },
        },
    }
]

response = model.create_chat_completion(
    messages=[{"role": "user", "content": "What is the weather in Shanghai?"}],
    tools=tools,
    tool_choice="auto",
)

tool_calls = response["choices"][0]["message"].get("tool_calls")

Tool syntax and automatic selection depend on the model's chat template. Pass a specific tool_choice object to force one named function when the active handler supports it.

create_chat_completion_openai_v1

This method accepts the same arguments as create_chat_completion() but returns typed objects from the OpenAI Python package instead of dictionaries. Install the optional dependency first with pip install openai.

response = model.create_chat_completion_openai_v1(
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
    max_tokens=64,
)

print(response.choices[0].message.content)

With stream=True, it returns an iterator of typed ChatCompletionChunk objects.

The OpenAI SDK's typed finish_reason currently does not include the local "abort" extension. If an interrupted response is converted through this method, the SDK may raise a validation error. Use create_chat_completion() directly when the caller needs to handle finish_reason="abort".

create_completion / __call__

Generates standard text completion from a raw string prompt.

For OpenAI naming compatibility, both methods accept presence_penalty as an optional alias. Internally it is normalized to the primary present_penalty parameter only when present_penalty remains at its default value.

import llama_cpp

model = llama_cpp.Llama(model_path="models/llama-3-8b.gguf")
output = model("The capital of Japan is", max_tokens=10, stop=["\n"])
print(output["choices"][0]["text"])

generate

A low-level generator yielding token IDs one by one. Highly customizable with sampling parameters, dynamic LoRA mounting, and control vectors.

import llama_cpp

model = llama_cpp.Llama(model_path="models/llama-3-8b.gguf")
tokens = model.tokenize(b"def fibonacci(n):")

for token in model.generate(tokens, top_k=40, top_p=0.95, temp=0.2):
    print(model.detokenize([token]).decode('utf-8'), end="", flush=True)

eval

Low-level method to ingest and evaluate a sequence of tokens. Used internally to update context memory and logits. When the token count exceeds n_ctx, it attempts Context Shifting by discarding older tokens while preserving up to n_keep leading tokens. It raises RuntimeError if the backend memory cannot be shifted or the incoming chunk cannot fit.

# Evaluates a chunk of tokens and updates internal state
model.eval(tokens=[1, 453, 234, 987], active_loras=[{"name": "coding_adapter", "scale": 1.0}])

If the native abort callback interrupts an in-flight llama_decode() call, eval() fully resets the context and raises the internal LlamaDecodeAbort exception. Direct low-level callers should treat that exception as the end of the current request. The full reset is necessary because llama.cpp may already have committed an unknown number of physical micro-batches while the Python token ledger still represents the pre-call state.

reset

Clears the evaluated sequence state owned by the active context. This includes native KV/recurrent memory, the Python token cursor, cached output boundaries, hybrid checkpoints, and speculative-engine state. It does not unload model weights, reset sampling configuration, or clear a separately configured prompt-cache object.

abort

Requests cancellation of active generation. abort() sets both the Python generation event and the native callback flag:

  • The Python event is checked at generation boundaries on every backend.
  • The native callback can interrupt an in-flight llama_decode() graph, but llama.cpp currently documents this path as CPU-only.
  • A new high-level completion clears the previous abort flags before starting.

For create_completion() and chat handlers that delegate to the standard text completion path, both streaming and non-streaming responses finish with "finish_reason": "abort". A native mid-decode abort also performs a full context reset, because completed micro-batches cannot be inferred reliably. Consequently, the interrupted request loses its reusable KV/checkpoint state and is not written to the configured prompt cache. Text already emitted by a stream remains valid output, but a later request must evaluate its prompt again.

For a native mid-decode abort, generate() ends iteration normally after the reset. If cancellation is observed only at a Python generation boundary, it can stop without forcing that full reset. Direct eval() and LlamaContext.decode() callers instead receive LlamaDecodeAbort; the latter is an advanced internal API and does not perform the high-level reset itself.

Call abort() from another thread, such as a timeout timer or UI cancellation handler. Do not use one Llama instance for concurrent generation requests; the abort flags and context state belong to the instance.

External Threadpool Management

attach_threadpool(threadpool, threadpool_batch=None) attaches externally created ggml_threadpool handles to the native context. The first pool is used for single-token generation. When threadpool_batch is omitted, llama.cpp uses the first pool for batch/prompt processing too.

detach_threadpool() waits for pending context work and removes both handles. Attaching also synchronizes first, so a pool cannot be replaced while work is still pending. The Python wrapper keeps strong references to the supplied objects while attached, but ownership stays with the caller: it neither creates nor frees the native pools.

The package currently exposes the opaque ggml_threadpool_p handle and the attach/detach calls, but not ggml_threadpool_new() or ggml_threadpool_free(). Obtain and release pools through the native extension or integration that created them. Detach before freeing an external pool, and keep the Llama context alive while the pool is attached. Applications that do not need shared or specially configured pools should rely on llama.cpp's automatic threadpool creation and the n_threads / n_threads_batch constructor parameters.

Runtime Logging Control

The Llama class exposes lightweight runtime helpers for adjusting native llama.cpp / ggml logging after initialization.

Note: Native backend logging is process-global because llama.cpp / ggml use a global log callback. Changing verbosity or log filters affects all Llama instances in the current Python process.

  • set_verbosity(verbosity: Union[int, str, bool, None]): Set native runtime log verbosity.
  • get_verbosity() -> int: Return the current native runtime log verbosity.
  • set_log_filters(filters: Sequence[str], case_sensitive: bool = True): Replace substring filters for native runtime logs.
  • add_log_filters(filters: Sequence[str]): Append substring filters.
  • get_log_filters() -> List[str]: Return the current substring filters.
  • clear_log_filters(): Clear all substring filters, including default filters.
  • reset_log_filters(): Restore default substring filters.
from llama_cpp import Llama

llm = Llama(
    model_path="models/qwen3.gguf",
    verbosity=3,  # llama.cpp-style info logs
)

# Temporarily enable debug-level native logs.
llm.set_verbosity(5)

# Suppress noisy backend messages by substring.
llm.add_log_filters([
    "CUDA Graph",
    "CUDA graph",
    "clip_model_loader: tensor",
])

# Return to quiet error-only logging.
llm.set_verbosity(1)

Dynamic LoRA Management

The Llama class allows you to load multiple LoRAs into VRAM and apply them dynamically per-generation or per-eval.

  • load_lora(name: str, path: str): Loads an adapter into VRAM (does not apply it yet).
  • unload_lora(name: str): Releases the specific LoRA from VRAM.
  • list_loras() -> List[str]: Returns names of all registered LoRAs.
  • unload_all_loras(): Forces VRAM release for all loaded adapters.

Best Practices & Common Patterns

  1. Context Shifting & Prompt Caching:

    By default, generate(reset=True) and create_completion() check for the longest matching prefix in existing context memory. To maximize speed, keep system prompts static and only append new dialogue to avoid re-evaluating the entire history. If the context limit is reached during eval, the model attempts a Context Shift, discarding older tokens while preserving up to n_keep leading tokens. Backends whose memory cannot shift raise an error instead.

  2. Structured Output: Use response_format for JSON or JSON Schema constrained output. See Structured JSON Output for the current API.

  3. Speculative Decoding:

    New code should pass SpecConfig through the speculative argument. This enables the stateful begin/draft/process/accept lifecycle, including verification batches, acceptance feedback, recurrent-state rollback, and per-run statistics.

    The current implementation is text-only and uses sequence ID 0. It supports built-in and external MTP, external DFlash, DFlash2, and DSpark drafts, plus the NGRAM_MAP_K and NGRAM_MAP_K4V lookup engines. Multimodal pseudo-tokens and MTMD embedding batches are not yet supported by this path.

    Built-in MTP heads

    from llama_cpp import Llama
from llama_cpp.llama_speculative import SpecConfig, SpeculativeType

llm = Llama(
    model_path="path/to/model-with-mtp.gguf",
    n_ctx=4096,
    n_batch=512,
    n_gpu_layers=-1,
    speculative=SpecConfig(
        spec_type=SpeculativeType.DRAFT_MTP,
        draft_n_max=2,
        draft_p_min=0.0,
    ),
)

    Omitting draft_model_path makes Llama enable target MTP tensor loading automatically. MTP has been tested with built-in and external MTP models from the Qwen3.5, Qwen3.6, and Qwen3.8 families. A gemma4 target paired with an external, compatible gemma4-assistant GGUF has also been tested; the engine handles its target-context and shared-KV workflow automatically. All of these stateful MTP paths are currently text-only. For Qwen3.8 27B, draft_n_max=2 is a good starting point, but the best value depends on the backend, GPU, quantization, prompt, and sampling settings. Use examples.benchmark.benchmark_speculative to tune it in the deployment environment.

    External MTP model

    llm = Llama(
    model_path="path/to/target.gguf",
    n_batch=512,
    n_gpu_layers=-1,
    speculative=SpecConfig(
        spec_type=SpeculativeType.DRAFT_MTP,
        draft_model_path="path/to/mtp.gguf",
        draft_n_max=2,
        draft_n_gpu_layers="all",
    ),
)

    External DFlash, DFlash2, or DSpark model

    llm = Llama(
    model_path="path/to/target.gguf",
    n_ctx=8192,
    n_batch=512,
    n_gpu_layers="all",
    speculative=SpecConfig(
        # DFlash2 uses DRAFT_DFLASH and is detected from selector metadata.
        spec_type=SpeculativeType.DRAFT_DFLASH,  # or DRAFT_DSPARK
        draft_model_path="path/to/dflash-dflash2-or-dspark.gguf",
        draft_n_max=7,
        draft_p_min=0.0,
        draft_n_gpu_layers="all",
        # Used by DFlash v1/DSpark; remains inactive for DFlash2.
        draft_backend_sampling=True,
    ),
)

    DFlash, DFlash2, and DSpark require a compatible external draft GGUF. Their effective draft length is limited by the GGUF's trained block size. DFlash2 uses DRAFT_DFLASH and is selected when dflash.selector_top_k is positive. draft_p_min filters token probability for DFlash, selector-transition probability for DFlash2, and predicted acceptance confidence for DSpark. DFlash2 reads unmasked selector rows and does not activate backend vocabulary sampling. Benchmark draft length and threshold together; longer blocks only help when their additional accepted tokens outweigh verification cost.

    N-gram lookup

    llm = Llama(
    model_path="path/to/model.gguf",
    n_batch=512,
    speculative=SpecConfig(
        spec_type=SpeculativeType.NGRAM_MAP_K,
        ngram_size_n=8,
        ngram_size_m=16,
    ),
)

    MTP and n-gram draft lengths are independent: draft-family engines use draft_n_max, while K/K4V use ngram_size_m. The implementation keeps the complete [last_verified_token, draft...] verification batch together and limits the effective draft length to n_batch - 1.

    After a generation, llm.last_speculative_stats exposes acceptance, phase timing, checkpoint, rollback, TTFT, and sustained-generation measurements. See [Llama Speculative Decoding] for configuration details, supported engines, benchmark commands, and the exact meaning of each statistic.

  4. Dynamic LoRA Routing:

    You can load multiple LoRAs using load_lora() at startup. Then, pass the active_loras parameter to .generate(), .create_completion(), or .create_chat_completion() to dynamically apply them to specific queries without reloading the base model.

    Multi-LoRA Dynamic Switching Example:

    Load multiple adapters and apply them selectively without reloading the base model.

    llm = Llama(model_path="base_model.gguf")
llm.load_lora("coding", "codellama_adapter.gguf")
llm.load_lora("story", "storywriter_adapter.gguf")
llm.load_lora("sql_expert", "adapters/sql_lora.gguf")

# Use coding adapter
llm.create_completion("def sort:", active_loras=[{"name": "coding", "scale": 1.0}])

# Use story adapter
llm.create_completion("Once upon a time", active_loras=[{"name": "story", "scale": 0.9}])

# Use sql adapter
llm.create_completion("SELECT *", active_loras=[{"name": "sql_expert", "scale": 0.8}])

  5. Hybrid & Recurrent Architectures:

    The class natively detects Hybrid/Recurrent models (for example LFM2VL/LFM2.5VL, Qwen3.5/3.6, Mamba, RWKV, or specialized SWA models such as Gemma3/4) and automatically enables the HybridCheckpointCache.

    Unlike regular Transformer KV caches, Hybrid/Recurrent model memory cannot always be safely truncated token-by-token. The wrapper therefore saves periodic sequence-state checkpoints during long context prefill, allowing rollback to a verified prefix without corrupting recurrent state.

    HybridCheckpointCache supports two checkpoint storage modes:

    • Host checkpoint mode (checkpoint_on_device=False, default): checkpoint payloads are serialized into Python-owned bytes. This supports multiple historical checkpoints per seq_id, which is useful for multi-turn reuse and deeper rollback history.
    • Device checkpoint mode (checkpoint_on_device=True): checkpoint tensor payloads are stored in llama_context-owned device buffers via LLAMA_STATE_SEQ_FLAGS_ON_DEVICE. Python only keeps the host-visible serialized portion. This reduces device-to-host tensor copy overhead, but only one active checkpoint per seq_id is safe because device payloads are keyed by seq_id.

    These prompt-cache checkpoints are distinct from the native recurrent snapshots used by MTP speculative decoding. MTP reserves recurrent snapshot slots from draft_n_max. N-gram engines do not own a native draft context, so rejection on a Hybrid/Recurrent target depends on HybridCheckpointCache; keep ctx_checkpoints greater than zero for that combination, preferably with on-device storage.

    Tips: If you are using a hybrid multimodal model for ComfyUI nodes or single-turn API wrappers without stateful speculative decoding or multi-turn rollback, initialize your Llama instance with ctx_checkpoints=0:

    llm = Llama(
    model_path="./Qwen3.5-VL-9B.gguf",
    chat_handler=MTMDChatHandler(clip_model_path="./mmproj.gguf"),
    n_ctx=4096,
    ctx_checkpoints=0  # Disable checkpoints for zero-latency single-turn fast paths
)

    For long prompts on GPU-backed Hybrid/Recurrent models, you can enable device-backed checkpoints to reduce device-to-host copy overhead:

    llm = Llama(
    model_path="./Qwen3.6-27B.gguf",
    n_ctx=32768,
    n_gpu_layers=-1,
    ctx_checkpoints=16,
    checkpoint_interval=4096,
    checkpoint_on_device=True
)

    Use checkpoint_on_device=False if you need multiple historical checkpoints for the same seq_id. Use checkpoint_on_device=True when fast rollback/checkpointing is more important than keeping many historical checkpoint payloads. Do not disable checkpoints when combining n-gram speculation with a Hybrid/Recurrent target.

  6. Assistant Prefill:

    llama-cpp-python supports Assistant Prefill for message continuation through the assistant_prefill=True parameter of create_chat_completion().

    The default chat-format path copies the input message list, removes the last assistant message before rendering the preceding history, and appends that message's text to the formatted prompt. If the list is empty, the final role is not assistant, or its content is empty, prefill is ignored (with a warning when verbose logging is enabled). Custom chat handlers may implement different behavior.

    from llama_cpp import Llama

llm = Llama(model_path="path/to/model.gguf")

# An interrupted/partial conversation
messages = [
    {"role": "user", "content": "What are the first 5 planets in the solar system?"},
    {"role": "assistant", "content": "The first 5 planets in our solar system are:\n1. Mercury\n2."}
]

# Seamlessly continue the generation
response = llm.create_chat_completion(
    messages=messages,
    max_tokens=50,
    assistant_prefill=True # <--- Enables seamless continuation
)

prefilled_text = messages[-1]["content"]
# The generated text continues after the supplied "2." prefix.
generated_text = response["choices"][0]["message"]["content"]

print(prefilled_text + generated_text)

  7. Interrupting Reasoning & Assistant Prefill (Time-boxing):

    Use abort() alongside assistant_prefill=True to request cancellation when a reasoning model (such as Qwen or DeepSeek) exceeds a time limit, then inject bridge text and request continuation as a final answer.

    A native mid-decode abort resets the context, so the continuation request below re-evaluates the reconstructed message history rather than reusing the interrupted request's KV state. The partial text collected by the caller is preserved in the assistant-prefill message.

    import threading
from llama_cpp import Llama

llm = Llama(model_path="Qwen3.6-27B.gguf", n_ctx=4096, n_gpu_layers=-1)

def run_controlled_generation(prompt: str, timeout_seconds: int = 10):
    messages = [{"role": "user", "content": prompt}]

    # 1. Set a time bomb to interrupt long <think> phases
    def timeout_handler():
        llm.abort()

    timer = threading.Timer(timeout_seconds, timeout_handler)
    timer.start()

    stream = llm.create_chat_completion(
        messages=messages, max_tokens=2048, stream=True
    )

    partial_response = ""
    finish_reason = None

    for chunk in stream:
        finish_reason = chunk["choices"][0].get("finish_reason")

        if finish_reason is not None and finish_reason != "abort":
            timer.cancel()
            break

        if finish_reason == "abort":
            break

        delta = chunk["choices"][0]["delta"].get("content", "")
        if delta:
            partial_response += delta
            print(delta, end="", flush=True)

    # 2. Forced Intervention and Prefill Continuation
    if finish_reason == "abort":
        # Inject bridge text to forcefully close the reasoning tag
        bridge_text = "\n...Wait, I have thought long enough, let's start answering the user.\n</think>\n\n"
        print(bridge_text, end="", flush=True)

        prefilled_content = partial_response + bridge_text
        messages.append({"role": "assistant", "content": prefilled_content})

        # Use assistant_prefill=True to seamlessly continue the text block
        stream_part2 = llm.create_chat_completion(
            messages=messages,
            max_tokens=2048,
            stream=True,
            assistant_prefill=True
        )

        for chunk in stream_part2:
            delta = chunk["choices"][0]["delta"].get("content", "")
            if delta:
                print(delta, end="", flush=True)

run_controlled_generation("Explain quantum mechanics in a way that relates to bugs in code.", timeout_seconds=8)

  8. Runtime Logging & Backend Noise Filtering:

    Llama supports fine-grained native llama.cpp / ggml logging through verbosity. This is more precise than the legacy verbose boolean flag.

    from llama_cpp import Llama

# Legacy behavior:
# verbose=False -> error-only logs
llm_quiet = Llama(
    model_path="models/qwen3.gguf",
    verbose=False,
)

# Recommended precise logging:
# 0 = output, 1 = error, 2 = warning, 3 = info, 4 = trace, 5 = debug
llm = Llama(
    model_path="models/qwen3.gguf",
    verbosity=3,  # llama.cpp-style default info logs
)

    For low-level debugging, use verbosity=5. By default, the logger may suppress known noisy backend messages such as CUDA Graph reuse logs. Pass log_filters=[] to disable all substring filtering.

    llm = Llama(
    model_path="models/qwen3.gguf",
    verbosity=5,
    log_filters=[],  # show all debug logs, including normally filtered ones
)

    To suppress additional noisy messages, pass substring filters:

    llm = Llama(
    model_path="models/qwen3.gguf",
    verbosity=5,
    log_filters=[
        "CUDA Graph id",
        "clip_model_loader: tensor",
        "ggml_cuda_graph_update_required",
    ],
)

    You can also adjust logging at runtime:

    llm.set_verbosity(5)
llm.add_log_filters(["llama_perf_context_print"])

# Later, return to warning-level logs.
llm.set_verbosity(2)

    Important: native backend logging is process-global. Runtime changes affect all Llama instances in the same Python process.

    verbose=False vs. verbosity=0: These have distinct behaviors.

    • verbose=False silences Python wrapper prints but not backend diagnostics; like if self.verbose: print()
    • verbosity=0 silences all backend non-error output.

Embeddings

The Llama embedding methods are maintained and use streaming batches. Create the model with embeddings=True before calling them.

from llama_cpp import Llama, LLAMA_POOLING_TYPE_UNSPECIFIED

llm = Llama(
    model_path="path/to/embedding-model.gguf",
    embeddings=True,
    pooling_type=LLAMA_POOLING_TYPE_UNSPECIFIED,
    n_batch=512,
    n_ubatch=512,
    n_seq_max=8,
    kv_unified=True,
)

try:
    # Raw sequence embeddings with explicit L2 normalization.
    vectors = llm.embed(["query", "document"], normalize=2)

    # OpenAI-compatible response.
    response = llm.create_embedding(
        ["query", "document"],
        normalize=True,
    )
finally:
    llm.close()

embed(input, normalize=False, truncate=True, separator=None, return_count=False)

Generate raw embedding values for strings or pre-tokenized inputs.

Parameter Type Default Description
input Union[str, List[str], List[List[int]]] Required A single string, a list of strings, or a list containing pre-tokenized token-ID lists.
normalize Union[bool, int] False False returns raw values, while True applies L2 normalization. Integer modes are listed below. Rank outputs are not normalized.
truncate bool True Truncate each input to the smaller of the context capacity and logical batch capacity. If disabled, an input longer than n_batch raises ValueError.
separator Optional[str] None Split a single string into multiple independent inputs. When set, the result uses the batch return shape.
return_count bool False Return (result, total_token_count) instead of only the embedding result.

Normalization modes follow the llama.cpp embedding example:

Value Behavior
False or -1 No normalization
True or 2 Euclidean/L2 normalization
0 Scale by the maximum absolute value to a maximum magnitude of 32760
1 Taxicab/L1 normalization
Integer greater than 2 Corresponding p-norm normalization

Unlike LlamaEmbedding.embed(), the standard Llama.embed() method defaults to raw, unnormalized output for backward compatibility.

The return shape depends on the input and pooling type:

Input / pooling mode Return shape
Single string with sequence pooling List[float]
String list or separator-split string with sequence pooling List[List[float]]
LLAMA_POOLING_TYPE_NONE One token embedding matrix per input: List[List[float]] for a single string or List[List[List[float]]] for a batch
LLAMA_POOLING_TYPE_RANK with one classifier output A scalar for a single string or a list of scalars for a batch
LLAMA_POOLING_TYPE_RANK with multiple classifier outputs A classifier vector for each input
Any mode with return_count=True (result, total_token_count)

Use LLAMA_POOLING_TYPE_UNSPECIFIED for ordinary sentence embeddings unless the model documentation requires a specific sequence pooling strategy. LLAMA_POOLING_TYPE_NONE is token-level output and should not be used when one vector per input document is expected.

create_embedding(input, model=None, normalize=False, truncate=True)

Wrap sequence or token-level embedding output in an OpenAI-compatible response:

{
    "object": "list",
    "data": [
        {
            "object": "embedding",
            "embedding": [...],
            "index": 0,
        }
    ],
    "model": "path/to/embedding-model.gguf",
    "usage": {
        "prompt_tokens": 12,
        "total_tokens": 12,
    },
}
Parameter Type Default Description
input Union[str, List[str]] Required One string or a list of strings.
model Optional[str] None Model name placed in the response. Defaults to model_path.
normalize Union[bool, int] False Passed directly to embed().
truncate bool True Passed directly to embed().

For parallel batches, n_seq_max must cover every sequence ID active in a single decode batch. The default n_seq_max=1 is valid and processes multiple inputs sequentially. Increasing it allows more inputs to be decoded in parallel; for example, n_seq_max=8 permits IDs 0 through 7 in one batch. n_batch limits logical input tokens, n_ubatch controls the physical token batch, and n_seq_max limits independent sequences.

LlamaEmbedding remains available as the specialized convenience class. It automatically enables embedding-oriented context options, defaults to L2 normalization, provides additional output formats, and adds the rank() helper for formatting query/document pairs.

OpenAI compatibility: use sequence pooling when calling create_embedding() through an OpenAI-compatible client. Token-level pooling (LLAMA_POOLING_TYPE_NONE) produces nested token vectors rather than the single flat vector normally expected for each input.

Related Links

  • [Index-Home]
  • [Llama Cache] - Implementing disk or RAM-based prompt caching (LlamaRAMCache, TrieCache, HybridCheckpointCache).
  • [Llama Embedding] - Dedicated class for text embeddings and reranking.
  • [Llama Speculative Decoding] - Configuring stateful MTP, DFlash, DFlash2, DSpark, and n-gram speculative engines, rollback, statistics, and benchmarks.
  • [llama.cpp ctypes Bindings] - Source pointers for the low-level llama.cpp and ggml bindings.
  • [DFlash2 Speculative Decoding] - Runnable Qwen3.8 DFlash2 configuration, validation, benchmark, and tuning workflow.
  • [[ChatHandlers]] - Customizing LlamaChatCompletionHandler for function calling and vision/omni models (e.g., [[Gemma4ChatHandler]], [[Qwen35ChatHandler]]).