HybridLanguageModel API Reference

The HybridLanguageModel<D> struct combines n-gram and embedding models for robust language modeling with OOV handling.

Overview

Hybrid models leverage the strengths of both approaches:

N-gram models provide accurate probabilities for seen n-grams
Embedding models provide semantic similarity for OOV words
Configurable interpolation balances the two components

Type Parameters

Parameter	Description
`D`	Dictionary backend implementing `MutableMappedDictionary<Value = NgramEntry> + Send + Sync`

Construction

From Pre-trained Components

use libgrammstein::hybrid::{HybridLanguageModel, HybridConfig};
use libgrammstein::ngram::NgramModel;
use libgrammstein::embedding::SubwordEmbedding;

let ngram_model = NgramModel::load("ngram.bin")?;
let embedding_model = SubwordEmbedding::load("embeddings.bin")?;

// With custom configuration
let config = HybridConfig::default();
let hybrid = HybridLanguageModel::new(ngram_model, embedding_model, config);

// With defaults
let hybrid = HybridLanguageModel::with_defaults(ngram_model, embedding_model);

Loading from File

use libgrammstein::hybrid::HybridLanguageModel;
use liblevenshtein::dictionary::dynamic_dawg_char::DynamicDawgChar;

// Binary format (requires serde-extras feature)
let model: HybridLanguageModel<DynamicDawgChar<NgramEntry>> =
    HybridLanguageModel::load("hybrid.bin")?;

// Portable format
let model = HybridLanguageModel::load_portable(
    "hybrid.portable.bin",
    || DynamicDawgChar::new()
)?;

Configuration

HybridConfig

use libgrammstein::hybrid::{HybridConfig, InterpolationStrategy};

let config = HybridConfig {
    strategy: InterpolationStrategy::Linear { alpha: 0.8 },
    cache_size: 50_000,
    embedding_smoothing: 1e-8,
    temperature: 1.0,
};

Field	Type	Default	Description
`strategy`	`InterpolationStrategy`	`Linear { alpha: 0.8 }`	How to combine scores
`cache_size`	`usize`	`50_000`	LRU cache size for scores
`embedding_smoothing`	`f64`	`1e-8`	Smoothing for embedding probabilities
`temperature`	`f64`	`1.0`	Temperature for similarity-to-probability

Interpolation Strategies

Linear Interpolation

Combines probabilities: P = α * P_ngram + (1-α) * P_embedding

InterpolationStrategy::Linear { alpha: 0.8 }

Log-Linear Interpolation

Combines log probabilities: log P = α * log P_ngram + (1-α) * log P_embedding

InterpolationStrategy::LogLinear { alpha: 0.7 }

N-gram with Embedding Fallback

Uses n-gram for known words, embedding for OOV:

InterpolationStrategy::NgramWithEmbeddingFallback

Dynamic Weighting

Adjusts weight based on context length:

InterpolationStrategy::Dynamic {
    base_alpha: 0.5,       // Base weight for n-gram
    alpha_per_context: 0.1, // Additional weight per context word
    max_alpha: 0.9,        // Maximum n-gram weight
}

Methods

Scoring

`score(word, context) -> f64`

Score a word given context using the configured interpolation strategy.

let score = model.score("fox", &["the", "quick", "brown"]);
println!("log P(fox | the quick brown) = {:.4}", score);

Returns: Log probability of the word given context.

`sentence_log_prob(words) -> f64`

Compute total log probability of a sentence.

let log_prob = model.sentence_log_prob(&["the", "quick", "brown", "fox"]);

`perplexity(words) -> f64`

Compute perplexity of a sentence.

let ppl = model.perplexity(&["the", "quick", "brown", "fox"]);
println!("Perplexity: {:.2}", ppl);

Lower perplexity indicates better model fit.

Prediction

`predict_next(context, candidates) -> Option<(String, f64)>`

Find the most likely next word from candidates.

let candidates = ["fox", "dog", "cat", "bird"];
if let Some((word, score)) = model.predict_next(&["the", "quick"], &candidates) {
    println!("Best: {} (score: {:.4})", word, score);
}

Component Access

`ngram_model() -> &NgramModel<D>`

Get reference to the n-gram component.

let ngram = model.ngram_model();
println!("N-gram order: {}", ngram.order());

`embedding_model() -> &SubwordEmbedding`

Get reference to the embedding component.

let embedding = model.embedding_model();
println!("Embedding dim: {}", embedding.dim());

`config() -> &HybridConfig`

Get reference to the configuration.

let config = model.config();
println!("Cache size: {}", config.cache_size);

Cache Management

`clear_cache()`

Clear the score cache.

model.clear_cache();

Serialization (requires `serde-extras` feature)

`save(path) -> Result<()>`

Save model to binary file (requires D: Serialize).

model.save("hybrid.bin")?;

`load(path) -> Result<Self>`

Load model from binary file.

let model: HybridLanguageModel<DynamicDawgChar<NgramEntry>> =
    HybridLanguageModel::load("hybrid.bin")?;

`save_portable(path) -> Result<()>`

Save in portable format (works with any dictionary backend).

model.save_portable("hybrid.portable.bin")?;

`load_portable(path, factory) -> Result<Self>`

Load from portable format with dictionary factory.

let model = HybridLanguageModel::load_portable(
    "hybrid.portable.bin",
    || DynamicDawgChar::new()
)?;

How Embedding Probabilities Work

The embedding model converts similarity to probability:

Compute context vector (average of context word embeddings)
Compute cosine similarity between word and context
Apply temperature scaling: scaled_sim = similarity / temperature
Convert to log probability: log_prob = scaled_sim - 1.0

For OOV words:

Subword embeddings provide the word vector
Even completely unseen words get reasonable probabilities

Performance Considerations

Cache Size
- Larger cache = better performance for repeated queries
- Trade-off with memory usage
Interpolation Strategy
- Linear: Simple, well-understood
- LogLinear: Better for combining log-space models
- Fallback: Fast for known words
- Dynamic: Adapts to context availability
Temperature
- Lower temperature = sharper probability distribution
- Higher temperature = smoother distribution

Example: Complete Workflow

use libgrammstein::hybrid::{HybridLanguageModel, HybridConfig, InterpolationStrategy};
use libgrammstein::ngram::{NgramModel, TrainerBuilder, NgramEntry};
use libgrammstein::embedding::EmbeddingTrainerBuilder;
use libgrammstein::corpus::PlaintextReader;
use liblevenshtein::dictionary::dynamic_dawg_char::DynamicDawgChar;

fn main() -> libgrammstein::Result<()> {
    // 1. Train components
    let reader = PlaintextReader::from_file("corpus.txt")?;

    let ngram_model = TrainerBuilder::new(DynamicDawgChar::new())
        .order(5)
        .train(&reader)?;

    let reader2 = PlaintextReader::from_file("corpus.txt")?;
    let embedding_model = EmbeddingTrainerBuilder::new()
        .dim(100)
        .epochs(5)
        .train(&reader2)?;

    // 2. Create hybrid model
    let config = HybridConfig {
        strategy: InterpolationStrategy::Linear { alpha: 0.7 },
        ..Default::default()
    };
    let hybrid = HybridLanguageModel::new(ngram_model, embedding_model, config);

    // 3. Score sentences
    let sentence = ["the", "quick", "brown", "fox"];
    let log_prob = hybrid.sentence_log_prob(&sentence);
    let ppl = hybrid.perplexity(&sentence);

    println!("Log probability: {:.4}", log_prob);
    println!("Perplexity: {:.2}", ppl);

    // 4. Handle OOV words
    let oov_score = hybrid.score("xyzzy", &["magic", "word"]);
    println!("OOV word score: {:.4}", oov_score);  // Still gets reasonable score

    // 5. Save model
    hybrid.save("hybrid.bin")?;

    Ok(())
}

Use Cases

Spell Correction Ranking

// Score correction candidates
let candidates = ["their", "there", "they're"];
let context = ["put", "it", "over"];

let mut scored: Vec<_> = candidates.iter()
    .map(|c| (c, model.score(c, &context)))
    .collect();

scored.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap());

println!("Best correction: {}", scored[0].0);

Language Model Perplexity Evaluation

fn evaluate_corpus(model: &HybridLanguageModel<D>, sentences: &[Vec<&str>]) -> f64 {
    let total_log_prob: f64 = sentences.iter()
        .map(|s| model.sentence_log_prob(s))
        .sum();

    let total_words: usize = sentences.iter()
        .map(|s| s.len())
        .sum();

    (-total_log_prob / total_words as f64).exp()
}

Uh oh!

FilesExpand file tree

hybrid.md

Latest commit

History