llammer-pipeline

A WFST spelling-correction pipeline for Rust — tokenize text, build a weighted correction lattice, and decode the most likely fix with Viterbi, with optional phonetic rescoring and conversation-context awareness.

llammer-pipeline is the extracted, library form of llammer's correction stack. It corrects a line of possibly-misspelled text and returns the fix plus a structured account of every change and how confident it is — using a principled tropical-semiring lattice and exact shortest-path decoding rather than ad-hoc heuristics.

• License: MIT OR Apache-2.0 · Edition: 2021 · Status: 0.1.0
• Full documentation: docs/ — architecture, theory, API reference, guides, and design notes.

---

How it works

"i wnat to hava"  ─tokenize─▶  correction lattice  ─viterbi─▶  "i want to have"

Raw text is tokenized (preserving punctuation and case); each token contributes candidate spellings as parallel, weighted edges to a lattice; an optional layer reweights those edges; and a Viterbi decode selects the minimum-cost path, which is reassembled with the original punctuation and case.

Text view

 "i wnat to hava"
        │ tokenize
        ▼
   tokenizer ─▶ lattice build ─▶ layer pipeline ─▶ decode ─▶ reconstruct
  (case/punct) (candidate edges) (phonetic, opt) (viterbi/nbest) (re-case)
        │  uses liblevenshtein (DynamicDawgChar · Transducer)
        │  and  lling-llang    (LatticeBuilder · TropicalWeight · LayerPipeline · viterbi/nbest)
        ▼
 CorrectionResult { text, confidence, corrections, alternatives }

Features

• Punctuation- and case-preserving tokenization with lossless per-token reconstruction ("Teh," → "The,").
• Damerau-Levenshtein candidate generation over a DAWG dictionary — adjacent transpositions (teh → the) count as a single edit.
• A tropical-semiring correction lattice decoded exactly by Viterbi in $`O(\lvert V \rvert + \lvert E \rvert)`$ time, with N-best alternatives.
• Optional phonetic rescoring using verified Zompist spelling-to-sound rules.
• Conversation-context awareness — boost confidence from recent dialogue.
• A structured result — corrected text, per-word corrections with distances and confidences, and ranked alternatives — all serde-serializable.
• An embedded default dictionary, so it works with zero configuration.

Quick start

use llammer_pipeline::{LatticeCorrectionPipeline, LatticePipelineConfig};

fn main() -> anyhow::Result<()> {
    // Uses the embedded default dictionary; no files needed.
    let config = LatticePipelineConfig::default();
    let mut pipeline = LatticeCorrectionPipeline::new(config)?;

    let result = pipeline.correct("i wnat to hava")?;

    println!("{}", result.text);          // "i want to have"
    println!("changed: {}", result.changed);      // true
    println!("confidence: {:.2}", result.confidence);  // 0.67
    for c in &result.corrections {
        println!("  {} → {} (distance {})", c.original, c.corrected, c.edit_distance);
    }
    // wnat → want (distance 1)
    // hava → have (distance 1)
    Ok(())
}

The correction quality depends on the dictionary. The embedded default is a ~400-word common-English list, so it fixes everyday typos well; point dictionary_path at a domain word list for domain text (see custom dictionaries).

See docs/guides/getting-started.md for a step-by-step walkthrough and docs/guides/interpreting-results.md for reading the output.

Installation

Not published to crates.io. llammer-pipeline depends on sibling path crates and is consumed within the f1r3fly workspace layout. Its two dependencies — liblevenshtein and lling-llang — must be present as siblings (../liblevenshtein-rust, ../lling-llang).

Add it as a path dependency from a crate laid out beside it:

[dependencies]
llammer-pipeline = { path = "../llammer-pipeline" }

Feature flags:

# Enable phonetic rescoring (off by default):
llammer-pipeline = { path = "../llammer-pipeline", features = ["phonetic-rescore"] }

# Everything:
llammer-pipeline = { path = "../llammer-pipeline", features = ["full"] }

Feature	Default	Effect
default	✓	Fuzzy candidates + Viterbi/N-best decoding.
phonetic-rescore	off	Compiles the phonetic rescoring layer (see the note in configuration).
full	off	Umbrella; currently equals phonetic-rescore.

Realized today vs. 🚧 Roadmap

The docs describe realized behavior precisely and mark planned work as 🚧 Roadmap so nothing over-promises:

• Realized: tokenization; Damerau-Levenshtein candidates; the tropical lattice; Viterbi + N-best; optional phonetic rescoring; the conversation/context model with confidence blending.
• 🚧 Roadmap (declared but not yet executed): CFG filtering, language-model reranking, beam search, neural rescoring, and the WfstComposition / NeuralRescoring / ContextAware builder components. See docs/design/design-rationale.md.

Documentation

Section	Contents
Architecture	Module structure, end-to-end data flow, and the dependency seam.
Concepts	Self-contained theory: lattices, semirings, Viterbi, Levenshtein automata, cost model, phonetics, context.
Components	Per-module API reference.
Guides	Getting started, configuration, contextual usage, dictionaries, results, serialization.
Design	Rationale, engineering, performance, security, testing.
Glossary · Notation · Bibliography	Terms, symbols, and DOI-checked references.

Building

cargo build                       # default features
cargo build --features full       # all features
cargo test                        # run the unit tests
make diagrams-check               # validate the documentation diagrams

The make targets for diagrams (diagrams, diagrams-force, diagrams-check) render the documentation's PlantUML/Graphviz/D2 sources; see docs/diagrams/README.md.

License

Licensed under either of MIT or Apache License, Version 2.0 at your option.