Skip to content

README.md

Latest commit

 

History

History
274 lines (207 loc) · 13 KB

README.md

File metadata and controls

274 lines (207 loc) · 13 KB
Raw

libgrammstein Documentation

libgrammstein is a hybrid language model library combining N-gram models with subword embeddings. It is designed to integrate with lling-llang for WFST-based text correction and normalization.

What is libgrammstein?

libgrammstein provides:

  • N-gram Language Model: Statistical word sequence prediction using Modified Kneser-Ney smoothing
  • Subword Embeddings: FastText-style embeddings for handling out-of-vocabulary words
  • Hybrid Model: Combines both approaches for robust scoring
  • WFST Integration: Implements lling-llang's LanguageModel trait for lattice rescoring
┌─────────────────────────────────────────────────────────────────┐
│                        libgrammstein                             │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌─────────────────┐   ┌─────────────────┐                     │
│  │   N-gram Model  │   │    Subword      │                     │
│  │                 │   │   Embeddings    │                     │
│  │  Modified KN    │   │   FastText-     │                     │
│  │  smoothing      │   │   style         │                     │
│  └────────┬────────┘   └────────┬────────┘                     │
│           │                     │                               │
│           └──────────┬──────────┘                               │
│                      ▼                                          │
│           ┌─────────────────────┐                               │
│           │   HybridLanguage    │                               │
│           │       Model         │                               │
│           │                     │                               │
│           │  Implements         │                               │
│           │  LanguageModel      │                               │
│           │  trait              │                               │
│           └─────────────────────┘                               │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Quick Start

use libgrammstein::prelude::*;

// Load a trained hybrid model
let model = HybridLanguageModel::load("model.bin")?;

// Score a token sequence
let log_prob = model.score_sequence(&["the", "quick", "brown", "fox"]);

// Score a continuation
let next_prob = model.score_continuation(&["the", "quick"], "brown");

println!("Sequence log probability: {}", log_prob);
println!("P(brown | the quick): {}", next_prob.exp());

Training a Model

use libgrammstein::prelude::*;
use libgrammstein::corpus::PlaintextReader;
use libgrammstein::ngram::TrainerBuilder;

// Stream corpus from directory
let reader = PlaintextReader::from_directory("./corpus")?;

// Train N-gram model
let ngram = TrainerBuilder::new()
    .order(5)
    .train(&reader)?;

// Train subword embeddings
let embedding = EmbeddingTrainer::new()
    .dimension(100)
    .epochs(20)
    .train(&reader)?;

// Combine into hybrid model
let hybrid = HybridLanguageModel::new(ngram, embedding);
hybrid.save("model.bin")?;

Documentation Structure

Architecture

Components

N-gram Model

Subword Embeddings

Hybrid Model

Text Generation

  • Text Generation - Autoregressive text generation with sampling strategies

Corpus Processing

  • Overview - Streaming corpus architecture
  • Streaming - Memory-efficient processing
  • Formats - Wikipedia, Gutenberg, plaintext

Acoustic Processing

Neural Module

  • Overview - ModernBERT-based neural components
  • Model - ModernBERT model wrapper and inference
  • Embedder - Document and query embedding
  • Rescorer - Neural rescoring for beam search
  • Summarizer - Extractive summarization with MMR
  • Cache - KV cache for efficient inference

RAG (Retrieval-Augmented Generation)

  • Overview - Document indexing and retrieval
  • Document - Document structures and metadata
  • Backend - Retrieval backends (Exact, HNSW)
  • Index - RagIndex with topic integration
  • Retriever - Query and retrieval pipeline
  • Builder - Index construction workflow

Topic Extraction

Paradigm Detection

Code Embeddings

Subtree Mining

Code Correction

Integration

lling-llang

liblevenshtein

Training

API Reference

Examples

Prerequisites

  • Rust: 1.75+ (2024 edition)
  • liblevenshtein-rust: Dictionary backends
  • Corpus data: Wikipedia dumps, Project Gutenberg, or custom text files

Features

[dependencies]
libgrammstein = { version = "0.1", features = ["lling-llang-integration", "serde"] }
Feature Description
default Core N-gram and embedding functionality
lling-llang-integration Implements lling-llang's LanguageModel trait
serde Model serialization/deserialization
async Async corpus streaming (Tokio)
acoustic Audio feature extraction (MFCC, filterbank)
candle-model Candle-based neural acoustic models
phonetic Phonetic-aware embeddings with Zompist rules
neural-rescore ModernBERT embeddings, rescoring, and summarization
rag RAG indexing with topic extraction
code Core code correction (lexical corrector only)
code-python Python language support
code-rust Rust language support
code-javascript JavaScript language support
code-rholang Rholang language support
code-metta MeTTa language support
code-neural Neural code embeddings (UniXcoder, GraphCodeBERT)

Related Projects

License

MIT OR Apache-2.0