German Tokenizer

A high-performance German text tokenizer for search and NLP applications. Specializes in compound word decomposition, a critical feature for German text processing.

Features

• Compound word decomposition: Splits German compounds into constituent parts (e.g., "Brandschutzkonzept" → ["brand", "schutz", "konzept"])
• FST-based dictionary lookups: Uses finite state transducers for O(n) dictionary lookups where n is word length
• Configurable normalization pipeline: NFKD decomposition, lowercase, ß→ss conversion, German stemming, and more
• LRU cache: 100k entry cache for compound splits (~10MB memory)
• Structured output: Returns per-word {whole, parts} so callers can use the umlaut-preserved whole word, the normalized compound parts, or both
• Runtime dictionary updates: Add or remove words without restarting

Installation

go get github.com/kerem-kaynak/german-tokenizer/v2

Quick Start

package main

import (
    "fmt"
    "github.com/kerem-kaynak/german-tokenizer/v2/pkg/tokenizer"
)

func main() {
    tok, err := tokenizer.NewTokenizer("path/to/dictionary.txt", tokenizer.Config{
        Cache: true,
        Normalizers: tokenizer.NormalizerConfig{
            NFKDDecompose:        true,
            RemoveControlChars:   true,
            Lowercase:            true,
            NormalizeQuotes:      true,
            ExpandLigatures:      true,
            ConvertEszett:        true,
            RemoveCombiningMarks: true,
            StemGerman:           true,
        },
    })
    if err != nil {
        panic(err)
    }
    defer tok.Close()

    tokens := tok.Tokenize("Brandschutzkonzept")
    fmt.Println(tokens)
    // Output: [{brandschutzkonzept [brand schutz konzept]}]
}

Dictionary

The tokenizer requires a dictionary of German compound word components. A dictionary with ~15,000 words is included at dictionaries/german_compound_word_components.txt.

Dictionary source: The included dictionary is derived from uschindler/german-decompounder, which was created based on Björn Jacke's igerman98 dictionary. The dictionary contains component parts commonly used to form German compound words (not the compounds themselves).

You can also use your own dictionary - one word per line, lowercase.

Runtime Dictionary Updates

Words can be added or removed at runtime. Changes are immediately persisted to disk and the FST is rebuilt:

// Add a word - FST is rebuilt immediately
err := tok.AddWord("neueswort")

// Remove a word - FST is rebuilt immediately
err := tok.RemoveWord("alteswort")

Configuration

All configuration is explicit. No hidden defaults.

Config struct

type Config struct {
    Cache       bool             // Enable LRU cache for compound splits
    Normalizers NormalizerConfig // Which normalizers to apply
}

type NormalizerConfig struct {
    NFKDDecompose        bool // Unicode NFKD decomposition
    RemoveControlChars   bool // Remove control characters
    Lowercase            bool // Convert to lowercase
    NormalizeQuotes      bool // Normalize „" » « to ASCII quotes
    ExpandLigatures      bool // æ→ae, œ→oe
    ConvertEszett        bool // ß→ss
    RemoveCombiningMarks bool // Remove combining diacritics (ä→a after NFKD)
    StemGerman           bool // Apply Snowball German stemmer (blevesearch/snowballstem)
}

Example configurations

Full normalization (search indexing):

tokenizer.Config{
    Cache: true,
    Normalizers: tokenizer.NormalizerConfig{
        NFKDDecompose:        true,
        RemoveControlChars:   true,
        Lowercase:            true,
        NormalizeQuotes:      true,
        ExpandLigatures:      true,
        ConvertEszett:        true,
        RemoveCombiningMarks: true,
        StemGerman:           true,
    },
}

Preserve umlauts (exact matching):

tokenizer.Config{
    Cache: true,
    Normalizers: tokenizer.NormalizerConfig{
        NFKDDecompose:        false,
        RemoveControlChars:   true,
        Lowercase:            true,
        NormalizeQuotes:      true,
        ExpandLigatures:      false,
        ConvertEszett:        false,
        RemoveCombiningMarks: false,
        StemGerman:           false,
    },
}

No cache (memory constrained):

tokenizer.Config{
    Cache:       false, // Disable cache
    Normalizers: //   ...
}

How It Works

1. Word Splitting

Input text is split into words using Unicode letter/number detection:

"Der Brandschutzkonzept" → ["Der", "Brandschutzkonzept"]

2. Compound Decomposition

Each word is decomposed using a greedy left-to-right algorithm:

"Brandschutzkonzept"
  ├─ Try "Brandschutzkonzept" → not in dictionary
  ├─ Try "Brandschutzkonzep" → not in dictionary
  ├─ ...
  ├─ Try "Brand" → IN DICTIONARY ✓
  │   └─ Recurse on "schutzkonzept"
  │       ├─ Try "Schutz" → IN DICTIONARY ✓
  │       │   └─ Recurse on "konzept"
  │       │       └─ Try "Konzept" → IN DICTIONARY ✓
  └─ Result: ["brand", "schutz", "konzept"]

Dictionary lookups use:

1. Direct FST lookup
2. Umlaut normalization (ä→a, ö→o, ü→u, ß→ss)
3. Suffix stripping for inflected forms

3. Token Output

For each detected word, the tokenizer emits one WordTokens entry:

• Whole: lowercase original, umlauts preserved (via LowercaseOnly)
• Parts: each compound segment, fully normalized and stemmed, in splitter order

Input: "Wärmedämmung"
Output: [{Whole: "wärmedämmung", Parts: ["warm", "dammung"]}]

No cross-word deduplication — each input word produces exactly one entry, in input order. Callers pick whichever fields they need (e.g. index-side: Whole

• Parts; query-filter side: Parts only).

4. Normalization Pipeline

Each segment passes through the configured normalizers in order:

"Größe"
  → NFKD: "Gro\u0308ße" (ö decomposed to o + combining umlaut)
  → Lowercase: "gro\u0308ße"
  → ConvertEszett: "gro\u0308sse"
  → RemoveCombiningMarks: "grosse"
  → StemGerman: "gross"

Stemming uses the official Snowball German algorithm via blevesearch/snowballstem (the Snowball compiler's generated Go output, as used by bleve). It conflates inflected forms (Dämmungen and Dämmung both → dammung, Wärme → warm) but is deliberately conservative with derivational suffixes: -ung is only stripped from words long enough to have a region R2 (Bezeichnung → bezeichn, but Dämmung → dammung, not damm). Running the stemmer after umlaut/ß folding is output-equivalent to canonical Snowball, which performs the same folding internally (prelude ß→ss, postlude ä/ö/ü→a/o/u) — pinned against the official snowball-data corpus in TestStemGerman_SnowballGroundTruth.

5. FST Dictionary

The dictionary uses a Finite State Transducer (FST) via blevesearch/vellum:

• Memory efficient: FST is smaller than a hash map for large dictionaries
• Fast lookups: O(n) where n is the word length, not dictionary size
• Prefix queries: Can efficiently find all words with a given prefix

6. LRU Cache

Compound splits are cached using an LRU cache (100k entries, ~10MB):

• Cache hit: Return cached result immediately
• Cache miss: Compute split, store in cache
• Eviction: Least recently used entries are evicted when cache is full

CLI Tools

Tokenize

# Build all binaries
make build

# Tokenize a single input
make run TEXT="Brandschutzkonzept"
# Output: [{"whole":"brandschutzkonzept","parts":["brand","schutz","konzept"]}]

# Interactive mode
make demo
> Wärmedämmung
  [{"whole":"wärmedämmung","parts":["warm","dammung"]}]

Dictionary Management

# Show dictionary statistics
make dict-stats

# Check if a word exists
make dict-contains WORD=haus

# Add a word
make dict-add WORD=neueswort

# Remove a word
make dict-remove WORD=alteswort

Throughput Benchmarking

make throughput

Performance

Benchmarks on Apple M4 Pro:

Operation	Throughput	Latency
Single word tokenization	710k ops/sec	1.4μs
Long compound tokenization	440k ops/sec	2.3μs
Sentence (10 words)	180k ops/sec	5.6μs
Dictionary lookup	8.1M ops/sec	123ns
Normalizer (full pipeline)	1.2M ops/sec	824ns
Cache hit	59M ops/sec	17ns

Run benchmarks on your hardware:

# Go micro-benchmarks (per-function timing)
make bench

# Throughput test (words/sec with colored output)
make throughput

API Reference

Tokenizer

// Create tokenizer
tok, err := tokenizer.NewTokenizer(dictPath string, cfg Config) (*Tokenizer, error)

// Tokenize text — one WordTokens per input word, in input order, no dedup
tokens := tok.Tokenize(text string) []WordTokens

// Per-word output type (JSON tags shown for wire use)
type WordTokens struct {
    Whole string   `json:"whole"` // lowercase original, umlauts preserved
    Parts []string `json:"parts"` // normalized compound segments, splitter order
}

// Dictionary management (FST rebuilt immediately, persisted to disk)
err := tok.AddWord(word string) error
err := tok.RemoveWord(word string) error

// Cache management
tok.CacheSize() int
tok.ClearCache()
tok.CacheEnabled() bool

// Info
tok.DictionaryWordCount() int

// Cleanup
tok.Close() error

Normalizer (standalone)

// Create with all normalizers
norm := tokenizer.NewNormalizer()

// Create with specific normalizers
norm := tokenizer.NewNormalizerWithSteps(
    tokenizer.NFKDDecompose,
    tokenizer.Lowercase,
    tokenizer.StemGerman,
)

// Normalize text
result := norm.Normalize(text string) string

// Lowercase only (preserves umlauts)
result := norm.LowercaseOnly(text string) string

Individual normalizer functions

All normalizer functions are exported and can be used standalone:

tokenizer.NFKDDecompose(s string) string
tokenizer.RemoveControlChars(s string) string
tokenizer.Lowercase(s string) string
tokenizer.NormalizeQuotes(s string) string
tokenizer.ExpandLigatures(s string) string
tokenizer.ConvertEszett(s string) string
tokenizer.RemoveCombiningMarks(s string) string
tokenizer.StemGerman(s string) string

Development

# Run tests
make test

# Run micro-benchmarks
make bench

# Run throughput test
make throughput

# Build binaries
make build

# Format code
make fmt

# Lint
make lint

License

MIT License - see LICENSE