Skip to content

basic-correction.md

Latest commit

 

History

History
340 lines (263 loc) · 13.7 KB

basic-correction.md

File metadata and controls

340 lines (263 loc) · 13.7 KB
Raw

Basic Correction Examples

The llammer correct subcommand corrects a single string (or, in batch mode, a file) using the llammer-pipeline lattice corrector: it tokenizes the input, proposes dictionary-constrained Damerau–Levenshtein candidates for each out-of-vocabulary word, and decodes the lowest-cost path with the Viterbi algorithm over the tropical (min, +) semiring. This page shows the command by example, using output captured from the real binary. For the full architecture see the pipeline overview; for every flag see the CLI reference.

llammer correct data flow: tokenize, dictionary-constrained Damerau–Levenshtein candidates, Viterbi decode over the tropical semiring, case restoration and reconstruction — single-string and batch paths

Before you start: two things about the output

Two properties of the shipped binary shape every transcript below.

  1. The dictionary is a small, embedded, English-only word list. correct builds its pipeline configuration directly from CLI arguments with no dictionary path, and it never reads a config file, so it always uses the embedded llammer-pipeline/data/basic_dictionary.txt — roughly 360 unique common English words. There is no CLI flag that gives correct a different or larger dictionary. A word that is not in that list is treated as a spelling error and is either left unchanged (if no close dictionary word exists) or replaced by the nearest dictionary word — even if the input was itself a perfectly valid English word. This is honest as-built behavior; see Limitations.

  2. A log line is printed to standard output. main.rs initializes tracing with the default filter llammer=info,libgrammstein=warn,lling_llang=warn, and the formatter writes to stdout. Every run therefore emits a line such as:

    $ llammer correct "custmer"
2026-07-13T03:11:58.804055Z  INFO llammer::cli::commands::correct: Loading models...
  [original] custmer
[corrected] customer

    Because the line is on stdout, 2>/dev/null does not remove it. To silence logging — which you must do before piping machine-readable output into jq, cut, or awk — set RUST_LOG=off (or RUST_LOG=warn). Every transcript below is shown with logging quieted so the correction output stands alone.

Single-string correction

Pass the text as the positional TEXT argument. The default text format prints two lines when the correction changed the input — a dim [original] line and a bold-green [corrected] line — and prints just the text when nothing changed.

$ RUST_LOG=off llammer correct "custmer"
  [original] custmer
[corrected] customer

A multi-word sentence is corrected token by token, preserving surrounding spacing and punctuation:

$ RUST_LOG=off llammer correct "thier problm"
  [original] thier problm
[corrected] their problem

$ RUST_LOG=off llammer correct "custmer servce"
  [original] custmer servce
[corrected] customer service

$ RUST_LOG=off llammer correct "wrok on the projcet"
  [original] wrok on the projcet
[corrected] work on the project

$ RUST_LOG=off llammer correct "documnet and infrmation"
  [original] documnet and infrmation
[corrected] document and information

When every word is already in the dictionary, the input passes through unchanged and a single plain line is printed:

$ RUST_LOG=off llammer correct "hello world"
hello world

$ RUST_LOG=off llammer correct "please help me find the book"
please help me find the book

Case and punctuation are preserved

The tokenizer peels leading and trailing punctuation off each word, remembers its case pattern, and re-applies both after correction, so capitalization, quotes, commas, and terminal periods survive:

$ RUST_LOG=off llammer correct "Recieve"
  [original] Recieve
[corrected] Receive

Unicode words are handled correctly; a valid non-ASCII word that is out-of-vocabulary (the embedded list is small) is simply left unchanged rather than mangled:

$ RUST_LOG=off llammer correct "Müller"
Müller

Purely numeric tokens and one-character tokens are always kept verbatim (they are never treated as spelling errors).

Output formats

--format selects the output shape: text (default), json, or tsv.

JSON

For a single string, --format json emits a pretty-printed object with exactly three keys — changed, corrected, and original — serialized in alphabetical order. There is no confidence field and no corrections array in this output.

$ RUST_LOG=off llammer correct --format json "custmer servce"
{
  "changed": true,
  "corrected": "customer service",
  "original": "custmer servce"
}

changed is false when the input was already correct:

$ RUST_LOG=off llammer correct --format json "hello world"
{
  "changed": false,
  "corrected": "hello world",
  "original": "hello world"
}

TSV

--format tsv prints a single tab-separated line, original⇥corrected (no header, no third column):

$ RUST_LOG=off llammer correct --format tsv "custmer servce"
custmer servce	customer service

An unchanged input repeats the text on both sides:

$ RUST_LOG=off llammer correct --format tsv "hello world"
hello world	hello world

Gallery: error types the corrector handles

The transducer uses Damerau–Levenshtein distance, so a single adjacent transposition, insertion, deletion, or substitution all cost one edit. The examples below are real outputs and land on the intended word because each target is in the embedded dictionary.

Error class Example (input ⇒ output) Edit that explains it
Transposition (adjacent swap) thier ⇒ their, wrold ⇒ world, recieve ⇒ receive, wrok ⇒ work one swap of neighboring letters
Deletion (a letter is missing) custmer ⇒ customer, problm ⇒ problem, adress ⇒ address, mananger ⇒ manager insert the missing letter
Insertion (an extra letter) gooodbye ⇒ goodbye, helllo ⇒ hello, buisness ⇒ business delete the extra letter
Substitution (a wrong letter) problam ⇒ problem, documant ⇒ document, managar ⇒ manager, servace ⇒ service replace one letter

Multi-error inputs are corrected as long as each word stays within the edit-distance budget:

$ RUST_LOG=off llammer correct "adress and phnoe"
  [original] adress and phnoe
[corrected] address and phone

$ RUST_LOG=off llammer correct "gooodbye wrold"
  [original] gooodbye wrold
[corrected] goodbye world

Options that influence candidate generation

--max-edit-distance <N> (default 2)

Controls how far the Damerau–Levenshtein transducer searches for candidate words. A larger radius admits more candidates (and can change the winning path); a smaller radius is faster and stricter. For a word one edit from its target, 1, 2, and 3 all agree:

$ RUST_LOG=off llammer correct --max-edit-distance 1 "documnet"
  [original] documnet
[corrected] document

$ RUST_LOG=off llammer correct --max-edit-distance 3 "documnet"
  [original] documnet
[corrected] document

--language <CODE> (default en-US)

Sets the language tag stored on the pipeline configuration. It is parsed and retained, but because correct always loads the embedded English dictionary and has no flag to supply a language-specific word list, changing the language does not change the correction result today. It is documented here so the behavior is not surprising:

$ RUST_LOG=off llammer correct --language fr "custmer"
  [original] custmer
[corrected] customer

Planned / not yet implemented. Per-language dictionaries (so that --language de or --language fr selects a matching word list) are not wired into the correct command. Track this in the roadmap.

Flags that are accepted but currently ignored

The following flags parse successfully (for backward CLI compatibility) but have no effect on the output today. They are documented so you know not to rely on them.

Flag Why it is inert
--neural-rescore Maps to the pipeline's enable_lm_rerank. The LM-rerank layer is a stub — the source literally leaves a // LM rerank layer would be added here placeholder — so it adds no layer and changes nothing.
--beam-width <N> (default 100) Stored on the configuration, but the CLI decodes with exact Viterbi, which does not prune, so the beam width is never consulted.
--insertion-penalty <F> (default 3.0) The lattice scores every edit with a single edit_weight; there is no separate insertion cost.
--deletion-penalty <F> (default 3.0) Same — there is no separate deletion cost.
-m/--models <DIR> (env LLAMMER_MODELS) correct never loads models from disk; the value is ignored.
-c/--config <PATH>, -v/--verbose (global) Parsed but never consumed. Logging is driven only by RUST_LOG; correct does not read a config file.

All of these can be combined and the result is identical to the plain command:

$ RUST_LOG=off llammer correct --neural-rescore --beam-width 500 \
    --insertion-penalty 1.0 --deletion-penalty 9.0 --models /nope "custmer servce"
  [original] custmer servce
[corrected] customer service

How a correction is scored

For each token the corrector adds edges to a lattice: an in-dictionary word gets cost $0$; an out-of-vocabulary original is added at a deliberately high cost of $\texttt{max\_edit\_distance} + 1$ (so any real correction outranks keeping the typo); and each Damerau–Levenshtein candidate at distance $d$ is added at cost

$$w(e) = d \times \texttt{edit\_weight}$$

with edit_weight = 0.3 by default. Over the tropical semiring, lower weight is better, and Viterbi returns the minimum-cost path:

$$\hat{p} = \arg\min_{p \,\in\, \mathrm{paths}(\mathcal{L})} \; \sum_{e \in p} w(e)$$

When a per-word correction is reported (in the REPL and in the internal result), its confidence is derived from the chosen edit distance:

$$\text{confidence} = 1 - \frac{d}{\texttt{max\_edit\_distance} + 1}$$

so a distance-1 correction reports $1 - \tfrac{1}{3} \approx 67\%$ and a distance-2 correction reports $1 - \tfrac{2}{3} \approx 33\%$. The single-string correct command does not print these per-word figures — they surface in the REPL transcripts — but they explain the scoring.

Limitations: small dictionary and no language model

Because the dictionary is small and the language-model rerank layer is a stub, the corrector has no notion of context and no way to prefer one equidistant candidate over another on linguistic grounds. Edit-distance ties are broken by the phonetic rescoring layer alone, which produces occasional surprising picks. These are real outputs:

$ RUST_LOG=off llammer correct "helo world"
  [original] helo world
[corrected] help world

$ RUST_LOG=off llammer correct "teh quikc brown fox"
  [original] teh quikc brown fox
[corrected] be quikc brown for

Here helo is one edit from both help and hello, and the tie resolves to help; teh resolves to be rather than the; and the valid words quikc, brown, fox are handled as follows — quikc and brown have no close dictionary word and are kept, while fox is not in the 363-word list and is one substitution from the dictionary word for, so it is "corrected" to for. A larger or domain-specific dictionary plus a real language-model rerank layer would resolve these; both are tracked in the roadmap.

What correct does not support

  • No standard input. correct reads only the positional TEXT argument or an --input FILE. It does not treat - as stdin, and there is no echo … | llammer correct mode. Use --input for files (see Batch Processing).

  • No input at all is an error. Running correct with neither TEXT nor --input exits non-zero:

    $ RUST_LOG=off llammer correct
Error: No input provided. Use TEXT argument or --input flag.
$ echo $?
1

See Also

References

  1. Damerau, F. J. (1964). A technique for computer detection and correction of spelling errors. Communications of the ACM, 7(3), 171–176. https://doi.org/10.1145/363958.363994
  2. Levenshtein, V. I. (1966). Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8), 707–710.
  3. Schulz, K., & Mihov, S. (2002). Fast string correction with Levenshtein automata. International Journal on Document Analysis and Recognition, 5(1), 67–85. https://doi.org/10.1007/s10032-002-0082-8
  4. Viterbi, A. J. (1967). Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory, 13(2), 260–269. https://doi.org/10.1109/TIT.1967.1054010
  5. Mohri, M., Pereira, F., & Riley, M. (2002). Weighted finite-state transducers in speech recognition. Computer Speech & Language, 16(1), 69–88. https://doi.org/10.1006/csla.2001.0184