batch-processing.md

Batch Processing Examples

llammer correct can process a whole file at once: with --input FILE it reads one sentence per line, corrects each line with the same lattice pipeline used for a single string, and either writes the results to --output FILE or prints them to the terminal. llammer has no directory, recursion, or parallelism features of its own, so larger workflows are built by combining correct with ordinary shell tools. This page shows the file interface and a set of verified shell recipes. For the single-string form and output shapes see Basic Correction; for every flag see the CLI reference.

Logging is on stdout. As with single-string correction, every run prints a … INFO llammer::cli::commands::correct: Loading models... line to stdout (not stderr), so 2>/dev/null will not remove it. Prefix any command that pipes machine-readable output into jq, cut, or awk with RUST_LOG=off. All piping recipes below already do this.

File in, file out

Create an input file with one sentence per line:

$ cat sentences.txt
thier problm
custmer servce
wrok on the projcet
documnet and infrmation
hello world

Correct it into an output file. In the default text format the output file contains only the corrected line for each input line:

$ RUST_LOG=off llammer correct --input sentences.txt --output corrected.txt
$ cat corrected.txt
their problem
customer service
work on the project
document and information
hello world

Short and long flags are interchangeable: -i/--input and -o/--output.

Progress bar

While it works, correct renders an indicatif progress bar to the terminal (standard error), driven by the template {spinner} [{elapsed_precise}] [{bar:40.cyan/blue}] {pos}/{len} ({eta}). For a large file you will see a frame like:

⠹ [00:00:07] [########################################] 10000/10000 (0s)

Because the bar is drawn to standard error, redirecting stderr (or piping only stdout to another tool) hides it without affecting the corrected output.

Output formats to a file

--format chooses the on-disk shape. The batch shapes differ from the single-string shapes, so read this carefully.

Text (default)

One corrected line per input line, as shown above.

JSON — newline-delimited objects

Batch JSON is newline-delimited JSON (NDJSON / JSON Lines): one compact object per input line, each with exactly two alphabetically-ordered keys, corrected and original. There is no enclosing array, and no changed or confidence field (those exist only in the single-string JSON and the internal result, respectively).

$ RUST_LOG=off llammer correct --input sentences.txt --output results.json --format json
$ cat results.json
{"corrected":"their problem","original":"thier problm"}
{"corrected":"customer service","original":"custmer servce"}
{"corrected":"work on the project","original":"wrok on the projcet"}
{"corrected":"document and information","original":"documnet and infrmation"}
{"corrected":"hello world","original":"hello world"}

TSV

One tab-separated original⇥corrected line per input line (no header):

$ RUST_LOG=off llammer correct --input sentences.txt --output results.tsv --format tsv
$ cat results.tsv
thier problm	their problem
custmer servce	customer service
wrok on the projcet	work on the project
documnet and infrmation	document and information
hello world	hello world

Output to the terminal (no --output)

Omit --output to stream results to stdout. In text format each line is printed the same way as a single-string correction — a dim [original]/bold-green [corrected] pair when the line changed, and a single plain line when it did not:

$ RUST_LOG=off llammer correct --input sentences.txt
  [original] thier problm
[corrected] their problem
  [original] custmer servce
[corrected] customer service
  [original] wrok on the projcet
[corrected] work on the project
  [original] documnet and infrmation
[corrected] document and information
hello world

With --format json or --format tsv and no --output, the same NDJSON / TSV shapes shown above are written to stdout instead of a file, which is what makes the jq and awk recipes below possible.

Shell recipes

llammer intentionally keeps the CLI small; orchestration is the shell's job.

Filter with jq over the NDJSON stream

Because the JSON is one object per line, jq reads it as a stream. Emit only the lines that actually changed, as original⇥corrected pairs:

$ RUST_LOG=off llammer correct --input sentences.txt --format json \
    | jq -r 'select(.corrected != .original) | "\(.original)\t\(.corrected)"'
thier problm	their problem
custmer servce	customer service
wrok on the projcet	work on the project
documnet and infrmation	document and information

Count how many lines changed by slurping the stream into an array with -s:

$ RUST_LOG=off llammer correct --input sentences.txt --format json \
    | jq -s 'map(select(.corrected != .original)) | length'
4

Batch JSON carries no confidence field, so you cannot filter batch output by confidence. If you need per-word confidences, correct strings one at a time in the REPL, whose display includes them.

Filter with awk/cut over TSV

The TSV form is convenient for line-oriented tools. Print only changed lines:

$ RUST_LOG=off llammer correct --input sentences.txt --format tsv \
    | awk -F'\t' '$1 != $2'
thier problm	their problem
custmer servce	customer service
wrok on the projcet	work on the project
documnet and infrmation	document and information

Extract just the corrected column:

$ RUST_LOG=off llammer correct --input sentences.txt --format tsv | cut -f2
their problem
customer service
work on the project
document and information
hello world

Split a large corpus, process the pieces, and rejoin

correct processes a file on a single thread, so for a very large corpus split it into chunks, correct each chunk, and concatenate the results back in order:

# Split into 10k-line chunks named chunk_aa, chunk_ab, …
split -l 10000 large-corpus.txt chunk_

# Correct each chunk to a matching corrected_ file
for f in chunk_*; do
    RUST_LOG=off llammer correct --input "$f" --output "corrected_$f"
done

# Reassemble in the original order and clean up
cat corrected_chunk_* > corrected-corpus.txt
rm chunk_* corrected_chunk_*

Run chunks in parallel with GNU parallel

To use multiple cores, hand the chunks to GNU parallel, which runs one llammer process per chunk:

split -l 10000 large-corpus.txt chunk_

# {} is the chunk path; {/} is its basename for the output name
ls chunk_* | parallel 'RUST_LOG=off llammer correct --input {} --output corrected_{/}'

cat corrected_chunk_* > corrected-corpus.txt

Process every file in a directory

llammer has no recursive or directory mode, so drive it with a glob or find. A flat directory:

mkdir -p corrected
for f in input/*.txt; do
    RUST_LOG=off llammer correct --input "$f" --output "corrected/$(basename "$f")"
done

A directory tree, recreating the layout under corrected/ with find:

find docs -type f -name '*.txt' | while IFS= read -r f; do
    dest="corrected/$f"
    mkdir -p "$(dirname "$dest")"
    RUST_LOG=off llammer correct --input "$f" --output "$dest"
done

For many files across cores, combine find with parallel:

find input -type f -name '*.txt' \
    | parallel 'RUST_LOG=off llammer correct --input {} --output corrected/{/}'

Robust scripting

Guard against missing inputs and propagate failures — correct exits non-zero on a missing input file or when no input is supplied:

#!/usr/bin/env bash
set -euo pipefail

input="$1"
output="$2"

if [[ ! -f "$input" ]]; then
    echo "Error: input file not found: $input" >&2
    exit 1
fi

if ! RUST_LOG=off llammer correct --input "$input" --output "$output"; then
    echo "Error: correction failed for $input" >&2
    exit 1
fi

echo "Corrected $input -> $output"

What batch mode does not have

llammer ships none of the following; use the shell patterns above instead. All are tracked in the roadmap.

• No --batch-size flag. Lines are streamed one at a time; there is no internal batching knob.
• No built-in recursion or directory input. --input takes exactly one file. Use find/globs as shown.
• No internal parallelism. A single correct run is single-threaded; parallelism comes from running multiple processes (e.g. with parallel).
• No standard-input mode. correct does not read - or a pipe; it reads the TEXT argument or an --input file only.
• No language-specific dictionaries. Batch correction uses the same embedded, English-only 363-word dictionary as single-string correction; --language is stored but does not change results. See Basic Correction → Limitations.

See Also

• Basic Correction — single-string form, output shapes, and limitations.
• REPL Modes — interactive correction with per-word confidences.
• CLI Reference — every flag and default.
• REPL Overview — interactive-mode architecture.
• Roadmap — planned batching, directory, and parallelism features.

References

1. Damerau, F. J. (1964). A technique for computer detection and correction of spelling errors. Communications of the ACM, 7(3), 171–176. https://doi.org/10.1145/363958.363994
2. Levenshtein, V. I. (1966). Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8), 707–710.
3. Viterbi, A. J. (1967). Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory, 13(2), 260–269. https://doi.org/10.1109/TIT.1967.1054010