lingua-detect

A modular, dependency-free statistical language identification engine for pure Java (17+). Detects the language of a text using character n-gram frequency profiles and vector similarity, in the spirit of the classic Cavnar & Trenkle (1994) approach.

Features

• Character n-gram profiling (trigrams by default, configurable)
• Pluggable similarity strategy: Cosine Similarity (default) or Out-Of-Place / rank-order distance (Cavnar & Trenkle)
• 6 built-in reference languages: English, German, French, Spanish, Italian, Portuguese
• Lightweight, dependency-free profile serialization (export/import as text)
• Interactive CLI with a train command to add new languages at runtime from any sample text file no code changes or recompilation needed
• Zero third-party dependencies standard library only

Project layout

src/main/java/com/linguadetect/
├── model/
│   ├── Language.java          enum of supported languages
│   ├── NGramProfile.java      immutable frequency-vector wrapper + stats
│   └── DetectionResult.java   immutable record: prediction + rankings
├── profiling/
│   └── ProfileBuilder.java    text -> cleaned -> n-grams -> frequencies
├── similarity/
│   ├── SimilarityMetric.java       strategy interface
│   ├── CosineSimilarityMetric.java default strategy
│   └── OutOfPlaceMetric.java       Cavnar & Trenkle rank-order distance
├── core/
│   └── LanguageDetector.java  holds profiles, runs comparisons
├── repository/
│   └── ProfileRepository.java built-in corpora + text-format (de)serialization
└── cli/
    └── Main.java              interactive Scanner-based CLI

Build & run (plain javac, no build tool required)

# from the project root
mkdir -p out
javac -d out $(find src -name "*.java")
java -cp out com.linguadetect.cli.Main

Usage

> Bonjour, comment allez-vous aujourd'hui ?
--------------------------------------------------
Predicted language : French
Confidence         : 41.23%
Top candidates:
  1. French       similarity = 0.8931
  2. Italian      similarity = 0.6120
  3. Spanish      similarity = 0.5894
  4. Portuguese   similarity = 0.5602
  5. English      similarity = 0.3011
--------------------------------------------------

Switching the similarity metric

> metric outofplace
Switched to the Out-Of-Place (rank order) metric.

Training a new language profile at runtime

> train --lang=JAPANESE --file=sample.txt
Trained and loaded profile for Japanese from 'sample.txt' (300 distinct n-grams).
Saved to /home/user/lingua-detect/profiles/japanese.profile.txt

The resulting profile is persisted under profiles/ and is automatically reloaded the next time the CLI starts, so JAPANESE becomes a permanent detectable language from then on. Any value from the Language enum can be used with train; to support a brand-new language not already listed, add it as a new enum constant in Language.java first.

Extending the engine

• Add a language with a curated built-in corpus: add a sample text constant and a profiles.put(...) line in ProfileRepository.loadBuiltInProfiles().
• Add a language ad hoc: use the train CLI command shown above.
• Add a new similarity strategy: implement SimilarityMetric and call detector.setSimilarityMetric(...).
• Change n-gram length: construct LanguageDetector with new LanguageDetector(4, new CosineSimilarityMetric()) and rebuild any reference profiles with the matching n.

Notes on accuracy

The bundled sample corpora are short (a few hundred words each), written specifically for this project to keep the repository dependency-free and license-clean. For production-grade accuracy, replace ProfileRepository's built-in samples with much larger corpora (tens of thousands of words per language) the algorithm itself does not change.