GRIS — Interpretable MT Evaluation

Grammatical Interpretable Translation Scoring
An interpretable dependency-based approach for machine translation evaluation.

---

GRIS provides two complementary metrics:

• GRIS-DepScore — dependency-structure matching using the Hungarian algorithm.
• GRIS-SynGram — subtree path n-gram matching.

Install

Core installation

pip install .

Installs the core GRIS-DepScore and GRIS-SynGram metrics.

Evaluation dependencies

pip install ".[eval]"

Adds dependencies such as pandas, SacreBLEU, and BERTScore for Evaluate.py.

Diagnostic dashboard

pip install ".[dashboard]"

Adds Streamlit for the diagnostic dashboard.

Optional COMET comparison

pip install ".[comet]"

Adds the optional COMET comparison metric.

Development installation

For development, use an editable installation so code changes are picked up immediately:

pip install -e ".[eval,dashboard]"

---

Quick Start

After installation, run the example:

python examples/basic_example.py

The example demonstrates how to compute a GRIS score for a hypothesis/reference translation pair.

You can also use GRIS directly from Python:

from gris import compute_DepScore_emb, compute_syntactic_ngram_metric

dep = compute_DepScore_emb(
    hyps=["Der Hund lief schnell."],
    refs=["Der Hund rannte schnell."],
    lang="de"
)

print(dep)

---

Interpretability

GRIS can provide a detailed, step-by-step explanation of how a dependency-based score is calculated.

The explanation can include:

• matched dependency edges
• similarity scores
• linguistic bonuses
• precision, recall, and Fβ
• language blend weight
• sentence-level penalties

Print the explanation

from gris import compute_DepScore_emb

score = compute_DepScore_emb(
    hyps=["Der Hund lief schnell."],
    refs=["Der Hund rannte schnell."],
    lang="de",
    explain=True,
)

Return structured explanation

The explanation can also be returned as structured data:

from gris import compute_DepScore_emb, explain_dep_score

score, details = compute_DepScore_emb(
    hyps=["Der Hund lief schnell."],
    refs=["Der Hund rannte schnell."],
    lang="de",
    return_details=True,
)

explain_dep_score(details[0])

print(details[0]["matched"])

---

Command-Line Usage

GRIS provides a command-line interface for scoring translation files.

Standard scoring

gris-score --hyp hyps.txt --ref refs.txt --lang de

Scoring with explanation

gris-score --hyp hyps.txt --ref refs.txt --lang de --explain

---

Diagnostic Dashboard

Install the dashboard dependencies:

pip install ".[dashboard]"

Then run:

streamlit run $(python -c "import gris, os; print(os.path.join(os.path.dirname(gris.__file__), 'dashboard_corrected.py'))")

The dashboard provides an interactive and interpretable view of the GRIS scoring process.

---

Project Structure

GRIS/
├── gris/
│   ├── ...
│   ├── parser_constituency.py
│   ├── model_cache.py
│   └── dashboard_corrected.py
├── examples/
│   └── basic_example.py
├── figures/
│   └── banner.png
├── Evaluate.py
├── pyproject.toml
├── README.md
└── ...

---