nl-two-layer-content-moderation-gates

An educational Python project for two-layer content moderation gates using Detoxify and a local Ollama LLM reviewer.

The first layer scores text with a local toxicity classifier. The second layer asks a local LLM for a structured review only when policy says the case is ambiguous, flagged, blocked, or sensitive. Deterministic code owns thresholds, self-harm escalation, final decisions, exit codes, and JSON audit evidence.

Overview

This repository demonstrates a production-shaped moderation control loop:

1. Load typed configuration from JSON with environment overrides.
2. Read a frozen moderation case set.
3. Score each input with Detoxify.
4. Apply deterministic flag, block, and ambiguous-review thresholds.
5. Escalate self-harm or crisis-language signals outside the toxicity labels.
6. Optionally ask a local Ollama model for one structured JSON review.
7. Keep the final moderation decision in code.
8. Persist the complete decision path as an audit artifact.

The classifier and LLM provide signals. They do not own the moderation policy.

What This Project Demonstrates

• Python-native Detoxify integration
• Local Ollama review through the OpenAI-compatible /v1/chat/completions endpoint
• Typed Pydantic configuration and result models
• Local endpoint allowlisting for the LLM reviewer
• Deterministic thresholds for allow, flag, block, and escalate
• Separate escalation for self-harm and crisis-language signals
• Fail-closed behavior when the LLM review fails
• Replayable JSON cases
• JSON audit records for inspection
• Tests that exercise the policy layer without requiring Detoxify weights or Ollama

Prerequisites

• Python 3.11 or later
• Ollama installed locally
• A local Ollama chat model for review, such as llama3.2:3b

Detoxify downloads model weights through its normal PyTorch and Transformers path on first use. The tests do not require those weights.

Quick Start

Create the Python environment from the repository root:

python -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install -e ".[dev]"

Start Ollama and pull the configured review model:

ollama pull llama3.2:3b
ollama serve

Run the sample gate:

moderation-gate --config config/local-ollama.json --cases data/moderation_cases.json

The command returns exit code 2 when any case is blocked or escalated. That makes the sample usable as a CI-style gate if the case set represents content that must remain acceptable.

Configuration

Configuration is explicit JSON:

{
  "detoxify": {
    "model_name": "unbiased",
    "device": "cpu"
  },
  "llm": {
    "enabled": true,
    "base_url": "http://localhost:11434/v1",
    "api_key": "ollama",
    "model": "llama3.2:3b",
    "timeout_seconds": 45,
    "review_mode": "ambiguous"
  },
  "policy": {
    "max_input_chars": 4000,
    "flag_thresholds": {
      "toxicity": 0.45,
      "severe_toxicity": 0.25,
      "threat": 0.30
    },
    "block_thresholds": {
      "toxicity": 0.85,
      "severe_toxicity": 0.60,
      "threat": 0.70
    },
    "ambiguous_margin": 0.08,
    "self_harm_terms": ["kill myself", "end my life", "suicide"]
  }
}

The following environment variables override local model settings:

MODGATE_LLM_MODEL
MODGATE_LLM_BASE_URL
MODGATE_LLM_API_KEY
MODGATE_DETOXIFY_DEVICE

The LLM endpoint must remain local: localhost, 127.0.0.1, or ::1.

How It Works

Detoxify scores first

DetoxifyClassifier lazily loads the configured Detoxify model and converts returned labels into the policy label names used by the rest of the app. For example, older toxic and severe_toxic names are normalized to toxicity and severe_toxicity.

Policy owns the first decision

ModerationPolicy checks empty input, maximum length, self-harm terms, block thresholds, flag thresholds, and the ambiguous review band. This pass produces an initial decision and a reason list before any LLM review happens.

The LLM reviews selected cases

OllamaReviewer calls the local OpenAI-compatible Ollama endpoint with JSON mode enabled. It asks for a small object containing:

• category
• recommended_decision
• confidence
• rationale

If the request fails, the review returns escalate. Moderation systems should not turn unavailable review infrastructure into silent approval.

Code keeps the final gate

The final decision keeps the stricter result. The LLM can raise a decision from allow to flag, block, or escalate, but it cannot lower a deterministic block into an allow decision.

Project Structure

.
|-- config/
|   `-- local-ollama.json
|-- data/
|   `-- moderation_cases.json
|-- src/content_moderation_gates/
|   |-- audit.py
|   |-- classifier.py
|   |-- cli.py
|   |-- config.py
|   |-- engine.py
|   |-- models.py
|   |-- policy.py
|   `-- reviewer.py
|-- tests/
|-- LICENSE
|-- README.md
`-- pyproject.toml

Control Guarantees

• The LLM review endpoint must be local.
• Configuration rejects unknown fields.
• Empty and oversized inputs fail before model review.
• Toxicity thresholds are deterministic policy settings.
• Self-harm language is treated as an escalation category, not as toxicity.
• LLM review failure escalates instead of allowing by default.
• The final decision preserves the stricter policy outcome.
• Audit records include classifier scores, review output, labels, reasons, and final decision.

Tests

Run the deterministic test suite without Detoxify weights or an Ollama server:

pytest

The tests use fake classifier and reviewer adapters so policy behavior stays fast and reproducible.

License

This project is licensed under the MIT License. See LICENSE.

Contributing

Contributions are welcome for improvements within current project scope.

Suggested areas:

• Additional deterministic sensitive-content detectors
• More reviewer schemas
• Calibration reports for named Detoxify thresholds
• Batch-mode audit reports
• Provider adapters for other local OpenAI-compatible runtimes