autocritic

Critic-card-driven image evaluation using historical art theory and multimodal LLMs.

Load a critic card distilled from a given art theory book. The critic evaluates images through that theorist's lens, producing structured qualitative feedback with numeric scores. An optional improvement loop feeds critiques back into a generative system, steering parameter changes through LLM-driven translation — no hardcoded mapping tables.

How it works

                    ┌──────────────┐
                    │  Critic Card │  (JSON: Wölfflin, Kandinsky, Arnheim, ...)
                    │  ─ thesis    │
                    │  ─ criteria  │
                    │  ─ indicators│
                    └──────┬───────┘
                           │
    ┌─────────┐    ┌───────▼───────┐    ┌───────────────┐
    │  Image  │───▶│  LLM Vision   │───▶│ ScoredCritique│
    └─────────┘    │  (critique)   │    │ ─ axis scores │
                   └───────────────┘    │ ─ directives  │
                                        └───────┬───────┘
                                                │
                              ┌──────────────────▼──────────────────┐
                              │         LLM Translator              │
                              │  critique → parameter deltas        │
                              │  (no hardcoded mapping — LLM reasons│
                              │   about which params to change)     │
                              └──────────────────┬──────────────────┘
                                                 │
                                        ┌────────▼────────┐
                                        │    Generator    │
                                        │ (rewriteDrawer, │
                                        │  morphogenesis, │
                                        │   your system)  │
                                        └─────────────────┘

Installation

pip install -e ".[dev]"

# For OpenAI models:
pip install -e ".[openai]"

# For Anthropic models:
pip install -e ".[anthropic]"

# For the rewriteDrawer adapter:
pip install -e ".[rewriter]"

# Everything:
pip install -e ".[all-llms,rewriter,dev]"

Requires Python 3.11+.

Quickstart

Evaluate a single image

from autocritic import load_critic, critique_image

critic = load_critic("critics/wolfflin.json")
result = critique_image(critic, "path/to/image.png")
print(result.summary())

Score an image with numeric axis scores

from autocritic import load_critic, score_critique

critic = load_critic("critics/arnheim.json")
scored = score_critique(critic, "path/to/image.png", model="openai:gpt-5.4-mini")

print(f"Composite: {scored.composite_score:.2f}")
for axis in scored.axis_scores:
    print(f"  {axis.label}: {axis.score:.2f} — {axis.reasoning}")

Critique from the command line

The repo includes 5 CC0 sample images from The Met's Open Access collection in tests/fixtures/samples/ — use them to try things out immediately.

# Critique a sample image — no generator needed
python3 -m autocritic critique tests/fixtures/samples/Friedrich.jpg --critic wolfflin --model openai:gpt-5.4-mini

# Try different theoretical lenses on the same image
python3 -m autocritic critique tests/fixtures/samples/Hiroshige.jpg --critic kandinsky --model openai:gpt-5.4-mini
python3 -m autocritic critique tests/fixtures/samples/Goltzius.jpg --critic arnheim --model openai:gpt-5.4-mini

# Critique your own images
python3 -m autocritic critique your-image.png --critic ruskin --model openai:gpt-5.4-mini

# See available critics
python3 -m autocritic list

Run the improvement loop

The improvement loop connects autocritic to a generative system. It generates an image, critiques it, translates the critique into parameter changes, and repeats.

# Start your generator server first (e.g. rewriteDrawer), then:
python3 -m autocritic run --critic wolfflin --generator rewriter --model openai:gpt-5.4-mini --iterations 5

# Try different critics to get different aesthetic directions
python3 -m autocritic run --critic kandinsky --generator rewriter --model openai:gpt-5.4-mini --iterations 5

# Add an intent to guide the critique
python3 -m autocritic run --critic arnheim --generator rewriter --model openai:gpt-5.4-mini --iterations 5 --intent "organic branching structure"

Results are saved to runs/ with a contact sheet and narrative summary.

Evaluate critic cards

The eval harness measures critic card quality and critique usefulness without manual inspection.

# Programmatic checks: structure, vocabulary, ground truth (fast, no API key)
python3 -m autocritic eval

# Usefulness checks: completeness, lens vocabulary, expected terms, coherence
# (requires cached fixtures — first run generates them via API)
python3 -m autocritic eval --suite usefulness --model openai:gpt-5.4

# Re-run usefulness evals against cached fixtures (instant, no API calls)
python3 -m autocritic eval --suite usefulness

# Regenerate fixtures after changing a critic card
python3 -m autocritic eval --suite usefulness --refresh --model openai:gpt-5.4

# Run all suites for a single critic
python3 -m autocritic eval --suite all --critic wolfflin

# Include LLM-as-judge evals (actionability + anti-generic checks)
python3 -m autocritic eval --suite usefulness --judge --model openai:gpt-5.4

Results are saved as timestamped JSON in evals/results/. See python3 -m autocritic eval --help for all options.

Other commands

# Validate critic card files
python3 -m autocritic validate critics/*.json

# Generate a contact sheet for an existing run
python3 -m autocritic report runs/rewriter_1774718482/

# List available critic cards
python3 -m autocritic list

Available critics

Card	Theorist	Book	Type
albers	Josef Albers	Interaction of Color	5 core concepts (unipolar)
arnheim	Rudolf Arnheim	Art and Visual Perception	4 principles (unipolar)
dow	Arthur Wesley Dow	Composition	3 elements + 5 principles (unipolar)
gombrich	Ernst Gombrich	The Sense of Order	5 core concepts (unipolar)
hildebrand	Adolf Hildebrand	The Problem of Form	5 core concepts (unipolar)
kandinsky	Wassily Kandinsky	Point and Line to Plane	3 elements + 4 principles (unipolar)
klee	Paul Klee	Pedagogical Sketchbook	5 core concepts (unipolar)
ruskin	John Ruskin	The Elements of Drawing	5 core concepts (unipolar)
wolfflin	Heinrich Wölfflin	Principles of Art History	5 criteria (bipolar)
worringer	Wilhelm Worringer	Abstraction and Empathy	2 impulses + 3 dimensions (unipolar)

Bipolar critics (Wölfflin) score on a -1 to +1 spectrum between two poles — neither pole is superior. Unipolar critics score 0 to 1 on presence/strength of each concept. Cards can declare "scoring_mode": "bipolar" explicitly; otherwise the mode is inferred from indicator structure (pole_a/pole_b).

Authoring new critics

See AUTHORING.md for the full guide to creating your own critic cards, and schemas/ for the JSON schema and template.

Quick overview:

1. Start from schemas/template_card.json
2. Fill in the theorist's evaluative framework
3. Validate: python3 -m autocritic validate critics/your_card.json

Architecture

src/autocritic/
├── critic.py        # Load critic cards, generate qualitative critiques
├── scoring.py       # Extend critiques with numeric axis scores
├── translator.py    # LLM-driven critique → parameter delta translation
├── loop.py          # Generate → critique → translate → adjust → repeat
├── report.py        # Contact sheets and narrative summaries
├── validate.py      # Critic card validation against schema
├── llm.py           # Multi-provider LLM routing (OpenAI, Anthropic, etc.)
├── __main__.py      # CLI entry point
└── adapters/
    └── rewriter.py  # rewriteDrawer HTTP client and parameter space

critics/             # 10 critic cards (JSON)
schemas/             # JSON schema and template for authoring
evals/
├── judge.py         # LLM-as-judge and programmatic eval primitives
├── runner.py        # Eval suite runner and result persistence
├── usefulness.py    # Usefulness eval suite (fixture-based)
├── ground_truth/    # Per-theorist ground truth (10 modules)
├── fixtures/        # Test images and cached critique fixtures
└── rubrics/         # Eval rubric definitions
scripts/             # Distillation scripts (book → critic card)
tests/               # 654+ tests (loop, scoring, validation, evals)

Key concepts

• Critic card: A JSON file encoding one art theorist's evaluative framework — thesis, criteria, indicators, diagnostic questions, feedback directions, anti-goals. The card tells the LLM how to see from that theorist's perspective.

• ParamSpace: A generator-agnostic description of tunable parameters. Any generative system that defines a ParamSpace can plug into the improvement loop.

• LLM-driven translation: Instead of hardcoded mapping tables from critique terms to parameter names, the LLM reads the critique, looks at the parameter space, and reasons about what to change. This is the core design principle — flexibility over rigidity.

• Damped deltas: Parameter changes are applied with damping (new = current + damping * (target - current)) to prevent oscillation. Rejected iterations halve the deltas before retrying.

License

MIT. See LICENSE.

The critic cards in critics/ are derived from published art history and theory texts. The cards contain original analysis and selected citations under fair use. The source texts themselves are not included in this repository.