Skip to content

CLAUDE.md

Latest commit

 

History

History
251 lines (196 loc) · 8.85 KB

CLAUDE.md

File metadata and controls

251 lines (196 loc) · 8.85 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

This repository contains the architectural design and implementation guidelines for the Atomic Evaluation Engine (AEE) - a deterministic, zero-leakage AI evaluation system built on the Atoms of Thought (AoT) philosophy using the Agno framework.

Core Concept: Replace probabilistic "vibes-based" AI evaluation with deterministic logic gates (Atoms) composed into workflows (Molecules) that produce structured, auditable outputs without narrative reasoning.

Philosophy: Atoms of Thought (AoT)

Core Principles

  1. No Leakage: Agents return structured data (Pydantic models) only - never narrative text or explanations
  2. Binary Truth: Verification is boolean (True/False), not a probabilistic scale
  3. Composability: Complex tasks decompose into single-purpose Atoms that chain into Molecules
  4. Auditability: Every atomic decision is logged with input/output pairs for debugging

The Atom Pattern

An Atom is a single-purpose cognitive unit:

  • One Purpose: Single, well-defined transformation or check
  • Explicit Inputs: Strict Pydantic input schema
  • One Verifiable Output: Strict Pydantic output schema
  • No Narrative Reasoning: Internal deliberation never exposed in output

The Molecule Pattern

A Molecule is a workflow that chains Atoms:

  • Decomposer Atom → List[Claims] → Verifier Atom → Boolean Results → Scorer Atom → Final Score
  • Each step passes structured data to the next
  • Python logic handles iteration/mapping, not LLMs

Agno Framework Implementation

Agent Configuration (Critical Parameters)

All Agno agents in this project MUST use these parameters:

from agno.agent import Agent
from agno.models.openai import OpenAIChat

agent = Agent(
    model=OpenAIChat(id="gpt-4o-mini", temperature=0),  # Zero temp for determinism
    description="Clear, single-purpose description",
    response_model=MyPydanticSchema,      # MANDATORY: Enforces structured output
    show_full_reasoning=False,             # CRITICAL: Prevents narrative leakage
    markdown=False,                        # Prevents formatting that implies narrative
    instructions=["Specific step-by-step instructions"]
)

Key Parameters:

  • response_model: Enforces strict Pydantic output schema (no free-form text)
  • show_full_reasoning=False: Hard-codes silence, preventing internal reasoning from leaking
  • temperature=0: Maximizes determinism
  • markdown=False: Prevents narrative-style formatting

Workflow Orchestration

Use Workflow to chain Atoms, not monolithic agents:

from agno.workflow.workflow import Workflow
from agno.workflow.step import Step

class AtomicWorkflow(Workflow):
    decomposer: Step = Step(agent=decomposer_agent)
    verifier: Step = Step(agent=verifier_agent)
    scorer: Step = Step(executor=custom_scoring_function)  # Python logic, not LLM

    def run(self, input_data):
        claims = self.decomposer.run(input_data)
        results = [self.verifier.run(c) for c in claims.content]
        final = self.scorer.run(results)
        return final

Architecture: Three-Atom Evaluation Pipeline

Atom 1: Decomposer (Entropy Reduction)

  • Input: Raw text to evaluate
  • Output: List[AtomicClaim] (single, falsifiable statements)
  • Instructions: "Break complex text into atomic, falsifiable claims. Split compound sentences. Ignore fluff. Preserve specific numbers/dates exactly."

Atom 2: Verifier (Boolean Logic Gate)

  • Input: AtomicClaim + ground_truth document
  • Output: VerificationResult (boolean + evidence quote)
  • Instructions: "You are a Boolean Truth Function. Return True if ground truth explicitly supports claim, False otherwise. Do not hallucinate support."

Atom 3: Scorer (Deterministic Calculator)

  • Input: List[VerificationResult]
  • Output: FinalScore (0-100 integer + PASS/FAIL status)
  • Instructions: "Calculate percentage of True results. Apply Zero Tolerance rule: any critical claim marked False = automatic FAIL."

Development Commands

Python Environment Setup

# Use uv (preferred) or poetry for dependency management
uv sync                          # Install dependencies
source .venv/bin/activate        # Activate virtual environment (if not auto-activated)

Testing

uv run pytest                    # Run all tests
uv run pytest -q                 # Quiet mode (before PRs)
uv run pytest tests/atoms/       # Test specific module
uv run pytest -k test_verifier   # Run specific test
uv run pytest --cov=.            # With coverage report

Running Examples

uv run python examples/demo.py               # Basic demo (should run without API keys)
uv run python examples/medical_eval.py       # Medical hallucination detection example

Database (for audit trails)

docker compose up db             # Start PostgreSQL for logging atomic traces

Project Structure

AtomsOfThought/
├── AGENTS.md           # Repository guidelines (build/test/commit)
├── notes1.md           # Original AoT architecture and PRD
├── notes1a.md          # Agno framework alignment guide
├── docs/               # Additional ADR-style documentation
├── src/                # Runtime code (when implemented)
│   ├── atoms/          # Individual atom agents
│   ├── workflows/      # Molecule orchestrations
│   └── schemas.py      # Pydantic type system
├── examples/           # Runnable demos
├── tests/              # pytest test suites
│   └── atoms/          # Unit tests for each atom
└── scripts/            # One-off automation

Testing Requirements

Golden Set Testing

Create datasets of (Claim, Context, Expected_Bool) for regression testing:

# Example: Testing the Verifier Atom
test_cases = [
    {
        "claim": "Users usually click 'Buy'",
        "context": "Click-through rate is 2%.",
        "expected": False  # Hallucination
    },
    {
        "claim": "The rate is 2%",
        "context": "Click-through rate is 2%.",
        "expected": True
    }
]

Coverage Requirements

  • Target: >90% code coverage for all modules
  • Each Atom requires dedicated unit tests
  • At least one workflow-level integration test for orchestration changes
  • Hallucination regression cases for Verifier Atom

Schemas and Type System

Define all data structures in schemas.py first:

from pydantic import BaseModel, Field
from typing import List, Literal

class InputContext(BaseModel):
    ground_truth: str
    generated_text: str
    criteria: str

class AtomicClaim(BaseModel):
    claim_text: str = Field(..., description="A single factual statement")
    criticality: Literal["high", "low"] = "high"

class VerificationResult(BaseModel):
    claim: AtomicClaim
    is_supported: bool
    evidence_quote: str = Field(..., description="Exact substring from ground truth")

class FinalScore(BaseModel):
    score: int = Field(..., ge=0, le=100)
    status: Literal["PASS", "FAIL"]
    failed_claims: List[VerificationResult]

Commit Guidelines

  • Format: Imperative present tense with scope prefix: agents: add verifier schema
  • Before Committing: Run uv run pytest -q to ensure all tests pass
  • Include: Before/after artifacts (JSON outputs or logs) for agent behavior changes
  • Never Commit: Secrets, API keys, or large data files

Common Pitfalls to Avoid

  1. Using narrative agents: Never create agents that "think out loud" or explain reasoning in output
  2. Skipping response_model: All agents MUST have strict Pydantic output schemas
  3. Forgetting show_full_reasoning=False: This causes reasoning leakage
  4. Monolithic prompts: One giant prompt asking for evaluation (Bad) vs. chained Atoms (Good)
  5. LLM-based iteration: Loops/mapping should happen in Python, not inside LLM calls

Advanced Features (Phase 3+)

Parallel Verification

Use asyncio.gather to verify claims concurrently:

results = await asyncio.gather(*[verifier.run(c) for c in claims])

Database Audit Trail

Configure agents with PostgreSQL storage:

from agno.db.postgres import PostgresDb

agent = Agent(
    storage=PostgresDb(...),
    save_run=True,  # Saves input/output of every run
    ...
)

Cultural Knowledge (Self-Learning)

For long-term learning across runs:

agent = Agent(
    add_culture_to_context=True,        # Load learned best practices
    update_cultural_knowledge=True,      # Auto-update knowledge from successful runs
    ...
)

References

  • Agno Documentation: Formerly Phidata, focus on Agents, Workflows, structured output
  • Related Files:
    • AGENTS.md: Build/test/commit guidelines
    • notes1.md: Original AoT architecture and PRD
    • notes1a.md: Agno framework implementation details