Darwin

Autonomous AI Research Laboratory for Document Intelligence

Evolution through selection — experiments evolve until the best model survives.

pip install darwin-lab

---

Why "Darwin"?

Named after Charles Darwin and his foundational theory of evolution through natural selection ("survival of the fittest"), this platform applies evolutionary principles to deep learning research.

In biological evolution:

1. Organisms mutate and adapt to environmental pressures.
2. Unfit traits are eliminated through empirical failure.
3. Only the strongest variants survive to the next generation.

In Darwin:

1. Model variants, augmentations, and hyperparameters act as competing traits.
2. The Failure Analyzer identifies environmental pressures (rotation distortion, low OCR confidence, scan artifacts).
3. The Hypothesis Generator formulates targeted mutations for the next generation.
4. Less fit architectures are discarded, and only the highest-performing pipeline with verified accuracy survives.

---

The Problem

Today, improving a document AI model requires manual, repetitive human labor:

• Read papers -> choose an architecture
• Fine-tune -> compare benchmarks
• Analyze failure samples -> tweak hyperparameters
• Retrain -> draft research reports

Darwin automates 80% of that workflow.

---

Quickstart

from darwin import ResearchLab

lab = ResearchLab(
    objective="Improve OCR robustness under motion blur for identity document verification"
)
lab.run()

Darwin autonomously executes:

1. Paper Search: Queries arXiv and Semantic Scholar for SOTA literature on your objective.
2. Dataset Discovery: Selects the optimal dataset from a curated catalog (RVLCDIP, FUNSD, CORD, DocLayNet).
3. Model Selection: Chooses the best architecture (LayoutLMv3, Donut, Florence-2, TrOCR).
4. Training: Fine-tunes via LoRA/PEFT with document-specific augmentations.
5. Failure Analysis: Quantifies error patterns across metadata features (rotation, resolution, OCR confidence).
6. Hypothesis Generation: Formulates structured, testable hypotheses for the next experiment iteration.
7. Selection Loop: Iterates until accuracy gains plateau or max iterations are reached.
8. Reporting: Generates a comprehensive research report in Markdown.

---

Architecture

                    User Research Goal
                            │
                            ▼
              ┌──────────────────────────┐
              │   Darwin Orchestrator    │
              │  (LangGraph StateGraph   │
              │    + Checkpointing)      │
              └──────────────────────────┘
                            │
         ┌──────────────────┼──────────────────┐
         ▼                  ▼                  ▼
    Paper Agent        Dataset Agent      Model Agent
   arXiv + S2 RAG     HF Hub Catalog     LLM Selection
         │                  │                  │
         └──────────────────┼──────────────────┘
                            ▼
              ┌──────────────────────────┐
              │   Training Orchestrator  │
              │   HuggingFace Trainer    │
              │   + PEFT / LoRA          │
              │   + Augmentations        │
              └──────────────────────────┘
                            │
                            ▼
              ┌──────────────────────────┐
              │   Experiment Tracker     │
              │   MLflow Integration     │
              └──────────────────────────┘
                            │
                            ▼
              ┌──────────────────────────┐
              │    Failure Analyzer      │ <-- Natural Selection Engine
              │  Rotation x Quality x    │
              │  OCR confidence x ...    │
              └──────────────────────────┘
                            │
                            ▼
              ┌──────────────────────────┐
              │   Hypothesis Generator   │
              │   LLM Guided Mutation    │
              │   Papers + Failures      │
              │   -> Testable Hypothesis │
              └──────────────────────────┘
                            │
                     ┌──────┴──────┐
                     ▼             ▼
                 Continue?     Generate
                   (yes)        Report
                     │
                     └──> Training Orchestrator (next generation)

---

Example Research Cycle

Objective: Detect forged identity documents under real-world degradations.

--- Iteration 1 (Generation 1) -----------------------------------
Model:    LayoutLMv3-Base
Accuracy: 91.3%   Macro F1: 88.7%

Failure Analysis:
  - Documents rotated >15 deg:     error rate = 34%  (48 samples)
  - OCR confidence <0.6:           error rate = 41%  (39 samples)
  - Handwritten text samples:      error rate = 37%  (22 samples)

Hypothesis H-001 [confidence: medium]:
  "Adding synthetic handwriting augmentation + stronger rotation (+/-30 deg)
   will improve accuracy on handwritten documents from 78.3% to ~85%."
  Predicted improvement: +3.5%

--- Iteration 2 (Generation 2) -----------------------------------
Model:    LayoutLMv3-Base  (LoRA rank 16, augs: rotation + handwriting)
Accuracy: 94.8%   Macro F1: 92.1%   (+3.5%)

Failure Analysis:
  - Low JPEG quality (<50):        error rate = 28%  (61 samples)
  - Stamp overlay:                 error rate = 22%  (18 samples)

Hypothesis H-002 [confidence: high]:
  "Switching to Donut (OCR-free) eliminates the OCR confidence failure
   mode entirely. JPEG augmentation (quality 20-95) addresses scan artifacts."
  Based on: [2111.15664] Donut: OCR-free Document Understanding
  Predicted improvement: +2.8%

--- Iteration 3 (Selection Fittest) -------------------------------
Model:    Donut-Base  (LoRA rank 8, augs: JPEG + rotation + erasing)
Accuracy: 96.1%   Macro F1: 93.8%   (+1.3%)

Research report generated: experiments/reports/darwin_report_20240725.md

---

Failure Analyzer (Natural Selection Engine)

Standard tools output static metrics. Darwin diagnoses underlying failure conditions:

# Standard Output:
"Accuracy: 91.3% -> needs improvement"

# Darwin Output:
{
  "failure_conditions": [
    {"condition": "Documents rotated >15 deg",   "error_rate": 0.34, "n": 48},
    {"condition": "OCR confidence <0.6",         "error_rate": 0.41, "n": 39},
    {"condition": "Face area <18% of image",     "error_rate": 0.28, "n": 55},
    {"condition": "Handwritten text samples",    "error_rate": 0.37, "n": 22}
  ],
  "suggested_fixes": [
    "Add rotation augmentation (up to +/-30 deg)",
    "Switch to Donut (OCR-free) to eliminate OCR dependency",
    "Add synthetic handwriting overlay augmentation",
    "Use random_erasing to simulate stamp occlusion"
  ]
}

---

Supported Architectures

Model	Family	Requires OCR	Primary Domain
microsoft/layoutlmv3-base	LayoutLMv3	Yes	Forms, structured classification
naver-clova-ix/donut-base	Donut	No	End-to-end receipt / OCR extraction
microsoft/Florence-2-base	Florence-2	No	Multi-task vision, VQA, fraud
microsoft/trocr-base-printed	TrOCR	No	Printed text recognition
microsoft/trocr-base-handwritten	TrOCR	No	Handwritten text recognition

---

Training Modes

Mode	Description	GPU Needed?	Best For
demo	Instant simulation with synthetic metrics	No	Fast UI / CLI verification
hybrid (default)	Real inference + realistic fine-tuning simulation	No	Local Mac / laptop workflow
full	Real PyTorch + PEFT/LoRA fine-tuning with HF Trainer	Yes	Full GPU execution

Real GPU Training on Google Colab

To run full GPU model fine-tuning (training_mode="full"), use our Google Colab notebook:

examples/Darwin_Colab_GPU_Training.ipynb

1. Open the notebook in Google Colab.
2. Select a T4 GPU runtime (Runtime -> Change runtime type -> T4 GPU).
3. Upload darwin_code.zip or set your GOOGLE_API_KEY to run full PyTorch LoRA fine-tuning.

---

Configuration

# configs/default.yaml
llm:
  provider: gemini           # gemini | openai | ollama
  model_name: gemini-2.5-flash
  temperature: 0.1

training:
  mode: hybrid               # demo | hybrid | full
  default_lora_rank: 8
  default_learning_rate: 2e-4

max_iterations: 3
min_improvement_delta: 0.005  # Stop if improvement < 0.5%

Or via environment variables:

DARWIN_LLM_PROVIDER=gemini
DARWIN_TRAINING_MODE=hybrid
DARWIN_MAX_ITERATIONS=3
GOOGLE_API_KEY=your_key_here

---

Commands

# Install
make install-dev

# Run a research cycle (CLI)
darwin run "Improve OCR robustness under motion blur" --mode hybrid --iterations 3

# Run the demo (no GPU needed)
make demo

# Launch the dashboard
make dashboard                    # http://localhost:8501

# View MLflow experiments
make mlflow-ui                    # http://localhost:5000

# Run tests
make test

# Check environment
darwin status

---

Repository Structure

darwin/
├── darwin/
│   ├── __init__.py             # Public API: ResearchLab
│   ├── cli.py                  # CLI commands
│   ├── core/
│   │   ├── state.py            # DarwinState TypedDict (shared state)
│   │   ├── config.py           # Pydantic config (LLM + training + MLflow)
│   │   └── orchestrator.py     # LangGraph StateGraph (orchestrator engine)
│   ├── agents/
│   │   ├── paper_agent.py      # arXiv + Semantic Scholar + RAG
│   │   ├── dataset_agent.py    # HuggingFace Hub dataset discovery
│   │   └── model_agent.py      # LLM architecture selection
│   ├── trainer/
│   │   ├── hf_trainer.py       # HuggingFace Trainer + LoRA
│   │   ├── augmentations.py    # Document degradation pipeline
│   │   └── lora_config.py      # PEFT configurations per model family
│   ├── evaluator/
│   │   └── evaluator.py        # Confusion matrix + precision/recall
│   ├── analyzer/
│   │   └── failure_analyzer.py # Systematic error pattern detection
│   ├── tracker/
│   │   └── mlflow_tracker.py   # MLflow logging + artifact generation
│   ├── hypothesis/
│   │   └── generator.py        # LLM hypothesis generation engine
│   ├── reporter/
│   │   └── report_generator.py # Markdown research report renderer
│   └── dashboard/
│       └── app.py              # Streamlit monitoring dashboard
├── configs/                    # YAML configuration files
├── examples/                   # Quickstart, demo, and Colab GPU notebook
├── tests/                      # Pytest unit test suite
└── pyproject.toml              # Package configuration (pip install darwin-lab)

---

Technology Stack

Component	Technology
Agent Orchestration	LangGraph (StateGraph + Checkpointing)
LLM Backbone	Gemini 2.5 Flash / GPT-4o / Ollama (configurable)
Paper Retrieval	arxiv + semanticscholar Python libraries
Model Fine-Tuning	HuggingFace Trainer + PEFT/LoRA + Accelerate
Experiment Tracking	MLflow (metrics, artifacts, model registry)
Augmentations	Albumentations + PIL
Dashboard	Streamlit + Plotly
Package	pyproject.toml (pip install darwin-lab)
Config	Pydantic v2 + YAML
Testing	pytest + pytest-mock

---

License

MIT License — see LICENSE for details.

---

Made with ❤️ by Sarvagna