Skip to content

README.md

Latest commit

 

History

History
134 lines (104 loc) · 4.66 KB

README.md

File metadata and controls

134 lines (104 loc) · 4.66 KB
Raw

Synthetic Population Experiments

This workspace is organized to handle multiple experiments with different configurations, data, and analysis requirements. The LLMPopulationGenerator package is installed as an editable dependency (pip install -e), allowing direct imports while keeping your custom schemas, prompts, and data separate.

Directory Structure

Synthetic-Population-Experiments/
├── requirements.txt              # Python dependencies
├── .env                         # Environment variables (API keys, etc.)
├── .env.example                 # Example environment variable template
├── README.md                    # This file
├── setup_shared_files.py        # Validation script for package installation
│
├── shared/                      # Shared resources across experiments
│   ├── utils/                   # Utility functions and classes
│   │   ├── experiment_utils.py  # General experiment utilities
│   │   ├── fit_metrics.py       # Demographic fit metric calculations
│   │   ├── model_factory.py     # LLM model creation factory
│   │   ├── plot_config.py       # Shared plotting configuration
│   │   ├── srmse_calculator.py  # SRMSE calculation utilities
│   │   └── validation_rules.py  # Data validation rules
│   ├── prompts/                 # Custom prompt templates
│   └── schemas/                 # Custom JSON schemas
│
├── experiments/                 # Individual experiments
│   ├── 01_llm_benchmark/       # LLM benchmarking experiment
│   ├── 02_global/              # Global population generation
│   ├── 03_newcastle/           # Newcastle case study
│   ├── 04_dar_es_salaam/       # Dar es Salaam case study
│   ├── 05_prompt_sensitivity/  # Prompt sensitivity analysis
│   ├── 06_no_feedback/         # Generation without feedback
│   ├── 07_method_benchmark/    # Method benchmarking
│   └── 08_batch_generation/    # Batch generation experiment
│
├── data_repository/             # Canonical input data and data dictionary
├── checkpoints/                 # Saved experiment checkpoints
└── figures_for_publication/     # Publication-ready figures

Each experiment follows a common structure:

<experiment>/
├── config/             # Experiment-specific configuration
├── data/               # Experiment input/processed data
├── scripts/            # Experiment scripts
└── outputs/            # Generated results and reports

Setup Instructions

  1. Install Dependencies

    cd Synthetic-Population-Experiments
pip install -r requirements.txt

    Then clone and install the LLMPopulationGenerator package in editable mode:

    git clone git@github.com:MJones235/LLM-Population-Generator.git
pip install -e LLM-Population-Generator

  2. Validate Package Installation

    python setup_shared_files.py

    This verifies that the LLMPopulationGenerator package can be imported and is properly installed.

  3. Add Your Custom Files

    • Copy or create prompts in shared/prompts/
    • Copy or create schemas in shared/schemas/

  4. Environment Variables Create a .env file with your API credentials:

    AZURE_OPENAI_API_KEY=your_azure_key_here
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_API_VERSION=2024-08-01-preview
AZURE_OPENAI_TEMPERATURE=0.7

Running the LLM Benchmark Experiment

  1. Navigate to experiment directory

    cd experiments/01_llm_benchmark

  2. Configure models to test Edit config/llm_models.yaml to add/remove models

  3. Configure experiment parameters
    Edit config/experiment_config.yaml for generation settings

  4. Run the benchmark

    python scripts/run_benchmark.py

Experiment Results

Results are saved to experiments/01_llm_benchmark/outputs/ with:

  • Individual model results in timestamped folders
  • Population data in JSON and CSV formats
  • Cost analysis and failure statistics
  • Comprehensive experiment summary

Adding New Experiments

To create a new experiment:

  1. Create directory: experiments/09_new_experiment/
  2. Copy structure from an existing experiment (e.g. 01_llm_benchmark/)
  3. Update configurations for your specific needs
  4. Implement experiment-specific scripts
  5. Use shared utilities and data as needed

Analysis

Each experiment includes:

  • Automated cost tracking
  • Failure analysis and statistics
  • Demographic distribution analysis
  • Model comparison metrics
  • Exportable reports and summaries