Inspect Evals Scoring

Scoring infrastructure for the Inspect Evals dashboard.

Project Purpose

inspect_evals_scoring is designed to:

• Provide infrastructure for running evaluations at scale using AWS Batch
• Offer a unified CLI interface for managing evaluation jobs
• Support processing and summarizing (i.e. creating aggregated metrics for) evaluation results
• Enable promotion of evaluation logs to different environments
• Facilitate integration with dashboard visualization tools

CLI Usage

This package provides a unified command-line interface (CLI) through the ies command, which organizes all functionality into logical subcommands.

The run commands take most of the same parameters as those passed to inspect eval or inspect eval-set.

Basic Usage

# Run AWS Batch evaluation
ies run aws --tasks=inspect_evals/pubmedqa --model=openai/gpt-4o-mini

# Run local evaluation
ies run local --tasks=inspect_evals/pubmedqa --model=openai/gpt-4o-mini

# Monitor a job
ies batch monitor-job 5b2152b4-7c28-4d95-9303-45ebe8b08a9b

# Terminate a job
ies batch terminate-job 5b2152b4-7c28-4d95-9303-45ebe8b08a9b

# Check job status
ies status check --run-id 2023-07-05-10-55-42

# Check logs directly
ies status logs --log-dir s3://bucket/logs/working/2023-07-05-10-55-42

# Promote logs
ies promote --run-id 2023-07-05-10-55-42

Command Structure

• ies run: Commands for running evaluations

  • aws: Run evaluation on AWS Batch
  • local: Run evaluation locally

• ies batch: AWS Batch job management commands

  • monitor-job: Monitor a specific AWS Batch job
  • terminate-job: Terminate an AWS Batch job

• ies status: Commands for checking status

  • check: Check status of a job by run ID
  • logs: Check logs directly from a log directory
  • list: List recent jobs

• ies promote: Promote logs to the dashboard

• ies sandbox: Sandbox utilities

Docker Setup

This project includes Docker support for running evaluations in a containerized environment with Docker-in-Docker capabilities. This is primarily used as the base image for the AWS Batch job, but can also be run locally for development and testing.

For detailed instructions on the Docker setup, see the aws/README.md file.

Running Evaluations

You can run evaluations using the CLI, either locally or on AWS Batch. In both cases, the actual evaluation is run by calling inspect_ai.eval_set. The ies script provides a wrapper around inspect_ai.eval_set to handle the configuration and logging.

Local

ies run local --tasks=inspect_evals/pubmedqa --model=anthropic/claude-3-7-sonnet-20250219

Running evals locally will output the logs to the same s3 location as AWS Batch. It is therefore possible (and often useful for debugging) to retry a run locally that has failed on AWS Batch, by passing the --run-id flag:

ies run local --tasks=inspect_evals/pubmedqa --model=anthropic/claude-3-7-sonnet-20250219 --run-id="${RUN_ID}"

Running evaluations on AWS Batch

Use the ies run aws command to run an evaluation job using the Docker-in-Docker setup:

ies run aws --tasks=inspect_evals/pubmedqa --model=openai/gpt-4o-mini

The script will output:

1. The job ID for tracking the job status
2. A direct link to the AWS Batch console where you can monitor the job's status and details
3. A direct link to the CloudWatch logs where you can monitor the job's output in real-time

If you pass the --monitor flag, the script will also:

1. Monitor the job's progress in real-time
2. Output the job's logs in real-time
3. Output the job's results when the job is complete

It is possible to pass multiple models and/or multiple tasks to the job at a time, making use of inspect eval-set load balancing and retry functionality:

ies run aws --tasks=inspect_evals/pubmedqa,inspect_evals/humaneval --model=openai/gpt-4o-mini,openai/gpt-4o

Retrieving Results

Once the job has started, you will be see a Run ID in the output. This is a unique identifier for the run, of the form YYYY-MM-DD-hh-mm-ss-{random_hash}.

It is useful to save it to an environment variable, for example:

export RUN_ID="2025-02-26T05-43-33+00-00-f8ea"

The evaluation logs will be automatically uploaded to your S3 bucket in the following structure:

s3://${S3_BUCKET_NAME}/logs/working/${RUN_ID}/filename.eval

You can download the logs using the AWS CLI:

aws s3 cp s3://${S3_BUCKET_NAME}/logs/working/${RUN_ID}/filename.eval .

Summarizing Results

After running evaluations, you can summarize the results using:

ies summarize --run-id="${RUN_ID}"

This will create one .eval.dashboard.json file for each .eval log file in the same directory as the logs, for use by the Inspect Evals dashboard, and one zip file for each .eval log file.

Promoting Results

After summarizing the results, you can promote them to the staging area of the dashboard using:

ies promote --run-id="${RUN_ID}"

This will move the .eval.dashboard.json and zip files to the staging area of the dashboard, in the following structure:

s3://${S3_BUCKET_NAME}/logs/stage/{task_name}/{model_name}/{RUN_ID}/

Configuration

Environment Variables

This project uses environment variables for configuration. Copy the .env.template file to .env and set the appropriate values:

cp .env.template .env
# Edit .env with your preferred editor

Key environment variables:

• S3_BUCKET_NAME: The S3 bucket used for storing evaluation logs and results
• OPENAI_API_KEY: Your OpenAI API key for running evaluations

Other environment variables are listed and explained in .env.template.

Development Notes

• The Docker container uses a virtual environment to handle Python 3.12's externally managed environment feature
• AWS credentials are mounted from your local ~/.aws directory
• The Docker daemon is started internally using the entrypoint script from the base image
• The container runs in privileged mode to enable Docker-in-Docker functionality
• All S3 bucket references are centralized and sourced from the S3_BUCKET_NAME environment variable