Skip to content

README.md

Latest commit

 

History

History
137 lines (101 loc) · 3.88 KB

README.md

File metadata and controls

137 lines (101 loc) · 3.88 KB
Raw

Running Inspect Evals on AWS Batch

This directory contains the necessary files to run Inspect Evals on AWS Batch.

Prerequisites

  1. AWS CLI configured with appropriate credentials
  2. Docker installed locally

Setup Steps

Create the S3 bucket using CloudFormation

# Define the bucket name or create one
export S3_BUCKET_NAME="your-bucket-name"
# Or use a test bucket
# export S3_BUCKET_NAME="inspect-evals-test-$(aws sts get-caller-identity --query Account --output text)"

# Deploy the S3 bucket stack
aws cloudformation deploy \
  --template-file cloudformation/s3_bucket.cfn.yaml \
  --stack-name inspect-evals-s3 \
  --parameter-overrides \
    BucketName="$S3_BUCKET_NAME" \
    Environment=dev \
    EnableVersioning=true

Deploy the IAM role

aws cloudformation deploy \
  --template-file cloudformation/iam_role.cfn.yaml \
  --stack-name inspect-evals-iam \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides Environment=dev

Deploy the Secrets Manager configuration

aws cloudformation deploy \
  --template-file cloudformation/secrets.cfn.yaml \
  --stack-name inspect-evals-secrets \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides Environment=dev

Deploy AWS Batch resources

# Get default VPC subnets (or use your preferred VPC/subnets)
vpc_id=$(aws ec2 describe-vpcs --filters "Name=isDefault,Values=true" --query "Vpcs[0].VpcId" --output text)
subnet_ids=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=$vpc_id" --query "Subnets[*].SubnetId" --output text | tr ' ' ',')

# Deploy the AWS Batch stack
aws cloudformation deploy \
  --template-file cloudformation/batch.cfn.yaml \
  --stack-name inspect-evals-batch \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides \
    Environment=dev \
    VpcId=$vpc_id \
    SubnetIds=$subnet_ids

Set up the OpenAI API key and other API keys in Secrets Manager

The secrets_manager.py script provides commands for managing the required API keys. See the .env.template file and secrets_manager.py for more details:

# Source the environment
. ./.env

# Store or update the API key
python secrets_manager.py --env-var OPENAI_API_KEY --value "$OPENAI_API_KEY" --environment dev

# Verify the API key is stored correctly
python secrets_manager.py --env-var OPENAI_API_KEY --get --environment dev

Do this for all the keys in .env that you need to use in the batch jobs.

Build and push the Docker image

Use the provided script to build and push the Docker-in-Docker image to ECR:

./build_and_push_dind.sh

This script will:

  1. Build the Docker image using the dind.Dockerfile in the parent directory
  2. Create an ECR repository if it doesn't exist
  3. Log in to ECR
  4. Tag and push the image to ECR

This will create a Docker image with:

  • Ubuntu 24.04 (Noble Numbat) base image
  • Python 3.12
  • All required dependencies
  • Docker-in-Docker capabilities using the internal Docker daemon

Note that Docker-in-Docker is not compatible with AWS Fargate, so the job definition exclusively supports EC2 compute environments.

Register the Batch job definition

# Activate the virtual environment
source ../.venv/bin/activate

# Register the job definition
./register_job_definition.py

The Python script will:

  1. Automatically retrieve AWS account info and role ARNs from CloudFormation
  2. Create a job definition with the correct configuration for EC2 compute environments
  3. Register it with AWS Batch
  4. Output the job definition ARN and revision number

The job definition includes:

  • Docker-in-Docker support for container isolation
  • IAM roles configured via CloudFormation
  • CloudWatch logging configuration
  • Access to all necessary API keys from AWS Secrets Manager
  • Automatic upload of evaluation logs to S3
  • Environment-specific configuration

At this point you should have everything set up to run evaluations.