Skip to content

workflow.md

Latest commit

 

History

History
100 lines (62 loc) · 2.58 KB

workflow.md

File metadata and controls

100 lines (62 loc) · 2.58 KB
Raw

Workflow for Running AWS Batch Evals

This document outlines the step-by-step process for running evaluation jobs on AWS Batch, handling failures, and promoting successful runs to stage.

1. Identify Eval and Models

  • Choose the evaluation you want to run and the set of models to evaluate.

2. Construct AWS Batch Command

  • Prepare the bash command to launch the AWS Batch job.
cd ~/your_development_path/inspect_evals_scoring
. .venv/bin/activate

export EVAL_TASK=inspect_evals/gaia_level3

export MODELS=\
openai/o3-mini,\
openai/o1-mini,\
openrouter/deepseek/deepseek-r1,\
openrouter/meta-llama/llama-3.3-70b-instruct,\
mistral/mistral-small-latest

3. Start and Monitor the Batch Job

  • Execute the command to start the AWS Batch job.
ies run aws --tasks="${EVAL_TASK}" --model="${MODELS}" --log-level=info --limit=20000 --summarize=true --request-memory 32
  • Monitor the job's status through the AWS console or CLI.

4. Wait for Job Completion

  • Wait for the job to either succeed or fail.

5. Capture the Run ID

  • Once the job finishes, export the run ID from the output:

    # Completed all tasks in 's3://inspect-evals-dashboard/logs/working/2025-04-11-04-22-40-ffef8b5c/' successfully

export RUN_ID=2025-04-11-04-22-40-ffef8b5c

6. Handle Failures

  • If the job failed, check the logs:

    ies status check-logs --run-id="${RUN_ID}"

7. Investigate and Adjust Models

  • Review the errors in the logs.
  • If errors are due to unsupported features (e.g., "model x can't use tools" or "model x can't accept images"), delete those logs from S3 and update the $MODELS environment variable to exclude problematic models.

8. Resume the Run

  • Restart the run with the previous command, appending the --resume-run-id flag:

    ies run aws --tasks="${EVAL_TASK}" --model="${MODELS}" --log-level=info --limit=20000 --summarize=true --request-memory 32 --resume-run-id="${RUN_ID}"

9. Repeat as Needed

  • Repeat steps 4–8 until the job completes successfully.

10. Summarize the Run

  • If you did not already pass --summarize in the initial command, summarize the run with:

    ies run summarize-run --run-id="${RUN_ID}"

11. Promote to Stage

  • Promote the successful run to stage:

    ies promote promote-run-to-stage --run-id="${RUN_ID}"

Tip: Keep a record of your commands and RUN_IDs for traceability and easier troubleshooting.

To track the completed runs, you can use:

aws s3 ls s3://${S3_BUCKET_NAME}/logs/stage/ --recursive | sort -k 1 | grep dashboard