Parallelized Eva Evaluation

[achi2023]

Purpose

This PR introduces a Slurm script that parallelizes the evaluation for OpenMidnight on the Eva evaluation suite. The default parameters are set for ViT Base.

Usage

Run the packed eval script as follows:
sbatch OpenMidnight/slurm_run_evals_packed.sbatch
/path/to/sweep_or_eval
/admin/home/achihoub/openmidnight_eval_results

Optional parameters include:

1. CKPT_PARALLELISM: number of ckpts to run in parallel
2. TASK_PARALLELISM: number of tasks to run in parallel
3. MAX_CHECKPOINTS: max number of ckpts for this run
4. TASK_DATASETS: specific datasets to run from the Eva suite
5. BACKBONE_MODEL, MODEL_NAME, IN_FEATURES: Dino model type, model size, number of input features
6. N_DATA_WORKERS, PREDICT_BATCH_SIZE

[CLAassistant]

All committers have signed the CLA.

[PaulScotti]

Thanks Anis, but can you clean this up so I can push to main?

Why do you have both a slurm_run_evals.sbatch and slurm_run_evals_packed.sbatch? I dont understand the difference of how these are meant to be used

Also generally looks like lots of AI-generated code that is not necessary e.g., there are lines of code like "This is the failure mode behind the CUDA OOMs in job 33308" which has no meaning to other people. Also lots of new lines of code that I'm not sure are necessary and which I don't know how to interpret

stuff like

if [[ ! -f "${REPO_ROOT}/.venv/bin/activate" ]]; then
    echo "Virtualenv not found: ${REPO_ROOT}/.venv/bin/activate"
    exit 1
  fi

are not necessary; a PR should strictly implement the intended feature and leave everything unchanged that can be left unchanged

[achi2023]

Hi Paul,

To answer your question, slurm_run_evals.sbatch was added by accident. The packed script is what will parallelize the eva evals. I will take down slurm_run_evals.sbatch so that there is no confusion. I will also clean up the packed script to get rid of the unnecessary code asap.