aiq_mit_ta2

This project provides HELM run_spec and scenario components for the MIT AIQ TA2 team provided datasets.

Setup

As a dependency

In your pyproject.toml dependencies, you can add:

"aiq_mit_ta2 @ git+https://github.com/AIQ-Kitware/aiq_mit_ta2"

Or to install directly with pip:

pip install git+https://github.com/AIQ-Kitware/aiq_mit_ta2.git

Python environment

(If running / testing directly)

We use the uv tool for environment management. Installation instructions: https://docs.astral.sh/uv/#installation

uv venv --python 3.11 --seed .venv-311-example
source .venv-311-example/bin/activate
uv pip install -e .

Example invocations

Once installed, you can test the different datasets with the following example commands..

Arithmetic

helm-run -r 'arithmetic:format_type=numeric,model=openai/gpt2' -m 10 --suite test

Permutation

helm-run -r 'permutation_composition:num_items=3,story_length=5,model=openai/gpt2' -m 10 --suite test

EWoK core

Important: Access to the EWoK data requires accepting the terms of use via the HuggingFace page: https://huggingface.co/datasets/ewok-core/ewok-core-1.0 Notably:

• ⚠️ PLEASE DO NOT distribute any of the EWoK materials or derivatives publicly in plain-text! Any materials should appear in password-protected ZIP files or behind gated authentication mechanisms such as Huggingface datasets.
• ⚠️ Any use of EWoK materials in pretraining/training of models requires EXPLICIT ACKNOWLEDGMENT!

helm-run -r 'ewok_core:split=test,domain=physical-relations,model=openai/gpt2' -m 10 --suite test