GenomeHarness

GenomeHarness is an agent-facing fine-tuning search harness for genomic foundation models. It is designed for Codex/OpenCode-style interactive agents: the agent makes scientific proposal decisions, while the Python harness executes trials, enforces protocol safety, records state, schedules GPUs, and renders reports.

Repository Layout

GenomeHarness/
  AGENTS.md                 # supervisor-agent entry protocol
  skills/                   # repo-native agent behavior protocols
  genomeft/                 # Python harness package
  configs/templates/        # path-safe templates users should edit
  data/benchmarks/          # empty public data placeholder
  scripts/                  # benchmark preparation helpers
  plans/                    # active system design plan
  pyproject.toml

Runtime outputs such as runs/, checkpoints, reports, local datasets, model weights, and ablation results are intentionally excluded from this repository.

Installation

Create a fresh Python environment:

conda create -n genomeharness python=3.10 -y
conda activate genomeharness

Install PyTorch for your CUDA version. For example:

# CUDA 12.1 example
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

# Or choose the command matching your machine from https://pytorch.org/get-started/locally/

Install GenomeHarness in editable mode:

cd /path/to/GenomeHarness
pip install -e .

Optional but recommended environment variables:

export GENOMEHARNESS_MODEL_ROOT=/path/to/models
export GENOMEHARNESS_BENCHMARK_ROOT=/path/to/benchmarks
export GENOMEHARNESS_RUNS_DIR=/path/to/runs
export GENOMEHARNESS_PYTHON=$(which python)

Check the CLI:

genharness --help

Prepare Models

Place model checkpoints under your model root, for example:

/path/to/models/
  DNABERT-2-117M/
    config.json
    pytorch_model.bin or model.safetensors
    tokenizer files...

You can also pass a direct model path to every command:

--model /path/to/models/DNABERT-2-117M

Prepare Benchmarks

GenomeHarness expects a benchmark root containing processed CSV splits and a manifest. The public repository only includes templates; it does not include downloaded datasets.

To prepare supported public datasets:

python scripts/prepare_benchmarks.py \
  --root /path/to/benchmarks \
  --only nt

For all supported benchmark families:

python scripts/prepare_benchmarks.py \
  --root /path/to/benchmarks \
  --only all

Expected layout:

/path/to/benchmarks/
  manifests/
    all_benchmarks.summary.csv
  processed/
    nt/
      tasks/
        H2AFZ/
          train.csv
          validation.csv
          test.csv

Each split CSV must contain a sequence column (sequence or seq) and a label column. See configs/templates/benchmark_summary.template.csv if you want to provide a custom benchmark manually.

Agent-First Usage

The intended user interface is natural language inside a Codex/OpenCode session started from this repository.

Example user request:

Use DNABERT-2-117M to improve performance on nt/H2AFZ with GenomeHarness.

The agent should:

1. read AGENTS.md
2. read the relevant skill under skills/
3. run a read-only genharness plan --dry-run --json
4. show a Campaign Plan Preview
5. wait for user confirmation
6. start/attach the durable runner
7. resolve PROPOSE, FREEZE, and REPAIR requests through the skills
8. continue until DONE, FAILED, or required user authorization

For an NT sequential suite, ask for nt-suite. Do not use bare nt; in this project nt can also refer to model naming and is intentionally ambiguous.

Manual CLI Quick Start

Manual commands are useful for debugging or for environments without an agent shell. The official workflow still requires that a new campaign be previewed before execution.

Plan A Single Task

genharness plan --dry-run --json \
  --model /path/to/models/DNABERT-2-117M \
  --benchmark nt/H2AFZ \
  --benchmark-root /path/to/benchmarks \
  --scope campaign \
  --search-active-wall-time-cap 1h \
  --max-search-gpus 4 \
  --target-active-search-slots 4

Start A Single Campaign

Create a campaign config from configs/templates/single_task_campaign.yaml, replace /path/to/..., then:

genharness preflight --campaign /path/to/runs/campaigns/nt_H2AFZ/config.yaml
genharness start --campaign /path/to/runs/campaigns/nt_H2AFZ/config.yaml
genharness advance --campaign /path/to/runs/campaigns/nt_H2AFZ --wait

When advance --wait returns a PROPOSE request, read the packet in the campaign packets/ directory and submit a proposal JSON:

genharness submit-proposals \
  --campaign /path/to/runs/campaigns/nt_H2AFZ \
  --proposal /path/to/runs/campaigns/nt_H2AFZ/proposals/proposal_0001.json

Then attach again:

genharness advance --campaign /path/to/runs/campaigns/nt_H2AFZ --wait

Run A Sequential Suite

Create a suite over a benchmark family:

genharness suite-init \
  --suite /path/to/runs/suites/DNABERT-2-117M_nt-suite_v15 \
  --model /path/to/models/DNABERT-2-117M \
  --benchmark-family nt \
  --benchmark-root /path/to/benchmarks \
  --execute \
  --search-active-wall-time-cap 1h \
  --max-search-gpus 4 \
  --target-active-search-slots 4

Start or attach the next incomplete task:

genharness suite-start-next \
  --suite /path/to/runs/suites/DNABERT-2-117M_nt-suite_v15

Check suite status:

genharness suite-status \
  --suite /path/to/runs/suites/DNABERT-2-117M_nt-suite_v15

Protocol Summary

Search:

• search seed 42
• full train data and validation reward
• raw validation MCC is the reward
• no active Level1/Level2 promotion path
• no patience/min_delta early stop in v15
• wall-time launch deadline, then drain already-started trials

Confirmation/final:

• confirmation seeds [42, 43, 44]
• search and confirmation never read test
• freeze happens before any final test metric is observed
• final evaluates the frozen recipe once on the held-out test split
• if any harness final seed fails, the campaign cannot be marked DONE

GPU scheduling:

• default max_search_gpus=4
• default search_max_parallel_trials=4
• one trial per selected GPU
• dynamic GPU choice per launch
• do not automatically kill other users' processes

State:

• state.sqlite is the only mutable truth source inside a campaign
• packets, proposal files, reports, and briefs are derived or auditable artifacts
• runner/trials survive Codex/OpenCode session disconnects

Reports And Cleanup

Generate or refresh a report:

genharness report --campaign /path/to/runs/campaigns/nt_H2AFZ

Reports include:

• baseline/final comparison
• search trajectory
• UCT/MCTS decision evidence
• evolutionary recipe tree
• frozen recipe and confirmation/final metrics
• runtime/resource summary
• failure/repair summary

By default, after report generation the harness may delete checkpoint directories under the campaign trials/ tree to save disk space. It keeps state.sqlite, metrics, packets, proposals, and reports. Set keep_checkpoints: true or disable cleanup_checkpoints_after_report in your campaign config if you want to retain checkpoints.

Resume And Recovery

Read status:

genharness status --campaign /path/to/runs/campaigns/nt_H2AFZ --brief

If a session disconnects, ask it to continue in the current session. The agent should read the brief/status and attach.