Forecasting Downstream Performance of LLMs With Proxy Metrics

Proxy metrics for ranking large language models without running full benchmark evaluations.

Given a small set of expert reasoning trajectories, this code computes cheap next-token prediction metrics on candidate models that correlate well with downstream accuracy on the underlying benchmarks — letting you rank candidates with one short forward pass per problem.

Install

pip install -e .

Python 3.10+ required. vLLM expects a CUDA-capable GPU.

Pipeline

The three stages below feed into each other. You can stop at any stage.

1. Score candidate models on benchmarks

python scripts/score_models.py \
    --base-model meta-llama/Meta-Llama-3-8B-Instruct \
    --data aime --average-seeds

python scripts/export_scores.py

Writes per-problem TSVs to ./outputs/<run>/ and per-benchmark summary CSVs/plots to ./scores/.

2. Generate expert trajectories

python scripts/get_expert_trajectories.py \
    --model Qwen/Qwen3-Next-80B-A3B-Instruct \
    --model-short-name qwen3next \
    --backend vllm --task aime --grade

Writes one directory per (task, expert) containing trajectories.parquet + a backward-compatible JSONL to ./expert_outputs/. Use --backend together for API-based generation.

3. Compute proxy metrics on a candidate model

python scripts/score_proxy_metric.py \\
    --expert-dir ./expert_outputs \\
    --base-model Qwen/Qwen3-1.7B \\
    --data aime --mode suffix

Writes per-problem and aggregate metrics (cross-entropy, top-k accuracy, entropy, rank, margin, etc., across multiple token weighting schemes) to ./proxy_results/ntp_results/<run>/.

4. Fit a proxy ranker

python scripts/fit_sampled_linear_proxy.py --tasks aime gpqa hmmt supergpqa mmlupro
python scripts/fit_sampled_sparse_proxy.py --tasks aime gpqa hmmt supergpqa mmlupro

Cross-validated fits of linear RankSVM and sparse subset selection over the proxy metric features.

[Coming Soon] Code for datadecide and extrapolation experiments.

Citation

If you find our paper or code useful for your work, please consider citing:

@misc{patel2026forecastingdownstreamperformancellms,
      title={Forecasting Downstream Performance of LLMs With Proxy Metrics}, 
      author={Arkil Patel and Siva Reddy and Marius Mosbach and Dzmitry Bahdanau},
      year={2026},
      eprint={2605.18607},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.18607}, 
}