snlp

Forecasting eval pipeline. Takes a CSV of forecasting questions, runs them through Bedrock-hosted LLMs with self-consistency sampling, scores the predictions (Brier, ECE, optimism bias, etc.), and produces results sliceable by category, entity, language, and model.

Built for comparing forecasting bias across models from different labs and training cultures.

Setup

pip install -r requirements.txt

Credentials

This project uses a Bedrock API key. To provide it, copy the example env file and add your key:

cp .env.example .env
# then edit .env and set your Bedrock key:
# AWS_BEDROCK_KEY=ABSK...

The project loads environment variables from the process environment. You can also pass an API key on the command line with --api-key.

Checking your access

To verify credentials work, run:

python list_models.py claude

If you see a table of models, you're good. If you get an auth error, double-check your credentials.

Finding models

Not sure which model ID to use? The model search utility queries Bedrock and returns ready-to-use IDs:

python list_models.py                    # all text models
python list_models.py claude             # search by name
python list_models.py --provider meta    # filter by provider
python list_models.py llama -v           # verbose with details
python list_models.py --json             # JSON output for scripting

Copy a model ID from the output and pass it as --model.

Many newer models (Claude 3.5+, Llama 3.2+, etc.) require cross-region inference profile IDs -- the us. or global. prefixed versions. list_models.py handles this automatically. If you get an error like "Invocation of model ID with on-demand throughput isn't supported", you need the cross-region ID; run list_models.py to find it.

Quick start

python run_eval.py -i questions.csv -m bedrock/us.anthropic.claude-3-5-haiku-20241022-v1:0

The pipeline will:

1. Load and validate your CSV
2. Estimate cost and ask for confirmation (skip with -y)
3. Run N samples per question at the configured temperature
4. Extract probability/value from each response
5. Aggregate (mean, median, std) across samples
6. Score against ground truth if provided
7. Write results, scores, and a calibration plot

More examples

# Full options
python run_eval.py \
  -i questions.csv \
  -o results/ \
  -m bedrock/us.anthropic.claude-3-5-haiku-20241022-v1:0 \
  --samples 5 \
  --temperature 0.7 \
  --concurrency 10 \
  --prompt-style forecastbench \
  --resume \
  -y

# Re-score existing results (no inference)
python run_eval.py --score-only -o results/

# Compare results across models
python run_eval.py --compare results/haiku/results.csv results/sonnet/results.csv

CLI flags

Flag	Short	Description
--input	-i	Path to input CSV
--output	-o	Output directory (default: results/)
--model	-m	Model ID (get these from list_models.py)
--samples	-n	Self-consistency samples per question (default: 5)
--temperature	-t	Sampling temperature (default: 0.7)
--concurrency		Max parallel API calls (default: 10)
--prompt-style		default or forecastbench
--prompt-template		Path to custom prompt file (overrides --prompt-style)
--api-key		Bedrock API key (alternative to IAM credentials)
--resume		Resume from checkpoint if a previous run was interrupted
--yes	-y	Skip cost confirmation
--score-only		Re-score existing results without running inference
--compare		Compare multiple result CSV files side by side
--config		Path to custom config YAML

Input CSV

Column	Required	Notes
question_id	yes	unique identifier
question_text	yes	the forecasting question
question_type	yes	binary or numerical
resolution_date	no	YYYY-MM-DD
ground_truth	no	0/1 for binary, number for numerical (needed for scoring)
category	no	for slicing, e.g. "economics", "geopolitics"
entity	no	for slicing, e.g. "US", "China"
language	no	for slicing, e.g. "en", "zh"

Any extra columns are preserved as metadata. See examples/ for sample files.

Output

Written to the output directory:

• results.csv -- per-question predictions with sample values, reasoning traces, token counts, and cost
• scores.csv -- aggregate metrics grouped by overall, category, entity, language, and question type
• calibration.png -- reliability diagram for binary predictions

Output CSV columns

question_id, question_text, question_type, model, ground_truth, aggregated_value, median_value, std_value, num_samples, sample_values, reasoning_traces, category, entity, language, total_tokens, cost_usd, timestamp

Scores CSV columns

group_key, model, n, question_type, brier_score, brier_reliability, brier_resolution, brier_uncertainty, log_score, ece, optimism_bias, mae, rmse, mape

Metrics

Binary questions:

• Brier score + Murphy decomposition (reliability, resolution, uncertainty)
• Log score (probabilities clipped to [0.01, 0.99] for stability)
• ECE (expected calibration error, uniform binning)
• Optimism bias (mean predicted - mean actual; positive = overconfident)

Numerical questions:

• MAE, RMSE, MAPE

Metrics are computed only over questions that have ground_truth values.

Self-consistency sampling

Each question is run N times (configurable with --samples) at the given temperature. The pipeline extracts a probability or numerical estimate from each response, then reports the mean, median, and standard deviation across samples. This reduces variance from any single model call.

Prompt templates

Two built-in styles in prompts/:

• default -- direct, short
• forecastbench -- superforecaster-style, references Tetlock

Use --prompt-style forecastbench to switch, or --prompt-template path/to/your.txt for a custom template. Templates support these placeholders: {question_text}, {resolution_info}, {resolution_date}, {category}, {entity}, {language}.

To add a new built-in style, drop files named binary_yourstyle.txt and numerical_yourstyle.txt into prompts/.

Resume / crash recovery

Pass --resume to pick up where a previous run left off. Progress is checkpointed per-question as JSONL. If inference is interrupted (network error, Ctrl-C, etc.), re-run with --resume and it'll skip already-completed questions. The checkpoint is automatically cleaned up after a successful run.

Cost estimation

Before starting inference, the pipeline estimates total cost based on question count, sample count, and model pricing. If the estimate exceeds $10 (configurable in config.yaml), it prints a warning and asks for confirmation. Pass -y to skip the prompt.

Experiments

Experiments live in experiments/. Each one is a self-contained directory with its own questions, run script, and README. See experiments/README.md for the full guide on adding new experiments.

Current experiments:

#	Name	Description
001	Baseline calibration	How well-calibrated is Haiku on binary forecasting?
002	Cross-model comparison	Do models from different labs show different biases?
003	Prompt style ablation	Does prompt style affect accuracy and calibration?

To run an experiment:

cd experiments/001_baseline_calibration
bash run.sh

To add a new experiment, see the experiments guide.

Config

Defaults live in config.yaml:

samples: 5
temperature: 0.7
concurrency: 10
prompt_style: default
cost_warning_threshold: 10.0
model: bedrock/us.anthropic.claude-3-5-haiku-20241022-v1:0
output_dir: results
probability_clip_min: 0.01
probability_clip_max: 0.99
ece_num_bins: 10

CLI flags override config file values.

Project layout

run_eval.py              # entry point
list_models.py           # model search utility
config.yaml              # default config
requirements.txt
snlp/
  cli.py                 # argument parsing, pipeline orchestration
  config.py              # YAML config loading
  models.py              # dataclasses (Question, SampleResult, QuestionResult)
  io.py                  # CSV read/write + validation
  inference.py           # LiteLLM async calls + self-consistency
  extraction.py          # regex parsing of model output
  scoring.py             # Brier, log score, ECE, MAE, RMSE, MAPE, etc.
  analysis.py            # group-wise slicing + comparison tables
  plotting.py            # calibration curves + bar charts
  prompts.py             # prompt template loading
  resume.py              # JSONL checkpoint for crash recovery
prompts/                 # prompt template .txt files
examples/                # sample input CSVs
experiments/             # experiment directories (see experiments/README.md)