KnowledgeGain: Evaluating and Optimizing Science News Generation for Reader Learning.

KnowledgeGain (KGain) evaluates science communication by measuring how much a reader's question-answering accuracy improves after reading an article:

KGain = post-reading accuracy - pre-reading accuracy

The repository also includes LLMSim, a prompt-based simulated reader calibrated to human answer distributions. LLMSim uses behavioral personas, memory bottleneck, an explicit “I do not know” option, and verbalized answer sampling to estimate KGain at scale. The resulting score can be used to rank generated articles, filter training data, and evaluate science-news generators.

Highlights

• Reader-centered evaluation metric based on pre/post comprehension.
• Question generation for factual recall and basic inference.
• Human-calibrated LLM reader simulation.
• Data and scripts for KGain-filtered SFT.
• A 2,747-instance abstract–QA–news corpus.

The results show that professional science news produced knowledge gains comparable to paper abstracts and larger gains than tweets/threads. Articles selected with the simulated KGain signal also improved human normalized learning gain over a strong agentic baseline.

Repository structure

data/                 Human-study, question-generation, and derived datasets
evaluation/           Human and automated evaluation data/scripts
plots/                Plotting outputs and utilities
src/
  scripts/            Main LLMSim, filtering, training, and evaluation pipeline
  data/               Prepared JSONL datasets for training and evaluation
  qgen.py              Comprehension-question generation
  run_kgain_llmsim.py  Human-study-style LLMSim runner

Most scripts use paths relative to src/, so run the examples below from that directory.

Setup

git clone https://github.com/lamps-lab/knowledge-gain.git
cd knowledge-gain

python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install openai torch transformers datasets peft accelerate trl \
  scikit-learn pandas numpy scipy matplotlib

Most of the workflows require:

• OPENAI_API_KEY for question generation and OpenAI-backed LLMSIM runs.
• Access to Hugging Face model weights.
• A CUDA-capable GPU for practical fine-tuning and local-model evaluation.

Quick start

1. Run a small LLMSIM test

cd src
export OPENAI_API_KEY="..."

RESUME_RUN_DIR="" \
N_STUDENTS=2 \
LIMIT_ITEMS=1 \
LIMIT_QUESTIONS=2 \
python run_kgain_llmsim.py \
  --llmsim scripts/llmsim.py \
  --questions ../evaluation/generated_questions.json \
  --eval ../evaluation/eval_dataset.json \
  --personas scripts/persona_cards.json

Outputs are written to a timestamped directory under src/runs/, including predictions, memory traces, a run configuration, and summary distributions.

2. Score candidate articles with LLMSIM

cd src
PYTHONPATH=scripts python scripts/score_best_articles_llmsim.py \
  --input data/judged_teachers_news.jsonl \
  --out data/judged_teachers_news_llmsim.jsonl \
  --n_simulated_readers 5 \
  --pre_cache cache/pre_cache_train.jsonl

3. Train the KGain-filtered generator

A prepared filtered dataset is included in src/data/:

cd src
python scripts/train_sft_news.py \
  --data data/sft_kg_filtered_top75.jsonl \
  --model Qwen/Qwen3-4B-Instruct-2507 \
  --out outputs/sft_kg_filtered_top75

4. Evaluate a trained adapter

cd src
PYTHONPATH=scripts python scripts/eval_model_llmsim.py \
  --data data/test_llmsim_s2.jsonl \
  --base_model Qwen/Qwen3-4B-Instruct-2507 \
  --adapter outputs/sft_kg_filtered_top75 \
  --out results/eval_sft_kg_filtered_top75.jsonl \
  --n_simulated_readers 5 \
  --max_examples 300 \
  --use_qas \
  --pre_cache cache/pre_cache_eval.jsonl

See src/scripts/commands.sh for the fuller data-splitting, filtering, SFT, and evaluation workflow.

Question generation

src/qgen.py generates six multiple-choice questions per abstract:

1. Two extractive true/false questions.
2. Two extractive multiple-choice questions.
3. Two inferential multiple-choice questions.

Every question includes an explicit “I do not know” option so the evaluation can distinguish uncertainty from an incorrect belief.

Notes

This is research code and several scripts assume the repository's current directory layout, model names, and local output folders. Check paths and resource requirements before launching large experiments. LLMSim is intended as a scalable development-time proxy, not a replacement for human pre/post evaluation.