Factual Reasoning

Paper: Learning to Reason for Factuality

📦 Companion Repositories

We release three artifacts accompanying the paper. You are currently in the Training Code repo.

Artifact	Where to find it
🧑‍💻 Training Code (this repo)	https://github.com/facebookresearch/factual_reasoning
📊 Training Data	https://huggingface.co/datasets/facebook/factual_reasoning
🧪 ScalableVeriScore (reward verifier server)	https://github.com/facebookresearch/ScalableVeriScore

⚠️ The GRPO stage in this repo requires a running VeriScore server from the ScalableVeriScore repo. See the VeriScore section below.

Overview

This repository contains the training code for the paper. It covers:

1. SFT with fairseq2 lm instruction_finetune.
2. DPO with fairseq2 lm preference_finetune.
3. GRPO with fairseq2 lm online_finetune and VeriScore rewards.

The scripts are designed to run from an existing interactive SLURM allocation. Start compute with salloc ..., then invoke the scripts directly. They do not submit sbatch jobs.

Environment

Install with install_env.sh.

Activate using

source env/bin/activate

Data Preparation

python scripts/prepare_data.py --max-examples 1000 --output-dir data

Expected JSONL schemas:

• SFT generic_instruction: src, tgt.
• DPO generic_preference: src, tgt_chosen, tgt_rejected.
• GRPO prompt_dataset: src, question; question is preserved through dataset.extras.keep_jsonl_keys for VeriScore.

All paths below are relative to the repo root. cd into the repo before running any of the scripts.

SFT

./run_sft.sh \
  --config configs/sft.yaml \
  --data data/sft/train.jsonl \
  --output-dir <OUTPUT_DIR>/sft_run

start_sft_training.sh delegates to run_sft.sh for compatibility.

Replace <OUTPUT_DIR> with a writable checkpoint directory.

DPO

Point DPO at the SFT checkpoint directory and model names expected by the fairseq2 asset registry:

./run_dpo.sh \
  --config configs/dpo.yaml \
  --data data/dpo/ \
  --checkpoint-dir <OUTPUT_DIR>/sft_run/checkpoints \
  --model-name checkpoint_step_94 \
  --reference-model-name checkpoint_step_94 \
  --output-dir <OUTPUT_DIR>/dpo_run

GRPO

GRPO uses Ray and vLLM actors. Run it inside an existing allocation with enough nodes for training plus Ray/vLLM. The launcher uses the last allocated nodes for Ray and the remaining nodes for fairseq2 training.

./start_training_wray_ray_multinode.sh \
  --config configs/grpo.yaml \
  --data data/grpo/ \
  --output-dir <OUTPUT_DIR>/grpo_run \
  --restart-ray \
  --num-ray-nodes 1

Optional overrides:

• --model-path PATH updates the policy and reference vLLM actor model paths.
• --tokenizer PATH updates the reward tokenizer and policy/reference actor tokenizers.
• Arguments after -- are forwarded to fairseq2 lm online_finetune.

run_grpo.sh is a thin alias for start_training_wray_ray_multinode.sh.

VeriScore

GRPO uses criterion.config.reward.name=veriscore_reward and posts to criterion.config.reward.config.veriscore_server_addr. For real training runs you must stand up a VeriScore server from the companion ScalableVeriScore repository and point veriscore_server_addr at it. Ensure that service is reachable from the training nodes before launching GRPO.

For wiring/smoke tests only, this repo ships scripts/mock_veriscore_server.py, which returns deterministic n_supported and n_all_claims values. It validates fairseq2 / Ray / reward plumbing only and must not be used for actual training — it does not produce meaningful reward signal. Use the real server from ScalableVeriScore for any production GRPO run.

Smoke Tests

Prepare tiny data and run SFT/DPO smoke stages:

./run_smoke_test.sh --skip-grpo

Run GRPO smoke separately inside an interactive allocation with Ray/vLLM resources:

./run_smoke_test.sh --skip-sft --skip-dpo

The GRPO smoke path starts the mock VeriScore server on http://127.0.0.1:42000/veriscore, uses configs/grpo_smoke.yaml, and calls start_training_wray_ray_multinode.sh so it exercises the same Ray orchestration as production. As noted above, the mock server is for plumbing validation only — substitute the real ScalableVeriScore server for any run whose rewards you care about.

Troubleshooting

• Missing datasets: activate the local env/ virtualenv and install datasets before running scripts/prepare_data.py.
• Missing model paths: pass --model-path and --tokenizer to the GRPO launcher or override fairseq2 config keys after --.
• Stale entrypoint errors: use fairseq2 lm online_finetune; the old online_finetune_game name is not used by these scripts.
• Ray connectivity: use --restart-ray, verify allocated node DNS resolution, and check ray status --address HOST:PORT.
• VeriScore unreachable: confirm criterion.config.reward.config.veriscore_server_addr is reachable from training nodes; use the mock server only for smoke wiring tests, and a real ScalableVeriScore server for production.

License

factual_reasoning is licensed under the Creative Commons Attribution-NonCommercial 4.0 International (CC-BY-NC 4.0) license, as found in the LICENSE file.

Contributing

See CONTRIBUTING.md for how to help, and CODE_OF_CONDUCT.md for community guidelines.

Citation

If you use this repo, please cite the following paper:

@inproceedings{chen2025learning,
      title={Learning to Reason for Factuality},
      author={Xilun Chen and Ilia Kulikov and Vincent-Pierre Berges and Barlas Oğuz and Rulin Shao and Gargi Ghosh and Jason Weston and Wen-tau Yih},
      year={2025},
      booktitle={Forty-third International Conference on Machine Learning (ICML 2026)},
      url={https://arxiv.org/abs/2508.05618},
}