Circuit Claims Depend on What Is Extracted and How It Is Compared

Official code for the paper Circuit Claims Depend on What Is Extracted and How It Is Compared by Yang Sheng and Jie Fu.

This repository contains the core scripts used for the controlled circuit-reproducibility study.

Structure

• data/: synthetic logic-task generation and tokenizer training.
• training/: supervised pretraining and GRPO fine-tuning scripts.
• evaluation/: task-level evaluation scripts.
• circuits/: circuit extraction, factorized graph extraction, and fixed-loss pruning sweeps.
• runtime/inference/: minimal model and hook utilities required by the scripts above.

Large artifacts such as checkpoints, generated data, logs, and result tables are intentionally not included.

The runtime/inference/ utilities are adapted from OpenAI's Apache-2.0 circuit_sparsity codebase, with local adaptations for this study; see LICENSE and NOTICE.

Environment

pip install -r requirements.txt

Typical Workflow

python data/generate_tokenizer_corpus.py
python data/train_tokenizer.py
python training/train_sft.py --config dense
python training/train_grpo.py --model-type custom --model-path <checkpoint> --config-path <config> --output-dir artifacts/results/grpo_run
python evaluation/evaluate.py --results-dir artifacts/results/checkpoints
python circuits/circuit_extraction.py --checkpoint <checkpoint>/best_model.pt --config <checkpoint>/config.json --output-dir artifacts/results/circuits --task-group composition

The factorized query/key and fixed-loss pruning scripts in circuits/ use the same checkpoint/config/output-dir convention.

License

Apache License 2.0 (see LICENSE).

Citation

@misc{sheng2026circuitclaimsdependextracted,
      title={Circuit Claims Depend on What Is Extracted and How It Is Compared},
      author={Yang Sheng and Jie Fu},
      year={2026},
      eprint={2607.18921},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2607.18921},
}