Agent-ProSAT: Auto-Generating Reasoning CoT via Self-Correcting Programmatic Data Synthesis & Validation Loops

[habanwer]

Hello NVIDIA Community & Developers:

I’ve recently open-sourced Agent-Prosat, heavily inspired by Karpathy’s autoresearch, this pipeline tackles the bottleneck of manually curating quality Chain-of-Thought (CoT) datasets or the costs of model-driven forms by redirecting AI from CoT generation (liable to hallucination or model architecture or capabilities), into coding of solver thar can be improved by iteration or through human-in-the-loop, to programmatic review of the underlying code, and better control token utilisation in SFT/RL pipelines.

Agent-ProSAT utilizes programmatic, self-correcting loops with programmatic harness that provide guardrails while the agent codes the solvers, then assess it, against gound truth and against held-out sample (with blinded/unblinded programmatic script that the agent is prohibited from tampering) to eventually edit prosat.py that solves the various puzzle types to incorporate into generation of the CoT reasoning traces.

I’ve been benchmarking this on the "Alice in Wonderland" (AIW) logic puzzles from NVIDIA’s Nemotron Reasoning Challenge to stress-test how the loops handle edge-case spatial and logical constraints. Output dataset has been released on Hugging Face:

The Features are:

• Programmatic Data Synthesis: Autonomous generation of multi-step CoT traces.
• Validation Loops: Built-in, self-correcting reflection to validate logical soundness.
• Agentic Curation: Karpathy's autoresearch-style iterative workflows directly into data formatting.

I would love to collaborate with others working on synthetic data and agentic workflows. Specifically:

• CoT Schemas: What structural formats are you finding most effective when prepping synthetic reasoning traces for downstream fine-tuning?
• Validation Overhead: How are you balancing token-efficiency and accuracy during automated self-reflection steps?

More Info, codebase, case-study on Nemotron Reasoning/AIW and the ProSAT CoT dataset are available under:

• GitHub Repository: https://github.com/The-Adimension/Agent-ProSAT/
• Kaggle Notebook: https://www.kaggle.com/code/habanwer/nemotron-atlas (SFT on AIW ProSAT CoT)
• Hugging Face Dataset & code: https://huggingface.co/datasets/The-Adimension/prosat-nemotron-alice-in-wonderland-traces
• Framework behind the work: https://doi.org/10.1093/ehjimp/qyaf038

If you are experimenting with programmatic reasoning data or iterative agentic loops, I’d love to hear from you, and most importantly to validate and review the work I share with the community!

Best,
Shehab Anwer, MD
Cardiologist, Data Engineering & AI enthusiast.