feat: QLoRA fine-tuning (int4/int8) + DPO + PPO for small LLMs

[feiyuehchen]

Summary

• New ops: cross_entropy_per_token (fused Metal kernel, unreduced per-row output for DPO/PPO log-prob) and log_sigmoid (numerically stable, mx.compile-fused)
• metalgrad.finetune subpackage: LoRA/QLoRA adapters, model definitions (Qwen2.5, Llama 3.2, DeepSeek-Coder) using metalgrad ops, SFT/DPO/PPO trainers with OOM protection
• Apple Silicon OOM protection: pre-flight memory estimation, gradient checkpointing (default on), forced mx.eval(), dynamic batch truncation, rollout buffer cap, memory logging
• Version consolidation: __version__ and pyproject.toml both at 0.1.0, added CHANGELOG.md, updated docs/scope.md

Supported models (int4/int8 QLoRA)

Model	int4 memory	LoRA trainable %
Qwen2.5-0.5B	~0.5 GB	2.8%
Qwen2.5-1.5B	~1.3 GB	—
Llama-3.2-1B	~0.9 GB	—
DeepSeek-Coder-1.3B	~1.1 GB	—

Training pipelines

# SFT
python scripts/finetune_sft.py --model Qwen/Qwen2.5-0.5B --dataset tatsu-lab/alpaca --bits 4

# DPO
python scripts/finetune_dpo.py --model Qwen/Qwen2.5-0.5B --dataset Anthropic/hh-rlhf --bits 4

# PPO
python scripts/finetune_ppo.py --model Qwen/Qwen2.5-0.5B --bits 4 --reward heuristic

Test plan

• All 40+ gradcheck tests pass (including 6 new for cross_entropy_per_token and log_sigmoid)
• LoRA: forward shape correct, gradients only flow through adapters, freeze/unfreeze works
• Model: Qwen/Llama/DeepSeek int4 forward produces correct shape logits
• SFT: synthetic data training loop converges (loss 12.2 → 3.8), memory stable at ~780MB
• DPO: synthetic preference training converges (loss 0.28 → 0.001, margin 1.1 → 7.0)
• PPO: synthetic rollout + training loop runs without error, memory stable
• Real data validation pending (needs model weight download + pip install datasets)

🤖 Generated with Claude Code