Switch to Qwen3-32B with LoRA

[kfallah]

Summary

• Switch from Qwen3-14B full fine-tuning to Qwen3-32B with LoRA (rank=32, alpha=64, all-linear)
• LR increased to 5e-5 (standard for LoRA)
• Added memory optimizations: gradient checkpointing, use_shm, param_offload, layered_summon, load_format=safetensors
• Kept gpu_memory_utilization=0.85, tensor_parallel=2, sequence_parallel=2, batch_size=16

Key config changes

Setting	Before	After
Model	Qwen/Qwen3-14B	Qwen/Qwen3-32B
LoRA rank	N/A (full FT)	32
LoRA alpha	N/A	64
LR	1e-5	5e-5
load_format	default	safetensors
layered_summon	off	True
gradient_checkpointing	off	True

Test plan

• Verify Qwen/Qwen3-32B is downloadable on remote
• Verify training launches without OOM on 2x H100
• Monitor first few steps for healthy entropy/pg_loss

🤖 Generated with Claude Code