StableAdamW with Muon orthogonalisation for matrix parameters.
-
Updated
Jun 9, 2026 - Python
#
language-model-training
Here are 3 public repositories matching this topic...
OpenAI parameter-golf challenge: train the smallest LM that fits in 16MB. My SP8192 frontier submission (openai/parameter-golf#1887): 11L x 512d w/ targeted middle recurrence, MuonEq-R optimizer, int6 GPTQ+SDClip, Brotli-11 compression, legal score-first TTT. 8xH100 DDP.
openai quantization ddp model-compression distributed-training llm gptq h100 language-model-training muon-optimizer
-
Updated
Apr 28, 2026 - Python
Comprehensive resources and scripts for training and fine-tuning Large Language Models (LLMs) from scratch using Hugging Face Transformers, litGPT, and custom GPT implementations with PyTorch and PyTorch Lightning.
machine-learning pytorch fine-tuning huggingface pytorch-lightning large-language-models llm finetuning-llms litgpt language-model-training
-
Updated
Jun 22, 2025 - Jupyter Notebook
Improve this page
Add a description, image, and links to the language-model-training topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the language-model-training topic, visit your repo's landing page and select "manage topics."