---

AlignTune is the definitive modular ecosystem for the utility-driven post-training of Large Language Models. Built for both researchers and production engineers, it abstracts the complexity of disparate training backends into a single, high-performance interface focused on maximizing model reasoning, coding, and mathematical capabilities.

The Backend Factory routes each training call to whichever backend (TRL, Unsloth, or ES) has the best kernel support for your algorithm and hardware, so you write one config and don't have to track backend-specific quirks yourself.

Core Features

Multi-Backend Architecture: Route the same training call across TRL (the default, most-tested path), Unsloth (2x speed / 60% memory), or ES (gradient-free) backends.

Complete RLHF Coverage: 13+ SFT/RL algorithms including SFT, DPO, Online-DPO, PPO, GRPO, GSPO, DAPO, Dr. GRPO, GBMPO, Counterfactual GRPO, PACE, ORPO, and SPIN — plus RAFT and Distillation (Standard/SDFT).

Full CLI Surface: A typer-based CLI covering training, recipes, config validation, system diagnostics, cost/VRAM advising, model merging, an interactive training inspector, export/quantization verification, LoRA adapter management, multi-stage compositions, and Indic-language evaluation. See CLI Reference.

Production-Ready: every algorithm has been run end-to-end on real models and datasets (see Changelog for specifics), config values are validated before a run starts, and failures raise specific errors instead of failing silently.

🏛️ The 9 pillars of AlignTune Utility

AlignTune is built upon nine foundational architectural pillars that ensure production excellence and maximum model utility.

1. Unified Backend Factory: A single API to toggle natively between TRL (Reliability), Unsloth (2x Speed/60% Memory), and ES (Gradient-free).
2. Advanced Parameterization: State-of-the-art PEFT suite featuring MoA (Mixture of Adapters), Text2LoRA, and Doc2LoRA.
3. Long Context Infrastructure: Native support for RoPE scaling (Linear, NTK, YaRN), S2-Attention, Sliding Window Attention, and Sequence Packing.
4. Utility Advisor: aligntune advise: deterministic VRAM/time/cost/carbon estimation and algorithm recommendation based on your hardware profile, no GPU required.
5. Production Compositions: aligntune compose orchestrates multi-stage pipelines (e.g. SFT → MoA → ES → DPO → audit) with automated checkpoint threading between stages.
6. Factuality & Alignment Auditing: Specialized probe sets for hallucination detection and reasoning accuracy across domain-specific data in BFSI, Legal, and Healthcare, plus an AlignmentAuditor for tracking alignment metrics during training.
7. Model Merging Hub: Native aligntune merge integration for linear and task-arithmetic merging (via mergekit), plus a dependency-free LoRA adapter merge.
8. Verifiable-Reward RL (RLVR): Built-in verifiable reward functions (math, code execution, SQL, JSON-schema, regex) for GRPO-family training, plus SPIN self-play fine-tuning, reducing reliance on human preference labels for reasoning tasks.
9. Regional Utility (Indic-Plus): Script-aware tokenizer extension for Devanagari, Tamil, Bengali, Telugu, Kannada, and Malayalam, and an aligntune indic-eval CLI.

🏗️ Technical Manifest (Feature Mapping)

For developers and researchers, here is the direct mapping of features to the AlignTune core:

Feature	Implementation Path	Technical Detail
Backend Selection	aligntune.core.backend_factory.BackendFactory	Dynamic routing to TRL/Unsloth/ES kernels.
Adapters (PEFT)	aligntune.core.adapters/	MoA, Text2LoRA/Doc2LoRA.
RoPE / Packing	aligntune.core.long_context/	RoPE (YaRN/NTK), S2/Sliding-Window Attention, and packing kernels.
Resource Advisor	aligntune.core.advisor / aligntune.cli.advise	Deterministic VRAM, time, cost, and carbon profiling.
Compositions	aligntune.core.composition/	Multi-stage pipeline orchestration (aligntune compose).
Merging Hub	aligntune.core.merge/	linear/task-arithmetic via mergekit, plus LoRA merge.
RAFT	aligntune.core.backend_factory.create_raft_trainer	Retrieval-augmented SFT with golden/distractor document context.
Verifiable Rewards	aligntune.rewards.verifiable	Math, code-execution, SQL, JSON-schema, and regex reward functions for RLVR.
Indic Tokenization	aligntune.core.tokenization/	Script-aware BPE vocabulary expansion (continued-BPE, naive extension, pruning).
Distillation	aligntune.core.distill/	Standard and SDFT (self-distillation) methods.
Alignment Auditing	aligntune.eval.alignment_auditor	AlignmentAuditor / AlignmentAuditCallback for BFSI/Legal/Healthcare probes.
ES Rollout	aligntune.core.rollout/	HFRolloutBackend / VLLMRolloutBackend: pluggable generation engines for Evolution Strategies.
CLI	aligntune.cli.unified	12 command groups: train, recipes, validate, diagnose, advise, merge, aligner, export, verify-export, adapters, compose, indic-eval.

Quick Start

Supervised Fine-Tuning (SFT)

from aligntune.core.backend_factory import create_sft_trainer

# Create and train SFT model
trainer = create_sft_trainer(
    model_name="unsloth/llama-3-8b-bnb-4bit",
    dataset_name="tatsu-lab/alpaca",
    backend="unsloth",  # High-speed specialized kernels
    num_epochs=3,
    max_steps=-1,
    batch_size=4,
    learning_rate=5e-5
)

# Train the model
trainer.train()

# Evaluate
metrics = trainer.evaluate()
print(metrics)

Reinforcement Learning (DPO)

from aligntune.core.backend_factory import create_rl_trainer

# Create and train DPO model
trainer = create_rl_trainer(
    model_name="Qwen/Qwen3-0.6B",
    dataset_name="Anthropic/hh-rlhf",
    algorithm="dpo",   # Swap for "ppo", "grpo", "pace", "raft", etc.
    backend="trl",      # Route to TRL for scale
    num_epochs=1,
    batch_size=4,
    learning_rate=5e-5
)

# Train the model
trainer.train()

The CLI, equivalently

aligntune train --model unsloth/llama-3-8b-bnb-4bit --dataset tatsu-lab/alpaca --backend unsloth --type sft --epochs 3

# Before committing GPU time, sanity-check the plan:
aligntune advise estimate --model Qwen/Qwen2.5-7B --dataset-size 10000 --algorithm grpo
aligntune validate config my_config.yaml

Supported Algorithms

AlignTune supports 13+ state-of-the-art SFT/RL algorithms with intelligent backend routing.

Algorithm	TRL	Unsloth	Description
SFT	✅	✅	Standard Supervised Fine-Tuning
DPO	✅	✅	Direct Preference Optimization
Online-DPO	✅	✅	Iterative/Online variant of DPO
PPO	✅	✅	Proximal Policy Optimization
GRPO	✅	✅	Group-Relative Policy Optimization
GBMPO	✅	✅	Group-Based Mirror PO — unified config, 4 divergence types (L2/L2KL/ProbL2/ProbL2KL)
Counterfactual GRPO	✅	✅	Counterfactual variant of Group-Relative PO
PACE	✅	✅	High-efficiency Baseline-Optimized Learning
GSPO	✅	✅	Group Sequence Policy Optimization
DAPO	✅	✅	Decoupled-Clip Dynamic-Sampling PO
Dr. GRPO	✅	✅	GRPO Done Right (Unbiased variant)
SPIN	✅	✅	Self-Play Fine-Tuning
ORPO	✅	✅	Odds-Ratio Preference Optimization
RAFT	✅	✅	Retrieval Augmented Fine-Tuning (document-grounded SFT)
Distillation	✅	✅	Knowledge Distillation (Standard, SDFT)

See the Algorithm Zoo for the full comparison table and selection guide.

Installation

pip install aligntune

Or from source:

git clone https://github.com/Lexsi-Labs/aligntune.git
cd aligntune
pip install -e .          # or:  uv pip install -e .

Either command pulls every runtime dependency (CuratorKIT for data curation, plus the full ML stack) from pyproject.toml - no extra steps.

mergekit, unsloth, unsloth_zoo, and tokenizer-extension are vendored inside the aligntune package itself (under third_party/, built into the same wheel rather than installed as separate distributions), so there is no second install command and no --no-deps dance:

• mergekit (model merging) - two small patches for this project's transformers/pydantic versions; see third_party/mergekit/PATCH_NOTES.md.
• unsloth / unsloth_zoo - Unsloth's published metadata caps transformers<=5.5.0 and trl<=0.24.0 while this project pins transformers==5.14.1 and trl==1.7.1; vendoring sidesteps that cap. See third_party/unsloth/PATCH_NOTES.md and third_party/unsloth_zoo/PATCH_NOTES.md.
• tokenizer-extension (Indic/multilingual vocabulary extension) - vendored verbatim; it is not published on PyPI. See third_party/tokenizer-extension/PATCH_NOTES.md.

Requirements

• Python 3.11+
• PyTorch 2.0+
• CUDA-compatible GPU (recommended for faster training)

Local Notebooks

Beyond the Colab demos below, notebooks/ ships 46 local runnable notebooks — one per algorithm/technique (SFT, every RL algorithm, adapters, merging, long-context, tokenization). Each is a quick, tiny-model smoke test, runnable straight from the repo:

jupyter notebook notebooks/

Full breakdown by number range: docs/notebooks.md.

Demo Notebooks

Interactive Colab notebooks covering SFT, RL, distillation, tokenization, embedding, and long-context workflows.

Category	Backend	Algorithm / Technique	Model	Dataset	Link
SFT	TRL	SFT	Qwen/Qwen3-4B-Instruct-2507	sohamb37lexsi/bitext-wealth-management-llm-chatbot-splits
SFT	TRL	SFT	Qwen3-4B-Instruct	sohamb37lexsi/bitext-retail-banking-llm-chatbot-splits
SFT	Unsloth	SFT	Qwen/Qwen2.5-0.5B-Instruct	bebechien/MobileGameNPC
SFT	TRL	SFT	google/txgemma-2b-predict	trialbench_adverse-event-rate-prediction
SFT	Unsloth	SFT	Qwen/Qwen2.5-0.5B-Instruct	bebechien/MobileGameNP
RL	Unsloth	DPO	microsoft/phi-2	argilla/distilabel-intel-orca-dpo-pairs
RL	TRL	DPO	google/gemma-2-2b-it	Anthropic/hh-rlhf
RL	TRL	DPO	sohamb37lexsi/wealth_management_Qwen3-4B-Instruct-2507	sohamb37lexsi/bitext_wealth_management_preference_data
RL	Unsloth	PPO	Qwen/Qwen2.5-0.5B-Instruct	HuggingFaceH4/ultrachat_200k
RL	TRL	PPO	EleutherAI/pythia-1.4b	CarperAI/openai_summarize_tldr
RL	TRL	GRPO (Coding)	Qwen/Qwen3-4B	google-research-datasets/mbpp
RL	Unsloth	GRPO (Math)	meta-llama/Llama-3.2-3B-Instruct	openai/gsm8k
RL	TRL	GRPO	meta-llama/Llama-3.2-3B-Instruct	openai/gsm8k
RL	Unsloth	DRGRPO	Qwen/Qwen2.5-3B-Instruct	yahma/alpaca-cleaned
RL	TRL	DRGRPO	Qwen/Qwen2-0.5B-Instruct	AI-MO/NuminaMath-TIR
RL	Unsloth	GSPO	Qwen/Qwen3-1.7B	CyberNative/Code_Vulnerability_Security_DPO
RL	TRL	GSPO	meta-llama/Llama-3.2-3B-Instruct	HuggingFaceH4/ultrachat_200k
RL	Unsloth	DAPO	microsoft/Phi-3.5-mini-instruct	HuggingFaceH4/ultrachat_200k
RL	TRL	DAPO	meta-llama/Llama-3.2-3B-Instruct	google-research-datasets/mbpp
RL	TRL	SPIN	umarigan/llama3.2-1B-fin	gbharti/finance-alpaca
RL	TRL	ORPO	meta-llama/Llama-3.2-1B-Instruct (then SFT output)	bhavyagoyal-lexsi/orpo-ds
Distillation	TRL	Offline knowledge distillation (teacher: OpenDataArena/ODA-Fin-SFT-8B)	Qwen/Qwen3-1.7B	bhavyagoyal-lexsi/FINANCE-offline-online-distillation-data
Distillation	TRL	Online knowledge distillation (teacher: nvidia/Riva-Translate-4B-Instruct-v2)	tencent/Hy-MT2-1.8B	Helsinki-NLP/opus-100 (en-hi)
Distillation	TRL	Self-distillation fine-tuning (SDFT, self-teacher)	LiquidAI/LFM2.5-2.6B	—
Tokenization	—	Vocabulary extension (continued BPE, Arabic)	gpt2	M-A-D/Mixed-Arabic-Datasets-Repo
Embedding	TRL	Tokenizer + embedding adaptation, then LoRA SFT (Marathi)	LiquidAI/LFM2.5-1.2B-Instruct	ai4bharat/sangraha + saillab/alpaca-marathi-cleaned
Long context	TRL	RoPE scaling (context-window extension)	HuggingFaceTB/SmolLM-1.7B-Instruct	bavest/fin-llama-dataset
Long context	TRL	Sliding-window attention (SWA)	Qwen/Qwen2.5-3B	Sakshi1307/FindSUM
Long context	TRL	Shifted sparse attention (S2-Attn)	meta-llama/Llama-3.2-3B-Instruct	mrSoul7766/ECTSum + ChanceFocus/flare-edtsum

Further Resources

• CLI Reference: All 12 command groups, train, recipes, validate, diagnose, advise, merge, aligner, export, verify-export, adapters, compose, indic-eval.
• Algorithm Zoo: full comparison table and selection guide for every supported alignment method.
• Novelty Frontiers: 2026 research roadmap (forward-looking, not yet implemented).

Documentation

• Getting Started: Installation, setup, and basic usage
• User Guide: In-depth tutorials for SFT and RL training
• API Reference: Complete Python API and class/method details
• CLI Reference: Full command-line interface reference
• Examples: End-to-end code examples
• Advanced Topics: Architecture, custom backends, and performance optimization
• Notebooks: Interactive Colab notebooks and local Jupyter notebooks
• Hyperparameter Reference: Every configuration parameter, organized by training type and algorithm
• Unsloth Compatibility: Supported versions, known per-algorithm issues, and troubleshooting
• Changelog: Detailed, PR-by-PR change history

Key Capabilities

• Multiple Training Paradigms: SFT, DPO, PPO, GRPO, RAFT, and 15+ other RL algorithms
• Backend Flexibility: TRL, Unsloth, and ES backends with automatic fallback
• Reward Model Training: Train custom reward models, including verifiable (RLVR) rewards
• Comprehensive Evaluation: Multi-level evaluation with lm-eval integration and Indic-language benchmarks
• Production Ready: Model serialization, reproducible training, and deployment-ready pipelines (GGUF/Ollama/HF Hub export)
• Extensible Architecture: Modular design for easy integration of custom algorithms and backends

🧬 Regional Specialization (Indic-Plus)

AlignTune is uniquely architected for South Asian linguistic utility, moving beyond simple evaluation:

• Tokenizer Extender: Native extension of base model vocabularies (Llama-3/Mistral) using script-aware BPE merges for Devanagari, Tamil, Bengali, Telugu, Kannada, and Malayalam.
• Utility Benchmarks: Post-training validation via aligntune indic-eval against MILU, IndicXTREME, and IndicGenBench.

See Indic / Regional Post-Training for details.

Architecture

AlignTune uses a flexible backend architecture:

flowchart TD
    Factory[Backend Factory] --> TRL[TRL Backend]
    Factory --> Unsloth[Unsloth Backend]
    Factory --> ES[ES Backend]
    TRL --> TRL_Algos[13+ SFT/RL Algorithms]
    Unsloth --> Unsloth_Algos[13+ SFT/RL Algorithms]
    ES --> ES_Algos[Gradient-free Adapter Search]
    Factory --> Compose[Composition Runner]
    Compose --> Stage1[Stage: SFT] --> Stage2[Stage: Adapters/MoA] --> Stage3[Stage: RL/ES] --> Stage4[Stage: Audit]

Loading

TRL Backend: SFT, DPO, Online-DPO, PPO, GRPO, GSPO, DAPO, Dr. GRPO, GBMPO, Counterfactual GRPO, PACE, ORPO, SPIN, RAFT, Distillation

Unsloth Backend: SFT, DPO, Online-DPO, PPO, GRPO, GSPO, DAPO, Dr. GRPO, GBMPO, PACE, ORPO, SPIN, RAFT (see Supported Algorithms)

See Architecture for details.

Contributing

We welcome contributions! See our Contributing Guide for details.

License

This project is released under the Lexsi Labs Source Available License (LSAL) v1.1. Please cite appropriately if used in academic or production projects. See the LICENSE.md file for details, and THIRD_PARTY_LICENSES.md for vendored/third-party components (mergekit, CuratorKIT, Unsloth, etc.).

Key Points:

• Free for Research & Learning: Use, modify, and study for personal, academic, or research purposes
• Source Available: Full access to source code
• Commercial Use Restricted: Requires separate commercial license
• Contact: For commercial licensing, partnership, or redistribution rights, contact support@lexsi.ai

This is not an open-source license as defined by OSI, but provides broad access for non-commercial use.

Citation

If you use AlignTune in your research, please cite:

BibTeX:

@software{alignTune2025,
  title        = {{AlignTune}: Modular Toolkit for Post-Training Alignment of Large Language Models},
  author       = {Lyngkhoi, R E Zera Marveen and Goyal, Bhavya and Chawla, Chirag and Bhattacharjee, Soham and Avaiya, Utsav and Kadiyala, Ram Mohan Rao and Khandoga, Mykola and Yuan, Rui and Sankarapu, Vinay Kumar and Seth, Pratinav},
  year         = {2025},
  note         = {Equal contribution: Bhavya Goyal, R E Zera Marveen Lyngkhoi, Chirag Chawla, Pratinav Seth},
  organization = {Lexsi Labs},
  url          = {https://github.com/Lexsi-Labs/aligntune}
}

Plain Text:

Lyngkhoi, R. E. Z. M., Goyal, B., Chawla, C., Bhattacharjee, S., Avaiya, U., Kadiyala, R. M. R.,
Khandoga, M., Yuan, R., Sankarapu, V. K., & Seth, P. (2025). AlignTune: Modular Toolkit for
Post-Training Alignment of Large Language Models. Lexsi Labs. https://github.com/Lexsi-Labs/aligntune

*Equal contribution: Bhavya Goyal, R E Zera Marveen Lyngkhoi, Chirag Chawla, Pratinav Seth

Acknowledgments

AlignTune is built upon the excellent work of the following projects:

• HuggingFace Transformers - Model architectures and tokenizers
• TRL - Transformer Reinforcement Learning library
• Unsloth - Fast and memory-efficient training
• HuggingFace Datasets - Dataset loading and processing
• mergekit - linear/task-arithmetic model merging

Support

• Documentation: aligntune.lexsi.ai/
• GitHub Issues: github.com/Lexsi-Labs/aligntune/issues
• Discussions: github.com/Lexsi-Labs/aligntune/discussions
• Email: hello@lexsi.ai
• Discord: Discord Lexsi Labs

Contact

https://www.lexsi.ai

Paris 🇫🇷 · Mumbai 🇮🇳 · London 🇬🇧