Inject LoRA, train, merge. A small, readable PEFT helper for transformer LMs.
loraflux is a tiny library for parameter-efficient finetuning. It injects
low-rank adapters into the layers you care about, trains only those, and can
fold them back into the base weights when you are done — no extra inference
cost, no framework lock-in.
Most of the time you don't need a giant framework to attach a few LoRA matrices. You need something you can read in an afternoon and trust. That's the goal here.
pip install -e .from transformers import AutoModelForCausalLM
from loraflux import inject_lora, mark_only_lora_as_trainable
model = AutoModelForCausalLM.from_pretrained("gpt2")
inject_lora(model, "gpt2", r=8, alpha=16, dropout=0.05)
mark_only_lora_as_trainable(model)
# ... train with your own loop, or use loraflux.Trainer ...Merge the adapter back when you're done:
from loraflux import merge_adapter
merge_adapter(model) # base weights now contain the updateLoRALinearwith rank / alpha scaling, dropout and zero-initB- Target modules by suffix, regex, or per-architecture presets
- Tiny
Trainer: gradient accumulation, clipping, warmup + cosine schedule - Save adapters only, or merge + export a standalone checkpoint
- A small CLI:
loraflux finetune | merge | inspect
GPT-2 small, single epoch on a 20k-line instruction set (1x RTX 3090):
| method | trainable params | peak mem | val ppl |
|---|---|---|---|
| full finetune | 124M (100%) | 11.4 GB | 18.2 |
| loraflux r=8 | 0.30M (0.24%) | 6.1 GB | 18.9 |
| loraflux r=16 | 0.59M (0.47%) | 6.3 GB | 18.6 |
Numbers are indicative, not a rigorous study.
- v0.1 (2024-03) —
LoRALinear, suffix-based injection - v0.2 (2024-07) — presets, merge/export, CLI, public API
- v0.3 (2024-09) — safetensors adapters, config validation
- v0.4 (2024-11) — planned: 8-bit base weights, adapter hub upload
- v0.5 (2024-12) — planned: QLoRA-style quantised finetuning
- DoRA / weight-decomposed variant
- Multiple named adapters on one model
- Gradient checkpointing helper
Apache-2.0 © 2024 Wu Yunling