Bend the latent space with your own sound: the complete open-source text-to-audio pipeline for musicians.
Fragmenta is an open-source app that brings the complete generative AI text-to-audio pipeline to musicians: intuitive dataset creation, LoRA training, generation, audio editing, and live performance. After the initial installation, the app runs fully offline, and your data never leaves your device.
Built on Stable Audio 3, Fragmenta is designed for all musicians, and especially for experimental music and sonic arts practitioners, giving them the ability to shape the latent space with their own audio and musical data, no coding required. It reflects the small-data, model-bending, and artist-first approaches to AI that are central to my PhD research.
Compatibility: The beta version's engine has been removed; if you need it, use the
v0.1.x-legacytag.
- Desktop app with a lightweight
pywebviewwindow and a pre-built React frontend - Bulk auto-annotation — generate text prompts for your audio files via DSP analysis (Basic) or AI tagging with LAION-CLAP (Rich), with optional user-defined vocabulary
- Project-aware LoRA training with configurable rank, steps, learning rate, batch size, checkpoint frequency, and precision — trains directly on a Dataset Workbench project
- LoRA adapters — train LoRA, DoRA, or BoRA adapters (plus low-VRAM
-xsvariants) on top of a frozen*-basecheckpoint for consumer GPUs; stack up to 4 at once with per-slot strength, bypass, and reorder at generation time - Text-to-audio generation — variable-length clips (up to 120s small / 380s medium), with CFG scale, inference steps, seed control, and a multi-LoRA stack
- Audio editing — style transfer (audio-to-audio), region inpainting, and clip extension/continuation. Temporarily hidden: these are implemented, but the upstream Stable Audio 3 runtime does not yet produce usable results for them, so the controls are switched off rather than shipped broken. They return in a future release once upstream support lands.
- Checkpoint Manager — pick and download individual SA3 checkpoints (Small Music/SFX, Medium, and the matching
*-basemodels) with per-item progress and hardware-compatibility hints - Performance Mode — a 4-channel live sampler: per-channel effects (gain, pan, filter, delay, reverb), master dBFS metering, bars-mode generation, launch quantization (standalone or via Ableton Link), persistent sessions, named presets, and MIDI learn (see Performance Mode)
- Real-time GPU memory monitoring
Works on Windows, macOS, and Linux. Small Music/SFX models run on CPU, Apple Silicon, or GPU; Medium needs an NVIDIA GPU. After a one-time model download everything runs offline and on-device — the in-app Checkpoint Manager handles the downloads (see Authenticate & Download Models).
Requirements: Python 3.11 (download) — newer versions (3.12, 3.13) won't install the dependencies. On Mac, Fragmenta supports Apple Silicon (M1 or newer) on macOS 14 (Sonoma or newer); Intel Macs are not supported.
Open your Terminal or PowerShell and paste the following:
git clone https://github.com/MAz-Codes/fragmenta.git
cd fragmentaRun the installer for your platform:
| Platform | Command |
|---|---|
| Linux | ./fragmenta.sh |
| macOS | fragmenta.command |
| Windows | ./fragmenta.bat |
The installer verifies Python 3.11 is available, sets up a virtual environment, installs all dependencies, and launches Fragmenta. The first run takes a while; subsequent launches are faster — re-run the same script to start the app. The install is fully isolated: deleting the folder removes everything except Python itself.
The app guides you through downloading models and authenticating with HuggingFace.
HuggingFace Token Requirement: the Stable Audio 3 weights are gated. You must:
- Accept their license on the model page (while logged into your HF account).
- Create a classic Read token and copy it into Fragmenta.
Datasets are built in the Dataset Workbench (the Dataset tab) — an in-app workspace that takes you from raw audio to a captioned, training-ready dataset without spreadsheets or leaving the app. Everything is scoped to a project: a folder of audio clips and their text prompts.
-
Create or load a project — start fresh, or reopen an existing one to keep editing.
-
Add audio — ingest a folder by copy (duplicates the originals, safe) or symlink (references them in place, saves disk).
-
Inspect & slice — every clip shows a waveform and plays back inline. Slice long files into training-sized segments.
-
Annotate — type prompts by hand, or auto-annotate every clip (or just selected ones):
Tier Adds Cost Basic tempo (BPM) + musical key, via librosa instant, CPU Rich Basic + zero-shot genre / mood / instruments via LAION-CLAP ~2.35 GB one-time download A prompt template (Music / Instrument / SFX) shapes how the tags are formatted, and the Rich tier's CLAP vocabulary is editable so tags match your own taxonomy.
-
Check & create — a health strip flags empty or duplicate prompts, sub-1 s clips, and unsupported formats. Save keeps a restartable draft; Create Dataset writes the final, training-ready form. Optionally pre-encode latents here for a 5–10× per-step training speed-up.
The finished project shows up directly in the Training tab — no export step.
Training produces a LoRA adapter against a *-base checkpoint (sa3-small-music-base, sa3-small-sfx-base, or sa3-medium-base).
| Base | VRAM (LoRA) | When to pick it |
|---|---|---|
sa3-small-*-base |
~2.5 GB (≈2 GB with -xs) |
Laptop / consumer GPU; fast iteration |
sa3-medium-base |
~6.5 GB (≈5.5 GB with bf16) |
Best fidelity; needs an NVIDIA GPU |
Pick a project from the Dataset Workbench, choose an adapter type (dora-rows is the recommended default), set rank/steps/learning rate, and Start. Each step_<n>.safetensors checkpoint appears in the LoRA picker as it's written.
Use base or distilled models to generate audio from text prompts — up to 120 s on the Small models, 380 s on Medium. If you've trained a LoRA, pick one and dial in a multiplier to control how strongly the adapter biases the output.
Settings:
- CFG scale — how closely the model follows your prompt (higher = stricter)
- Inference steps — diffusion steps per generation. More = higher quality, proportionally slower. Locked to 8 for the distilled small model.
- Seed — control randomness; same seed produces identical results
- Batch generation — generate multiple variations at once
This is a 4-channel diffusion sampler designed for live performance use. Each channel has its own prompt, generated clip, transport, and effect chain.
Per-channel controls:
- Text prompt and Generate (batch 1–4 for variations)
- Gain, pan, bipolar filter (one knob: low-pass below center, high-pass above), delay, reverb
- Mute, solo, loop toggle
- Waveform display and per-channel level meter
- Fragment history — every clip a channel generates is kept; audition it, star to keep, or drag it onto the waveform to load
Master controls:
- Master fader with peak-hold dBFS metering
- Play All / Stop All, and Record the master output to WAV
- Master tempo (BPM) — drives bars-mode generation and launch quantization
- Launch quantization (
Q) — snap launches to the grid (None, 1/32 → 8 Bars) - Master reverb (selectable impulse response) and tempo-synced delay division
- Ableton Link toggle — sync tempo with any Link-enabled app on the local network. The small Link binding (~1–2 MB) isn't bundled; the app offers to install it the first time you enable Link.
Bars-mode generation:
Switch a channel from sec to bars and pick a length (1, 2, 4, 8, or 16 bars). The clip is rendered to that bar count at the master BPM.
Launch quantization:
With Q set to anything other than None, channel launches snap to the next quantum on the bar grid. Works standalone via an internal beat clock, or in sync with Ableton Link if enabled. The first launch (when nothing is playing) fires immediately and anchors the transport; subsequent launches snap.
Sessions and presets:
- Every panel setting (BPM, knobs, prompts, model choice, etc.) auto-persists, so reloads and tab switches don't lose state.
- Click the floppy-disk icon in the toolbar to save the current configuration as a named preset, or load a saved one. Restore defaults wipes the session and MIDI mappings if you want to start clean.
MIDI: Click MIDI in the toolbar to enter learn mode, then click any UI element and move a hardware knob/button to bind it. Mappings persist across sessions.
Other:
- Prompt injection — optional toggles append the master tempo (BPM), a prompt key, and a time signature to each prompt (BPM is skipped if the prompt already names one). Set from the bottom bar.
- Shared inference steps and seed with the Generation page.
Real generation times measured in Fragmenta — full clip (sampling + VAE decode), warm model, batch size 1. Yours will vary with hardware, prompt, and settings.
Test machines: NVIDIA RTX 5080 · AMD Ryzen 5 7600 (CPU, 12 threads) · Apple M1 (8-core)
Models: Small = Music or SFX (same 433M architecture, near-identical speed; up to 120 s). Medium = 1.4B, NVIDIA GPU only (requires Flash Attention 2; up to 380 s). All values in seconds.
| Device | Inference steps | 10 s | 30 s | 60 s | 120 s |
|---|---|---|---|---|---|
| GPU (RTX 5080) | 8 (distilled) | 0.3 | 0.3 | 0.3 | 0.4 |
| GPU (RTX 5080) | 50 (base) | 1.4 | 1.4 | 1.6 | 2.7 |
| GPU (RTX 5080) | 250 (base) | 6.7 | 6.6 | 7.5 | 12.5 |
| CPU (Ryzen 5 7600) | 8 (distilled) | 4.7 | 8.3 | 14.7 | 27.1 |
| CPU (Ryzen 5 7600) | 50 (base) | 40.0 | 74.6 | 139.1 | 277.0 |
| CPU (Ryzen 5 7600) | 250 (base) | 197.4 | 402.9 | 691.5 | 1301.3 |
| CPU (Apple M1) | 8 (distilled) | 5.6 | 8.9 | 14.1 | 27.8 |
| CPU (Apple M1) | 50 (base) | 43.0 | 83.9 | 134.9 | 548.1 |
| CPU (Apple M1) | 250 (base) | 210.7 | 398.5 | 668.3 | 2515.9 |
| Inference steps | Device | 10 s | 30 s | 60 s | 120 s | 250 s | 380 s |
|---|---|---|---|---|---|---|---|
| 8 (distilled) | GPU (RTX 5080) | 0.4 | 0.5 | 0.6 | 1.1 | 2.3 | 5.7 |
| 50 (base) | GPU (RTX 5080) | 2.0 | 2.9 | 4.3 | 8.1 | 18.6 | 29.5 |
| 250 (base) | GPU (RTX 5080) | 9.9 | 13.7 | 19.9 | 39 | 90 | 143 |
fragmenta/
├── app/
│ ├── backend/ # Flask API server
│ ├── frontend/
│ │ ├── build/ # Pre-built React app (served by Flask)
│ │ └── src/ # React source (only needed for development)
│ └── core/ # Core logic (generation, training, model management)
├── vendor/ # Bundled third-party code
│ └── stable-audio-3/ # Stable Audio 3 (pinned snapshot; see UPSTREAM.md)
├── models/ # Model configs and checkpoints
├── utils/ # Utility modules
├── config/ # Configuration files
├── fragmenta.sh # Linux — first-time setup and subsequent launches
├── fragmenta.bat # Windows — first-time setup and subsequent launches
├── fragmenta.command # macOS — first-time setup and subsequent launches
├── install.py # Creates the venv + installs dependencies (run by the launch scripts)
└── start.py # Starts the backend + desktop window (run by install.py --launch)
| Problem | Solution |
|---|---|
| pywebview / GTK error on Linux | Install WebKitGTK / GTK runtime packages, then rerun the installer |
| Flash-Attention won't install | Small models run without it; Medium requires it (NVIDIA GPU). On Windows, install the prebuilt wheel provided with the project |
| GPU memory issues | Use the Free GPU button in the header, or reduce batch size |
| Performance panel feels stuck or its state is wrong | Open the preset menu and click Restore defaults |
| Multi-channel audio in Performance mode (macOS) | The native macOS window can't route audio to separate output devices. Open Fragmenta in your browser instead: http://localhost:5001 |
| Import errors | Verify Python 3.11 is installed — newer versions are not supported. Download Python 3.11.9. If you previously ran the installer with the wrong Python version, delete the venv/ folder and rerun. |
If you use Fragmenta in your research, please cite it. You can use the Cite this repository button on GitHub (powered by CITATION.cff), or:
Azimi, M. (2026). Fragmenta (Version 1.0.0) [Computer software]. https://doi.org/10.5281/zenodo.20692998
BibTeX:
@software{azimi_fragmenta_2026,
author = {Azimi, Misagh},
title = {Fragmenta},
year = {2026},
version = {1.0.0},
doi = {10.5281/zenodo.20692998},
url = {https://github.com/MAz-Codes/Fragmenta}
}The DOI is a Zenodo concept DOI — it always resolves to the latest release.
Fragmenta is free, open-source, and developed only by me as part of my PhD research on artist-first AI music tools. If it inspires you or powers your creativity, consider sponsoring its development. It directly funds maintenance, testing hardware, and documentation.
Before proposing a substantial change, please read the contribution guidelines and file an issue for discussion before opening a pull request.
Copyright 2025-2026 Misagh Azimi
Licensed under the GNU Affero General Public License v3.0 — see LICENSE for the full text.
What this means in practice:
- You can use, study, modify, and redistribute Fragmenta freely.
- If you distribute Fragmenta (modified or not), or run a modified version as a hosted/network service, you must release your source code under the same AGPL-3.0 license.
- You must keep the copyright notice and attribution.
If you want to use Fragmenta under different terms (e.g. embedded in a closed-source commercial product), please reach out — as the sole copyright holder I'm open to discussing alternative licensing.
Fragmenta is Powered by Stability AI, using the Stable Audio 3 model family.
"This Stability AI Model is licensed under the Stability AI Community License, Copyright © Stability AI Ltd. All Rights Reserved"
- Vendored code — the Stable Audio 3 inference/training code is bundled under
vendor/stable-audio-3/(pinned snapshot; see UPSTREAM.md) under the MIT License, © 2026 Stability AI — see vendor/stable-audio-3/LICENSE. - Model weights — not redistributed with Fragmenta; you download them from Hugging Face and accept their license then. Governed by the Stability AI Community License (free use up to USD $1M annual revenue; above that an enterprise license from Stability AI is required), and they include a T5Gemma text encoder under the Gemma Terms of Use. LoRA adapters you train on the
*-basecheckpoints are Derivative Works under the same Community License. - Impulse responses — the bundled convolution-reverb impulse responses under
app/frontend/public/ir/are original, synthesised by Fragmenta, with no external licensing.
Fragmenta also depends on many open-source libraries. See NOTICE.md for the complete attribution and license list.