title	Large Music Model
emoji	🎹
colorFrom	indigo
colorTo	blue
sdk	docker
app_port	7860
pinned	false
hf_oauth	true

Large Music Model (LMM)

A small symbolic-music Transformer that does three things with one set of weights:

Continue	you upload an unfinished piece, it writes the rest
Arrange	you upload a finished piece, it writes a part for another instrument
Steer	you pick composer / genre / mood / style tags, the output follows them
Edit	drag, draw and delete notes in a piano-roll editor
Rewrite	select bars and have the model redo just that region
Learn your style	upload your MIDI or recordings, get your own tag

Those are not three models. They are three masking patterns over one model. See lmm/masking.py — it is the file that makes the whole design work, and the first one to read.

---

Quick start

pip install -r requirements.txt      # Windows: see SETUP.md first
python scripts/run_demo.py           # 7 stages, ~90s, CPU
uvicorn api.main:app --port 8000     # then open http://localhost:8000

Full Windows instructions, including what you need installed: SETUP.md.

That builds a synthetic corpus, trains a tiny model on CPU (~2 min), and writes one example of each mode to out/demo/. The music will be bad. That is the point: it proves every stage connects to the next, so that later problems are data or model problems rather than plumbing problems.

Then open notebooks/01_explore_midi.ipynb and actually look at the data.

Doing it for real

# 1. put real MIDI in data/raw/ (see data/tags.py for where to get it)
# 2. optional but recommended: produce data/raw/tags.json for tag steering
python -m data.tags   # or write your own join

# 3. build the corpus
python -m data.build_corpus --raw data/raw --out data/processed --bpe-vocab 2000

# 4. train (Colab T4, ~4-6 h; resume freely, it checkpoints on a timer)
python -m lmm.train --config configs/train.yaml
python -m lmm.train --config configs/train.yaml --resume

# 5. generate
python scripts/generate.py --ckpt out/run1/last.pt --mode new \
    --tags genre=jazz,mood=calm --guidance 2.0 --out out/new.mid

python scripts/generate.py --ckpt out/run1/last.pt --mode add-track \
    --input my_song.mid --family bass --out out/with_bass.mid

# 6. evaluate — always against the baseline, never alone
python -m lmm.baseline_markov --corpus data/raw
python -m eval.objective --real data/held_out --generated out/samples \
    --ckpt out/run1/last.pt --shards data/processed/shards

# 7. serve — UI at /, API docs at /docs
uvicorn api.main:app --port 8000

Training on a free Colab GPU: notebooks/02_train_colab.ipynb. Deploying a public URL (Google Cloud Run): DEPLOY.md.

Layout

configs/          model + training hyperparameters (yaml)
data/
  filter.py       quality gate for raw MIDI — run standalone to see rejection reasons
  tags.py         where genre/mood/composer labels come from, and licensing notes
  build_corpus.py raw MIDI -> tokenizer + tokenized shards
lmm/
  tokenizer.py    miditok wrapper; owns the [CTRL] BOS TRACK PROG ... layout
  masking.py    ★ the three training objectives + the inference prompt builders
  model.py        decoder-only Transformer (RMSNorm, RoPE, SwiGLU), ~25M params
  dataset.py      shard storage + batching; masking happens at batch time
  train.py        training loop, built to survive Colab disconnects
  sample.py       top-p sampling + classifier-free guidance + KV cache
  checkpoint.py   one place that rebuilds a model from a checkpoint
  baseline_markov.py   order-N Markov chain — your week-1 demo and your baseline
  style.py        ★ user-trained styles by textual inversion (frozen model)
  regions.py      bar-accurate region regeneration for the editor
  transcribe.py   audio -> MIDI via basic-pitch (optional dependency)
eval/objective.py descriptors, distribution comparison, held-out perplexity
api/main.py       FastAPI: serves the UI at /, plus /generate /continue /add-track /tags /styles
api/store.py      SQLite: generation history + style vectors
api/auth.py       Sign in with Hugging Face (optional)
api/jobs.py       one background worker for style training
web/index.html    the front end — single file, no build step, no framework
scripts/          run_demo.py, generate.py, make_demo_midi.py
notebooks/        01_explore_midi.ipynb, 02_train_colab.ipynb, 03_train_styles.ipynb
Dockerfile        one image for Cloud Run and HF Spaces; CPU-only torch
SETUP.md          Windows setup + troubleshooting
DEPLOY.md         publishing the model, then deploying to Cloud Run

The four decisions worth knowing about

One model, three jobs. Training mixes 55% continue-the-suffix, 30% mask-a- whole-track, 15% mask-a-span. Infilling works in a causal model by reordering: the hidden material is moved after a <SEP> token so the model can see all the surrounding context before predicting it. lmm/masking.py.

Control-token dropout. Each tag is randomly dropped 15% of the time during training. This gives the model a real unconditional mode, which is what lets classifier-free guidance at sampling time turn weak tag adherence into strong tag adherence — no extra training. Set it to 0 and the tag selector will feel like it does nothing. lmm/masking.py, lmm/sample.py.

BPE on the token stream. Roughly halves sequence length, which roughly doubles how much music fits in the context window. Free quality. data/build_corpus.py.

~25M parameters, deliberately. Big enough to learn the corpus you can actually assemble, small enough to run inference on CPU — which is what makes free hosting possible. lmm/model.py.

The front end

Three views — Generate, Editor, History — in plain HTML, CSS and ES modules. No npm, no build step, no framework. Open / once the server is running.

web/index.html    markup + styles
web/js/midi.js    MIDI reader AND writer (the editor has to send edits back)
web/js/editor.js  canvas piano-roll editor with undo/redo
web/js/app.js     wiring

Two decisions worth knowing about:

The piano roll is ours, the audio is not. We parse and draw MIDI in JavaScript; the CDN <midi-player> is only for sound. A blocked CDN — school wifi, a locked-down network — costs you playback, not the whole interface.

"Regenerate selected bars" is the feature to demo. Select bars on the ruler, pick a track, and the model rewrites only that region while keeping everything around it. That is INFILL_SPAN from training aimed where the user points, and it is undoable like any other edit. See lmm/regions.py.

User-trained styles

Users can teach the model their own style — and it does not involve retraining anything.

The technique is textual inversion: freeze all 35M weights and learn k input vectors (default 4) that steer the model toward the user's uploads. What you get per style:

~2 KB	vs ~2 MB for a LoRA, ~140 MB for a fine-tune
minutes on CPU	only 2048 numbers receive gradients
one shared model	no per-request adapter loading; free hosting survives
composes with tags	applied where control tokens live, so guidance steers it

What it cannot do is change how the model writes. If a style needs behaviour the base model never learned, prompt tuning will not find it — that is LoRA's job. LORA_NOTES at the bottom of lmm/style.py documents exactly what that takes; the API already carries a method field so it slots in without a breaking change.

Two ways to train:

• In the app — Styles tab. Upload MIDI (or audio, if basic-pitch is installed), it queues on a single background worker, chip appears when ready.
• In batch on a GPU — notebooks/03_train_styles.ipynb. One folder per person, twenty styles in a few minutes, import the .npy files. This is the workshop path.

Every trained style is scored against held-out pieces from the same upload. If it does not lower loss, the UI says so rather than handing someone a chip that does nothing.

Styles are tied to the base model — the vectors live in that model's embedding space. Retrain the model and styles must be retrained; the app checks d_model and refuses a mismatch rather than producing quiet nonsense.

Accounts and history

Every generation is saved with the settings that produced it — sampling is stochastic, so a good result you did not download is otherwise gone for good.

Anonymous visitors get history immediately, keyed to a session cookie; no login wall in front of the thing you want people to try. If sign-in is enabled (hf_oauth: true on the Space), signing in adopts that anonymous history rather than discarding it. See api/auth.py and api/store.py.

A note on composer tags

Tag values come from the training corpus, so the chip list grows the moment you train on real data — MAESTRO alone gives ~60 real composers.

Named living composers are deliberately not the recommended path. It requires a corpus of copyrighted work as MIDI, which is a real takedown risk on a public app. The style namespace (cinematic, epic, minimal, driving, …) exists to capture the sound people are reaching for when they name an artist, without that exposure — and data/tags.py:style_from_features derives it from the notes themselves, so it works on any corpus.

Limitations

• Symbolic only. It reads and writes MIDI, not audio. Rendering to sound needs a synthesizer or soundfont.
• No per-user fine-tuning. Uploads condition the model in-context; they do not retrain it. LoRA adapters are the realistic next step.
• Tag steering is only as good as the tag metadata, which for scraped corpora is noisy Last.fm crowd labels.
• Long-range structure (verse/chorus/recapitulation) is beyond a 1024-token context. The model writes convincing phrases, not convincing forms.

Data provenance

Read the licensing section of data/tags.py before deploying anything publicly. Lakh MIDI is scraped and unlicensed; that is normal for research and murky for a hosted app. Whichever way you go, document the choice.