English · العربية · Español · Français · 日本語 · 한국어 · Tiếng Việt · 中文 (简体) · 中文（繁體） · Deutsch · Русский

Musia

AI song localization: extract the human voice, stems, lyrics, beats, and chords from a song, then prepare the path toward singable multilingual re-singing.

Musia is a local-first research prototype for AI music localization. The current MVP takes an input song, separates it into the four Demucs stems bass, drums, vocals, and other, creates an instrumental mix, aliases the vocal as human_sound, transcribes lyrics, estimates beats, and produces Chordify-style chord segments.

Donate	PayPal	Stripe

What It Produces

input song
-> source/input.wav
-> stems/bass.wav
-> stems/drums.wav
-> stems/vocals.wav
-> stems/other.wav
-> stems/instrumental.wav
-> stems/human_sound.wav
-> analysis/lyrics.json + lyrics.txt
-> analysis/beats.json + beats.csv
-> analysis/chords.json + chords.csv
-> manifest.json + REPORT.md

instrumental.wav is mixed from bass + drums + other. human_sound.wav is the isolated vocals.wav stem.

Current Contents

Path	Purpose
musia/	Local Python analysis toolkit.
package.json	npm package metadata for the musia CLI/webapp wrapper.
bin/musia.js	npm executable that launches Musia Studio and Python workflows.
website/	Static Fun Lazying Art media website with a synced player, covers, lyrics/subtitles, pinyin, furigana, chords, and share metadata.
SETUP_AND_TEST.md	Fresh-clone setup, smoke test, matrix test, and script map.
scripts/setup_and_smoke_test.sh	One-command core setup plus open-song smoke test.
scripts/bootstrap_musia.sh	Creates the conda environment and installs the local stack.
scripts/download_open_songs.py	Downloads free/open test songs.
scripts/run_pipeline.py	Runs separation, transcription, beats, chords, and report generation.
scripts/test_open_songs_matrix.sh	Runs the EN/ZH/JA open-song analysis matrix.
scripts/install_research_repos.sh	Shallow-clones optional research repositories into third_party/.
scripts/download_quality_backends.sh	Downloads high-quality research backends and model weights into ignored local folders.
scripts/install_quality_envs.sh	Creates isolated repo-local conda/uv environments for large singing and music models.
scripts/run_soulx_env.sh	Runs SoulX-Singer with the cloned repo on PYTHONPATH.
scripts/run_soulx_svs.sh	Runs SoulX-Singer SVS inference with prompt and target metadata.
scripts/soulx_rewrite_metadata.py	Rewrites SoulX metadata with Mandarin target lyrics while preserving note/F0 timing.
scripts/mix_vocal_with_instrumental.sh	Mixes a generated vocal with an instrumental stem and loudness-normalizes it.
scripts/run_localization_performance_pipeline.sh	Dedicated EN/ZH localization performance demo using DeepSeek or OpenAI for lyric/check packages.
scripts/musia_create.py	Creative CLI for idea, lyrics, chords, notation, reference audio, and localization project planning.
scripts/musia_song_workbench.py	Reusable song generation, correction, review, and LALACHAN handoff workbench.
scripts/musia_studio_web.py	Local Musia Studio web app.
scripts/start_musia_studio_tmux.sh	Starts Musia Studio in tmux.
scripts/musia_quality_check.py	Checks generated audio duration, levels, ASR, and lyric overlap.
scripts/validate_fun_media_site.py	Validates the fun.lazying.media.v1 website catalog, manifests, assets, and language tracks.
scripts/run_moss_music_env.sh	Runs MOSS-Music with the required FFmpeg, TorchCodec, and CUDA library paths.
scripts/musia_lyricfit_openai.py	Optional OpenAI-powered lyric adaptation helper.
references/	Architecture, deep research, and local setup notes.
TODO.md	Build checklist and next engineering steps.

Quick Start

Fresh clone, setup, and smoke test:

git clone https://github.com/lachlanchen/Musia.git
cd Musia
bash scripts/setup_and_smoke_test.sh

Manual core commands:

bash scripts/bootstrap_musia.sh
PYTHONNOUSERSITE=1 conda run -n musia python scripts/download_open_songs.py --id danny-boy-1917
PYTHONNOUSERSITE=1 conda run -n musia python scripts/run_pipeline.py data/open_songs/danny-boy-1917/original.ogg --run-name smoke-danny --max-duration 45 --asr-model tiny

Results are written to:

data/runs/<run-name>/

Generated audio, downloaded songs, model weights, and third-party clones are ignored by git.

For the full setup/test guide, see SETUP_AND_TEST.md.

Fun Lazying Art Website

The public media site lives in website/ and is intended for songs, localized songs, MVs, short films, covers, synced lyrics, subtitles, chords, stems, and shareable media pages.

The default catalog item is One Sky, Three Lights, a mixed English, Mandarin-pinyin, and Japanese-romaji song with trilingual native-script lyrics, pinyin, furigana, chord timing, and translation-word highlighting. The catalog also includes Small Under the Red Sky and Rain Day Full Song; independent full-song renders use per-vocal trilingual lyric sets when the generated versions do not sing identical lines.

npm run website:validate
python3 -m http.server 8778 --directory website

Local URL:

http://127.0.0.1:8778/

Public URL after GitHub Pages deploy:

https://fun.lazying.art

The reusable data protocol is documented in references/musia-website-json-format.md. The full song-generation and website-publishing workflow is documented in references/musia-song-generation-and-website-runbook.md.

Song Workbench

For repeatable song generation and review:

musia song init --title "My Song" --vocal-language ja --lyrics-file lyrics.txt
data/creative_projects/<song-id>/commands.sh generate
data/creative_projects/<song-id>/commands.sh review
musia song handoff --project-dir data/creative_projects/<song-id> --audio data/creative_projects/<song-id>/final/selected.mp3

See references/musia-song-workbench.md and references/lalachan-song-first-video-workflow.md.

npm CLI And Webapp

Musia also ships as an npm package wrapper around the local Python/conda engine.

From this repo:

npm install
npm link
musia doctor
musia studio --tmux

From npm:

npm install -g @lazyingart/musia
musia doctor

Or without linking:

npm run check
node bin/musia.js studio --tmux

Useful npm/CLI commands:

musia setup
musia models
musia plan --title "My Song" --idea "A cinematic song about..." --provider deepseek
musia soulx-verse --title "Rain Day" --idea "A rainy bilingual musical short film verse"
musia new-session --title "My Song" --cwd ./my-song-folder
musia chat --cwd ./my-song-folder --mode chat "What should I do next?"
musia chat --cwd ./my-song-folder --mode worker "Analyze this project and register artifacts."
musia resume-session <session-id> --cwd ./another-folder
musia pipeline data/open_songs/danny-boy-1917/original.ogg --run-name npm-smoke --max-duration 45
npm run pack:dry-run

The npm CLI uses conda run -n musia python ... by default. Override with:

MUSIA_CONDA_ENV=musia
MUSIA_PYTHON=/path/to/python
MUSIA_NO_CONDA=1

Legacy note: older local setups may still have the previous Musai / musai name or MUSAI_* environment variables. Treat those as compatibility fallbacks only. New commands, docs, and environments should use Musia, musia, and MUSIA_*.

Run the dedicated localization performance demo after the smoke/matrix artifacts exist:

MUSIA_LYRIC_PROVIDER=deepseek \
DEEPSEEK_MODEL=deepseek-reasoner \
scripts/run_localization_performance_pipeline.sh

If your OpenAI account has access to GPT-5.5-compatible chat completions, the same demo can be run with:

scripts/run_localization_performance_pipeline.sh --provider openai --model gpt-5.5

Create original music projects from ideas, lyrics, chords, notation, or reference recordings:

PYTHONNOUSERSITE=1 conda run -n musia python scripts/musia_create.py plan \
  --title "My Song" \
  --idea "A hopeful acoustic song about..." \
  --lyrics-file lyrics.txt \
  --chords "C G Am F" \
  --provider deepseek

Start the local web app:

scripts/start_musia_studio_tmux.sh

Musia Studio and the CLI share sessions. A working folder gets a .musia/sessions/<session-id>/session.json pointer, while messages and jobs stay in data/studio/, so you can create or resume music sessions from either web or CLI.

See references/musia-creative-studio.md.

For the new control-level model and the SoulX vocal-only workflow, see references/musia-control-and-soulx-workflow.md. For original short SoulX verse generation, see references/soulx-verse-tool.md and references/lalachan-musia-musical-short-film-handoff.md. For the full installed capability map and practical workflows, see references/musia-full-capability-guide.md.

Local Validation

The local smoke test on an open Wikimedia Commons recording passed on a machine with an NVIDIA RTX 4090 D:

PYTHONNOUSERSITE=1 conda run -n musia python scripts/run_pipeline.py data/open_songs/danny-boy-1917/original.ogg --run-name smoke-danny-120-fixed --max-duration 120 --asr-model base.en --language en --demucs-device cuda

Recorded result:

• Four stems: bass, drums, vocals, other
• Additional audio: instrumental, human_sound
• Tempo estimate: 129.20 BPM
• Beat count: 257
• Chord segments: 132
• Lyrics status: ok

See references/local-setup-and-test-report.md.

A broader open-song matrix and first-verse Chinese localization test were run on 2026-06-28:

MAX_DURATION=60 ASR_MODEL=small DEMUCS_DEVICE=cuda scripts/test_open_songs_matrix.sh

See references/open-song-test-report-2026-06-28.md for exact input songs, output paths, SoulX/YingMusic attempts, and the current quality gate results.

The dedicated DeepSeek/OpenAI localization performance script was also run on 2026-06-28. See references/localization-performance-demo-2026-06-28.md.

High-Quality Backends

The local machine has staged heavier research backends for better vocal quality and music understanding:

• Strict song localization: SoulX-Singer and YingMusic-Singer-Plus.
• Full-song alternatives: ACE-Step 1.5, YuE, DiffRhythm, SongGen, HeartMuLa.
• Music understanding and QA: MOSS-Music.

See references/high-quality-vocal-backends.md and references/local-quality-backend-install-status.md. For the practical control modes, including vocal-only SoulX generation, melody/旋律 generation, full production, controlled material-based generation, and licensed localization, see references/musia-control-and-soulx-workflow.md. For the local SoulX vocal-only outputs that sounded usable, see references/soulx-good-vocal-runs-2026-06-28.md.

For the broader EN/ZH/JP model map, including Chinese-company repos and Microsoft Muzic, see references/en-zh-jp-music-model-repos.md.

Architecture Direction

Musia is not only translation plus TTS. The intended full pipeline is:

song upload
-> rights / ownership check
-> vocal + instrumental separation
-> lyrics transcription
-> word / phoneme timing
-> melody / pitch extraction
-> singable lyric adaptation
-> AI singing synthesis
-> optional voice/timbre conversion
-> mixing + mastering
-> music-player interface

The current repository implements the first local analysis layer. Singing synthesis with YingMusic-Singer-Plus, SoulX-Singer, and related models is staged as a research integration because those models require larger weights, license checks, and separate GPU-worker packaging.

Citation

If you use Musia in research, cite the repository. GitHub reads CITATION.cff and shows a Cite this repository panel on the repo page.

@software{chen_musia_2026,
  author = {Chen, Lachlan},
  title = {Musia: Local-first AI song localization and music analysis},
  year = {2026},
  url = {https://github.com/lachlanchen/Musia}
}

Status

Musia is early research software. The local pipeline works for testing and artifact generation, but the chord detector is a lightweight baseline and the singable re-singing layer is not production-ready yet. Use songs you own, public-domain songs, licensed songs, or creator-uploaded material.