MuScriptor

Multi-Instrument Music Transcription to MIDI and Sheet Music

---

🎵 Overview

MuScriptor is a state-of-the-art open-source multi-instrument music transcription system developed by Kyutai and Mirelo. It analyzes audio recordings (MP3, WAV, FLAC, OGG) and transcribes them into high-accuracy multi-track MIDI, interactive piano-roll visualizations, and engraved sheet music (MusicXML/PDF).

✨ Key Features

• Multi-Instrument Audio Transcription: Separates and transcribes pitch, duration, and onset timings for multiple instruments simultaneously.
• Full Audio Transport Controls: Professional player transport cluster (Play, Pause, Stop, Jump to Start, -5s Reverse, +5s Fast-Forward) with intuitive keyboard hotkeys (Space, Home/K, ←/J, →/L, Esc).
• Automated Guitar Tab & Chord Transcription: Automatically generates text-based guitar tablature and chord charts (guitar_tabs_chords.txt) alongside traditional notation scores.
• Synth String Vocal Representation: Detected vocal/voice tracks are synthesized using a Synth String soundfont patch (GM Program 50) for smooth audio previewing and DAW playback.
• Expanded Audio & Video Container Support: Built-in support for MP4, M4A, AAC, WEBM, MOV, OGG, FLAC, MP3, and WAV files with automatic multi-backend decoding (_read_non_wav_file).
• Interactive Piano-Roll & 8-Bit Retro Mode: Real-time note editing, stem controls, track isolation, and low-resolution 8-bit chiptune synthesis mode.
• Pure-Python Audio Fallback: Robust handling for standard and custom audio formats even in restricted environments without native SoundFont binaries.
• Sheet Music & Tablature Engraving: Generates quantized MusicXML, full score PDFs, and instrument tablatures using MuseScore 4.

---

⚡ Key Enhancements vs Upstream

This repository (Richie086/muscriptor) includes several core improvements over the original Kyutai base repository:

1. Full Player Transport Control Suite:

   • Upstream: Simple Play/Pause toggle without navigation or seeking.
   • This Project: Comprehensive transport toolbar (Controls.tsx) supporting jump to start, 5-second rewind/forward, full stop/reset, and global hotkeys (Space, Home/K, ←/J, →/L, Esc).

2. Automated Guitar Tablature & Chord Transcriber:

   • Upstream: Standard notation output without text-based tablature or chord progression extraction.
   • This Project: Added automated tab & chord generation (muscriptor/utils/tabs.py), creating formatted ASCII guitar tabs (guitar_tabs_chords.txt) during sheet music exports.

3. Vocal Track Synth String Representation:

   • Upstream: Default voice patch rendering.
   • This Project: Re-mapped vocal tracks (voice) to General MIDI Program 50 (Synth Strings 1) in both WebAudio (audio.ts) and MIDI export (notes.py) for a rich, continuous tone.

4. Expanded Audio & Video Container Ingestion:

   • Upstream: Primary support focused on WAV audio formats with errors on certain MP3/compressed streams in restrictive environments.
   • This Project: Added support for MP4, M4A, AAC, WEBM, OGG, FLAC, MP3, and WAV files with PyAV, soundfile, ffmpeg CLI, and torchaudio stream extraction via _read_non_wav_file in muscriptor/utils/audio.py.

5. Pure-Python Audio Processing Fallback & 8-Bit Mode:

   • Upstream: Required native system dependencies (fluidsynth) to process audio soundfonts and auralizations.
   • This Project: Implements a pure-Python additive synthesizer fallback (auralization.py) and a retro 8-bit chiptune synthesis engine for low-res audio rendering.

6. Interactive Export Hub & Client-Side MIDI Encoding:

   • Upstream: Basic inline export buttons in the output bar.
   • This Project: Adds a comprehensive Export Hub modal interface (ExportDialog.tsx) and a custom client-side midiEncoder.ts that serializes live piano-roll notes into multi-track Standard MIDI (.mid) files directly in the browser.

---

🚀 Quickstart & Local Servers

Prerequisites

• Python: 3.10 or newer
• uv: Installed via curl -sSf https://astral.sh/uv/install.sh or pip
• Node.js & pnpm: Node.js >= 18 and pnpm >= 9
• MuseScore 4 (Optional for PDF sheet music output): Download from musescore.org

1. Web UI & Production Backend Server

To start the integrated server hosting both the web client and the FastAPI backend:

uv run muscriptor serve --port 8222

• Local Output URL: http://127.0.0.1:8222

2. Frontend Hot-Reloading Development Server

If you are modifying the React frontend:

cd web
npx pnpm install
npx pnpm dev

• Local Dev URL: http://localhost:5173

---

🛠️ Local Build Instructions

Building the Web Frontend

Compile TypeScript and bundle static production assets into muscriptor/web_dist/:

cd web
npx pnpm run build

Building Python Wheel & Source Packages

Build local distribution packages (includes bundled web_dist assets):

uv build

The output artifacts will be saved in dist/:

• dist/muscriptor-0.3.0-py3-none-any.whl
• dist/muscriptor-0.3.0.tar.gz

---

💻 Command-Line Interface (CLI)

Transcribe to MIDI

uv run muscriptor transcribe path/to/song.wav

Transcribe to Sheet Music (PDF & MusicXML)

uv run muscriptor transcribe path/to/song.wav --format sheets --output score/

Output Structure:

score/
├── score.mid                       # Quantized multi-track MIDI
├── score.musicxml                  # Engraved score as MusicXML
├── full_score.pdf                  # Full ensemble score PDF
├── 01_electric_guitar.pdf          # Individual instrument score
└── 01_electric_guitar_tab.pdf      # Tablature PDF

---

📊 Model Variants

MuScriptor model weights are hosted under the MuScriptor HuggingFace Organization.

Variant	Parameters	Layers	Hidden Dim	Recommended Environment
small	103M	14	768	CPU-only / Laptops
medium (Default)	307M	24	1024	Apple Silicon / Standard GPU
large	1.4B	48	1536	High-end NVIDIA GPU

Before first use, log into HuggingFace:

uvx hf auth login
# or set environment variable
export HF_TOKEN=hf_...

---

📂 Project Architecture

muscriptor/
├── muscriptor/                     # Python Core & Server Package
│   ├── server.py                   # FastAPI SSE stream & HTTP routes
│   ├── model.py                    # Transformer transcription pipeline
│   ├── utils/                      # Audio loading, auralization & MIDI helpers
│   └── web_dist/                   # Built production frontend static assets
├── web/                            # React + TypeScript + Vite Frontend
│   ├── src/
│   │   ├── components/             # PianoRoll, Controls, ExportDialog, OutputBar
│   │   ├── hooks/                  # Transcription state & SSE stream handler
│   │   └── App.tsx                 # Main UI entrypoint
│   ├── package.json
│   └── vite.config.ts
├── dist/                           # Built Python wheel & sdist packages
├── pyproject.toml                  # Python package configuration (Hatchling)
└── README.md                       # Documentation

---

📄 License

• Code: Released under the MIT License.
• Model Weights: Released under CC BY-NC 4.0 License.

---

Maintained by Richie086 • Project Board: MuScriptor Board