Add Voice Cloning (MLX) - #1
Open
dbrkn wants to merge 6 commits into
Open
Conversation
SpeakerEncoder (ECAPA x-vector), SpeechEncoder (Mimi), and SpeechEncoderRVQ CoreML wrappers with the reference-tiling rule, two-branch RVQ (acoustic residual reset), and padded-tail trim; an Accelerate mel front-end matching the Python reference (reflect pad, periodic Hann, slaney filterbank, log clamp) golden-tested to <5e-3; VoiceClonePrompt as the encoder output consumed by generation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dual-track ICL prefix mirroring the Python reference (control block with the x-vector in the speaker slot, reference transcript + synthesis text over codec PAD, reference RVQ frame embeddings under text PAD), prompt-fit validation against the CodeDecoder KV budget, x-vector-only mode via a speaker-slot override, GenerationOptions.voiceClone, textEOS constant. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tts.cloneVoice(referenceAudio:referenceText:xVectorOnly:) with lazy voice-clone encoder loading; 0.6b-base model variant + per-component variant overrides (config + CLI); singleFunction SpeechDecoder mode for single-function asset exports; AVFoundation reference-audio loader; argmax-cli --ref-audio/--ref-text/--x-vector-only; Makefile download support for base-family assets. All encoding happens on-device. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dbrkn
force-pushed
the
berkin/voice-clone-mlx
branch
from
July 21, 2026 15:13
ba60f9d to
c2819ee
Compare
….7b-base preset The download config was rebuilt from the model preset alone, silently fetching the default versionDir/variants while loadModels() resolved the configured ones — any variant override without a local --models-path failed with 'No .mlmodelc found'. The voice-clone encoder assets were also never part of the download patterns. - setupModels: carry versionDir + all component variants + revision into the download config. - TTSKitConfig.voiceCloneDownloadPatterns: encoder-asset globs; the CLI appends them to the download when --ref-audio is set. - Base-family presets now map to the published base variant layout (W8A16-kv_len_512 talker, W8A16-kv_len_16 MCD, W16A16 text projector, single-function speech decoder) instead of the custom-voice names that do not exist under 12hz-*-base. - New 1.7b-base preset (12hz-1.7b-base carries the full voice-clone asset set); --speech-decoder-mode defaults to singleFunction for -base presets. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ckend Opt-in (default coreml unchanged): the reference clip is encoded by MLX-Swift ports of the ECAPA speaker encoder and Mimi/RVQ encoder — variable-length references with no fixed window (memory-capped at 120s; peak Metal memory scales ~90MB per reference second). Parity vs the Python mlx-audio reference: x-vector cosine 1.000000 and 100% exact RVQ codes on both an 8s and a 32s reference ([16,101] / [16,403] shapes). CoreML encoders remain the default and the only path below macOS 14. Package macOS floor 13->14 (mlx-swift's minimum; TTS generation already required macOS 15 at runtime). MLX products are macOS-conditional deps, so iOS/watchOS builds are unaffected. graft-mlx-metallib Makefile target covers command-line SwiftPM's Metal-shader limitation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dbrkn
force-pushed
the
berkin/voice-clone-mlx
branch
from
July 21, 2026 16:29
c2819ee to
2a10189
Compare
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds voice cloning support to TTSKit using the MLX based voice cloning encoder assets.