Add Voice Cloning (CoreML) - #2
Open
dbrkn wants to merge 4 commits into
Open
Conversation
dbrkn
force-pushed
the
berkin/voice-clone-coreml
branch
from
July 20, 2026 14:11
9951843 to
c64d748
Compare
SpeakerEncoder (ECAPA x-vector), SpeechEncoder (Mimi), and SpeechEncoderRVQ CoreML wrappers with the reference-tiling rule, two-branch RVQ (acoustic residual reset), and padded-tail trim; an Accelerate mel front-end matching the Python reference (reflect pad, periodic Hann, slaney filterbank, log clamp) golden-tested to <5e-3; VoiceClonePrompt as the encoder output consumed by generation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dual-track ICL prefix mirroring the Python reference (control block with the x-vector in the speaker slot, reference transcript + synthesis text over codec PAD, reference RVQ frame embeddings under text PAD), prompt-fit validation against the CodeDecoder KV budget, x-vector-only mode via a speaker-slot override, GenerationOptions.voiceClone, textEOS constant. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tts.cloneVoice(referenceAudio:referenceText:xVectorOnly:) with lazy voice-clone encoder loading; 0.6b-base model variant + per-component variant overrides (config + CLI); singleFunction SpeechDecoder mode for single-function asset exports; AVFoundation reference-audio loader; argmax-cli --ref-audio/--ref-text/--x-vector-only; Makefile download support for base-family assets. All encoding happens on-device. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dbrkn
force-pushed
the
berkin/voice-clone-coreml
branch
from
July 21, 2026 12:56
f75887d to
99611a3
Compare
….7b-base preset The download config was rebuilt from the model preset alone, silently fetching the default versionDir/variants while loadModels() resolved the configured ones — any variant override without a local --models-path failed with 'No .mlmodelc found'. The voice-clone encoder assets were also never part of the download patterns. - setupModels: carry versionDir + all component variants + revision into the download config. - TTSKitConfig.voiceCloneDownloadPatterns: encoder-asset globs; the CLI appends them to the download when --ref-audio is set. - Base-family presets now map to the published base variant layout (W8A16-kv_len_512 talker, W8A16-kv_len_16 MCD, W16A16 text projector, single-function speech decoder) instead of the custom-voice names that do not exist under 12hz-*-base. - New 1.7b-base preset (12hz-1.7b-base carries the full voice-clone asset set); --speech-decoder-mode defaults to singleFunction for -base presets. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds voice cloning support to TTSKit using the CoreML based voice cloning encoder assets.