NEUTRINO Multi-Version Support: Add v2.3.x & v3.1.x compatibility, fix various issues From PR#2136 and PR#2132 - #2232
Closed
27704534 wants to merge 36 commits into
Closed
NEUTRINO Multi-Version Support: Add v2.3.x & v3.1.x compatibility, fix various issues From PR#2136 and PR#2132#223227704534 wants to merge 36 commits into
27704534 wants to merge 36 commits into
Conversation
Changes have been implemented to support NEUTRINO type singers and renderers. - Added USingerType.Neutrino to ClassicSingerLoader.cs. - Added NeutrinoLabelPhonemizer.cs and implemented label generation. - Added NeutrinoRenderer.cs and implemented audio rendering. - Added NeutrinoServerLauncher.cs and implemented server management. - Added NeutrinoSinger.cs and implemented NEUTRINO singers. - Added the NEUTRINO renderer to Renderers.cs. - Added USingerType.Neutrino to USinger.cs. - Added locale settings and background process launching to ProcessRunner.cs. - Added TomlData.cs and implemented TOML file reading. - Added TomlDataTests.cs and implemented tests for TOML utilities. - Added the NEUTRINO type option to SingersViewModel.cs.
The test assertions within the MeasureForwardBackwardAreComputedPerBar method have been fixed. The expected values for the indices e0, e1, and e2 have been modified, and the following items were updated: - Forward Index (e10): Fixed expected values for e0[9], e1[9], and e2[9]. - Backward Index (e11): Fixed expected values for e0[10], e1[10], and e2[10]. - Forward Percent (e16): Fixed expected values for e1[15] and e2[15]. - Backward Percent (e17): Fixed expected values for e0[16] and e1[16]. As a result, the test's expected values have been updated to align with the specifications.
…nting everything in the subclasses Change HTSLabelRenderer: SetUp method to abstract The SetUp method has been changed from virtual to abstract. As a result, all subclasses are now required to implement SetUp. The original logic within the SetUp method (initialization of phoneDict, language settings, dictionary loading, etc.) has been removed, and these responsibilities are now delegated to the subclasses.
Added a process to adjust the pitch based on the toneShift value of phrase.phones[0] when generating RenderPitchResult. This allows the pitch to be shifted in units of 12-interval semitones, ensuring accurate pitch calculation.
In NeutrinoServerLauncher.cs, I changed the log output to string interpolation to improve readability. In ProcessRunner.cs, I removed the GetLanguageEnvironmentValue method and set the LANG environment variable to a fixed value of "ja_JP.utf8". Additionally, I updated the code to dynamically control RedirectStandardOutput based on the DebugSwitch.
Path handling has been unified into a platform-independent format using Path.Join. This removes Windows-style path separators and improves code readability and portability.
The conditions for the IsSyllableVowelExtensionNote method have been expanded to recognize lyrics starting with specific symbols as vowel extension notes. Additionally, the calculation of phonemeDuration within the ProcessPart method has been removed, and a logic to directly calculate startMs and endMs has been introduced. In phoneme timing calculations, new logic considering headMs and phrase.positionMs has been added, and a process to adjust the end time of existing monoLabels has been implemented. This prevents overlaps and inconsistencies, improving the accuracy of timing. Furthermore, the startMs of the monoLabel at the end of a phrase has been changed to sentenceDurMs - tailMs to ensure that the timing of the entire phrase is accurately reflected.
The conditions for the IsSyllableVowelExtensionNote method have been expanded to recognize lyrics starting with specific symbols as vowel extension notes. Additionally, the calculation of phonemeDuration within the ProcessPart method has been removed, and a logic to directly calculate startMs and endMs has been introduced. In phoneme timing calculations, new logic considering headMs and phrase.positionMs has been added, and a process to adjust the end time of existing monoLabels has been implemented. This prevents overlaps and inconsistencies, improving the accuracy of timing. Furthermore, the startMs of the monoLabel at the end of a phrase has been changed to sentenceDurMs - tailMs to ensure that the timing of the entire phrase is accurately reflected.
…ote determination conditions In the GetSymbols method of HTSLabelPhonemizer.cs, note.lyric is now passed directly to g2p.Query without converting it to lowercase, making it possible to distinguish between uppercase and lowercase letters. In HTSLabelRenderer.cs, an overload that accepts a single symbol has been added to the makeHtsNote method. In the IsSyllableVowelExtensionNote method, the determination condition for vowel extension notes has been changed to only +~ or +*, making the determination more strict.
This now allows for the regeneration of parameters when the pitch is changed.
Modified path strings within ArgParam to be enclosed in double quotes. This fix ensures correct processing even when paths contain spaces. Updated multiple locations within the SendScore method of NeutrinoLabelPhonemizer.cs and the Render method of NeutrinoRenderer.cs.
The monoLabel struct has been changed to public, and a customizable virtual method CustomMonoLabel has been added. This allows subclasses to override the processing logic for monoLabels_. Additionally, in NeutrinoRenderer.cs, CustomMonoLabel is overridden to implement a process that rounds label timings to 10ms increments when singerVersion is "v2.7".
…n for Version 3 to 3.2.
In HTSLabelPhonemizer.cs, the dictionary loading process within the SetSinger method has been moved before the g2p loading process to clarify dependencies. The following changes were made to HtsLabelPhonemizerTest.cs: - Added the DictionaryLoadedBeforeG2p property to the DummyHtsLabelPhonemizer class. - Set tablePath to "oto.ini" in the constructor. - Overrode the LoadG2p method to verify the dictionary loading state. - Added a new test, DictionaryIsLoadedBeforeG2p, to ensure that the dictionary is loaded before g2p.
…h the processing of the new Shuri HTSLabelPhonemizer.
Contributor
Author
|
这个修改仍然不太稳定，需要再修改修改 |
Contributor
Author
最好不要同时渲染多个轨道，以及多个后端版本
在渲染过程，最好不要播放预览
最好在刚启动程序时选择NEUTRINO渲染
|
Contributor
|
As pull request #2136 has not been merged, please submit the fix to my repository. |
Contributor
Author
|
sorry, I will do it |
Contributor
Author
|
Sorry,The current version has way too many issues.I'll submit it to your repo after I finish polishing it.I messed up |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on PR #2136 and #2132 to improve NEUTRINO engine multi-version support, adding compatibility with v2.3.x and v3.1.x, and fixing various issues with command parameters, encoding, and paths.
Thank rokujyushi very much.
Fixed Issues
Overly Strict Version Checks
The original code only supported v2.7.x and v3.0.x, explicitly excluding v3.1.x, which prevented v3.1.x voicebanks from working.
Fix: Relaxed version checks to v2.x and v3.x general support, covering all major versions including v2.3.x, v2.7.x, v3.0.x, and v3.1.x.
Incorrect v3 Command Parameters
The original v3 branch used the wrong pitch shift parameter -k and was missing the thread count parameter.
Fix: Changed pitch shift parameter to -f, added -n thread count parameter, aligned with official v3.1.x command-line format.
Incorrect v2 Command Parameters
The original v2 branch used the wrong thread count parameter -p and included an unnecessary -a parameter.
Fix: Changed thread count parameter to -o, removed unnecessary -a parameter, aligned with official v2.3.x script format.
Dictionary File Encoding Error
The original code loaded NEUTRINO engine dictionary files using the voicebank's TextFileEncoding, but files in the dic directory are UTF-8 encoded regardless of voicebank encoding.
Fix: Unified dictionary loading to use Encoding.UTF8, preventing dictionary loading errors with Japanese voicebanks (shift_jis encoded).
Incorrect Model Path
The original code used singer.Location as the model directory, but NEUTRINO voicebanks usually have info.toml in a subdirectory (e.g., KIRITAN/), causing model files to not be found.
Fix: Added modelDir field to store the actual model directory path where info.toml is located.
Subdirectory Voicebank Recognition
The original code only looked for info.toml in the voicebank root directory, not supporting voicebank structures with models stored in subdirectories.
Fix: Added subdirectory traversal to find info.toml, supporting the common "voicebank_name/model_name/info.toml" structure.
info.toml Version Parsing
The original code only read the version from acoustic.version, but v3 format info.toml has version at the root level.
Fix: Root-level version takes priority, with acoustic.version as fallback, compatible with both v2 and v3 info.toml formats.
G2P Null Safety
The original code directly accessed dictionary keys, which would throw KeyNotFoundException and crash when some keys were missing in certain voicebanks.
Fix: Added null checks in LoadG2p and PhraseAdjustments, using GetValueOrDefault for safe access.
Engine Directory Lookup Optimization
The original code only looked for NEUTRINO_v27 and NEUTRINO_v3 directory names, not supporting generic naming.
Fix: Added lookup priority for generic directory names like NEUTRINO_v2, making it easier for users to manage different engine versions.
Files Changed
OpenUtau.Core/Neutrino/NeutrinoSinger.cs
OpenUtau.Core/Neutrino/NeutrinoLabelPhonemizer.cs
OpenUtau.Core/Neutrino/NeutrinoRenderer.cs
OpenUtau.Core/Classic/VoicebankInstaller.cs
OpenUtau/ViewModels/SingerSetupViewModel.cs
Compatibility
All changes are fully backward compatible and do not affect existing v2.7.x and v3.0.x voicebanks. The fixed code automatically detects the voicebank version and selects the corresponding command-line parameters.
Current Issues
Rendering is slow when multiple tracks with different voicebanks and different engine versions are rendered simultaneously.
Using multiple similar NEUTRINO backends simultaneously may cause bugs, especially with server-based versions.
There is currently no feature to specify which backend to use for a voicebank. As a result, multi-backend compatibility may be poor. It is recommended to manually rename engine directories and match voicebank versions with backend versions to avoid unexpected issues.
The phonemizer currently only supports basic kana and does not handle cases like めい correctly. The phoneme recognition for ん may also be incorrect.
There is no dedicated settings page for configuring NEUTRINO-related parameters.
I will demonstrate the installation process, usage, and the results later.