Skip to content

piper: Exclude re-downloadable catalog voices from backups - #4829

Closed
tronikos wants to merge 1 commit into
home-assistant:masterfrom
tronikos:piper-exclude-catalog-voices-from-backups
Closed

tronikos wants to merge 1 commit into
home-assistant:masterfrom
tronikos:piper-exclude-catalog-voices-from-backups

Conversation

@tronikos

@tronikos tronikos commented Sep 21, 2026 •

Copy link
Copy Markdown
Member

2.5.2 stopped excluding "*.onnx" from backups, because the new web interface can upload voices and an uploaded voice exists nowhere else. That is right for uploads, but it also put catalog voices back into every backup: a single model is 20-110 MB, and the add-on downloads it again from HuggingFace whenever it is missing. An install with two voices adds over 130 MB to each backup for files that a restore would fetch anyway.

wyoming-piper writes downloads and uploads to the same --download-dir, so the two cannot be separated by configuration. Instead, sort them at startup: a voice whose md5 matches what the catalog publishes for that name is moved to /data/catalog_voices, which config.yaml excludes, and that directory is passed as another --data-dir so the voice keeps working and is not downloaded again. After a restore it is simply absent, which is the state of a fresh install, and it is downloaded on demand.

Matching on the hash rather than the name keeps a custom voice that reuses a catalog name - a fine-tune exported as en_US-lessac-medium.onnx, say - in /data and in backups. Names are checked before hashing, so a custom voice under a custom name is never read.

--download-dir stays /data, so a voice downloaded while the add-on runs is moved only at the next start. That way the failure mode is one larger backup rather than an upload that is missing from one.

Summary by CodeRabbit

  • Bug Fixes

    • Catalog voice models are again excluded from backups after being moved to a separate storage location.
    • Uploaded and customized voices remain included in backups, even when they use catalog voice names.
    • Catalog voices are relocated only when they match the official catalog files, preventing accidental exclusion of modified voices.
    • Required catalog voices are still available for restoration and can be downloaded again when needed.
  • Documentation

    • Updated backup behavior documentation and changelog for version 2.5.3.

@coderabbitai

coderabbitai Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: c67a28c0-cd17-4668-a634-aec24b121806

📥 Commits

Reviewing files that changed from the base of the PR and between ceb8001 and b16bb00.

📒 Files selected for processing (5)
  • piper/CHANGELOG.md
  • piper/DOCS.md
  • piper/config.yaml
  • piper/rootfs/etc/s6-overlay/s6-rc.d/piper/run
  • piper/rootfs/usr/local/bin/sort_catalog_voices.py

Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.


📝 Walkthrough

Walkthrough

Changes

Catalog voice backup handling

Layer / File(s) Summary
Catalog voice identification and relocation
piper/rootfs/usr/local/bin/sort_catalog_voices.py
The new script matches catalog voice names and md5 digests. It moves matching .onnx and .json pairs into /data/catalog_voices.
Startup sorting and voice serving
piper/rootfs/etc/s6-overlay/s6-rc.d/piper/run
Piper runs the sorter before startup, logs its output, and serves the catalog directory through an additional --data-dir argument.
Backup exclusion and release documentation
piper/config.yaml, piper/DOCS.md, piper/CHANGELOG.md
The catalog directory is excluded from backups. The add-on version and documentation describe catalog and uploaded voice handling.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant PiperRun
  participant SortCatalogVoices
  participant WyomingPiper
  PiperRun->>SortCatalogVoices: Sort matching voices before startup
  SortCatalogVoices-->>PiperRun: Return moved voice summary
  PiperRun->>WyomingPiper: Start with /data and /data/catalog_voices
Loading

Suggested reviewers: synesthesiam

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 1 files. (4 skipped: 4… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the primary change: excluding re-downloadable Piper catalog voices from backups.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 1 files. (4 skipped: 4 unsupported.)

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

2.5.2 stopped excluding "*.onnx" from backups, because the new web
interface can upload voices and an uploaded voice exists nowhere else.
That is right for uploads, but it also put catalog voices back into every
backup: a single model is 20-110 MB, and the add-on downloads it again
from HuggingFace whenever it is missing. An install with two voices adds
over 130 MB to each backup for files that a restore would fetch anyway.

wyoming-piper writes downloads and uploads to the same --download-dir, so
the two cannot be separated by configuration. Instead, sort them at
startup: a voice whose md5 matches what the catalog publishes for that
name is moved to /data/catalog_voices, which config.yaml excludes, and
that directory is passed as another --data-dir so the voice keeps working
and is not downloaded again. After a restore it is simply absent, which is
the state of a fresh install, and it is downloaded on demand.

Matching on the hash rather than the name keeps a custom voice that reuses
a catalog name - a fine-tune exported as en_US-lessac-medium.onnx, say -
in /data and in backups. Names are checked before hashing, so a custom
voice under a custom name is never read.

--download-dir stays /data, so a voice downloaded while the add-on runs is
moved only at the next start. That way the failure mode is one larger
backup rather than an upload that is missing from one.
@tronikos
tronikos force-pushed the piper-exclude-catalog-voices-from-backups branch from b16bb00 to e362b92 Compare September 28, 2026 01:06
@sairon

sairon commented Nov 4, 2026

Copy link
Copy Markdown
Member

Hi @tronikos, the code for the app your PR was targeting was migrated to home-assistant/app-piper. Please submit a new PR there.

Please leave out the version bump and the CHANGELOG.md entry. The version now comes from the GitHub release, and the PR title goes into the release notes. Thank you!

@tronikos

tronikos commented Nov 9, 2026

Copy link
Copy Markdown
Member Author

Thanks @sairon! I've submitted a new PR at home-assistant/app-piper#4 (also updated so catalog voices downloaded at runtime go straight to /data/catalog_voices rather than waiting for the next restart).

One question: home-assistant/app-piper currently shows 0 watchers and didn't auto-assign a reviewer (CODEOWNERS isn't set up there). How are issues and PRs in these new split app repos monitored so they don't go unnoticed?

@tronikos
tronikos deleted the piper-exclude-catalog-voices-from-backups branch November 9, 2026 02:51
@sairon

sairon commented Nov 9, 2026

Copy link
Copy Markdown
Member

@tronikos Apparently GH doesn't count people with "Custom" watch settings as a watchers, because I do have monitoring for PRs enabled there and it still shows 0 :) Otherwise it's up those who feel responsible for the app, ideally @synesthesiam should subscribe there (or create CODEOWNERS, which was not defined even in this repo) for the voice apps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants