feat(repositories): add S3 storage class support

[kamarkiewicz]

Hi! Thanks for the great project.

This is my attempt to address #724 as I needed S3 storage class support myself as a mirror.
I have successfully tested the creation of the new repository with GLACIER (pointing to french Scaleway S3 Object Storage), set this to mirror my main backup and triggered a new backup.

I'd really appreciate a review.

Screenshots:

[CLAassistant]

All committers have signed the CLA.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: a8501d03-5427-4267-b68e-b5d65d2e3617

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[nicotsx]

Thank you @kamarkiewicz for the contribution. However, I have some concerns regarding the usage of this feature. I don't know Scaleway's glacier but for example with Amazon Glacier this wouldn't end up well.

When you upload stuff to glacier you usually have a commitment to store the data for X days. Zerobyte would fail to execute the usual pruning operations and the delete snapshot operations. For restoring we also have an issue. These storage classes usually need you to ask for retrieval and then wait X hours for the data to be available. Restic would fail internally since it cannot read the data and you wouldn't be able to restore at all.

Also restic still fetches data during backups to read parent snapshots metadata which will cause the same problem as above during backups and even if scaleway allows you to fetch directly it has probably a very high egress cost (it's usually the tradeoff with these cheaper storage options) that might cause a bad surprise to the user that thought they were only appending data.

I don't want to gatekeep this feature but I'm also not sure I want to broadly open this low-level setting that might confuse users into thinking they can simply use any storage class and that it'll work.

If scaleway works differently, I'd prefer to have a dedicated Scaleway S3 option in the dropdown with the supported (and tested) storage classes available in a dropdown. Like ONEZONE_IA or GLACIER if they work well with restic

Were you able to restore files as well with Glacier storage class?

[kamarkiewicz]

I find your concerns fully valid. Definitely, the cold storage is not for non-technical users due to complex pricing and restoration requirements. I think the usual hot storage cycle of backup/check/prune/restore does not apply there.

The restoration process would be manual, to use CLI to Restore the Glacier-class data/ objects to Standard Multi-AZ first. The Scaleway docs advise to use AWS CLI for doing so: https://www.scaleway.com/en/docs/object-storage/how-to/restore-an-object-from-glacier/#how-to-restore-all-objects-in-a-bucket

Restic has experimental support for cold storage https://restic.readthedocs.io/en/stable/faq.html#are-cold-storages-supported

For advanced users, who want a cold storage just-in-case mirror only of the actual Zerobyte backup, what is your advice?

[kamarkiewicz]

I would suggest adding the ⁠EXPERIMENTAL=s3.storage-class⁠ env var, so this option can be hidden by default, yet remain accessible at one's own risk. What are your thoughts on this approach?

[nicotsx]

This is a fair tradeoff. But I would still not be very comfortable with it. Yes being hidden behind a flag makes it less likely a user will mistakenly use this class with wrong expectations but at the same time it changes the repository contract we keep as developers. It changes from "all repos are operated the same way" to "some repo might not be compatible with all features". This is the kind of maintenance tax that later bites us back because for each new feature (and existing ones) we have to keep remembering "what if the repo is of class X" and every part of the app starts to branch into multiple cases. If we provide the option, having the app simply break in some workflows for these users is not a good experience imo.

To give you a concrete example, let's say I'm planning to add a dedicated repository feature to automatically check data integrity (described here #273) I would have to think about how do we handle gracefully the error if it's a glacier-type repo (because you can't perform the check). If we do not model that correctly, users will likely report bugs where the real issue is that the repository is using an exotic storage class

I think the escape hatch is not the right approach. I would be more open to either:

• Provider-specific storage classes that still work with all normal restic operations. With a dedicated form in the app for it
• A dedicated cold/append-only repository mode (also with a dedicated form and not s3 with one extra field), if restic and the provider flow can support it well enough. We'd have to explicitly disable features like restore or prune etc. for this kind of repo or create new patterns with different states (restore requested, waiting for provider, etc). So this would be a bigger change

For now I think this kind of cold mirror is better done outside Zerobyte or later as a first-class cold repository mode.