Borg backup of all apps failing at `hook_exec(`

[eauchat]

Since last yunohost system upgrade (passing to bookworm), I'm having this strange thing that borg backup keep failing every night with logs ending with the following:

  - 2025-01-23 02:31:58,975: DEBUG - + /var/www/borg/venv/bin/borg create --stats '::auto_<appName>-{now}' .
  - 2025-01-23 02:32:05,339: DEBUG - + retcode=2
  - 2025-01-23 02:32:05,339: DEBUG - + popd
  - 2025-01-23 02:32:05,339: DEBUG - /etc/yunohost/hooks.d/backup_method
  - 2025-01-23 02:32:05,339: DEBUG - + [[ 2 == 1 ]]
  - 2025-01-23 02:32:05,339: DEBUG - + [[ 2 != 0 ]]
  - 2025-01-23 02:32:05,339: DEBUG - + exit 2
  - 2025-01-23 02:32:05,440: ERROR - Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app
  - Traceback (most recent call last):
  -   File "/usr/lib/python3/dist-packages/yunohost/hook.py", line 301, in hook_callback
  -     hook_return = hook_exec(
  -                   ^^^^^^^^^^
  -   File "/usr/lib/python3/dist-packages/yunohost/hook.py", line 421, in hook_exec
  -     raise YunohostError("hook_exec_failed", path=path)
  - yunohost.utils.error.YunohostError: Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app

Each app's backup fails with this samge message (including yunohost system config and data).
I can't really figure out the meaning of the problem, those logs seem very vague to me.

Yunohost is the latest version, borg_ynh as well.
And I tried running yunohost app upgrade borg --force with no effect.

The strangest thing to me, is that I have in the same server another borg instance which sends backup of the same content to another server, and this is working fine.
On top of that, another yunohost server is sending without issues backups to the target borg server for which this one is failing.
So it seems the problem is really locate in this specific borg_ynh instance.

I had a look at the /etc/yunohost/hooks.d/backup_method/05-borg_app script and it's content and permissions are analogous to /etc/yunohost/hooks.d/backup_method/05-borg__2_app (apart from target server obviously). So nothing seems wrong in their either.

If anyone has a clue on what could be happening or how I may investigate the issue, it's very welcome :)

[orhtej2]

There's a more detailed log hidden under /var/log/borg, names follow yyMMdd_hhmm.{err,log} pattern (so today five past midnight would produce 250123_0005.err and 250123_0005.log).
Perhaps there are more clues there? Could you share content of these (if no PII is inside)?

[eauchat]

Thanks for your answer @orhtej2, unfortunately the logs in /var/log/borg aren't very satisfying.
The .log file contains:

INFO Collecting files to be backed up for borg…
INFO Declaring files to be backed up...
INFO Backup script completed for borg. (YunoHost will then actually copy those files to the archive).
INFO Creating a backup archive from the collected files…
INFO The archive will contain about 131.8KB of data.
INFO The operation 'Create a backup archive' could not be completed. Please share the full log of this operation using the command 'yunohost log share 20250128-013111-backup_create' to get help

in loop for every app, system data and config, and nothing more.
The .err file contains the following:

ERROR Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app
ERROR Custom backup method could not get past the 'backup' step
ERROR Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app
ERROR Custom backup method could not get past the 'backup' step
ERROR Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app
ERROR Custom backup method could not get past the 'backup' step
ERROR Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app
ERROR Custom backup method could not get past the 'backup' step
ERROR Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app
ERROR Custom backup method could not get past the 'backup' step
ERROR Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app
ERROR Custom backup method could not get past the 'backup' step
ERROR Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app
ERROR Custom backup method could not get past the 'backup' step
ERROR Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app
ERROR Custom backup method could not get past the 'backup' step

[orhtej2]

This actually seems to be a problem with backup script rather than borg itself. What are you trying to back up again?

[eauchat]

Hmm what do you mean with "backup script"? Do you mean the borg_ynh backup script, or yunohost backup mechanism?

I'm simply backing up yunohost system configs, data and yunohost installed apps. What confuses me is that I have a second borg_ynh instance on the same server backing up the same contents to another remote, and that one works fine.

[eauchat]

Ok I realized there were more info in borg general log file: /var/log/borg/borg.log, of which existence I had forgotten.
The messages there were pretty clear: "The storage quota has been exceeded".
I didn't figure out this quota issue earlier because checking borg info on the repo, it would display less than the quota while the real size would actually be more, as explained in #190.
Here is a PR to fix this emailing issue: #193.