Qubes 4.3 backup restore randomly fails to decrypt different chunks with "scrypt: Passphrase is incorrect"

[guillaumedebruyne]

Qubes OS release

Qubes OS 4.3

Relevant package versions:

qubes-core-dom0-4.3.51-1.fc41.noarch
qubes-core-admin-client-4.3.34-1.fc41.noarch
qubes-core-dom0-linux-4.3.22-1.fc41.x86_64
scrypt-1.3.3-1.fc41.x86_64
kernel: 6.18.46-1.qubes.fc41.x86_64
CPU: AMD Ryzen 5 1400

The backup uses format version 4:

version=4
encrypted=True
compressed=True
compression-filter=gzip
hmac-algorithm=scrypt

Brief summary

An encrypted Qubes backup is successfully created, but verification/restoration intermittently fails with:

scrypt: Passphrase is incorrect

The important point is that the failing encrypted chunk changes between verification attempts on the same backup archive using the same passphrase.

Observed failures have included:

qubes.xml.000.enc
vm17/root.img.000.enc
vm17/root.img.004.enc
vm17/root.img.006.enc

A chunk reported as failing by qvm-backup-restore can subsequently be successfully decrypted manually in dom0.

I have not identified the root cause.

Steps to reproduce

1. Create an encrypted Qubes backup stored in a dedicated AppVM (backup).

2. Verify the backup using either the GUI integrity verification or:

qvm-backup-restore -v --verify-only \
  --passphrase-file /tmp/qubes-backup-pass \
  --dest-vm backup \
  backup-test/qubes-backup-2026-09-03T092223

3. The verification fails with, for example:

Getting new file: vm17/root.img.000.enc
...
qvm-backup-restore: error: failed to decrypt .../vm17/root.img.000.enc:
b'scrypt: Passphrase is incorrect\n'

4. Run the same verification again, with exactly the same archive and passphrase.

The failure can occur on a different encrypted chunk. Previous runs failed on qubes.xml.000.enc, vm17/root.img.004.enc, and vm17/root.img.006.enc.

A simple ASCII-only passphrase was also tested to rule out special-character or encoding issues.

Expected behavior

A successfully created backup should consistently pass --verify-only when using the correct passphrase.

Actual behavior

Verification intermittently fails on apparently varying encrypted chunks with:

scrypt: Passphrase is incorrect

The failure does not appear to be deterministic corruption of one specific chunk.

Additional information

I performed several tests to isolate the problem.

Archive integrity

qubes.xml.000.enc was independently extracted three times from the same backup. All three copies had exactly the same SHA-256:

10fa88a9e5f32d710b3659e6bc47c7a5cf74da64a2583b79e52b2f83ece556e9

The chunk was then transferred from the backup AppVM to dom0 using qvm-run --pass-io. Its SHA-256 remained identical.

The same qubes.xml.000.enc was successfully decrypted several times directly in dom0 using the Qubes v4-derived scrypt passphrase, producing identical decrypted outputs.

Testing a chunk immediately after qvm-backup-restore rejected it

One verification run specifically failed on:

vm17/root.img.000.enc

I manually extracted that exact chunk from the same backup.

Size:

101M

SHA-256:

5944fca8298d5bb3408f22574efa7afa18174899a23f4096318042b08c49daba

After transferring it to dom0, the SHA-256 remained identical.

The exact chunk that qvm-backup-restore had just rejected was then successfully decrypted manually in dom0 with:

scrypt dec -f /tmp/root000.enc /tmp/root000.dec

using the appropriate Qubes v4-derived passphrase.

Testing Qubes' own launch_scrypt()

I also tested the function used by Qubes directly:

from qubesadmin.backup.restore import launch_scrypt

p = launch_scrypt(
    'dec',
    '/tmp/root000.enc',
    '/tmp/root-test.dec',
    '<Qubes-v4-derived-passphrase>'
)

p.communicate()
p.returncode

Result:

(None, b'')
0

Therefore launch_scrypt() successfully decrypted the exact encrypted chunk that had failed during the complete qvm-backup-restore verification.

Other tests

A direct 256 MiB scrypt enc / scrypt dec round-trip in dom0 succeeded and cmp confirmed identical data.

Memtest86+ was run for approximately 13 hours:

4/4 passes
0 errors
PASS

The problem was also reproduced with the backup stored in the backup AppVM private volume on the system NVMe, rather than on the separate SATA backup disk.

So far, the individually tested path appears stable:

backup archive
 -> tar extraction
 -> transfer from backup AppVM to dom0
 -> scrypt
 -> decrypted data

The failure has only been reproduced when using the complete Qubes backup restore/verification pipeline.

I have not determined whether the issue is in the retrieval/unpacking pipeline, synchronization between its components, or elsewhere.

I can perform additional diagnostic tests or provide additional logs if needed.

[crass]

@guillaumedebruyne How big is the volume that is failing to restore? If its large it could be #5393, which needs to be verified on newer Qubes versions (last verified in 4.1).

[guillaumedebruyne]

@guillaumedebruyne How big is the volume that is failing to restore? If its large it could be #5393, which needs to be verified on newer Qubes versions (last verified in 4.1).

@crass The failing VM is actually quite small:

root volume: 20 GiB provisioned, ~7.8 GiB used
private volume: 2 GiB provisioned, ~618 MiB used

So it doesn't appear to be a particularly large-volume case like #5393.

Also, repeated restore/verify attempts of the same backup failed on different encrypted chunks (qubes.xml.000.enc and different root.img.00x.enc chunks), while the individual failing chunks could be extracted and decrypted successfully when tested separately.

I'm happy to run additional tests if you'd like to check whether this could nevertheless be related to #5393 on Qubes 4.3.

[crass]

@guillaumedebruyne How big is the volume that is failing to restore? If its large it could be #5393, which needs to be verified on newer Qubes versions (last verified in 4.1).

@crass The failing VM is actually quite small:

root volume: 20 GiB provisioned, ~7.8 GiB used private volume: 2 GiB provisioned, ~618 MiB used

So it doesn't appear to be a particularly large-volume case like #5393.

I suspected as much. But I forgot to mention that really I wanted you to check that the backup size looked correct. In #5393, it looks like the backup is only partial. But reading again, it looks like you're able to independently verify that scrypt decrypts the the full backup contents, correct?

Also, repeated restore/verify attempts of the same backup failed on different encrypted chunks (qubes.xml.000.enc and different root.img.00x.enc chunks),

Its not clear to me that this isn't the case with #5393, but seems likely true also, although there I think it was the last chunk of a partial backup that always failed.

while the individual failing chunks could be extracted and decrypted successfully when tested separately.

This also might be the case #5393, but I had initially assumed not.

I'm happy to run additional tests if you'd like to check whether this could nevertheless be related to #5393 on Qubes 4.3.

Not necessarily related to #5393, but it looks like you should be able to run the verify-only command within the qube with the backups. Have you tested that?