`meson dist` results in invalid tar archives on Windows when the repository contains symlinks

[dnicolodi]

Mostly reporting this so that other do not have to rediscover it.

With Python 3.15, on Windows with support for symlinks enabled, when the git repository of the project to be packed contains symbolic links, tar archives produced by meson dist are invalid: the symlink target paths use the \ path separator rather than the / path separator.

The reason for this is that meson dist executes git archive, executes the dist scripts, and the repacks the resulting tar archive. AFAICT git dist produces correct tar archives. Python 3.15 has been patched to substitute / path separators in symlink targets with \, but the converse normalization has not been implemented when creating tar archives. This results in the symlink having Windows path separators rather than the required POSIX path separators.

The bug is not visible with Python 3.14 and earlier because the lack of normalization on tarfile.TarFile.extract() created symlinks with the wrong destination path to be usable on Windows, but that resulted in the right byte sequence when read back to repack the archive. Reproducing this also requires a Windows system with symlink support enabled (as are the GitHub Actions Windows runners).

I noticed this when testing meson-python with Python 3.15 mesonbuild/meson-python#855

Here is the CPython PR that broke the thing python/cpython#138309

I think this is a bug in Pyhton and I have reported it python/cpython#151669 with a fix python/cpython#151671 However, it seems an uphill battle to have this fixed python/cpython#138309 (comment)

I currently don't see a way to work around this in Meson, without restructuring the code at the base of meson dist significantly, monkey patching shutil.make_archive() or tarfile.TarFile.add(), or similar heavy handed changes.

[eli-schwartz]

If I understand correctly, using tarfile on windows to create an archive has always been wrong by default, but previously when taking a git generated tar, it was possible to roundtrip (unpack and repack) while preserving the target location set by git archive. And the roundtripping now causes a problem because unpacking will "destroy" information?

[dnicolodi]

@eli-schwartz Correct.

[dnicolodi]

One possible solution for Meson would be to register a tarfile.extract_filter that undoes the normalization on Windows on Python 3.15 and later, to restore the old behavior (which is wrong if you want to use the symlinks, but it guarantees correct round-tripping of the archive content). tarfile does not have a global registry for TarInfo filters for the creating of an archive, and shtutil.make_archive() does not allow to pass one.

However, this still break if the dist script to be executed on the extracted content of the archive produced by git archive depend on being able to resolve the symlink destinations. But this was broken before Python 3.15 and no one complained (so far).

[bonzini]

@dnicolodi's patch was committed by upstream CPython