Paths.json should store placeholder offset and file permissions (executable)

[wolfv]

Checklist

• I added a descriptive title
• I searched open requests and couldn't find a duplicate

What is the idea?

To speed up installation (linking) we should store the exact offets of prefix_placehodler locations (so we don't need to search for the string again) and the permissions (so we do not need to call stat on the original file).

That shoudl give a bit of a speed boost.

When offset data is present, we should use a "faster" replacement function. Just note that replacement in binaries is not as trivial as text files due to the null terminator handling.

It should be implemented in a backwards-compatible way so that conda can still read, and install these packages.

Why is this needed?

No response

What should happen?

No response

Additional Context

No response

[baszalmstra]

• Should we just store an array of byte offsets?
• Should we only store whether the file should be executable? I dont think any other missions matter right?
• Should we create a CEP for this as well?

[wolfv]

I think yes to all three, although I am most unsure about whether other permissions might matter. BUt I think read and read-write shoudl anyways be controlled by user settings, maybe?

[baszalmstra]

THis would require a version bump of the paths.json right? Because only then can we assume that the absense of the executable permissions field means its not executable.

[Dagmar-Dinjens]

Is this still open to be picked up?
I have been working on something similar from the standpoint of RattlerFS (#1182) and would be able to implement this in the complete structure.

[wolfv]

Hi @Dagmar-Dinjens - I think it would be great to pick this up, yes!

We could start with an experimental support in our paths.json struct (optional fields that are not serialized if not present).

Or we add a PathsV2 struct and increase the schema field and parse either or... and then also add supprot for the V2 struct where needed. The original PathsJson could then become an enum (that is "internally tagged" in serde-lingo.

[Dagmar-Dinjens]

These are good suggestion I'll go and look at them and decide.

[iamthebot]

This would be super useful and happy to work on this if you don't have time @Dagmar-Dinjens. We actually pre-compute offsets in our internal serialization engine for this very reason. But having this in the paths.json makes this way easier.

[Dagmar-Dinjens]

The PR #2032 implements the first step of this. Other things to work on would be the rattler-build side
prefix-dev/rattler-build#528
and during downloading to the cache upgrading the paths.json