ObjectStorageProvider: add write back support

[Deleted user (ghost)]

Description

When we succesfully restore a binary package from the cloud storage we should cache it locally to avoid downloading it again.

This feature works at these conditions:

• we have configured binary caches on the file system with write permissions (default,write, default,readwrite, files,...,write, files,...,readwrite)
• we have configured binary caches on the cloud storage with read permissions (x-aws,...,read, x-aws,...,readwrite, x-cos,...,read, x-cos,...,readwrite, x-gcs,...,read, x-gcs,...,readwrite)

To cache it, it just moves the restored zip to the local cache directory instead of deleting it.

Design

flowchart TD
    subgraph Dev["Developer Machine"]
        A[Checked-out C++ Code]
        B["VCPKG Build Process"]
        C["Local Binary Cache (.zip files)"]
    end

    subgraph AWS["Build of the external component (CMake)"]
        D[GitHub checkout]
        E[CMake config]
        F[CMake build]
    end

    A --> B
    B -->|Check| C
    C -->|Hit| BDone[Use from Local Cache]
    C -->|Miss| FCheck[Check S3 Cache]
    FCheck -->|Hit| FDownload[Download from S3]
    FDownload --> B
    FDownload -.-> CUpdate
    FCheck -->|Miss| D
    D --> E
    E --> F
    F --> CUpdate["Update Local Cache"]
    CUpdate --> B
    BDone --> B

    style BDone fill:#ccffcc,stroke:#333
    style FDownload fill:#ccffcc,stroke:#333
    style CUpdate fill:#ccffcc,stroke:#333
    linkStyle 6 stroke:red,stroke-width:3px,stroke-dasharray: 5, 5

Loading

Rational

We have our developers compiling with their machines inside our offices around the world and they also often work from home. To speed-up their job we store precompiled externals in archives on AWS S3.

For externals we have an internal solution that uses local hard-drive as first layer cache and S3 as second layer. When something is not found localy, it is downloaded and then left on the local hard-drive for a while.

A part of the compilation time optimization, this is also helping a lot into keeping egress costs under control:

This is an example of a month where costs were higher than usual because for some few days the feature was not working after an upgrade of our tool.

This small PR is implementing with fews lines of code a similar strategy to VCPKG.

[Osyotr (Osyotr)]

Related: #1406

[Deleted user (ghost)]

There are 3 PRs trying to implement write back support to this tool.

This is a must-have feature in many circumstances.

What distinguish this solution is the following:

• it is in a mergeable status.
• it is very small.
• it doesn't require much testing (it is already working very nicely on our side).
• but it is limited to Cloud Storage backends (which also reduces the things to be tested).

What about taking in this now and then looking forward for a more robust solution based on the other proposals in the mid-long term?

At the end this is the most efficient solution possible on the backends we are targeting:

• just move/rename the packages on local binary caches after aving used them successfully.

No extra copies, just one fast, atomic I/O operation.

[Deleted user (ghost)]

Billy O'Neal (@BillyONeal) what do you think about my last comment?

[Steve Brain (EvilMcStevil)]

This looks like a very positive pr, that just does local caching where it can, why is it not accepted? I understand all of the issues about the general write back support problem, but this fixes 90% of my use case, such that I have a script that does this manually already, which I run regularly, but it pulls in way more than i need or want.

[Deleted user (ghost)]

• I have just resolved all merge conflicts.
• Please consider this as an intermediate solution, it would reduce EGRES costs and download time to many people. On our site it has been pretty succesful so far.

[Daniel Gehriger (dgehriger)]

Billy O'Neal (@BillyONeal) : would you be able to give some feedback on this PR, and why it isn't being considered?

[Nick D'Ademo (nickdademo)]

Billy O'Neal (@BillyONeal) : would you be able to give some feedback on this PR, and why it isn't being considered?

This looks super useful and is a must-have IMO. Looking forward to seeing this merged soon... 🤞

[Nick D'Ademo (nickdademo)]

@rressi-at-globus
out of curiosity, how do you provide vcpkg.exe with this patch to your team of developers? by using a fork of the vcpkg repo with the patch applied? (assuming vcpkg repo is a submodule in your project)

[Deleted user (ghost)]

• This solution is obsolete now as the baseline code changed totally now.

[Michael Leuchtenburg (dyfrgi)]

@rressi-at-globus is there an equivalent solution in place or do we need to rewrite it again?

[Billy O'Neal (BillyONeal)]

@rressi-at-globus is there an equivalent solution in place or do we need to rewrite it again?

I don't believe we have an equivalent right now. I had to look at this area recently and am trying to include an equivalent feature but no promises.