|
| 1 | +# Setting up a cache server (Harbor + Redis) |
| 2 | + |
| 3 | +The distributed cache has two server-side halves, which fit comfortably on a |
| 4 | +single small host: |
| 5 | + |
| 6 | +- **An OCI registry** stores the `cache.url: oci://...` entries as |
| 7 | + chunk-deduplicated artifacts. This guide uses [Harbor](https://goharbor.io): |
| 8 | + unlike a plain CNCF `distribution` registry it has per-project access |
| 9 | + control, **tag retention policies** (the only expiry mechanism for the |
| 10 | + `oci://` backend, which has no TTL) and scheduled garbage collection. Any |
| 11 | + OCI registry works the same from the client's point of view. |
| 12 | +- **A Redis instance** serves the `cache.lock: redis://...` distributed locks |
| 13 | + (and, optionally, `redis://` cache entries for small blobs). |
| 14 | + |
| 15 | +Sizing: Harbor wants 4 GB of RAM and 2 vCPUs minimum, plus disk for the cache |
| 16 | +volume (chunks are zstd-compressed and deduplicated, so plan for the size of |
| 17 | +one full cache generation, not one per build). The lock Redis is capped at |
| 18 | +64 MB and is negligible. |
| 19 | + |
| 20 | +## Harbor: OCI cache storage |
| 21 | + |
| 22 | +### Install |
| 23 | + |
| 24 | +Harbor ships as a docker-compose bundle. On a host with docker and the |
| 25 | +compose plugin: |
| 26 | + |
| 27 | +```sh |
| 28 | +curl -LO https://github.com/goharbor/harbor/releases/download/v2.13.0/harbor-offline-installer-v2.13.0.tgz |
| 29 | +tar xzf harbor-offline-installer-v2.13.0.tgz && cd harbor |
| 30 | +cp harbor.yml.tmpl harbor.yml |
| 31 | +``` |
| 32 | + |
| 33 | +Edit `harbor.yml` — the relevant keys: |
| 34 | + |
| 35 | +```yaml |
| 36 | +hostname: harbor.example.com # or the host IP |
| 37 | +https: |
| 38 | + port: 443 |
| 39 | + certificate: /etc/harbor/certs/harbor.crt |
| 40 | + private_key: /etc/harbor/certs/harbor.key |
| 41 | +harbor_admin_password: <initial admin password> |
| 42 | +data_volume: /srv/harbor # blobs end up here |
| 43 | +``` |
| 44 | +
|
| 45 | +With a self-signed or private-CA certificate, keep the CA file around: every |
| 46 | +client needs it (the `?ca=` URL parameter or `TASK_CACHE_OCI_CA`). |
| 47 | + |
| 48 | +```sh |
| 49 | +sudo ./install.sh |
| 50 | +``` |
| 51 | + |
| 52 | +`install.sh` generates the compose file and starts Harbor; it is restarted on |
| 53 | +boot through its own `docker-compose` restart policies. |
| 54 | + |
| 55 | +### Project and robot account |
| 56 | + |
| 57 | +In the Harbor UI (or via the API): |
| 58 | + |
| 59 | +1. Create a **private project** named `task-cache`. |
| 60 | +2. In the project, create a **robot account** (e.g. `ci`) with `pull` and |
| 61 | + `push` permission on repositories. The full username Harbor generates is |
| 62 | + `robot$task-cache+ci`; the secret is shown once. |
| 63 | + |
| 64 | +The robot credentials are what CI uses — set them as (masked) CI variables, |
| 65 | +not in the Taskfile (see below). |
| 66 | + |
| 67 | +### Retention and garbage collection |
| 68 | + |
| 69 | +The `oci://` backend never deletes anything: each cache key is a tag, and old |
| 70 | +tags accumulate until the registry prunes them. Two scheduled jobs do that: |
| 71 | + |
| 72 | +1. **Tag retention** (project → *Policy* → *Tag Retention*): add a rule such |
| 73 | + as "retain the artifacts pushed within the last 14 days" (or "retain the |
| 74 | + most recently pushed 50 artifacts") applied to all repositories of the |
| 75 | + project, and schedule it daily. Use the dry-run button to check the rule |
| 76 | + before letting it delete. |
| 77 | +2. **Garbage collection** (*Administration* → *Clean Up* → *Garbage |
| 78 | + Collection*): retention only deletes manifests; GC is what reclaims the |
| 79 | + chunk blobs no longer referenced by any manifest. Schedule it (e.g. |
| 80 | + weekly), with *delete untagged artifacts* enabled. |
| 81 | + |
| 82 | +## Redis: distributed locks |
| 83 | + |
| 84 | +Run a **dedicated, minimal** Redis for the locks — do not reuse Harbor's |
| 85 | +internal one (it is not exposed and is sized for Harbor's own job queues). |
| 86 | +The configuration is deliberately spartan, because locks are short-TTL leases: |
| 87 | + |
| 88 | +- **no persistence** (`save ""`, `appendonly no`): losing locks on a restart |
| 89 | + just makes waiters re-acquire them; |
| 90 | +- **small memory cap with `noeviction`**: evicting a lock key would silently |
| 91 | + break mutual exclusion — better to refuse writes; |
| 92 | +- **password auth**: the lock URL embeds it. |
| 93 | + |
| 94 | +```sh |
| 95 | +PASS=$(openssl rand -hex 24) |
| 96 | +mkdir -p /etc/redis-lock |
| 97 | +cat >/etc/redis-lock/redis.conf <<EOF |
| 98 | +requirepass $PASS |
| 99 | +save "" |
| 100 | +appendonly no |
| 101 | +maxmemory 64mb |
| 102 | +maxmemory-policy noeviction |
| 103 | +EOF |
| 104 | +docker run -d --name redis-lock --restart always \ |
| 105 | + -p 6379:6379 \ |
| 106 | + -v /etc/redis-lock/redis.conf:/usr/local/etc/redis/redis.conf:ro \ |
| 107 | + redis:8.0 redis-server /usr/local/etc/redis/redis.conf |
| 108 | +``` |
| 109 | + |
| 110 | +The resulting lock URL is `redis://:$PASS@<host>:6379`. Smoke test (compare |
| 111 | +the replies — `redis-cli` exits 0 even on `NOAUTH`/`WRONGPASS` error replies): |
| 112 | + |
| 113 | +```sh |
| 114 | +docker exec -e REDISCLI_AUTH=$PASS redis-lock redis-cli ping # PONG |
| 115 | +docker exec redis-lock redis-cli ping # NOAUTH error |
| 116 | +docker exec -e REDISCLI_AUTH=$PASS redis-lock redis-cli set l 1 NX PX 5000 # OK |
| 117 | +``` |
| 118 | + |
| 119 | +## Wiring it into a Taskfile |
| 120 | + |
| 121 | +```yaml |
| 122 | +tasks: |
| 123 | + build: |
| 124 | + sources: |
| 125 | + - src/** |
| 126 | + generates: |
| 127 | + - dist/** |
| 128 | + cache: |
| 129 | + enabled: '{{ne .CI_CACHE_REDIS_URL ""}}' |
| 130 | + url: 'oci://harbor.example.com/task-cache/build:{{urlsafe .TASK}}-{{.CHECKSUM}}' |
| 131 | + lock: 'redis://{{.CI_CACHE_REDIS_URL}}/lock:{{urlsafe .TASK}}-{{.CHECKSUM}}' |
| 132 | + cmds: |
| 133 | + - ./build.sh |
| 134 | +``` |
| 135 | + |
| 136 | +Notes: |
| 137 | + |
| 138 | +- In Harbor the repository path must start with the project name: |
| 139 | + `task-cache/build` is the repository `build` in the project `task-cache`. |
| 140 | + The tag carries the cache key (`[A-Za-z0-9._-]`, 128 chars max). |
| 141 | +- Keep the registry credentials out of the Taskfile: export |
| 142 | + `TASK_CACHE_OCI_USER`, `TASK_CACHE_OCI_PASSWORD` and `TASK_CACHE_OCI_CA` |
| 143 | + in the environment (masked CI variables, with the CA as a *file* variable). |
| 144 | + The robot username contains a `$`, so single-quote it in shell: |
| 145 | + `export TASK_CACHE_OCI_USER='robot$task-cache+ci'`. |
| 146 | +- The Redis URL (with its password) should likewise come from a masked CI |
| 147 | + variable, e.g. `CI_CACHE_REDIS_URL=:<password>@<host>:6379`. |
| 148 | + |
| 149 | +## Verifying the setup |
| 150 | + |
| 151 | +Run a cached task twice — the first run pushes, the second restores without |
| 152 | +executing: |
| 153 | + |
| 154 | +```sh |
| 155 | +task build # task: "build" saved to cache (pushed 42/42 chunks, 13.5 MB) |
| 156 | +rm -rf dist .task |
| 157 | +task build # task: "build" restored from cache |
| 158 | +``` |
| 159 | + |
| 160 | +The pushed entries are visible with `oras` or in the Harbor UI: |
| 161 | + |
| 162 | +```sh |
| 163 | +oras repo tags --ca-file harbor-ca.crt -u 'robot$task-cache+ci' \ |
| 164 | + harbor.example.com/task-cache/build |
| 165 | +``` |
0 commit comments