diff --git a/README.md b/README.md index 19b10ad..8986c07 100644 --- a/README.md +++ b/README.md @@ -3,28 +3,26 @@ [![Latest release](https://img.shields.io/github/v/tag/0lut/gitmirrorcache?sort=semver&label=release)](https://github.com/0lut/gitmirrorcache/tags) [![Container image](https://img.shields.io/badge/ghcr.io-0lut%2Fgitmirrorcache-blue?logo=docker)](https://github.com/0lut/gitmirrorcache/pkgs/container/gitmirrorcache) -A read-only Git fetch cache that sits between clone-heavy automation — CI -runners, coding agents, sandboxes, build farms — and allowlisted HTTPS -upstreams addressable as `host/owner/repo` (GitHub-style remotes, including -top-level GitLab and Bitbucket repos). Instead of hammering the upstream with -thousands of identical clones, clients fetch from the cache: an S3-compatible object store is the durable source of truth, and -local disk is just a disposable hot layer that can be rebuilt at any time. The -result is faster clones, fewer upstream rate-limit headaches, and a cache you -can throw away without losing anything. +A read-only Git fetch cache for clone-heavy automation (CI runners, coding +agents, build farms). It sits in front of allowlisted HTTPS upstreams +addressable as `host/owner/repo`, so clients fetch from the cache instead of +hitting the upstream with thousands of identical clones. An S3-compatible +object store is the durable source of truth; local disk is a disposable hot +layer that can be rebuilt at any time. It exposes two interfaces: -- A **materialization API** (`/v1/materialize`, `/v1/resolve`) for warming and +- A materialization API (`/v1/materialize`, `/v1/resolve`) for warming and inspecting cache state by commit, branch, or default branch. -- A **read-through Git remote** at `/git/{host}/{owner}/{repo}.git` that - standard Git clients can clone and fetch from over Smart HTTP, including - shallow and blobless fetches. Pushes (`git-receive-pack`) are rejected. +- A read-through Git remote at `/git/{host}/{owner}/{repo}.git` that standard + Git clients can clone and fetch from over Smart HTTP, including shallow and + blobless fetches. Pushes (`git-receive-pack`) are rejected. On a cold miss, the server proxies upstream's response straight to the client -(so first-clone latency stays close to a direct clone) while warming the cache -in the background. Once the cache is warm, pack data is served from the local -bare repo; ref advertisements and branch/default-branch selectors still verify -refs against upstream so clients never see stale tips. +and warms the cache in the background, so first-clone latency stays close to a +direct clone. Once the cache is warm, pack data is served from the local bare +repo. Ref advertisements and branch selectors still verify refs against +upstream, so clients never see stale tips. ## Quick Start @@ -86,7 +84,7 @@ Clients can opt out of cold-miss proxying per request with the The server reads either a TOML config file (set `GIT_CACHE_CONFIG` to its path; see `config/*.example.toml`) or individual environment variables. When `GIT_CACHE_CONFIG` is set, the file wins and the other configuration variables -are ignored — except S3 credentials (`GIT_CACHE_S3_ACCESS_KEY`, +are ignored. The exception is S3 credentials (`GIT_CACHE_S3_ACCESS_KEY`, `GIT_CACHE_S3_SECRET_KEY`, `GIT_CACHE_S3_SESSION_TOKEN`, `GIT_CACHE_S3_REGION`), which are always read from the environment and never from the file. @@ -95,11 +93,11 @@ from the file. | Variable | Default | What it does | | --- | --- | --- | -| `GIT_CACHE_CONFIG` | – | Path to a TOML config file. If set, other `GIT_CACHE_*` variables are ignored, except the S3 credential variables noted above. | +| `GIT_CACHE_CONFIG` | unset | Path to a TOML config file. If set, other `GIT_CACHE_*` variables are ignored, except the S3 credential variables noted above. | | `GIT_CACHE_BIND_ADDR` | `127.0.0.1:8080` | Address and port the HTTP server listens on. | | `GIT_CACHE_ROOT` | `./cache` | Directory for the local hot cache (bare repos, temp files, repo index). | | `GIT_CACHE_ALLOWED_UPSTREAM_HOSTS` | `github.com` | Comma-separated allowlist of upstream hosts the cache will talk to (e.g. `github.com,gitlab.com,git.internal.example`). | -| `GIT_CACHE_UPSTREAM_ROOT` | – | Optional local directory of bare upstream repos, used instead of real network upstreams (mainly for tests and local dev). | +| `GIT_CACHE_UPSTREAM_ROOT` | unset | Optional local directory of bare upstream repos, used instead of real network upstreams (mainly for tests and local dev). | | `GIT_CACHE_RATE_LIMIT_PER_MINUTE` | `120` | Global rate limit for materialize requests. | ### Object store @@ -108,11 +106,11 @@ from the file. | --- | --- | --- | | `GIT_CACHE_OBJECT_STORE_KIND` | `local` | `local` (filesystem) or `s3`. S3 requires building with the `s3` feature. | | `GIT_CACHE_OBJECT_STORE_ROOT` | `./tmp/object-store` | Root directory for the `local` object store. | -| `GIT_CACHE_S3_BUCKET` | – | S3 bucket name. Required when kind is `s3`. | +| `GIT_CACHE_S3_BUCKET` | unset | S3 bucket name. Required when kind is `s3`. | | `GIT_CACHE_S3_PREFIX` | `repos` | Key prefix inside the bucket. A schema-version suffix is appended automatically (e.g. `repos` stores under `repos-v3`). | -| `GIT_CACHE_S3_ENDPOINT` | – | Custom S3 endpoint for compatible stores (MinIO, Cloudflare R2, ...). Enables path-style addressing. | +| `GIT_CACHE_S3_ENDPOINT` | unset | Custom S3 endpoint for compatible stores (MinIO, Cloudflare R2, ...). Enables path-style addressing. | | `GIT_CACHE_S3_REGION` | falls back to `AWS_REGION` / `AWS_DEFAULT_REGION` | AWS region for the bucket. | -| `GIT_CACHE_S3_ACCESS_KEY` / `GIT_CACHE_S3_SECRET_KEY` | – | Static S3 credentials. If unset, the standard AWS credential chain is used (env vars, profiles, IAM roles, workload identity). | +| `GIT_CACHE_S3_ACCESS_KEY` / `GIT_CACHE_S3_SECRET_KEY` | unset | Static S3 credentials. If unset, the standard AWS credential chain is used (env vars, profiles, IAM roles, workload identity). | | `GIT_CACHE_S3_SESSION_TOKEN` | falls back to `AWS_SESSION_TOKEN` | Session token for temporary credentials. | ### Git remote @@ -134,13 +132,13 @@ from the file. | `GIT_CACHE_MAX_CONCURRENT_GIT_PROCESSES` | `64` | Semaphore limiting concurrent Git subprocesses (the main CPU/memory knob). | | `GIT_CACHE_ASYNC_MATERIALIZE_CONCURRENCY` | `2` | Concurrency for background materialization work. | | `GIT_CACHE_USE_GITOXIDE` | `true` | Use in-process gitoxide for local read-only Git operations; disable as a kill switch to fall back to the `git` binary. | -| `GIT_CACHE_UPSTREAM_AUTH_TOKEN_ENV` | – | Name of another env var holding a deployment-wide upstream token, injected via Git config env (never argv or manifests). | +| `GIT_CACHE_UPSTREAM_AUTH_TOKEN_ENV` | unset | Name of another env var holding a deployment-wide upstream token, injected via Git config env (never argv or manifests). | ### Disk | Variable | Default | What it does | | --- | --- | --- | -| `GIT_CACHE_DISK_QUOTA_BYTES` | 10 GiB | Hot-cache disk quota; LRU eviction keeps usage under this. The Helm chart sets this to 100 GiB to match its default 100Gi PVC — keep the quota at or below the volume size. | +| `GIT_CACHE_DISK_QUOTA_BYTES` | 10 GiB | Hot-cache disk quota; LRU eviction keeps usage under this. The Helm chart sets this to 100 GiB to match its default 100Gi PVC. Keep the quota at or below the volume size. | | `GIT_CACHE_DISK_MIN_FREE_BYTES` | 1 GiB | Minimum free disk space to preserve on the cache volume. | | `GIT_CACHE_DISK_ACCESS_FLUSH_SECS` | `60` | How often buffered repo-access timestamps are flushed to the on-disk index. | @@ -158,14 +156,13 @@ from the file. ### Helm (Kubernetes) -The project ships as a Helm chart that lives in this repository at +The Helm chart lives at [`deploy/helm/gitmirrorcache`](deploy/helm/gitmirrorcache). Release charts are -published to GHCR as OCI artifacts alongside the container image. The chart runs -the server as a StatefulSet with a persistent volume for the hot cache and an -hourly CronJob for compaction: +published to GHCR as OCI artifacts alongside the container image. The chart +runs the server as a StatefulSet with a persistent volume for the hot cache +and an hourly CronJob for compaction. -Helm install commands take both a release name and a chart reference. From the -repository root, use the chart directory as the chart reference: +From the repository root: ```sh helm install git-cache deploy/helm/gitmirrorcache \ @@ -184,9 +181,9 @@ helm install git-cache "${CHART_REF}" \ --set config.objectStore.s3.bucket=my-git-cache-bucket ``` -For release tag `vX.Y.Z`, use chart version `X.Y.Z`. See the +Release tag `vX.Y.Z` maps to chart version `X.Y.Z`. See the [chart README](deploy/helm/gitmirrorcache/README.md) for credentials, local -checkout installs, sizing, and scaling guidance. +checkout installs, sizing, and scaling. ### AWS (ECS on EC2) @@ -203,16 +200,16 @@ Preview stacks for any branch, tag, or commit can be deployed behind a shared ALB with `scripts/aws/deploy-preview.sh`. See [docs/deployment.md](docs/deployment.md). -The production target pulls the latest published GHCR image for this repository -and skips local Docker/ECR image builds: +The production target pulls the latest published GHCR image and skips local +Docker/ECR image builds: ```sh AWS_REGION=us-west-2 scripts/aws/deploy-prod.sh ``` -The Cloudflare static front door serves the landing page at `gitcache.sh` and -proxies `/git/*` plus API paths to the production ALB, so public clones use the -apex hostname: +The Cloudflare front door serves the landing page at `gitcache.sh` and proxies +`/git/*` plus API paths to the production ALB, so public clones use the apex +hostname: ```sh AWS_REGION=us-west-2 scripts/cloudflare/deploy-static-site.sh @@ -222,8 +219,7 @@ git clone https://gitcache.sh/git/github.com/org/repo.git ### Docker Prebuilt multi-arch images are published to `ghcr.io/0lut/gitmirrorcache` on -`v*` release tags — the badge at the top of this README always shows the -newest release: +`v*` release tags: ```sh docker pull ghcr.io/0lut/gitmirrorcache:latest # most recent release @@ -238,11 +234,11 @@ testing there is a MinIO compose file: docker compose -f docker-compose.minio.yml up -d ``` -### Bare metal / anything else +### Bare metal -It's a single binary plus the `git` CLI. Build with +It is a single binary plus the `git` CLI. Build with `cargo build --release -p git-cache-api --features s3`, point it at a config -file or env vars, and put it behind whatever process supervisor you like. +file or env vars, and run it under any process supervisor. ## CLI @@ -270,7 +266,7 @@ RUN_GITHUB_INTEGRATION=1 python3 -m unittest -v integration_tests.test_git_remot ``` See [integration_tests/README.md](integration_tests/README.md) for MinIO/S3 -variants. The S3 adapter itself is feature-gated: +variants. The S3 adapter is feature-gated: ```sh cargo test -p git-cache-objectstore --features s3