RFC: Resolve runtime Ansible collections from Forgejo mirrors, not the EE image

[znerol2]

Proposes: an amendment to ADR-0034
Tracks: dmfdeploy/dmfdeploy#443

Summary

ADR-0034 bans public Galaxy at runtime. Today the ban holds only because the AWX execution environment (EE) image carries the collections. This RFC proposes that AWX project sync resolves runtime collections from git mirrors in the platform's own Forgejo, pinned by tag or commit. The EE keeps only the collections the bootstrap path needs before Forgejo and AWX exist. The same mirror mechanic can serve the next dependency type.

Context

ADR-0034 decided four things: the Organization Galaxy credentials are internal or empty, the internal source is a Forgejo git mirror, collections are baked into the EE, and collections/requirements.yml stays declarative and pinned. It deferred Zot-OCI. The accepted ADR calls the EE bake an interim step and says an opaque EE is not the permanent shape.

The baked image is the weakness of that interim fix:

• A version bump needs an image rebuild and republish.
• requirements.yml in the project no longer lists what runs. The awx-integration role pushes collections: [] to match the image.
• You cannot read the dependency record from the project. You have to inspect the image.

Issue #443 shows the cost of the original problem. A public Galaxy outage failed a workload launch while nothing else changed.

Proposal

Split the collections by when they are needed.

1. The bootstrap set stays in the EE image. dmf-init runs these collections before Forgejo exists, so they cannot come from it.
2. The runtime set leaves the EE. AWX project sync installs it from Forgejo mirrors.

Mirror naming. Each collection gets a Forgejo repo named mirrors/<namespace>.<name>, filled by Forgejo's pull-mirror feature from the upstream source.

Pinning. requirements.yml references the mirror as a git source:

collections:
  - name: https://<forgejo-host>/mirrors/netbox.netbox.git
    type: git
    version: v3.23.0   # tag or full commit SHA

Use a full commit SHA when you need a pin that a tag move cannot change.

How requirements.yml changes. The awx-integration role renders the list from one variable instead of the empty list it pushes today. Each entry holds a name, an upstream URL, and a pin. The remaining runtime entries are listed in the table below.

Galaxy credentials. The default organization carries no Galaxy credential, or only an internal one. The role sets this explicitly and a check asserts it.

One reusable mechanic. One list variable, for example dmf_dependency_mirrors, drives both the Forgejo mirror creation and the rendered requirements file. Each entry has a kind (collection today). To add a dependency, add one entry. A new dependency type adds a new kind and a renderer for it, not a new mechanism.

Zot-OCI stays deferred, as ADR-0034 says.

Collection sets

The classification comes from the playbooks that dmf-init runs (the bootstrap-sandbox-* wrappers in dmf-infra), the launcher playbooks in dmf-runbooks, and the awx-integration role.

Collection	Set	Proof
kubernetes.core	Bootstrap (also used by AWX launchers)	dmf-infra roles/base/cert-manager/tasks/main.yml:37; dmf-runbooks playbooks/launch-nmos-cpp.yml:165
community.general	Bootstrap	dmf-infra roles/base/audit-log-archival/tasks/main.yml:53 (ini_file); roles/stack/operator/awx-integration/tasks/main.yml:39 (random_string)
ansible.posix	Bootstrap	dmf-infra roles/base/harden/tasks/main.yml:30 (authorized_key)
netbox.netbox	Runtime	dmf-infra roles/stack/operator/awx-integration/tasks/main.yml:772 (nb_inventory, AWX inventory source)
community.docker	Unused, so removable	EE ee/requirements.yml:26-29 cites playbook 630, but 630 copies images with skopeo; no community.docker call exists in dmf-infra, dmf-media, or dmf-runbooks

The bootstrap trio matches dmf-init's own requirements-collections.yml:17-22.

Consequences and trade-offs

• Forgejo becomes a project-sync dependency. This is already true for the project source itself (ADR-0014), so it adds no new failure domain.
• Image size and opacity improve. The EE shrinks, and requirements.yml in the project lists what a job runs.
• Sync-time failure mode. If a mirror or tag is missing, project_update fails with a git clone or version error that names the mirror URL. Today an outage fails the sync with an HTTP 502 from a third party. The new message points at a platform-owned repo you can fix.
• First-run ordering. On a fresh facility the mirrors must exist before the first project sync that needs them. The bootstrap chain creates Forgejo (620, 692) before the AWX integration play (693). Mirror creation goes into that window.
• Mirror fill needs upstream egress once, at seed time. Air-gapped installs seed mirrors from a bundle. See open question 3.
• Version bumps become a mirror-list edit, with no image rebuild.

What the release gate checks

The gate asserts that no runtime resolves from public Galaxy and that the EE image holds only the bootstrap set. It checks:

1. Organization Galaxy credentials are empty or internal.
2. Every entry in the rendered requirements.yml is a type: git source on the platform's Forgejo, with a tag or SHA pin.
3. ansible-galaxy collection list inside the published EE image shows only the bootstrap set.
4. With egress to galaxy.ansible.com blocked, a fresh project_update and one catalog launch succeed.

Checks 1 to 3 are static. Check 4 runs on the sandbox profile (ADR-0031).

Migration steps

Each step is one PR.

1. Add the dmf_dependency_mirrors variable and the mirror-creation task for kind: collection.
2. Render collections/requirements.yml from the variable, with netbox.netbox as the first entry.
3. Set the organization Galaxy credentials explicitly, and add the static checks 1 and 2.
4. Remove netbox.netbox and community.docker from ee/requirements.yml; rebuild and publish the EE.
5. Pin the new EE digest on the catalog job templates and the inventory source.
6. Add the image-content check (3) and the blocked-egress check (4) to the release gate.
7. Amend ADR-0034 (layer 3 and the "Bake into the EE" alternative) and flip the plan frontmatter.

Alternatives considered

• Keep baking everything. Loses: version bumps need a rebuild, and the image, not the project, becomes the dependency record.
• Vendor collections into the project repo. Loses: it grows the repo, hides upstream provenance, and still runs ansible-galaxy per sync.
• Zot-OCI. Loses for now: ansible-galaxy OCI support is not first-class. Revisit when it is.
• Self-hosted Galaxy server. Loses: a new heavy service for a handful of collections.

Open questions

1. dmf-init bakes its own copy of the bootstrap trio. Does the EE still need them, or only kubernetes.core for the launchers?
2. Pin by tag, or require a commit SHA for every entry?
3. How are mirrors seeded on an air-gapped install: from a bundle, or fetched once at install time?
4. Should the project stop updating on launch (scm_update_on_launch), as ADR-0034 asks us to reconsider?
5. Who owns bumps to the mirror list and the cadence?

Not yet verified

• An AWX inventory update may not see collections that the project sync installed. If it does not, netbox.netbox cannot leave the EE. Test this before migration step 4.
• No existing code creates Forgejo pull mirrors, so mirror creation and seeding is new work, not an extension.
• The live state of the organization's Galaxy credentials is unconfirmed, and no code sets them today.

[znerol2]

Answers to the open questions (owner, 2026-11-05):

1. Bootstrap set in the EE. Keep only what the AWX launchers call, which is kubernetes.core. community.general and ansible.posix stay in dmf-init's own bootstrap copy and leave the EE.
2. Pins. Every entry pins a full commit SHA. Put the upstream tag in a comment beside it for readability. A tag can move, and a SHA cannot.
3. Air-gapped installs. Seed the mirrors from a bundle that ships with the install. Fetching once at install time would break the case this exists for.
4. scm_update_on_launch. Turn it off. Sync the project when it changes, so a launch does not depend on Forgejo being reachable at that moment.
5. Ownership and cadence. The platform maintainers own the mirror list. Bump monthly, and right away for security fixes. Every bump is a PR to the one list.

With these settled, the next step is the ADR-0034 amendment and the plan update, then migration step 1. The "Not yet verified" item about inventory updates seeing project-installed collections gets tested before migration step 4, as the RFC says.