Refactor Docker build pipeline: matrix-driven selection + reusable workflow

[valITino]

Summary

Restructure the Docker image build pipeline to decouple image selection from build execution. Introduces a canonical component registry (.github/build-matrix.json), a selection script that maps changed files to affected images, and a reusable build workflow. This enables:

• Per-component builds on push to main/master (via ci.yml): only rebuild images whose source files changed
• Full or selective builds on release tags and manual dispatch (via build-and-push.yml): rebuild all images for a tag, or pick one via workflow dispatch
• Single source of truth: all image metadata, Dockerfile paths, and change-detection globs live in one JSON file

Key Changes

• .github/build-matrix.json (new): Canonical registry of all publishable Docker images with metadata (service name, Dockerfile path, tag prefix, change-detection globs)

• scripts/select_build_matrix.py (new): Selection script that reads the registry and outputs a compact JSON matrix for GitHub Actions. Supports three modes:

  • --all: select every component (release tags, manual "build all")
  • --service NAME: select one component by name (manual dispatch)
  • --changed FILE: read changed paths from stdin/file and select components whose globs match

• .github/workflows/_build-images.yml (new): Reusable workflow that builds and pushes a single Docker image. Extracted from the original build-and-push.yml to avoid duplication between ci.yml and build-and-push.yml.

• .github/workflows/build-and-push.yml (refactored): Simplified to a two-job workflow:

  • select: resolves the build matrix from the registry (all images for a tag, or one via dispatch)
  • call-build: calls the reusable workflow once per selected image
  • Removed hardcoded matrix; removed workflow_run trigger (per-component builds now happen in ci.yml)

• .github/workflows/ci.yml (enhanced): Added two new jobs:

  • detect-changes: maps changed files to affected images using the selection script
  • call-build: builds only the affected images after tests pass on main/master (push-only guard)

• tests/test_docker_publish_matrix.py (expanded): Added comprehensive tests for the new selection logic:

  • Registry validation (Dockerfiles exist, self-reference, JSON booleans)
  • Dispatch choice options match all services
  • Selection script modes (--all, --service, --changed)
  • File-to-component mapping (exact paths and recursive globs)
  • Multi-component union behavior

• CLAUDE.md & README.md: Updated to document the new workflow structure and registry-driven approach

Implementation Details

• Change detection: Uses glob patterns (exact path or prefix/** recursive) to map changed files to components. A component rebuilds if any of its paths globs match a changed file.
• No-op safety: If no components match the changed files (e.g., docs-only changes), any_changed is false and the build job is skipped.
• Reusable workflow inputs: The _build-images.yml workflow accepts service name, Dockerfile path, tag prefix, and scout flag as inputs, allowing both ci.yml and build-and-push.yml to call it with the same interface.
• Backward compatibility: Release tags and manual dispatch still work as before; per-component builds on push are now automatic and efficient.

https://claude.ai/code/session_01B2rvA2p3F2H9HRuiE74F3j