feat(ai): add opt-in AISIX gateway canary

[Tobias Varlemann (tvarlemann)]

What

• add AISIX v0.8.1 as an experimental, opt-in internal canary beside LiteLLM
• build pinned upstream AISIX natively for AMD64 and ARM64 with SBOM and provenance
• publish the OpenAI-compatible model subset through resource-name-scoped Secrets and RBAC
• add unit tests, a hermetic AISIX/LiteLLM contract-test Job, rollback documentation, and a replacement decision gate
• pin the verified multi-architecture image digest in the module base

Why

This collects real compatibility and operational evidence before replacing LiteLLM. AISIX is disabled by default, has no public route, and every production application continues to use LiteLLM.

Safety and impact

• fresh installations keep the same enabled module set
• unsupported providers remain on LiteLLM and are reported without credentials
• the caller key appears only as a SHA-256 hash in AISIX resources
• provider keys are stored separately from the declarative resource document
• the controller may update only two pre-created Secrets in aisix-system
• AISIX runs non-root with a read-only root filesystem and no service-account token
• Kubernetes service links are disabled so generated AISIX_PORT_* variables cannot override AISIX configuration
• rollback is simply disabling the optional module

Image evidence

• multi-architecture image workflow: passed
• native linux/amd64 and linux/arm64 builds: passed
• published OCI index contains both platform manifests
• pinned digest: ghcr.io/qualityminds/magicstick-aisix@sha256:007cb3c8865e26ac7535bd51253172eca0039176e891be064409cc6dc4ec976c

Rancher ARM64 end-to-end evidence

Tested on the local Rancher Desktop k3s ARM64 node:

• public GHCR pull by digest: passed
• startup with an empty fresh-install model catalog: passed
• readiness/liveness and hardened pod security settings: passed
• credential/config Secret update and rolling reload: passed
• unauthenticated request rejection: passed
• unknown model returns 404: passed
• two temporary models visible through AISIX and LiteLLM: passed
• chat completion through both gateways: passed
• SSE streaming through both gateways: passed
• embeddings through both gateways: passed
• contract Job: succeeded=1 failed=0
• temporary models removed from LiteLLM after the test: passed
• all AISIX/test resources removed and the original cluster state restored: passed

The in-cluster test exposed the Kubernetes service-link collision and the PR now contains the verified fix.

Automated checks

• final public release checks: passed
• 4 Python unit tests and Python byte-compilation: passed
• Ansible playbook syntax and required collections: passed in CI
• all 21 documented Kustomize targets: passed locally and in CI
• current-tree and full-history gitleaks scans: passed locally and in CI
• AISIX v0.8.1 CLI validation of generated resources: passed
• workflow YAML and module-catalog JSON parsing: passed

Test after deploying this branch

1. Open Modules in the dashboard.
2. Enable AISIX (Experimental); no image override is required.
3. Wait for app-aisix and Deployment/aisix to become ready.
4. Optionally apply magic-cluster/apps/ai/aisix/tests and follow Job/aisix-contract-test.
5. Disable AISIX at any time to remove the canary; production clients remain on LiteLLM.

Decision status

This PR makes the comparison testable; it does not approve a production cutover. The documented decision gate still requires real-client provider coverage, tool/structured-output tests, failure tests, benchmarks, and a 72-hour soak before a later neutral ai-gateway migration can be considered.