Add optional manual CI workflow: integration tests matrix (gemini/gpt/claude)

[VinShurik]

QuickApps version

latest

Area

Tests / CI, Build / dependencies / tooling, Documentation

What is the current state?

• Integration tests can be run locally via make integration_test MODEL=, but there is no CI workflow to run integration tests.
• Model selection is ad-hoc; running multiple models requires manual repetition.
• The Makefile references MODEL_SHORT_NAME for JUnit output, but it is not defined in-repo, making JUnit naming fragile.

Why is this debt?

• No easy way to validate integration behavior across provider families (Gemini/GPT/Claude) in a consistent, repeatable CI run.
• Harder to reproduce integration failures and to compare behavior across models.
• CI signal is missing for changes that affect orchestration + tool interactions (even though the workflow should remain optional/manual to avoid cost/noise).

Proposed remediation

1. Add a canonical 3-model list in the Makefile (single source of truth), e.g. INTEGRATION_TEST_MODELS with defaults for:
   • gemini-2.5-pro
   • gpt-5.2-2025-12-11
   • anthropic.claude-opus-4-6-v1
2. Add Make targets:
   • print-integration-test-models (prints one model per line; used by CI matrix)
   • integration_test_ci MODEL= (runs full integration test flow for a single model; fails if MODEL unset)
   • (optional) integration_test_all (local sequential loop over INTEGRATION_TEST_MODELS)
3. Fix JUnit output naming by deriving a filesystem-safe suffix from MODEL (e.g. replace : and /) instead of relying on MODEL_SHORT_NAME.
4. Add a new GitHub Actions workflow file:
   • .github/workflows/integration-tests.yml
   • workflow_dispatch only (manual trigger; optional; not a PR gate)
   • prepare job reads models from make print-integration-test-models and produces a JSON matrix
   • matrix job runs 3 models in parallel, each invoking make integration_test_ci MODEL=${{ matrix.model }}
   • upload JUnit XML artifacts per model
5. Update docs:
   • .claude/rules/integration-testing.md, src/tests/integration_tests/README.md, CLAUDE.md to document the new targets and manual CI workflow.

Alternatives considered

• Add integration tests to the PR workflow (pr.yml) as a required check: rejected (cost/noise; needs live secrets; should stay optional/manual).
• Run all 3 models sequentially in one CI job: rejected (slower; no parallelism).

Additional information

CI secrets required for live runs: DIAL_URL, DIAL_API_KEY, plus py-interpreter variables if session tests run (PY_INTERPRETER_URL, PY_INTERPRETER_API_KEY, PY_INTERPRETER_LOCAL_RUN=true).