[Meta] Advanced Test Scheme and Onboarding Plan for Performance & Integration Testing

[nicmostert]

[Meta] Simplified Test Strategy & Implementation Checklist

Background

This meta-issue replaces the earlier, overly complex test scheme proposal. The actual code in main now uses a much simpler, more robust testing pattern that prioritizes deterministic, offline, and CI-friendly testing for all endpoints. The approach is visible in tests/test_schemas/test_responses/test_measurement_list_response.py, tests/conftest.py, and the use of the tests/mock_data/ directory. This issue tracks the rollout of this pattern to all other tests and endpoints.

New Testing Strategy (as implemented)

• All unit tests use only mock/anonymized data checked into the repo (in tests/mock_data/).
  • No network or remote dependencies for unit tests.
  • Enables fully offline runs for CI and local development.
  • Mock data can be updated or expanded as needed for new test cases.
• Integration tests use cached fixture data or, optionally, remote servers.
  • Integration tests are not run by default and require explicit CLI flags or markers.
  • Skipping logic/pytest markers ensure no accidental remote or slow tests during CI runs.
• Test configuration and selection is via CLI flags and pytest markers.
  • See pytest --help for mode options (offline/unit/integration/performance/etc).
  • Tests are grouped cleanly and skip with clear reasons if run in the wrong mode.
• Fixtures for tests are created via simple factory functions, with mock data loaded per endpoint/case.
  • See test_measurement_list_response.py for a fully worked example.
  • Extending to new endpoints is a matter of copying the pattern and supplying new mock data.

Action Items (to complete rollout)

• For each remaining endpoint or response type, convert existing tests to use the new mock-data/fixture pattern (as in test_measurement_list_response.py).
• Ensure all unit tests are fully offline and deterministic (no live requests, only checked-in mock data).
• Add/expand mock data files in tests/mock_data/ as needed for other endpoints and edge cases.
• Standardize test selection and skipping logic across all tests (using the same marker/flag pattern as in conftest.py).
• Update developer documentation (README/test docs) to describe this new, simpler strategy, including how to:
  • Add a new endpoint test using the mock-data pattern.
  • Add new mock data.
  • Run tests in different modes (unit/integration).
• Remove or archive any references, scripts, or docs referring to the old three-source (remote/cached/mocked) test strategy.

Why this approach?

• Reliable and fast: All unit tests are deterministic and run the same on every machine and in CI, with no network dependencies.
• Easy onboarding: New contributors only need to add mock data and copy the fixture pattern to add coverage for new endpoints.
• Flexible: Integration and remote tests are still possible, but always opt-in and never run by default.
• Simple maintenance: All test logic is in one place, and adding new scenarios does not require changes to the core test harness.

---

All further discussion, coordination, and checklist tracking for the new testing approach should happen on this issue. When all endpoints are covered and docs are updated, this issue will close.

[nicmostert]

Okay @copilot, I've rethought this whole thing now. This scheme was way too complicated. I've implemented a much simpler test strategy and pushed it to main. Check out tests/test_schemas/test_responses/test_measurement_list_response.py and tests/conftest.py. to see how i did it. I added mock data in the tests/mock_data/ dir which is used for fully offline unit tests.

The task now would be to implement that strategy on all the other tests and endpoints, where appropriate. Can you look at all this and edit this issue to match this new testing strat? (That's an honest question I'm not sure what you can and cannot do).

[nicmostert]

You pushed an issue to my repo again, you muppet. I updated the issue.