Fixture-gated tests can vanish silently, same failure mode as #55

[kmatzen]

tests/test_video_reader.cpp guards test cases on fixture availability:

#ifdef HAVE_SEEK_FIXTURE
...
#endif
#ifdef HAVE_HDR_FIXTURE
...
#endif

and CMakeLists.txt only defines those when the corresponding ffmpeg invocation at configure time returns 0. If the encode fails — libx265 absent, a filter unavailable in that ffmpeg build, a syntax change in a future ffmpeg — the tests are compiled out and the suite reports success with fewer cases than the author intended.

That is structurally the same problem as #55: a green result that quietly means less than it appears to. #55 was "zero tests ran"; this is "an unknown subset ran".

Currently benign — I checked the CI logs for #51 and neither the seek fixture could not be generated nor the libx265 not available message appears, so both fixtures do build on Linux and macOS today. The concern is that nothing would tell us if that stopped being true.

Options, roughly in order of cost:

1. Print the fixture status positively at configure time (-- seek fixture: enabled) so the log states what is covered rather than only what is missing.
2. Assert a minimum test count in CI: framewright_tests --list-tests | wc -l, compared against an expected number, or Catch2's --warn NoTests.
3. Make fixture generation a hard failure on platforms where the codec is expected to be present, and keep the graceful skip only for genuinely optional ones like HDR/libx265.

(3) is the strongest but needs a decision about which fixtures are mandatory. (1) is nearly free and would at least make the gap visible to anyone reading the log.

Found while verifying that the new [seek] cases from #51 actually execute in CI — the suite finished in 0.94s, which was fast enough to be worth confirming they had not been skipped.

[kmatzen]

Partly addressed by #63, which makes CI fail if any fixture reports DISABLED at configure time. All fixture status messages now use a uniform NAME fixture: enabled/DISABLED form, and CI installs every dependency they need, so a disabled fixture there is a real problem rather than graceful degradation. Verified in both directions — all three fixtures report enabled on Linux and both macOS runners, and the guard trips on a synthesised DISABLED line.

Leaving this open for the part I deliberately skipped: asserting a minimum test-case count.

My reasoning for not doing it, for whoever picks this up:

• A hardcoded count needs bumping on every test addition, which trains people to update the number reflexively — including when it drops for a bad reason.
• It fails with "the number changed", not "coverage was lost", so the diagnostic is weak.
• Requiring the fixtures targets the actual mechanism by which the suite can silently shrink, which is what this issue is about.

That said, it does catch cases the fixture guard cannot — a #ifdef gated on something other than a fixture, or a TEST_CASE accidentally deleted. If you want belt and braces, framewright_tests --list-tests ends with a parseable N test cases line and would be a few lines in the workflow.

Reasonable to close this as sufficiently handled if you would rather not carry the count.