test: cover RESUME and ensemble+TTA prediction end-to-end - #32
Merged
Conversation
Group tests by the unit they cover instead of by bug-fix batch: - add tests/conftest.py (autouse KONFAI env defaults, write_config and image_attributes fixtures), replacing 12 copy-pasted module-level os.environ.setdefault blocks that violated the monkeypatch rule - fold the audit/fixes grab-bags and single-test files into per-module files: config, patching, data_manager, transform, augmentation, network, blocks, schedulers, models, dataset, measure, main_cli - merge 5 duplicated tests into their surviving twin, keeping every assertion (config missing-file/invalid-yaml/invalid-literal errors, Data._split tail dedup) 38 unit files instead of 60; 343 -> 338 collected tests (the 5 exact duplicates); per-module coverage is byte-identical before/after.
- fold test_trainer_checkpoints/test_early_stopping/test_ema_convention/ test_resume_lr_override/test_resume_https_checkpoint into test_trainer.py and test_runtime_guards/test_runtime_progress_ddp into test_runtime.py, all tests moved verbatim (18 and 13 collected IDs unchanged) - fix the remaining ruff violations across tests/ (import order, unused noqa, ClassVar on mutable class defaults, zip strict, raw regex match) and reformat the flagged files - mark tests/integration with pytest.mark.integration so -m selection works
Extend ruff check/format targets with tests/ in the CI jobs and in both pixi task blocks so test-suite hygiene is gated like source code.
Two previously untested user workflows now run for real on the tiny synthesis harness: - test_konfai_resume.py: TRAIN 2 epochs, RESUME to 4 via the CLI, then PREDICTION with the resumed checkpoint. Continuity is proven from checkpoint metadata: prior checkpoints preserved (no workspace wipe), epoch/it counters and AdamW step counts continue from the loaded checkpoint instead of restarting, weights keep training, and the final model predicts finite volumes. - test_konfai_ensemble_tta.py: PREDICTION with a 2-checkpoint ensemble (ModelComposite, Mean combine) plus a deterministic Flip TTA. The pointwise TinySynth model makes exact oracles possible: the ensemble must equal (A+B)/2 of the single-model baselines, and a Concat+Sum TTA reduction must yield A+B — proving both members and both TTA branches actually execute and the inverse flip is applied.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds end-to-end coverage for the two highest-risk user workflows that had none (stacked on #31 for the shared conftest and integration marker; will retarget to
mainonce #31 merges):tests/integration/test_konfai_resume.pykonfai TRAIN(2 epochs, tiny synthesis harness,save_checkpoint_mode: ALL) →konfai RESUMEwith the config extended to 4 epochs →konfai PREDICTIONwith the resumed checkpoint. Continuity is asserted from checkpoint metadata, each check chosen to fail on a specific broken behaviour:Checkpoints/<name>and fails this;it = it_end + 1/epoch = epoch_end— fails ifTrainer._load()doesn't restore counters;it_end + (4 − epoch_end) × its_per_epoch— distinguishes a genuine resume from both a silent restart and a resume that ignores the extended epoch count;stepcounters equal the total iteration count across both runs — proves the optimizer state itself round-tripped;tests/integration/test_konfai_ensemble_tta.pyTrains once, derives a second ensemble member by shifting every weight, then runs PREDICTION with
--models A B(ModelComposite,Meancombine) plus a deterministicFlipTTA.TinySynthNetis pointwise (1×1 conv + tanh), so a correctly inverted geometric TTA is an exact identity — turning the assertions into sharp numeric oracles:(A+B)/2of the single-model baselines (atol 5e-3, measured residual 6e-5) — fails if the combine is wrong or the TTA inverse flip is skipped (deviation would be ~24× the tolerance);Concatreduction +Sumtransform yields ≡A+B— the only numeric proof that both TTA branches materialized at runtime (a silently dropped branch gives(A+B)/2, off by 2×);Verification
ruff check/format --checkclean at CI scope.