Harden test logistics

.github/workflows/ci.yml

@@ -23,12 +23,16 @@ jobs:
       - name: Install dependencies
         run: uv sync --all-extras
 
-      - name: Run unit tests
-        run: uv run pytest tests/unit/ -q
+      - name: Run fast tests
+        run: >-
+          uv run pytest -q -ra
+          --cov=mlx_speech
+          --cov-report=term-missing:skip-covered
         env:
           # GitHub's virtualized macOS runner reports Metal but aborts custom kernels.
           MLX_SPEECH_DISABLE_CUSTOM_METAL: "1"
 
-      - name: Run checkpoint tests
-        # Skips cleanly when local checkpoints are absent, which is always on CI.
-        run: uv run pytest tests/checkpoint/ -q
+      - name: Validate opt-in test collection
+        run: >-
+          uv run pytest --collect-only -q
+          tests/checkpoint/ tests/runtime/ tests/integration/

.gitignore

@@ -14,6 +14,7 @@
 .hypothesis/
 .coverage
 .coverage.*
+coverage.xml
 htmlcov/
 __pycache__/
 *.py[cod]

README.md

@@ -195,10 +195,14 @@ artifact without PyTorch or `mlx-audio`.
 git clone https://github.com/appautomaton/mlx-speech.git
 cd mlx-speech
 uv sync
-uv run pytest tests/unit/
+uv run pytest
 uv run ruff check .
 ```
 
+The default command runs the fast, artifact-free tier. See
+[`tests/README.md`](tests/README.md) for checkpoint, runtime, integration,
+fixture, and coverage gates.
+
 ```text
 mlx-speech/
   src/mlx_speech/    library code

pyproject.toml

@@ -22,6 +22,7 @@ dependencies = [
 [dependency-groups]
 dev = [
     "pytest>=8.3,<9",
+    "pytest-cov>=7.1,<8",
     "ruff>=0.11,<0.12",
 ]
 
@@ -30,6 +31,7 @@ requires = ["uv_build>=0.11.2,<0.12"]
 build-backend = "uv_build"
 
 [tool.pytest.ini_options]
+addopts = ["--strict-config", "--strict-markers"]
 testpaths = ["tests"]
 markers = [
     "checkpoint: needs local model checkpoints (skip if absent)",
@@ -38,6 +40,16 @@ markers = [
     "local_integration: test requires local model artifacts or repo-specific runtime assets",
 ]
 
+[tool.coverage.run]
+branch = true
+source = ["mlx_speech"]
+
+[tool.coverage.report]
+fail_under = 72.8
+precision = 1
+show_missing = true
+skip_empty = true
+
 [tool.ruff]
 target-version = "py313"
 exclude = [".references", ".venv"]

tests/README.md

@@ -0,0 +1,108 @@
+# Testing
+
+Tests protect existing runtime behavior. Reorganizing tests, fixtures, CI, or
+coverage does not authorize changes under `src/`. If test cleanup exposes a
+production defect, report it and fix it in a separate behavior change.
+
+## Tiers
+
+| Tier | Purpose | Local artifacts | Default CI |
+| --- | --- | --- | --- |
+| `unit/` | Pure logic, tiny MLX models, synthetic checkpoints, and bounded oracle fixtures | No | Yes |
+| `checkpoint/` | Loading and alignment against real local checkpoint assets | Yes | Collection only |
+| `runtime/` | Real-weight forward, inference, streaming, and model-level parity | Yes | Collection only |
+| `integration/` | Public API through waveform or transcript output | Yes, plus evaluation inputs | Collection only |
+
+The default command runs only the fast tier:
+
+```bash
+pytest
+```
+
+Run an opt-in tier by naming its directory:
+
+```bash
+pytest tests/checkpoint/
+pytest tests/runtime/
+RUN_LOCAL_INTEGRATION=1 pytest tests/integration/
+```
+
+When an opt-in run is acting as a required gate, make skips fail the session:
+
+```bash
+MLX_SPEECH_REQUIRE_CHECKPOINTS=1 pytest tests/runtime/
+```
+
+Use this inexpensive command to validate imports and collection across every
+tier without running model inference:
+
+```bash
+pytest --collect-only -q tests/unit/ tests/checkpoint/ tests/runtime/ tests/integration/
+```
+
+Do not put `test_*.py` files directly under `tests/`. Tier directories are
+Python packages so identically named tests in different tiers cannot collide
+during combined collection.
+
+## Placement rules
+
+A test belongs in `unit/` only when it is deterministic and needs no network,
+local model directory, upstream checkout, or optional Torch environment. Prefer
+real tiny MLX modules and tiny safetensors over mocks.
+
+Checkpoint tests validate real artifact layout, keys, shapes, storage/runtime
+dtypes, quantization metadata, and strict alignment. Pure remapping and loader
+branches still belong in `unit/` and should use synthetic checkpoint files.
+
+Runtime tests load real weights and exercise component or inference behavior.
+Integration tests cross the public API boundary and must reach waveform or
+transcript output. A finite, non-empty waveform assertion is a smoke test, not
+a quality gate; WER, CER, speaker similarity, numeric parity, memory, and timing
+regressions need their own explicit metrics.
+
+Every bug fix adds the smallest test that fails before the fix. Put that test in
+the lowest tier capable of reproducing the bug, then add a higher-tier contract
+test only when the failure can cross a component boundary.
+
+## Test doubles
+
+Use stubs or fakes at expensive and nondeterministic boundaries: Hub access,
+network calls, subprocesses, clocks, file writers, tokenizers, and heavyweight
+model adapters. A test double should reject unexpected calls and preserve the
+boundary's input/output contract.
+
+Do not replace the behavior under test. Loader tests use real tiny files;
+generation-state tests use a tiny model or a narrow adapter fake; numeric model
+tests run the real layer implementation.
+
+## Golden fixtures
+
+Golden fixtures isolate the MLX test from the upstream reference environment.
+They avoid a full checkpoint only when the fixture contains the required small
+weights or the tested component is weight-free.
+
+Committed oracle fixtures must record:
+
+- deterministic input construction and seed;
+- reference repository revision and dependency versions;
+- array shapes, dtypes, hashes, and bounded file sizes;
+- a tolerance chosen for the numeric quantity being compared;
+- capture and regeneration commands.
+
+Use exact equality for discrete schedules and tokens, `allclose` for stable
+tensor math, and scale-aware metrics such as correlation or relative RMSE for
+waveforms and complex spectra. Do not regenerate a fixture merely because a
+test failed. Review the behavioral difference and the pinned reference first.
+
+## Coverage
+
+CI measures line and branch coverage for `mlx_speech`. The floor in
+`pyproject.toml` is the measured fast-suite baseline, not a claim that every
+model family has sufficient behavioral coverage. It may move upward with new
+tests and must not be lowered to make CI pass.
+
+New or changed production behavior needs direct coverage for success, boundary,
+and failure paths. Model-family reviews should trace config parsing, checkpoint
+mapping, component parity, generation state, real runtime inference, public API
+output, and quality/performance gates instead of relying on a single aggregate
+percentage.

tests/checkpoint/__init__.py

@@ -0,0 +1 @@
+"""Tests that require local checkpoint artifacts."""

tests/checkpoint/test_cohere_asr_tokenizer_checkpoint.py

@@ -0,0 +1,34 @@
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+from mlx_speech.models.cohere_asr.tokenizer import CohereAsrTokenizer
+
+
+MODEL_DIR = Path("models/cohere/cohere_transcribe/original")
+
+pytestmark = [
+    pytest.mark.checkpoint,
+    pytest.mark.skipif(
+        not MODEL_DIR.is_dir(),
+        reason="Cohere ASR tokenizer assets are not present",
+    ),
+]
+
+
+def test_tokenizer_prompt_ids_support_punctuation_and_itn() -> None:
+    tokenizer = CohereAsrTokenizer.from_dir(MODEL_DIR)
+
+    default_prompt = tokenizer.get_decoder_prompt_ids("en")
+    no_punctuation_prompt = tokenizer.get_decoder_prompt_ids("en", punctuation=False)
+    itn_prompt = tokenizer.get_decoder_prompt_ids("en", itn=True)
+
+    assert len(default_prompt) == 10
+    assert default_prompt[6] == 5  # <|pnc|>
+    assert no_punctuation_prompt[6] == 6  # <|nopnc|>
+    assert default_prompt[7] == 9  # <|noitn|>
+    assert itn_prompt[7] == 8  # <|itn|>
+    assert default_prompt[8:] == [11, 13]  # <|notimestamp|>, <|nodiarize|>
+    assert itn_prompt[:7] == default_prompt[:7]

tests/checkpoint/test_moss_delay_checkpoint.py

@@ -0,0 +1,26 @@
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+from mlx_speech.models.moss_delay import load_moss_tts_delay_model
+
+MODEL_DIR = Path("models/openmoss/moss_ttsd/mlx-int8")
+
+pytestmark = [
+    pytest.mark.checkpoint,
+    pytest.mark.skipif(
+        not MODEL_DIR.is_dir(),
+        reason="MOSS-TTSD checkpoint is not present",
+    ),
+]
+
+
+def test_default_ttsd_runtime_loads_quantized_mlx_model() -> None:
+    loaded = load_moss_tts_delay_model()
+
+    assert loaded.alignment_report.is_exact_match
+    assert loaded.model.config.n_vq == 16
+    assert loaded.model.language_model.config.num_hidden_layers == 36
+    assert loaded.quantization is not None

tests/test_moss_sound_effect_integration.py → tests/checkpoint/test_moss_sound_effect_checkpoint.py

@@ -1,10 +1,23 @@
 from __future__ import annotations
 
+from pathlib import Path
+
 import pytest
 
-from mlx_speech.models.moss_delay import load_moss_sound_effect_model, resolve_moss_sound_effect_model_dir
+from mlx_speech.models.moss_delay import (
+    load_moss_sound_effect_model,
+    resolve_moss_sound_effect_model_dir,
+)
+
+MODEL_DIR = Path("models/openmoss/moss_sound_effect/mlx-4bit")
 
-pytestmark = pytest.mark.local_integration
+pytestmark = [
+    pytest.mark.checkpoint,
+    pytest.mark.skipif(
+        not MODEL_DIR.is_dir(),
+        reason="MOSS sound-effect checkpoint is not present",
+    ),
+]
 
 
 def test_default_moss_sound_effect_runtime_loads_quantized_mlx_model() -> None:

tests/checkpoint/test_vibevoice_checkpoint_local.py

@@ -0,0 +1,35 @@
+"""Local VibeVoice checkpoint loading and alignment coverage."""
+
+from pathlib import Path
+
+import pytest
+
+from mlx_speech.models.vibevoice.checkpoint import (
+    load_vibevoice_checkpoint,
+    load_vibevoice_model,
+)
+
+
+MODEL_DIR = Path("models/vibevoice/mlx-int8")
+HAS_CHECKPOINT = MODEL_DIR.is_dir() and any(MODEL_DIR.glob("*.safetensors"))
+
+pytestmark = [
+    pytest.mark.checkpoint,
+    pytest.mark.skipif(
+        not HAS_CHECKPOINT,
+        reason="VibeVoice checkpoint is not present",
+    ),
+]
+
+
+def test_load_checkpoint() -> None:
+    checkpoint = load_vibevoice_checkpoint(MODEL_DIR)
+
+    assert checkpoint.key_count > 0
+    assert checkpoint.config.model_type == "vibevoice"
+
+
+def test_model_alignment() -> None:
+    loaded = load_vibevoice_model(MODEL_DIR, strict=False)
+
+    assert loaded.alignment_report.is_exact_match

tests/checkpoint/test_vibevoice_config_checkpoint.py

@@ -0,0 +1,40 @@
+"""VibeVoice configuration coverage against local upstream assets."""
+
+from pathlib import Path
+
+import pytest
+
+from mlx_speech.models.vibevoice.config import VibeVoiceConfig
+
+
+MODEL_DIR = Path("models/vibevoice/original")
+
+pytestmark = [
+    pytest.mark.checkpoint,
+    pytest.mark.skipif(
+        not (MODEL_DIR / "config.json").is_file(),
+        reason="VibeVoice original config is not present",
+    ),
+]
+
+
+def test_from_path() -> None:
+    config = VibeVoiceConfig.from_path(MODEL_DIR)
+
+    assert config.model_type == "vibevoice"
+    assert config.hidden_size == 3584
+    assert config.language_config.num_hidden_layers == 28
+    assert config.acoustic_tokenizer_config.vae_dim == 64
+    assert config.semantic_tokenizer_config.vae_dim == 128
+    assert config.diffusion_config.head_layers == 4
+
+
+def test_round_trip() -> None:
+    config = VibeVoiceConfig.from_path(MODEL_DIR)
+    restored = VibeVoiceConfig.from_dict(config.to_dict())
+
+    assert restored.hidden_size == config.hidden_size
+    assert (
+        restored.acoustic_tokenizer_config.vae_dim
+        == config.acoustic_tokenizer_config.vae_dim
+    )

tests/conftest.py

@@ -36,18 +36,6 @@ def _requested_tiers(root: Path, args: tuple[str, ...]) -> set[str]:
     return enabled
 
 
-def _is_explicit_target(path: Path, root: Path, args: tuple[str, ...]) -> bool:
-    for arg in args:
-        if not arg or arg.startswith("-"):
-            continue
-        candidate = Path(arg)
-        if not candidate.is_absolute():
-            candidate = (root / candidate).resolve()
-        if candidate == path:
-            return True
-    return False
-
-
 def pytest_ignore_collect(collection_path: Path, config) -> bool:  # type: ignore[no-untyped-def]
     root = Path(str(config.rootpath)).resolve()
     args = tuple(config.invocation_params.args)
@@ -57,14 +45,6 @@ def pytest_ignore_collect(collection_path: Path, config) -> bool:  # type: ignor
         tier_dir = (root / "tests" / tier).resolve()
         if path == tier_dir or tier_dir in path.parents:
             return tier not in enabled
-    if path.parent == (root / "tests").resolve() and path.suffix == ".py":
-        text = path.read_text(encoding="utf-8")
-        if (
-            'Path("models/' in text
-            or 'MODEL_DIR = "models/' in text
-            or "pytest.mark.local_integration" in text
-        ):
-            return not _is_explicit_target(path, root, args)
     return False
 
 

tests/integration/__init__.py

@@ -0,0 +1 @@
+"""End-to-end tests that exercise public speech APIs."""