mcdc-coverage

Modified Condition/Decision Coverage for Python.

coverage.py measures statement and branch coverage. Neither is the criterion the certified safety-critical tiers ask for. DO-178C Level A, IEC 62304 Class C, and IEC 61508 SIL 3 and 4 require MC/DC: every atomic condition shown to independently affect the outcome of its decision.

No mainstream open-source Python tool measures it. The tools that do are commercial: VectorCAST, Parasoft, Squish COCO, RapiCover.

The gap, in two test cases

def admit(is_member, has_ticket):
    if is_member and has_ticket:
        return "in"
    return "out"

assert admit(True, True) == "in"
assert admit(False, False) == "out"

That is 100% statement coverage, 100% branch coverage, and 100% decision coverage. The decision was taken both ways. Under MC/DC it proves one condition out of two, because nothing shows that has_ticket independently changes the result.

$ pytest --mcdc sample_pkg.logic

  sample_pkg.logic                                1/2      50.0%
  TOTAL conditions                                1/2      50.0%
  decisions fully covered: 0/1
  decision coverage:  1/1  100.0%
  condition coverage: 1/2   50.0%

  conditions still needing an independence pair:
    sample_pkg.logic:5  `has_ticket`
        add a case with is_member=True, `has_ticket`=False, expecting the
        decision to be False (an existing run has `has_ticket`=True giving True).

Decision coverage reads 100% while MC/DC reads 50%. That is the gap, and it is why offering the weaker number as evidence for the stronger requirement does not hold.

It also names the missing vector. Finding out a condition is uncovered is the easy half; working out which case is missing, through several conditions and short-circuiting, is the part that costs an afternoon.

Install

Not on PyPI yet. Install from source:

pip install git+https://github.com/Wayfinder-Systems-Group/mcdc-coverage

Use

It installs a pytest plugin, so it measures inside the test run you already have, with your markers, your options and your conftest:

pytest --mcdc yourpkg --mcdc-fail-under 90
pytest --mcdc yourpkg.core --mcdc-html mcdc.html --mcdc-json mcdc.json

Naming a package instruments everything under it. Listing modules by hand does not scale, and the module someone forgets to list is the one nobody measures.

pytest option	Meaning
--mcdc MODULE	instrument this module or package, repeatable
--mcdc-fail-under PCT	exit 1 below this condition-coverage percentage
--mcdc-json PATH	machine-readable result for CI or an evidence package
--mcdc-html PATH	annotated report, self-contained, no network
--mcdc-raw PATH	raw observations, for merging with other runs
--mcdc-show N	how many uncovered conditions to explain

There is also a standalone runner, which spawns the test runner itself:

mcdc-coverage --modules yourpkg --tests tests --import-root src \
  --fail-under 90 --json mcdc.json --html mcdc.html

It drives pytest by default and unittest on request:

mcdc-coverage --modules yourpkg --tests tests --runner unittest

Three criteria, not one

DO-178C Level A asks for statement, decision and MC/DC. Two of the three fall straight out of observations already recorded, so all three are reported.

Criterion	Satisfied when
Decision	each decision has been taken both ways
Condition	each condition has been seen both true and false
MC/DC	each condition is shown to independently affect the outcome

Statement coverage stays coverage.py's job. Run both.

Merging runs

Coverage collected from unit, integration and system suites has to be combined at the observation level. Summaries cannot be merged: whether a condition is covered depends on a pair of observations, and the two halves of that pair may come from different suites entirely.

pytest tests/unit   --mcdc yourpkg.core --mcdc-raw unit.json
pytest tests/system --mcdc yourpkg.core --mcdc-raw system.json
mcdc-coverage --combine unit.json system.json --json merged.json --html merged.html

A run showing 50% and a run showing 0% can merge to 100%, which is exactly why adding percentages is wrong.

Merging refuses dumps whose decisions have different condition counts, because that means the source moved between runs and the union would describe a revision that never existed.

Which test proved which condition

Under the plugin, every observation is attributed to the test that produced it, so the artifact answers what a reviewer asks. Not "what is the percentage" but "which two tests prove this condition, and what would prove the one that is unproven".

Every artifact also names the build that produced it (tool, tool_version), so a defect found later in the measurement tool can be traced to the artifacts it affected. This tool shipped two such defects in a single day.

"proved_by": [
  {
    "condition": "is_member",
    "module": "sample_pkg.logic",
    "line": 5,
    "tests": ["tests/test_logic.py::test_two_vectors_only"]
  }
]

What counts as a decision

DO-178C defines a decision as a boolean expression composed of conditions, not as an if statement. All of these are instrumented:

Form	Example
if / elif	if a and b:
while	while a and b:
conditional expression	1 if a and b else 0
assert	assert a or b
return of a boolean expression	return a and b
assignment of one	x = a or b
match case guard	case 1 if a and b:
comprehension filter	[x for x in xs if x > 0 and x < 10]
lambda body	lambda a, b: a and b

Nested and/or flatten into a single decision: (a and b) or c is one decision with three conditions. not of an atom is one condition, not two.

Excluding a decision, with a reason

if watchdog_tripped and not recovered:  # pragma: no mcdc: cannot be forced in test
    emergency_stop()

Defensive branches that cannot be exercised are real. Without a way to exclude them a team never reaches a clean gate, and a gate nobody can pass gets deleted.

The exclusion is recorded with its reason and appears in the JSON and the HTML report. A coverage hole an auditor accepts is one justified by analysis, and a justification that leaves no trace in the artifact is not a justification. The pragma applies to the decision on that line only.

Exit codes

Code	Meaning
0	the floor was met, or no floor was set
1	below the floor, or nothing was instrumented
2	usage error

Decisions it cannot measure, reported rather than dropped

Pattern alternation (case 1 | 2:) is a decision and cannot be instrumented, because patterns are not expressions and there is nowhere to attach a recording call. It is reported under "found but not measurable" in the terminal, the JSON and the HTML rather than quietly omitted.

A silently uncounted decision is the worst outcome available. The percentage looks complete and a reviewer has no way to know what was skipped.

Correctness under overlapping evaluations

Condition values are recorded on a per-thread stack, one frame per evaluation, so a decision evaluated concurrently or re-entrantly still produces correct observations. Three cases are pinned by tests:

Case	What happened without frames
Two threads through one decision	an observation came out missing a condition
A condition that recurses into its own decision	the outer frame's conditions were lost
A condition that raises	partial values leaked into the next evaluation

Each produced a wrong observation rather than a missing one, which is worse. A wrong observation can pair with a real one and report a condition as proven when nothing proved it.

Limits, stated rather than discovered

• Statement coverage is not measured. Use coverage.py.
• Requires pytest and a suite to observe. It reports what your tests exercised.
• Does not run under pytest-xdist. Workers record observations in their own processes and those never reach the controller, so a number computed there would cover part of the suite and read as a regression. The plugin refuses the run rather than reporting a fraction. Use --mcdc-raw per shard and --combine instead.
• Targets must be named exactly and must not already be imported and bound before the hook installs. When none of the targets end up instrumented the run fails and says so, rather than printing a percentage computed over nothing.
• Measurement, not a qualification. Producing MC/DC evidence is one input to a certification argument, and this tool is not qualified under DO-330.

Contributing

See CONTRIBUTING.md. The most useful contribution is a decision form that is silently uncounted.

License

Apache License 2.0. Maintained by Wayfinder Systems Group.