benchmark_support_matrix.md

Benchmark Support Matrix

Canonical benchmark support metadata for yolozu benchmark and tools/benchmark_model.py.

This matrix describes benchmark artifacts, not every standalone exporter utility.

Legend

• real: Benchmark runs the backend/export/eval path and writes real predictions/eval artifacts when the required model artifact and runtime are present; otherwise the report is skipped.
• artifact-real: Benchmark consumes a backend-specific artifact and writes the real outputs identified by the eval and parity columns without claiming YOLOZU ran backend inference.
• placeholder: Benchmark writes an explicit non-real artifact for dry-run planning, a skipped or failed prerequisite, or a component that is not yet attached.
• skipped: Benchmark records a component that is not real because a prerequisite is missing or the comparison is not implemented; see the row notes and report reason.
• unsupported/skipped: Task or format is visible in the interface contract but real benchmark wiring is not shipped; benchmark runs report skipped.

Runtime Requirements

Format	Support state	Runtime requirements	License/runtime notes
torch	conditional real	Python runtime plus task-specific torch/Ultralytics dependencies for detect; artifact-backed tasks consume supplied files.	No runtime is bundled by benchmark mode; users provide model artifacts and optional framework dependencies.
onnx	conditional real	ONNX Runtime and an .onnx artifact for detect; artifact-backed tasks consume supplied files.	ONNX Runtime is optional and environment-provided.
engine	conditional real	Linux, GPU, TensorRT/CUDA bindings, and an .engine or .plan artifact for detect; artifact-backed tasks consume supplied files.	TensorRT/CUDA are optional external runtimes; benchmark reports skipped when unavailable.
torchscript	conditional real	Local PyTorch and a TorchScript artifact for detect; artifact-backed tasks consume supplied files.	PyTorch is optional for this lane and supplied by the runtime environment.
openvino	conditional real	OpenVINO runtime and an OpenVINO IR .xml artifact for detect; artifact-backed tasks consume supplied files.	OpenVINO is optional and environment-provided; detect reports skipped when the runtime is unavailable, while artifact-backed tasks do not invoke the OpenVINO runtime.
executorch	unsupported/skipped	Standalone exporter utilities may exist, but benchmark orchestration is not wired.	Visible for planning; benchmark reports benchmark_format_not_wired.
opencv_dnn	unsupported/skipped	Standalone OpenCV-DNN exporters may exist, but benchmark orchestration is not wired.	Visible for planning; benchmark reports benchmark_format_not_wired.

Task Semantics

Task	Metric family	Surface	Semantics
detect	bbox_map	mainstream	real backend eval with auto or dataset_pass_wall_time when a supported runtime and artifact are available; explicit artifact_eval is rejected before writes
segmentation	mask_map	mainstream	artifact-backed real eval/parity for real backend formats
classification	topk_accuracy	mainstream	artifact-backed real eval/parity with unique sample ids, finite class-aligned score vectors, validated optional ordered class vocabularies, score-difference thresholds, and top-k agreement diagnostics
obb	obb_map	mainstream	artifact-backed real eval/parity with unique image ids, resolved class-id range checks, finite normalized geometry, [0,1] scores, valid empty detection lists, confidence-ranked per-class rotated-IoU AP, separately averaged recall, and rotated-IoU/score parity diagnostics
keypoints	oks_map	mainstream	artifact-backed real eval/parity for real backend formats
depth	depth_error	yolozu-native	artifact-backed real eval/parity for real backend formats
pose6d	pose6d_error	yolozu-native	artifact-backed real eval/parity for real backend formats

Backend Flag Applicability

Task/source validation is applied before per-format flag applicability, after auto resolves to an effective latency source. Within a valid task/source lane, default backend flag values are always accepted: --batch 1, --no-half, --no-nms.

Task scope	Requested latency source	Effective latency source	Formats	Accepted non-default flags	Rejected non-default flags	Behavior
classification, obb, segmentation, keypoints, depth, pose6d	auto	artifact_eval	torch, onnx, engine, torchscript, openvino	none	--half, --batch, --nms	auto resolves to artifact_eval; reject non-default backend execution flags before writing benchmark artifacts.
detect	artifact_eval	rejected before execution	torch, onnx, engine, torchscript, openvino, executorch, opencv_dnn	none	none	No prepared detection-artifact evaluation path is implemented; reject before report, artifact, or backend writes and direct users to auto or dataset_pass_wall_time for backend execution.
classification, obb, segmentation, keypoints, depth, pose6d	artifact_eval	artifact_eval	torch, onnx, engine, torchscript, openvino, executorch, opencv_dnn	none	--half, --batch, --nms	Supported artifact_eval tasks consume prepared artifacts; reject backend precision, batching, and NMS options before writing benchmark artifacts.
non-dry-run classification, obb, segmentation, keypoints, depth, pose6d	dataset_pass_wall_time	invalid for artifact-backed tasks	torch, onnx, engine, torchscript, openvino, executorch, opencv_dnn	none	none	Reject before report or backend writes; use --latency-source auto or artifact_eval.
any task with an otherwise valid non-artifact_eval source	auto, synthetic_step, dataset_pass_wall_time	not artifact_eval	torch	--half, --batch, --nms	none	Accepted values are forwarded by torch detect execution or recorded in the benchmark report for planning-only execution.
any task with an otherwise valid non-artifact_eval source	auto, synthetic_step, dataset_pass_wall_time	not artifact_eval	onnx, engine, torchscript, openvino, executorch, opencv_dnn	none	--half, --batch, --nms	Current format paths do not consume these backend execution flags, so non-default values fail early.

Artifact Support

Format	Task	Inference artifact	Eval artifact	Parity artifact	Notes
torch	detect	real or skipped	real or skipped	real when comparable	Uses export_predictions_ultralytics.py plus eval_suite.py.
torch	segmentation	artifact-real	real	real when comparable	Consumes backend mask-prediction artifacts.
torch	classification	artifact-real	real	real when comparable	Consumes backend classification score artifacts, evaluates top-k metrics, and compares aligned class scores without claiming YOLOZU ran backend inference.
torch	obb	artifact-real	real	real when comparable	Consumes backend rotated-box artifacts, reports 101-point interpolated per-class AP plus separately averaged recall, and compares matched rotated boxes without claiming YOLOZU ran backend inference.
torch	keypoints	artifact-real	real	real when comparable	Consumes backend predictions artifacts.
torch	depth	artifact-real	real	real when comparable	Consumes backend depth artifacts.
torch	pose6d	artifact-real	real	real when comparable	Consumes backend pose predictions artifacts.
onnx	detect	real or skipped	real or skipped	real when comparable	Requires ONNX Runtime and an .onnx artifact or --onnx-model.
onnx	segmentation	artifact-real	real	real when comparable	Consumes backend mask-prediction artifacts.
onnx	classification	artifact-real	real	real when comparable	Consumes backend classification score artifacts, evaluates top-k metrics, and compares aligned class scores without claiming YOLOZU ran backend inference.
onnx	obb	artifact-real	real	real when comparable	Consumes backend rotated-box artifacts, reports 101-point interpolated per-class AP plus separately averaged recall, and compares matched rotated boxes without claiming YOLOZU ran backend inference.
onnx	keypoints	artifact-real	real	real when comparable	Consumes backend predictions artifacts.
onnx	depth	artifact-real	real	real when comparable	Consumes backend depth artifacts.
onnx	pose6d	artifact-real	real	real when comparable	Consumes backend pose predictions artifacts.
engine	detect	real or skipped	real or skipped	real when comparable	Requires Linux, GPU, TensorRT/CUDA bindings, and .engine/.plan.
engine	segmentation	artifact-real	real	real when comparable	Consumes backend mask-prediction artifacts.
engine	classification	artifact-real	real	real when comparable	Consumes backend classification score artifacts, evaluates top-k metrics, and compares aligned class scores without claiming YOLOZU ran backend inference.
engine	obb	artifact-real	real	real when comparable	Consumes backend rotated-box artifacts, reports 101-point interpolated per-class AP plus separately averaged recall, and compares matched rotated boxes without claiming YOLOZU ran backend inference.
engine	keypoints	artifact-real	real	real when comparable	Consumes backend predictions artifacts.
engine	depth	artifact-real	real	real when comparable	Consumes backend depth artifacts.
engine	pose6d	artifact-real	real	real when comparable	Consumes backend pose predictions artifacts.
torchscript	detect	real or skipped	real or skipped	real when comparable	Uses local PyTorch and the declared combined-output decode path in export_predictions_torchscript.py.
torchscript	segmentation	artifact-real	real	real when comparable	Consumes backend mask-prediction artifacts.
torchscript	classification	artifact-real	real	real when comparable	Consumes backend classification score artifacts, evaluates top-k metrics, and compares aligned class scores without claiming YOLOZU ran backend inference.
torchscript	obb	artifact-real	real	real when comparable	Consumes backend rotated-box artifacts, reports 101-point interpolated per-class AP plus separately averaged recall, and compares matched rotated boxes without claiming YOLOZU ran backend inference.
torchscript	keypoints	artifact-real	real	real when comparable	Consumes backend predictions artifacts.
torchscript	depth	artifact-real	real	real when comparable	Consumes backend depth artifacts.
torchscript	pose6d	artifact-real	real	real when comparable	Consumes backend pose predictions artifacts.
openvino	detect	real or skipped	real or skipped	real when comparable	Requires OpenVINO runtime and an OpenVINO IR .xml artifact; reports missing runtime/artifact as skipped.
openvino	segmentation	artifact-real	real	real when comparable	Consumes backend mask-prediction artifacts.
openvino	classification	artifact-real	real	real when comparable	Consumes backend classification score artifacts, evaluates top-k metrics, and compares aligned class scores without claiming YOLOZU ran backend inference.
openvino	obb	artifact-real	real	real when comparable	Consumes backend rotated-box artifacts, reports 101-point interpolated per-class AP plus separately averaged recall, and compares matched rotated boxes without claiming YOLOZU ran backend inference.
openvino	keypoints	artifact-real	real	real when comparable	Consumes backend predictions artifacts.
openvino	depth	artifact-real	real	real when comparable	Consumes backend depth artifacts.
openvino	pose6d	artifact-real	real	real when comparable	Consumes backend pose predictions artifacts.
executorch	detect	skipped	skipped	skipped	Standalone export_predictions_executorch.py has a declared runtime-output decode path; benchmark orchestration reports benchmark_format_not_wired.
executorch	segmentation	skipped	skipped	skipped	Benchmark artifact lane is not shipped; benchmark reports benchmark_format_not_wired.
executorch	classification	skipped	skipped	skipped	Benchmark format lane is not shipped; benchmark reports benchmark_format_not_wired.
executorch	obb	skipped	skipped	skipped	Benchmark format lane is not shipped; benchmark reports benchmark_format_not_wired.
executorch	keypoints	skipped	skipped	skipped	Benchmark artifact lane is not shipped; benchmark reports benchmark_format_not_wired.
executorch	depth	skipped	skipped	skipped	Benchmark artifact lane is not shipped; benchmark reports benchmark_format_not_wired.
executorch	pose6d	skipped	skipped	skipped	Benchmark artifact lane is not shipped; benchmark reports benchmark_format_not_wired.
opencv_dnn	detect	skipped	skipped	skipped	Standalone OpenCV-DNN exporters exist; benchmark orchestration reports benchmark_format_not_wired.
opencv_dnn	segmentation	skipped	skipped	skipped	Benchmark artifact lane is not shipped; benchmark reports benchmark_format_not_wired.
opencv_dnn	classification	skipped	skipped	skipped	Benchmark format lane is not shipped; benchmark reports benchmark_format_not_wired.
opencv_dnn	obb	skipped	skipped	skipped	Benchmark format lane is not shipped; benchmark reports benchmark_format_not_wired.
opencv_dnn	keypoints	skipped	skipped	skipped	Benchmark artifact lane is not shipped; benchmark reports benchmark_format_not_wired.
opencv_dnn	depth	skipped	skipped	skipped	Benchmark artifact lane is not shipped; benchmark reports benchmark_format_not_wired.
opencv_dnn	pose6d	skipped	skipped	skipped	Benchmark artifact lane is not shipped; benchmark reports benchmark_format_not_wired.

Sync Rule

When CLI behavior changes, update the metadata source above and regenerate this page. Keep these files synchronized in the same PR:

• yolozu/cli_entry.py
• docs/benchmark_mode.md
• docs/benchmark_mode_spec_parity_target.md
• docs/benchmark_mode_gap_audit.md
• docs/benchmark_backend_runtime_matrix.md
• docs/generated/cli_reference.md
• docs/README.md
• docs/tools_index.md
• manual/chapters/09_parity_bench_protocols.tex
• tools/manifest.json
• yolozu/data/manifest/tools_manifest.json