Recommended integration patterns for D-FINE and DEIMv2 (Edge DETRs)

[acode-x]

Hi folks!

We are building a multi-class anomaly detection pipeline deployed on low-power ARM64 CPUs using ONNXRuntime + XNNPACK. To adhere to enterprise software licensing policies, we are using <12M parameter real-time transformer detectors: D-FINE (Nano/Small) and DEIMv2 (Pico/Nano).
They are natively NMS-free and output direct 1-to-1 object queries.

Few queries on top of my mind:

1. For ONNX Runtime outputs, is manually populating sv.Detections(xyxy=..., confidence=..., class_id=...) from NumPy arrays the expected pattern?

2. Since these models output direct 1-to-1 query predictions without requiring a Non-Maximum Suppression (NMS) step, are best practices when feeding these predictions directly into sv.ByteTrack?

3. For high-resolution micro-defect detection (1000+pixel), is it better to run high-res inputs directly through the model (leveraging their token-based attention scaling) or pair them with Supervision's InferenceSlicer?

4. Since, they are transformer based models, am curious if xnnpack can really provide any acceleration.

Any recommended integration patterns or guidance would be really helpful!

[capCafu]

Can give you 1 and 2 with reasonable confidence — I'll leave 3 and 4 to someone who's actually deployed these on ARM.

1. Yep, that's the normal way round, not a workaround. Detections is just a dataclass: xyxy required, confidence / class_id / mask / tracker_id / data optional. The from_* helpers are only adapters building the same thing.

2. Main thing is to threshold the queries yourself before building the Detections. ByteTrack doesn't throw low scores away — it uses them on purpose:

remain_inds = scores >= self.track_activation_threshold
inds_low  = scores > 0.1
inds_high = scores < self.track_activation_threshold
inds_second = np.logical_and(inds_low, inds_high)

Anything between 0.1 and track_activation_threshold (0.25 by default) gets a second association pass, and that 0.1 is hard-coded, not a constructor argument. Hand it all your queries raw and a lot of junk boxes get a shot at extending your tracks. You're right that you can skip NMS — ByteTrack does none of its own — and minimum_consecutive_frames > 1 helps kill one-frame ghosts.

[acode-x]

Understood. Thanks for the detailed explanation! @capCafu

[Y1D1R]

Adding to the above, on Q3 and Q4 which are still open.

Q4 (XNNPACK with transformer detectors): expect little gain, and measure it.
The XNNPACK EP only takes a short list of ops: Conv, ConvTranspose, MaxPool, AveragePool, Softmax, Resize (bilinear), Gemm, MatMul (2D only) and their QLinear variants (docs). Everything else falls back to the CPU EP. In D-FINE / DEIMv2 that means the conv backbone can go to XNNPACK, but LayerNorm, the batched MatMuls of attention, GridSample in the deformable decoder, etc. stay on the CPU EP. You end up with a partitioned graph, and the default CPU EP (MLAS) already has NEON kernels on ARM64, so the difference is often small and can even be negative.

If you test it, use the threading setup the docs recommend, otherwise the two threadpools fight each other:

so = ort.SessionOptions()
so.intra_op_num_threads = 1
so.add_session_config_entry("session.intra_op.allow_spinning", "0")
sess = ort.InferenceSession(
    "model.onnx", so,
    providers=[("XnnpackExecutionProvider", {"intra_op_num_threads": 4}), "CPUExecutionProvider"],
)

Then compare against plain CPUExecutionProvider with intra_op_num_threads = <physical cores>. Setting so.log_severity_level = 0 prints the node placement so you can see how much of the graph XNNPACK actually took.

Q3 (full high-res input vs InferenceSlicer): slice, for two reasons.

1. Attention cost grows quadratically with the token count, so pushing 1000+ px inputs through the encoder is exactly what a low-power CPU cannot afford. These models are also trained at a fixed 640 input, and accuracy drops when you move far from the training resolution without fine-tuning.
2. Downscaling the full frame to 640 destroys micro-defects. A 6 px defect in a 2000 px image becomes about 2 px.

So keep the ONNX export at its native static shape and tile:

slicer = sv.InferenceSlicer(
    callback=run_onnx,          # your ORT call returning sv.Detections
    slice_wh=640,
    overlap_wh=100,
    overlap_filter=sv.OverlapFilter.NON_MAX_SUPPRESSION,
    iou_threshold=0.5,
)

Note that even though the model itself is NMS-free, you still need the overlap filter here: the same defect seen in two overlapping tiles gives two detections, and only the slicer can merge them. Latency scales with the number of tiles, so on a fixed-camera inspection setup crop to the ROI first to cut the tile count. Keep thread_workers=1 if ORT already uses all cores.

[acode-x]

Thanks a lot for the detailed answer! @Y1D1R .

[Y1D1R]

You're welcome!