Skip to content

perf: ARM CPU conv kernels for DocTR detection (rebased onto temp-9341)#162

Open
olyasir wants to merge 1 commit into
temp-9341from
perf/doctr-detection-cpu-kernels-9341
Open

perf: ARM CPU conv kernels for DocTR detection (rebased onto temp-9341)#162
olyasir wants to merge 1 commit into
temp-9341from
perf/doctr-detection-cpu-kernels-9341

Conversation

@olyasir

@olyasir olyasir commented Jun 19, 2026

Copy link
Copy Markdown

What

ARM CPU conv kernels for the DocTR DBNet detector, rebased from #152 (which
targeted temp-8828) onto temp-9341.

Speeds up DocTR detection when it runs on the ggml CPU backend — the hybrid
path used on Mali, where full-Vulkan detection is pathological. All kernel
changes are ARM-CPU only and bit-identical in their numerical formulas; GPU
backends and non-ARM CPU paths are untouched.

Changes vs temp-9341

ops.cpp / ops.h (conv kernels, applied unchanged from #152):

  • NEON-vectorised conv_2d_dw_whcn (was fully scalar).
  • Row-parallel im2col + a NEON-tiled, two-level-blocked 1×1 im2col transpose;
    falls back to the channel-strided split when rows_total < nthreads so
    1-D / OH==1 convs stay multi-threaded.
  • Parallelised conv_transpose_2d and a contiguous-row UPSCALE fast path.
  • ggml_compute_forward_conv_2d_fused / ggml_compute_forward_add_unary_fused:
    CONV_2D → ADD(bias) [→ UNARY] and standalone ADD(bias) → UNARY in one pass.

ggml-cpu.c (the one reconciliation):

tests/test-backend-ops.cpp:

  • CONV_2D_BIAS_ACT and ADD_BIAS_ACT cases covering both fusion paths
    (relu / hardswish, with and without the trailing activation). The reference
    CPU backend runs with use_ref=1 (fusion disabled), so these compare the
    fused result against the unfused reference.

Testing

  • test-backend-ops test -b CPU: 15759/15759 OK (clang-22, x86 host),
    including the new fusion cases.
  • NEON kernel paths are exercised by aarch64 CI (compiled out on x86).

Measured on a Pixel 9 Pro (Mali-G715), DocTR clinical_chemistry, warm:
detection 1.42s → ~0.83s on the CPU backend, output unchanged (boxes=197,
12/12 expected keywords).

Speeds up the DocTR DBNet detector when it runs on the ggml CPU backend
(the hybrid path used on Mali, where full-Vulkan detection is pathological).
All kernel changes are ARM-CPU only and bit-identical in their numerical
formulas; GPU backends and non-ARM CPU paths are untouched.

Rebased from PR #152 (was branched off temp-8828) onto temp-9341. The conv
GEMM / im2col / depthwise / conv_transpose / upscale kernels apply unchanged;
the only reconciliation is the graph-compute fusion, which now plugs into
temp-9341's ggml_cpu_try_fuse_ops() dispatcher (alongside RMS_NORM+MUL)
instead of the old inline block in ggml_graph_compute_thread. The F16-src1
mul_mat handling from #152 is dropped: it only existed to keep mul_mat
identical to #152's older base and is unreachable for OCR (f16/f16) and LLM
(f32 activations), so temp-9341's mul_mat is left untouched.

ops.cpp / ops.h:
- NEON-vectorised conv_2d_dw_whcn (was fully scalar).
- Row-parallel im2col plus a NEON-tiled, two-level-blocked 1x1 im2col
  transpose (vectorised f32->f16, paired-channel 32-bit stores, L1-resident
  tiles). Falls back to the channel-strided split when rows_total < nthreads
  so 1-D / OH==1 convs stay multi-threaded.
- Parallelised conv_transpose_2d and a contiguous-row UPSCALE fast path.
- ggml_compute_forward_conv_2d_fused / ggml_compute_forward_add_unary_fused:
  CONV_2D -> ADD(bias) [-> UNARY] and standalone ADD(bias) -> UNARY collapsed
  into a single pass (bias + activation applied while the result is hot).

ggml-cpu.c:
- CONV_2D -> ADD -> UNARY(relu|hardswish) epilogue fusion and standalone
  ADD -> UNARY pair fusion, added as cases in ggml_cpu_try_fuse_ops().

tests:
- test-backend-ops: CONV_2D_BIAS_ACT and ADD_BIAS_ACT cases covering the two
  fusion paths (relu / hardswish, with and without activation). The reference
  CPU backend runs with use_ref=1 (fusion disabled), so the cases compare the
  fused result against the unfused reference.

Measured on a Pixel 9 Pro (Mali-G715), DocTR clinical_chemistry, warm:
detection 1.42s -> ~0.83s on the CPU backend, output unchanged (boxes=197,
12/12 expected keywords).
@olyasir olyasir requested a review from a team as a code owner June 19, 2026 05:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant