perf: ARM CPU conv kernels for DocTR detection (rebased onto temp-9341)#162
Open
olyasir wants to merge 1 commit into
Open
perf: ARM CPU conv kernels for DocTR detection (rebased onto temp-9341)#162olyasir wants to merge 1 commit into
olyasir wants to merge 1 commit into
Conversation
Speeds up the DocTR DBNet detector when it runs on the ggml CPU backend (the hybrid path used on Mali, where full-Vulkan detection is pathological). All kernel changes are ARM-CPU only and bit-identical in their numerical formulas; GPU backends and non-ARM CPU paths are untouched. Rebased from PR #152 (was branched off temp-8828) onto temp-9341. The conv GEMM / im2col / depthwise / conv_transpose / upscale kernels apply unchanged; the only reconciliation is the graph-compute fusion, which now plugs into temp-9341's ggml_cpu_try_fuse_ops() dispatcher (alongside RMS_NORM+MUL) instead of the old inline block in ggml_graph_compute_thread. The F16-src1 mul_mat handling from #152 is dropped: it only existed to keep mul_mat identical to #152's older base and is unreachable for OCR (f16/f16) and LLM (f32 activations), so temp-9341's mul_mat is left untouched. ops.cpp / ops.h: - NEON-vectorised conv_2d_dw_whcn (was fully scalar). - Row-parallel im2col plus a NEON-tiled, two-level-blocked 1x1 im2col transpose (vectorised f32->f16, paired-channel 32-bit stores, L1-resident tiles). Falls back to the channel-strided split when rows_total < nthreads so 1-D / OH==1 convs stay multi-threaded. - Parallelised conv_transpose_2d and a contiguous-row UPSCALE fast path. - ggml_compute_forward_conv_2d_fused / ggml_compute_forward_add_unary_fused: CONV_2D -> ADD(bias) [-> UNARY] and standalone ADD(bias) -> UNARY collapsed into a single pass (bias + activation applied while the result is hot). ggml-cpu.c: - CONV_2D -> ADD -> UNARY(relu|hardswish) epilogue fusion and standalone ADD -> UNARY pair fusion, added as cases in ggml_cpu_try_fuse_ops(). tests: - test-backend-ops: CONV_2D_BIAS_ACT and ADD_BIAS_ACT cases covering the two fusion paths (relu / hardswish, with and without activation). The reference CPU backend runs with use_ref=1 (fusion disabled), so the cases compare the fused result against the unfused reference. Measured on a Pixel 9 Pro (Mali-G715), DocTR clinical_chemistry, warm: detection 1.42s -> ~0.83s on the CPU backend, output unchanged (boxes=197, 12/12 expected keywords).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ARM CPU conv kernels for the DocTR DBNet detector, rebased from #152 (which
targeted
temp-8828) ontotemp-9341.Speeds up DocTR detection when it runs on the ggml CPU backend — the hybrid
path used on Mali, where full-Vulkan detection is pathological. All kernel
changes are ARM-CPU only and bit-identical in their numerical formulas; GPU
backends and non-ARM CPU paths are untouched.
Changes vs
temp-9341ops.cpp/ops.h(conv kernels, applied unchanged from #152):conv_2d_dw_whcn(was fully scalar).falls back to the channel-strided split when
rows_total < nthreadsso1-D /
OH==1convs stay multi-threaded.conv_transpose_2dand a contiguous-row UPSCALE fast path.ggml_compute_forward_conv_2d_fused/ggml_compute_forward_add_unary_fused:CONV_2D → ADD(bias) [→ UNARY] and standalone ADD(bias) → UNARY in one pass.
ggml-cpu.c(the one reconciliation):ADD → UNARY pair fusion now plug into
temp-9341'sggml_cpu_try_fuse_ops()dispatcher (next to RMS_NORM+MUL), instead of perf: ARM CPU conv kernels for DocTR detection #152's inline block in
ggml_graph_compute_thread.mul_mathandling from perf: ARM CPU conv kernels for DocTR detection #152 is dropped — it only existed tokeep
mul_matidentical to perf: ARM CPU conv kernels for DocTR detection #152's older base, and is unreachable for OCR(f16/f16) and LLM (f32 activations).
temp-9341'smul_matis left as-is.tests/test-backend-ops.cpp:CONV_2D_BIAS_ACTandADD_BIAS_ACTcases covering both fusion paths(relu / hardswish, with and without the trailing activation). The reference
CPU backend runs with
use_ref=1(fusion disabled), so these compare thefused result against the unfused reference.
Testing
test-backend-ops test -b CPU: 15759/15759 OK (clang-22, x86 host),including the new fusion cases.
Measured on a Pixel 9 Pro (Mali-G715), DocTR clinical_chemistry, warm:
detection 1.42s → ~0.83s on the CPU backend, output unchanged (boxes=197,
12/12 expected keywords).