perf: ARM CPU conv kernels for DocTR detection#152
Open
olyasir wants to merge 4 commits into
Open
Conversation
Speeds up the DocTR DBNet detector when it runs on the ggml CPU backend (the hybrid path used on Mali, where full-Vulkan detection is pathological). All changes are ARM-CPU only and bit-identical in their numerical formulas; GPU backends and non-ARM CPU paths are untouched. ggml-cpu.c: - Re-enable llamafile/tinyBLAS for the F16xF16 im2col GEMM on builds that compile in SVE/i8mm (upstream #undef'd GGML_USE_LLAMAFILE there, which left conv GEMMs on the naive path). Runtime kill switch OCR_NO_LLAMAFILE=1 for A/B measurement. - CONV_2D->ADD->UNARY epilogue fusion and standalone ADD->UNARY pair fusion in the graph-compute loop (bias + activation applied in one pass). ops.cpp / ops.h: - NEON-vectorised conv_2d_dw_whcn (was fully scalar). - Row-parallel im2col plus a NEON-tiled, two-level-blocked 1x1 im2col transpose (vectorised f32->f16, paired-channel 32-bit stores, L1-resident tiles). Falls back to the channel-strided split when rows_total < nthreads so 1-D / OH==1 convs stay multi-threaded. - Parallelised conv_transpose_2d and a contiguous-row UPSCALE fast path. Measured on a Pixel 9 Pro (Mali-G715), DocTR clinical_chemistry, warm: detection 1.42s -> ~0.83s on the CPU backend, output unchanged (boxes=197, 12/12 expected keywords).
Remove the env-gated [CPUPROF] per-op wall-clock profiler from ggml_graph_compute_thread (state decl, per-node timing, and the summary log line). It was development instrumentation only (active under OCR_CPU_PROF=1) and has no effect on the kernels. The CONV_2D->ADD->UNARY and ADD->UNARY fusion paths this PR adds are unchanged.
Revert two pieces of experimentation residue to upstream, leaving only the conv-kernel gains: - llamafile/tinyBLAS re-enable (the SVE/i8mm #undef removal, the ggml_ocr_llamafile_enabled gate, OCR_NO_LLAMAFILE, the two call-site guards). The port builds GGML_LLAMAFILE=OFF so this is compiled out, and llamafile measured lf==nolf on ARM for this workload — no gain, and it would re-enable tinyBLAS for LLM on ARM if the flag were ever flipped. - F16-src1 handling in mul_mat (assert widening + the f16->f32 chunked branch). It only enabled an f16 im2col against quantized (Q8_0) conv weights; the Q8 detector path was dropped, so it is unreachable for OCR (src1 matches vec_dot_type for f16/f16) and for LLM (f32 activations). Kept: im2col row-parallel + 1x1 NEON fast path, depthwise NEON, conv_transpose_2d parallelization, upscale fast path, and the CONV_2D->ADD->UNARY / ADD->UNARY fusion.
Re-add the F16 src1 path in ggml_compute_forward_mul_mat (assert widening + the f16->f32 chunked conversion) that an earlier cleanup removed. mtmd / finetuning can feed F16 activations into a quantized mul_mat, so this path is not safely dead. Restoring it makes the mul_mat code identical to the original #152 for every consumer, eliminating any LLM behavioural difference. Only the profiler removal + the compiled-out llamafile revert remain on top of #152.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ARM CPU conv-kernel optimizations for the DocTR DBNet detector when it runs on
the ggml CPU backend. This is the hybrid path used on Mali devices (Pixel 9
Pro / Immortalis), where full-Vulkan detection is pathologically slow, so the
addon routes detection to the CPU and keeps recognition on Vulkan.
All changes are ARM-CPU only and bit-identical in their numerical
formulas. GPU backends and non-ARM CPU paths are untouched.
Changes (3 files,
ggml/src/ggml-cpu/)ggml-cpu.ccompile in SVE / i8mm. Upstream
#undefsGGML_USE_LLAMAFILEwhen those arepresent (the repacked quant paths are preferred for LLM workloads), which left
the large conv GEMMs on the naive path. Runtime kill switch
OCR_NO_LLAMAFILE=1for A/B measurement.CONV_2D→ADD→UNARYepilogue fusion and standaloneADD→UNARYpair fusion inthe graph-compute loop (bias + activation applied in a single pass).
ops.cpp/ops.hconv_2d_dw_whcn(was fully scalar).(vectorised f32→f16, paired-channel 32-bit stores, L1-resident tiles). Falls
back to the channel-strided split when
rows_total < nthreads, so 1-D /OH==1convs stay multi-threaded.conv_transpose_2dand a contiguous-rowUPSCALEfast path.Measured
Pixel 9 Pro (Mali-G715), DocTR
clinical_chemistry, warm, CPU backend:Output unchanged:
boxes=197, 12/12 expected clinical-chemistry keywords.Base
Branched from and targeting
temp-8828(currently at the PR #148 MetalCONV_2D_DW merge,
7bcd140f). The@qvac/ocr-ggmloverlay port will pin thisbranch to validate end-to-end on Device Farm before it's cut as a tag + a
registry port.