Rebase b9840#176
Open
makaveli10 wants to merge 330 commits into
Open
Conversation
Co-authored-by: Lubosz Sarnecki <lubosz.sarnecki@collabora.com> Co-authored-by: Guilherme Gallo <guilherme.gallo@collabora.com> --- Rebase on b8828: * Shader includes are now .glsl Rebase on b9840: Use src0_needs_f32 for the src0 f32-conversion guard. The "Add dequantization to out_prod cuda kernel" fix renamed the predicate to src0_needs_f32 = (src0->type != GGML_TYPE_F32) The old name no longer exists (won't compile), and the intent is to also convert F16 — not just quantized types — into a contiguous f32 buffer before the f32-only cublasSgemm. Needed qvac-b9840 because OUT_PROD is the MUL_MAT backward op exercised by this branch's LoRA/training path with F16 weights; on rocBLAS (AMD/RADV) reinterpreting F16 as f32 halves the leading dimension and is rejected with CUBLAS_STATUS_INVALID_VALUE (NVIDIA silently computes garbage).
…split architecture Co-authored-by: Lubosz Sarnecki <lubosz.sarnecki@collabora.com>
Signed-off-by: Marcus Edel <marcus.edel@collabora.com>
Signed-off-by: Marcus Edel <marcus.edel@collabora.com>
Signed-off-by: Marcus Edel <marcus.edel@collabora.com>
Signed-off-by: Marcus Edel <marcus.edel@collabora.com>
Rebase on b8828: * Update FLOAT_TYPEV2 type. * Don't move MXFP4 block in between. * Rename tq_utils.comp to .glsl. Since tq_utils an include only shader it now must be suffixed with glsl, otherwise it will be picked up by the glob upstream introduced in e29acf7 and produce a CMake warning. Fixes: ``` CMake Warning: Expected depfile does not exist. ggml/src/ggml-vulkan/tq_utils.comp.cpp.d ``` Co-authored-by: Lubosz Sarnecki <lubosz.sarnecki@collabora.com> Signed-off-by: Marcus Edel <marcus.edel@collabora.com>
… backward pass. Signed-off-by: Marcus Edel <marcus.edel@collabora.com>
…erride during training. vulkan: revert graph_optimize skip for prompt processing Co-authored-by: vineet <vineet.suryan@collabora.com> Signed-off-by: Marcus Edel <marcus.edel@collabora.com>
Rebase on b8828: * Use .glsl include for tq_utils. Co-authored-by: Lubosz Sarnecki <lubosz.sarnecki@collabora.com> Signed-off-by: Marcus Edel <marcus.edel@collabora.com>
Signed-off-by: vineet <vineet.suryan@collabora.com>
Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
Add llama_opt_epoch_resume function for the resume-from-batch use case and update callers accordingly. Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
- Add overflow and error checks for snprintf when generating LoRA tensor names - Encapsulate tensor pointer validation within llama_lora_init_tensor_weights() and return bool to simplify the caller Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
…ining Signed-off-by: makaveli10 <vineet.suryan@collabora.com> Co-authored-by: Guilherme Gallo <guilherme.gallo@collabora.com> b9692 rebase: - Restrict the OUTPUT LoRA target to the non-block output tensor and exclude norm tensors, so it no longer matches per-block or *norm weights.
Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
…backslash Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
…ry (Metal/CUDA) The FA-gate defaulted efficient_fa=false, so on non-Vulkan GPU backends (Metal, CUDA) — where ggml_backend_supports_efficient_fa is not implemented — flash attention was wrongly disabled. The resulting explicit-attention clip path overflows the pre-sized compute buffer at high n_pos (image_tile_mode=disabled + image_max_tokens=4096), hitting GGML_ASSERT in ggml-backend.cpp:2043 (SIGABRT) on darwin/iOS Metal integration tests. Fix: default efficient_fa=true and disable only when a backend affirmatively reports non-efficient FA. Only ggml-vulkan implements the query (returns false for VK_VENDOR_ID_ARM / non-coopmat), so the Mali FA-disable win is unchanged; Metal/CUDA/CPU keep their efficient FA. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…pCount Review fix (PR tetherto#174): the fused-norm change dispatched GGML_OP_NORM as a direct {ne01, ne02, ne03} grid; on large row counts ne01 can exceed maxComputeWorkGroupCount[0] (spec minimum 65535) and trip the GGML_ASSERT in ggml_vk_dispatch_pipeline, where the previous flattened/tiled dispatch handled arbitrary ggml_nrows. Restore the flattened {512, 512, N} row tiling on the host (same group as SOFT_MAX/SUM_ROWS) and reconstruct {row, channel, sample} in norm.comp from the flat workgroup id (formula shared with soft_max.comp), with a workgroup-uniform bounds return for the tiling round-up. dst offset is unchanged: flat_row == (samp*nchannels + channel)*nrows + row by construction. No behavioural change for in-range shapes; the fusion's dispatch-count reduction is untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…udget-aware) The warmup-time hard-disable for GPU projectors without efficient (coopmat) flash attention replaced AUTO/ENABLED with DISABLED, which short-circuited the budget-aware AUTO heuristic in clip_resolve_flash_attn_type(). At high n_pos (image_tile_mode=disabled with image_max_tokens=4096 -> 16384 ViT patches) the forced explicit attention path materializes an O(n^2 * n_head) score matrix, growing RSS to ~12 GB and getting the process lmkd-killed on Pixel 9 Pro (runQwen35ImageTileModeTokensTest). Downgrade to AUTO instead and record the inefficiency in clip_ctx::fa_backend_inefficient, which now also enables the AUTO cutoff default (previously Mali-detection only) so any non-coopmat backend gets the per-image budget decision: explicit attention below the cutoff (fast on scalar-FA GPUs), memory-frugal scalar FA at/above it or when the explicit scratch would not fit device memory. Explicit user DISABLED is still honored, and MTMD_CLIP_AUTO_FA_MIN_KV still overrides the cutoff.
…q-chunking) On Galaxy S25 Ultra (Adreno 830, OpenCL) the monolithic 16384-patch ViT encode (image_tile_mode=disabled, image_max_tokens=4096) faults the GPU near the end of the encode (Adreno-GSL log_gpu_snapshot fires before any decode work reaches the device), after which the driver aborts the process from cl_a8x_cmdbuf_mgr_submit_ibs (os_exit) on the next submission. Two unbounded behaviours plausibly drive the fault and both are bounded here: - ggml_backend_opencl_graph_compute enqueued entire graphs (thousands of nodes, ~48 s of GPU work for the failing encode) with no intra-graph flush. Now clFlush every GGML_OPENCL_FLUSH_INTERVAL nodes (default 64, 0 disables) so the GSL command-buffer manager receives bounded batches. clFlush submits without stalling the host. - ggml_cl_flash_attn issued one dispatch covering all q rows; at n_q = n_kv = 16384 every workgroup loops the full KV, making a single very long kernel. Now chunked along q rows at GGML_OPENCL_FA_MAX_NQ rows per dispatch (default 4096, 0 disables) with a clFlush between chunks. The split is exact: the kernel resolves its q row relative to the Q/O/mask base offsets, is_causal is always 0 (masking is explicit) and alibi/sinks depend only on the head index, so shifting the row base via byte offsets while shrinking n_q is mathematically identical. No .cl kernel changes. The 512-token image-chunk decode is not implicated: the S25 VLM benchmark ran 304 full-512-row ubatch decodes cleanly. Only the giant monolithic encode (5-8x beyond anything previously run on this backend) triggers the fault.
…, tests Addresses the pre-merge review findings on the two QVAC-21914 crash-fix commits (P1/P2 performance, C1/C2 correctness, S1/S2 robustness, K nits): - ggml-opencl: gate the periodic graph flush on accumulated estimated WORK (GGML_OPENCL_FLUSH_WORK_MB, default 512 MB) instead of a bare node counter. Per-token LLM decode graphs never reach the budget by construction, so the decode hot path stays submission-free; the 16k-patch encode still flushes dozens of times. Single touch point in graph_compute (no more per-fusion-branch duplication). - ggml-opencl: both tunables move onto ggml_backend_opencl_context, resolved once at init with strtol-based parsing (clamp, warn on garbage instead of silently disabling the mitigation) and GGML_LOG_INFO'd like the file's other env knobs. FA chunking reads the context field. - ggml-opencl: GGML_ASSERT(is_causal == 0) before the FA chunk loop — the kernel's causal-boundary formula needs the TOTAL n_q, so chunks after the first would silently corrupt output if causal FA were ever enabled here; keep the invariant loud. Explicit n_q == 0 guard. - clip: rework the AUTO cutoff memory clamp. Total memory now provides the STABLE fast-path clamp (explicit scratch <= total/4); free memory (a volatile, load-dependent number) may only lower the cutoff further via the hard-fit requirement (scratch <= free), never the old free/2 heuristic that silently pushed normal-size Mali images onto the ~2.6x slower scalar-FA path under momentary memory pressure. No memory info at all now fails SAFE at a conservative 2048-patch cap instead of trusting the raw 4096 default (~3.2 GB scratch at n_head=16). The arithmetic is extracted into clip_fa_effective_min_kv() (pure, exposed via clip.h for tests). - tests: test-clip-fa-cutoff (pure CPU, locks in the fast path, the P2 regression guard, the fail-safe cap and edge cases; passing) and test-opencl-fa-chunking (chunked-vs-CPU numerical parity over unchunked / exact-chunk / partial-last-chunk / n_q==1, masked and unmasked, with GGML_OPENCL_FA_MAX_NQ=64 and a 1 MB flush budget; self-skips without a capable OpenCL device — PoCL lacks FP16, so it executes on Adreno-class hardware). - clip warmup comment: note ggml-opencl also lands in the "no efficient-FA query" bucket and its giant-encode fault is handled by the submission bounding inside that backend. GGML_OPENCL_FLUSH_INTERVAL (node-count knob) is replaced by GGML_OPENCL_FLUSH_WORK_MB; GGML_OPENCL_FA_MAX_NQ semantics unchanged.
The pure AUTO-budget helper was defined out-of-line in clip.cpp and declared in the internal clip.h. On Windows mtmd builds as a shared library exporting only the MTMD_API-decorated public API; the internal clip_* symbols are absent from mtmd.lib, so test-clip-fa-cutoff (the first cross-DLL-boundary consumer of a clip_* symbol) failed to link (LNK2019). Linux/macOS export all default-visibility symbols, so it linked there. Move the function inline into clip.h (with its NO_MEMINFO_CAP constant); the test and clip.cpp both compile their own copy — no DLL export of an internal helper. CLIP_AUTO_FA_MIN_KV_MALI_DEFAULT stays in clip.cpp (its only user). Verified: mtmd + test-clip-fa-cutoff build and the test passes.
…uard - parse_env_i64: GGML_LOG_WARN when an in-range-but-too-large GGML_OPENCL_FLUSH_WORK_MB / GGML_OPENCL_FA_MAX_NQ is clamped to max, matching the file's convention of logging every overridden value (previously the clamp was silent). - test-clip-fa-cutoff: the n_head=0 case passed total_mem==free_mem==0, which short-circuits to the NO_MEMINFO cap before any sqrt(.../n_head) branch runs — the div-by-zero guard was never exercised. Pass 16 GB total so the total-memory clamp runs with n_head=0; without the guard the (int)sqrt(x/0) path would now fail the assertion. Both from yingying0906's review; Windows/CPU-only surface, no Android behavior change.
The graph_compute work-budget flush ran BEFORE the current node was dispatched: it accounted the node's work, and on crossing the budget flushed (submitting only the prior batch) then reset the counter to 0 — so the crossing node started a fresh batch. A large op, or the last large segment of the graph, could therefore begin an unflushed batch and be submitted unbounded at the implicit end-of-graph finish, defeating the bound. Move the budget check below the dispatch (convert the fused-op continue chain to if/else so every path reaches one touch point), so the node that crosses the budget is part of the flushed batch. Reported by @gianni-cor.
Non-behavioral cleanups from the PR tetherto#181 review pass: - ggml-opencl graph_compute: correct the flush-cadence comment. The old "per-token decode hot path submission-free by construction" claim was false for multi-GB models — a decode step streams the whole model, so its estimated work crosses the default 512 MB budget a few times per token. Reworded to state that accurately (cost is negligible in practice since clFlush is non-blocking, and it is tunable/zeroable to make decode fully submission-free). Mechanism unchanged. - ggml_cl_flash_attn: GGML_ASSERT(q->ne[1] <= INT32_MAX) before the int n_q truncation, since the q-chunk loop accumulates into an int and derives cl_ulong offsets from it (defensive; not reachable with real shapes). - Consistency: normalize the ticket tag to bare `QVAC-21914` (drop the `qvac ` prefix) in clip.h and tests/CMakeLists.txt, matching the .cpp files and the fork's QVAC-21257 precedent. - tests/CMakeLists.txt: move the unconditional test-opencl-fa-chunking registration up beside test-copy-tbq-subgroups (its self-skipping sibling) instead of sitting right after the LLAMA_MTMD endif() where it read as MTMD-gated; add a comment noting it deliberately does not link mtmd. No functional change to the fix; local mtmd + both tests build, test-clip-fa-cutoff passes.
…so >512-row dispatches don't write past the destination tensor and tolerate a broken system SPIRV-Headers package Signed-off-by: Marcus Edel <marcus.edel@collabora.com>
…on-contiguous (permuted) inputs read the right columns Signed-off-by: Marcus Edel <marcus.edel@collabora.com>
…macOS big-stack helper so model loads on secondary threads don't overflow their 512 KiB stacks Signed-off-by: Marcus Edel <marcus.edel@collabora.com>
Only claim OUT_PROD support for src types ggml_get_to_fp32_cuda can requantize. ggml_cuda_out_prod converts non-F32 srcs to F32 before the f32-only cuBLAS GEMM and aborts on e.g. TQ2_0 which has no CUDA dequantizer.
This commit was lost in the b9518 rebase. vulkan: fix count_equal_masked device-lost hang on RADV/GFX1151. Reduce shared memory usage and accumulate with a 32-bit atomic instead of a 64-bit one; collapse the per-element predicate and drop the dead D_TYPE=int64_t define from the count_equal_masked_i32 generation. Signed-off-by: Marcus Edel <marcus.edel@collabora.com> Co-authored-by: Guilherme Gallo <guilherme.gallo@collabora.com>
The OpenCL kernels are compiled with -cl-finite-math-only and -cl-fast-relaxed-math, which let the compiler assume no Inf/NaN. The flash-attention online softmax initialises its running max to -INFINITY and masks padded scores with -INFINITY, so finite-math miscompiles the init/masking path. Compile the flash-attention programs with a relaxed option set that drops -cl-fast-relaxed-math, -cl-finite-math-only and -cl-unsafe-math-optimizations (keeping -cl-mad-enable for speed) so the -inf sentinels behave correctly. Also harden the strip: erase every occurrence of each flag (not just the first) and GGML_ASSERT that no finite-math/fast-math/unsafe-math flag survived, so a future compile_opts spelling/spacing change fails loudly at load time instead of silently reintroducing the -INFINITY miscompile. Re-ported onto b9840's rewritten OpenCL flash-attn (upstream PR ggml-org#14987 + follow-ups): the original per-dim kernel-compile loop is gone, so the strip is applied once in ggml_opencl_fa_compile_opts(), the single site every FA variant (F16/F32/F32_F16/Q8_0/Q4_0/PRE and _SPLIT) is compiled through. Squashed re-port of 0cbe362 + c1dace7 (finite-math part).
The flash-attention dispatch inferred causal masking from shape with `is_causal = (mask == NULL && n_q > 1 && n_q == n_kv)`. A null mask means no masking, i.e. bidirectional attention (the SigLIP vision and embedding encoders), while causal attention always supplies an explicit causal mask in this codebase (llama-graph.cpp build_attn passes a kq_mask filled with -INFINITY). The heuristic therefore wrongly made the bidirectional Qwen3-VL vision tower attend causally, so each patch only saw earlier patches and the image embedding was corrupted. Set is_causal = 0 unconditionally; causality is always expressed via the explicit mask. This cannot regress the LLM, which already passes a real causal mask (is_causal was already 0 for it) and relies on that mask. Document the invariant in ggml_cl_flash_attn: a null mask is treated as bidirectional, so any caller needing causal masking must supply an explicit causal mask rather than relying on shape inference. Re-ported onto b9840's rewritten OpenCL flash-attn; b9840's own q-chunking path already GGML_ASSERTs is_causal == 0, so this is consistent with the existing code. Squashed re-port of 51dbb17 + c1dace7 (is_causal part).
…uard upscale zero dims The f32/f16 flash-attention kernels load K/V tiles into local memory, barrier, read them, then loop to overwrite the tiles for the next K/V block without a trailing barrier. Out-of-range lanes (the last partial BLOCK_M block) `continue` past the read and race ahead into the next tile load while active lanes are still reading l_k/l_v. With n_kv > BLOCK_N (e.g. the bidirectional vision tower, n_kv=247) this corrupts the shared tiles. Add a trailing barrier(CLK_LOCAL_MEM_FENCE) at the end of the K/V block loop and guard the score computation with `if (my_query_row < n_q)` instead of an early continue. flash_attn_f32_f16.cl already uses that guard + trailing barrier after b9840's redesign, so it is left unchanged. Also guard zero source dimensions in ggml_cl_upscale: the sf* scale factors divide by the source dims, so a zero source dim yields +inf; the existing early-exit only covered zero destination dims. Re-ported onto b9840's rewritten OpenCL flash-attn (b9840 widened the score unroll to j += 4; only the divergence guard + trailing barrier are re-applied, the body is unchanged). Squashed re-port of dc64397 + b7ad6d4. The barrier fix is a GPU-scheduling race whose only proof is on-device; the original b7ad6d4 validated on S25 Ultra / Adreno 830 (Qwen3-VL GPU vision projector matches CPU exactly, Delta 0.0 pp, ~26% faster on encode). Re-verify on-device before merge.
Upstream master's ubuntu-wasm job is red with the same error: github.com/ggml-org/llama.cpp/actions/runs/29361861822/job/87183687962 Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
…of GDN-back op After the optimizations applied to gated_deltanet_back op in the "optimize gated_delta_net_back with subgroup ops" commit, the sc_carry scratch region (S_v * S_v) is no longer used so it can be removed from scratch allocation / stride offets.
…cupancy - Rewrite phase B as a coalesced row-parallel pass and reconstruct exp(g)*S_prev instead of reloading it, cutting per-token scratch traffic (6 -> 4 streams) with coalesced reads - Store u/sd from the scans instead of recomputing, dropping a matrix read and reduction in phase A2 - Remove the per-token workgroup barriers from the serial scans - Run several subgroups per workgroup to hide the token loop's latency - Factor out the shared clustered-add reduction helper (no behavior change)
iancris
reviewed
Jul 16, 2026
iancris
reviewed
Jul 16, 2026
iancris
reviewed
Jul 16, 2026
iancris
reviewed
Jul 16, 2026
iancris
reviewed
Jul 16, 2026
| } else if (op->src[0]->type == GGML_TYPE_Q5_0 || op->src[0]->type == GGML_TYPE_Q5_1 || | ||
| op->src[0]->type == GGML_TYPE_MXFP4 || | ||
| op->src[0]->type == GGML_TYPE_IQ4_NL || | ||
| op->src[0]->type == GGML_TYPE_Q4_K || |
Author
There was a problem hiding this comment.
It was removed because with Q4_1 and Q4_K, opencl was getting OOM issues on Adreno. This happened because more operators were implemented in opencl, which in turn meant more operators were offloaded to opencl kernels, leading to increased memory usage.
by removing it, opencl tests passed on Adreno again
cc @infinitalo
There was a problem hiding this comment.
got it. however, the OOM issue needs to get fixed at the root cause since we usually use Q4_K_M model variants for mobile. if Q4_K gets removed then it will be offloaded to CPU instead. please benchmark using qwen3.5/gemma4 Q4_K_M and compare against existing qwen3.5/gemma4 Q8_0 results. thanks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebase tag: https://github.com/ggml-org/llama.cpp/releases/tag/b9840