Skip to content

Add XPU SYCL GGUF ops for dequantize and A8 matmul#1

Draft
zhenwei-intel with Copilot wants to merge 5 commits into
mainfrom
copilot/implement-xpu-gguf-operators
Draft

Add XPU SYCL GGUF ops for dequantize and A8 matmul#1
zhenwei-intel with Copilot wants to merge 5 commits into
mainfrom
copilot/implement-xpu-gguf-operators

Conversation

Copilot AI commented Mar 14, 2026

Copy link
Copy Markdown

This PR adds the minimum GGUF/XPU operator surface needed to cover ggml_dequantize, ggml_mul_mat_vec_a8, and ggml_mul_mat_a8. The implementation now provides these operators through the native _C extension using a SYCL kernel path aligned with the existing XPU kernel style in this repository.

  • What changed

    • Added GGUF operator registrations for:
      • ggml_dequantize
      • ggml_mul_mat_vec_a8
      • ggml_mul_mat_a8
    • Exported the new interfaces from vllm_xpu_kernels and exposed them through the test op registry.
    • Wired the new implementation into the _C extension build and torch bindings.
  • Implementation

    • Replaced the earlier Python compatibility fallback with a native C++/SYCL implementation in csrc/gguf.cpp.
    • Added _C extension declarations and registrations in:
      • csrc/ops.h
      • csrc/torch_bindings.cpp
      • CMakeLists.txt
    • Kept vllm_xpu_kernels/gguf_interface.py as a thin Python wrapper over the registered native ops.
    • The current native GGUF path supports:
      • Q4_0
      • Q8_0
  • Dependency surface

    • Removed the previous Python gguf runtime dependency from requirements.txt, since the operator path is now implemented natively in the extension.
  • Focused coverage

    • Updated targeted tests to validate:
      • packed Q4_0 / Q8_0 reference construction
      • dequantization behavior
      • vector-style A8 matmul path
      • matrix-style A8 matmul path
    • Tests skip XPU-only execution when the XPU device or built extension is unavailable.

Example usage:

import torch
from vllm_xpu_kernels import (
    ggml_dequantize,
    ggml_mul_mat_a8,
    ggml_mul_mat_vec_a8,
)

weight = torch.tensor(..., device="xpu", dtype=torch.uint8)
x = torch.randn(4, 4096, device="xpu", dtype=torch.float16)

deq = ggml_dequantize(weight, quant_type, m, n, torch.float16)
y_vec = ggml_mul_mat_vec_a8(weight, x[:1], quant_type, row_count)
y_mat = ggml_mul_mat_a8(weight, x, quant_type, row_count)

📍 Connect Copilot coding agent with Jira, Azure Boards or Linear to delegate work to Copilot in one click without leaving your project management tool.

Copilot AI and others added 2 commits March 14, 2026 10:30
Co-authored-by: zhenwei-intel <109187816+zhenwei-intel@users.noreply.github.com>
Co-authored-by: zhenwei-intel <109187816+zhenwei-intel@users.noreply.github.com>
Copilot AI changed the title [WIP] Add basic XPU GGUF operators including dequantize and multiplication Add XPU GGUF compatibility ops for dequantize and A8 matmul Mar 14, 2026
Copilot AI requested a review from zhenwei-intel March 14, 2026 10:33
@zhenwei-intel

Copy link
Copy Markdown
Owner

@copilot 参考fp8_quant.cpp的实现,要用sycl kernel实现

Copilot AI and others added 2 commits March 14, 2026 10:50
Co-authored-by: zhenwei-intel <109187816+zhenwei-intel@users.noreply.github.com>
Co-authored-by: zhenwei-intel <109187816+zhenwei-intel@users.noreply.github.com>

Copilot AI commented Mar 14, 2026

Copy link
Copy Markdown
Author

@copilot 参考fp8_quant.cpp的实现,要用sycl kernel实现

已改成基于 C++/_C 扩展的 SYCL kernel 实现,并去掉了之前的 Python fallback 路径,见 f1c50ad。当前新增了原生 ggml_dequantizeggml_mul_mat_vec_a8ggml_mul_mat_a8 的注册与实现,测试也同步调整了。

Copilot AI changed the title Add XPU GGUF compatibility ops for dequantize and A8 matmul Add XPU SYCL GGUF ops for dequantize and A8 matmul Mar 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants