[Feature] Add native Qwen3-14B A8W8 serving path

[vegetabledoww]

Summary

Track the native Qwen3-14B A8W8 serving integration introduced by PR #48.

This serving-side work connects the Qwen3-14B A8W8 PyPTO kernels from pypto-lib into an end-to-end native serving path. The combined branch can run a full A8W8 decode flow, but follow-up validation and performance tracking are still needed before treating this path as a stable fast path.

Motivation / Use Case

Qwen3-14B A8W8 serving should have a native path that can exercise the PyPTO A8W8 kernels end to end, including model initialization, decode execution, output validation, and performance measurement.

The current work is useful as the integration point for kernel-side optimization in pypto-lib. It also provides a realistic serving benchmark for validating decode TPOT improvements, output quality, and regression risk.

Current behavior

• The matching pypto-lib PR provides the Qwen3-14B A8W8 kernel path.
• The serving-side PR wires this path into native Qwen3-14B A8W8 serving.
• The combined setup can produce normal Chinese output after the Q RoPE equivalence issue in the fused QK norm path was isolated and corrected on the kernel/model side.
• The latest observed baseline is around 360-371 ms/token TPOT, with one corrected fused-QK-norm path observed around 429.6 ms/token.

Measurement basis

The current performance bottleneck appears to be decode task fanout and scheduling overhead rather than a single serving-side control-flow issue.

From the paired kernel-side profiling:

• Device wall time is around 356 ms/step.
• Each decode step emits about 38,705 AICore task records.
• Average task runtime is about 45 us.
• Dispatch-to-finish latency is about 61 us.
• The largest static fanout source is MLP gate/up/silu/down, at about 224 groups per layer.

This means the serving path should keep reporting end-to-end TPOT and output quality while the lower-level PyPTO kernel work reduces task fanout.

Proposed behavior

The native Qwen3-14B A8W8 serving path should provide:

• A clear way to enable the native A8W8 path for Qwen3-14B.
• Stable end-to-end decode execution with the matching pypto-lib kernels.
• Reproducible output-quality checks for short decode runs, such as 16-token and 48-token smoke tests.
• Reproducible TPOT reporting so kernel-side changes can be evaluated against the same serving path.
• Guardrails around experimental kernel flags so correctness-sensitive fast paths can remain default-off until validated.

Known risks

• Experimental fused kernel flags can affect output quality. One previous repeated-output regression was traced to the fused QK norm branch using a non-equivalent batched Q RoPE path.
• Some apparent model-DSL optimizations do not reduce end-to-end TPOT, even when they look promising from static submit counts.
• Backend buffer limits currently block some larger-tile MLP attempts, so the serving path should not assume every kernel-side fast path is production-ready.

Suggested validation plan

1. Keep a default correctness path for Qwen3-14B A8W8 serving.
2. Add or document the exact command/config used for 16-token and 48-token output-quality checks.
3. Record TTFT, TPOT, and decode throughput for the default path and each experimental fast-path flag combination.
4. Cross-link serving measurements with the matching pypto-lib optimization issue.
5. Treat experimental flags as opt-in until both quality and TPOT improve.

Acceptance criteria

• Qwen3-14B A8W8 native serving can be enabled and run end to end with the matching pypto-lib kernels.
• 16-token and 48-token output-quality checks pass without repeated-output regression.
• TTFT, TPOT, and decode throughput are reported for the validated configuration.
• Experimental fast paths are either default-off or documented with their correctness/performance status.
• The serving issue is kept linked to the kernel-side optimization tracker.

Related

• pypto-serving PR: Add native Qwen3 14B A8W8 serving path #48
• Matching kernel-side PR: Add Qwen3 14B A8W8 kernels pypto-lib#642
• Kernel-side optimization tracker: [Tracking] Qwen3-14B A8W8 decode kernels and TPOT optimization pypto-lib#665