Add Qwen3 A8W8 L2 runner foundation

[vegetabledoww]

Summary

This is part 1 of splitting the native Qwen3-14B A8W8 serving integration into reviewable PRs under 1000 changed lines.

This PR adds the reusable L2 runtime foundation for the Qwen3-14B A8W8 path:

• add Qwen314BA8W8ModelRunner
• run A8W8 prefill/decode callables through the serving ModelRunner API
• manage A8W8 paged KV cache plus KV scale cache
• pack prefill/decode inputs into the block-table / slot-mapping layout expected by the A8W8 kernels
• cache Simpler L2 workers, registered callables, and worker-side tensor allocations
• add WorkerTensor.to_continuous_tensor() so worker-owned allocations can be passed into L2 orchestration calls

This PR does not register the A8W8 model format or change the user-facing npu_generate.py CLI. That wiring is intentionally left to the follow-up PR so each diff stays below 1000 lines.

Split Plan

1. This PR: L2 runner foundation only.
2. Follow-up PR: A8W8 loader, executor, CLI dispatch, pto_isa_commit plumbing, and tests.

Current size:

2 files changed, 779 insertions(+)

Related

Part of #52.

Related PRs:

• Add native Qwen3 14B A8W8 serving path #48: original full native Qwen3-14B A8W8 serving integration, now split so the runner foundation can be reviewed first.
• Qwen3-14B serving: device-side embedding and sampling #47: Qwen3-14B serving device-side embedding/sampling support, already in the base branch and used as the serving path this work builds on.

Validation

python -m py_compile \
  examples/model/qwen3_14b/runner/npu_runner_a8w8.py \
  python/runtime/worker.py

Also validated through the stacked follow-up branch:

8-token A8W8 generation:  task_20260708_185411_31111876715, exit=0
48-token A8W8 generation: task_20260708_185004_293001430832, exit=0

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 36e3c415-9e80-4ac5-bd9d-e90c750687d0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR adds a new NPU runner module Qwen314BA8W8ModelRunner for Qwen3-14B that manages device KV cache, dispatches compiled PyPTO kernels via an L2 worker stack for prefill/decode, and computes logits on host. A supporting to_continuous_tensor() method is added to WorkerTensor in the runtime worker module.

Changes

Qwen3-14B A8W8 NPU Runner

Layer / File(s)	Summary
Module constants, data shapes, and worker tensor support examples/model/qwen3_14b/runner/npu_runner_a8w8.py, python/runtime/worker.py	Defines trace-name mapping, internal dataclasses for kernel weights/callables/packed inputs, and adds WorkerTensor.to_continuous_tensor() with a compatibility import fallback for ContinuousTensor.
Runner init and KV cache lifecycle examples/model/qwen3_14b/runner/npu_runner_a8w8.py	Constructs Qwen314BA8W8ModelRunner, allocates/frees A8W8 KV scale pages via init_kv_cache/close_kv_cache, and validates KV page bounds on host before device dispatch.
L2 dispatch plumbing and tensor upload examples/model/qwen3_14b/runner/npu_runner_a8w8.py	Builds worker tensor views, orchestration args (_build_l2_orch_args), runs/caches compiled programs via L2 workers (_run_l2_program, _ensure_l2_program), handles CPU→device child tensor uploads, and releases resources on close().
Prefill input packing and execution examples/model/qwen3_14b/runner/npu_runner_a8w8.py	Packs variable-length prefill requests with paged KV addressing, runs the chunked prefill program per layer chunk, and projects logits via _project_logits_host().
Decode input packing and execution examples/model/qwen3_14b/runner/npu_runner_a8w8.py	Packs fixed-batch decode requests with padded rows and paged KV addressing, validates bounds, dispatches the fused decode program, and returns trimmed logits/hidden states.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant Qwen314BA8W8ModelRunner
  participant L2Worker
  participant KVCache

  Caller->>Qwen314BA8W8ModelRunner: run_prefill(model, batch)
  Qwen314BA8W8ModelRunner->>Qwen314BA8W8ModelRunner: _prepare_prefill_inputs(batch)
  Qwen314BA8W8ModelRunner->>KVCache: _validate_kv_cache_bounds(block_table)
  loop per transformer-layer chunk
    Qwen314BA8W8ModelRunner->>L2Worker: _run_l2_program(prefill chunk)
    L2Worker-->>Qwen314BA8W8ModelRunner: chunk hidden states
  end
  Qwen314BA8W8ModelRunner->>Qwen314BA8W8ModelRunner: _project_logits_host()
  Qwen314BA8W8ModelRunner-->>Caller: PrefillResult(logits)

  Caller->>Qwen314BA8W8ModelRunner: run_decode(model, batch)
  Qwen314BA8W8ModelRunner->>Qwen314BA8W8ModelRunner: _prepare_decode_inputs(batch)
  Qwen314BA8W8ModelRunner->>KVCache: _validate_kv_cache_bounds(padded block_table)
  Qwen314BA8W8ModelRunner->>L2Worker: _run_l2_program(decode fused)
  L2Worker-->>Qwen314BA8W8ModelRunner: logits buffer, hidden states
  Qwen314BA8W8ModelRunner-->>Caller: DecodeResult(trimmed logits)

Loading

Possibly related PRs

• hw-native-sys/pypto-serving#24: The new runner's init_kv_cache/close_kv_cache/KV bounds validation extends PR #24's shift to NPU-resident KV cache management.

Poem

A rabbit hops through pages paged,
KV blocks aligned, addresses gauged.
Prefill chunks and decode rows,
Through L2 workers, logits flows.
🐰✨ Hop, dispatch, and cache with care —
Qwen3-14B now runs right there!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: adding the Qwen3 A8W8 L2 runner foundation.
Description check	✅ Passed	The description matches the changeset and accurately describes the new runner and WorkerTensor support.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

examples/model/qwen3_14b/runner/npu_runner_a8w8.py (1)

559-588: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Remove the unused helper or reuse it in close(). _release_l2_child_allocs() has no callers, and close() duplicates the same per-allocation worker.free(...) loop; consolidating that cleanup path would keep the release logic in one place.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@examples/model/qwen3_14b/runner/npu_runner_a8w8.py` around lines 559 - 588,
_Remove the dead helper or wire it into the shutdown path._
`_release_l2_child_allocs()` is currently unused, and `close()` repeats the same
per-allocation `worker.free(...)` and cache cleanup logic; refactor `close()` to
call `_release_l2_child_allocs()` for each runtime or fold the shared release
logic into that helper so the L2 allocation cleanup lives in one place.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@examples/model/qwen3_14b/runner/npu_runner_a8w8.py`:
- Around line 559-588: _Remove the dead helper or wire it into the shutdown
path._ `_release_l2_child_allocs()` is currently unused, and `close()` repeats
the same per-allocation `worker.free(...)` and cache cleanup logic; refactor
`close()` to call `_release_l2_child_allocs()` for each runtime or fold the
shared release logic into that helper so the L2 allocation cleanup lives in one
place.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3634c0c9-d693-4a1b-8f68-647c6eeea064

📥 Commits

Reviewing files that changed from the base of the PR and between 17da73a and e447dc0.

📒 Files selected for processing (2)

• examples/model/qwen3_14b/runner/npu_runner_a8w8.py
• python/runtime/worker.py

[gemini-code-assist]

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.