pypto-serving

PyPTO Serving is a small local inference stack for running Qwen3-14B generation with PyPTO kernels on Ascend NPUs. It includes a reusable Python runtime, Qwen3-14B executor glue, CLI entry points, and tests for batching and config handling.

Layout

python/
  cli/                         pypto-serving CLI implementation
  core/                        engine, scheduler, KV cache, model loading, async serving
  runtime/                     Simpler worker wrapper for NPU dispatch
pypto-lib/                     submodule providing Qwen3-14B PyPTO kernels
examples/
  pypto-serving                executable CLI wrapper
  model/qwen3_14b/
    cpu_generate.py            CPU reference generation example
    npu_generate.py            NPU generation/profiling example
    npu_serving.json           sample serving config
    runner/                    Qwen3 executors and runner glue
    src/                       PyPTO kernel/program builders
tests/                         CLI, batching, E2E serving, and benchmark tests

Quick Checks

Initialize the kernel submodule after cloning:

git submodule update --init --recursive

Run the unit tests:

python -m pytest tests/test_cli.py tests/test_batching.py

Show CLI help:

./examples/pypto-serving --help
python -m python.cli --help

NPU Generation

One-shot generation:

task-submit --device auto --max-time 0 --run \
  "python examples/model/qwen3_14b/npu_generate.py \
    --model-dir /path/to/Qwen3-14B \
    --prompt 'Huawei is' \
    --platform a2a3 \
    --max-seq-len 512 \
    --max-new-tokens 5"

Offline generation does not require the larger PTO2 ring settings used for concurrent HTTP serving.

Add --profile to print timing and write a Chrome trace when SA_PROFILE_OUTPUT or SA_PROFILE_LEVEL is set:

task-submit --device auto --max-time 0 --run \
  "SA_PROFILE_OUTPUT=/tmp/pypto-serving-profile-offline SA_PROFILE_LEVEL=verbose \
  python examples/model/qwen3_14b/npu_generate.py \
    --model-dir /path/to/Qwen3-14B \
    --prompt 'Huawei is' \
    --platform a2a3 \
    --max-seq-len 512 \
    --max-new-tokens 5 \
    --profile"

HTTP Serving (OpenAI-compatible API)

Start the serving server with a multiprocess worker. When launching through task-submit, keep --device {} so the selected NPU ID is substituted into the server command:

task-submit --device auto --run \
  "python -m python.cli.main \
    --model /path/to/Qwen3-14B \
    --backend npu \
    --platform a2a3 \
    --device {} \
    --port 8899"

Send a generation request after the server logs Application startup complete:

# Health check
curl --noproxy "*" http://127.0.0.1:8899/health

# Completion
curl --noproxy "*" http://127.0.0.1:8899/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Huawei is", "max_tokens": 32, "temperature": 0.0}'

# Streaming
curl --noproxy "*" http://127.0.0.1:8899/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Huawei is", "max_tokens": 32, "stream": true}'

# Chat completion
curl --noproxy "*" http://127.0.0.1:8899/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "What is 1+1?"}], "max_tokens": 32}'

Run the serving benchmark:

python tests/bench_serving.py --port 8899 --stream -n 8 -c 4 --max-tokens 16

Single-request HTTP serving does not require the larger PTO2 ring settings. For concurrent NPU serving, start the server with the larger PTO2 ring settings:

task-submit --device auto --run \
  "PTO2_RING_HEAP=4294967296 PTO2_RING_TASK_WINDOW=1048576 PTO2_RING_DEP_POOL=1048576 \
  python -m python.cli.main \
    --model /path/to/Qwen3-14B \
    --backend npu \
    --platform a2a3 \
    --device {} \
    --port 8899"

Without these settings, multi-request serving may return HTTP 200 while generating no tokens and logging worker runtime failures such as rtMalloc failed: 207001, 507018, or 507046.

Notes

• All model/device/runtime options are passed via CLI arguments. Run python -m python.cli.main --help for the full list.
• Generated kernel artifacts are written under build_output/ and are ignored by git.
• This repository expects PyPTO, CANN, torch, safetensors, transformers, and the local Ascend runtime environment to be available in the active Python environment.
• HTTP serving mode additionally requires fastapi, uvicorn, and pydantic. The benchmark script requires aiohttp.