安装组件版本信息
Ascend-docker-runtime_26.0.1_linux-aarch64.run
hami插件版本ascend-device-plugin:v1.4.0
https://github.com/Project-HAMi/HAMi/blob/master/examples/ascend/job-910C.yaml
参考上述模板定义任务,模型镜像改为模型镜像信息quay.io/ascend/vllm-ascend:v0.22.1rc1-a3
,无法启动提示crashloopbackoff
必须声明使用privileged: true
才能创建vnpu
What happened:
测试用例1
yaml声明
securityContext:
privileged: true
资源申明卡数,不申明显存数量
huawei.com/Ascend910C: "1"
资源返回,申请1张卡
环境变量ASCEND_VISIBLE_DEVICES返回 0-15 任意1个编号
例如返回
ASCEND_VISIBLE_DEVICES=13
测试用例2
yaml声明
移除securityContext
资源申明卡数,不申明显存数量
huawei.com/Ascend910C: "1"
资源返回2张卡
huawei.com/Ascend910C: "2"
环境变量ASCEND_VISIBLE_DEVICES返回 0-15 任意2个相邻的编号
ASCEND_VISIBLE_DEVICES=5,4
测试用例3
yaml声明
securityContext:
privileged: true
资源申明卡数,申明显存数量
huawei.com/Ascend910C: "1"
huawei.com/Ascend910C-memory: "16384"
资源返回,申请1张卡
huawei.com/Ascend910C: "1"
huawei.com/Ascend910C-memory: "16384"
ASCEND_VISIBLE_DEVICES=返回 0-15 任意1个的编号
ASCEND_VNPU_SPECS=vir05_1c_16g
查询对应的宿主机,有生成/dev/vdavinciXXX的虚拟实例
同时进入容器内执行
ls -lh /devdavinci*
查询到16张卡0-15,以及虚拟卡116
在镜像内使用vllm命令启动模型
通过环境变量ASCEND_RT_VISBLE_DEVICES,强行指定id,如果指定0-15,可以启动模型,但是等于直接使用某张物理卡,不符合需求。
使用虚拟卡116,则报错
信息
torch npu/csre/core/npu/sys ctrl/npu sys ctrl.cpp:165 NPU function error: aclInit, error code 1s 1070012026-08-26-06:02:31 (PID:77, Device:0, RankID:-I) ERR00100 PTA cd11 acl api failed
ror: Initialize:../
Invalid device ID.
Check whether the device ID is valid.
77) 2026-08-26- 06, 02,31.744.611 Invai1d Argumn ( E0)S The aggumnt 1. inVa1 14 .Reson: EGe Devag eKeuEion fal led, the contont 1s nl e
PID:
Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
TraceBack (most recent call last):
input error deviceId:0 is err:0x7010003(FUNC:SetDefaultDeviceId) ([FILE :api error.cc] [LINE:5432)
rtSetDefaultDeviceId execution failed, reason-device id error [FUNC:FuneErrorReasonl (FILE error message manage.ccl (LINE:65
set default device id failed, ret:107001 [FUNC:ReportCallError [FILE: log inner.cpp] [LINE:148)
IFrocessl [DefaultDevice]process RandleDefaultDevice failed[FUNC:RepprtInnerError] [FILE: log inner.Cpp] (LINE:132)
ctx is NULL![FUNC:GetDevErrMsgl [FILE:api impl.Cc] (LINE:6108)
The arqument is invalid. Reason: rtGetDevMsg execution failed, the context is a null pointer
测试用例4
yaml声明
移除securityContext
资源申明卡数,申明显存数量
huawei.com/Ascend910C: "1"
huawei.com/Ascend910C-memory: "16384"
启动pod崩溃,报错信息
failed to create shim task:OCI runtime create failed: unable to retrieve OCI runtime error
(open /run/containerd/io.containrerd.runtime.v2.task/k8s.io/qwen-vllm-16g/log.json:no such file or directory):
/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime did not terminate succesfully: exit status 1
查询资源申请状态
kubectl describe pod XXXX -o yaml
显示
huawei.com/Ascend910C: "2"
huawei.com/Ascend910C-memory: "16384"
What you expected to happen:
可以正常创建vnpu,并使用对应vnpu运行模型
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Please include the following information:
- The Pod manifest, or at least its accelerator resource requests and limits
- Pod events and relevant accelerator-related node annotations
- Output from the vendor's diagnostic tool, when available (for example,
nvidia-smi -a for NVIDIA devices)
- Your container runtime configuration (for example,
/etc/docker/daemon.json or the relevant containerd configuration)
- Logs from the device plugin for the affected accelerator
- The hami-scheduler logs
- The kubelet logs from the affected node (for example,
sudo journalctl -r -u kubelet)
- Relevant kernel output from
dmesg
Security note: Remove credentials, tokens, registry secrets, private image names, and other sensitive information before posting manifests, configuration files, or logs.
Environment:
- HAMi version:2.9.0
- Kubernetes version:1.34.10
- Accelerator vendor and model:
- HAMi device backend and requested resource names (for example,
nvidia.com/gpu):
- Accelerator driver and runtime version:
- Container runtime and version:containerd2.3.0
- Workload image and tag (redact private image names and state when redacted):
- Operating system and kernel version (for example, the output of
uname -a):kylinv10 sp3
- Others:
安装组件版本信息
Ascend-docker-runtime_26.0.1_linux-aarch64.run
hami插件版本ascend-device-plugin:v1.4.0
https://github.com/Project-HAMi/HAMi/blob/master/examples/ascend/job-910C.yaml
参考上述模板定义任务,模型镜像改为模型镜像信息quay.io/ascend/vllm-ascend:v0.22.1rc1-a3
,无法启动提示crashloopbackoff
必须声明使用privileged: true
才能创建vnpu
What happened:
测试用例1
yaml声明
securityContext:
privileged: true
资源申明卡数,不申明显存数量
huawei.com/Ascend910C: "1"
资源返回,申请1张卡
环境变量ASCEND_VISIBLE_DEVICES返回 0-15 任意1个编号
例如返回
ASCEND_VISIBLE_DEVICES=13
测试用例2
yaml声明
移除securityContext
资源申明卡数,不申明显存数量
huawei.com/Ascend910C: "1"
资源返回2张卡
huawei.com/Ascend910C: "2"
环境变量ASCEND_VISIBLE_DEVICES返回 0-15 任意2个相邻的编号
ASCEND_VISIBLE_DEVICES=5,4
测试用例3
yaml声明
securityContext:
privileged: true
资源申明卡数,申明显存数量
huawei.com/Ascend910C: "1"
huawei.com/Ascend910C-memory: "16384"
资源返回,申请1张卡
huawei.com/Ascend910C: "1"
huawei.com/Ascend910C-memory: "16384"
ASCEND_VISIBLE_DEVICES=返回 0-15 任意1个的编号
ASCEND_VNPU_SPECS=vir05_1c_16g
查询对应的宿主机,有生成/dev/vdavinciXXX的虚拟实例
同时进入容器内执行
ls -lh /devdavinci*
查询到16张卡0-15,以及虚拟卡116
在镜像内使用vllm命令启动模型
通过环境变量ASCEND_RT_VISBLE_DEVICES,强行指定id,如果指定0-15,可以启动模型,但是等于直接使用某张物理卡,不符合需求。
使用虚拟卡116,则报错
信息
torch npu/csre/core/npu/sys ctrl/npu sys ctrl.cpp:165 NPU function error: aclInit, error code 1s 1070012026-08-26-06:02:31 (PID:77, Device:0, RankID:-I) ERR00100 PTA cd11 acl api failed
ror: Initialize:../
Invalid device ID.
Check whether the device ID is valid.
77) 2026-08-26- 06, 02,31.744.611 Invai1d Argumn ( E0)S The aggumnt 1. inVa1 14 .Reson: EGe Devag eKeuEion fal led, the contont 1s nl e
PID:
Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
TraceBack (most recent call last):
input error deviceId:0 is err:0x7010003(FUNC:SetDefaultDeviceId) ([FILE :api error.cc] [LINE:5432)
rtSetDefaultDeviceId execution failed, reason-device id error [FUNC:FuneErrorReasonl (FILE error message manage.ccl (LINE:65
set default device id failed, ret:107001 [FUNC:ReportCallError [FILE: log inner.cpp] [LINE:148)
IFrocessl [DefaultDevice]process RandleDefaultDevice failed[FUNC:RepprtInnerError] [FILE: log inner.Cpp] (LINE:132)
ctx is NULL![FUNC:GetDevErrMsgl [FILE:api impl.Cc] (LINE:6108)
The arqument is invalid. Reason: rtGetDevMsg execution failed, the context is a null pointer
测试用例4
yaml声明
移除securityContext
资源申明卡数,申明显存数量
huawei.com/Ascend910C: "1"
huawei.com/Ascend910C-memory: "16384"
启动pod崩溃,报错信息
failed to create shim task:OCI runtime create failed: unable to retrieve OCI runtime error
(open /run/containerd/io.containrerd.runtime.v2.task/k8s.io/qwen-vllm-16g/log.json:no such file or directory):
/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime did not terminate succesfully: exit status 1
查询资源申请状态
kubectl describe pod XXXX -o yaml
显示
huawei.com/Ascend910C: "2"
huawei.com/Ascend910C-memory: "16384"
What you expected to happen:
可以正常创建vnpu,并使用对应vnpu运行模型
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Please include the following information:
nvidia-smi -afor NVIDIA devices)/etc/docker/daemon.jsonor the relevant containerd configuration)sudo journalctl -r -u kubelet)dmesgEnvironment:
nvidia.com/gpu):uname -a):kylinv10 sp3