Skip to content

910Cvnpu适配问题 #2842

Description

@sawchen

安装组件版本信息
Ascend-docker-runtime_26.0.1_linux-aarch64.run
hami插件版本ascend-device-plugin:v1.4.0

https://github.com/Project-HAMi/HAMi/blob/master/examples/ascend/job-910C.yaml
参考上述模板定义任务,模型镜像改为模型镜像信息quay.io/ascend/vllm-ascend:v0.22.1rc1-a3
,无法启动提示crashloopbackoff
必须声明使用privileged: true
才能创建vnpu
What happened:
测试用例1
yaml声明
securityContext:
privileged: true
资源申明卡数,不申明显存数量
huawei.com/Ascend910C: "1"

资源返回,申请1张卡
环境变量ASCEND_VISIBLE_DEVICES返回 0-15 任意1个编号
例如返回
ASCEND_VISIBLE_DEVICES=13

测试用例2
yaml声明
移除securityContext

资源申明卡数,不申明显存数量
huawei.com/Ascend910C: "1"

资源返回2张卡
huawei.com/Ascend910C: "2"
环境变量ASCEND_VISIBLE_DEVICES返回 0-15 任意2个相邻的编号
ASCEND_VISIBLE_DEVICES=5,4

测试用例3
yaml声明
securityContext:
privileged: true
资源申明卡数,申明显存数量
huawei.com/Ascend910C: "1"
huawei.com/Ascend910C-memory: "16384"

资源返回,申请1张卡
huawei.com/Ascend910C: "1"
huawei.com/Ascend910C-memory: "16384"
ASCEND_VISIBLE_DEVICES=返回 0-15 任意1个的编号
ASCEND_VNPU_SPECS=vir05_1c_16g

查询对应的宿主机,有生成/dev/vdavinciXXX的虚拟实例
同时进入容器内执行
ls -lh /devdavinci*
查询到16张卡0-15,以及虚拟卡116

在镜像内使用vllm命令启动模型
通过环境变量ASCEND_RT_VISBLE_DEVICES,强行指定id,如果指定0-15,可以启动模型,但是等于直接使用某张物理卡,不符合需求。
使用虚拟卡116,则报错
信息
torch npu/csre/core/npu/sys ctrl/npu sys ctrl.cpp:165 NPU function error: aclInit, error code 1s 1070012026-08-26-06:02:31 (PID:77, Device:0, RankID:-I) ERR00100 PTA cd11 acl api failed
ror: Initialize:../
Invalid device ID.
Check whether the device ID is valid.
77) 2026-08-26- 06, 02,31.744.611 Invai1d Argumn ( E0)S The aggumnt 1. inVa1 14 .Reson: EGe Devag eKeuEion fal led, the contont 1s nl e
PID:
Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
TraceBack (most recent call last):
input error deviceId:0 is err:0x7010003(FUNC:SetDefaultDeviceId) ([FILE :api error.cc] [LINE:5432)
rtSetDefaultDeviceId execution failed, reason-device id error [FUNC:FuneErrorReasonl (FILE error message manage.ccl (LINE:65
set default device id failed, ret:107001 [FUNC:ReportCallError [FILE: log inner.cpp] [LINE:148)
IFrocessl [DefaultDevice]process RandleDefaultDevice failed[FUNC:RepprtInnerError] [FILE: log inner.Cpp] (LINE:132)
ctx is NULL![FUNC:GetDevErrMsgl [FILE:api impl.Cc] (LINE:6108)
The arqument is invalid. Reason: rtGetDevMsg execution failed, the context is a null pointer

测试用例4
yaml声明
移除securityContext

资源申明卡数,申明显存数量
huawei.com/Ascend910C: "1"
huawei.com/Ascend910C-memory: "16384"

启动pod崩溃,报错信息
failed to create shim task:OCI runtime create failed: unable to retrieve OCI runtime error
(open /run/containerd/io.containrerd.runtime.v2.task/k8s.io/qwen-vllm-16g/log.json:no such file or directory):
/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime did not terminate succesfully: exit status 1

查询资源申请状态
kubectl describe pod XXXX -o yaml
显示
huawei.com/Ascend910C: "2"
huawei.com/Ascend910C-memory: "16384"

What you expected to happen:
可以正常创建vnpu,并使用对应vnpu运行模型
How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Please include the following information:

  • The Pod manifest, or at least its accelerator resource requests and limits
  • Pod events and relevant accelerator-related node annotations
  • Output from the vendor's diagnostic tool, when available (for example, nvidia-smi -a for NVIDIA devices)
  • Your container runtime configuration (for example, /etc/docker/daemon.json or the relevant containerd configuration)
  • Logs from the device plugin for the affected accelerator
  • The hami-scheduler logs
  • The kubelet logs from the affected node (for example, sudo journalctl -r -u kubelet)
  • Relevant kernel output from dmesg

Security note: Remove credentials, tokens, registry secrets, private image names, and other sensitive information before posting manifests, configuration files, or logs.

Environment:

  • HAMi version:2.9.0
  • Kubernetes version:1.34.10
  • Accelerator vendor and model:
  • HAMi device backend and requested resource names (for example, nvidia.com/gpu):
  • Accelerator driver and runtime version:
  • Container runtime and version:containerd2.3.0
  • Workload image and tag (redact private image names and state when redacted):
  • Operating system and kernel version (for example, the output of uname -a):kylinv10 sp3
  • Others:

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions