Skip to main content
Enterprise

Huawei Ascend

Ascend Device Plugin registers Huawei Ascend NPUs as Kubernetes resources. The HAMi scheduler selects devices for workloads. This guide uses Chart 0.4.0 and application v1.5.0 to install the plugin alongside HAMi, configure slicing modes, and verify a workload.

The plugin supports three modes: template (hardware templates), hami-core (runtime soft slicing), and enpu. Check the requirements for the selected mode, configure HAMi scheduling, deploy the standalone plugin, and run a workload.

Prerequisites​

  • Kubernetes 1.20 or later, and Helm 3 with OCI chart support.
  • HAMi Enterprise is installed. See Installation for deployment options. Ascend device plugin and the NVIDIA (HAMi) device plugin can coexist in the same cluster, managing Ascend NPUs and NVIDIA GPUs respectively.
  • Complete HAMi Enterprise license activation before use. This requirement applies to all slicing modes described in this guide.
  • Ascend drivers and ascend-docker-runtime installed on each Ascend node, with an ascend handler configured in the container runtime. Creating a RuntimeClass does not install or configure the handler.
  • npu-smi is available on the node and accessible in the plugin container through the driver directory or the /usr/local/sbin mount.

Ascend driver and CANN versions​

Driver and CANN requirements

The HAMi integration requires Ascend driver 25.5 or later on the node. Prepare CANN in the workload image for the selected slicing backend:

ModeCANN requirement
templateUse a workload CANN version compatible with the node driver and device model. The soft-slicing runtime build versions below do not apply to template mode.
hami-coreThe bundled libvnpu runtime is built with CANN 8.5.2. The workload image's CANN environment must be compatible with this runtime.
enpuThe bundled ubs-virt-enpu/vCANN-RT 1.0.0 runtime is built with CANN 9.1.0. The runtime artifacts must match the workload image's CANN version and meet the corresponding CANN/HDK requirements.

CANN 8.5.2 and 9.1.0 identify the two runtimes' build environments. Compatibility with other CANN versions requires validation of the node driver, workload image, and selected runtime together. Confirm this combination before deploying workloads or upgrading CANN.

template mode​

The driver must support virtualization templates for the device model. HAMi selects the smallest template that satisfies the memory request. Template names and capacities in the device configuration must match the driver.

hami-core mode​

This mode uses libvnpu to control memory and compute. It currently requires ARM nodes and the HAMi scheduler. The driver and workload CANN environment must meet the version requirements above.

The plugin enables device-share for chips during startup on hami-core nodes. If a busy chip temporarily refuses the switch, the plugin withholds that chip from kubelet and HAMi, then retries after the workload finishes. Other chips remain available. Other switching errors still cause plugin startup to fail.

enpu mode​

The integration supports A2/910B and A3/910C. Enable device sharing as required by the paired runtime. Each container can request a share of only one physical DIE.

Set node mode hamiVnpuMode: enpu, and use huawei.com/vnpu-mode: enpu and runtimeClassName: ascend on workload Pods. Prepare vCANN-RT assets compatible with CANN/HDK. Workload images also need /usr/bin/systemd-detect-virt and its dependencies. A3/910C nodes must use independent DIE mode (INDEP_POLICY); A2/910B does not need that setting.

The plugin installs runtime assets from its image into /usr/local/enpu/vcann-rt on the node. If existing files differ, it refuses to overwrite them and fails startup. Before replacing runtime assets, stop ENPU workloads and pause the device plugin on the target node. Back up and move these three files out of that directory: lib/libvruntime.so, tools/enpu-monitor, and ld.so.preload. Keep the hami-core runtime assets. Update the plugin image through the existing Helm release, then resume the plugin to install the new files. To roll back, stop workloads and pause the plugin again, restore the backed-up files and previous plugin image, then resume the plugin.

The bundled ENPU runtime provides basic soft slicing without mem-swap. Basic soft slicing does not require enpu-manager. For mem-swap, prepare a runtime that supports it, deploy a compatible enpu-manager separately and configure enpu.managerURL and enpu.managerConfigRoot. managerURL must reach the manager on the plugin's own node. The plugin does not use hostNetwork, so do not use 127.0.0.1 or a Service that can route to another node.

Configure Ascend scheduling​

These values belong to the HAMi chart. Save the configuration for the selected mode as hami-ascend-values.yaml, merge it with the existing HAMi release values, and update the deployment. All three modes require devices.ascend.enabled: true; this does not install the Ascend plugin. Confirm that the HAMi version supports fields such as hamiVnpuMode before using them.

template configuration​

devices:
ascend:
enabled: true
runtimeClassName: ascend
hamiVnpuMode: template

Template mode derives capacity from device templates. HAMi manages model definitions, resource names, and templates.

hami-core configuration​

devices:
ascend:
enabled: true
runtimeClassName: ascend
hamiVnpuMode: hami-core
deviceSplitCount: 10
hamiCorePolicy: spatial

deviceSplitCount limits the number of virtual devices per physical NPU and defaults to 10. Workload Pods still request their memory and compute shares.

enpu configuration​

devices:
ascend:
enabled: true
runtimeClassName: ascend
hamiVnpuMode: enpu
deviceSplitCount: 10
enpuPolicy: elastic

ENPU caps the effective slot count at 100. enpuPolicy supports fixed-share, elastic, and best-effort. Pods can override the default with huawei.com/enpu-policy; workloads on the same physical DIE must use the same policy.

Update HAMi configuration​

See Installation for installation and upgrade options, and merge the file into the values used by that deployment method. runtimeClassName: ascend lets the HAMi webhook add a RuntimeClass to Ascend workloads; its name must match the resource created or reused by the plugin.

An explicit mode takes precedence over the legacy hamiVnpuCore boolean, and node modes can override the global mode. Stop or reschedule affected workloads before switching modes.

Deploy Ascend Device Plugin​

Deploy the plugin in the same namespace as HAMi. If another Helm release or raw YAML already manages the plugin, update that deployment to avoid duplicate DaemonSets, RBAC resources, and node ConfigMaps.

Label device nodes​

kubectl label node <ascend-node> ascend=on --overwrite

Run this for each target node. The label selects where the plugin runs; global or node configuration determines the slicing mode.

Prepare plugin values​

Find the namespace and name of the ConfigMap containing device-config.yaml:

kubectl get configmap -A \
-l app.kubernetes.io/component=hami-scheduler \
-o go-template='{{range .items}}{{if index .data "device-config.yaml"}}{{.metadata.namespace}}{{"\t"}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}'

For a HAMi release named hami, the ConfigMap is usually hami-scheduler-device. Install the plugin in the same namespace as this ConfigMap.

Save the following as ascend-plugin-values.yaml. These values belong to the Ascend plugin chart and are maintained separately from HAMi values. Adjust the ConfigMap name to match the discovery output:

config:
create: false
existingDeviceConfigMapName: hami-scheduler-device
runtimeClass:
create: true
name: ascend
handler: ascend
nodeSelector:
ascend: "on"
nodeConfig: |-
nodes: []

The chart creates RBAC, a ServiceAccount, a DaemonSet, a RuntimeClass, and a node ConfigMap. If a suitable ascend RuntimeClass already exists, set runtimeClass.create: false. With nodes: [], nodes inherit the global HAMi mode.

caution

config.create: false reuses HAMi's global device configuration. Manage that configuration through the HAMi release instead of applying another device ConfigMap. In this mode, the plugin chart's hamiVnpuMode, deviceSplitCount, and hamiVnpuCore.defaultPolicy do not override HAMi configuration.

Configure node modes and device counts​

For a cluster using multiple modes, replace nodeConfig in the plugin values with the actual node list:

nodeConfig: |-
nodes:
- name: ascend-node-1
hamiVnpuMode: template
- name: ascend-node-2
hamiVnpuMode: hami-core
hamiCorePolicy: spatial
vDeviceCount: 8
- name: ascend-node-3
hamiVnpuMode: enpu
vDeviceCount: 8

Node modes override the global mode. A positive vDeviceCount overrides the advertised count, with ENPU still capped at 100. Names must match Kubernetes nodes. nodeConfig is replaced as a complete YAML string, so preserve other entries. Chart updates to it trigger a plugin rollout. Each node must meet its mode requirements; do not mix hami-core and ENPU on one physical DIE.

Install and verify the plugin​

Upgrading existing hami-core workloads

Follow the Chart 0.4.0 upgrade guidance to stop affected workloads, migrate local shared-memory state, and recreate workloads after upgrading the plugin. A plugin rollout alone does not update the runtime library or mounts in running containers.

Set the namespace to the actual HAMi namespace, then install:

HAMI_NAMESPACE=hami-system
helm upgrade --install ascend-device-plugin \
oci://ghcr.io/dynamia-ai/charts/ascend-device-plugin \
--version 0.4.0 \
--namespace "$HAMI_NAMESPACE" \
--values ascend-plugin-values.yaml

The chart defaults to ghcr.io/dynamia-ai/ascend-device-plugin:v1.5.0. Use image.repository and image.tag for another registry; image.digest takes precedence over the tag. Preserve existing settings in the values file during upgrades.

kubectl -n "$HAMI_NAMESPACE" rollout status \
daemonset/hami-ascend-device-plugin --timeout=180s
kubectl -n "$HAMI_NAMESPACE" get pods \
-l app.kubernetes.io/instance=ascend-device-plugin -o wide
kubectl get node <ascend-node> -o jsonpath='{.status.allocatable}{"\n"}'

Each target node should have a ready plugin Pod and advertise a huawei.com/* device count for its actual model. Soft slicing advertises 10 virtual device slots per healthy NPU by default; node-level vDeviceCount can override this count. Available slots do not guarantee enough memory or compute for that many concurrent requests.

After HAMi updates the shared device ConfigMap, restart the plugin to reload it:

kubectl -n "$HAMI_NAMESPACE" rollout restart daemonset/hami-ascend-device-plugin
kubectl -n "$HAMI_NAMESPACE" rollout status \
daemonset/hami-ascend-device-plugin --timeout=180s

Run Huawei Ascend workloads​

The examples use scheduler hami-scheduler and RuntimeClass ascend; adjust them for the deployment. Replace workload image placeholders and ensure CANN, driver, and slicing runtime compatibility. PyTorch compute examples also require torch_npu.

Resource names come from vnpus.configs in the shared device-config.yaml and must match the actual chip model. For Ascend 910B3:

ResourceDescription
huawei.com/Ascend910B3Number of requested devices
huawei.com/Ascend910B3-memoryMemory quota per device, in MiB
huawei.com/Ascend910B3-coreCompute share per device for soft slicing, with positive values from 1 to 100; enforcement depends on the policy

Requesting only the device count uses the full device memory. To share one device, each container requests one device and an explicit memory quota. In hami-core mode, omitting -core or setting it to 0 reserves no compute in the scheduler and gives the runtime a limit of 100. To request a compute share, set it explicitly and add huawei.com/vnpu-mode: hami-core.

For multiple devices in hami-core mode, memory and compute quotas apply independently to each device. A count of 2 with -memory: 16384 means 16 GiB on each device. All selected devices must have identical quotas.

The bundled configuration also includes resource names for Ascend910A, Ascend910B2, Ascend910B4-1, Ascend910B4, Ascend310P, and Ascend910C. Before requesting compute, confirm that the model entry contains resourceCoreName and that the scheduler and plugin use the same definition. A model entry does not imply support for every slicing backend.

template mode​

Run these Pods on template-mode nodes. The 910B3 example requests 16 GiB and the 310P example requests 3 GiB, matching bundled template capacities. Pods without a mode annotation can also match hami-core nodes; use a node selector or affinity to restrict these examples to template nodes in a mixed-mode cluster.

Ascend 910B3​

apiVersion: v1
kind: Pod
metadata:
name: ascend-template-910b3
spec:
schedulerName: hami-scheduler
runtimeClassName: ascend
containers:
- name: workload
image: registry.example.com/your-project/ascend-workload:replace-me
command: ["sh", "-c", "sleep 86400"]
resources:
limits:
huawei.com/Ascend910B3: "1"
huawei.com/Ascend910B3-memory: "16384"

Ascend 310P​

apiVersion: v1
kind: Pod
metadata:
name: ascend-template-310p
spec:
schedulerName: hami-scheduler
runtimeClassName: ascend
containers:
- name: workload
image: registry.example.com/your-project/ascend-workload:replace-me
command: ["sh", "-c", "sleep 86400"]
resources:
limits:
huawei.com/Ascend310P: "1"
huawei.com/Ascend310P-memory: "3072"

hami-core mode​

Single-device memory and compute quotas​

This test targets a hami-core node and requests 4 GiB of memory and a 50% compute share on an Ascend 910B3. It checks quota injection and a real NPU matrix multiplication. Replace all three resource names for another model and confirm the HAMi scheduler name; the example uses hami-scheduler.

Prepare a workload image containing compatible CANN, PyTorch, and torch_npu. Replace the image placeholder and save this as ascend-smoke.yaml:

apiVersion: v1
kind: Pod
metadata:
name: ascend-smoke
annotations:
huawei.com/vnpu-mode: hami-core
huawei.com/npu-sched-policy: spatial
spec:
schedulerName: hami-scheduler
runtimeClassName: ascend
restartPolicy: Never
containers:
- name: pytorch
image: registry.example.com/your-project/pytorch-ascend:replace-me
env:
- name: TZ
value: UTC
command: ["python3", "-u", "-c"]
args:
- |
import os
import torch
import torch_npu
for key in ("NPU_MEMORY_LIMIT_MB", "NPU_COMPUTE_LIMIT", "NPU_SCHED_POLICY"):
print(key, os.environ.get(key))
with open("/proc/self/maps") as maps:
assert "/hami-vnpu-core/libvnpu.so" in maps.read(), "libvnpu is not loaded"
print("libvnpu loaded")
torch.npu.set_device(0)
a = torch.ones((1024, 1024), device="npu:0")
result = torch.mm(a, a)
torch.npu.synchronize()
value = result[0, 0].item()
assert value == 1024.0, value
print("NPU result:", value)
resources:
limits:
huawei.com/Ascend910B3: "1"
huawei.com/Ascend910B3-memory: "4096"
huawei.com/Ascend910B3-core: "50"

Create the Pod and check the result:

kubectl apply -f ascend-smoke.yaml
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/ascend-smoke --timeout=300s
kubectl logs ascend-smoke
CheckExpected
Pod phaseSucceeded
NPU_MEMORY_LIMIT_MB4096
NPU_COMPUTE_LIMIT50
NPU_SCHED_POLICYspatial
Runtime librarylibvnpu loaded
Compute resultNPU result: 1024.0

This verifies scheduling, runtime injection, and NPU compute. It does not verify rejection above the memory limit, isolation between workloads sharing a device, or throughput ratios. Clean up after the test:

kubectl delete pod ascend-smoke

Compute policies​

The default policy is spatial. Precedence is the Pod annotation huawei.com/npu-sched-policy, then node-level hamiCorePolicy, then global vnpus.hamiCorePolicy.

PolicyBehavior
spatialSets cooperative Cube/Vector core budgets through CANN; this is not an exclusive hardware partition and does not guarantee proportional inference throughput
dynamicSchedules time by relative shares and allows borrowing idle compute
fixedControls the target percentage through run and sleep intervals
overdraftAdds bounded borrowing while idle to dynamic time scheduling

Keep the plugin chart value hamiVnpuCore.coreUtilizationPolicy at default to use these policies. An explicit container NPU_CORE_UTILIZATION_POLICY takes precedence.

Multi-device parallel inference​

Keep huawei.com/vnpu-mode: hami-core on multi-device workloads and specify quotas per device. For example, request 32 GiB and a 50% compute share on each of two 64 GiB 910B3 devices:

resources:
limits:
huawei.com/Ascend910B3: "2"
huawei.com/Ascend910B3-memory: "32768"
huawei.com/Ascend910B3-core: "50"

This requests 64 GiB in total. For TP=2 inference with vLLM-Ascend, size the memory budget using the per-device ratio of 32/64 and leave room for runtime overhead. Prepare the workload image, model files, and storage separately.

enpu mode​

This example requests 16 GiB of memory and a 20% compute share on a 910B3. The elastic policy can borrow idle compute, so 20% is not a strict ceiling. The image must meet the ENPU requirements above and include PyTorch and torch_npu.

apiVersion: v1
kind: Pod
metadata:
name: ascend-enpu-smoke
annotations:
huawei.com/vnpu-mode: enpu
huawei.com/enpu-policy: elastic
spec:
schedulerName: hami-scheduler
runtimeClassName: ascend
restartPolicy: Never
containers:
- name: pytorch
image: registry.example.com/your-project/pytorch-ascend:replace-me
command: ["python3", "-u", "-c"]
args:
- |
import torch
import torch_npu
torch.npu.set_device(0)
a = torch.ones((1024, 1024), device="npu:0")
result = torch.mm(a, a)
torch.npu.synchronize()
value = result[0, 0].item()
assert value == 1024.0, value
print("ENPU result:", value)
resources:
limits:
huawei.com/Ascend910B3: "1"
huawei.com/Ascend910B3-memory: "16384"
huawei.com/Ascend910B3-core: "20"

Save as ascend-enpu-smoke.yaml, then run and check the output:

kubectl apply -f ascend-enpu-smoke.yaml
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/ascend-enpu-smoke --timeout=300s
kubectl logs ascend-enpu-smoke
kubectl delete pod ascend-enpu-smoke

Expect ENPU result: 1024.0. For 910C, replace all three resource names and verify independent DIE mode. Use logical device npu:0 inside the container.

Usage notes​

  • Resource names must match the shared configuration and actual model. A model definition does not imply support for every backend.
  • Template examples request one device per container. hami-core supports multiple devices with per-device quotas; ENPU supports one physical DIE per container.
  • The template examples use sleep to allow allocation inspection and do not perform NPU compute. The hami-core and ENPU examples check the compute path, not sharing isolation or performance.
  • Recreate workload Pods as required when replacing runtime libraries, switching modes, or renewing licenses. Restarting an application process alone does not guarantee a new allocation.

Monitoring​

NPU hardware metrics​

Deploy npu-exporter using the AscendHub installation instructions and configure Prometheus to scrape it. Like NVIDIA's DCGM Exporter, npu-exporter collects hardware metrics for Ascend NPUs. Select a version suitable for the device model and driver, and follow the official deployment and scrape configuration instructions.

Soft-slicing and container metrics​

Ascend device plugin provides soft-slicing and container-level metrics that complement the hardware metrics from npu-exporter. The vnpuMonitor values below configure metrics collection resources for the plugin itself; they do not install npu-exporter.

hami-core and ENPU nodes expose metrics at :9395/metrics. Select the plugin Pod on the target node and forward its port in one terminal:

kubectl -n "$HAMI_NAMESPACE" get pods \
-l app.kubernetes.io/instance=ascend-device-plugin -o wide
kubectl -n "$HAMI_NAMESPACE" port-forward "pod/<plugin-pod>" 9395:9395

Check the endpoint in another terminal:

curl -fsS http://localhost:9395/metrics

With Prometheus Operator, set vnpuMonitor.enabled=true. This creates a ServiceMonitor and PrometheusRule by default. Install the corresponding CRDs first, and adjust the namespaces and discovery labels under vnpuMonitor.serviceMonitor.* and vnpuMonitor.prometheusRule.*. To create only the Service, set both create values to false.

Troubleshooting​

kubectl -n "$HAMI_NAMESPACE" logs \
-l app.kubernetes.io/instance=ascend-device-plugin \
-c device-plugin --tail=200
kubectl describe pod ascend-smoke
SymptomCheck
No plugin PodNode label ascend=on, scheduling taints, and available resources
ConfigMap mount failureConfigMap name, namespace, and the device-config.yaml key
RuntimeClass or handler errorThe ascend RuntimeClass and node container runtime handler match
Startup failure or no device resourcesDriver, model configuration, device filters, and device-share errors in logs; confirm npu-smi is accessible inside the plugin container
Workload Pending or allocation failureActual resource names, node mode, HAMi scheduler name, remaining memory, and slot count
License errorValid grants for the allocated devices in Node annotation hami.io/ascend-license; workload Pods recreated after renewal
ENPU allocation failureCANN/runtime compatibility, systemd-detect-virt, independent DIE mode on 910C, and runtime file conflicts

Uninstall​

Stop Ascend workloads that depend on this plugin before removing its release:

helm uninstall ascend-device-plugin --namespace "$HAMI_NAMESPACE"

Uninstalling does not delete the shared device ConfigMap managed by HAMi or automatically remove runtime libraries and data on the nodes. A RuntimeClass managed by this chart is deleted with the release; check that other workloads do not depend on it before uninstalling.