Huawei Ascend
Ascend Device Plugin registers Huawei Ascend NPUs as Kubernetes resources. The HAMi scheduler selects devices for workloads. This guide uses Chart 0.4.0 and application v1.5.0 to install the plugin alongside HAMi, configure slicing modes, and verify a workload.
The plugin supports three modes: template (hardware templates), hami-core (runtime soft slicing), and enpu. Check the requirements for the selected mode, configure HAMi scheduling, deploy the standalone plugin, and run a workload.
Prerequisites
- Kubernetes 1.20 or later, and Helm 3 with OCI chart support.
- HAMi Enterprise is installed. See Installation for deployment options. Ascend device plugin and the NVIDIA (HAMi) device plugin can coexist in the same cluster, managing Ascend NPUs and NVIDIA GPUs respectively.
- Complete HAMi Enterprise license activation before use. This requirement applies to all slicing modes described in this guide.
- Ascend drivers and ascend-docker-runtime installed on each Ascend node, with an
ascendhandler configured in the container runtime. Creating a RuntimeClass does not install or configure the handler. npu-smiis available on the node and accessible in the plugin container through the driver directory or the/usr/local/sbinmount.
Ascend driver and CANN versions
The HAMi integration requires Ascend driver 25.5 or later on the node. Prepare CANN in the workload image for the selected slicing backend:
| Mode | CANN requirement |
|---|---|
template | Use a workload CANN version compatible with the node driver and device model. The soft-slicing runtime build versions below do not apply to template mode. |
hami-core | The bundled libvnpu runtime is built with CANN 8.5.2. The workload image's CANN environment must be compatible with this runtime. |
enpu | The bundled ubs-virt-enpu/vCANN-RT 1.0.0 runtime is built with CANN 9.1.0. The runtime artifacts must match the workload image's CANN version and meet the corresponding CANN/HDK requirements. |
CANN 8.5.2 and 9.1.0 identify the two runtimes' build environments. Compatibility with other CANN versions requires validation of the node driver, workload image, and selected runtime together. Confirm this combination before deploying workloads or upgrading CANN.
template mode
The driver must support virtualization templates for the device model. HAMi selects the smallest template that satisfies the memory request. Template names and capacities in the device configuration must match the driver.
hami-core mode
This mode uses libvnpu to control memory and compute. It currently requires ARM nodes and the HAMi scheduler. The driver and workload CANN environment must meet the version requirements above.
The plugin enables device-share for chips during startup on hami-core nodes. If a busy chip temporarily refuses the switch, the plugin withholds that chip from kubelet and HAMi, then retries after the workload finishes. Other chips remain available. Other switching errors still cause plugin startup to fail.
enpu mode
The integration supports A2/910B and A3/910C. Enable device sharing as required by the paired runtime. Each container can request a share of only one physical DIE.
Set node mode hamiVnpuMode: enpu, and use huawei.com/vnpu-mode: enpu and runtimeClassName: ascend on workload
Pods. Prepare vCANN-RT assets compatible with CANN/HDK. Workload images also need /usr/bin/systemd-detect-virt and its
dependencies. A3/910C nodes must use independent DIE mode (INDEP_POLICY); A2/910B does not need that setting.
The plugin installs runtime assets from its image into /usr/local/enpu/vcann-rt on the node. If existing files differ,
it refuses to overwrite them and fails startup. Before replacing runtime assets, stop ENPU workloads and pause the
device plugin on the target node. Back up and move these three files out of that directory: lib/libvruntime.so,
tools/enpu-monitor, and ld.so.preload. Keep the hami-core runtime assets. Update the plugin image through the
existing Helm release, then resume the plugin to install the new files. To roll back, stop workloads and pause the
plugin again, restore the backed-up files and previous plugin image, then resume the plugin.
The bundled ENPU runtime provides basic soft slicing without mem-swap. Basic soft slicing does not require enpu-manager.
For mem-swap, prepare a runtime that supports it, deploy a compatible enpu-manager separately and configure
enpu.managerURL and enpu.managerConfigRoot. managerURL must reach the manager on the plugin's own node. The plugin
does not use hostNetwork, so do not use 127.0.0.1 or a Service that can route to another node.
Configure Ascend scheduling
These values belong to the HAMi chart. Save the configuration for the selected mode as hami-ascend-values.yaml,
merge it with the existing HAMi release values, and update the deployment. All three modes require
devices.ascend.enabled: true; this does not install the Ascend plugin. Confirm that the HAMi version supports fields
such as hamiVnpuMode before using them.
template configuration
devices:
ascend:
enabled: true
runtimeClassName: ascend
hamiVnpuMode: template
Template mode derives capacity from device templates. HAMi manages model definitions, resource names, and templates.
hami-core configuration
devices:
ascend:
enabled: true
runtimeClassName: ascend
hamiVnpuMode: hami-core
deviceSplitCount: 10
hamiCorePolicy: spatial
deviceSplitCount limits the number of virtual devices per physical NPU and defaults to 10. Workload Pods still
request their memory and compute shares.
enpu configuration
devices:
ascend:
enabled: true
runtimeClassName: ascend
hamiVnpuMode: enpu
deviceSplitCount: 10
enpuPolicy: elastic
ENPU caps the effective slot count at 100. enpuPolicy supports fixed-share, elastic, and best-effort. Pods can
override the default with huawei.com/enpu-policy; workloads on the same physical DIE must use the same policy.
Update HAMi configuration
See Installation for installation and upgrade options, and merge the file into the values
used by that deployment method. runtimeClassName: ascend lets the HAMi webhook add a RuntimeClass to Ascend workloads;
its name must match the resource created or reused by the plugin.
An explicit mode takes precedence over the legacy hamiVnpuCore boolean, and node modes can override the global mode.
Stop or reschedule affected workloads before switching modes.
Deploy Ascend Device Plugin
Deploy the plugin in the same namespace as HAMi. If another Helm release or raw YAML already manages the plugin, update that deployment to avoid duplicate DaemonSets, RBAC resources, and node ConfigMaps.
Label device nodes
kubectl label node <ascend-node> ascend=on --overwrite
Run this for each target node. The label selects where the plugin runs; global or node configuration determines the slicing mode.
Prepare plugin values
Find the namespace and name of the ConfigMap containing device-config.yaml:
kubectl get configmap -A \
-l app.kubernetes.io/component=hami-scheduler \
-o go-template='{{range .items}}{{if index .data "device-config.yaml"}}{{.metadata.namespace}}{{"\t"}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}'
For a HAMi release named hami, the ConfigMap is usually hami-scheduler-device. Install the plugin in the same
namespace as this ConfigMap.
Save the following as ascend-plugin-values.yaml. These values belong to the Ascend plugin chart and are maintained
separately from HAMi values. Adjust the ConfigMap name to match the discovery output:
config:
create: false
existingDeviceConfigMapName: hami-scheduler-device
runtimeClass:
create: true
name: ascend
handler: ascend
nodeSelector:
ascend: "on"
nodeConfig: |-
nodes: []
The chart creates RBAC, a ServiceAccount, a DaemonSet, a RuntimeClass, and a node ConfigMap. If a suitable ascend
RuntimeClass already exists, set runtimeClass.create: false. With nodes: [], nodes inherit the global HAMi mode.
config.create: false reuses HAMi's global device configuration. Manage that configuration through the HAMi release
instead of applying another device ConfigMap. In this mode, the plugin chart's hamiVnpuMode, deviceSplitCount, and
hamiVnpuCore.defaultPolicy do not override HAMi configuration.
Configure node modes and device counts
For a cluster using multiple modes, replace nodeConfig in the plugin values with the actual node list:
nodeConfig: |-
nodes:
- name: ascend-node-1
hamiVnpuMode: template
- name: ascend-node-2
hamiVnpuMode: hami-core
hamiCorePolicy: spatial
vDeviceCount: 8
- name: ascend-node-3
hamiVnpuMode: enpu
vDeviceCount: 8
Node modes override the global mode. A positive vDeviceCount overrides the advertised count, with ENPU still capped at
100. Names must match Kubernetes nodes. nodeConfig is replaced as a complete YAML string, so preserve other entries.
Chart updates to it trigger a plugin rollout. Each node must meet its mode requirements; do not mix hami-core and ENPU
on one physical DIE.
Install and verify the plugin
Follow the Chart 0.4.0 upgrade guidance to stop affected workloads, migrate local shared-memory state, and recreate workloads after upgrading the plugin. A plugin rollout alone does not update the runtime library or mounts in running containers.
Set the namespace to the actual HAMi namespace, then install:
HAMI_NAMESPACE=hami-system
helm upgrade --install ascend-device-plugin \
oci://ghcr.io/dynamia-ai/charts/ascend-device-plugin \
--version 0.4.0 \
--namespace "$HAMI_NAMESPACE" \
--values ascend-plugin-values.yaml
The chart defaults to ghcr.io/dynamia-ai/ascend-device-plugin:v1.5.0. Use image.repository and image.tag for
another registry; image.digest takes precedence over the tag. Preserve existing settings in the values file during
upgrades.
kubectl -n "$HAMI_NAMESPACE" rollout status \
daemonset/hami-ascend-device-plugin --timeout=180s
kubectl -n "$HAMI_NAMESPACE" get pods \
-l app.kubernetes.io/instance=ascend-device-plugin -o wide
kubectl get node <ascend-node> -o jsonpath='{.status.allocatable}{"\n"}'
Each target node should have a ready plugin Pod and advertise a huawei.com/* device count for its actual model. Soft
slicing advertises 10 virtual device slots per healthy NPU by default; node-level vDeviceCount can override this
count. Available slots do not guarantee enough memory or compute for that many concurrent requests.
After HAMi updates the shared device ConfigMap, restart the plugin to reload it:
kubectl -n "$HAMI_NAMESPACE" rollout restart daemonset/hami-ascend-device-plugin
kubectl -n "$HAMI_NAMESPACE" rollout status \
daemonset/hami-ascend-device-plugin --timeout=180s
Run Huawei Ascend workloads
The examples use scheduler hami-scheduler and RuntimeClass ascend; adjust them for the deployment. Replace workload
image placeholders and ensure CANN, driver, and slicing runtime compatibility. PyTorch compute examples also require
torch_npu.
Resource names come from vnpus.configs in the shared device-config.yaml and must match the actual chip model. For
Ascend 910B3:
| Resource | Description |
|---|---|
huawei.com/Ascend910B3 | Number of requested devices |
huawei.com/Ascend910B3-memory | Memory quota per device, in MiB |
huawei.com/Ascend910B3-core | Compute share per device for soft slicing, with positive values from 1 to 100; enforcement depends on the policy |
Requesting only the device count uses the full device memory. To share one device, each container requests one device
and an explicit memory quota. In hami-core mode, omitting -core or setting it to 0 reserves no compute in the
scheduler and gives the runtime a limit of 100. To request a compute share, set it explicitly and add
huawei.com/vnpu-mode: hami-core.
For multiple devices in hami-core mode, memory and compute quotas apply independently to each device. A count of 2
with -memory: 16384 means 16 GiB on each device. All selected devices must have identical quotas.
The bundled configuration also includes resource names for Ascend910A, Ascend910B2, Ascend910B4-1, Ascend910B4,
Ascend310P, and Ascend910C. Before requesting compute, confirm that the model entry contains resourceCoreName and
that the scheduler and plugin use the same definition. A model entry does not imply support for every slicing backend.
template mode
Run these Pods on template-mode nodes. The 910B3 example requests 16 GiB and the 310P example requests 3 GiB, matching bundled template capacities. Pods without a mode annotation can also match hami-core nodes; use a node selector or affinity to restrict these examples to template nodes in a mixed-mode cluster.
Ascend 910B3
apiVersion: v1
kind: Pod
metadata:
name: ascend-template-910b3
spec:
schedulerName: hami-scheduler
runtimeClassName: ascend
containers:
- name: workload
image: registry.example.com/your-project/ascend-workload:replace-me
command: ["sh", "-c", "sleep 86400"]
resources:
limits:
huawei.com/Ascend910B3: "1"
huawei.com/Ascend910B3-memory: "16384"
Ascend 310P
apiVersion: v1
kind: Pod
metadata:
name: ascend-template-310p
spec:
schedulerName: hami-scheduler
runtimeClassName: ascend
containers:
- name: workload
image: registry.example.com/your-project/ascend-workload:replace-me
command: ["sh", "-c", "sleep 86400"]
resources:
limits:
huawei.com/Ascend310P: "1"
huawei.com/Ascend310P-memory: "3072"
hami-core mode
Single-device memory and compute quotas
This test targets a hami-core node and requests 4 GiB of memory and a 50% compute share on an Ascend 910B3. It checks
quota injection and a real NPU matrix multiplication. Replace all three resource names for another model and confirm the
HAMi scheduler name; the example uses hami-scheduler.
Prepare a workload image containing compatible CANN, PyTorch, and torch_npu. Replace the image placeholder and save
this as ascend-smoke.yaml:
apiVersion: v1
kind: Pod
metadata:
name: ascend-smoke
annotations:
huawei.com/vnpu-mode: hami-core
huawei.com/npu-sched-policy: spatial
spec:
schedulerName: hami-scheduler
runtimeClassName: ascend
restartPolicy: Never
containers:
- name: pytorch
image: registry.example.com/your-project/pytorch-ascend:replace-me
env:
- name: TZ
value: UTC
command: ["python3", "-u", "-c"]
args:
- |
import os
import torch
import torch_npu
for key in ("NPU_MEMORY_LIMIT_MB", "NPU_COMPUTE_LIMIT", "NPU_SCHED_POLICY"):
print(key, os.environ.get(key))
with open("/proc/self/maps") as maps:
assert "/hami-vnpu-core/libvnpu.so" in maps.read(), "libvnpu is not loaded"
print("libvnpu loaded")
torch.npu.set_device(0)
a = torch.ones((1024, 1024), device="npu:0")
result = torch.mm(a, a)
torch.npu.synchronize()
value = result[0, 0].item()
assert value == 1024.0, value
print("NPU result:", value)
resources:
limits:
huawei.com/Ascend910B3: "1"
huawei.com/Ascend910B3-memory: "4096"
huawei.com/Ascend910B3-core: "50"
Create the Pod and check the result:
kubectl apply -f ascend-smoke.yaml
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/ascend-smoke --timeout=300s
kubectl logs ascend-smoke
| Check | Expected |
|---|---|
| Pod phase | Succeeded |
NPU_MEMORY_LIMIT_MB | 4096 |
NPU_COMPUTE_LIMIT | 50 |
NPU_SCHED_POLICY | spatial |
| Runtime library | libvnpu loaded |
| Compute result | NPU result: 1024.0 |
This verifies scheduling, runtime injection, and NPU compute. It does not verify rejection above the memory limit, isolation between workloads sharing a device, or throughput ratios. Clean up after the test:
kubectl delete pod ascend-smoke
Compute policies
The default policy is spatial. Precedence is the Pod annotation huawei.com/npu-sched-policy, then node-level
hamiCorePolicy, then global vnpus.hamiCorePolicy.
| Policy | Behavior |
|---|---|
spatial | Sets cooperative Cube/Vector core budgets through CANN; this is not an exclusive hardware partition and does not guarantee proportional inference throughput |
dynamic | Schedules time by relative shares and allows borrowing idle compute |
fixed | Controls the target percentage through run and sleep intervals |
overdraft | Adds bounded borrowing while idle to dynamic time scheduling |
Keep the plugin chart value hamiVnpuCore.coreUtilizationPolicy at default to use these policies. An explicit
container NPU_CORE_UTILIZATION_POLICY takes precedence.
Multi-device parallel inference
Keep huawei.com/vnpu-mode: hami-core on multi-device workloads and specify quotas per device. For example, request 32
GiB and a 50% compute share on each of two 64 GiB 910B3 devices:
resources:
limits:
huawei.com/Ascend910B3: "2"
huawei.com/Ascend910B3-memory: "32768"
huawei.com/Ascend910B3-core: "50"
This requests 64 GiB in total. For TP=2 inference with vLLM-Ascend, size the memory budget using the per-device ratio of 32/64 and leave room for runtime overhead. Prepare the workload image, model files, and storage separately.
enpu mode
This example requests 16 GiB of memory and a 20% compute share on a 910B3. The elastic policy can borrow idle compute,
so 20% is not a strict ceiling. The image must meet the ENPU requirements above and include PyTorch and torch_npu.
apiVersion: v1
kind: Pod
metadata:
name: ascend-enpu-smoke
annotations:
huawei.com/vnpu-mode: enpu
huawei.com/enpu-policy: elastic
spec:
schedulerName: hami-scheduler
runtimeClassName: ascend
restartPolicy: Never
containers:
- name: pytorch
image: registry.example.com/your-project/pytorch-ascend:replace-me
command: ["python3", "-u", "-c"]
args:
- |
import torch
import torch_npu
torch.npu.set_device(0)
a = torch.ones((1024, 1024), device="npu:0")
result = torch.mm(a, a)
torch.npu.synchronize()
value = result[0, 0].item()
assert value == 1024.0, value
print("ENPU result:", value)
resources:
limits:
huawei.com/Ascend910B3: "1"
huawei.com/Ascend910B3-memory: "16384"
huawei.com/Ascend910B3-core: "20"
Save as ascend-enpu-smoke.yaml, then run and check the output:
kubectl apply -f ascend-enpu-smoke.yaml
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/ascend-enpu-smoke --timeout=300s
kubectl logs ascend-enpu-smoke
kubectl delete pod ascend-enpu-smoke
Expect ENPU result: 1024.0. For 910C, replace all three resource names and verify independent DIE mode. Use logical
device npu:0 inside the container.
Usage notes
- Resource names must match the shared configuration and actual model. A model definition does not imply support for every backend.
- Template examples request one device per container. hami-core supports multiple devices with per-device quotas; ENPU supports one physical DIE per container.
- The template examples use
sleepto allow allocation inspection and do not perform NPU compute. The hami-core and ENPU examples check the compute path, not sharing isolation or performance. - Recreate workload Pods as required when replacing runtime libraries, switching modes, or renewing licenses. Restarting an application process alone does not guarantee a new allocation.
Monitoring
NPU hardware metrics
Deploy npu-exporter using the AscendHub installation instructions and configure Prometheus to scrape it. Like NVIDIA's DCGM Exporter, npu-exporter collects hardware metrics for Ascend NPUs. Select a version suitable for the device model and driver, and follow the official deployment and scrape configuration instructions.
Soft-slicing and container metrics
Ascend device plugin provides soft-slicing and container-level metrics that complement the hardware metrics from npu-exporter. The vnpuMonitor values below configure metrics collection resources for the plugin itself; they do not install npu-exporter.
hami-core and ENPU nodes expose metrics at :9395/metrics. Select the plugin Pod on the target node and forward its
port in one terminal:
kubectl -n "$HAMI_NAMESPACE" get pods \
-l app.kubernetes.io/instance=ascend-device-plugin -o wide
kubectl -n "$HAMI_NAMESPACE" port-forward "pod/<plugin-pod>" 9395:9395
Check the endpoint in another terminal:
curl -fsS http://localhost:9395/metrics
With Prometheus Operator, set vnpuMonitor.enabled=true. This creates a ServiceMonitor and PrometheusRule by default.
Install the corresponding CRDs first, and adjust the namespaces and discovery labels under
vnpuMonitor.serviceMonitor.* and vnpuMonitor.prometheusRule.*. To create only the Service, set both create values
to false.
Troubleshooting
kubectl -n "$HAMI_NAMESPACE" logs \
-l app.kubernetes.io/instance=ascend-device-plugin \
-c device-plugin --tail=200
kubectl describe pod ascend-smoke
| Symptom | Check |
|---|---|
| No plugin Pod | Node label ascend=on, scheduling taints, and available resources |
| ConfigMap mount failure | ConfigMap name, namespace, and the device-config.yaml key |
| RuntimeClass or handler error | The ascend RuntimeClass and node container runtime handler match |
| Startup failure or no device resources | Driver, model configuration, device filters, and device-share errors in logs; confirm npu-smi is accessible inside the plugin container |
| Workload Pending or allocation failure | Actual resource names, node mode, HAMi scheduler name, remaining memory, and slot count |
| License error | Valid grants for the allocated devices in Node annotation hami.io/ascend-license; workload Pods recreated after renewal |
| ENPU allocation failure | CANN/runtime compatibility, systemd-detect-virt, independent DIE mode on 910C, and runtime file conflicts |
Uninstall
Stop Ascend workloads that depend on this plugin before removing its release:
helm uninstall ascend-device-plugin --namespace "$HAMI_NAMESPACE"
Uninstalling does not delete the shared device ConfigMap managed by HAMi or automatically remove runtime libraries and data on the nodes. A RuntimeClass managed by this chart is deleted with the release; check that other workloads do not depend on it before uninstalling.