Dynamia ascend-device-plugin Chart 0.4.0
Release date: 2026-11-09
Chart 0.4.0 delivers application v1.5.0 with three Enterprise Ascend capabilities on the open-source baselines: dynamic memory, Turbo mode, and task priority. They address runtime memory growth, full compute access for exclusive workloads, and prioritized execution on a shared NPU. Fixes and hardware support relative to the previous release are listed under Changes.
| Version | Previous | Current |
|---|---|---|
| Chart | 0.3.1 | 0.4.0 |
| appVersion | v1.4.3 | v1.5.0 |
Open-source baselines
This release integrates features from the following open-source baselines and bundles the commercial libvnpu v1.5.0 runtime. Deployment configuration, resource management, and HAMi Enterprise integration retain their commercial implementations.
| Component | Open-source baseline | Delivery scope |
|---|---|---|
| Ascend Device Plugin | Project-HAMi/ascend-device-plugin v1.5.0, reviewed through 9325759 | Template-based partitioning, per-card quotas across multiple NPUs in hami-core, ENPU integration, and the device-management fixes and Ascend950DT whole-card allocation described below. |
| hami-vnpu-core | a4f52f0 | The preload, memory-query, compute-timing, and shared-state permission fixes listed below, with Enterprise compute policies, task priority, and memory expansion retained. |
Multi-card soft partitioning and ENPU were already delivered in Chart 0.3.1, with upstream sources ascend-device-plugin #147 and hami-vnpu-core #44. This release retains those capabilities; additions and fixes are listed under Changes.
HAMi Enterprise compatibility
| Feature or mode | Minimum HAMi Enterprise version | Companion requirements and upgrade guidance |
|---|---|---|
Template-based hardware partitioning and Ascend950DT whole-card allocation, with the scheduler and plugin sharing vnpus.configs | v2.9.0-rc0 | The existing scheduler can be retained. Ascend950DT requires an additional device configuration and template mode on the node. Upgrading the plugin Chart alone does not update the configuration managed by HAMi. |
Static soft partitioning with explicit huawei.com/vnpu-mode: hami-core, including Enterprise compute policies, task priority, and Turbo | v2.10.0-r2 | Requires the companion runtime authorization introduced with application v1.4.3. Deployments already meeting the Chart 0.3.0 requirements can retain their scheduler; the runtime-state migration below is still required. |
ENPU scheduling or hami-core compute oversell with deviceCoreScaling greater than 1 | v2.10.0-r5 | Requires the corresponding scheduler backend and resource accounting, with matching shared-device and node configuration. |
| Automatic memory expansion | v2.10.0-r2 | Applies only to an ordinary container explicitly using hami-core with one allocated vNPU. Expansion approval and admission protection must be enabled; the companion Chart requires Kubernetes 1.30 or later. Disabled approval, rejection, or timeout does not increase the memory limit. |
Enterprise extensions
Dynamic memory, Turbo mode, and task priority are implemented by the Ascend plugin and commercial libvnpu together. Dynamic memory also requires HAMi Enterprise scheduler approval. Their release history is recorded in Chart 0.2.2; the following sections describe their use and companion requirements in this release.
All three use hami-core soft partitioning with the matching runtime injected. Meet the version and license requirements in the compatibility table before configuring them in workload containers.
Dynamic memory: expand at runtime
Dynamic memory serves workloads whose memory demand grows with input size or execution phase. A workload can start with an initial quota and request a higher limit during execution, reducing the need to reserve its entire peak requirement at startup.
When a controlled allocation would exceed the current quota, the runtime requests expansion. HAMi Enterprise checks the target against allocatable device memory and namespace quotas. After approval, the runtime rechecks the quota and attempts the allocation without restarting the application process to raise the limit. The target is rounded up to 1 GiB; the original Pod resource declaration and AI Core quota stay unchanged.
For example, a container with an initial 4 GiB quota can request 5 GiB when an allocation raises its required capacity to that amount. Approval depends on remaining allocatable memory and quota. Expansion is upward only; it does not swap memory out or guarantee capacity beyond the allocatable amount. Rejection, timeout, or an underlying allocation failure can still result in OOM or the native allocation error.
Merge this fragment into an existing workload Pod, requesting one vNPU and its initial memory quota:
metadata:
annotations:
huawei.com/vnpu-mode: hami-core
spec:
containers:
- name: workload
env:
- name: NPU_MEMORY_AUTO_ADJUST
value: "true"
Only ordinary workload containers with one allocated vNPU are eligible. Init containers and a single container assigned multiple vNPUs are outside the scope. The runtime waits up to 45 seconds for an allocation by default; NPU_MEMORY_AUTO_ADJUST_TIMEOUT_MS can change this limit, up to 300000 milliseconds. This setting does not extend the companion approval process's own deadline. See enable memory expansion for approval and admission configuration.
Turbo mode: full compute access for exclusive workloads
Turbo serves trusted, compute-exclusive training or batch workloads. It removes runtime compute throttling, submission waits, and compute timing so a workload can use the device's available compute. Memory limits, accounting, and approved dynamic memory expansion remain active. Turbo does not add physical compute capacity or reserve an exclusive device; the deployment must provide the required exclusive-use conditions.
Request one vNPU and 100% of AI Core for the workload container, then set:
env:
- name: NPU_TURBO_MODE
value: "true"
Turbo is disabled by default and must be configured before the application process starts. Recreate the Pod to change modes; all processes sharing a local memory-state region must use consistent settings. Turbo takes precedence over NPU_CORE_UTILIZATION_POLICY=force and does not participate in task-priority arbitration. Use the controlled configuration below when workloads need to share compute by priority.
Task priority: prioritized execution on a shared NPU
Task priority serves mixed workloads, such as inference and training, on the same physical NPU. When high-priority inference starts submitting compute, lower-priority training pauses further submissions to let inference run first. Training can continue when the higher-priority workload stops submitting. The handoff preserves application processes and allocated memory, without evicting or restarting the training workload.
Set NPU_TASK_PRIORITY in the workload container. Its default is 1; smaller unsigned integers have higher priority, so 0 outranks 1. It determines execution order among controlled tasks. Compute shares still come from resource allocation and compute policy; do not confuse it with the compute-share value NPU_PRIORITY.
For example, configure two containers sharing a physical NPU as follows, and use their resource requests to specify memory and compute shares:
# Training container
env:
- name: NPU_CORE_UTILIZATION_POLICY
value: force
- name: NPU_TASK_PRIORITY
value: "1"
# Inference container
env:
- name: NPU_CORE_UTILIZATION_POLICY
value: force
- name: NPU_TASK_PRIORITY
value: "0"
The example explicitly selects force for controlled time-slice execution while retaining the average duty-cycle limit derived from each allocation. A high-priority workload's rest interval still permits lower-priority work to run. The Chart's default default / spatial configuration uses spatial compute budgets and should not be treated as the time-slice priority configuration shown here.
Preemption occurs at intercepted compute-submission boundaries. Already submitted kernels or model tasks finish naturally, so fixed handoff latency is not guaranteed. This controls runtime compute order on a shared physical NPU; it does not replace Kubernetes Pod scheduling priority or free memory held by lower-priority tasks. Sustained high-priority demand can leave lower-priority tasks waiting indefinitely. This release does not provide independent priority arbitration for multiple devices assigned to one container.
Combining the capabilities
| Combination | Behavior and conditions |
|---|---|
| Dynamic memory and task priority | Can be combined. High-priority tasks still need available memory; raising priority does not reclaim another task's memory, and expansion remains subject to approval and capacity. |
| Dynamic memory and Turbo | Can be combined. Turbo removes compute control; memory limits and expansion approval still apply. |
| Turbo and task priority | Turbo tasks do not participate in priority arbitration. To share compute by priority, disable Turbo for participating tasks and use a controlled compute policy. NPU_CORE_UTILIZATION_POLICY=disable also skips priority arbitration. |
Deployment and monitoring also provide image pinning with image.digest and the vnpuMonitor metrics service, ServiceMonitor, and PrometheusRule configuration.
Changes
The following additions and fixes compare against Chart 0.3.1 / application v1.4.3.
Chart changes
- [Upstream sync] The bundled
deviceConfigadds Ascend950DT with resource namehuawei.com/Ascend950DT,98304MiB of memory,32AI cores, and an emptytemplateslist. With the defaultconfig.create=false, the deployment still uses an existing device ConfigMap. See the upgrade guidance to apply the new configuration. Source: ascend-device-plugin #149.
Application changes
- [Upstream sync] Corrects compute submission across multiple NPUs and contexts. Tasks follow the calling thread's current device and context, with timing state maintained separately for each context. Retiring a context preserves memory accounting for live processes on the same physical card. Multi-card allocation was already available in the previous release; this release corrects its runtime behavior. Source: hami-vnpu-core #44, with the related routing and context handling completed in this release.
- [Upstream sync] Loads CANN/DCMI dependencies when needed, fixing startup failures for ordinary commands such as
bashwhen the runtime library is preloaded before Ascend dependencies are available. Source: hami-vnpu-core #48. - [Upstream sync] Prevents memory queries during application initialization from initializing spatial compute limits too early. Memory queries preserve the native memory type, capacity, and error result while limiting successful results to the current card's quota. Small or empty native memory pools no longer appear to have more available memory than they actually provide. Sources: hami-vnpu-core #50, #56.
- [Upstream sync] Corrects compute scheduling waits and timing: planned scheduling windows are bounded, incomplete or delayed timing samples are excluded, and DCMI queries run in a separate process with timeouts. A pending request no longer replaces the stream used to measure the previous batch. Sources: hami-vnpu-core #38, #52; only the corresponding fixes are included in this release.
- [Upstream sync] Fixes
Permission deniederrors caused by global shared-state file permissions when containers with different user IDs share a physical NPU. New files permit access across users; existing files require the action described below. Source: hami-vnpu-core #57. - [Upstream sync] On
hami-corenodes, a busy chip that temporarily refuses to enable sharing no longer causes the entire plugin to exit at startup. The plugin withholds that chip from kubelet and HAMi while other chips remain available. It retries after the workload finishes and advertises the chip once the switch succeeds. Other types of switching errors still fail startup. Source: ascend-device-plugin #137. - [Upstream sync] Enables periodic reclamation of orphaned idle vNPUs by default, checking Pod state, kubelet allocation records, and driver state. The default check interval is
60seconds, with a10-minute grace period and3consecutive confirmations required. Devices are not reclaimed while a Pod is using them or allocation state is uncertain. Startup and restart cleanup use the same checks. Source: ascend-device-plugin #142. - [Upstream sync] Adds DCMI v2 discovery and whole-card allocation for Ascend950DT. Each physical card advertises one device and requires
templatemode with an empty template list. This release does not provide soft partitioning for this model. A transient discovery failure retains known devices as unhealthy so they are not allocated. Source: ascend-device-plugin #149.
Upgrade guidance
Migrate soft-partitioned workloads from Chart 0.3.1
The bundled libvnpu v1.5.0 changes the local shared-memory protocol from version 3 to 4. Existing local state cannot be reused directly. Rolling the device plugin alone does not update the library or mounts in running containers.
- Record each affected container's actual
NPU_LOCAL_SHM_PATH,NPU_GLOBAL_SHM_PATH, and host mount mappings. Stop the old soft-partitioned workloads being upgraded and confirm that every process using the local files to be migrated has exited. - Remove the old local state files owned by those workloads, or use fresh local paths. The default container path is
/hami-vnpu-shmem/vnpu_local_shmem.device-<physical-device-ID>, mapped to/usr/local/hami-vnpu-core/containers/<Pod-UID>_<container-name>/vnpu_local_shmem.device-<physical-device-ID>on the host. Follow the actual mounts ifHOOK_PATHor other paths are customized. Do not delete files that running processes still use. - Existing global state files do not have their permissions repaired automatically. For sharing across users, check that the participating containers can read and write the corresponding files. To use the product's default permissions, change the identified files in place to
0666. The default host path is/usr/local/hami-shared-region/global_registry.device-<physical-device-ID>; the suffix identifies the physical device, not its logical index inside the container. If recreating a global file, first stop every workload sharing it. - Upgrade the plugin. Confirm that the matching runtime library and
vnpu-dcmihelper are installed and that node devices have registered again, then recreate the workloads. The default allocation flow creates a local state directory for each new container and mounts the matching library and helper. Do not let old and new libraries share the same local state file.
Check images and startup arguments
- The default plugin image is
ghcr.io/dynamia-ai/ascend-device-plugin:v1.5.0. Update a pinnedimage.tag. Ifimage.digestis set, update it to the target image digest or clear it; the digest takes precedence over the tag. - Deployments that explicitly set
--enable_periodic_idle_vnpu_cleanup=falseretain that setting. To enable periodic cleanup, change it totrueor remove the override. The grace period and confirmation count can be adjusted with--idle_vnpu_cleanup_grace_period=10mand--idle_vnpu_cleanup_confirmations=3. Preserve the existing configuration-file arguments when adding these options to the Chart'sdaemonSet.args, and restart the plugin after updating them. - Before upgrading a
hami-coredeployment from an older application, meet thev2.10.0-r2companion requirement and complete license activation. This requirement was introduced with applicationv1.4.3; deployments already meeting it do not need another Enterprise upgrade for the runtime fixes in this release.
Use Enterprise memory expansion
Meet the companion requirements in the compatibility table. Set devices.ascend.enabled=true and scheduler.admissionWebhook.enabled=true in the HAMi Enterprise Chart, and confirm that the vNPU expansion-validation webhook is enabled. With the 2.10 series, also set dra.enabled=false. Chart 2.11.0-r0 removes the DRA integration and rejects both dra and hami-dra configuration, including dra.enabled=false; omit those sections when using that version.
After migrating runtime state, set the literal NPU_MEMORY_AUTO_ADJUST=true in an ordinary container, explicitly set huawei.com/vnpu-mode: hami-core on the Pod, and allocate exactly one vNPU to that container. Existing deployments that meet these conditions do not need another Enterprise upgrade. Init containers and a single container assigned multiple vNPUs are outside this automatic-expansion scope.
Configure Ascend950DT
With the default config.create=false, add the following entry to vnpus.configs in the device-config.yaml used by both the scheduler and plugin. Preserve the entries for other models. Compared with the plugin's bundled entry, the shared configuration also needs a memory resource name for the Enterprise scheduler:
- chipName: Ascend950DT
commonWord: Ascend950DT
resourceName: huawei.com/Ascend950DT
resourceMemoryName: huawei.com/Ascend950DT-memory
memoryAllocatable: 98304
memoryCapacity: 98304
memoryFactor: 1
aiCore: 32
templates: []
These defaults describe a device with 96 GiB of memory and 32 AI cores. Adjust them for other hardware specifications; the plugin validates memory capacity and AI core count. If the scheduler injects a RuntimeClass, include the cluster's actual runtimeClassName in this entry.
Append huawei.com/Ascend950DT and huawei.com/Ascend950DT-memory to the HAMi Enterprise Chart's devices.ascend.customresources list, retaining the existing resource names. In the Ascend950DT node configuration, set hamiVnpuMode to template and vDeviceCount to 1. Node configuration can override a global soft-partitioning mode. Ensure that both the scheduler and plugin load the updated configuration before creating workloads with matching whole-card huawei.com/Ascend950DT counts in requests and limits.
With config.create=true, the bundled configuration includes this model. The additional scheduler fields above are still required when sharing that configuration with Enterprise. Deployments that replace the entire deviceConfig string must add the configuration themselves.
See the Chart and appVersion mapping or the software center.
Values reference
The parameter descriptions below are preserved from the README bundled with Chart 0.4.0. The original values.yaml from that package defines its defaults.
Expand values table
| Key | Type | Default | Description |
|---|---|---|---|
| config.create | bool | false | Create the bundled device configuration ConfigMap for an installation without HAMi. Leave disabled when HAMi owns the device configuration. |
| config.deviceConfigMapName | string | "hami-scheduler-device" | Name of the device configuration ConfigMap created when config.create=true. |
| config.existingDeviceConfigMapName | string | "hami-scheduler-device" | HAMi-managed device configuration ConfigMap to reuse when config.create=false. The ConfigMap must be in the release namespace. |
| daemonSet.args[0] | string | "--config_file" | |
| daemonSet.args[1] | string | "/device-config.yaml" | |
| daemonSet.args[2] | string | "--v=4" | |
| daemonSet.args[3] | string | "--node_config_file" | |
| daemonSet.args[4] | string | "/node-config.yaml" | |
| daemonSet.name | string | "hami-ascend-device-plugin" | Device plugin DaemonSet name. |
| deviceSplitCount | int | 10 | Number of virtual devices advertised per physical Ascend NPU in hami-core or enpu mode when this chart creates the device configuration. A positive node-level vDeviceCount overrides it; ENPU caps the count at 100. |
| enpu.configRoot | string | "/var/lib/hami-enpu" | Host directory for per-container ENPU configs. |
| enpu.defaultPolicy | string | "elastic" | Default ENPU scheduling policy. |
| enpu.exposeAllDevices | bool | false | Expose all davinci devices; keep false for device isolation. |
| enpu.managerConfigRoot | string | "" | Manager config_dir or its vcann-rt subdirectory; empty uses configRoot. |
| enpu.managerURL | string | "" | Optional node-local enpu-manager REST endpoint. |
| enpu.monitorPath | string | "/usr/local/enpu/vcann-rt/tools/enpu-monitor" | Host path to enpu-monitor. |
| enpu.preloadPath | string | "/usr/local/enpu/vcann-rt/ld.so.preload" | Host path to ld.so.preload. |
| enpu.runtimePath | string | "/usr/local/enpu/vcann-rt/lib/libvruntime.so" | Host path to libvruntime.so. |
| enpu.systemdDetectVirtPath | string | "" | Optional host binary; prefer providing it in the workload image. |
| fullnameOverride | string | "" | Override the fully qualified resource name. |
| hamiVnpuCore.coreUtilizationPolicy | string | "default" | Default utilization override for containers without NPU_CORE_UTILIZATION_POLICY. default permits hamiCorePolicy (spatial by default); explicit force and disable retain precedence. |
| hamiVnpuCore.defaultPolicy | string | "spatial" | Default hami-core compute policy: spatial, dynamic, fixed, or overdraft. Pod huawei.com/npu-sched-policy and node hamiCorePolicy override it. |
| hamiVnpuCore.deviceCoreScaling | float | 1 | hami-core compute oversell ratio. The plugin advertises Devcore = round(100 * deviceCoreScaling) so HAMi can admit more than 100% of -core on one card. Values below 1 are not supported. |
| hamiVnpuCore.enabled | bool | false | Enable hami-vnpu-core in the bundled device configuration when config.create=true. This mode requires the HAMi scheduler. |
| hamiVnpuMode | string | "" | Default backend: template, hami-core (alias hamiCore), or enpu. Empty uses template, or hami-core if the deprecated hamiVnpuCore.enabled is true. Node mode takes priority. |
| image.digest | string | "" | Optional immutable OCI manifest digest. When set, it takes precedence over image.tag. |
| image.pullPolicy | string | "IfNotPresent" | Kubernetes image pull policy. |
| image.repository | string | "ghcr.io/dynamia-ai/ascend-device-plugin" | Container image repository. |
| image.tag | string | "" | Container image tag. Defaults to the chart appVersion when empty. |
| nameOverride | string | "" | Override the chart name used in resource names. |
| nodeConfig | string | "nodes: []" | Per-node hami-vnpu-core configuration written to the node ConfigMap. |
| nodeConfigMap.create | bool | true | Create the per-node configuration ConfigMap. |
| nodeConfigMap.name | string | "hami-device-node-config" | Per-node configuration ConfigMap name. |
| nodeSelector.ascend | string | "on" | Node label value used to schedule the device plugin. |
| rbac.name | string | "hami-ascend" | Name shared by the chart-managed RBAC resources. |
| resources.limits.cpu | string | "500m" | |
| resources.limits.memory | string | "500Mi" | |
| resources.requests.cpu | string | "500m" | |
| resources.requests.memory | string | "500Mi" | |
| runtimeClass.create | bool | true | Create the Ascend RuntimeClass. |
| runtimeClass.handler | string | "ascend" | Container runtime handler used by the RuntimeClass. |
| runtimeClass.name | string | "ascend" | RuntimeClass resource name. |
| serviceAccount.create | bool | true | Create a ServiceAccount for the device plugin. |
| serviceAccount.name | string | "hami-ascend" | ServiceAccount name. |
| vnpuMonitor.enabled | bool | false | Expose device plugin metrics and create optional Prometheus Operator resources. |
| vnpuMonitor.prometheusRule.create | bool | true | Create a PrometheusRule when vnpuMonitor.enabled=true. |
| vnpuMonitor.prometheusRule.groupLabels.vendor | string | "ascend" | |
| vnpuMonitor.prometheusRule.groupName | string | "ascend-vnpu" | Recording-rule group name. |
| vnpuMonitor.prometheusRule.interval | string | "15s" | Recording-rule evaluation interval. |
| vnpuMonitor.prometheusRule.labels.release | string | "prometheus" | |
| vnpuMonitor.prometheusRule.labels.role | string | "recording-rules" | |
| vnpuMonitor.prometheusRule.labels.vendor | string | "ascend" | |
| vnpuMonitor.prometheusRule.name | string | "hami-ascend-vnpu-monitor" | PrometheusRule resource name. |
| vnpuMonitor.prometheusRule.namespace | string | "monitoring" | Namespace for the PrometheusRule. |
| vnpuMonitor.service.name | string | "hami-ascend-device-plugin-metrics" | Headless Service that exposes the device plugin metrics port. |
| vnpuMonitor.serviceMonitor.create | bool | true | Create a ServiceMonitor when vnpuMonitor.enabled=true. |
| vnpuMonitor.serviceMonitor.interval | string | "15s" | Prometheus scrape interval. |
| vnpuMonitor.serviceMonitor.labels.release | string | "prometheus" | |
| vnpuMonitor.serviceMonitor.name | string | "hami-ascend-vnpu-monitor" | ServiceMonitor resource name. |
| vnpuMonitor.serviceMonitor.namespace | string | "monitoring" | Namespace for the ServiceMonitor. |
| vnpuMonitor.serviceMonitor.path | string | "/metrics" | Metrics HTTP path. |