Skip to main content

HAMi Enterprise 2.9.0-rc0

HAMi Enterprise v2.9.0-rc0 adds AMD memory and compute-share requests, an Ascend soft-partition scheduling interface, and Biren device support. It also changes monitoring metric names, the device-configuration structure, and traffic routing for scheduler leader and follower instances. Existing deployments should carefully review monitoring rules, custom device configuration, and companion device plugins.

Version information​

ItemValue
Enterprise releaseHAMi Enterprise v2.9.0-rc0
Release date2026-05-15
Enterprise upgrade baselinev2.8.0-rc1
Open-source baselineHAMi v2.9.0
Helm Charthami, version 2.9.0-rc0
Chart appVersion2.9.0-rc0
Default Enterprise component image tagv2.9.0-rc0

Software changes​

Changes compared with Enterprise v2.8.0-rc1​

  • Device scheduling: AMD switches from recognizing full cards based on node capacity to reading device-plugin registration information and adds memory- and compute-share requests. Ascend adds scheduling and runtime configuration interfaces for soft partitioning. Biren adds scheduling support for full-card and die modes, plus UUID selection. Each device type requires a separately deployed matching device plugin; the main Chart does not include these plugins.
  • Monitoring interfaces: Common scheduler and vGPU monitor metrics now use the hami_ prefix, and label names are updated. A compatibility switch for old metrics is added and disabled by default; it does not cover the three memory-detail metrics. See “Monitoring rule migration” for the complete mapping.
  • GPU memory auto-expansion: This existing feature now has an explicit enablement condition. It processes expansion feedback only when CUDA_MEMORY_AUTO_ADJUST: "true" is declared directly on a workload container.
  • Scheduler leader/follower routing: Readiness probes can now succeed on both leader and follower instances. The Service selects the leader using hami.io/scheduler-role: leader. Existing leader election remains available, but readiness no longer indicates that an instance is the leader.
  • Allocation and recovery: Fixes index preservation in multi-container device annotations, CDI file updates after MIG reconfiguration, and some state handling for terminating Pods and device-registration recovery. Also fixes scoring for percentage-based memory requests and handling of zero capacity.

Enterprise extensions compared with open-source HAMi v2.9.0​

CapabilityEnterprise behavior and requirements
Enterprise license managementContinues to manage licenses by device instance or device-model count and expose expiration status. Confirm license support for each device backend against the delivery plan.
NVIDIA container allocation and memory auto-expansionContinues to pass device, memory, and compute allocations to containers through the Downward API and to process memory-expansion requests and allocation updates. Expansion requires a compatible Enterprise runtime, is limited by allocatable memory recorded by the scheduler, and applies only to regular workload containers.
AMD share schedulingExtends the open-source baseline's full-card handling with registered-device information, memory and compute requests, model filtering, and allocation-information coordination. Requires the updated device-registration protocol.
Alibaba PPU scheduling integrationContinues PPU resource requests, device selection, and namespace quota checks. Requires a separate companion PPU plugin and runtime. See “AMD and PPU integration” for the limitations of the default Chart resource mapping in this release.
AWS Neuron resource accountingContinues Enterprise accounting semantics for core counts and memory shares, which differ from the open-source baseline. When migrating from open source, verify existing quotas, resource statistics, and device-type mappings.

Except for AMD share scheduling, the capabilities above were already provided in earlier Enterprise releases. The NVIDIA Downward API allocation path also already existed; useDownward and postNvidiaSmi remain enabled in the default device configuration.

Upstream changes included in this release​

The following are the major upstream changes included in this release. Follow this page's upgrade notes for configuration and Enterprise usage boundaries.

ChangeImpact on usage and integrationUpstream PR
Biren device schedulingAdds scheduling integration for full-card and die modes and UUID selection; requires the vendor device plugin.#1645
Webhook label filteringLimits admission processing by namespace and Pod labels while retaining existing exclusion conditions.#1653
Monitoring metric normalizationRenames metrics and labels and provides compatibility output for some old metrics.#1644
Ascend soft partitioning and runtime interfaceAdds HAMi-core mode, compute requests, RuntimeClass, and environment-variable handling. Runtime libraries and the device plugin must be provided separately.#1771, #1804, #1812, #1837, #1738
Preload configuration for a custom library directoryUses devicePlugin.libPath for preload files instead of continuing to reference a fixed default directory.#1714
Resource usage by terminating PodsRetains device-usage records while a Pod is being deleted to avoid releasing resources too early from the scheduler cache.#1719
Zero-capacity scoring protectionHandles devices or nodes with zero capacity to prevent invalid scores.#1820
Device-registration recoveryFixes handshake-time parsing and allows a deleted device to register again after the cooldown period.#1816, #1843
MIG and CDI integrationUpdates and checks CDI files after MIG configuration changes in the applicable allocation path; fixes the A100 80 GB MIG profile.#1826, #1847

Chart changes​

  • Adds legacyMetrics, defaulting to false. When set to true, the scheduler and vGPU monitor both append the supported legacy metrics to their output.
  • Adds scheduler.admissionWebhook.namespaceSelector and objectSelector, supporting matchLabels and matchExpressions. Custom conditions apply together with existing conditions; namespaces or Pods labeled hami.io/webhook: ignore are still excluded.
  • A non-empty component-level image.tag for the scheduler, NVIDIA Device Plugin, or vGPU monitor takes precedence over global.imageTag; an empty component-level tag falls back to the global tag. The settings are scheduler.extender.image.tag, devicePlugin.image.tag, and devicePlugin.monitor.image.tag.
  • Consolidates scheduling-configuration templates for older and newer Kubernetes versions. For Kubernetes 1.22 and later, the scheduling ConfigMap name no longer includes -newversion; Chart-managed mounts are updated accordingly. Update deployments that reference the old ConfigMap name directly.
  • Sets the NVIDIA Device Plugin DaemonSet to hostNetwork: false. Configurations that collect monitoring data directly from the host network address must use the monitor Service endpoint instead and review related network policies. Existing Service and ServiceMonitor interfaces remain available.
  • Adds devices.ascend.hamiVnpuCore (default false) and devices.ascend.runtimeClassName (default empty), and adds Ascend compute-resource and 910C device configuration. devices.ascend.enabled remains disabled by default.
  • scheduler.overwriteEnv is now used to generate NVIDIA device configuration and is also used for the new Ascend configuration. It remains "false" by default. Enabling it affects the device environment variables visible to containers that did not request the corresponding device; check existing environment-variable conventions before enabling it.
  • Updates the optional hami-dra subchart from 0.1.0 to 0.2.0. dra.enabled remains disabled by default. Enabling it switches to the DRA deployment path and does not install the existing scheduler extender. DRA driver images have independent version settings and are not determined by the main application tag.

Upgrade notes​

Package and companion components​

Obtain hami-2.9.0-rc0.tgz, the Enterprise images, and the plugins and runtimes for the devices in use through the agreed delivery channel. The Chart remains named hami. Keep the existing Helm release name and namespace during the upgrade, and merge existing custom values with the new version's configuration.

If component-level image tags were pinned to an older version, changing only global.imageTag will not override them. Update the three component-level tags or clear them to use the global tag.

Monitoring rule migration​

Before upgrading, update dashboards, alerts, recording rules, and label filters. During migration, set legacyMetrics: true to emit both new metrics and supported legacy metrics; disable legacy output after migration. Deployments that start binaries directly must separately set --legacy-metrics=true for the scheduler and vGPU monitor.

Old scheduler metricNew metric
GPUDeviceMemoryLimithami_gpu_memory_limit_bytes
GPUDeviceCoreLimithami_gpu_core_limit_ratio
GPUDeviceMemoryAllocatedhami_gpu_memory_allocated_bytes
GPUDeviceSharedNumhami_gpu_shared_count
GPUDeviceCoreAllocatedhami_gpu_core_allocated_ratio
nodeGPUOverviewhami_node_gpu_overview
nodeGPUMemoryPercentagehami_node_gpu_memory_allocated_ratio
nodeGPUMigInstancehami_node_gpu_mig_instance_info
vGPUMemoryAllocatedhami_vgpu_memory_allocated_bytes
vGPUCoreAllocatedhami_vgpu_core_allocated_ratio
QuotaUsedhami_resource_quota_used
Old monitor metricNew metricIs the old name retained by the compatibility switch?
HostGPUMemoryUsagehami_host_gpu_memory_used_bytesYes
HostCoreUtilizationhami_host_gpu_utilization_ratioYes
vGPU_device_memory_usage_in_byteshami_vgpu_memory_used_bytesYes
vGPU_device_memory_limit_in_byteshami_vgpu_memory_limit_bytesYes
Device_memory_desc_of_containerhami_container_device_memory_bytesYes
Device_utilization_desc_of_containerhami_container_device_utilization_ratioYes
Device_last_kernel_of_containerhami_container_last_kernel_elapsed_secondsYes
MigInfohami_mig_device_infoYes
vGPU_device_memory_context_size_byteshami_vgpu_memory_context_bytesNo
vGPU_device_memory_module_size_byteshami_vgpu_memory_module_bytesNo
vGPU_device_memory_buffer_size_byteshami_vgpu_memory_buffer_bytesNo

New metrics also use the following label names. Update query filters, grouping, and joins accordingly.

Old labelNew label
nodeid, nodenamenode
deviceuuid, deviceidx, devicetypedevice_uuid, device_index, device_type
devicecores, sharedcontainers, devicememorylimitdevice_cores, shared_containers, device_memory_limit
podnamespace, quotanamespacenamespace
podname, ctrname, containeridxpod, container, container_index
quotaName, mignamequota_name, mig_name
vdeviceid, instanceidvdevice_index, instance_id
context, module, data in memory descriptionscontext_size, module_size, buffer_size

The compatibility output for GPUDeviceSharedNum also adds the devicetype label. The Enterprise metric vGPULicenseRemains retains its existing name and labels.

Do not change thresholds uniformly based only on the _ratio suffix. hami_host_gpu_utilization_ratio still reports utilization from 0 to 100. Compute-limit and allocation metrics retain the device backend's values; a regular NVIDIA full card uses 100. hami_node_gpu_memory_allocated_ratio is allocated memory divided by total memory. For memory breakdowns, use the separate hami_vgpu_memory_*_bytes metrics instead of numeric labels in the memory-description metric.

GPU memory auto-expansion​

For regular workload containers that use this feature, declare the following environment variable directly in the workload template and apply it through a normal rolling update or container recreation:

env:
- name: CUDA_MEMORY_AUTO_ADJUST
value: "true"

This release checks for a directly declared string value in the Pod. Providing the variable indirectly through valueFrom or envFrom does not meet the enablement condition. Expansion is constrained by allocatable memory; it does not expand nodes automatically and does not guarantee that CUDA out-of-memory errors will be eliminated.

Ascend configuration migration and soft-partitioning requirements​

Device configuration generated by the Chart adopts the new structure automatically. If you provide files/device-config.yaml in the Chart, move the old vnpus list to vnpus.configs and preserve the existing chip configurations. The custom file replaces the entire default device configuration; entries are not merged individually.

vnpus:
hamiVnpuCore: false
configs: [] # Move the complete chip configurations from the old vnpus list here

Soft partitioning requires the companion Ascend device plugin, runtime, and HAMi-vNPU-core. Set devices.ascend.hamiVnpuCore: true to declare that nodes support this mode by default. The hami-vnpu-core node annotation takes precedence; only the string "true" enables it. Workloads must also set the Pod annotation huawei.com/vnpu-mode: hami-core and request the corresponding huawei.com/<device-type>-core compute resource; for example, a value of 40 means 40%. Nodes using soft partitioning reject some memory requests made in the legacy vNPU mode. In mixed deployments, use node annotations to distinguish modes and avoid switching nodes that still run legacy-mode workloads.

devices.ascend.runtimeClassName is added only when the Pod does not already specify a RuntimeClass. The container must provide bash, the companion runtime libraries, and /hami-vnpu-core/limiter. An existing postStart hook is not automatically combined with the limiter startup command; arrange to start /hami-vnpu-core/limiter in the background from the existing hook. These scheduling settings do not replace installation of the companion device plugin and runtime.

AMD and PPU integration​

The AMD plugin must report hami.io/node-amd-register; setting Node capacity alone is no longer sufficient for this release to recognize devices. The memory resource name changes from amd.com/gpu-memory to amd.com/gpumem. Platforms that integrate allocation information must also update the device-type identifier from AMDGPU to amd.

In container resources.limits, amd.com/gpumem is measured in MiB, and amd.com/gpucores: 40 requests 40% of the total compute capacity reported by the device. If memory is omitted or has a non-positive value, the full-card memory is reserved; if compute is also omitted, full-card compute is reserved as well.

To request an AMD compute share, add amd.com/gpucores to Helm values because it is not included in the default list:

devices:
amd:
customresources:
- amd.com/gpu
- amd.com/gpumem
- amd.com/gpucores

The default scheduler managedResources in this Chart does not include the previous PPU mapping, and devices.alibaba.enabled: true does not add those entries. Existing PPU deployments must add and retain alibaba.com/ppu, alibaba.com/ppu-memory, and alibaba.com/gpu-core, with ignoredByScheduler: true for each, under extenders[].managedResources in the Chart or custom scheduler configuration, then replace the existing configuration. Do not overwrite an existing PPU resource mapping with this release's default configuration.

Health-check difference from the open-source baseline​

This release does not include open-source HAMi v2.9.0 #1879. For devices such as Ascend that use the shared health check, a registration handshake that remains in Requesting for more than 60 seconds may cause the device cache to be cleared and affect scheduling for new tasks, even if Kubernetes still reports allocatable resources for the device. When diagnosing this, check both device-plugin registration and scheduler device state; Node resource counts alone do not confirm successful registration.

Init-container limitations​

This release improves init-container request parsing and allocation-index handling, but GPU admission injection, the Enterprise Downward API path, and monitoring are still incomplete. Workloads with GPU init containers should not be considered fully supported. GPU memory auto-expansion also applies only to regular workload containers.

See the Chart and appVersion mapping or the software center.

Values reference​

The following parameter descriptions are taken from the README bundled with the Chart 2.9.0-rc0 release package and are preserved as published. Use this version's values.yaml as the source of default configuration.

Expand values table

Global Configuration

ParameterDescriptionDefault Value
global.imageRegistryGlobal Docker image registry""
global.imagePullSecretsGlobal Docker image pull secrets[]
global.imageTagImage tag"v2.8.0"
global.gpuHookPathGPU Hook path/usr/local
global.labelsGlobal labels{}
global.annotationsGlobal annotations{}
global.managedNodeSelectorEnableWhether to enable managed node selectorfalse
global.managedNodeSelector.usageManaged node selector usage"gpu"
nameOverrideName override""
fullnameOverrideFull name override""
namespaceOverrideNamespace override""

Resource Name Configuration

NVIDIA GPU Resources

ParameterDescriptionDefault Value
resourceNameGPU resource name"nvidia.com/gpu"
resourceMemGPU memory resource name"nvidia.com/gpumem"
resourceMemPercentageGPU memory percentage resource name"nvidia.com/gpumem-percentage"
resourceCoresGPU core resource name"nvidia.com/gpucores"
resourcePriorityGPU priority resource name"nvidia.com/priority"

Cambricon MLU Resources

ParameterDescriptionDefault Value
mluResourceNameMLU resource name"cambricon.com/vmlu"
mluResourceMemMLU memory resource name"cambricon.com/mlu.smlu.vmemory"
mluResourceCoresMLU core resource name"cambricon.com/mlu.smlu.vcore"

Hygon DCU Resources

ParameterDescriptionDefault Value
dcuResourceNameDCU resource name"hygon.com/dcunum"
dcuResourceMemDCU memory resource name"hygon.com/dcumem"
dcuResourceCoresDCU core resource name"hygon.com/dcucores"

Metax GPU Resources

ParameterDescriptionDefault Value
metaxResourceNameGPU resource name"metax-tech.com/sgpu"
metaxResourceCoreGPU core resource name"metax-tech.com/vcore"
metaxResourceMemGPU memory resource name"metax-tech.com/vmemory"
metaxsGPUTopologyAwareGPU topology awareness"false"

Enflame GCU Resources

ParameterDescriptionDefault Value
enflameResourceNameVGCUvGCU resource name"enflame.com/vgcu"
enflameResourceNameVGCUPercentagevGCU percentage resource name"enflame.com/vgcu-percentage"

Kunlunxin XPU Resources

ParameterDescriptionDefault Value
kunlunResourceNameXPU resource name"kunlunxin.com/xpu"

Scheduler Configuration

ParameterDescriptionDefault Value
schedulerNameScheduler name"hami-scheduler"
scheduler.nodeNameDefine node name, scheduler will schedule to this node""
scheduler.overwriteEnvWhether to overwrite environment variables"false"
scheduler.defaultSchedulerPolicy.nodeSchedulerPolicyNode scheduler policybinpack
scheduler.defaultSchedulerPolicy.gpuSchedulerPolicyGPU scheduler policyspread
scheduler.metricsBindAddressMetrics bind address":9395"
scheduler.forceOverwriteDefaultSchedulerWhether to force overwrite default schedulertrue
scheduler.livenessProbeWhether to enable liveness probefalse
scheduler.leaderElectWhether to enable leader electiontrue
scheduler.replicasNumber of replicas1

Kube Scheduler Configuration

ParameterDescriptionDefault Value
scheduler.kubeScheduler.enabledWhether to run kube-scheduler container in scheduler podtrue
scheduler.kubeScheduler.image.registryKube scheduler image registry"registry.cn-hangzhou.aliyuncs.com"
scheduler.kubeScheduler.image.repositoryKube scheduler image repository"google_containers/kube-scheduler"
scheduler.kubeScheduler.image.tagKube scheduler image tag""
scheduler.kubeScheduler.image.pullPolicyKube scheduler image pull policyIfNotPresent
scheduler.kubeScheduler.image.pullSecretsKube scheduler image pull secrets[]
scheduler.kubeScheduler.extraNewArgsExtra new arguments["--config=/config/config.yaml", "-v=4"]
scheduler.kubeScheduler.extraArgsExtra arguments["--policy-config-file=/config/config.json", "-v=4"]

Extender Configuration

ParameterDescriptionDefault Value
scheduler.extender.image.registryScheduler extender image registry"docker.io"
scheduler.extender.image.repositoryScheduler extender image repository"projecthami/hami"
scheduler.extender.image.tagScheduler extender image tag""
scheduler.extender.image.pullPolicyScheduler extender image pull policyIfNotPresent
scheduler.extender.image.pullSecretsScheduler extender image pull secrets[]
scheduler.extender.extraArgsScheduler extender extra arguments["--debug", "-v=4"]

Admission Webhook Configuration

ParameterDescriptionDefault Value
scheduler.admissionWebhook.enabledWhether to enable admission webhooktrue
scheduler.admissionWebhook.customURL.enabledWhether to enable custom URLfalse
scheduler.admissionWebhook.customURL.hostCustom URL host127.0.0.1
scheduler.admissionWebhook.customURL.portCustom URL port31998
scheduler.admissionWebhook.customURL.pathCustom URL path/webhook
scheduler.admissionWebhook.reinvocationPolicyReinvocation policyNever
scheduler.admissionWebhook.failurePolicyFailure policyIgnore

TLS Certificate Configuration

ParameterDescriptionDefault Value
scheduler.certManager.enabledWhether to use cert-manager to generate self-signed certificatesfalse
scheduler.patch.enabledWhether to use kube-webhook-certgen to generate self-signed certificatestrue
scheduler.patch.image.registryCertgen image registry"docker.io"
scheduler.patch.image.repositoryCertgen image repository"jettech/kube-webhook-certgen"
scheduler.patch.image.tagCertgen image tag"v1.5.2"
scheduler.patch.image.pullPolicyCertgen image pull policyIfNotPresent
scheduler.patch.imageNew.registryNew certgen image registry"docker.io"
scheduler.patch.imageNew.repositoryNew certgen image repository"liangjw/kube-webhook-certgen"
scheduler.patch.imageNew.tagNew certgen image tag"v1.1.1"

Scheduler Service Configuration

ParameterDescriptionDefault Value
scheduler.service.typeService typeNodePort
scheduler.service.httpPortHTTP port443
scheduler.service.schedulerPortScheduler NodePort31998
scheduler.service.monitorPortMonitor port31993
scheduler.service.monitorTargetPortMonitor target port9395

Device Plugin Configuration

ParameterDescriptionDefault Value
devicePlugin.image.registryDevice plugin image registry"docker.io"
devicePlugin.image.repositoryDevice plugin image repository"projecthami/hami"
devicePlugin.image.tagDevice plugin image tag""
devicePlugin.image.pullPolicyDevice plugin image pull policyIfNotPresent
devicePlugin.image.pullSecretsDevice plugin image pull secrets[]

Monitor Configuration

ParameterDescriptionDefault Value
devicePlugin.monitor.image.registryMonitor image registry"docker.io"
devicePlugin.monitor.image.repositoryMonitor image repository"projecthami/hami"
devicePlugin.monitor.image.tagMonitor image tag""
devicePlugin.monitor.image.pullPolicyMonitor image pull policyIfNotPresent
devicePlugin.monitor.image.pullSecretsMonitor image pull secrets[]
devicePlugin.monitor.ctrPathContainer path/usr/local/vgpu/containers
devicePlugin.monitor.extraArgsMonitor extra arguments["-v=4"]
devicePlugin.monitor.extraEnvsMonitor extra environments{}

Device Plugin Other Configuration

ParameterDescriptionDefault Value
devicePlugin.deviceSplitCountInteger type, default value: 10. Maximum number of tasks assigned to a single GPU device10
devicePlugin.deviceMemoryScalingDevice memory scaling ratio1
devicePlugin.deviceCoreScalingDevice core scaling ratio1
devicePlugin.runtimeClassNameRuntime class name""
devicePlugin.createRuntimeClassWhether to create runtime classfalse
devicePlugin.migStrategyString type, "none" means ignore MIG functionality, "mixed" means allocate MIG devices through independent resources"none"
devicePlugin.disablecorelimitString type, "true" means disable core limit, "false" means enable core limit"false"
devicePlugin.passDeviceSpecsEnabledWhether to enable passing device specsfalse
devicePlugin.extraArgsDevice plugin extra arguments["-v=4"]
devicePlugin.nodeConfiguration.configNode configuration for device plugin by jsonAn example of default configuration.
devicePlugin.nodeConfiguration.externalConfigNameNode configuration for device plugin by external congimap""
devicePlugin.extraEnvsDevice plugin extra environments{}

Device Plugin Service Configuration

ParameterDescriptionDefault Value
devicePlugin.service.typeService typeNodePort
devicePlugin.service.httpPortHTTP port31992

Device Plugin Deployment Configuration

ParameterDescriptionDefault Value
devicePlugin.pluginPathPlugin path/var/lib/kubelet/device-plugins
devicePlugin.libPathLibrary path/usr/local/vgpu
devicePlugin.nvidiaNodeSelectorNVIDIA node selector{"gpu": "on"}
devicePlugin.updateStrategy.typeUpdate strategy typeRollingUpdate
devicePlugin.updateStrategy.rollingUpdate.maxUnavailableMaximum unavailable count1

Device Configuration

AWS Neuron

ParameterDescriptionDefault Value
devices.awsneuron.customresourcesCustom resources["aws.amazon.com/neuron", "aws.amazon.com/neuroncore"]

Kunlunxin

ParameterDescriptionDefault Value
devices.kunlun.enabledWhether to enabletrue
devices.kunlun.customresourcesCustom resources["kunlunxin.com/xpu"]

Mthreads

ParameterDescriptionDefault Value
devices.mthreads.enabledWhether to enabletrue
devices.mthreads.customresourcesCustom resources["mthreads.com/vgpu"]

NVIDIA

ParameterDescriptionDefault Value
devices.nvidia.gpuCorePolicyGPU core policydefault
devices.nvidia.libCudaLogLevelCUDA library log level1

Huawei Ascend

ParameterDescriptionDefault Value
devices.ascend.enabledWhether to enablefalse
devices.ascend.imageImage""
devices.ascend.imagePullPolicyImage pull policyIfNotPresent
devices.ascend.extraArgsExtra arguments[]
devices.ascend.nodeSelectorNode selector{"ascend": "on"}
devices.ascend.tolerationsTolerations[]
devices.ascend.customresourcesCustom resources["huawei.com/Ascend910A", "huawei.com/Ascend910A-memory", ...]

Iluvatar

ParameterDescriptionDefault Value
devices.iluvatar.enabledWhether to enablefalse
devices.iluvatar.customresourcesCustom resources["iluvatar.ai/BI-V150-vgpu", "iluvatar.ai/BI-V150.vMem","iluvatar.ai/BI-V150.vCore", ...]