HAMi Enterprise 2.8.0-rc0

Chart version	Declared appVersion	Release date
2.8.0-rc0	2.8.0-rc0	2026-03-09

This is a pre-release. This page uses HAMi Enterprise v2.7.1-rc0 as the upgrade baseline and describes Enterprise extensions relative to open-source HAMi v2.8.0.

Software changes

Upgrade changes and upstream synchronization

This release synchronizes the following upstream changes while retaining Enterprise device allocation, runtime memory adjustment, and license management capabilities.

Change	Impact on usage and integration	Upstream PR
Scheduler leader-follower coordination	The HAMi scheduler extender follows the kube-scheduler leader in the same Pod. Only the leader registers node devices and passes its readiness probe. After taking over leadership, an instance synchronizes device state before handling scheduling filter requests. See below for multi-replica configuration and readiness behavior.	HAMi #1553
Webhook certificate hot reload	Watches for changes to mounted TLS certificates and private keys and loads updated certificates, reducing the need to restart the scheduler during certificate rotation. Webhook calls may still fail while files are being propagated or loaded.	HAMi #1573
NVIDIA multi-container quota fix	When checking subsequent GPU containers in the same Pod, the scheduler now includes memory and compute requests from earlier containers. For example, if a namespace has 2000 MiB of memory quota remaining, two containers requesting 1024 MiB each can no longer both pass separate quota checks.	HAMi #1579
Clear metrics after node deletion	Removes device and usage records when a node is deleted, so the scheduler no longer emits stale metrics for that node. Historical samples already retained by the monitoring system are unaffected.	HAMi #1597
Build-info metric and version output	The scheduler and vGPU monitor add hami_build_info, whose value is always 1. Build information is exposed through the version, revision, build_date, go_version, compiler, and platform labels. The version command also switches to multi-field output.	HAMi #1581
Kunlunxin vXPU memory check	Checks whether a requested memory amount exceeds the device's available capacity even when no other task has allocated the device, preventing an oversized request from passing the initial allocation check.	HAMi #1569
Optional DRA monitoring	The companion HAMi-DRA subchart is updated to 0.1.0 and includes a monitoring component. The regular device-plugin deployment remains the default path.	HAMi #1580
Mock device-plugin update	The default image version for the optional mock device plugin changes from 0.1.0 to 1.0.1, affecting validation environments where the plugin is enabled.	HAMi #1570

Enterprise NVIDIA allocation annotations also add checks for empty allocation lists and out-of-range container indexes to avoid accessing nonexistent devices or containers.

Enterprise extensions compared with open-source HAMi v2.8.0

The following capabilities already existed in Enterprise v2.7.1-rc0 for the regular NVIDIA device-plugin path and continue to be available in this release.

• Pass device-allocation results through the Downward API: Pass GPU UUIDs, memory, and compute limits from Pod annotations into the corresponding containers. The NVIDIA allocation path can therefore avoid relying on a node lock to find the Pod awaiting allocation. nvidia.useDownward is true in the default device configuration; it is not a new installation parameter in this release.
• Adjust memory limits for running workloads: The monitor can submit memory-adjustment requests reported by the runtime to the scheduler. The scheduler checks remaining device capacity, updates allocation records and quota usage, and the monitor then synchronizes the approved memory limit to the runtime. This requires the companion Enterprise runtime and monitor; it does not modify the Pod's resources.limits.
• Device license management: Device license state is included in NVIDIA scheduling decisions. The scheduler exposes remaining license time in seconds through vGPULicenseRemains, which can be used to configure expiration alerts. Obtain and renew licenses through the agreed delivery process.
• Container memory-detail metrics: In addition to total memory usage and limits, Enterprise monitoring exposes hami_container_device_memory_context_bytes, hami_container_device_memory_data_bytes, hami_container_device_memory_module_bytes, and hami_container_device_memory_offset_bytes for context, data, module, and offset usage, respectively. Values are in bytes and represent current values; use Gauge semantics.

Chart changes

• The main Chart and appVersion are both 2.8.0-rc0; the default application image tag changes to v2.8.0-rc0. Deployments that override image tags explicitly should verify the version of each component.
• scheduler.leaderElect remains true by default and scheduler.replicas remains 1. This release connects the elected-leader state to the HAMi scheduler extender, adds a /readyz probe and list/watch permissions for Leases, and requires scheduler Pods to be spread across nodes by kubernetes.io/hostname. When leader election is disabled, the Chart fixes the scheduler replica count at one.
• dra.enabled remains false by default. The optional hami-dra subchart is updated from 0.0.1 to 0.1.0 and adds hami-dra.monitor.enabled: true. This monitoring switch only applies when the DRA path is enabled. Enabling DRA switches deployment paths and does not deploy the regular scheduler and NVIDIA device plugin from the main Chart.
• mockDevicePlugin.enabled remains false by default. When enabled, it uses mockDevicePlugin.image.tag: "1.0.1".

Upgrade notes

Check scheduler replicas and readiness

The default single-replica deployment does not become multi-replica automatically during an upgrade. To deploy a leader and follower, use the following values and ensure at least two nodes satisfy the node selector:

scheduler:
  leaderElect: true
  replicas: 2

With leader election enabled, scheduler Pods must run on different nodes. A single-node deployment or rolling update may not have enough available nodes, leaving a new replica in Pending. Before upgrading, reserve schedulable capacity for the new replica or adjust leader-election settings for the deployment. The anti-affinity rule matches the hami-scheduler component label in the same namespace; also check node placement of other installations.

The follower's /readyz returns 503; only the leader returns 200 and is included in the Service's ready endpoints. Deployment automation and alerts should use the Lease holder and leader endpoint to determine availability rather than requiring every leader/follower replica to be ready. If you manage RBAC yourself, add list and watch permissions for Leases to the scheduler.

If you disable the bundled kube-scheduler and connect an independently deployed scheduler, do not reuse the leader-election settings above without adjustment. The extender matches the Lease holder against its own hostname; a separate scheduler usually does not meet this condition, leaving the extender unready and preventing device registration. For a single-extender deployment, use the following values and retain the external scheduler integration settings:

scheduler:
  kubeScheduler:
    enabled: false
  leaderElect: false
  livenessProbe: false

This combination disables the bundled liveness probe, as required by this Chart's templates, and fixes the extender replica count at one. It disables the extender's dependency on the bundled scheduler's leader-election state; it does not affect leader election in the external scheduler.

Check certificate rotation and quotas

Certificate hot reload depends on the updated certificate files being fully propagated to the mounted volume. scheduler.admissionWebhook.failurePolicy remains Ignore by default, so a failed webhook call may allow a Pod to be created without complete HAMi injection. If your deployment requires creation to be blocked when the webhook call fails, configure Fail and account for related Pod creation being blocked while the webhook is unavailable.

After upgrading, workloads that relied on the old multi-container quota calculation may fail scheduling if their total request exceeds the namespace quota. Before upgrading, check the total GPU memory and compute requests across all GPU containers in each Pod against ResourceQuota.

Preserve Enterprise device configuration and monitoring semantics

The default device configuration sets both nvidia.useDownward and nvidia.postNvidiaSmi to true. The latter adds a postStart hook that runs nvidia-smi in containers requesting NVIDIA devices. The container image must provide that command, and verify whether an existing postStart hook would be replaced. To change these fields, edit the device configuration used by this Chart version; they are not top-level Helm parameters.

If you provide files/device-config.yaml in the Chart, it replaces the entire default device configuration rather than merging individual entries. Preserve any Enterprise fields your deployment uses. After changing the configuration, deploy the scheduler and device plugin with the updated configuration, then recreate workloads whose injected settings need to change.

Use hami_build_info to check component build information. Scripts that call scheduler version or vGPUmonitor version should parse the new multi-field output. Existing Enterprise memory-detail and license metrics remain available.

Obtain the package, companion Enterprise runtime, and license materials through the agreed delivery channel. See the Chart and appVersion mapping or the software center.

Values reference

The following parameter descriptions are taken from the README bundled with the Chart 2.8.0-rc0 release package and are preserved as published. Use this version's values.yaml as the source of default configuration.

Expand values table

Global Configuration

Parameter	Description	Default Value
global.imageRegistry	Global Docker image registry	""
global.imagePullSecrets	Global Docker image pull secrets	[]
global.imageTag	Image tag	"v2.8.0"
global.gpuHookPath	GPU Hook path	/usr/local
global.labels	Global labels	{}
global.annotations	Global annotations	{}
global.managedNodeSelectorEnable	Whether to enable managed node selector	false
global.managedNodeSelector.usage	Managed node selector usage	"gpu"
nameOverride	Name override	""
fullnameOverride	Full name override	""
namespaceOverride	Namespace override	""

Resource Name Configuration

NVIDIA GPU Resources

Parameter	Description	Default Value
resourceName	GPU resource name	"nvidia.com/gpu"
resourceMem	GPU memory resource name	"nvidia.com/gpumem"
resourceMemPercentage	GPU memory percentage resource name	"nvidia.com/gpumem-percentage"
resourceCores	GPU core resource name	"nvidia.com/gpucores"
resourcePriority	GPU priority resource name	"nvidia.com/priority"

Cambricon MLU Resources

Parameter	Description	Default Value
mluResourceName	MLU resource name	"cambricon.com/vmlu"
mluResourceMem	MLU memory resource name	"cambricon.com/mlu.smlu.vmemory"
mluResourceCores	MLU core resource name	"cambricon.com/mlu.smlu.vcore"

Hygon DCU Resources

Parameter	Description	Default Value
dcuResourceName	DCU resource name	"hygon.com/dcunum"
dcuResourceMem	DCU memory resource name	"hygon.com/dcumem"
dcuResourceCores	DCU core resource name	"hygon.com/dcucores"

Metax GPU Resources

Parameter	Description	Default Value
metaxResourceName	GPU resource name	"metax-tech.com/sgpu"
metaxResourceCore	GPU core resource name	"metax-tech.com/vcore"
metaxResourceMem	GPU memory resource name	"metax-tech.com/vmemory"
metaxsGPUTopologyAware	GPU topology awareness	"false"

Enflame GCU Resources

Parameter	Description	Default Value
enflameResourceNameVGCU	vGCU resource name	"enflame.com/vgcu"
enflameResourceNameVGCUPercentage	vGCU percentage resource name	"enflame.com/vgcu-percentage"

Kunlunxin XPU Resources

Parameter	Description	Default Value
kunlunResourceName	XPU resource name	"kunlunxin.com/xpu"

Scheduler Configuration

Parameter	Description	Default Value
schedulerName	Scheduler name	"hami-scheduler"
scheduler.nodeName	Define node name, scheduler will schedule to this node	""
scheduler.overwriteEnv	Whether to overwrite environment variables	"false"
scheduler.defaultSchedulerPolicy.nodeSchedulerPolicy	Node scheduler policy	binpack
scheduler.defaultSchedulerPolicy.gpuSchedulerPolicy	GPU scheduler policy	spread
scheduler.metricsBindAddress	Metrics bind address	":9395"
scheduler.forceOverwriteDefaultScheduler	Whether to force overwrite default scheduler	true
scheduler.livenessProbe	Whether to enable liveness probe	false
scheduler.leaderElect	Whether to enable leader election	true
scheduler.replicas	Number of replicas	1

Kube Scheduler Configuration

Parameter	Description	Default Value
scheduler.kubeScheduler.enabled	Whether to run kube-scheduler container in scheduler pod	true
scheduler.kubeScheduler.image.registry	Kube scheduler image registry	"registry.cn-hangzhou.aliyuncs.com"
scheduler.kubeScheduler.image.repository	Kube scheduler image repository	"google_containers/kube-scheduler"
scheduler.kubeScheduler.image.tag	Kube scheduler image tag	""
scheduler.kubeScheduler.image.pullPolicy	Kube scheduler image pull policy	IfNotPresent
scheduler.kubeScheduler.image.pullSecrets	Kube scheduler image pull secrets	[]
scheduler.kubeScheduler.extraNewArgs	Extra new arguments	["--config=/config/config.yaml", "-v=4"]
scheduler.kubeScheduler.extraArgs	Extra arguments	["--policy-config-file=/config/config.json", "-v=4"]

Extender Configuration

Parameter	Description	Default Value
scheduler.extender.image.registry	Scheduler extender image registry	"docker.io"
scheduler.extender.image.repository	Scheduler extender image repository	"projecthami/hami"
scheduler.extender.image.tag	Scheduler extender image tag	""
scheduler.extender.image.pullPolicy	Scheduler extender image pull policy	IfNotPresent
scheduler.extender.image.pullSecrets	Scheduler extender image pull secrets	[]
scheduler.extender.extraArgs	Scheduler extender extra arguments	["--debug", "-v=4"]

Admission Webhook Configuration

Parameter	Description	Default Value
scheduler.admissionWebhook.enabled	Whether to enable admission webhook	true
scheduler.admissionWebhook.customURL.enabled	Whether to enable custom URL	false
scheduler.admissionWebhook.customURL.host	Custom URL host	127.0.0.1
scheduler.admissionWebhook.customURL.port	Custom URL port	31998
scheduler.admissionWebhook.customURL.path	Custom URL path	/webhook
scheduler.admissionWebhook.reinvocationPolicy	Reinvocation policy	Never
scheduler.admissionWebhook.failurePolicy	Failure policy	Ignore

TLS Certificate Configuration

Parameter	Description	Default Value
scheduler.certManager.enabled	Whether to use cert-manager to generate self-signed certificates	false
scheduler.patch.enabled	Whether to use kube-webhook-certgen to generate self-signed certificates	true
scheduler.patch.image.registry	Certgen image registry	"docker.io"
scheduler.patch.image.repository	Certgen image repository	"jettech/kube-webhook-certgen"
scheduler.patch.image.tag	Certgen image tag	"v1.5.2"
scheduler.patch.image.pullPolicy	Certgen image pull policy	IfNotPresent
scheduler.patch.imageNew.registry	New certgen image registry	"docker.io"
scheduler.patch.imageNew.repository	New certgen image repository	"liangjw/kube-webhook-certgen"
scheduler.patch.imageNew.tag	New certgen image tag	"v1.1.1"

Scheduler Service Configuration

Parameter	Description	Default Value
scheduler.service.type	Service type	NodePort
scheduler.service.httpPort	HTTP port	443
scheduler.service.schedulerPort	Scheduler NodePort	31998
scheduler.service.monitorPort	Monitor port	31993
scheduler.service.monitorTargetPort	Monitor target port	9395

Device Plugin Configuration

Parameter	Description	Default Value
devicePlugin.image.registry	Device plugin image registry	"docker.io"
devicePlugin.image.repository	Device plugin image repository	"projecthami/hami"
devicePlugin.image.tag	Device plugin image tag	""
devicePlugin.image.pullPolicy	Device plugin image pull policy	IfNotPresent
devicePlugin.image.pullSecrets	Device plugin image pull secrets	[]

Monitor Configuration

Parameter	Description	Default Value
devicePlugin.monitor.image.registry	Monitor image registry	"docker.io"
devicePlugin.monitor.image.repository	Monitor image repository	"projecthami/hami"
devicePlugin.monitor.image.tag	Monitor image tag	""
devicePlugin.monitor.image.pullPolicy	Monitor image pull policy	IfNotPresent
devicePlugin.monitor.image.pullSecrets	Monitor image pull secrets	[]
devicePlugin.monitor.ctrPath	Container path	/usr/local/vgpu/containers
devicePlugin.monitor.extraArgs	Monitor extra arguments	["-v=4"]
devicePlugin.monitor.extraEnvs	Monitor extra environments	{}

Device Plugin Other Configuration

Parameter	Description	Default Value
devicePlugin.deviceSplitCount	Integer type, default value: 10. Maximum number of tasks assigned to a single GPU device	10
devicePlugin.deviceMemoryScaling	Device memory scaling ratio	1
devicePlugin.deviceCoreScaling	Device core scaling ratio	1
devicePlugin.runtimeClassName	Runtime class name	""
devicePlugin.createRuntimeClass	Whether to create runtime class	false
devicePlugin.migStrategy	String type, "none" means ignore MIG functionality, "mixed" means allocate MIG devices through independent resources	"none"
devicePlugin.disablecorelimit	String type, "true" means disable core limit, "false" means enable core limit	"false"
devicePlugin.passDeviceSpecsEnabled	Whether to enable passing device specs	false
devicePlugin.extraArgs	Device plugin extra arguments	["-v=4"]
devicePlugin.nodeConfiguration.config	Node configuration for device plugin by json	An example of default configuration.
devicePlugin.nodeConfiguration.externalConfigName	Node configuration for device plugin by external congimap	""
devicePlugin.extraEnvs	Device plugin extra environments	{}

Device Plugin Service Configuration

Parameter	Description	Default Value
devicePlugin.service.type	Service type	NodePort
devicePlugin.service.httpPort	HTTP port	31992

Device Plugin Deployment Configuration

Parameter	Description	Default Value
devicePlugin.pluginPath	Plugin path	/var/lib/kubelet/device-plugins
devicePlugin.libPath	Library path	/usr/local/vgpu
devicePlugin.nvidiaNodeSelector	NVIDIA node selector	{"gpu": "on"}
devicePlugin.updateStrategy.type	Update strategy type	RollingUpdate
devicePlugin.updateStrategy.rollingUpdate.maxUnavailable	Maximum unavailable count	1

Device Configuration

AWS Neuron

Parameter	Description	Default Value
devices.awsneuron.customresources	Custom resources	["aws.amazon.com/neuron", "aws.amazon.com/neuroncore"]

Kunlunxin

Parameter	Description	Default Value
devices.kunlun.enabled	Whether to enable	true
devices.kunlun.customresources	Custom resources	["kunlunxin.com/xpu"]

Mthreads

Parameter	Description	Default Value
devices.mthreads.enabled	Whether to enable	true
devices.mthreads.customresources	Custom resources	["mthreads.com/vgpu"]

NVIDIA

Parameter	Description	Default Value
devices.nvidia.gpuCorePolicy	GPU core policy	default
devices.nvidia.libCudaLogLevel	CUDA library log level	1

Huawei Ascend

Parameter	Description	Default Value
devices.ascend.enabled	Whether to enable	false
devices.ascend.image	Image	""
devices.ascend.imagePullPolicy	Image pull policy	IfNotPresent
devices.ascend.extraArgs	Extra arguments	[]
devices.ascend.nodeSelector	Node selector	{"ascend": "on"}
devices.ascend.tolerations	Tolerations	[]
devices.ascend.customresources	Custom resources	["huawei.com/Ascend910A", "huawei.com/Ascend910A-memory", ...]

Iluvatar

Parameter	Description	Default Value
devices.iluvatar.enabled	Whether to enable	false
devices.iluvatar.customresources	Custom resources	["iluvatar.ai/BI-V150-vgpu", "iluvatar.ai/BI-V150.vMem","iluvatar.ai/BI-V150.vCore", ...]