Skip to main content

HAMi Enterprise 2.7.1-rc0

Chart versionDeclared appVersionRelease date
2.7.1-rc02.7.1-rc02026-01-16

This is Dynamia's first HAMi Enterprise release and is a pre-release. This page uses open-source HAMi v2.7.0 as its baseline.

Software changes​

Enterprise extensions compared with open-source HAMi v2.7.0​

Pass NVIDIA device-allocation results through the Downward API. The scheduler writes GPU UUIDs, memory limits, and compute limits to Pod annotations and then passes them to the corresponding containers through the Kubernetes Downward API. The regular NVIDIA vGPU allocation path can therefore skip the step that relies on a node lock to find the Pod awaiting allocation. nvidia.useDownward is true in the default device configuration. The companion device plugin, Enterprise runtime library, and NVIDIA container runtime environment are still required.

Adjust the memory limit of a running container. The Enterprise monitor receives memory-adjustment requests reported by the runtime and submits them to the scheduler, which checks available device capacity. After the scheduler updates allocation records and quota usage, the monitor synchronizes the approved limit back to the runtime. This capability requires the companion Enterprise runtime, monitor, and scheduler. It does not modify the Pod's resources.limits, and it does not mean all workloads will expand automatically. The first release does not additionally check the namespace ResourceQuota limit when deciding whether to expand memory; do not rely on it as a hard quota boundary for runtime expansion.

Manage NVIDIA device licenses. The scheduler includes device-license validity in device selection, and the monitor synchronizes license updates. The scheduler metric vGPULicenseRemains reports the remaining license time in seconds and can be used to configure expiration alerts. Obtain license materials for installation and renewal through the agreed delivery channel.

Expose container memory details. In addition to total memory usage and limits, Enterprise monitoring provides the following metrics in bytes. These are current values and should be treated as Gauges:

MetricMeaning
hami_container_device_memory_context_bytesContext memory usage
hami_container_device_memory_data_bytesData memory usage
hami_container_device_memory_module_bytesModule memory usage
hami_container_device_memory_offset_bytesOffset memory usage

These metrics associate containers and devices using the podnamespace, podname, ctrname, vdeviceid, and deviceuuid labels. Use these exact label names in collection rules.

Upstream changes included in this release​

This release also includes the following community changes made after open-source HAMi v2.7.0.

ChangeImpact on usage and integrationUpstream PR
Multi-model configuration for Iluvatar devicesConfigure device count, memory, and compute resource names separately for BI-V100, BI-V150, MR-V100, and MR-V50. Requires the matching GPU Manager and devices.iluvatar.enabled: true (default false). For example, BI-V150 uses iluvatar.ai/BI-V150-vgpu, iluvatar.ai/BI-V150.vMem, and iluvatar.ai/BI-V150.vCore.HAMi #1399
Online/offline task classification for Metax sGPUThe Pod annotation metax-tech.com/sgpu-app-class accepts online or offline to distinguish device selection for these task types. When this annotation is set, each applicable container can request only one sGPU. Task execution priority still depends on the companion Metax components.HAMi #1436
AMD MI300X full-card resourcesRequires a separate compatible AMD device plugin. HAMi uses the reported amd.com/gpu capacity to include full-card allocations in scheduler records and metrics. Memory accounting is fixed at 192000 MiB per card. These metrics represent scheduler allocation amounts, not real-time memory usage or utilization telemetry. This release does not provide AMD memory or compute partitioning.HAMi #1472
Mixed-size selection for dynamic MIGWhen a partition profile includes multiple instance sizes, matches each request to a suitable instance, fixing cases where a small-memory request incorrectly occupied a larger instance.HAMi #1518
Quota usage cleanupReclaims quota usage when a Pod enters a terminating state. When a ResourceQuota is updated, clears the old configuration first so removed quota entries do not continue to impose stale limits.HAMi #1400
Track GPU counts per nodeTracks NVIDIA GPU counts reported by each node separately, preventing nodes from overwriting each other's device-count records.HAMi #1555
Node-lock annotation parsingReturns a clear error when a node-lock annotation contains a delimiter but has an invalid number of fields; the legacy timestamp-only format remains parseable. The Enterprise default Downward API path does not depend on this node-lock allocation flow.HAMi #1560
Node configuration and selector fixesWrites node configuration to the ConfigMap as JSON text and consistently uses devicePlugin.nvidiaNodeSelector as the NVIDIA node selector. Check key names when migrating old values.HAMi #1558, HAMi #1546
Optional mock device pluginInstalls a mock device plugin for scheduling validation without a GPU. It is disabled by default; mock devices do not perform real GPU computation.HAMi #1534
Optional DRA installation pathThe main Chart includes the hami-dra subchart 0.0.1, which can switch deployment to the Kubernetes Dynamic Resource Allocation path. It is disabled by default. See below for the behavior when switching paths.HAMi #1542

Chart changes​

The first main Chart is named hami; the default application image is ghcr.io/dynamia-ai/hami:v2.7.1-rc0. This tag is used for the scheduler extender, NVIDIA device plugin, and vGPU monitor. The bundled kube-scheduler, certificate helper, and optional subcharts use their own image settings.

SettingDefaultDescription
schedulerNamehami-schedulerScheduler name used by workloads
scheduler.replicas1Deploy one scheduler Pod by default
devicePlugin.nvidiaNodeSelectorgpu: "on"Deploy the NVIDIA device plugin to nodes with this label
devicePlugin.deviceSplitCount10Default number of device-sharing slices
devicePlugin.deviceMemoryScaling1Do not scale up the default allocatable memory capacity
devicePlugin.deviceListStrategyenvvarPass the device list through environment variables by default
mockDevicePlugin.enabledfalseWhen enabled, use mock plugin image tag 0.1.0 by default
dra.enabledfalseWhen enabled, use the DRA subchart instead of the regular scheduler and NVIDIA device plugin from the main Chart

The DRA path must meet the requirements of the companion driver and Kubernetes DRA. Enterprise extension notes for the regular device-plugin path do not automatically apply to it.

Upgrade notes​

Install the first release or migrate from open source​

Use the delivered hami-2.7.1-rc0.tgz, the companion Enterprise runtime, and license materials. If you use a custom image registry or explicit image tags, check the versions of the scheduler extender, device plugin, and monitor together; do not replace only one component.

When migrating from an open-source deployment, check node selection, scheduler integration, and device resource requests against this release's configuration. If old values use devicePlugin.nvidianodeSelector, change the key to devicePlugin.nvidiaNodeSelector. The default selector is gpu: "on"; omitting a custom selector may deploy the plugin to the wrong set of nodes.

The old Iluvatar settings iluvatarResourceName, iluvatarResourceMem, and iluvatarResourceCore are no longer used by the Chart, and old values are not migrated automatically. The former default resources iluvatar.ai/vgpu, iluvatar.ai/vcuda-memory, and iluvatar.ai/vcuda-core are replaced with model-specific resource names. Set devices.iluvatar.enabled: true, update Pod device-count, memory, and compute requests for the actual model, and deploy the companion GPU Manager with the same resource names. Replacing only the Chart while keeping old Pod resource names does not complete the migration.

devicePlugin.nodeConfiguration.config accepts JSON text. Set name to the actual node name; the default example your-node-name does not automatically match a node in your cluster. When devicePlugin.nodeConfiguration.externalConfigName is set, the Chart uses the specified ConfigMap and does not create this node configuration.

Check the Downward API and startup hook​

The default device configuration sets both nvidia.useDownward: true and nvidia.postNvidiaSmi: true. With the Downward API enabled, the latter adds a postStart hook that runs nvidia-smi in containers requesting NVIDIA devices. The application image must provide this command. An existing postStart hook is replaced, not merged automatically.

To disable the startup hook, keep nvidia.useDownward: true in the device configuration and set nvidia.postNvidiaSmi to false. These two fields are not top-level Helm parameters in this release. If you use a custom files/device-config.yaml, it replaces the entire default device configuration; preserve the other device fields your deployment uses rather than supplying only these two. Update the scheduler and device plugin with the modified configuration, then recreate workload Pods whose injection behavior must change.

The first release handles NVIDIA allocation annotations using the regular-container list and does not yet include the later protections for empty allocation lists and out-of-range container indexes. Combinations involving non-GPU containers, init containers, or native sidecars should not be considered fully supported on this basis.

Check quotas and monitoring integrations​

The first release does not include the NVIDIA multi-container quota-aggregation fix. Multiple GPU containers in one Pod may each pass their individual checks even when their combined request exceeds the namespace's remaining quota. Deployments that need this fix can refer to Chart 2.8.0-rc0. The quota boundary for runtime memory adjustment is described above.

Memory-detail metrics come from the vGPU monitor; vGPULicenseRemains comes from the scheduler. Configure collection for both components and use current-value semantics in dashboards and alerts. This release still retains older metrics such as HostGPUMemoryUsage and HostCoreUtilization; do not remove their collection rules based on a removal list from a later release.

See the Chart and appVersion mapping or the software center.

Values reference​

The following parameter descriptions are taken from the README bundled with the Chart 2.7.1-rc0 release package and are preserved as published. Use this version's values.yaml as the source of default configuration.

Expand values table

Global Configuration

ParameterDescriptionDefault Value
global.imageRegistryGlobal Docker image registry""
global.imagePullSecretsGlobal Docker image pull secrets[]
global.imageTagImage tag"v2.7.1"
global.gpuHookPathGPU Hook path/usr/local
global.labelsGlobal labels{}
global.annotationsGlobal annotations{}
global.managedNodeSelectorEnableWhether to enable managed node selectorfalse
global.managedNodeSelector.usageManaged node selector usage"gpu"
nameOverrideName override""
fullnameOverrideFull name override""
namespaceOverrideNamespace override""

Resource Name Configuration

NVIDIA GPU Resources

ParameterDescriptionDefault Value
resourceNameGPU resource name"nvidia.com/gpu"
resourceMemGPU memory resource name"nvidia.com/gpumem"
resourceMemPercentageGPU memory percentage resource name"nvidia.com/gpumem-percentage"
resourceCoresGPU core resource name"nvidia.com/gpucores"
resourcePriorityGPU priority resource name"nvidia.com/priority"

Cambricon MLU Resources

ParameterDescriptionDefault Value
mluResourceNameMLU resource name"cambricon.com/vmlu"
mluResourceMemMLU memory resource name"cambricon.com/mlu.smlu.vmemory"
mluResourceCoresMLU core resource name"cambricon.com/mlu.smlu.vcore"

Hygon DCU Resources

ParameterDescriptionDefault Value
dcuResourceNameDCU resource name"hygon.com/dcunum"
dcuResourceMemDCU memory resource name"hygon.com/dcumem"
dcuResourceCoresDCU core resource name"hygon.com/dcucores"

Metax GPU Resources

ParameterDescriptionDefault Value
metaxResourceNameGPU resource name"metax-tech.com/sgpu"
metaxResourceCoreGPU core resource name"metax-tech.com/vcore"
metaxResourceMemGPU memory resource name"metax-tech.com/vmemory"
metaxsGPUTopologyAwareGPU topology awareness"false"

Enflame GCU Resources

ParameterDescriptionDefault Value
enflameResourceNameVGCUvGCU resource name"enflame.com/vgcu"
enflameResourceNameVGCUPercentagevGCU percentage resource name"enflame.com/vgcu-percentage"

Kunlunxin XPU Resources

ParameterDescriptionDefault Value
kunlunResourceNameXPU resource name"kunlunxin.com/xpu"

Scheduler Configuration

ParameterDescriptionDefault Value
schedulerNameScheduler name"hami-scheduler"
scheduler.nodeNameDefine node name, scheduler will schedule to this node""
scheduler.overwriteEnvWhether to overwrite environment variables"false"
scheduler.defaultSchedulerPolicy.nodeSchedulerPolicyNode scheduler policybinpack
scheduler.defaultSchedulerPolicy.gpuSchedulerPolicyGPU scheduler policyspread
scheduler.metricsBindAddressMetrics bind address":9395"
scheduler.forceOverwriteDefaultSchedulerWhether to force overwrite default schedulertrue
scheduler.livenessProbeWhether to enable liveness probefalse
scheduler.leaderElectWhether to enable leader electiontrue
scheduler.replicasNumber of replicas1

Kube Scheduler Configuration

ParameterDescriptionDefault Value
scheduler.kubeScheduler.enabledWhether to run kube-scheduler container in scheduler podtrue
scheduler.kubeScheduler.image.registryKube scheduler image registry"registry.cn-hangzhou.aliyuncs.com"
scheduler.kubeScheduler.image.repositoryKube scheduler image repository"google_containers/kube-scheduler"
scheduler.kubeScheduler.image.tagKube scheduler image tag""
scheduler.kubeScheduler.image.pullPolicyKube scheduler image pull policyIfNotPresent
scheduler.kubeScheduler.image.pullSecretsKube scheduler image pull secrets[]
scheduler.kubeScheduler.extraNewArgsExtra new arguments["--config=/config/config.yaml", "-v=4"]
scheduler.kubeScheduler.extraArgsExtra arguments["--policy-config-file=/config/config.json", "-v=4"]

Extender Configuration

ParameterDescriptionDefault Value
scheduler.extender.image.registryScheduler extender image registry"docker.io"
scheduler.extender.image.repositoryScheduler extender image repository"projecthami/hami"
scheduler.extender.image.tagScheduler extender image tag""
scheduler.extender.image.pullPolicyScheduler extender image pull policyIfNotPresent
scheduler.extender.image.pullSecretsScheduler extender image pull secrets[]
scheduler.extender.extraArgsScheduler extender extra arguments["--debug", "-v=4"]

Admission Webhook Configuration

ParameterDescriptionDefault Value
scheduler.admissionWebhook.enabledWhether to enable admission webhooktrue
scheduler.admissionWebhook.customURL.enabledWhether to enable custom URLfalse
scheduler.admissionWebhook.customURL.hostCustom URL host127.0.0.1
scheduler.admissionWebhook.customURL.portCustom URL port31998
scheduler.admissionWebhook.customURL.pathCustom URL path/webhook
scheduler.admissionWebhook.reinvocationPolicyReinvocation policyNever
scheduler.admissionWebhook.failurePolicyFailure policyIgnore

TLS Certificate Configuration

ParameterDescriptionDefault Value
scheduler.certManager.enabledWhether to use cert-manager to generate self-signed certificatesfalse
scheduler.patch.enabledWhether to use kube-webhook-certgen to generate self-signed certificatestrue
scheduler.patch.image.registryCertgen image registry"docker.io"
scheduler.patch.image.repositoryCertgen image repository"jettech/kube-webhook-certgen"
scheduler.patch.image.tagCertgen image tag"v1.5.2"
scheduler.patch.image.pullPolicyCertgen image pull policyIfNotPresent
scheduler.patch.imageNew.registryNew certgen image registry"docker.io"
scheduler.patch.imageNew.repositoryNew certgen image repository"liangjw/kube-webhook-certgen"
scheduler.patch.imageNew.tagNew certgen image tag"v1.1.1"

Scheduler Service Configuration

ParameterDescriptionDefault Value
scheduler.service.typeService typeNodePort
scheduler.service.httpPortHTTP port443
scheduler.service.schedulerPortScheduler NodePort31998
scheduler.service.monitorPortMonitor port31993
scheduler.service.monitorTargetPortMonitor target port9395

Device Plugin Configuration

ParameterDescriptionDefault Value
devicePlugin.image.registryDevice plugin image registry"docker.io"
devicePlugin.image.repositoryDevice plugin image repository"projecthami/hami"
devicePlugin.image.tagDevice plugin image tag""
devicePlugin.image.pullPolicyDevice plugin image pull policyIfNotPresent
devicePlugin.image.pullSecretsDevice plugin image pull secrets[]

Monitor Configuration

ParameterDescriptionDefault Value
devicePlugin.monitor.image.registryMonitor image registry"docker.io"
devicePlugin.monitor.image.repositoryMonitor image repository"projecthami/hami"
devicePlugin.monitor.image.tagMonitor image tag""
devicePlugin.monitor.image.pullPolicyMonitor image pull policyIfNotPresent
devicePlugin.monitor.image.pullSecretsMonitor image pull secrets[]
devicePlugin.monitor.ctrPathContainer path/usr/local/vgpu/containers
devicePlugin.monitor.extraArgsMonitor extra arguments["-v=4"]
devicePlugin.monitor.extraEnvsMonitor extra environments{}

Device Plugin Other Configuration

ParameterDescriptionDefault Value
devicePlugin.deviceSplitCountInteger type, default value: 10. Maximum number of tasks assigned to a single GPU device10
devicePlugin.deviceMemoryScalingDevice memory scaling ratio1
devicePlugin.deviceCoreScalingDevice core scaling ratio1
devicePlugin.runtimeClassNameRuntime class name""
devicePlugin.createRuntimeClassWhether to create runtime classfalse
devicePlugin.migStrategyString type, "none" means ignore MIG functionality, "mixed" means allocate MIG devices through independent resources"none"
devicePlugin.disablecorelimitString type, "true" means disable core limit, "false" means enable core limit"false"
devicePlugin.passDeviceSpecsEnabledWhether to enable passing device specsfalse
devicePlugin.extraArgsDevice plugin extra arguments["-v=4"]
devicePlugin.nodeConfiguration.configNode configuration for device plugin by jsonAn example of default configuration.
devicePlugin.nodeConfiguration.externalConfigNameNode configuration for device plugin by external congimap""
devicePlugin.extraEnvsDevice plugin extra environments{}

Device Plugin Service Configuration

ParameterDescriptionDefault Value
devicePlugin.service.typeService typeNodePort
devicePlugin.service.httpPortHTTP port31992

Device Plugin Deployment Configuration

ParameterDescriptionDefault Value
devicePlugin.pluginPathPlugin path/var/lib/kubelet/device-plugins
devicePlugin.libPathLibrary path/usr/local/vgpu
devicePlugin.nvidiaNodeSelectorNVIDIA node selector{"gpu": "on"}
devicePlugin.updateStrategy.typeUpdate strategy typeRollingUpdate
devicePlugin.updateStrategy.rollingUpdate.maxUnavailableMaximum unavailable count1

Device Configuration

AWS Neuron

ParameterDescriptionDefault Value
devices.awsneuron.customresourcesCustom resources["aws.amazon.com/neuron", "aws.amazon.com/neuroncore"]

Kunlunxin

ParameterDescriptionDefault Value
devices.kunlun.enabledWhether to enabletrue
devices.kunlun.customresourcesCustom resources["kunlunxin.com/xpu"]

Mthreads

ParameterDescriptionDefault Value
devices.mthreads.enabledWhether to enabletrue
devices.mthreads.customresourcesCustom resources["mthreads.com/vgpu"]

NVIDIA

ParameterDescriptionDefault Value
devices.nvidia.gpuCorePolicyGPU core policydefault
devices.nvidia.libCudaLogLevelCUDA library log level1

Huawei Ascend

ParameterDescriptionDefault Value
devices.ascend.enabledWhether to enablefalse
devices.ascend.imageImage""
devices.ascend.imagePullPolicyImage pull policyIfNotPresent
devices.ascend.extraArgsExtra arguments[]
devices.ascend.nodeSelectorNode selector{"ascend": "on"}
devices.ascend.tolerationsTolerations[]
devices.ascend.customresourcesCustom resources["huawei.com/Ascend910A", "huawei.com/Ascend910A-memory", ...]

Iluvatar

ParameterDescriptionDefault Value
devices.iluvatar.enabledWhether to enablefalse
devices.iluvatar.customresourcesCustom resources["iluvatar.ai/BI-V150-vgpu", "iluvatar.ai/BI-V150.vMem","iluvatar.ai/BI-V150.vCore", ...]