Skip to main content

HAMi Enterprise 2.10.0-r4

Chart versionDeclared appVersionRelease date
2.10.0-r42.10.0-r42026-10-15

This is a patch release. Changes below are compared with HAMi Enterprise v2.10.0-r3.

Software changes​

  • Device annotations for NVIDIA multi-container workloads: Fixes a mismatch between device-allocation positions and containers in Pods containing init containers, sidecars, and GPU workload containers when scheduler.useDownward is enabled. GPU devices, memory, and compute limits are now annotated on the containers that actually request the devices. Containers that do not request devices do not receive these annotations.
  • NVIDIA node-registration cleanup: When the device plugin exits normally, it removes the node registration and related annotations that it created. During a rolling update, an old instance does not remove registration information already taken over by a new instance. This cleanup does not apply to forced termination.
  • Iluvatar device-resource defaults: For a single-device request without an explicit vCore value, the default is now the full device's 100 vCores; an explicitly configured single-device vCore value is unchanged. Multi-device requests use a limit of 100 vCores per device. Before upgrading, review workloads that use custom vCore limits for multi-device requests.

Chart changes​

  • Environment variables for NVIDIA device-plugin and monitor containers: devicePlugin.extraEnvs and devicePlugin.monitor.extraEnvs now use Kubernetes env lists and both default to []. This fixes a type mismatch when environment variables are configured as a list and makes the two settings consistent.
  • The main Chart in this release is 2.10.0-r4. The companion CRD Chart 0.4.2 and Resource Pool Manager Chart 0.0.6 have not changed versions since v2.10.0-r3.

Upgrade notes​

Before upgrading, check custom values for devicePlugin.extraEnvs and devicePlugin.monitor.extraEnvs. If they are still written as {} or another map, change them to a list. Use [] when no additional environment variables are needed. Example with non-empty values:

devicePlugin:
extraEnvs:
- name: EXAMPLE_DEVICE_PLUGIN_ENV
value: example
monitor:
extraEnvs:
- name: EXAMPLE_MONITOR_ENV
value: example

Before upgrading, also review resource manifests and quota expectations for Iluvatar workloads that rely on the old behavior when no vCore value is specified, or use custom vCore limits for multi-device requests.

Obtain the package and component manifest through the agreed delivery channel. See the Chart and appVersion mapping or the software center.

Values reference​

The following parameter descriptions are taken from the README bundled with the Chart 2.10.0-r4 release package and are preserved as published. Use this version's values.yaml as the source of default configuration.

Expand values table
KeyTypeDefaultDescription
birenResourceNamestring"birentech.com/gpu"Biren GPU resource name
device-config.contentstring""
devicePlugin.cdiVendorstring""CDI vendor for generated specs and device annotations (default: k8s.device-plugin.nvidia.com)
devicePlugin.createRuntimeClassboolfalseWhether to create runtime class
devicePlugin.deviceCoreScalingint1Device core scaling factor
devicePlugin.deviceListStrategystring"envvar"Device list strategy. Supported values: envvar, volume-mounts, cdi-annotations
devicePlugin.deviceMemoryScalingint1Device memory scaling factor
devicePlugin.deviceSplitCountint10Number of slices to split each GPU into
devicePlugin.disablecorelimitstring"false"Disable core limit
devicePlugin.enableNumaTopologyboolfalseAdvertise the physical GPU's NUMA node on each vGPU replica
devicePlugin.enabledbooltrueEnable device plugin
devicePlugin.extraArgslist["-v=4"]Extra arguments for device plugin
devicePlugin.extraEnvslist[]Extra environment variables for device plugin as a Kubernetes env list.
devicePlugin.gdrcopyEnabledboolfalseEnable GDRCopy
devicePlugin.gdsEnabledboolfalseEnable GDS
devicePlugin.gpuOperatorToolkitReady.enabledboolfalseEnable GPU operator toolkit ready check
devicePlugin.gpuOperatorToolkitReady.hostPathstring"/run/nvidia/validations"Host path for GPU operator toolkit ready check
devicePlugin.gpuOperatorToolkitReady.securityContextobject{"privileged":true,"runAsUser":0}Security context for the GPU operator toolkit validation init container
devicePlugin.hostNetworkboolfalseShare the host network namespace with the device plugin pod
devicePlugin.hostPIDbooltrueShare the host PID namespace with the device plugin pod
devicePlugin.image.pullPolicystring"IfNotPresent"Device plugin image pull policy
devicePlugin.image.pullSecretslist[]Specify docker-registry secret names as an array
devicePlugin.image.registrystring"ghcr.io"Device plugin image registry
devicePlugin.image.repositorystring"dynamia-ai/hami-enterprise"Device plugin image repository
devicePlugin.image.tagstring""Device plugin image tag (immutable tags are recommended)
devicePlugin.libPathstring"/usr/local/vgpu"Library path for vGPU
devicePlugin.migStrategystring"none"MIG strategy. Supported values: none, single, mixed
devicePlugin.mofedEnabledboolfalseEnable Mellanox OFED
devicePlugin.monitor.ctrPathstring"/usr/local/vgpu/containers"Container path for monitor
devicePlugin.monitor.extraArgslist["-v=4"]Extra arguments for monitor
devicePlugin.monitor.extraEnvslist[]Extra environment variables for monitor as a Kubernetes env list.
devicePlugin.monitor.image.pullPolicystring"IfNotPresent"Monitor image pull policy
devicePlugin.monitor.image.pullSecretslist[]Specify docker-registry secret names as an array
devicePlugin.monitor.image.registrystring"ghcr.io"Monitor image registry
devicePlugin.monitor.image.repositorystring"dynamia-ai/hami-enterprise"Monitor image repository
devicePlugin.monitor.image.tagstring""Monitor image tag (immutable tags are recommended)
devicePlugin.monitor.resourcesobject{}Resources for monitor container
devicePlugin.monitor.resyncIntervalstring"5m"Resync interval for monitor
devicePlugin.monitor.securityContextobject{"allowPrivilegeEscalation":false,"capabilities":{"add":["SYS_ADMIN"],"drop":["ALL"]}}Security context for the monitor container
devicePlugin.nodeConfiguration.configstring"{\n \"nodeconfig\": [\n {\n \"name\": \"your-node-name\",\n \"operatingmode\": \"hami-core\",\n \"devicememoryscaling\": 1,\n \"devicesplitcount\": 10,\n \"preconfigureddevicememory\": 0,\n \"enablenumatopology\": false,\n \"migstrategy\": \"none\",\n \"filterdevices\": {\n \"uuid\": [],\n \"index\": []\n },\n \"enablegetpreferredallocation\": false\n }\n ]\n}\n"Node configuration for device plugin. If set, overrides the default config.json
devicePlugin.nodeConfiguration.externalConfigNamestring""Name of an existing ConfigMap to use for node configuration
devicePlugin.numaRefitobject{"caFile":"","caSecret":"","enabled":false,"schedulerEndpoint":"","tlsInsecure":true}NUMA alignment refit configuration
devicePlugin.numaRefit.caFilestring""Path to a CA bundle already mounted in the device-plugin Pod
devicePlugin.numaRefit.caSecretstring""Secret containing ca.crt, mounted at /etc/hami/numa-refit-ca
devicePlugin.numaRefit.enabledboolfalseEnable NUMA alignment refit
devicePlugin.numaRefit.schedulerEndpointstring""Scheduler base URL override. Defaults to the in-cluster Scheduler service
devicePlugin.numaRefit.tlsInsecurebooltrueSkip Scheduler TLS verification. Set false with caFile or caSecret to verify TLS
devicePlugin.nvidiaDriverRootstring""NVIDIA driver root on the host, mounted into the device plugin and monitor containers
devicePlugin.nvidiaHookPathstring""Path for NVIDIA hook
devicePlugin.nvidiaNodeSelectorobject{"gpu":"on"}Node selector for NVIDIA device plugin
devicePlugin.passDeviceSpecsEnabledboolfalseEnable passing device specs
devicePlugin.pluginPathstring"/var/lib/kubelet/device-plugins"Path for device plugin socket
devicePlugin.podAnnotationsobject{}Annotations for device plugin pod
devicePlugin.preConfiguredDeviceMemoryint0Pre-configured device memory in MB for GPUs that don't support memory query. Set to 0 for auto-detection
devicePlugin.resourcesobject{}Resources for device plugin container
devicePlugin.runtimeClassNamestring""Runtime class name to be used by the device plugin
devicePlugin.securityContextobject{"allowPrivilegeEscalation":true,"capabilities":{"add":["SYS_ADMIN"],"drop":["ALL"]},"privileged":true}Security context for the device plugin container
devicePlugin.service.annotationsobject{}Annotations for device plugin service
devicePlugin.service.httpPortint31992HTTP port for device plugin service
devicePlugin.service.labelsobject{}Labels for device plugin service
devicePlugin.service.typestring"NodePort"Service type for device plugin
devicePlugin.tolerationslist[{"effect":"NoSchedule","key":"nvidia.com/gpu","operator":"Exists"}]Tolerations for device plugin pod
devicePlugin.updateStrategy.rollingUpdate.maxUnavailableint1Maximum unavailable pods during update
devicePlugin.updateStrategy.typestring"RollingUpdate"Update strategy for device plugin DaemonSet
devices.alibaba.customresourceslist["alibaba.com/ppu","alibaba.com/ppu-memory","alibaba.com/gpu-core"]Custom resource definitions for Alibaba devices
devices.alibaba.enabledbooltrueEnable Alibaba PPU device
devices.alibaba.ppuCorePolicystring"default"PPU core policy for Alibaba devices
devices.amd.customresourceslist["amd.com/gpu","amd.com/gpumem","amd.com/gpucores"]Custom resource definitions for AMD devices
devices.ascend.customresourceslist["huawei.com/Ascend910A","huawei.com/Ascend910A-memory","huawei.com/Ascend910A-core","huawei.com/Ascend910B2","huawei.com/Ascend910B2-memory","huawei.com/Ascend910B2-core","huawei.com/Ascend910B3","huawei.com/Ascend910B3-memory","huawei.com/Ascend910B3-core","huawei.com/Ascend910B4","huawei.com/Ascend910B4-memory","huawei.com/Ascend910B4-core","huawei.com/Ascend910B4-1","huawei.com/Ascend910B4-1-memory","huawei.com/Ascend910B4-1-core","huawei.com/Ascend310P","huawei.com/Ascend310P-memory","huawei.com/Ascend310P-core","huawei.com/Ascend910C","huawei.com/Ascend910C-memory","huawei.com/Ascend910C-core"]Custom resource definitions for Ascend devices
devices.ascend.enabledboolfalseEnable Ascend device plugin
devices.ascend.extraArgslist[]Extra arguments for Ascend device plugin
devices.ascend.hamiVnpuCoreboolfalseEnable soft-partitioning based on hami-vnpu-core (node-level annotation takes higher priority)
devices.ascend.imagestring""Ascend device plugin image
devices.ascend.imagePullPolicystring"IfNotPresent"Image pull policy for Ascend device plugin
devices.ascend.nodeSelectorobject{"ascend":"on"}Node selector for Ascend device plugin
devices.ascend.runtimeClassNamestring""Runtime class name for Ascend NPU pods
devices.ascend.tolerationslist[]Tolerations for Ascend device plugin
devices.ascend.vnpuDeviceSplitCountint10Number of virtual devices advertised per physical Ascend NPU in soft-slicing mode. A positive node-level vDeviceCount overrides it.
devices.awsneuron.customresourceslist["aws.amazon.com/neuron","aws.amazon.com/neuroncore"]Custom resource definitions for AWS Neuron devices
devices.biren.customresourceslist["birentech.com/gpu"]Custom resource definitions for Biren devices
devices.biren.enabledbooltrueEnable Biren device plugin
devices.enflame.customresourceslist["enflame.com/drs-gcu","enflame.com/gcu-memory","enflame.com/gcu-core","enflame.com/gcu","enflame.com/vgcu","enflame.com/vgcu-percentage"]Custom resource definitions for Enflame devices
devices.enflame.enabledbooltrueEnable Enflame device plugin
devices.iluvatar.customresourceslist["iluvatar.ai/BI-V100-vgpu","iluvatar.ai/BI-V100.vCore","iluvatar.ai/BI-V100.vMem","iluvatar.ai/BI-V150-vgpu","iluvatar.ai/BI-V150.vCore","iluvatar.ai/BI-V150.vMem","iluvatar.ai/MR-V100-vgpu","iluvatar.ai/MR-V100.vCore","iluvatar.ai/MR-V100.vMem","iluvatar.ai/MR-V50-vgpu","iluvatar.ai/MR-V50.vCore","iluvatar.ai/MR-V50.vMem"]Custom resource definitions for Iluvatar devices
devices.iluvatar.enabledboolfalseEnable Iluvatar device plugin
devices.kunlun.customresourceslist["kunlunxin.com/xpu","kunlunxin.com/vxpu","kunlunxin.com/vxpu-memory"]Custom resource definitions for Kunlun devices
devices.kunlun.enabledbooltrueEnable Kunlun device plugin
devices.mthreads.customresourceslist["mthreads.com/vgpu"]Custom resource definitions for MThreads devices
devices.mthreads.enabledbooltrueEnable MThreads device plugin
devices.nvidia.gpuCorePolicystring"default"GPU core policy for NVIDIA devices
devices.nvidia.libCudaLogLevelint2libCUDA log level
devices.vastai.customresourceslist["vastaitech.com/va"]Custom resource definitions for Vastai devices
devices.vastai.enabledbooltrueEnable Vastai device plugin
dra.enabledboolfalseEnable DRA (Dynamic Resource Allocation)
enflameResourceNameDRSGCUstring"enflame.com/drs-gcu"Enflame DRS GCU resource name
enflameResourceNameGCUCorestring"enflame.com/gcu-core"Enflame GCU core resource name
enflameResourceNameGCUMemorystring"enflame.com/gcu-memory"Enflame GCU memory resource name
enflameResourceNameVGCUstring"enflame.com/vgcu"Enflame VGCU resource name
enflameResourceNameVGCUPercentagestring"enflame.com/vgcu-percentage"Enflame VGCU percentage resource name
fullnameOverridestring""Override the fullname of the chart
global.annotationsobject{}Global annotations to add to all resources
global.gpuHookPathstring"/usr/local"Path for GPU hook installation on nodes
global.imagePullSecretslist[]Global Docker image pull secrets
global.imageRegistrystring""Global Docker image registry
global.imageTagstring"v2.10.0-r4"Global image tag for HAMi Enterprise components
global.labelsobject{}Global labels to add to all resources
global.managedNodeSelectorobject{"usage":"gpu"}Node selector configuration for managed GPU nodes
global.managedNodeSelectorEnableboolfalseEnable managed node selector for GPU nodes
hami-dra.drivers.nvidia.containerDriverbooltrueEnable container driver for NVIDIA DRA
hami-dra.drivers.nvidia.enabledbooltrueEnable NVIDIA DRA driver
hami-dra.drivers.nvidia.image.repositorystring"ghcr.io/project-hami/k8s-dra-driver"NVIDIA DRA driver image repository
hami-dra.drivers.nvidia.image.tagstring"main"NVIDIA DRA driver image tag
hami-dra.monitor.enabledbooltrueEnable monitor for hami-dra
hcuResourceCoresstring"hygon.com/hcucores"Hygon HCU cores resource name
hcuResourceMemstring"hygon.com/hcumem"Hygon HCU memory resource name
hcuResourceNamestring"hygon.com/hcunum"Hygon HCU resource name
kunlunResourceNamestring"kunlunxin.com/xpu"Kunlun XPU resource name
kunlunResourceVCountNamestring"kunlunxin.com/vxpu"Kunlun virtual XPU count resource name
kunlunResourceVMemoryNamestring"kunlunxin.com/vxpu-memory"Kunlun virtual XPU memory resource name
legacyMetricsboolfalseEnable legacy metrics
metaxResourceCorestring"metax-tech.com/vcore"Metax sGPU core resource name
metaxResourceMemstring"metax-tech.com/vmemory"Metax sGPU memory resource name
metaxResourceNamestring"metax-tech.com/sgpu"Metax sGPU resource name
metaxsGPUTopologyAwarestring"false"Enable Metax sGPU topology awareness
mluResourceCoresstring"cambricon.com/mlu.smlu.vcore"MLU cores resource name
mluResourceMemstring"cambricon.com/mlu.smlu.vmemory"MLU memory resource name
mluResourceNamestring"cambricon.com/vmlu"MLU resource name
mockDevicePlugin.enabledboolfalseEnable mock device plugin
mockDevicePlugin.image.pullPolicystring"IfNotPresent"Mock device plugin image pull policy
mockDevicePlugin.image.pullSecretslist[]Specify docker-registry secret names as an array
mockDevicePlugin.image.registrystring"docker.io"Mock device plugin image registry
mockDevicePlugin.image.repositorystring"projecthami/mock-device-plugin"Mock device plugin image repository
mockDevicePlugin.image.tagstring"1.0.1"Mock device plugin image tag
nameOverridestring""Override the chart name
namespaceOverridestring""Override the namespace
openshift.securityContextConstraints.createbooltrueCreate the named device-plugin SCC and its use ClusterRole when OpenShift support is enabled. Set this to false only when both the SCC and system:openshift:scc:<name> ClusterRole already exist, such as for the built-in privileged SCC.
openshift.securityContextConstraints.namestring"hami-device-plugin"SCC granted to enabled device-plugin service accounts. When create=false, the matching system:openshift:scc:<name> ClusterRole must already exist. The built-in privileged SCC requires create=false.
platform.openshiftboolfalseEnable OpenShift-specific resources and handling
platform.openshiftVersionstring""OpenShift version used to resolve the Red Hat OpenShift hyperkube image tag when platform.openshift=true and scheduler.kubeScheduler.image.tag is empty
podSecurityPolicy.enabledboolfalseEnable PodSecurityPolicy
prometheus.enabledboolfalseEnable Prometheus metrics
resourceCoresstring"nvidia.com/gpucores"NVIDIA GPU cores resource name
resourceMemstring"nvidia.com/gpumem"NVIDIA GPU memory resource name
resourceMemPercentagestring"nvidia.com/gpumem-percentage"NVIDIA GPU memory percentage resource name
resourceNamestring"nvidia.com/gpu"NVIDIA GPU resource name
resourcePrioritystring"nvidia.com/priority"NVIDIA GPU priority resource name
scheduler.admissionWebhook.customURL.enabledboolfalseEnable custom webhook URL
scheduler.admissionWebhook.customURL.hoststring"127.0.0.1"Webhook host (must be an endpoint using https)
scheduler.admissionWebhook.customURL.pathstring"/webhook"Webhook path
scheduler.admissionWebhook.customURL.portint31998Webhook port
scheduler.admissionWebhook.enabledbooltrueEnable admission webhook. If disabled, pods must be configured with correct schedulerName manually
scheduler.admissionWebhook.failurePolicystring"Ignore"Failure policy for webhook
scheduler.admissionWebhook.manageNamespaceSelectorbooltrueWhether the chart renders and manages the webhook's namespaceSelector field
scheduler.admissionWebhook.namespaceSelectorobject{"matchExpressions":[],"matchLabels":{}}Namespace selector for webhook
scheduler.admissionWebhook.objectSelectorobject{"matchExpressions":[]}Object selector for webhook
scheduler.admissionWebhook.reinvocationPolicystring"Never"Reinvocation policy for webhook
scheduler.admissionWebhook.whitelistNamespaceslist[]Namespaces that the webhook will not be applied to
scheduler.certManager.enabledboolfalseEnable cert-manager for TLS certificate generation
scheduler.defaultSchedulerPolicy.gpuSchedulerPolicystring"spread"GPU-level scheduling policy. Supported values: binpack, spread
scheduler.defaultSchedulerPolicy.nodeSchedulerPolicystring"binpack"Node-level scheduling policy. Supported values: binpack, spread
scheduler.extender.extraArgslist["--debug","-v=4"]Extra arguments for scheduler extender
scheduler.extender.image.pullPolicystring"IfNotPresent"Scheduler extender image pull policy
scheduler.extender.image.pullSecretslist[]Specify docker-registry secret names as an array
scheduler.extender.image.registrystring"ghcr.io"Scheduler extender image registry
scheduler.extender.image.repositorystring"dynamia-ai/hami-enterprise"Scheduler extender image repository
scheduler.extender.image.tagstring""Scheduler extender image tag (immutable tags are recommended)
scheduler.extender.resourcesobject{}Resources for scheduler extender container
scheduler.forceOverwriteDefaultSchedulerbooltrueForce overwrite the default scheduler name in pods when it equals the K8s default scheduler name
scheduler.kubeScheduler.enabledbooltrueWhether to run kube-scheduler container in the scheduler pod
scheduler.kubeScheduler.extraArgslist["--policy-config-file=/config/config.json","-v=4"]Extra arguments for kube-scheduler (legacy config format)
scheduler.kubeScheduler.extraNewArgslist["--config=/config/config.yaml","-v=4"]Extra arguments for kube-scheduler (new config format)
scheduler.kubeScheduler.image.pullPolicystring"IfNotPresent"kube-scheduler image pull policy
scheduler.kubeScheduler.image.pullSecretslist[]Specify docker-registry secret names as an array
scheduler.kubeScheduler.image.registrystring"registry.cn-hangzhou.aliyuncs.com"kube-scheduler image registry
scheduler.kubeScheduler.image.repositorystring"google_containers/kube-scheduler"kube-scheduler image repository
scheduler.kubeScheduler.image.tagstring""kube-scheduler image tag (immutable tags are recommended)
scheduler.kubeScheduler.resourcesobject{}Resources for kube-scheduler container
scheduler.leaderElectbooltrueEnable leader election for scheduler
scheduler.livenessProbeboolfalseEnable liveness probe for scheduler
scheduler.metricsBindAddressstring":9395"Address to bind metrics server
scheduler.nodeLockExpirestring"5m"Node lock expiration time
scheduler.nodeNamestring""Node name for scheduler pod. If installing as default scheduler, set this to skip the schedule workflow
scheduler.overwriteEnvstring"false"Overwrite environment variables
scheduler.patch.enabledbooltrueEnable kube-webhook-certgen patch job
scheduler.patch.image.pullPolicystring"IfNotPresent"kube-webhook-certgen image pull policy
scheduler.patch.image.pullSecretslist[]Specify docker-registry secret names as an array
scheduler.patch.image.registrystring"docker.io"kube-webhook-certgen image registry
scheduler.patch.image.repositorystring"jettech/kube-webhook-certgen"kube-webhook-certgen image repository
scheduler.patch.image.tagstring"v1.5.2"kube-webhook-certgen image tag
scheduler.patch.imageNew.pullPolicystring"IfNotPresent"Kubernetes 1.22+ kube-webhook-certgen image pull policy
scheduler.patch.imageNew.pullSecretslist[]Specify docker-registry secret names as an array
scheduler.patch.imageNew.registrystring"registry.k8s.io"Kubernetes 1.22+ kube-webhook-certgen image registry
scheduler.patch.imageNew.repositorystring"ingress-nginx/kube-webhook-certgen"Kubernetes 1.22+ kube-webhook-certgen image repository
scheduler.patch.imageNew.tagstring"v1.6.9"Kubernetes 1.22+ kube-webhook-certgen image tag
scheduler.patch.nodeSelectorobject{}Node selector for patch job
scheduler.patch.podAnnotationsobject{}Annotations for patch job pod
scheduler.patch.priorityClassNamestring""Priority class name for patch job
scheduler.patch.runAsUserint2000User ID to run patch job as. OpenShift omits this value at render time so its SCC can allocate an allowed UID
scheduler.patch.tolerationslist[]Tolerations for patch job
scheduler.podAnnotationsobject{}Annotations for scheduler pod
scheduler.replicasint1Number of scheduler replicas (only effective when leaderElect is true)
scheduler.service.annotationsobject{}Annotations for scheduler service
scheduler.service.httpPortint443HTTP port for scheduler service
scheduler.service.httpTargetPortint9443Target port on which the scheduler extender listens
scheduler.service.labelsobject{}Labels for scheduler service
scheduler.service.monitorPortint31993Monitoring port for scheduler service
scheduler.service.monitorTargetPortint9395Target port for monitoring
scheduler.service.schedulerPortint31998NodePort for scheduler HTTP
scheduler.service.typestring"NodePort"Service type for scheduler
scheduler.tolerationslist[]Tolerations for scheduler pod
scheduler.useDownwardbooltrueWhether to enable useDownward in the generated scheduler device config for NVIDIA devices
schedulerNamestring"hami-scheduler"Name of the scheduler
selinux.enabledboolfalseRelabel shared vGPU host directories on SELinux-enabled Kubernetes nodes
selinux.levelstring"s0"SELinux level applied to shared vGPU host directories
selinux.typestring"container_file_t"SELinux type applied to shared vGPU host directories
vastaiResourceNamestring"vastaitech.com/va"Vastai resource name