跳到主要内容

HAMi Enterprise 2.8.0-rc1

Chart 版本Chart 声明的 appVersion发布日期
2.8.0-rc12.8.0-rc12026 年 3 月 16 日

本次为 patch 版本,以下变化以 HAMi Enterprise v2.8.0-rc0 为比较基线。

软件变更​

  • 阿里云 PPU 调度接入:调度器开始识别 PPU 节点注册信息和 alibaba.com/ppu、alibaba.com/ppu-memory、alibaba.com/gpu-core 资源。此 Chart 不安装 PPU 设备插件;使用该能力前,仍需由配套设备插件在节点上注册设备与资源。
  • 统一内存 NVIDIA GPU 的显存登记:当 NVML 不支持查询设备显存时,设备插件可使用预设显存容量登记设备。未设置正值时,这类设备会被跳过;NVML 可正常查询显存的设备继续使用查询结果。
  • Ascend 910C 模组分配:启用 Ascend 且设备配置包含 commonWord: Ascend910C 时,单容器申请 1 个 NPU 会调整为 2 个,以完整模组分配;其他奇数申请会被拒绝。发布 Chart 的默认设备配置尚无 910C 条目。
  • 显式指定调度器的 Pod 准入:修复 spec.schedulerName 已设为 HAMi 调度器名称时,准入 webhook 可能跳过设备处理的问题;该名称由 Chart 的 schedulerName 配置。
  • NVIDIA 节点状态与工作负载启动:调度器在可分配 GPU 数量归零时清理节点设备状态,并在设备注册标注变化时更新设备信息。NVIDIA 配置启用 postNvidiaSmi 且工作负载已有 Exec 型 postStart 命令时,原命令成功后才执行 nvidia-smi;串联命令需要容器镜像提供 sh。
  • NVIDIA 命名空间配额:Pod 准入时会累计各容器中声明为单卡的 NVIDIA 显存和算力请求,并检查命名空间已配置的相应 ResourceQuota;超出配额时返回 exceeding resource quota。此项准入检查不覆盖容器声明多卡的请求。
  • 监控指标调整:容器显存明细指标 hami_container_device_memory_context_bytes、hami_container_device_memory_module_bytes、hami_container_device_memory_data_bytes 分别改为 vGPU_device_memory_context_size_bytes、vGPU_device_memory_module_size_bytes、vGPU_device_memory_buffer_size_bytes;旧的 hami_container_device_memory_offset_bytes 没有同名替代项。调度器的 GPUDeviceMemoryLimit、GPUDeviceCoreLimit、GPUDeviceMemoryAllocated、GPUDeviceCoreAllocated、vGPULicenseRemains 增加 devicetype 标签;其中 GPUDeviceMemoryAllocated 还增加 devicecores 标签,nodeGPUOverview 也增加 devicecores 标签。

Chart 变更​

  • 新增 devicePlugin.preConfiguredDeviceMemory,单位为 MB,默认 0。它写入调度器设备配置;可通过 devicePlugin.nodeConfiguration.config 中对应节点的 preconfigureddevicememory 覆盖设备插件配置。节点级正值可覆盖通用配置。
  • 新增 devices.alibaba.enabled(默认 true)和 devices.alibaba.ppuCorePolicy(默认 default),用于 PPU 资源映射和算力策略配置;策略可取 default、force 或 disable。
  • 新增 prometheus.enabled,默认 false。设为 true 且集群已有 monitoring.coreos.com/v1/ServiceMonitor 资源类型时,Chart 会为调度器和 NVIDIA 设备插件创建 ServiceMonitor;Chart 不安装该资源类型。
  • 主 Chart 的 version 和 appVersion 均为 2.8.0-rc1,默认镜像标签调整为 v2.8.0-rc1。

升级说明​

在 NVML 无法查询显存的 NVIDIA 节点上,将 devicePlugin.preConfiguredDeviceMemory 设置为实际容量的 MB 数值。需要按节点设置不同容量时,在 devicePlugin.nodeConfiguration.config 中按节点名称填写正值 preconfigureddevicememory;其他 NVIDIA 节点可以保留默认值。

使用 Ascend 910C 的部署需在 devices.ascend.enabled: true 的基础上,在自定义 Chart 包中提供包含 commonWord: Ascend910C 和对应资源名的 files/device-config.yaml,并将这些资源名加入 devices.ascend.customresources。自定义文件会替换 Chart 生成的整份设备配置,原有设备条目需一并保留;默认 Chart 不含 910C 配置。已有 910C 工作负载应检查奇数 NPU 申请及单卡申请被调整为双 NPU 后的容量预期。

已有自定义 Exec 型 postStart 命令的 NVIDIA 工作负载,应确认镜像提供 sh,且原命令及其参数在 shell 中执行的结果符合预期;原命令失败时不会执行 nvidia-smi。

使用 Prometheus Operator 的部署可按需启用 prometheus.enabled。升级前检查依赖上述旧指标名或精确匹配标签的看板、告警和查询规则,并迁移到新指标及标签;依赖显存 offset 指标的规则需要单独调整。

安装包通过约定的交付渠道获取。查看Chart 与 appVersion 对应关系或软件中心。

Values 说明​

以下参数说明取自 Chart 2.8.0-rc1 发布包随附的 README,保留原文。默认配置以该版本的 values.yaml 为准。

展开参数表

Global Configuration

ParameterDescriptionDefault Value
global.imageRegistryGlobal Docker image registry""
global.imagePullSecretsGlobal Docker image pull secrets[]
global.imageTagImage tag"v2.8.0"
global.gpuHookPathGPU Hook path/usr/local
global.labelsGlobal labels{}
global.annotationsGlobal annotations{}
global.managedNodeSelectorEnableWhether to enable managed node selectorfalse
global.managedNodeSelector.usageManaged node selector usage"gpu"
nameOverrideName override""
fullnameOverrideFull name override""
namespaceOverrideNamespace override""

Resource Name Configuration

NVIDIA GPU Resources

ParameterDescriptionDefault Value
resourceNameGPU resource name"nvidia.com/gpu"
resourceMemGPU memory resource name"nvidia.com/gpumem"
resourceMemPercentageGPU memory percentage resource name"nvidia.com/gpumem-percentage"
resourceCoresGPU core resource name"nvidia.com/gpucores"
resourcePriorityGPU priority resource name"nvidia.com/priority"

Cambricon MLU Resources

ParameterDescriptionDefault Value
mluResourceNameMLU resource name"cambricon.com/vmlu"
mluResourceMemMLU memory resource name"cambricon.com/mlu.smlu.vmemory"
mluResourceCoresMLU core resource name"cambricon.com/mlu.smlu.vcore"

Hygon DCU Resources

ParameterDescriptionDefault Value
dcuResourceNameDCU resource name"hygon.com/dcunum"
dcuResourceMemDCU memory resource name"hygon.com/dcumem"
dcuResourceCoresDCU core resource name"hygon.com/dcucores"

Metax GPU Resources

ParameterDescriptionDefault Value
metaxResourceNameGPU resource name"metax-tech.com/sgpu"
metaxResourceCoreGPU core resource name"metax-tech.com/vcore"
metaxResourceMemGPU memory resource name"metax-tech.com/vmemory"
metaxsGPUTopologyAwareGPU topology awareness"false"

Enflame GCU Resources

ParameterDescriptionDefault Value
enflameResourceNameVGCUvGCU resource name"enflame.com/vgcu"
enflameResourceNameVGCUPercentagevGCU percentage resource name"enflame.com/vgcu-percentage"

Kunlunxin XPU Resources

ParameterDescriptionDefault Value
kunlunResourceNameXPU resource name"kunlunxin.com/xpu"

Scheduler Configuration

ParameterDescriptionDefault Value
schedulerNameScheduler name"hami-scheduler"
scheduler.nodeNameDefine node name, scheduler will schedule to this node""
scheduler.overwriteEnvWhether to overwrite environment variables"false"
scheduler.defaultSchedulerPolicy.nodeSchedulerPolicyNode scheduler policybinpack
scheduler.defaultSchedulerPolicy.gpuSchedulerPolicyGPU scheduler policyspread
scheduler.metricsBindAddressMetrics bind address":9395"
scheduler.forceOverwriteDefaultSchedulerWhether to force overwrite default schedulertrue
scheduler.livenessProbeWhether to enable liveness probefalse
scheduler.leaderElectWhether to enable leader electiontrue
scheduler.replicasNumber of replicas1

Kube Scheduler Configuration

ParameterDescriptionDefault Value
scheduler.kubeScheduler.enabledWhether to run kube-scheduler container in scheduler podtrue
scheduler.kubeScheduler.image.registryKube scheduler image registry"registry.cn-hangzhou.aliyuncs.com"
scheduler.kubeScheduler.image.repositoryKube scheduler image repository"google_containers/kube-scheduler"
scheduler.kubeScheduler.image.tagKube scheduler image tag""
scheduler.kubeScheduler.image.pullPolicyKube scheduler image pull policyIfNotPresent
scheduler.kubeScheduler.image.pullSecretsKube scheduler image pull secrets[]
scheduler.kubeScheduler.extraNewArgsExtra new arguments["--config=/config/config.yaml", "-v=4"]
scheduler.kubeScheduler.extraArgsExtra arguments["--policy-config-file=/config/config.json", "-v=4"]

Extender Configuration

ParameterDescriptionDefault Value
scheduler.extender.image.registryScheduler extender image registry"docker.io"
scheduler.extender.image.repositoryScheduler extender image repository"projecthami/hami"
scheduler.extender.image.tagScheduler extender image tag""
scheduler.extender.image.pullPolicyScheduler extender image pull policyIfNotPresent
scheduler.extender.image.pullSecretsScheduler extender image pull secrets[]
scheduler.extender.extraArgsScheduler extender extra arguments["--debug", "-v=4"]

Admission Webhook Configuration

ParameterDescriptionDefault Value
scheduler.admissionWebhook.enabledWhether to enable admission webhooktrue
scheduler.admissionWebhook.customURL.enabledWhether to enable custom URLfalse
scheduler.admissionWebhook.customURL.hostCustom URL host127.0.0.1
scheduler.admissionWebhook.customURL.portCustom URL port31998
scheduler.admissionWebhook.customURL.pathCustom URL path/webhook
scheduler.admissionWebhook.reinvocationPolicyReinvocation policyNever
scheduler.admissionWebhook.failurePolicyFailure policyIgnore

TLS Certificate Configuration

ParameterDescriptionDefault Value
scheduler.certManager.enabledWhether to use cert-manager to generate self-signed certificatesfalse
scheduler.patch.enabledWhether to use kube-webhook-certgen to generate self-signed certificatestrue
scheduler.patch.image.registryCertgen image registry"docker.io"
scheduler.patch.image.repositoryCertgen image repository"jettech/kube-webhook-certgen"
scheduler.patch.image.tagCertgen image tag"v1.5.2"
scheduler.patch.image.pullPolicyCertgen image pull policyIfNotPresent
scheduler.patch.imageNew.registryNew certgen image registry"docker.io"
scheduler.patch.imageNew.repositoryNew certgen image repository"liangjw/kube-webhook-certgen"
scheduler.patch.imageNew.tagNew certgen image tag"v1.1.1"

Scheduler Service Configuration

ParameterDescriptionDefault Value
scheduler.service.typeService typeNodePort
scheduler.service.httpPortHTTP port443
scheduler.service.schedulerPortScheduler NodePort31998
scheduler.service.monitorPortMonitor port31993
scheduler.service.monitorTargetPortMonitor target port9395

Device Plugin Configuration

ParameterDescriptionDefault Value
devicePlugin.image.registryDevice plugin image registry"docker.io"
devicePlugin.image.repositoryDevice plugin image repository"projecthami/hami"
devicePlugin.image.tagDevice plugin image tag""
devicePlugin.image.pullPolicyDevice plugin image pull policyIfNotPresent
devicePlugin.image.pullSecretsDevice plugin image pull secrets[]

Monitor Configuration

ParameterDescriptionDefault Value
devicePlugin.monitor.image.registryMonitor image registry"docker.io"
devicePlugin.monitor.image.repositoryMonitor image repository"projecthami/hami"
devicePlugin.monitor.image.tagMonitor image tag""
devicePlugin.monitor.image.pullPolicyMonitor image pull policyIfNotPresent
devicePlugin.monitor.image.pullSecretsMonitor image pull secrets[]
devicePlugin.monitor.ctrPathContainer path/usr/local/vgpu/containers
devicePlugin.monitor.extraArgsMonitor extra arguments["-v=4"]
devicePlugin.monitor.extraEnvsMonitor extra environments{}

Device Plugin Other Configuration

ParameterDescriptionDefault Value
devicePlugin.deviceSplitCountInteger type, default value: 10. Maximum number of tasks assigned to a single GPU device10
devicePlugin.deviceMemoryScalingDevice memory scaling ratio1
devicePlugin.deviceCoreScalingDevice core scaling ratio1
devicePlugin.runtimeClassNameRuntime class name""
devicePlugin.createRuntimeClassWhether to create runtime classfalse
devicePlugin.migStrategyString type, "none" means ignore MIG functionality, "mixed" means allocate MIG devices through independent resources"none"
devicePlugin.disablecorelimitString type, "true" means disable core limit, "false" means enable core limit"false"
devicePlugin.passDeviceSpecsEnabledWhether to enable passing device specsfalse
devicePlugin.extraArgsDevice plugin extra arguments["-v=4"]
devicePlugin.nodeConfiguration.configNode configuration for device plugin by jsonAn example of default configuration.
devicePlugin.nodeConfiguration.externalConfigNameNode configuration for device plugin by external congimap""
devicePlugin.extraEnvsDevice plugin extra environments{}

Device Plugin Service Configuration

ParameterDescriptionDefault Value
devicePlugin.service.typeService typeNodePort
devicePlugin.service.httpPortHTTP port31992

Device Plugin Deployment Configuration

ParameterDescriptionDefault Value
devicePlugin.pluginPathPlugin path/var/lib/kubelet/device-plugins
devicePlugin.libPathLibrary path/usr/local/vgpu
devicePlugin.nvidiaNodeSelectorNVIDIA node selector{"gpu": "on"}
devicePlugin.updateStrategy.typeUpdate strategy typeRollingUpdate
devicePlugin.updateStrategy.rollingUpdate.maxUnavailableMaximum unavailable count1

Device Configuration

AWS Neuron

ParameterDescriptionDefault Value
devices.awsneuron.customresourcesCustom resources["aws.amazon.com/neuron", "aws.amazon.com/neuroncore"]

Kunlunxin

ParameterDescriptionDefault Value
devices.kunlun.enabledWhether to enabletrue
devices.kunlun.customresourcesCustom resources["kunlunxin.com/xpu"]

Mthreads

ParameterDescriptionDefault Value
devices.mthreads.enabledWhether to enabletrue
devices.mthreads.customresourcesCustom resources["mthreads.com/vgpu"]

NVIDIA

ParameterDescriptionDefault Value
devices.nvidia.gpuCorePolicyGPU core policydefault
devices.nvidia.libCudaLogLevelCUDA library log level1

Huawei Ascend

ParameterDescriptionDefault Value
devices.ascend.enabledWhether to enablefalse
devices.ascend.imageImage""
devices.ascend.imagePullPolicyImage pull policyIfNotPresent
devices.ascend.extraArgsExtra arguments[]
devices.ascend.nodeSelectorNode selector{"ascend": "on"}
devices.ascend.tolerationsTolerations[]
devices.ascend.customresourcesCustom resources["huawei.com/Ascend910A", "huawei.com/Ascend910A-memory", ...]

Iluvatar

ParameterDescriptionDefault Value
devices.iluvatar.enabledWhether to enablefalse
devices.iluvatar.customresourcesCustom resources["iluvatar.ai/BI-V150-vgpu", "iluvatar.ai/BI-V150.vMem","iluvatar.ai/BI-V150.vCore", ...]