Enterprise

This guide explains how to deploy HAMi Enterprise with Helm on a Kubernetes cluster that can access the required Chart and image registries, configure NVIDIA, Ascend, PPU, and resource pool management as needed, and verify monitoring and workloads. It does not cover air-gapped installation.

💡 Installation does not equal activation. The components can run after Helm installation, but GPU virtualization and scheduling require license activation before they can be used.

Installation itself does not require a license. Complete the software deployment first, then apply for and import the license.

Without activation, vGPU partitioning and scheduling are unavailable, and functional verification will fail.

Why Choose Online Installation

Online installation is intended for teams that can operate Kubernetes, Helm, and image registries and need greater configuration flexibility. Component versions, deployment order, and values can be managed according to the organization infrastructure and change process.

For network-isolated environments, follow the HAMi Enterprise Air-Gap Deployment Guide. This guide covers online Helm installation where the required Chart and image registries are accessible.

Deployment Scope and Prerequisites

Component Selection

Component	When to Install	Notes
HAMi Enterprise	Every cluster	Core scheduler and device plugin. Required.
NVIDIA GPU Operator	Optional when NVIDIA nodes need GPU Operator to manage their GPU software stack; skip it if the nodes are already prepared	The default GPU Operator device plugin must not run together with the HAMi device plugin.
Ascend device plugin	Ascend nodes are used	Install after HAMi and reuse the HAMi device configuration ConfigMap.
Prometheus monitoring stack	HAMi or GPU metrics are required	Do not install another stack if the cluster already has a compatible monitoring system.
PPU device plugin and monitor	PPU nodes are used	Chart 0.0.5 and plugin v1.0.6; amd64 only. Prepare the PPU driver and runtime first.
Resource pool management (RPM)	Resource pool allocation and access authorization are needed	Requires Kubernetes ≥ 1.30 and HAMi extender scheduling. Device support depends on the manager version; this version does not support PPU resource pools.

Prerequisite Checks

Use a Kubernetes version supported by the selected Charts: HAMi requires Kubernetes ≥ 1.18; Ascend and PPU require ≥ 1.20; RPM requires ≥ 1.30. Provide an available default StorageClass. The administration host needs kubectl and Helm.
Prepare the NVIDIA driver and NVIDIA Container Toolkit on NVIDIA nodes.
Prepare the vendor driver, firmware, and runtime on Ascend nodes. Prepare the PPU driver and runtime on amd64 PPU nodes.
Disable NVIDIA GPU Operator’s default device plugin when HAMi supplies the NVIDIA device plugin. Set devicePlugin.enabled=false in the GPU Operator Helm values.
Before enabling monitoring integrations that create ServiceMonitor or PrometheusRule resources, confirm that the required monitoring.coreos.com CRDs exist. If they are missing, follow the monitoring stack section below to install a compatible monitoring stack before installing GPU Operator or platform components that create these resources. Reuse an existing compatible Prometheus or VictoriaMetrics setup where available; clusters that do not enable monitoring do not need to install a monitoring stack.

kubectl cluster-info
kubectl version
kubectl get nodes -o wide
kubectl get storageclass

Online Installation

Before running the commands below, confirm that the current kubeconfig context points to the target cluster. Keep the values file for each Chart under version control. Contact Dynamia.ai technical support for access to commercial Charts, mainland China image registries, and production values.

Install HAMi Enterprise

Install the HAMi core first. This command uses the published GHCR Chart 2.10.0-r4; the administration host must be able to access GHCR, and cluster nodes must be able to pull the required images. Create hami-enterprise-values.yaml with the minimum configuration below so that HAMi creates the hami-scheduler-device ConfigMap used by the Ascend and RPM examples. Merge the configuration for the selected devices into this file, preserve all other required values, and retain the complete file for future upgrades.

fullnameOverride: hami

helm install hami \
  oci://ghcr.io/dynamia-ai/charts/hami-enterprise \
  --version 2.10.0-r4 \
  --namespace hami-system \
  --create-namespace \
  -f hami-enterprise-values.yaml

See the HAMi Helm Chart Values Reference for the complete configuration.

For non-NVIDIA devices, enable the corresponding vendor under devices in the HAMi values. Ascend nodes require at least:

devices:
  ascend:
    enabled: true
    hamiVnpuCore: true

Install NVIDIA GPU Operator (NVIDIA Nodes Only)

NVIDIA GPU Operator is optional. If the nodes already have a compatible driver, NVIDIA Container Toolkit, and runtime configuration, you can skip it. See the HAMi NVIDIA GPU node prerequisites for the node setup and GPU Operator values. When installing or reusing GPU Operator, disable its bundled device plugin (devicePlugin.enabled=false) and CDI (cdi.enabled=false). This HAMi Enterprise deployment uses scheduler.useDownward, which is incompatible with CDI. Assess running GPU workloads before changing CDI settings on an existing cluster.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update

helm install --wait --generate-name \
  -n gpu-operator --create-namespace \
  nvidia/gpu-operator \
  --set devicePlugin.enabled=false \
  --set dcgmExporter.serviceMonitor.enabled=true \
  --set cdi.enabled=false \
  --version=v25.10.1

Enable GPU Nodes

The HAMi NVIDIA device plugin starts on nodes labeled gpu=on. Apply this label to NVIDIA nodes that HAMi should manage. Ascend components use their own nodeSelector; gpu=on is not their selection rule.

kubectl label nodes <node-name> gpu=on

Install the Ascend Device Plugin (Ascend Nodes Only)

Install the Ascend device plugin after the HAMi core is ready and devices.ascend.enabled and devices.ascend.hamiVnpuCore are enabled. Avoid installing duplicate device plugins. The plugin reuses the hami-scheduler-device ConfigMap created by HAMi. Save the following configuration as ascend-device-plugin.yaml.

nameOverride: "ascend-device-plugin"
fullnameOverride: "ascend-device-plugin"
image:
  repository: ghcr.io/dynamia-ai/ascend-device-plugin
  tag: "v1.4.2"
config:
  create: false
  existingDeviceConfigMapName: hami-scheduler-device

helm install ascend-device-plugin \
  oci://ghcr.io/dynamia-ai/charts/ascend-device-plugin \
  --version 0.2.2 \
  --namespace hami-system \
  -f ascend-device-plugin.yaml

Install the PPU Device Plugin and Monitor (amd64 PPU Nodes Only)

This section uses PPU Device Plugin Chart 0.0.5 and plugin v1.0.6, which support amd64 only. Install the PPU driver and runtime appropriate for the nodes, and confirm that the nodes can access the PPU devices. Avoid installing duplicate device plugins.

Prepare the Nodes and HAMi Configuration

kubectl label nodes <PPU_NODE> alibabappu=on --overwrite

Merge the following configuration into hami-enterprise-values.yaml, preserving the rest of the file. For a new installation, use the HAMi installation command above. For an existing installation, run the upgrade command below with the complete values file.

devices:
  alibaba:
    enabled: true

helm upgrade hami \
  oci://ghcr.io/dynamia-ai/charts/hami-enterprise \
  --version 2.10.0-r4 \
  --namespace hami-system \
  -f hami-enterprise-values.yaml \
  --wait

Configure and Install the PPU Chart

Save the following content as ppu-device-plugin.yaml. The PPU Chart deploys both the device plugin and the monitor. This example sets the monitoring Service to ClusterIP for collection by an in-cluster monitoring system.

nameOverride: "ppu-device-plugin"
fullnameOverride: "ppu-device-plugin"

image:
  repository: ghcr.io/dynamia-ai/ppu-device-plugin
  tag: "v1.0.6"
  pullPolicy: IfNotPresent

nodeSelector:
  alibabappu: "on"

plugin:
  enabled: true

monitor:
  enabled: true
  service:
    enabled: true
    type: ClusterIP

The PPU Chart creates a separate ConfigMap by default. Before deployment, check that the resource names and partitioning settings under the native Chart value config.data.alibaba are consistent with the HAMi scheduler configuration. Do not copy the Ascend ConfigMap configuration directly. Default resource names include alibaba.com/ppu, alibaba.com/ppu-memory, and alibaba.com/gpu-core.

helm install ppu-device-plugin \
  oci://ghcr.io/dynamia-ai/charts/ppu-vgpu-device-plugin \
  --version 0.0.5 \
  --namespace hami-system \
  -f ppu-device-plugin.yaml \
  --wait

Connect PPU Monitoring

The PPU monitor exposes the hami-system/ppu-device-plugin-monitor Service with labels app=ppu-vgpu-monitor and component=vgpu-monitor. Its metrics Service port is 31992 and targets container port 9394. The PPU Chart has no ServiceMonitor switch. When connecting HAMi Enterprise to an existing Prometheus installation, confirm that the ServiceMonitor CRD exists before creating the object below. HAMi AI Platform users should follow the platform-specific instructions after this example.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ppu-vgpu-monitor
  namespace: hami-system
  labels:
    release: prometheus
spec:
  jobLabel: app
  namespaceSelector:
    matchNames:
      - hami-system
  selector:
    matchLabels:
      app: ppu-vgpu-monitor
      component: vgpu-monitor
  endpoints:
    - port: metrics
      path: /metrics
      interval: 15s

For HAMi Enterprise with an existing Prometheus installation, create the ServiceMonitor shown here. For HAMi AI Platform, use the Kantaloupe Chart PPU monitoring switches in the HAMi AI Platform Online Deployment Guide. These switches also enable the recording rules required by the platform. Do not additionally create the ServiceMonitor shown here, to avoid duplicate scraping. For the manual ServiceMonitor, adjust release: prometheus to match the actual Prometheus serviceMonitorSelector, and confirm that it discovers ServiceMonitors in hami-system. Save the file as ppu-servicemonitor.yaml and run kubectl apply -f ppu-servicemonitor.yaml. VictoriaMetrics users should configure the corresponding VMServiceScrape or confirm that the Operator has converted the ServiceMonitor; avoid scraping the same Service twice.

Verify the PPU Components

kubectl get nodes -l alibabappu=on
kubectl -n hami-system get daemonsets ppu-device-plugin ppu-device-plugin-monitor
kubectl -n hami-system rollout status daemonset/ppu-device-plugin --timeout=180s
kubectl -n hami-system rollout status daemonset/ppu-device-plugin-monitor --timeout=180s
kubectl -n hami-system get endpointslice -l kubernetes.io/service-name=ppu-device-plugin-monitor
kubectl describe node <PPU_NODE>

The DESIRED and READY counts for both DaemonSets should equal the expected number of PPU nodes and be greater than zero. Confirm that the nodes have registered PPU resources, then run a workload that requests those resources to verify allocation and device access.

In Prometheus, query up{job="ppu-vgpu-monitor",namespace="hami-system"} and confirm that each scrape target reports 1. Query hami_host_gpu_utilization_ratio{job="ppu-vgpu-monitor"} and hami_host_gpu_memory_used_bytes{job="ppu-vgpu-monitor"} to confirm that real device metrics are present. Zero utilization can indicate an idle device.

Install Resource Pool Management (RPM, Optional)

RPM requires Kubernetes ≥ 1.30 and HAMi extender scheduling. Install HAMi first and confirm that the hami-system/hami-scheduler-device ConfigMap exists. Enable at least one device type in RPM. The available configuration switches include NVIDIA, Ascend, and Hygon; actual device support depends on the manager image version. This version does not support PPU resource pool management.

kubectl -n hami-system get configmap hami-scheduler-device

Install the Resource Pool CRDs

helm install hami-enterprise-crds \
  oci://ghcr.io/dynamia-ai/charts/hami-enterprise-crds \
  --version 0.4.2 \
  --namespace hami-system \
  --wait

kubectl wait --for=condition=Established --timeout=60s \
  crd/gpudevices.resource.hami.io \
  crd/resourcepools.resource.hami.io \
  crd/resourcepoolbindings.resource.hami.io

Configure and Install Resource Pool Manager

Save the following configuration as rpm-values.yaml. This example enables NVIDIA only. For other devices, confirm support in the selected manager version before changing the devices switches, and ensure that the HAMi device configuration contains the corresponding resources.

image:
  repository: ghcr.io/dynamia-ai/resource-pool-manager
  tag: "v0.0.2"

hami:
  schedulerName: hami-scheduler
  existingConfigMap: hami-scheduler-device

devices:
  nvidia:
    enabled: true
  ascend:
    enabled: false
  hygon:
    enabled: false

webhook:
  tls:
    certgen:
      image:
        repository: registry.k8s.io/ingress-nginx/kube-webhook-certgen
        tag: v1.6.9

During installation, the manager Chart uses Helm lookup to read the scheduler device configuration from the target cluster. The administration host must be able to access that cluster. A local helm template run cannot replace this step.

helm install hami-resource-pool-manager \
  oci://ghcr.io/dynamia-ai/charts/hami-resource-pool-manager \
  --version 0.0.6 \
  --namespace hami-system \
  -f rpm-values.yaml \
  --wait

After changing the scheduler device configuration, upgrade Resource Pool Manager with the complete manager values file to keep admission rules aligned with that configuration. The CRD Chart retains CRDs and their custom resources by default; uninstalling RPM does not delete this data.

Verify RPM

helm list -n hami-system
kubectl -n hami-system rollout status deployment/hami-resource-pool-manager --timeout=180s
kubectl get crd gpudevices.resource.hami.io resourcepools.resource.hami.io resourcepoolbindings.resource.hami.io
kubectl get gpudevices,resourcepools
kubectl get resourcepoolbindings -A

Confirm that both Helm releases are installed successfully, the manager Deployment is ready, and the device inventory is correct. After configuring resource pools and access rules, verify ResourcePool, ResourcePoolBinding, and workload admission behavior. Pod readiness alone does not prove that resource pool authorization is effective.

Install the Monitoring Stack (When Needed)

Monitoring is optional. Install kube-prometheus-stack only if metrics are needed and the cluster does not already have a compatible Prometheus or VictoriaMetrics installation. Otherwise, connect the existing monitoring system as described below.

helm install prometheus \
  oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack \
  --version 72.3.0 \
  --namespace monitoring \
  --create-namespace \
  --set alertmanager.enabled=false \
  --set grafana.enabled=false

Post-Installation Checks

Core Components

kubectl -n hami-system get pods
kubectl -n gpu-operator get pods
kubectl -n monitoring get pods

A Namespace for an optional component may not exist if that component was not installed. Pods for installed components should be Running or Completed.

Nodes and Monitoring

kubectl describe node <node-name>
kubectl api-resources --api-group=monitoring.coreos.com

For Prometheus, match ServiceMonitor.metadata.labels to Prometheus.spec.serviceMonitorSelector and check the namespace selector. For VictoriaMetrics Operator, VMAgent.spec.serviceScrapeSelector selects VMServiceScrape.metadata.labels; also check serviceScrapeNamespaceSelector and any ServiceMonitor conversion.

For PPU, also follow the component section to check raw utilization and memory metrics. Check that the relevant scrape targets report up=1, then query the current NVIDIA metrics and confirm that they return data:

DCGM_FI_DEV_GPU_UTIL
hami_host_gpu_utilization_ratio
hami_gpu_core_allocated_ratio

License Application and Activation

Request a License

Once the HAMi core components are ready, confirm that the device plugins or vendor components have reported all devices that require a license to the cluster before collecting license request information.

On the administration machine, ensure that kubectl and jq are available, the current kubecontext points to the target cluster, and you have permission to read the kube-system namespace and all Nodes.

Obtain collect-hami-license-info.sh from the offline delivery package, or download the script on a machine with internet access and transfer it to the target environment. Run the following command from the directory containing the script to generate the license request file.

bash collect-hami-license-info.sh > hami-license-info.json

The following example shows a generated hami-license-info.json file containing one NVIDIA A10 and two MLU cards.

{
  "esn": "96565d61-986a-4918-aafb-448ff6e3746b",
  "deviceInstances": [
    {
      "uuid": "GPU-ceee905d-48ac-93de-a81b-17c00e1e5e02",
      "deviceType": "NVIDIA A10"
    }
  ],
  "deviceModels": [
    {
      "deviceType": "MLU",
      "count": 2
    }
  ]
}

esn is the UID of the kube-system namespace and identifies the target cluster.
deviceInstances lists devices licensed by instance. Each entry contains the device UUID and model.
deviceModels lists devices that support only model-based licensing. count is the number of physical cards of that model across all nodes.

For clusters with only instance-licensed devices, deviceModels is empty. For clusters with only model-licensed devices, deviceInstances is empty. At least one device list must be nonempty in the license request.

Check the UUIDs, models, and card counts against the devices that require a license. Keep the deviceType values produced by the script. If devices are missing, check the reporting status of the device plugins or vendor components, along with node annotations and Capacity, then collect the information again.

The script uses the HAMi Chart resource names by default. If you have changed mluResourceCores or kunlunResourceName, set HAMI_MLU_CORE_RESOURCE or HAMI_KUNLUN_COUNT_RESOURCE, respectively, to the corresponding resource name before running the script.

Send the checked hami-license-info.json file to Dynamia presales or technical support to request a license.

Activate HAMi Enterprise

After receiving the license file, save it on an administration host that can access the target cluster, then create the license Secret:

kubectl create secret generic hami-license \
  --from-file=license=/path/to/license-file \
  -n hami-system

kubectl label secret hami-license \
  hami.io/license="true" \
  -n hami-system

kubectl get secret hami-license -n hami-system
kubectl get events --field-selector involvedObject.name=hami-license -n hami-system

A LicenseValid event indicates that license validation succeeded. For NVIDIA nodes, you can also inspect license registration:

kubectl get nodes -o custom-columns='NODE:.metadata.name,LICENSE:.metadata.annotations.hami\.io/nvidia-license'

Post-Activation Verification

kubectl -n hami-system get pods
kubectl describe node <gpu-node>
kubectl get events --field-selector involvedObject.name=hami-license -n hami-system
kubectl get nodes -o custom-columns='NODE:.metadata.name,LICENSE:.metadata.annotations.hami\.io/nvidia-license'

A LicenseValid event indicates that license validation succeeded. Confirm that the selected component Pods are Running or Completed and that accelerator resources are registered on the managed nodes.

Online Sample Workload Verification

Before running this example, confirm that the target cluster can pull nvidia/cuda:12.4.0-base-ubuntu22.04. If the cluster uses a private registry, mirror the image first and update the image reference in the example.

kubectl delete pod hami-smoke --ignore-not-found

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: hami-smoke
spec:
  restartPolicy: Never
  containers:
    - name: cuda
      image: nvidia/cuda:12.4.0-base-ubuntu22.04
      command: ["sh", "-c", "nvidia-smi && sleep 30"]
      resources:
        limits:
          nvidia.com/gpu: 1
          nvidia.com/gpumem: 2000
EOF

kubectl wait --for=condition=Ready pod/hami-smoke --timeout=180s
kubectl logs hami-smoke

If the Pod reaches Ready and the logs show GPU information, HAMi has scheduled the GPU and the NVIDIA runtime is available inside the container. Delete the test Pod after verification.

kubectl delete pod hami-smoke

Common Issues

Symptom	Checks and Resolution
`hami-device-plugin` is not running	Confirm that the node has the `gpu=on` label, then check `kubectl -n hami-system get pods`.
`hami-device-plugin` repeatedly restarts	Check whether NVIDIA GPU Operator still enables its default device plugin. Set devicePlugin.enabled=false in its Helm values.
Image pull fails	Inspect Pod events, the image reference and registry access. Verify the node can reach and authenticate to the configured image registry.
HAMi metrics are unavailable	Check whether the Prometheus or VictoriaMetrics selector matches the labels on the monitoring object.
A workload remains `Pending`	Check license activation, the `gpu=on` node label, available accelerator resources, and events from `kubectl describe pod`.

Support

Email: [email protected]
Sales / technical support: 400-026-7800
Customers with active commercial agreements should submit support requests through their dedicated support channel.

Deployment Scope and Prerequisites

Component Selection​

Prerequisite Checks​

Online Installation

Install HAMi Enterprise​

Install NVIDIA GPU Operator (NVIDIA Nodes Only)​

Enable GPU Nodes​

Install the Ascend Device Plugin (Ascend Nodes Only)​

Install the PPU Device Plugin and Monitor (amd64 PPU Nodes Only)​

Prepare the Nodes and HAMi Configuration​

Configure and Install the PPU Chart​

Connect PPU Monitoring​

Verify the PPU Components​

Install Resource Pool Management (RPM, Optional)​

Install the Resource Pool CRDs​

Configure and Install Resource Pool Manager​

Verify RPM​

Install the Monitoring Stack (When Needed)​