Skip to main content
Enterprise

HAMi Enterprise Air-Gap Deployment Guide

This guide is intended for SREs and platform engineers. It explains how to deploy HAMi Enterprise on Kubernetes with the v0.0.6 All-in-One air-gap package and Zarf v0.86.0, activate licenses, enable accelerator nodes, and verify workloads. Commands use Linux amd64; use the corresponding filenames for arm64. The PPU component is included only in amd64 packages.

This delivery package uses Zarf to perform image import, Helm chart installation, and subsequent upgrades in environments without external network access or with restricted networks, reducing the manual effort of syncing images and maintaining installation order.

Zarf is an application packaging and deployment tool for Kubernetes air-gap / semi-air-gap environments; it can bundle images, Helm charts, scripts, and deployment actions into a portable package.

Installation itself does not require a license. Complete the software deployment first, then apply for and import the license.

Without activation, vGPU partitioning and scheduling are unavailable, and functional verification will fail.

Air-Gap Package Contents​

The outer delivery package is named as follows:

hami-enterprise-v<VERSION>-airgap-<ARCH>.tar.gz
hami-enterprise-v<VERSION>-airgap-<ARCH>.tar.gz.sha256
hami-enterprise-slim-v<VERSION>-airgap-<ARCH>.tar.gz
hami-enterprise-slim-v<VERSION>-airgap-<ARCH>.tar.gz.sha256

The full hami-enterprise-v0.0.6-airgap-amd64.tar.gz contains the files below. Slim retains NVIDIA GPU Operator but excludes NVIDIA driver images and sample workloads. When using Slim, the GPU nodes must already have a compatible NVIDIA driver.

FilePurpose
zarf-linux-amd64Linux amd64 Zarf v0.86.0 CLI
zarf-init-amd64-v0.86.0.tar.zstZarf init air-gap package
hami-enterprise-v0.0.6-airgap-amd64.tar.zstHAMi Enterprise main deployment package
zarf-package-hami-example-gpu-burn-amd64-v0.0.2.tar.zstGPU Burn sample; full package only
zarf-package-hami-example-vllm-qwen-amd64-v0.0.4.tar.zstvLLM + Qwen sample; full package only
collect-hami-license-info.shLicense request information collection script
collect-cluster-info.shKubernetes and GPU inventory collector
COLLECT-CLUSTER-INFO.mdCollector prerequisites and commands
package-values.yamlPackage values template grouped by component; each Chart configuration path defaults to an empty object
PACKAGE-VALUES.mdPackage values format and migration guide
hami/README.mdHAMi Chart values examples and configuration reference

Prerequisite Checks​

  • Use Kubernetes 1.27 or later with an available default StorageClass. Selecting rpm requires Kubernetes 1.30 or later. The administration host needs kubectl and the bundled Zarf v0.86.0; inspect existing releases with zarf tools helm list -A.

For NVIDIA nodes, prepare a compatible driver and runtime configuration. The full package can deploy the optional GPU Operator with driver images. Slim also retains GPU Operator, but its NVIDIA driver images are omitted; install a compatible driver on the nodes before choosing Slim.

  • Prepare the vendor driver, firmware and runtime on Ascend nodes. The bundle provides optional ascend-device-plugin and npu-exporter components; they do not install node prerequisites.

If using GPU Operator alongside HAMi, disable the GPU Operator device plugin (gpu-operator.devicePlugin.enabled=false) and CDI (gpu-operator.cdi.enabled=false). The delivered package already uses these settings. HAMi Enterprise uses scheduler.useDownward, which is incompatible with CDI. Assess running GPU workloads before changing an existing Operator installation.

kubectl cluster-info
kubectl version
kubectl get nodes -o wide
kubectl get storageclass

Extract, Verify, and Install Zarf​

v0.0.6 is available from OSS and has not yet been uploaded to R2. Use the OSS download links provided with your delivery, together with the checksum file for the same product, version, and architecture. Replace <URL> and <SHA256_URL> below with those links.

# Download the delivery package and checksum file
curl -L -O <URL>
curl -L -O <SHA256_URL>

# Verify integrity
shasum -a 256 -c hami-enterprise-v0.0.6-airgap-amd64.tar.gz.sha256

# Extract the outer tar.gz
tar -xzf hami-enterprise-v0.0.6-airgap-amd64.tar.gz

# Enter the extracted directory
cd hami-enterprise-v0.0.6-airgap-amd64

The package includes the Linux Zarf v0.86.0 CLI and its matching init package. The example below uses amd64; for arm64, use zarf-linux-arm64 and the arm64 init package. Install the bundled CLI after entering the extracted directory:

chmod +x ./zarf-linux-amd64
sudo install -m 0755 ./zarf-linux-amd64 /usr/local/bin/zarf
zarf version

Zarf includes the Helm tool. For subsequent troubleshooting of Helm release, values, and chart status, use zarf tools helm to avoid relying on a separately installed Helm in the target environment:

zarf tools helm version
zarf tools helm list -A

Initialize Zarf​

Run zarf init only if the target cluster has not been initialized. Use the labeled policy to limit image rewriting to explicitly labeled namespaces or workloads. For an existing Zarf installation, inspect its Registry, StorageClass, and Agent mutation policy first; deploying the main package does not change that policy.

Use the Zarf built-in registry:

zarf init zarf-init-amd64-v0.86.0.tar.zst \
  --agent-mutation-policy=labeled \
  --confirm

Use an external registry:

zarf init zarf-init-amd64-v0.86.0.tar.zst \
  --agent-mutation-policy=labeled \
  --registry-url=harbor.example.com/zarf-amd64 \
  --registry-push-username=<username> \
  --registry-push-password=<password> \
  --confirm

💡 Mixed-architecture clusters must use separate registry prefixes. AMD64 and ARM64 clusters may share the same Harbor instance, but they must not share the same Zarf registry project or repository prefix. Images in a Zarf air-gap package are packaged for the target architecture. Deploying AMD64 and ARM64 packages sequentially to the same image path does not merge identical tags into a multi-platform manifest. A later push may replace the earlier tag and cause Pods on the other architecture to pull an incompatible image after a restart or rescheduling.

During cluster initialization, use an architecture-specific --registry-url for each architecture. For example, use registry.example.com/zarf-amd64 for AMD64 and registry.example.com/zarf-arm64 for ARM64. Create the corresponding Harbor projects or repository prefixes in advance, and ensure that the account used by Zarf has push and pull permissions.

External registry parameter descriptions:

ParameterDescription
--registry-urlExternal image registry address
--registry-push-usernameUsername used to push images
--registry-push-passwordPassword used to push images

After initialization, zarf package deploy imports the images bundled in the package and uses an admission webhook to rewrite image references for managed workloads to the Zarf registry. With the labeled policy, only resources labeled zarf.dev/agent: mutate, or resources in a namespace with that label, are processed; a label on the resource takes precedence over the namespace label. Before deployment, confirm that the namespaces for the selected components have this label. For offline workloads created separately, confirm that the namespace is labeled first. Label changes do not affect existing Pods; recreate the Pods to trigger rewriting.

Use --insecure-skip-tls-verify for the relevant command only when the TLS certificate of the target registry genuinely cannot be verified and the man-in-the-middle risk has been assessed. In production, fix the certificate chain or configure a trusted CA for the Zarf Agent instead of treating skipped TLS verification as a default.

Deploy HAMi Enterprise​

Workload components are disabled by default. Select the components to install with --components; passing --confirm alone does not install workload components. Providing values does not select a component.

The component list is as follows:

tools overwrites /usr/local/bin/jq and /usr/local/bin/nerdctl on the administration host. Removing Zarf does not restore these files. Record any existing files and how to restore them before selecting this component.

Component NameDescriptionRequiredRecommended installation
toolsInstalls jq and nerdctl. collect-hami-license-info.sh requires jq; without it, the script cannot collect the information required to issue a license.NoRecommended
hamiHAMi Enterprise Chart 2.10.0-r4; namespace hami-systemNoYes
rpmResource pool management: CRD Chart 0.4.2, manager Chart 0.0.6, manager image v0.0.2NoWhen resource pool management is needed and its prerequisites are met
ascend-device-pluginAscend Device Plugin Chart 0.2.2; image v1.4.2; uses the HAMi device ConfigMapNoAs needed for Ascend nodes
ppu-device-pluginPPU device plugin and monitor: Chart 0.0.5, image v1.0.6; amd64 onlyNoAs needed for PPU nodes
npu-exporterAscend NPU metrics exporter; see the shared configuration and monitoring instructions below.NoWhen NPU metrics are needed
prometheus-crdsPrometheus Operator CRDs, installed with Server-Side ApplyNoBefore prometheus
prometheuskube-prometheus-stack Helm chartNoAs needed
gpu-operatorNVIDIA GPU Operator; its device plugin is disabled in bundled valuesNoOptional for NVIDIA nodes in both full and Slim packages; Slim requires an existing compatible driver

Zarf v0.86.0 accepts --values=package-values.yaml. Starting with main package v0.0.5, each Chart configuration belongs under its component key; rpm.crds and rpm.manager configure the two RPM Charts separately. Omit --values when no overrides are needed. Values merge in this order: Chart defaults → bundled valuesFiles → the component’s package values. Configuration does not install a component; select it with --components.

Prepare Custom Values​

Start with the outer archive’s package-values.yaml template and read PACKAGE-VALUES.md. Component configuration paths are independent. The tools, Prometheus CRD, and Envoy Gateway CRD components have no Chart values mapping.

Package values keyChartAvailable in
hamiHAMi EnterpriseEnterprise
rpm.crds / rpm.managerHAMi Enterprise CRDs / Resource Pool ManagerEnterprise
ascend-device-pluginAscend Device PluginEnterprise
ppu-device-pluginPPU Device PluginEnterprise, amd64 only
npu-exporterNPU ExporterEnterprise
prometheuskube-prometheus-stackEnterprise
gpu-operatorNVIDIA GPU OperatorFull and Slim Enterprise packages

The hami/ examples use native Chart values. Do not pass them directly as package values. Starting with v0.0.5, old root-level keys such as scheduler, nodeSelector, and global are not forwarded to any Chart. Move them under the matching component key. If a previous file served several Charts, split its fields between those components. Common HAMi fields are listed below with their package values paths:

ParameterDescriptionDefault
hami.dra.enabledWhether to enable and deploy DRAfalse
hami.scheduler.leaderElectWhether to enable leader election for hami-scheduler across multiple nodes. Strongly recommended to disable for single-node clusters.true
hami.scheduler.replicasAdjust the number of hami-scheduler instances1
hami.scheduler.kubeScheduler.image.registryImage registry for the kube-scheduler image used by hami-schedulerregistry.cn-hangzhou.aliyuncs.com
hami.scheduler.kubeScheduler.image.repositoryImage repository name for the kube-scheduler image used by hami-schedulergoogle_containers/kube-scheduler
hami.scheduler.kubeScheduler.image.tagVersion of the kube-scheduler image used by hami-scheduler; should match the target cluster""

Minimal package-values.yaml example. For a single-node cluster, set hami.scheduler.leaderElect to false:

hami:
  dra:
    enabled: false
  scheduler:
    leaderElect: false

For non-NVIDIA devices, enable the corresponding vendor under hami.devices. For Ascend, set both switches:

hami:
  devices:
    ascend:
      enabled: true
      hamiVnpuCore: true

The kube-scheduler image selected by HAMi must be included in the delivered package and imported into the registry chosen at zarf init. Zarf-managed workloads in hami-system have their image references rewritten by the Agent. Setting a different registry in values does not add a missing image to the package. If the target Kubernetes version requires another image, contact technical support for a matching package.

This bundle includes only kube-scheduler:v1.37.0. HAMi selects a scheduler image tag for the target Kubernetes version by default. If the target cluster requires another version, obtain a delivery package containing the matching image and verify that the target offline registry can serve it. Changing only the image tag does not add an image to the package. Override the registry, repository, and tag under hami.scheduler.kubeScheduler.image when needed.

Before deployment, you can inspect the values and rendered manifests for the selected components with the same values file used for deployment. The example below selects HAMi core only.

PACKAGE=./hami-enterprise-v0.0.6-airgap-amd64.tar.zst
COMPONENTS=hami
VALUES_FILE=./package-values.yaml

zarf package inspect values-files "${PACKAGE}" \
  --components="${COMPONENTS}" --values="${VALUES_FILE}"
zarf package inspect manifests "${PACKAGE}" \
  --components="${COMPONENTS}" --values="${VALUES_FILE}" \
  > rendered.yaml

Execute Deployment​

Minimal installation, deploying only the HAMi Enterprise core components:

zarf package deploy hami-enterprise-v0.0.6-airgap-amd64.tar.zst \
  --components=hami \
  --values=package-values.yaml \
  --confirm

If a component takes a long time to complete, first identify whether it is importing images, running Helm hooks, or waiting for workload health checks. Diagnose the issue with zarf tools helm, Pod events, and logs. For incorrect values, fix the corresponding configuration in package-values.yaml, inspect it again, and rerun the same deployment command. If the operation exceeds the default 15 minutes, determine the cause before adjusting --timeout.

After an interrupted deployment, resolve the issue and resume with the same zarf package deploy ... --components=... --values=... command. Zarf will skip re-importing images when the digest has not changed; it will perform a Helm upgrade when Helm charts or values have changed.

If the target resources are already managed by another Helm release and you have confirmed that they should be transferred to the current Zarf package, review the resource scope and then use --take-ownership. Use --force-conflicts only for Server-Side Apply field ownership conflicts. It overwrites fields managed by other field managers, so use it only after confirming that this deployment may take over those fields; do not use it as a general retry option.

Optional: NVIDIA GPU Operator​

NVIDIA GPU Operator is optional, just like the Ascend components below. If the nodes already have a compatible driver, NVIDIA Container Toolkit and runtime configuration, you may skip it. See the HAMi NVIDIA GPU node prerequisites for the more detailed GPU Operator values. Do not enable CDI: this HAMi Enterprise package uses scheduler.useDownward. The bundled GPU Operator has CDI and its own device plugin disabled. Slim can deploy GPU Operator, but the GPU nodes must already have compatible drivers.

zarf package deploy hami-enterprise-v0.0.6-airgap-amd64.tar.zst \
  --components=gpu-operator \
  --values=package-values.yaml \
  --confirm

Enable GPU Nodes​

The HAMi NVIDIA device plugin starts on nodes labeled gpu=on. Apply this label to NVIDIA nodes that HAMi should manage. Ascend components use their own nodeSelector; gpu=on is not their selection rule.

kubectl label nodes <node-name> gpu=on

Optional: Ascend Components​

ascend-device-plugin and npu-exporter are optional and disabled by default in full and Slim packages for amd64 and arm64. This package includes Ascend Device Plugin Chart 0.2.2 and image v1.4.2. The plugin runs in hami-system and uses the existing hami-scheduler-device ConfigMap. The Exporter runs in the npu-exporter namespace. Avoid duplicate installations of existing components. Ascend-only clusters do not need GPU Operator; select it as needed for mixed-device clusters.

Prepare the Nodes

NPU Exporter supports ordinary Ascend compute nodes using containerd. Install the driver, firmware, and runtime on each target node first. Atlas 200I SoC A1 core boards require separate upstream manifests and startup scripts and are outside this component’s scope.

The default selector reuses the plugin’s ascend: "on" node label. No separate Exporter label is needed. The default containerd directory is /run/containerd. Override the Exporter settings only if the actual node labels or runtime path differ.

The installer creates the Exporter log directory and sets ownership to root:root and mode 0750. No manual directory setup on each node is needed. Installation uses the standard Zarf and Helm readiness checks. Verify metric collection and monitoring integration as described under “Verify the Deployment” below.

Configure and Deploy

The default node labels and containerd path need no extra configuration. Keep HAMi settings aligned with the target NPU model and partitioning mode. For a non-default containerd path, configure npu-exporter.hostPaths.containerd with the path used by the target nodes.

To add these components to a cluster where HAMi is already installed, select them as below. For a first installation, add them to the component combination for your edition. Also select prometheus-crds,prometheus when using the bundled monitoring stack. Use the main package for the selected version and architecture.

zarf package deploy <main-package.tar.zst> \
  --components=ascend-device-plugin,npu-exporter \
  --values=package-values.yaml --confirm

Omit --values if no overrides are needed. Select only npu-exporter if the plugin is already installed. Image changes require rebuilding the package with the new image; an offline deployment cannot pull an image that was not packaged.

NPU Exporter Monitoring​

The Exporter does not create a ServiceMonitor by default. Enable the Exporter’s optional ServiceMonitor. When using the bundled stack, install prometheus-crds,prometheus. After deployment, confirm the targets and metrics in Prometheus. Avoid two monitors scraping the same Exporter.

npu-exporter:
  serviceMonitor:
    enabled: true

Verify Ascend Components​

kubectl -n hami-system get daemonsets
kubectl -n npu-exporter get daemonset npu-exporter
kubectl -n npu-exporter rollout status daemonset/npu-exporter --timeout=180s
kubectl -n npu-exporter get pods -o wide
kubectl -n npu-exporter get endpointslice -l kubernetes.io/service-name=npu-exporter

Check the Exporter DaemonSet’s DESIRED and READY counts: both should equal the expected number of Ascend nodes and be greater than zero. Query up{job="npu-exporter",namespace="npu-exporter"} and npu_chip_info_utilization in Prometheus to confirm scraping and metrics for each node. Zero utilization is valid for idle devices. Also verify device-plugin registration and run a workload requesting the expected Ascend resources to confirm allocation and device access.

Exporter Pods being Ready does not prove that metrics are being collected. Prometheus configuration updates and the first scrape take some time after a ServiceMonitor is created. If metrics are missing, check the driver and DCMI. For scrape failures, check target discovery, configuration reload, ServiceMonitor labels, duplicate scraping, and network policies.

Maintenance and Non-default Environments

For Exporter troubleshooting, check that the node has a usable containerd.sock at the configured containerd directory. The component mounts the directory, rather than an individual socket file.

The NetworkPolicy allows TCP 8082 from Pods labeled app.kubernetes.io/name: prometheus across namespaces and denies outbound traffic by default; enforcement depends on the CNI. For other monitoring labels, override npu-exporter.networkPolicy.prometheusPodSelector. Configure standalone ServiceMonitor selector labels through npu-exporter.serviceMonitor.labels.

The Exporter runs as root in a privileged container and accesses host driver libraries, DCMI libraries, and the runtime socket. A read-only socket mount does not restrict runtime API calls. Keep DCMI libraries and their parent directories owned by root and not writable by group or others. Before upgrading the driver, stop workloads and then stop NPU Exporter.

Default host pathPurposeMount access
/usr/local/Ascend/driverDriver librariesRead-only
/usr/local/dcmiDCMI librariesRead-only
/sysDevice informationRead-only
/run/containerdContainerd and CRI socket accessRead-only directory mount
/etc/localtimeNode timezoneRead-only
/var/log/mindx-dl/npu-exporterExporter logsWritable
test -d /usr/local/Ascend/driver
test -d /usr/local/dcmi
test -S /run/containerd/containerd.sock

Removing the component does not remove host logs. The container root filesystem is read-only, /tmp uses emptyDir, and the ServiceAccount token is not mounted. These settings do not remove the privileged container’s host access.

Optional: PPU Components​

ppu-device-plugin is an optional component that is disabled by default. It is available in the amd64 packages for the Enterprise and AI Platform editions, including their Slim variants; arm64 packages do not include this component or its images. This package uses PPU Device Plugin Chart 0.0.5 and plugin v1.0.6 to deploy the device plugin and PPU monitor in hami-system. Avoid installing duplicate components if equivalent components are already present.

Prepare PPU Nodes and Configuration​

Install the PPU driver and runtime appropriate for each node, confirm that the node can access its PPU devices, and then add the node label:

kubectl label nodes <PPU_NODE> alibabappu=on --overwrite

Enable Alibaba PPU in HAMi and keep the plugin and monitoring enabled. Add the following configuration to package-values.yaml:

hami:
  devices:
    alibaba:
      enabled: true
ppu-device-plugin:
  nodeSelector:
    alibabappu: "on"
  plugin:
    enabled: true
  monitor:
    enabled: true
    service:
      enabled: true

The PPU Chart creates a separate ConfigMap by default. Before deployment, check that the resource names and partitioning configuration in ppu-device-plugin.config.data.alibaba match the HAMi scheduler configuration. Do not copy the Ascend ConfigMap configuration directly to PPU. The default resource names include alibaba.com/ppu, alibaba.com/ppu-memory, and alibaba.com/gpu-core. RPM does not support PPU in this version.

Deploy PPU Components​

For an initial deployment, select both hami and ppu-device-plugin. If HAMi is already installed and the Alibaba configuration is enabled, you can select only ppu-device-plugin:

zarf package deploy hami-enterprise-v0.0.6-airgap-amd64.tar.zst \
  --components=hami,ppu-device-plugin \
  --values=package-values.yaml \
  --confirm

Integrate PPU Monitoring​

By default, PPU monitor provides the hami-system/ppu-device-plugin-monitor Service with the labels app=ppu-vgpu-monitor and component=vgpu-monitor. Its metrics port is 31992, which maps to container port 9394. The PPU Chart does not provide a ServiceMonitor switch. To collect metrics, create a matching ServiceMonitor for your existing Prometheus instance. The following example uses release: prometheus, the default label for the Prometheus included in this package. For other monitoring systems, adjust the label to match their selectors.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ppu-vgpu-monitor
  namespace: hami-system
  labels:
    release: prometheus
spec:
  jobLabel: app
  namespaceSelector:
    matchNames:
      - hami-system
  selector:
    matchLabels:
      app: ppu-vgpu-monitor
      component: vgpu-monitor
  endpoints:
    - port: metrics
      path: /metrics
      interval: 15s

Save the example above as ppu-servicemonitor.yaml. Once the Prometheus CRDs are present and the monitoring system can discover ServiceMonitors in hami-system, run kubectl apply -f ppu-servicemonitor.yaml. For VictoriaMetrics, configure a corresponding VMServiceScrape or confirm that the Operator has converted the ServiceMonitor to a VMServiceScrape. Avoid scraping the same Service twice. To install Prometheus from this package, also select prometheus-crds,prometheus.

Verify PPU Components​

kubectl get nodes -l alibabappu=on
kubectl -n hami-system get daemonsets ppu-device-plugin ppu-device-plugin-monitor
kubectl -n hami-system rollout status daemonset/ppu-device-plugin --timeout=180s
kubectl -n hami-system rollout status daemonset/ppu-device-plugin-monitor --timeout=180s
kubectl -n hami-system get pods -o wide
kubectl -n hami-system get endpointslice -l kubernetes.io/service-name=ppu-device-plugin-monitor
kubectl describe node <PPU_NODE>

The DESIRED and READY counts for both DaemonSets should equal the expected number of PPU nodes and be greater than zero. Confirm that the nodes have registered PPU resources, then run a workload that requests PPU resources to verify allocation and device access.

In Prometheus, query up{job="ppu-vgpu-monitor",namespace="hami-system"} and confirm that the scrape target returns 1. Query hami_host_gpu_utilization_ratio{job="ppu-vgpu-monitor"} and hami_host_gpu_memory_used_bytes{job="ppu-vgpu-monitor"} to confirm that metrics from real devices are present. Utilization can be zero when devices are idle. A Ready Pod alone does not prove that metrics collection is working.

If raw metrics are missing, check the node drivers and the plugin and monitor logs. If scraping fails, check the Service labels, ports, EndpointSlices, ServiceMonitor selectors, and network policies. Perform workload acceptance checks after activating the license.

Optional: Resource Pool Management (RPM)​

rpm is not installed by default. It is available in the Enterprise and AI Platform editions, including their Slim packages. This component installs HAMi Enterprise CRD Chart 0.4.2 followed by Resource Pool Manager Chart 0.0.6 in hami-system. The manager image version is v0.0.2.

Before installation, confirm that Kubernetes is version 1.30 or later, HAMi uses extender scheduling mode, and the scheduler device ConfigMap exists in hami-system. RPM currently supports NVIDIA, Ascend, and Hygon; enable at least one device type. This version does not support resource pool management for PPU.

Configure RPM​

rpm:
  crds: {}
  manager:
    hami:
      existingConfigMap: hami-scheduler-device
      schedulerName: hami-scheduler
    devices:
      nvidia:
        enabled: true
      ascend:
        enabled: false
      hygon:
        enabled: false

Enable the device types present in your environment and keep these settings consistent with the HAMi device configuration. If the existing HAMi ConfigMap or scheduler has a different name, override the corresponding setting. RPM reads the device ConfigMap from the cluster during installation or upgrade. After changing the scheduler device configuration, redeploy rpm to update the webhook rules.

Deploy RPM​

Select hami,rpm when deploying HAMi and RPM for the first time. If HAMi is already installed, you can select only rpm.

zarf package deploy hami-enterprise-v0.0.6-airgap-amd64.tar.zst \
  --components=hami,rpm \
  --values=package-values.yaml \
  --confirm

Verify RPM​

zarf tools helm list -n hami-system
kubectl -n hami-system get deployments,pods
kubectl get crd | grep hami.io

After confirming that both Chart releases and the manager Pod are healthy, also verify resource pools, access authorization, and workload admission behavior. Offline rendering cannot read the target cluster’s ConfigMap and cannot replace acceptance checks on an actual installation. By default, the CRD Chart retains its CRDs and custom resources; uninstalling RPM does not delete this data.

Optional: Monitoring Integration​

Monitoring is optional. If the cluster already has compatible Prometheus or VictoriaMetrics monitoring, connect it using the selectors below. To install the bundled kube-prometheus-stack, select both monitoring components:

zarf package deploy hami-enterprise-v0.0.6-airgap-amd64.tar.zst \
  --components=prometheus-crds,prometheus \
  --values=package-values.yaml \
  --confirm
kubectl describe node <node-name>
kubectl api-resources --api-group=monitoring.coreos.com

For Prometheus, match ServiceMonitor.metadata.labels to Prometheus.spec.serviceMonitorSelector and check the namespace selector. For VictoriaMetrics Operator, VMAgent.spec.serviceScrapeSelector selects VMServiceScrape.metadata.labels; also check serviceScrapeNamespaceSelector and any ServiceMonitor conversion.

Confirm that your monitoring system collects the metrics for the selected hardware from HAMi, DCGM Exporter, NPU Exporter, or PPU monitor. The v0.0.6 bundle sets hami.legacyMetrics=false. First verify that the relevant scrape targets report up=1, then query the applicable current metrics. For NVIDIA, the following metrics should return data:

  • DCGM_FI_DEV_GPU_UTIL

  • hami_host_gpu_utilization_ratio

  • hami_gpu_core_allocated_ratio

Offline license collection: after deploying HAMi and the device plugins, follow the collection instructions below. The extracted delivery archive already contains collect-hami-license-info.sh; the download method requires internet access. Ensure that the administration host has kubectl and jq. Installing tools is recommended to provide jq; without it, the script cannot collect the information required to issue a license. Activate the license before validating workloads that require it.

Request a License​

Once the HAMi core components are ready, confirm that the device plugins or vendor components have reported all devices that require a license to the cluster before collecting license request information.

On the administration machine, ensure that kubectl and jq are available, the current kubecontext points to the target cluster, and you have permission to read the kube-system namespace and all Nodes.

Obtain collect-hami-license-info.sh from the offline delivery package, or download the script on a machine with internet access and transfer it to the target environment. Run the following command from the directory containing the script to generate the license request file.

bash collect-hami-license-info.sh > hami-license-info.json

The following example shows a generated hami-license-info.json file containing one NVIDIA A10 and two MLU cards.

{
  "esn": "96565d61-986a-4918-aafb-448ff6e3746b",
  "deviceInstances": [
    {
      "uuid": "GPU-ceee905d-48ac-93de-a81b-17c00e1e5e02",
      "deviceType": "NVIDIA A10"
    }
  ],
  "deviceModels": [
    {
      "deviceType": "MLU",
      "count": 2
    }
  ]
}

  • esn is the UID of the kube-system namespace and identifies the target cluster.

  • deviceInstances lists devices licensed by instance. Each entry contains the device UUID and model.

  • deviceModels lists devices that support only model-based licensing. count is the number of physical cards of that model across all nodes.

For clusters with only instance-licensed devices, deviceModels is empty. For clusters with only model-licensed devices, deviceInstances is empty. At least one device list must be nonempty in the license request.

Check the UUIDs, models, and card counts against the devices that require a license. Keep the deviceType values produced by the script. If devices are missing, check the reporting status of the device plugins or vendor components, along with node annotations and Capacity, then collect the information again.

The script uses the HAMi Chart resource names by default. If you have changed mluResourceCores or kunlunResourceName, set HAMI_MLU_CORE_RESOURCE or HAMI_KUNLUN_COUNT_RESOURCE, respectively, to the corresponding resource name before running the script.

Send the checked hami-license-info.json file to Dynamia presales or technical support to request a license.

Post-Activation Verification​

kubectl -n hami-system get pods
kubectl describe node <gpu-node>
kubectl get events --field-selector involvedObject.name=hami-license -n hami-system
kubectl get nodes -o custom-columns='NODE:.metadata.name,LICENSE:.metadata.annotations.hami\.io/nvidia-license'

A LicenseValid event indicates that license validation succeeded. Confirm that the selected component Pods are Running or Completed and that accelerator resources are registered on the managed nodes.

Sample Workload Verification​

The full amd64 outer package includes independent GPU Burn v0.0.2 and vLLM Qwen v0.0.4 Zarf sample packages; Slim does not include samples. Their versions are independent of main package v0.0.6. Verify the core components and real device allocation before deploying the samples. These two NVIDIA samples create workloads without a separate kubectl apply. Before deployment, confirm that the sample package may clean up resources with the same names left by earlier versions in default. vLLM Ascend is a separately provided arm64 image package and does not deploy workloads automatically.

GPU Burn Verification​

zarf package deploy zarf-package-hami-example-gpu-burn-amd64-v0.0.2.tar.zst --confirm

After deployment, check the Deployment / Pod status:

kubectl -n hami-example get deploy turbo-gpu-burn
kubectl -n hami-example get pods -l app=turbo-gpu-burn
kubectl -n hami-example logs -l app=turbo-gpu-burn --tail=50

This sample creates hami-example/turbo-gpu-burn Deployment; clean it up as needed after verification:

kubectl -n hami-example delete deploy turbo-gpu-burn

vLLM + Qwen Verification​

zarf package deploy zarf-package-hami-example-vllm-qwen-amd64-v0.0.4.tar.zst --confirm

After deployment, check the inference service status:

kubectl -n hami-example get deploy vllm-qwen3
kubectl -n hami-example get pods -l app=vllm-qwen3
kubectl -n hami-example get svc vllm-qwen3-webui

Once the Pod is ready, access Open WebUI at http://<node-ip>:30081/openwebui. Confirm that your firewall and cluster network policy allow port 30081.

# Get cluster node IP (any reachable node is fine)
kubectl get nodes -o wide

# Access via browser
# http://<node-ip>:30081/openwebui

Open WebUI is automatically connected to the vLLM sidecar in the same Pod; open the page to start interacting.

If the Pod remains Pending, first check whether the license is activated, whether GPU nodes are labeled with gpu=on, and whether the node GPU drivers are normal.

Troubleshooting​

For deployment failures, inspect Pods, events, and Zarf package status. Use zarf tools helm to inspect Helm releases. Diagnostic files may contain cluster configuration and logs; review them for sensitive information before sharing and do not include Secrets.

kubectl get pods -A | grep -E 'hami|gpu-operator|prometheus|vllm|gpu-burn'
kubectl get events -A --sort-by=.lastTimestamp | tail -50
zarf package list

Collect Kubernetes and GPU inventory with collect-cluster-info.sh; see COLLECT-CLUSTER-INFO.md. First install tools to provide jq. It does not install business components. Set PACKAGE to your Enterprise main package below; Slim uses hami-enterprise-slim-v0.0.6-airgap-amd64.tar.zst. For arm64, replace amd64 in both CLI and package names. Preserve the target kubeconfig when using sudo.

KUBECONFIG_PATH="${KUBECONFIG:-$HOME/.kube/config}"
COLLECTOR_PATH="$PWD/collect-cluster-info.sh"
ROOT_PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"
PACKAGE=./hami-enterprise-v0.0.6-airgap-amd64.tar.zst

sudo env PATH="$ROOT_PATH" sh -c 'command -v kubectl >/dev/null'
sudo env KUBECONFIG="$KUBECONFIG_PATH" PATH="$ROOT_PATH" \
  ./zarf-linux-amd64 package deploy "$PACKAGE" \
  --components=tools --confirm
sudo env PATH="$ROOT_PATH" sh -c 'command -v jq >/dev/null'
sudo env KUBECONFIG="$KUBECONFIG_PATH" PATH="$ROOT_PATH" \
  "$COLLECTOR_PATH" > cluster-info.json
sudo /usr/local/bin/jq empty cluster-info.json

Common Issues​

SymptomChecks and Resolution
hami-device-plugin is not runningConfirm that the node has the gpu=on label, then check kubectl -n hami-system get pods.
hami-device-plugin repeatedly restartsCheck whether NVIDIA GPU Operator still enables its default device plugin. Set the Chart key devicePlugin.enabled=false; in v0.0.5 package values, use gpu-operator.devicePlugin.enabled=false.
Image pull failsInspect Pod events, the image reference and registry access. For air-gap deployment, confirm the image is included in the package, the Namespace has zarf.dev/agent: mutate, and the Pod image was rewritten to the Zarf registry. Recreate the Pod after changing mutation labels.
HAMi metrics are unavailableCheck whether the Prometheus or VictoriaMetrics selector matches the labels on the monitoring object.
A workload remains PendingCheck license activation, the gpu=on node label, available accelerator resources, and events from kubectl describe pod.

Limitations of Installing Components with Zarf​

Control the Image Rewrite Scope Explicitly with Labels​

We recommend initializing Zarf with zarf init --agent-mutation-policy=labeled. This policy rewrites only resources labeled zarf.dev/agent: mutate, or resources in a namespace with that label. An individual workload that must retain its original image reference can be labeled zarf.dev/agent: ignore, and a resource label takes precedence over the namespace label. This allows image rewriting to be controlled per resource, even within the same cluster or namespace, without having the Agent manage every application namespace by default. Only images included in the Zarf package can be rewritten and pulled in an offline environment. After changing labels, recreate existing Pods because they are not updated automatically.

Manage Each Set of Resources Through a Single Delivery Pipeline​

Zarf manages the chart releases in a package through Helm. Managing the same Kubernetes resources in parallel through another native Helm workflow can still cause release ownership, field ownership, and upgrade ordering conflicts. Continue routine upgrades with a newer Zarf package. If existing resources must be transferred to Zarf, review the release and resource scope before using --take-ownership. Use --force-conflicts only for Server-Side Apply field conflicts that you have confirmed may be overwritten; it should not be used as a routine installation option.

Support

  • Email: [email protected]

  • Sales / technical support: 400-026-7800

  • Customers with active commercial agreements should submit support requests through their dedicated support channel.