Upgrade from Open Source Version

This guide describes how to upgrade a Helm-managed HAMi open source installation to HAMi Enterprise. Collect license information from the existing cluster, request and import the license, then perform the Helm upgrade. The upgrade keeps the existing Helm release and namespace, with particular attention to resource names, scheduler ports, and device plugin configuration.

This guide applies to NVIDIA GPU environments. If HAMi was installed using another method, you also need to migrate namespaces, or you need to enable other device types, contact Dynamia technical support to confirm the migration plan first.

Impact on workloads

The upgrade has two stages:

• Upgrade HAMi components: update the scheduler and device plugin, then check existing workload Pods.
• Update the runtime library used by workload Pods: recreate Pods during an agreed maintenance window so that new Pods use the Enterprise version of libvgpu.so.

Before upgrading, assess how scheduler and device plugin updates may affect newly created Pods, and schedule a maintenance window for recreating workload Pods. After upgrading, check existing workloads and verify GPU scheduling and runtime library versions in new Pods. Complete both checks and workload acceptance before closing the upgrade.

Prerequisites

Before you begin, confirm the following:

• The current HAMi installation is managed by Helm, and you have recorded its release name, namespace, Chart version, and application version.
• You have obtained the target HAMi Enterprise Chart package and corresponding images, and GPU nodes can pull those images.
• You have confirmed the upgrade path from the source version to the target version and the license scope with Dynamia technical support, and allowed time for license issuance.
• You have backed up the current configuration and retained the original Chart package and images for rollback.
• You have recorded workload Pod status, restart counts, and GPU workload validation results.

The commands in this guide use the following variables. Replace the example values with those for your environment. Keep the existing release and namespace during the upgrade.

RELEASE=hami
NAMESPACE=kube-system
CHART_PACKAGE=./hami-enterprise-<chart-version>.tgz
VALUES_FILE=./enterprise-values.yaml

Collect information and request a license

Request the license in advance using the existing HAMi open source cluster. You do not need to deploy Enterprise before collecting this information. The administration machine must have kubectl and jq installed, access to the target cluster, and permission to read the kube-system Namespace and Node information. Confirm that the GPUs to be licensed have been registered by the HAMi device plugin.

Obtain collect-hami-license-info.sh from the offline delivery package, or download the script on a machine with internet access and transfer it to the target environment. Run the following command from the directory containing the script to generate the license request file.

bash collect-hami-license-info.sh > hami-license-info.json

Check the GPU count, UUIDs, and models in deviceInstances in hami-license-info.json against all NVIDIA GPUs that require a license for this upgrade. If the list is empty or devices are missing, check device plugin registration and node annotations before collecting the information again.

The script also outputs deviceModels, which records devices that support only model-based licensing and their physical card counts. This field is empty in a cluster with only NVIDIA devices.

Send the verified hami-license-info.json to Dynamia presales or technical support to request a license. After receiving the license file, confirm that the licensed cluster, device scope, and validity period meet the upgrade requirements before importing it. See the License Application and Activation Guide for the application process.

Import the license before upgrading

Before running helm upgrade, import the license into the existing release's namespace. This guide uses $NAMESPACE for that namespace. Do not switch to the example namespace hami-system used in the license guide.

Set LICENSE_FILE to the actual file path. The following commands create or update the hami-license Secret and add the label that Enterprise uses to identify licenses:

LICENSE_FILE=./hami-license.lic

kubectl create secret generic hami-license \
  -n "$NAMESPACE" \
  --from-file=license="$LICENSE_FILE" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl label secret hami-license \
  -n "$NAMESPACE" \
  hami.io/license=true --overwrite

Check the license data entry and label in the Secret:

kubectl describe secret hami-license -n "$NAMESPACE"

Confirm that Labels includes hami.io/license=true and that Data contains a license entry with a byte count greater than 0. At this point, the license has been staged. HAMi open source does not emit the Enterprise LicenseValid event; check that event after the Enterprise components start. Do not upgrade until you have received the license and successfully imported the Secret, so that waiting for license issuance or import does not extend workload recovery time.

Record versions and back up configuration

helm version
kubectl version
helm list -A
helm history "$RELEASE" -n "$NAMESPACE"
helm get values "$RELEASE" -n "$NAMESPACE" -o yaml > hami-user-values.yaml
helm get values "$RELEASE" -n "$NAMESPACE" --all -o yaml > hami-computed-values.yaml
helm get manifest "$RELEASE" -n "$NAMESPACE" > hami-manifest-before.yaml
kubectl get pods,deploy,ds,svc -n "$NAMESPACE" -o wide

Record the successful revision before the upgrade from helm history. The hami-user-values.yaml file contains user-supplied configuration; hami-computed-values.yaml lets you inspect the complete configuration, including defaults. These backups may contain sensitive configuration and should be stored securely.

Record workload status before upgrading

Select representative GPU workload Pods and record their nodes, Pod UIDs, restart counts, and service responses. For containers using the HAMi runtime library, record its checksum:

kubectl get pod <pod-name> -n <workload-namespace> -o wide
kubectl get pod <pod-name> -n <workload-namespace> \
  -o jsonpath='{.metadata.uid}{"\n"}{.status.containerStatuses[*].restartCount}{"\n"}'
kubectl exec -n <workload-namespace> <pod-name> -c <container-name> -- \
  sha256sum /usr/local/vgpu/libvgpu.so

Use the actual mounted path of the runtime library. The container must have sha256sum available. Also run the workload's health checks or an inference request, and save the results for comparison after the upgrade.

Prepare the Enterprise configuration

Inspect the target Chart metadata and default values:

helm show chart "$CHART_PACKAGE"
helm show values "$CHART_PACKAGE" > enterprise-default-values.yaml

Create enterprise-values.yaml using the Enterprise Chart's configuration structure, migrating each custom setting from the open source installation. Do not use the complete open source values file directly as the Enterprise configuration, or use --reuse-values as a substitute for reviewing the configuration.

Settings to preserve and verify

Setting	Verification
Release name and namespace	Keep the existing values to avoid creating a separate deployment.
fullnameOverride	Check the rendered target Chart to ensure that existing resource names remain unchanged. The example value is hami; adjust it to match existing resource names and verify the rendered manifests.
scheduler.service.httpTargetPort	Check the current port and the listening configuration of both the old and new schedulers to avoid routing requests to the wrong port during the rolling update. Check the example value of 443 against the current configuration.
scheduler.kubeScheduler.image	Choose a kube-scheduler image compatible with the current Kubernetes version.
devicePlugin.runtimeClassName	Keep it consistent with the current NVIDIA container runtime configuration.
devicePlugin.nvidiaDriverRoot	Keep it consistent with the current GPU driver mount path.
devicePlugin.gpuOperatorToolkitReady.enabled	Verify it against the existing GPU Operator / Toolkit deployment.
devicePlugin.nvidiaNodeSelector	Preserve the actual GPU node selection criteria to avoid changing where the device plugin is deployed.
Image addresses and pull credentials	Verify the Enterprise image version, registry address, and required imagePullSecrets.

The following YAML provides an example upgrade configuration. Replace <enterprise-image-tag> and <kube-scheduler-tag> with the actual versions, and verify all other values against the existing environment.

fullnameOverride: hami

global:
  imageTag: "<enterprise-image-tag>"
  # For a registry that requires authentication, specify an existing Secret.
  # imagePullSecrets:
  #   - ghcr-pull-secret

devicePlugin:
  runtimeClassName: nvidia
  nvidiaDriverRoot: /
  gpuOperatorToolkitReady:
    enabled: true
  nvidiaNodeSelector:
    gpu: "on"

scheduler:
  kubeScheduler:
    image:
      registry: registry.k8s.io
      repository: kube-scheduler
      tag: "<kube-scheduler-tag>"
  service:
    httpTargetPort: 443

dra:
  enabled: false
devices:
  ascend:
    enabled: false
platform:
  openshift: false
selinux:
  enabled: false

This example applies to deployments without DRA, Ascend device support, OpenShift, or SELinux integration, so these settings are false. If your environment depends on any of these capabilities, preserve the required settings and confirm the appropriate upgrade plan first.

Review the rendered manifests

helm template "$RELEASE" "$CHART_PACKAGE" \
  -n "$NAMESPACE" \
  -f "$VALUES_FILE" \
  --is-upgrade > hami-manifest-after.yaml

diff -u hami-manifest-before.yaml hami-manifest-after.yaml

The diff command returns a nonzero exit code when it finds differences; this is expected. Check the following:

• Whether the names of existing Deployments, DaemonSets, Services, ConfigMaps, RBAC resources, or Webhooks have changed.
• Whether Service selectors, ports, and targetPort values match both old and new Pods during the rolling update.
• Whether the device plugin's node selection criteria, RuntimeClass, and driver paths are correct.
• Whether image addresses and versions match the delivery requirements.
• Whether added or removed resources are consistent with the upgrade plan.

Local rendering is only a manifest review and does not replace upgrade validation in the cluster. If you find unexpected resource renames, port mismatches, or deletions, correct the configuration before upgrading.

Perform the upgrade

Before upgrading, confirm that you have received the license and imported the hami-license Secret into $NAMESPACE, and that the data entry and label checks above pass. Perform the upgrade during the agreed maintenance window using the same release and namespace. The following command is for Helm 3:

helm upgrade "$RELEASE" "$CHART_PACKAGE" \
  -n "$NAMESPACE" \
  -f "$VALUES_FILE" \
  --atomic \
  --wait \
  --timeout 10m

For Helm 4, replace --atomic with --rollback-on-failure. Both options roll back a failed upgrade. The 10m timeout is an example; adjust it for the time needed to pull images and start components.

After the upgrade, check the release and component status:

helm status "$RELEASE" -n "$NAMESPACE"
helm history "$RELEASE" -n "$NAMESPACE"
kubectl get pods,deploy,ds,svc -n "$NAMESPACE" -o wide

Confirm that the HAMi scheduler and device plugin are using the target images and have completed their updates. If components are not ready, inspect their Pod events and logs before recreating workload Pods.

Verify the license after upgrading

After the Enterprise components start, check the validation events for the license imported before the upgrade:

kubectl get events -n "$NAMESPACE" \
  --field-selector involvedObject.name=hami-license

Confirm that a LicenseValid event appears and that the licensed product, scope, and expiration date are correct. The existence of a Secret alone does not prove that the license is valid. Resolve any license validation failures before validating new workload Pods.

Validate workloads and update the runtime library

Check existing workload Pods

Use the commands in "Record workload status before upgrading" to compare Pod UIDs, restart counts, service responses, and libvgpu.so checksums.

After checking existing workload Pods, validate recreated Pods. Check their scheduling results and runtime library versions to confirm that they use the target Enterprise runtime library.

Recreate workload Pods in stages

First select a workload that can be recreated during the maintenance window, and recreate its Pods using the appropriate controller and workload release process. For a Deployment, you can run:

kubectl rollout restart deployment/<deployment-name> -n <workload-namespace>
kubectl rollout status deployment/<deployment-name> -n <workload-namespace> --timeout=10m

Once the new Pods are ready, check their runtime library checksums against the target Enterprise deliverables. A changed checksum alone does not prove that the version is correct; confirm that the actual runtime library matches the target version.

Run an actual GPU task or inference request to validate resource requests, scheduling, and service responses. After validation passes, gradually recreate other workloads according to the workload release plan.

Acceptance checklist

• The Helm release is deployed, and HAMi components have been updated to the target version.
• License validation succeeds, with the expected scope and expiration date.
• Workload Pods retained during the upgrade have the expected status and service responses.
• New workload Pods can be scheduled with GPU resources and use the target Enterprise runtime library.
• Workload health checks and actual request validation pass.

Handle upgrade failures

If the upgrade command fails, check helm history and component status to determine whether automatic rollback succeeded. Do not assume that the environment has recovered just because the command has exited.

If a manual rollback is necessary, confirm the successful revision before the upgrade and assess the workload impact, then run:

helm rollback "$RELEASE" <previous-revision> \
  -n "$NAMESPACE" \
  --wait \
  --timeout 10m

After Helm rolls back, check component status and service responses again. If workload Pods were recreated after the upgrade, check their runtime library versions separately and arrange Pod recreation according to the rollback plan. Rolling back a Helm release does not mean that the workload runtime libraries have been restored.

If you encounter resource ownership conflicts, unexpected resource recreation, license validation failures, or GPU workload errors, retain the Helm history, values, Pod events, and logs, and contact Dynamia technical support through your customer support channel.