Upgrade Airgap Palette Installed with Kubernetes

This guide takes you through the process of upgrading a self-hosted airgap Palette instance installed on Kubernetes.

warning

Before upgrading Palette to a new major version, you must first update it to the latest patch version of the latest minor version available. Refer to the Supported Upgrade Paths section for details.

If your setup includes a PCG, you must also allow the PCG to upgrade automatically before each major or minor Palette upgrade.

Prerequisites

• Access to the x86 Linux jumpbox or bastion host with connectivity to the target platform where you have installed Palette.

• An OCI registry such as Harbor or AWS ECR configured and available to store the new Palette images and packs.

• Access to the latest Palette airgap setup binary. Refer to Access Palette for more details.

• kubectl and helm available in your system.

• Access to the kubeconfig file of the target Kubernetes cluster. You must be able to interact with the cluster through kubectl and have sufficient permissions to upgrade Palette. We recommend using a role with the cluster-admin permissions.

• unzip or a similar tool available in your system.

• Access to the latest Palette Helm Chart. Refer to Access Palette for more details.

Upgrade

1. Log in to the Linux environment from which you can access your self-hosted airgap Palette instance.

2. Authenticate with your OCI registry and get credentials to your spectro-packs and spectro-images repositories.

   info

   Be mindful of the auth token timeout. Uploading Palette images and packages to the OCI registry can take about an hour. If your auth token expires before everything is uploaded, you will need to re-authenticate and start the upload again.

   Select one of the following tabs for instructions on how to authenticate with Harbor and AWS ECR.

   Harbor

   Use the following command template to authenticate with your Harbor OCI registry through oras. Refer to the oras login docs for more details about the available CLI flags and examples.

   oras login X.X.X.X --user 'yourUserNameHere' --password 'yourPasswordHere'

   tip

   If your Harbor registry has a self-signed certificate, use the --insecure flag.

   oras login X.X.X.X --insecure --user 'yourUserNameHere' --password 'yourPasswordHere'

   AWS ECR

   To get the authentication commands for your specific registries, navigate to the AWS ECR console and click View push commands. Refer to the AWS ECR Authentication docs for more details.

   Alternatively, you can use the following command templates:

   • Authenticate with your private spectro-packs registry:

     • <ecr-region>: Replace with the region where your ECR repository is hosted.

     • <aws-account-id>: Replace with your AWS account ID.

       aws ecr get-login-password --region <ecr-region> | oras login --username AWS --password-stdin <aws-account-id>.dkr.ecr.<ecr-region>.amazonaws.com

       Consider the following example for reference.

       aws ecr get-login-password --region us-east-1 | oras login --username AWS --password-stdin 123456789.dkr.ecr.us-east-1.amazonaws.com

   • Authenticate with your public spectro-images registry:

     • <ecr-region>: Replace with the region where your ECR repository is hosted.

     • <public-ecr-url>: Replace with the URL to your public ECR registry.

       aws ecr-public get-login-password --region <ecr-region> | docker login --username AWS --password-stdin <public-ecr-url>

       Consider the following example for reference.

       aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/dfa987f

3. Prepare and export the environment variables required by the Palette airgap binary. Select one of the following tabs for instructions on how to prepare these variables for Harbor and AWS ECR.

   Harbor

   • OCI_IMAGE_REGISTRY: The IP address or domain name of the spectro-images registry.

   • OCI_IMAGE_BASE: The namespace or repository that hosts the images.

   • OCI_PACK_REGISTRY: The IP address or domain name of the spectro-packs registry.

   • OCI_PACK_BASE: The namespace or repository that hosts the packages.

     export OCI_IMAGE_REGISTRY=<harbor-endpoint>
     export OCI_IMAGE_BASE=spectro-images
     export OCI_PACK_REGISTRY=<harbor-endpoint>
     export OCI_PACK_BASE=spectro-packs

     Consider the following example for reference.

     export OCI_IMAGE_REGISTRY=example.internal.com
     export OCI_IMAGE_BASE=spectro-images
     export OCI_PACK_REGISTRY=10.10.100.48
     export OCI_PACK_BASE=spectro-packs

   AWS ECR

   • ECR_IMAGE_REGISTRY: The IP address or domain name of the public spectro-images registry.

   • ECR_IMAGE_BASE: The namespace or repository that hosts the images.

   • ECR_IMAGE_REGISTRY_REGION: The AWS region where the ECR registry is located.

   • ECR_PACK_REGISTRY: The IP address or domain name of the public spectro-packs registry.

   • ECR_PACK_BASE: The namespace or repository that hosts the packages.

   • ECR_PACK_REGISTRY_REGION: The AWS region where the ECR registry is located.

     export ECR_IMAGE_REGISTRY=<ecr-endpoint>
     export ECR_IMAGE_BASE=spectro-images
     export ECR_IMAGE_REGISTRY_REGION=<ecr-region>
     export ECR_PACK_REGISTRY=<ecr-endpoint>
     export ECR_PACK_BASE=spectro-packs
     export ECR_PACK_REGISTRY_REGION=<ecr-region>

     Consider the following example for reference.

     export ECR_IMAGE_REGISTRY=public.ecr.aws/1234567890
     export ECR_IMAGE_BASE=spectro-images
     export ECR_IMAGE_REGISTRY_REGION=us-east-1
     export ECR_PACK_REGISTRY=123456789.dkr.ecr.us-east-1.amazonaws.com
     export ECR_PACK_BASE=spectro-packs
     export ECR_PACK_REGISTRY_REGION=us-east-1

4. Download the airgap setup binary using the credentials our support team provided along with the binary version. Use the following command template to set up a VERSION variable you'll use in the following steps.

   export VERSION=<x.y.z>

   Consider the following example for reference.

   export VERSION=4.2.7

5. Use the following command template to download the new Palette airgap installation bin. Enter the username, password, and the Palette airgap installation URL you received from our support team. In the output file name, replace <version> with the Palette version you're downloading.

   curl --user <username>:<password> --output airgap-<version>.bin <url-to-airgap-installation-bin>

   Consider the following command example for reference.

   curl --user <username>:<password> --output airgap-4.2.7.bin https://software.spectrocloud.com/airgap-v4.2.7.bin

6. Use the following command template to execute the new Palette airgap installation bin.

   chmod +x airgap-v$VERSION.bin && ./airgap-v$VERSION.bin

   After the binary execution is done, you should receive the following success message (the example is shortened for brevity). As a result of starting the binary, you should have a spectro-manifests archive in your temporary directory.

   Verifying archive integrity...  100%   MD5 checksums are OK. All good.
   Uncompressing Airgap Setup - Version 4.0.17  100%
   Setting up Packs
   - Pushing Pack cni-calico:3.25.1
   ...
   Setting up Images
   - Pushing image docker.io/kindest/kindnetd:v20230227-15197099
   - Pushing image gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.22.8
   ...
   Preparing Manifests Archive
   Manifests are available in /tmp/spectro-manifests-1696971110.zip. Extract the archive to a file server to serve as a Spectro Cloud Repository
   Setup Completed

7. Move the spectro-manifests archive to a directory that your file server can access and use the following command template to unzip it.

   unzip spectro-manifests-<file-id>.zip -d /target/folder
   warning

   Do not remove or replace the existing files inside your target folder that is served by the file server. The previous content is necessary for the upgrade process.

8. Refer to the Additional Packs page and update the packages you are currently using. You must update each package separately.

   info

   Depending on your underlying infrastructure provider and Kubernetes distribution, you may need to modify the following Palette upgrade steps to match your environment.

9. Navigate to the directory with the Palette installation zip file. Unzip the file to a palette-install directory.

   unzip release-*.zip -d palette-install

10. Navigate to the release directory inside palette-install.

    cd palette-install/charts/release-*

11. In a code editor of your choice, open the extras/cert-manager/values.yaml file and replace the controllerImage, webhookImage, and amceResolverImage image URLs with your OCI image registry URLs.

    image:
    cainjectorImage: "gcr.io/spectro-images-public/release-fips/jetstack/cert-manager-cainjector:spectro-v1.11.0-20230427"
    controllerImage: "<your-oci-registry-url>/spectro-images-public/release-fips/jetstack/cert-manager-controller:spectro-v1.11.0-20230427"
    webhookImage: "<your-oci-registry-url>/spectro-images-public/release-fips/jetstack/cert-manager-webhook:spectro-v1.11.0-20230808"
    amceResolverImage: "<your-oci-registry-url>/spectro-images-public/release-fips/jetstack/cert-manager-acmesolver:spectro-v1.11.0-20230427"
    
    featureGates: "AdditionalCertificateOutputFormats=true"

    Consider the following example for reference.

    image:
    cainjectorImage: "gcr.io/spectro-images-public/release-fips/jetstack/cert-manager-cainjector:spectro-v1.11.0-20230427"
    controllerImage: "harbor.docs.spectro.dev/spectro-images-public/release-fips/jetstack/cert-manager-controller:spectro-v1.11.0-20230427"
    webhookImage: "harbor.docs.spectro.dev/spectro-images-public/release-fips/jetstack/cert-manager-webhook:spectro-v1.11.0-20230808"
    amceResolverImage: "harbor.docs.spectro.dev/spectro-images-public/release-fips/jetstack/cert-manager-acmesolver:spectro-v1.11.0-20230427"
    
    featureGates: "AdditionalCertificateOutputFormats=true"

12. Update the cert-manager chart using the following command.

    helm upgrade --values extras/cert-manager/values.yaml \
    cert-manager extras/cert-manager/cert-manager-*.tgz --install

    You should receive an output similar to the following.

    Release "cert-manager" has been upgraded. Happy Helming!
    NAME: cert-manager
    LAST DEPLOYED: Thu Feb 22 19:42:33 2024
    NAMESPACE: default
    STATUS: deployed
    REVISION: 2
    TEST SUITE: None

13. Prepare the Palette configuration file values.yaml. If you saved values.yaml used during the Palette installation, you can reuse it for the upgrade. Alternatively, follow the Kubernetes Installation Instructions to populate your values.yaml.

    warning

    Ensure that the values.yaml file is ready before proceeding. Specifically, make sure that the ociPackEcrRegistry and ociImageRegistry configurations include the parameters necessary to interact with your spectro-images and spectro-packs repositories.

14. Upgrade the image-swap chart with the following command. Point to the palette/values.yaml file from step twelve.

    helm upgrade --values palette/values.yaml \
    image-swap extras/image-swap/image-swap-*.tgz --install

    You should receive an output similar to the following.

    Release "image-swap" has been upgraded. Happy Helming!
    NAME: image-swap
    LAST DEPLOYED: Thu Feb 22 19:44:13 2024
    NAMESPACE: default
    STATUS: deployed
    REVISION: 2
    TEST SUITE: None

15. Upgrade the reach-system chart with the following command. Point to the palette/values.yaml file from step twelve.

    helm upgrade --values palette/values.yaml \
    reach-system extras/reach-system/reach-system-\*.tgz --install

    You should receive an output similar to the following.

    Release "reach-system" has been upgraded. Happy Helming!
    NAME: reach-system
    LAST DEPLOYED: Thu Feb 22 19:47:10 2024
    NAMESPACE: default
    STATUS: deployed
    REVISION: 2
    TEST SUITE: None

16. Upgrade Palette with the following command.

    helm upgrade --values palette/values.yaml \
    hubble palette/spectro-mgmt-plane-\*.tgz --install

    You should receive an output similar to the following.

    Release "hubble" has been upgraded. Happy Helming!
    NAME: hubble
    LAST DEPLOYED: Thu Feb 22 20:05:24 2024
    NAMESPACE: default
    STATUS: deployed
    REVISION: 2
    TEST SUITE: None

17. Use the following command to track the upgrade process.

    kubectl get pods --all-namespaces --watch
    tip

    For a more user-friendly experience, consider using K9s or a similar tool to track the upgrade.

    The upgrade usually takes up to five minutes. Palette is upgraded when the deployments in the namespaces cp-system, hubble-system, ingress-nginx, jet-system , and ui-system are in the Ready status.

Validate

1. In your terminal, issue the following command.

   helm ls

   You should receive an output with the version and other details of the currently deployed apps. Check the App Version column of cert-manager, image-swap, reach-system, and hubble to verify that they have the expected versions.

   NAME        	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART                   	APP VERSION
   cert-manager	default  	2       	2024-02-22 19:42:33.776829 +0100 CET	deployed	cert-manager-1.11.0     	1.11.0
   image-swap  	default  	2       	2024-02-22 19:44:13.209592 +0100 CET	deployed	image-swap-v1.5.2-spectro-4.1.1	1.5.2
   reach-system	default  	2       	2024-02-22 19:47:10.558061 +0100 CET	deployed	reach-system-4.2.0             	4.2.0
   hubble      	default  	2       	2024-02-22 19:47:24.085305 +0100 CET	deployed	spectro-mgmt-plane-4.2.7	4.2.7

2. Issue the following command to verify that the Palette clusters work as expected.

   kubectl get pods --all-namespaces --output custom-columns="NAMESPACE:metadata.namespace,NAME:metadata.name,STATUS:status.phase" \
   | grep -E '^(cp-system|hubble-system|ingress-nginx|jet-system|ui-system)\s'

   The command should return a list of deployments in the cp-system, hubble-system, ingress-nginx, jet-system, and ui-system namespaces. All deployments should have the status Running.

   cp-system       spectro-cp-ui-689984f88d-54wsw             Running
   hubble-system   auth-85b748cbf4-6drkn                      Running
   hubble-system   auth-85b748cbf4-dwhw2                      Running
   hubble-system   cloud-fb74b8558-lqjq5                      Running
   hubble-system   cloud-fb74b8558-zkfp5                      Running
   hubble-system   configserver-685fcc5b6d-t8f8h              Running
   hubble-system   event-68568f54c7-jzx5t                     Running
   hubble-system   event-68568f54c7-w9rnh                     Running
   hubble-system   foreq-6b689f54fb-vxjts                     Running
   hubble-system   hashboard-897bc9884-pxpvn                  Running
   hubble-system   hashboard-897bc9884-rmn69                  Running
   hubble-system   hutil-6d7c478c96-td8q4                     Running
   hubble-system   hutil-6d7c478c96-zjhk4                     Running
   hubble-system   mgmt-85dbf6bf9c-jbggc                      Running
   hubble-system   mongo-0                                    Running
   hubble-system   mongo-1                                    Running
   hubble-system   mongo-2                                    Running
   hubble-system   msgbroker-6c9b9fbf8b-mcsn5                 Running
   hubble-system   oci-proxy-7789cf9bd8-qcjkl                 Running
   hubble-system   packsync-28205220-bmzcg                    Succeeded
   hubble-system   spectrocluster-6c57f5775d-dcm2q            Running
   hubble-system   spectrocluster-6c57f5775d-gmdt2            Running
   hubble-system   spectrocluster-6c57f5775d-sxks5            Running
   hubble-system   system-686d77b947-8949z                    Running
   hubble-system   system-686d77b947-cgzx6                    Running
   hubble-system   timeseries-7865bc9c56-5q87l                Running
   hubble-system   timeseries-7865bc9c56-scncb                Running
   hubble-system   timeseries-7865bc9c56-sxmgb                Running
   hubble-system   user-5c9f6c6f4b-9dgqz                      Running
   hubble-system   user-5c9f6c6f4b-hxkj6                      Running
   ingress-nginx   ingress-nginx-controller-2txsv             Running
   ingress-nginx   ingress-nginx-controller-55pk2             Running
   ingress-nginx   ingress-nginx-controller-gmps9             Running
   jet-system      jet-6599b9856d-t9mr4                       Running
   ui-system       spectro-ui-76ffdf67fb-rkgx8                Running