[[s-kubernetes-upgrade]] === Upgrading a LINSTOR Deployment on Kubernetes A LINSTOR Deployment on Kubernets can be upgraded to a new release using Helm. Before upgrading to a new release, you should ensure you have an up-to-date backup of the LINSTOR database. If you are using the Etcd database packaged in the LINSTOR Chart, see <> IMPORTANT: Upgrades using the LINSTOR Etcd deployment require etcd to use persistent storage. Only follow these steps if Etcd was deployed using `etcd.persistentVolume.enabled=true` Upgrades will update to new versions of the following components: * LINSTOR operator deployment * LINSTOR Controller * LINSTOR Satellite * LINSTOR CSI Driver * Etcd * Stork Some versions require special steps, please take a look <> The main command to upgrade to a new LINSTOR operator version is: ---- helm repo update helm upgrade linstor-op linstor/linstor ---- If you used any customizations on the initial install, pass the same options to `helm upgrade`. The options currently in use can be retrieved from Helm. ---- # Retrieve the currently set options $ helm get values linstor-op USER-SUPPLIED VALUES: USER-SUPPLIED VALUES: null drbdRepoCred: drbdiocred operator: satelliteSet: kernelModuleInjectionImage: drbd.io/drbd9-rhel8:v9.0.28 storagePools: lvmThinPools: - devicePaths: - /dev/vdb name: thinpool thinVolume: thinpool volumeGroup: "" # Save current options $ helm get values linstor-op > orig.yaml # modify values here as needed. for example selecting a newer DRBD image $ vim orig.yaml # Start the upgrade $ helm upgrade linstor-op linstor/linstor -f orig.yaml ---- This triggers the rollout of new pods. After a short wait, all pods should be running and ready. Check that no errors are listed in the status section of LinstorControllers, LinstorSatelliteSets and LinstorCSIDrivers. IMPORTANT: During the upgrade process, provisioning of volumes and attach/detach operations might not work. Existing volumes and volumes already in use by a pod will continue to work without interruption. [[s-kubernetes-upgrade-k8s-backend]] ==== Upgrading a LINSTOR Cluster with Kubernetes backend Using the Kubernetes API as LINSTOR backend is quite new: there might be situations in which LINSTOR fails to upgrade resources, leaving the LINSTOR cluster in an unusable state. For this reason, we strongly recommend doing a manual backup of all LINSTOR resources before upgrading the cluster. In a future release, this step will be managed by the operator. 1. Stop the current controller: + ---- $ kubectl patch linstorcontroller linstor-cs '{"spec":{"replicas": 0}}' $ kubectl rollout status --watch deployment/linstor-cs-controller ---- 2. The following command will create a file `crds.yaml`, which stores the current state of all LINSTOR Custom Resource Definitions: + ---- $ kubectl get crds | grep -o ".*.internal.linstor.linbit.com" | xargs kubectl get crds -oyaml > crds.yaml ---- + 3. In addition to the definitions, the actual resources must be be backed up as well: + ---- $ kubectl get crds | grep -o ".*.internal.linstor.linbit.com" | xargs -i{} sh -c "kubectl get {} -oyaml > {}.yaml" ---- 4. Run the chart upgrade using `--set IHaveBackedUpAllMyLinstorResources=true` to acknowledge you have executed the above steps. [[s-kubernetes-upgrade-version]] ==== Upgrade instructions for specific versions Some versions require special steps, see below. ===== Upgrade to v1.7 No additional steps necessary. ===== Upgrade to v1.6 This versions introduces a new option to support Kubernetes distributions which use different state directories than the default of `/var/lib/kubelet`. A notable example is microk8s, which uses `/var/snap/microk8s/common/var/lib/kubelet`. To support this, a small addition to the `LinstorCSIDriver` CRD was necessary. As Helm does not upgrade CRDs on chart upgrade, instead please run: ---- $ helm repo update $ helm pull linstor/linstor --untar $ kubectl replace -f linstor/crds/ customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced ---- If you do not apply the new CRDs, you will get errors like the following: ---- Error: UPGRADE FAILED: error validating "": error validating data: ValidationError(LinstorCSIDriver.spec): unknown field "kubeletPath" in com.linbit.linstor.v1.LinstorCSIDriver.spec ---- If you previously used the included snapshot-controller to process `VolumeSnapshot` resources, you should replace it with the new charts provided by the Piraeus project. The <> has been updated to include instructions on how you can add the snapshot-controller to your cluster. ===== Upgrade to v1.5 This version introduces a <> component for DRBD resources. This requires a new image and a replacement of the existing `LinstorSatelliteSet` CRD. Helm does not upgrade the CRDs on a chart upgrade, instead please run: ---- $ helm repo update $ helm pull linstor/linstor --untar $ kubectl replace -f linstor/crds/ customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced ---- If you do not plan to use the provided <> you still need to apply the above steps, otherwise you will get an errors like the following ---- Error: UPGRADE FAILED: error validating "": error validating data: ValidationError(LinstorSatelliteSet.spec): unknown field "monitoringImage" in com.linbit.linstor.v1.LinstorSatelliteSet.spec ---- NOTE: Some Helm versions fail to set the monitoring image even after replacing the CRDs. In that case, the in-cluster LinstorSatelliteSet will show an empty `monitoringImage` value. Edit the resource using `kubectl edit linstorsatellitesets` and set the value to `drbd.io/drbd-reactor:v0.3.0` to enable monitoring. ===== Upgrade to v1.4 This version introduces a new default version for the Etcd image, so take extra care that Etcd is using persistent storage. *Upgrading the Etcd image without persistent storage will corrupt the cluster*. If you are upgrading an existing cluster without making use of new Helm options, no additional steps are necessary. If you plan to use the newly introduced `additionalProperties` and `additionalEnv` settings, you have to replace the installed CustomResourceDefinitions with newer versions. Helm does not upgrade the CRDs on a chart upgrade ---- $ helm pull linstor/linstor --untar $ kubectl replace -f linstor/crds/ customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced ---- ===== Upgrade to v1.3 No additional steps necessary. ===== Upgrade to v1.2 LINSTOR operator v1.2 is supported on Kubernetes 1.17+. If you are using an older Kubernetes distribution, you may need to change the default settings, for example [the CSI provisioner](https://kubernetes-csi.github.io/docs/external-provisioner.html). There is a known issue when updating the CSI components: the pods will not be updated to the newest image and the `errors` section of the LinstorCSIDrivers resource shows an error updating the DaemonSet. In this case, manually delete `deployment/linstor-op-csi-controller` and `daemonset/linstor-op-csi-node`. They will be re-created by the operator. [[s-kubernetes-etcd-backup]] ==== Creating Etcd Backups To create a backup of the Etcd database and store it on your control host, run: [source] ---- kubectl exec linstor-op-etcd-0 -- etcdctl snapshot save /tmp/save.db kubectl cp linstor-op-etcd-0:/tmp/save.db save.db ---- These commands will create a file `save.db` on the machine you are running `kubectl` from.