Debian Codegraph @

UG9/en/kubernetes-upgrade-guide.adoc

raw
1 [[s-kubernetes-upgrade]]
2 === Upgrading a LINSTOR Deployment on Kubernetes
3
4 A LINSTOR Deployment on Kubernets can be upgraded to a new release using Helm.
5
6 Before upgrading to a new release, you should ensure you have an up-to-date backup of the LINSTOR database.
7 If you are using the Etcd database packaged in the LINSTOR Chart, see <<s-kubernetes-etcd-backup,here>>
8
9 IMPORTANT: Upgrades using the LINSTOR Etcd deployment require etcd to use persistent storage. Only follow these steps if
10 Etcd was deployed using `etcd.persistentVolume.enabled=true`
11
12 Upgrades will update to new versions of the following components:
13
14 * LINSTOR operator deployment
15 * LINSTOR Controller
16 * LINSTOR Satellite
17 * LINSTOR CSI Driver
18 * Etcd
19 * Stork
20
21 Some versions require special steps, please take a look <<s-kubernetes-upgrade-version,here>>
22 The main command to upgrade to a new LINSTOR operator version is:
23
24 ----
25 helm repo update
26 helm upgrade linstor-op linstor/linstor
27 ----
28
29 If you used any customizations on the initial install, pass the same options to `helm upgrade`. The options currently
30 in use can be retrieved from Helm.
31
32 ----
33 # Retrieve the currently set options
34 $ helm get values linstor-op
35 USER-SUPPLIED VALUES:
36 USER-SUPPLIED VALUES: null
37 drbdRepoCred: drbdiocred
38 operator:
39 satelliteSet:
40 kernelModuleInjectionImage: drbd.io/drbd9-rhel8:v9.0.28
41 storagePools:
42 lvmThinPools:
43 - devicePaths:
44 - /dev/vdb
45 name: thinpool
46 thinVolume: thinpool
47 volumeGroup: ""
48 # Save current options
49 $ helm get values linstor-op > orig.yaml
50 # modify values here as needed. for example selecting a newer DRBD image
51 $ vim orig.yaml
52 # Start the upgrade
53 $ helm upgrade linstor-op linstor/linstor -f orig.yaml
54 ----
55
56 This triggers the rollout of new pods. After a short wait, all pods should be running and ready.
57 Check that no errors are listed in the status section of LinstorControllers, LinstorSatelliteSets and LinstorCSIDrivers.
58
59 IMPORTANT: During the upgrade process, provisioning of volumes and attach/detach operations might not work. Existing
60 volumes and volumes already in use by a pod will continue to work without interruption.
61
62 [[s-kubernetes-upgrade-k8s-backend]]
63 ==== Upgrading a LINSTOR Cluster with Kubernetes backend
64
65 Using the Kubernetes API as LINSTOR backend is quite new: there might be situations in which LINSTOR fails to upgrade
66 resources, leaving the LINSTOR cluster in an unusable state. For this reason, we strongly recommend doing a manual
67 backup of all LINSTOR resources before upgrading the cluster. In a future release, this step will be managed by the
68 operator.
69
70 1. Stop the current controller:
71 +
72 ----
73 $ kubectl patch linstorcontroller linstor-cs '{"spec":{"replicas": 0}}'
74 $ kubectl rollout status --watch deployment/linstor-cs-controller
75 ----
76 2. The following command will create a file `crds.yaml`, which stores the current state of all LINSTOR Custom Resource
77 Definitions:
78 +
79 ----
80 $ kubectl get crds | grep -o ".*.internal.linstor.linbit.com" | xargs kubectl get crds -oyaml > crds.yaml
81 ----
82 +
83 3. In addition to the definitions, the actual resources must be be backed up as well:
84 +
85 ----
86 $ kubectl get crds | grep -o ".*.internal.linstor.linbit.com" | xargs -i{} sh -c "kubectl get {} -oyaml > {}.yaml"
87 ----
88 4. Run the chart upgrade using `--set IHaveBackedUpAllMyLinstorResources=true` to acknowledge you have executed the
89 above steps.
90
91 [[s-kubernetes-upgrade-version]]
92 ==== Upgrade instructions for specific versions
93
94 Some versions require special steps, see below.
95
96 ===== Upgrade to v1.7
97
98 No additional steps necessary.
99
100 ===== Upgrade to v1.6
101
102 This versions introduces a new option to support Kubernetes distributions which use different state directories than the
103 default of `/var/lib/kubelet`. A notable example is microk8s, which uses `/var/snap/microk8s/common/var/lib/kubelet`.
104 To support this, a small addition to the `LinstorCSIDriver` CRD was necessary. As Helm does not upgrade CRDs on chart
105 upgrade, instead please run:
106
107 ----
108 $ helm repo update
109 $ helm pull linstor/linstor --untar
110 $ kubectl replace -f linstor/crds/
111 customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced
112 customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced
113 customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced
114 ----
115
116 If you do not apply the new CRDs, you will get errors like the following:
117
118 ----
119 Error: UPGRADE FAILED: error validating "": error validating data: ValidationError(LinstorCSIDriver.spec): unknown field "kubeletPath" in com.linbit.linstor.v1.LinstorCSIDriver.spec
120 ----
121
122 If you previously used the included snapshot-controller to process `VolumeSnapshot` resources, you should replace it
123 with the new charts provided by the Piraeus project. The <<s-kubernetes-add-snaphot-support,section on snapshots>> has
124 been updated to include instructions on how you can add the snapshot-controller to your cluster.
125
126 ===== Upgrade to v1.5
127
128 This version introduces a <<s-kubernetes-monitoring,monitoring>> component for DRBD resources. This requires a new image
129 and a replacement of the existing `LinstorSatelliteSet` CRD. Helm does not upgrade the CRDs on a chart upgrade,
130 instead please run:
131
132 ----
133 $ helm repo update
134 $ helm pull linstor/linstor --untar
135 $ kubectl replace -f linstor/crds/
136 customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced
137 customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced
138 customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced
139 ----
140
141 If you do not plan to use the provided <<s-kubernetes-monitoring,monitoring>> you still need to apply the above steps,
142 otherwise you will get an errors like the following
143
144 ----
145 Error: UPGRADE FAILED: error validating "": error validating data: ValidationError(LinstorSatelliteSet.spec): unknown field "monitoringImage" in com.linbit.linstor.v1.LinstorSatelliteSet.spec
146 ----
147
148 NOTE: Some Helm versions fail to set the monitoring image even after replacing the CRDs. In that case, the in-cluster
149 LinstorSatelliteSet will show an empty `monitoringImage` value. Edit the resource using
150 `kubectl edit linstorsatellitesets` and set the value to `drbd.io/drbd-reactor:v0.3.0` to enable monitoring.
151
152 ===== Upgrade to v1.4
153
154 This version introduces a new default version for the Etcd image, so take extra care that Etcd is using
155 persistent storage. *Upgrading the Etcd image without persistent storage will corrupt the cluster*.
156
157 If you are upgrading an existing cluster without making use of new Helm options, no additional steps are necessary.
158
159 If you plan to use the newly introduced `additionalProperties` and `additionalEnv` settings, you have to replace
160 the installed CustomResourceDefinitions with newer versions. Helm does not upgrade the CRDs on a chart upgrade
161
162 ----
163 $ helm pull linstor/linstor --untar
164 $ kubectl replace -f linstor/crds/
165 customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced
166 customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced
167 customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced
168 ----
169
170 ===== Upgrade to v1.3
171
172 No additional steps necessary.
173
174 ===== Upgrade to v1.2
175
176 LINSTOR operator v1.2 is supported on Kubernetes 1.17+. If you are using an older Kubernetes distribution, you may need
177 to change the default settings, for example [the CSI provisioner](https://kubernetes-csi.github.io/docs/external-provisioner.html).
178
179 There is a known issue when updating the CSI components: the pods will not be updated to the newest image and the
180 `errors` section of the LinstorCSIDrivers resource shows an error updating the DaemonSet. In this case, manually
181 delete `deployment/linstor-op-csi-controller` and `daemonset/linstor-op-csi-node`. They will be re-created by the
182 operator.
183
184 [[s-kubernetes-etcd-backup]]
185 ==== Creating Etcd Backups
186
187 To create a backup of the Etcd database and store it on your control host, run:
188
189 [source]
190 ----
191 kubectl exec linstor-op-etcd-0 -- etcdctl snapshot save /tmp/save.db
192 kubectl cp linstor-op-etcd-0:/tmp/save.db save.db
193 ----
194
195 These commands will create a file `save.db` on the machine you are running `kubectl` from.