Debian Codegraph @

UG9/en/kubernetes.adoc

raw
1 [[ch-kubernetes]]
2 == LINSTOR Volumes in Kubernetes
3
4 indexterm:[Kubernetes]This chapter describes the usage of LINSTOR in Kubernetes
5 as managed by the operator and with volumes provisioned using the
6 https://github.com/LINBIT/linstor-csi[LINSTOR CSI plugin].
7
8 This Chapter goes into great detail regarding all the install time
9 options and various configurations possible with LINSTOR and
10 Kubernetes. For those more interested in a "quick-start" for testing,
11 or those looking for some examples for reference. We have some
12 complete <<Helm Install Examples>> of a few common uses near the end
13 of the chapter.
14
15 [[s-kubernetes-overview]]
16 === Kubernetes Overview
17
18 _Kubernetes_ is a container orchestrator. Kubernetes defines the behavior of
19 containers and related services via declarative specifications. In this guide,
20 we'll focus on using `kubectl` to manipulate `.yaml` files that define the
21 specifications of Kubernetes objects.
22
23 [[s-kubernetes-deploy]]
24 === Deploying LINSTOR on Kubernetes
25
26 [[s-kubernetes-deploy-linstor-operator]]
27 ==== Deploying with the LINSTOR Operator
28
29 LINBIT provides a LINSTOR operator to commercial support customers.
30 The operator eases deployment of LINSTOR on Kubernetes by installing DRBD,
31 managing Satellite and Controller pods, and other related functions.
32
33 The operator itself is installed using a Helm v3 chart as follows:
34
35 * Create a kubernetes secret containing your my.linbit.com credentials:
36 +
37 ----
38 kubectl create secret docker-registry drbdiocred --docker-server=drbd.io --docker-username=<YOUR_LOGIN> --docker-email=<YOUR_EMAIL> --docker-password=<YOUR_PASSWORD>
39 ----
40 +
41 The name of this secret must match the one specified in the Helm values,
42 by default `drbdiocred`.
43
44 * Configure the LINSTOR database backend. By default, the chart configures Etcd as database backend. Starting with
45 1.7.0, the operator can also configure LINSTOR to use <<s-kubernetes-linstor-k8s-backend,Kubernetes as datastore>>
46 directly. If you go the Etcd route, you should configure persistent storage for it:
47 ** Use an existing storage provisioner with a default `StorageClass`.
48 ** <<s-kubernetes-etcd-hostpath-persistence,Use `hostPath` volumes>>.
49 ** Disable persistence, **for basic testing only**. This can be done by adding
50 `--set etcd.persistentVolume.enabled=false` to the `helm install` command below.
51
52 * Read <<s-kubernetes-storage, the storage guide>> and configure a basic storage setup for LINSTOR
53
54 * Read the <<s-kubernetes-securing-deployment,section on securing the deployment>> and configure as needed.
55
56 * Select the appropriate kernel module injector using `--set` with the `helm install` command in the final step.
57
58 ** Choose the injector according to the distribution you are using.
59 Select the latest version from one of `drbd9-rhel7`, `drbd9-rhel8`,... from http://drbd.io/ as appropriate.
60 The drbd9-rhel8 image should also be used for RHCOS (OpenShift). For the SUSE CaaS Platform use the SLES injector
61 that matches the base system of the CaaS Platform you are using (e.g., `drbd9-sles15sp1`). For example:
62 +
63 ----
64 operator.satelliteSet.kernelModuleInjectionImage=drbd.io/drbd9-rhel8:v9.0.30
65 ----
66
67 ** Only inject modules that are already present on the host machine. If a module is not found, it will be skipped.
68 +
69 ----
70 operator.satelliteSet.kernelModuleInjectionMode=DepsOnly
71 ----
72
73 ** Disable kernel module injection if you are installing DRBD by other means. Deprecated by `DepsOnly`
74 +
75 ----
76 operator.satelliteSet.kernelModuleInjectionMode=None
77 ----
78
79 * Finally create a Helm deployment named `linstor-op` that will set up everything.
80 +
81 ----
82 helm repo add linstor https://charts.linstor.io
83 helm install linstor-op linstor/linstor
84 ----
85 Further deployment customization is discussed in the <<s-kubernetes-advanced-deployments,advanced deployment section>>
86
87 [[s-kubernetes-linstor-k8s-backend]]
88 ===== LINSTOR k8s backend
89
90 WARNING: The Kubernetes backend for LINSTOR is still new, you might encounter issues, especially on LINSTOR version
91 upgrade. Please read <<s-kubernetes-upgrade-k8s-backend, the upgrade guide>> for more information.
92
93 Starting with LINSTOR 1.16.0, the Controller can use the Kubernetes API directly to persist its cluster state. To enable
94 this backend, use the following override file during the chart installation:
95
96 .k8s-backend.yaml
97 [source,yaml]
98 ----
99 etcd:
100 enabled: false
101 operator:
102 controller:
103 dbConnectionURL: k8s
104 ----
105
106 NOTE: It is currently **NOT** possible to migrate from an existing cluster with Etcd backend to the Kubernetes
107 backend.
108
109 [[s-kubernetes-etcd-hostpath-persistence]]
110 ===== LINSTOR etcd `hostPath` persistence
111
112 You can use the `pv-hostpath` Helm templates to create `hostPath` persistent
113 volumes. Create as many PVs as needed to satisfy your configured etcd
114 `replicas` (default 1).
115
116 Create the `hostPath` persistent volumes, substituting cluster node
117 names accordingly in the `nodes=` option:
118
119 ----
120 helm repo add linstor https://charts.linstor.io
121 helm install linstor-etcd linstor/pv-hostpath
122 ----
123
124 By default, a PV is created on every `control-plane` node. You can manually select the storage nodes by
125 passing `--set "nodes={<NODE0>,<NODE1>,<NODE2>}"` to the install command.
126
127 [[s-kubernetes-existing-database]]
128 ===== Using an existing database
129
130 LINSTOR can connect to an existing PostgreSQL, MariaDB or etcd database. For
131 instance, for a PostgreSQL instance with the following configuration:
132
133 ----
134 POSTGRES_DB: postgresdb
135 POSTGRES_USER: postgresadmin
136 POSTGRES_PASSWORD: admin123
137 ----
138
139 The Helm chart can be configured to use this database instead of deploying an
140 etcd cluster by adding the following to the Helm install command:
141
142 ----
143 --set etcd.enabled=false --set "operator.controller.dbConnectionURL=jdbc:postgresql://postgres/postgresdb?user=postgresadmin&password=admin123"
144 ----
145
146 [[s-kubernetes-storage]]
147 ==== Configuring storage
148
149 The LINSTOR operator can automate some basic storage set up for LINSTOR.
150
151 ===== Configuring storage pool creation
152
153 The LINSTOR operator can be used to create LINSTOR storage pools. Creation is under control of the
154 LinstorSatelliteSet resource:
155
156 [source]
157 ----
158 $ kubectl get LinstorSatelliteSet.linstor.linbit.com linstor-op-ns -o yaml
159 kind: LinstorSatelliteSet
160 metadata:
161 ..
162 spec:
163 ..
164 storagePools:
165 lvmPools:
166 - name: lvm-thick
167 volumeGroup: drbdpool
168 lvmThinPools:
169 - name: lvm-thin
170 thinVolume: thinpool
171 volumeGroup: ""
172 zfsPools:
173 - name: my-linstor-zpool
174 zPool: for-linstor
175 thin: true
176 ----
177
178 ====== At install time
179
180 At install time, by setting the value of `operator.satelliteSet.storagePools` when running helm install.
181
182 First create a file with the storage configuration like:
183
184 [source,yaml]
185 ----
186 operator:
187 satelliteSet:
188 storagePools:
189 lvmPools:
190 - name: lvm-thick
191 volumeGroup: drbdpool
192 ----
193
194 This file can be passed to the helm installation like this:
195
196 [source]
197 ----
198 helm install -f <file> linstor-op linstor/linstor
199 ----
200
201 ====== After install
202
203 On a cluster with the operator already configured (i.e. after `helm install`),
204 you can edit the LinstorSatelliteSet configuration like this:
205
206 [source]
207 ----
208 $ kubectl edit LinstorSatelliteSet.linstor.linbit.com <satellitesetname>
209 ----
210
211 The storage pool configuration can be updated like in the example above.
212
213 ===== Preparing physical devices
214
215 By default, LINSTOR expects the referenced VolumeGroups, ThinPools and so on to be present. You can use the
216 `devicePaths: []` option to let LINSTOR automatically prepare devices for the pool. Eligible for automatic configuration
217 are block devices that:
218
219 * Are a root device (no partition)
220 * do not contain partition information
221 * have more than 1 GiB
222
223 To enable automatic configuration of devices, set the `devicePaths` key on `storagePools` entries:
224
225 [source,yaml]
226 ----
227 storagePools:
228 lvmPools:
229 - name: lvm-thick
230 volumeGroup: drbdpool
231 devicePaths:
232 - /dev/vdb
233 lvmThinPools:
234 - name: lvm-thin
235 thinVolume: thinpool
236 volumeGroup: linstor_thinpool
237 devicePaths:
238 - /dev/vdc
239 - /dev/vdd
240 ----
241
242 Currently, this method supports creation of LVM and LVMTHIN storage pools.
243
244 ===== `lvmPools` configuration
245 * `name` name of the LINSTOR storage pool.Required
246 * `volumeGroup` name of the VG to create.Required
247 * `devicePaths` devices to configure for this pool.Must be empty and >= 1GiB to be recognized.Optional
248 * `raidLevel` LVM raid level.Optional
249 * `vdo` Enable [VDO] (requires VDO tools in the satellite).Optional
250 * `vdoLogicalSizeKib` Size of the created VG (expected to be bigger than the backing devices by using VDO).Optional
251 * `vdoSlabSizeKib` Slab size for VDO. Optional
252
253 [VDO]: https://www.redhat.com/en/blog/look-vdo-new-linux-compression-layer
254
255 ===== `lvmThinPools` configuration
256 * `name` name of the LINSTOR storage pool. Required
257 * `volumeGroup` VG to use for the thin pool. If you want to use `devicePaths`, you must set this to `""`.
258 This is required because LINSTOR does not allow configuration of the VG name when preparing devices.
259 * `thinVolume` name of the thinpool. Required
260 * `devicePaths` devices to configure for this pool. Must be empty and >= 1GiB to be recognized. Optional
261 * `raidLevel` LVM raid level. Optional
262
263 NOTE: The volume group created by LINSTOR for LVMTHIN pools will always follow the scheme "linstor_$THINPOOL".
264
265 ===== `zfsPools` configuration
266 * `name` name of the LINSTOR storage pool. Required
267 * `zPool` name of the zpool to use. Must already be present on all machines. Required
268 * `thin` `true` to use thin provisioning, `false` otherwise. Required
269
270 ===== Using `automaticStorageType` (DEPRECATED)
271
272 _ALL_ eligible devices will be prepared according to the value of `operator.satelliteSet.automaticStorageType`, unless
273 they are already prepared using the `storagePools` section. Devices are added to a storage pool based on the device
274 name (i.e. all `/dev/nvme1` devices will be part of the pool `autopool-nvme1`)
275
276 The possible values for `operator.satelliteSet.automaticStorageType`:
277
278 * `None` no automatic set up (default)
279 * `LVM` create a LVM (thick) storage pool
280 * `LVMTHIN` create a LVM thin storage pool
281 * `ZFS` create a ZFS based storage pool (**UNTESTED**)
282
283 [[s-kubernetes-securing-deployment]]
284 ==== Securing deployment
285 This section describes the different options for enabling security features available when
286 using this operator. The following guides assume the operator is installed <<s-kubernetes-deploy-linstor-operator,using Helm>>
287
288 ===== Secure communication with an existing etcd instance
289
290 Secure communication to an `etcd` instance can be enabled by providing a CA certificate to the operator in form of a
291 kubernetes secret. The secret has to contain the key `ca.pem` with the PEM encoded CA certificate as value.
292
293 The secret can then be passed to the controller by passing the following argument to `helm install`
294
295 ----
296 --set operator.controller.dbCertSecret=<secret name>
297 ----
298
299 ====== Authentication with `etcd` using certificates
300
301 If you want to use TLS certificates to authenticate with an `etcd` database, you need to set the following option on
302 helm install:
303
304 ----
305 --set operator.controller.dbUseClientCert=true
306 ----
307
308 If this option is active, the secret specified in the above section must contain two additional keys:
309
310 * `client.cert` PEM formatted certificate presented to `etcd` for authentication
311 * `client.key` private key **in PKCS8 format**, matching the above client certificate.
312
313 Keys can be converted into PKCS8 format using `openssl`:
314
315 ----
316 openssl pkcs8 -topk8 -nocrypt -in client-key.pem -out client-key.pkcs8
317 ----
318
319 ===== Configuring secure communication between LINSTOR components
320
321 The default communication between LINSTOR components is not secured by TLS. If this is needed for your setup,
322 follow these steps:
323
324 * Create private keys in the java keystore format, one for the controller, one for all satellites:
325 ----
326 keytool -keyalg rsa -keysize 2048 -genkey -keystore satellite-keys.jks -storepass linstor -alias satellite -dname "CN=XX, OU=satellite, O=Example, L=XX, ST=XX, C=X"
327 keytool -keyalg rsa -keysize 2048 -genkey -keystore control-keys.jks -storepass linstor -alias control -dname "CN=XX, OU=control, O=Example, L=XX, ST=XX, C=XX"
328 ----
329 * Create a trust store with the public keys that each component needs to trust:
330 * Controller needs to trust the satellites
331 * Nodes need to trust the controller
332 +
333 ----
334 keytool -importkeystore -srcstorepass linstor -deststorepass linstor -srckeystore control-keys.jks -destkeystore satellite-trust.jks
335 keytool -importkeystore -srcstorepass linstor -deststorepass linstor -srckeystore satellite-keys.jks -destkeystore control-trust.jks
336 ----
337
338 * Create kubernetes secrets that can be passed to the controller and satellite pods
339 +
340 ----
341 kubectl create secret generic control-secret --from-file=keystore.jks=control-keys.jks --from-file=certificates.jks=control-trust.jks
342 kubectl create secret generic satellite-secret --from-file=keystore.jks=satellite-keys.jks --from-file=certificates.jks=satellite-trust.jks
343 ----
344 * Pass the names of the created secrets to `helm install`
345 +
346 ----
347 --set operator.satelliteSet.sslSecret=satellite-secret --set operator.controller.sslSecret=control-secret
348 ----
349
350 IMPORTANT: It is currently **NOT** possible to change the keystore password. LINSTOR expects the passwords to be
351 `linstor`. This is a current limitation of LINSTOR.
352
353 ===== Configuring secure communications for the LINSTOR API
354
355 Various components need to talk to the LINSTOR controller via its REST interface. This interface can be
356 secured via HTTPS, which automatically includes authentication. For HTTPS+authentication to work, each component
357 needs access to:
358
359 * A private key
360 * A certificate based on the key
361 * A trusted certificate, used to verify that other components are trustworthy
362
363 The next sections will guide you through creating all required components.
364
365 ====== Creating the private keys
366
367 Private keys can be created using java's keytool
368
369 ----
370 keytool -keyalg rsa -keysize 2048 -genkey -keystore controller.pkcs12 -storetype pkcs12 -storepass linstor -ext san=dns:linstor-op-cs.default.svc -dname "CN=XX, OU=controller, O=Example, L=XX, ST=XX, C=X" -validity 5000
371 keytool -keyalg rsa -keysize 2048 -genkey -keystore client.pkcs12 -storetype pkcs12 -storepass linstor -dname "CN=XX, OU=client, O=Example, L=XX, ST=XX, C=XX" -validity 5000
372 ----
373
374 The clients need private keys and certificate in a different format, so we need to convert it
375
376 ----
377 openssl pkcs12 -in client.pkcs12 -passin pass:linstor -out client.cert -clcerts -nokeys
378 openssl pkcs12 -in client.pkcs12 -passin pass:linstor -out client.key -nocerts -nodes
379 ----
380
381 NOTE: The alias specified for the controller key (i.e. `-ext san=dns:linstor-op-cs.default.svc`) has to exactly match the
382 service name created by the operator. When using `helm`, this is always of the form `<release-name>-cs.<release-namespace>.svc`.
383
384 IMPORTANT: It is currently NOT possible to change the keystore password. LINSTOR expects the passwords to be linstor. This is a current limitation of LINSTOR
385
386 ====== Create the trusted certificates
387
388 For the controller to trust the clients, we can use the following command to create a truststore, importing the client certificate
389
390 ----
391 keytool -importkeystore -srcstorepass linstor -srckeystore client.pkcs12 -deststorepass linstor -deststoretype pkcs12 -destkeystore controller-trust.pkcs12
392 ----
393
394 For the client, we have to convert the controller certificate into a different format
395
396 ----
397 openssl pkcs12 -in controller.pkcs12 -passin pass:linstor -out ca.pem -clcerts -nokeys
398 ----
399
400 ====== Create Kubernetes secrets
401
402 Now you can create secrets for the controller and for clients:
403
404 ----
405 kubectl create secret generic http-controller --from-file=keystore.jks=controller.pkcs12 --from-file=truststore.jks=controller-trust.pkcs12
406 kubectl create secret generic http-client --from-file=ca.pem=ca.pem --from-file=client.cert=client.cert --from-file=client.key=client.key
407 ----
408
409 The names of the secrets can be passed to `helm install` to configure all clients to use https.
410
411 ----
412 --set linstorHttpsControllerSecret=http-controller --set linstorHttpsClientSecret=http-client
413 ----
414
415 ===== Automatically set the passphrase for LINSTOR
416
417 LINSTOR needs to store confidential data to support encrypted information. This data is protected by a master
418 passphrase. A passphrase is automatically generated on the first chart install.
419
420 If you want to use a custom passphrase, store it in a secret:
421
422 ----
423 kubectl create secret generic linstor-pass --from-literal=MASTER_PASSPHRASE=<password>
424 ----
425
426 On install, add the following arguments to the helm command:
427
428 ----
429 --set operator.controller.luksSecret=linstor-pass
430 ----
431
432 ===== Helm Install Examples
433
434 All the below examples use the following `sp-values.yaml` file. Feel
435 free to adjust this for your uses and environment. See <<Configuring storage pool creation>>
436 for further details.
437
438 -----
439 operator:
440 satelliteSet:
441 storagePools:
442 lvmThinPools:
443 - name: lvm-thin
444 thinVolume: thinpool
445 volumeGroup: ""
446 devicePaths:
447 - /dev/sdb
448 -----
449
450 Default install. Please note this does not setup any persistence for
451 the backing etcd key-value store.
452
453 WARNING: This is not suggested for any use
454 outside of testing.
455
456 -----
457 kubectl create secret docker-registry drbdiocred --docker-server=drbd.io --docker-username=<YOUR_LOGIN> --docker-password=<YOUR_PASSWORD>
458 helm repo add linstor https://charts.linstor.io
459 helm install linstor-op linstor/linstor
460 -----
461
462 Install with LINSTOR storage-pools defined at install via
463 `sp-values.yaml`, persistent hostPath volumes, 3 etcd replicas, and by
464 compiling the DRBD kernel modules for the host kernels.
465
466 This should be adequate for most basic deployments. Please note that
467 this deployment is not using the pre-compiled DRBD kernel modules just
468 to make this command more portable. Using the pre-compiled binaries
469 will make for a much faster install and deployment. Using the
470 `Compile` option would not be suggested for use in a large Kubernetes clusters.
471
472 -----
473 kubectl create secret docker-registry drbdiocred --docker-server=drbd.io --docker-username=<YOUR_LOGIN> --docker-password=<YOUR_PASSWORD>
474 helm repo add linstor https://charts.linstor.io
475 helm install linstor-etcd linstor/pv-hostpath --set "nodes={<NODE0>,<NODE1>,<NODE2>}"
476 helm install -f sp-values.yaml linstor-op linstor/linstor --set etcd.replicas=3 --set operator.satelliteSet.kernelModuleInjectionMode=Compile
477 -----
478
479 Install with LINSTOR storage-pools defined at install via
480 `sp-values.yaml`, use an already created PostgreSQL DB (preferably
481 clustered), instead of etcd, and use already compiled kernel modules for
482 DRBD. Additionally, we'll disable the Stork scheduler in this example.
483
484 The PostgreSQL database in this particular example is reachable via a
485 service endpoint named `postgres`. PostgreSQL itself is configured with
486 `POSTGRES_DB=postgresdb`, `POSTGRES_USER=postgresadmin`, and
487 `POSTGRES_PASSWORD=admin123`
488
489 -----
490 kubectl create secret docker-registry drbdiocred --docker-server=drbd.io --docker-username=<YOUR_LOGIN> --docker-email=<YOUR_EMAIL> --docker-password=<YOUR_PASSWORD>
491 helm repo add linstor https://charts.linstor.io
492 helm install -f sp-values.yaml linstor-op linstor/linstor --set etcd.enabled=false --set "operator.controller.dbConnectionURL=jdbc:postgresql://postgres/postgresdb?user=postgresadmin&password=admin123" --set stork.enabled=false
493 -----
494
495 [[s-kubernetes-helm-terminate]]
496 ===== Terminating Helm deployment
497
498 To protect the storage infrastructure of the cluster from accidentally deleting vital components, it is necessary to perform some manual steps before deleting a Helm deployment.
499
500 1. Delete all volume claims managed by LINSTOR components. You can use the following command to get a list of volume claims managed by LINSTOR. After checking that none of the listed volumes still hold needed data, you can delete them using the generated kubectl delete command.
501 +
502 ----
503 $ kubectl get pvc --all-namespaces -o=jsonpath='{range .items[?(@.metadata.annotations.volume\.beta\.kubernetes\.io/storage-provisioner=="linstor.csi.linbit.com")]}kubectl delete pvc --namespace {.metadata.namespace} {.metadata.name}{"\n"}{end}'
504 kubectl delete pvc --namespace default data-mysql-0
505 kubectl delete pvc --namespace default data-mysql-1
506 kubectl delete pvc --namespace default data-mysql-2
507 ----
508 +
509 WARNING: These volumes, once deleted, cannot be recovered.
510
511 2. Delete the LINSTOR controller and satellite resources.
512 +
513 Deployment of LINSTOR satellite and controller is controlled by the LinstorSatelliteSet and LinstorController resources. You can delete the resources associated with your deployment using kubectl
514 +
515 ----
516 kubectl delete linstorcontroller <helm-deploy-name>-cs
517 kubectl delete linstorsatelliteset <helm-deploy-name>-ns
518 ----
519 +
520 After a short wait, the controller and satellite pods should terminate. If they continue to run, you can check the above resources for errors (they are only removed after all associated pods terminate)
521
522 3. Delete the Helm deployment.
523 +
524 If you removed all PVCs and all LINSTOR pods have terminated, you can uninstall the helm deployment
525 +
526 ----
527 helm uninstall linstor-op
528 ----
529 +
530 NOTE: Due to the Helm's current policy, the Custom Resource Definitions named LinstorController and LinstorSatelliteSet will not be deleted by the command.
531 More information regarding Helm's current position on CRDs can be found
532 https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#method-1-let-helm-do-it-for-you[here].
533
534 [[s-kubernetes-advanced-deployments]]
535 ==== Advanced deployment options
536
537 The helm charts provide a set of further customization options for advanced use cases.
538
539 [source,yaml]
540 ----
541 global:
542 imagePullPolicy: IfNotPresent # empty pull policy means k8s default is used ("always" if tag == ":latest", "ifnotpresent" else) <1>
543 setSecurityContext: true # Force non-privileged containers to run as non-root users
544 # Dependency charts
545 etcd:
546 enabled: true
547 persistentVolume:
548 enabled: true
549 storage: 1Gi
550 replicas: 1 # How many instances of etcd will be added to the initial cluster. <2>
551 resources: {} # resource requirements for etcd containers <3>
552 image:
553 repository: gcr.io/etcd-development/etcd
554 tag: v3.4.15
555 stork:
556 enabled: false
557 storkImage: docker.io/openstorage/stork:2.6.5
558 schedulerImage: k8s.gcr.io/kube-scheduler-amd64
559 schedulerTag: ""
560 replicas: 1 <2>
561 storkResources: {} # resources requirements for the stork plugin containers <3>
562 schedulerResources: {} # resource requirements for the kube-scheduler containers <3>
563 podsecuritycontext: {}
564 csi:
565 enabled: true
566 pluginImage: "drbd.io/linstor-csi:v0.17.0"
567 csiAttacherImage: k8s.gcr.io/sig-storage/csi-attacher:v3.3.0
568 csiLivenessProbeImage: k8s.gcr.io/sig-storage/livenessprobe:v2.5.0
569 csiNodeDriverRegistrarImage: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0
570 csiProvisionerImage: k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0
571 csiSnapshotterImage: k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1
572 csiResizerImage: k8s.gcr.io/sig-storage/csi-resizer:v1.3.0
573 controllerReplicas: 1 <2>
574 nodeAffinity: {} <4>
575 nodeTolerations: [] <4>
576 controllerAffinity: {} <4>
577 controllerTolerations: [] <4>
578 enableTopology: true
579 resources: {} <3>
580 kubeletPath: /var/lib/kubelet <7>
581 priorityClassName: ""
582 drbdRepoCred: drbdiocred
583 linstorHttpsControllerSecret: "" # <- name of secret containing linstor server certificates+key.
584 linstorHttpsClientSecret: "" # <- name of secret containing linstor client certificates+key.
585 controllerEndpoint: "" # <- override to the generated controller endpoint. use if controller is not deployed via operator
586 psp:
587 privilegedRole: ""
588 unprivilegedRole: ""
589 operator:
590 replicas: 1 # <- number of replicas for the operator deployment <2>
591 image: "drbd.io/linstor-operator:v1.7.0"
592 affinity: {} <4>
593 tolerations: [] <4>
594 resources: {} <3>
595 podsecuritycontext: {}
596 controller:
597 enabled: true
598 controllerImage: "drbd.io/linstor-controller:v1.17.0"
599 dbConnectionURL: ""
600 luksSecret: ""
601 dbCertSecret: ""
602 dbUseClientCert: false
603 sslSecret: ""
604 affinity: {} <4>
605 tolerations: <4>
606 - key: node-role.kubernetes.io/master
607 operator: "Exists"
608 effect: "NoSchedule"
609 resources: {} <3>
610 replicas: 1 <2>
611 additionalEnv: [] <5>
612 additionalProperties: {} <6>
613 satelliteSet:
614 enabled: true
615 satelliteImage: "drbd.io/linstor-satellite:v1.17.0"
616 storagePools: {}
617 sslSecret: ""
618 automaticStorageType: None
619 affinity: {} <4>
620 tolerations: [] <4>
621 resources: {} <3>
622 monitoringImage: "drbd.io/drbd-reactor:v0.5.1"
623 kernelModuleInjectionImage: "drbd.io/drbd9-rhel7:v9.1.4"
624 kernelModuleInjectionMode: ShippedModules
625 kernelModuleInjectionResources: {} <3>
626 additionalEnv: [] <5>
627 haController:
628 enabled: true
629 image: drbd.io/linstor-k8s-ha-controller:v0.2.1
630 affinity: {} <4>
631 tolerations: [] <4>
632 resources: {} <3>
633 replicas: 1 <2>
634 ----
635 <1> Sets the pull policy for all images.
636 <2> Controls the number of replicas for each component.
637 <3> Set container resource requests and limits. See https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/[the kubernetes docs].
638 Most containers need a minimal amount of resources, except for:
639 * `etcd.resources` See the https://etcd.io/docs/v3.4.0/op-guide/hardware/[etcd docs]
640 * `operator.controller.resources` Around `700MiB` memory is required
641 * `operater.satelliteSet.resources` Around `700MiB` memory is required
642 * `operator.satelliteSet.kernelModuleInjectionResources` If kernel modules are compiled, 1GiB of memory is required.
643 <4> Affinity and toleration determine where pods are scheduled on the cluster. See the
644 https://kubernetes.io/docs/concepts/scheduling-eviction/[kubernetes docs on affinity and toleration].
645 This may be especially important for the `operator.satelliteSet` and `csi.node*` values. To schedule a pod using
646 a LINSTOR persistent volume, the node requires a running LINSTOR satellite and LINSTOR CSI pod.
647 <5> Sets additional environments variables to pass to the Linstor Controller and Satellites. Uses the same format as
648 https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/[the `env` value of a container]
649 <6> Sets additional properties on the Linstor Controller. Expects a simple mapping of `<property-key>: <value>`.
650 <7> Kubelet expects every CSI plugin to mount volumes under a specific subdirectory of it's own state directory. By
651 default, this state directory is `/var/lib/kubelet`. Some Kubernetes distributions use a different directory:
652 * microk8s: `/var/snap/microk8s/common/var/lib/kubelet`
653
654 [[s-kubernetes-ha-deployment]]
655 ===== High Availability Deployment
656 To create a High Availability deployment of all components, take a look at the https://github.com/piraeusdatastore/piraeus-operator/blob/b00fd34/doc/scheduling.md[upstream guide]
657 The default values are chosen so that scaling the components to multiple replicas ensures that the replicas are placed on different nodes. This ensures
658 that a single node failures will not interrupt the service.
659
660 [[s-kubernetes-monitoring]]
661 ===== Monitoring with Prometheus
662 Starting with Linstor Operator v1.5.0, you can use https://prometheus.io/[Prometheus] to monitor Linstor components.
663 The operator will set up monitoring containers along the existing components and make them available as a `Service`.
664
665 If you use the https://prometheus-operator.dev/[Prometheus Operator], the Linstor Operator will also set up the `ServiceMonitor`
666 instances. The metrics will automatically be collected by the Prometheus instance associated to the operator, assuming
667 https://prometheus-operator.dev/docs/kube/monitoring-other-namespaces/[watching the Piraeus namespace is enabled].
668
669 To disable exporting of metrics, set `operator.satelliteSet.monitoringImage` to an empty value.
670
671 ====== Linstor Controller Monitoring
672
673 The Linstor Controller exports cluster-wide metrics. Metrics are exported on the existing controller service, using the
674 path https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-linstor-monitoring[`/metrics`].
675
676 ====== DRBD Resource Monitoring
677
678 All satellites are bundled with a secondary container that uses https://github.com/LINBIT/drbd-reactor/[`drbd-reactor`]
679 to export metrics directly from DRBD. The metrics are available on port 9942, for convenience a headless service named
680 `<linstorsatelliteset-name>-monitoring` is provided.
681
682 If you want to disable the monitoring container, set `monitoringImage` to `""` in your LinstorSatelliteSet resource.
683
684 [[s-kubernetes-deploy-external-controller]]
685 ==== Deploying with an external LINSTOR controller
686
687 The operator can configure the satellites and CSI plugin to use an existing LINSTOR setup. This can be useful in cases
688 where the storage infrastructure is separate from the Kubernetes cluster. Volumes can be provisioned in diskless mode
689 on the Kubernetes nodes while the storage nodes will provide the backing disk storage.
690
691 To skip the creation of a LINSTOR Controller deployment and configure the other components to use your existing LINSTOR
692 Controller, use the following options when running `helm install`:
693
694 * `operator.controller.enabled=false` This disables creation of the `LinstorController` resource
695 * `operator.etcd.enabled=false` Since no LINSTOR Controller will run on Kubernetes, no database is required.
696 * `controllerEndpoint=<url-of-linstor-controller>` The HTTP endpoint of the existing LINSTOR Controller. For example: `http://linstor.storage.cluster:3370/`
697
698 After all pods are ready, you should see the Kubernetes cluster nodes as satellites in your LINSTOR setup.
699
700 IMPORTANT: Your kubernetes nodes must be reachable using their IP by the controller and storage nodes.
701
702 Create a storage class referencing an existing storage pool on your storage nodes.
703
704 [source,yaml]
705 ----
706 apiVersion: storage.k8s.io/v1
707 kind: StorageClass
708 metadata:
709 name: linstor-on-k8s
710 provisioner: linstor.csi.linbit.com
711 parameters:
712 autoPlace: "3"
713 storagePool: existing-storage-pool
714 resourceGroup: linstor-on-k8s
715 ----
716
717 You can provision new volumes by creating PVCs using your storage class. The volumes will first be placed only on nodes
718 with the given storage pool, i.e. your storage infrastructure. Once you want to use the volume in a pod, LINSTOR CSI
719 will create a diskless resource on the Kubernetes node and attach over the network to the diskful resource.
720
721 [[s-kubernetes-deploy-piraeus-operator]]
722 ==== Deploying with the Piraeus Operator
723
724 The community supported edition of the LINSTOR deployment in Kubernetes is
725 called Piraeus. The Piraeus project provides
726 https://github.com/piraeusdatastore/piraeus-operator[an operator] for
727 deployment.
728
729 [[s-kubernetes-linstor-interacting]]
730 === Interacting with LINSTOR in Kubernetes
731
732 The Controller pod includes a LINSTOR Client, making it easy to interact directly with LINSTOR.
733 For instance:
734
735 ----
736 kubectl exec deployment/linstor-op-cs-controller -- linstor storage-pool list
737 ----
738
739 For a convenient shortcut to the above command, download https://github.com/piraeusdatastore/kubectl-linstor/releases[`kubectl-linstor`]
740 and install it alongside `kubectl`. Then you can use `kubectl linstor` to get access to the complete Linstor CLI
741
742 ----
743 $ kubectl linstor node list
744 ╭───────────────────────────────────────────────────────────────────────────────────────────────╮
745 ┊ Node ┊ NodeType ┊ Addresses ┊ State ┊
746 ╞═══════════════════════════════════════════════════════════════════════════════════════════════╡
747 ┊ kube-node-01.test ┊ SATELLITE ┊ 10.43.224.26:3366 (PLAIN) ┊ Online ┊
748 ┊ kube-node-02.test ┊ SATELLITE ┊ 10.43.224.27:3366 (PLAIN) ┊ Online ┊
749 ┊ kube-node-03.test ┊ SATELLITE ┊ 10.43.224.28:3366 (PLAIN) ┊ Online ┊
750 ┊ linstor-op-cs-controller-85b4f757f5-kxdvn ┊ CONTROLLER ┊ 172.24.116.114:3366 (PLAIN) ┊ Online ┊
751 ╰───────────────────────────────────────────────────────────────────────────────────────────────╯
752 ----
753
754 It also expands references to PVCs to the matching Linstor resource
755
756 ----
757 $ kubectl linstor resource list -r pvc:my-namespace/demo-pvc-1 --all
758 pvc:my-namespace/demo-pvc-1 -> pvc-2f982fb4-bc05-4ee5-b15b-688b696c8526
759 ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
760 ┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
761 ╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
762 ┊ pvc-2f982fb4-bc05-4ee5-b15b-688b696c8526 ┊ kube-node-01.test ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2021-02-05 09:16:09 ┊
763 ┊ pvc-2f982fb4-bc05-4ee5-b15b-688b696c8526 ┊ kube-node-02.test ┊ 7000 ┊ Unused ┊ Ok ┊ TieBreaker ┊ 2021-02-05 09:16:08 ┊
764 ┊ pvc-2f982fb4-bc05-4ee5-b15b-688b696c8526 ┊ kube-node-03.test ┊ 7000 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2021-02-05 09:16:09 ┊
765 ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
766 ----
767
768 It also expands references of the form `pod:[<namespace>/]<podname>` into a list resources in use by the pod.
769
770 This should only be necessary for investigating problems and accessing advanced functionality.
771 Regular operation such as creating volumes should be achieved via the
772 <<s-kubernetes-basic-configuration-and-deployment,Kubernetes integration>>.
773
774 [[s-kubernetes-basic-configuration-and-deployment]]
775 === Basic Configuration and Deployment
776
777 Once all linstor-csi __Pod__s are up and running, we can provision volumes
778 using the usual Kubernetes workflows.
779
780 Configuring the behavior and properties of LINSTOR volumes deployed via Kubernetes
781 is accomplished via the use of __StorageClass__es.
782
783 IMPORTANT: the "resourceGroup" parameter is mandatory. Usually you want it to be unique and the same as the storage class name.
784
785 Here below is the simplest practical _StorageClass_ that can be used to deploy volumes:
786
787 .linstor-basic-sc.yaml
788 [source,yaml]
789 ----
790 apiVersion: storage.k8s.io/v1
791 kind: StorageClass
792 metadata:
793 # The name used to identify this StorageClass.
794 name: linstor-basic-storage-class
795 # The name used to match this StorageClass with a provisioner.
796 # linstor.csi.linbit.com is the name that the LINSTOR CSI plugin uses to identify itself
797 provisioner: linstor.csi.linbit.com
798 volumeBindingMode: WaitForFirstConsumer
799 parameters:
800 # LINSTOR will provision volumes from the drbdpool storage pool configured
801 # On the satellite nodes in the LINSTOR cluster specified in the plugin's deployment
802 storagePool: "drbdpool"
803 resourceGroup: "linstor-basic-storage-class"
804 # Setting a fstype is required for "fsGroup" permissions to work correctly.
805 # Currently supported: xfs/ext4
806 csi.storage.k8s.io/fstype: xfs
807 ----
808
809 We can create the _StorageClass_ with the following command:
810
811 ----
812 kubectl create -f linstor-basic-sc.yaml
813 ----
814
815 Now that our _StorageClass_ is created, we can now create a _PersistentVolumeClaim_
816 which can be used to provision volumes known both to Kubernetes and LINSTOR:
817
818 .my-first-linstor-volume-pvc.yaml
819 [source,yaml]
820 ----
821 kind: PersistentVolumeClaim
822 apiVersion: v1
823 metadata:
824 name: my-first-linstor-volume
825 spec:
826 storageClassName: linstor-basic-storage-class
827 accessModes:
828 - ReadWriteOnce
829 resources:
830 requests:
831 storage: 500Mi
832 ----
833
834 We can create the _PersistentVolumeClaim_ with the following command:
835
836 ----
837 kubectl create -f my-first-linstor-volume-pvc.yaml
838 ----
839
840 This will create a _PersistentVolumeClaim_, but no volume will be created just yet.
841 The storage class we used specified `volumeBindingMode: WaitForFirstConsumer`, which
842 means that the volume is only created once a workload starts using it. This ensures
843 that the volume is placed on the same node as the workload.
844
845 For our example, we create a simple Pod, which mounts or volume by referencing the
846 _PersistentVolumeClaim_.
847 .my-first-linstor-volume-pod.yaml
848 [source,yaml]
849 ----
850 apiVersion: v1
851 kind: Pod
852 metadata:
853 name: fedora
854 namespace: default
855 spec:
856 containers:
857 - name: fedora
858 image: fedora
859 command: [/bin/bash]
860 args: ["-c", "while true; do sleep 10; done"]
861 volumeMounts:
862 - name: my-first-linstor-volume
863 mountPath: /data
864 ports:
865 - containerPort: 80
866 volumes:
867 - name: my-first-linstor-volume
868 persistentVolumeClaim:
869 claimName: "my-first-linstor-volume"
870 ----
871
872 We can create the _Pod_ with the following command:
873
874 ----
875 kubectl create -f my-first-linstor-volume-pod.yaml
876 ----
877
878 Running `kubectl describe pod fedora` can be used to confirm that _Pod_
879 scheduling and volume attachment succeeded. Taking a look at the _PersistentVolumeClaim_,
880 we can see that it is now bound to a volume.
881
882 To remove a volume, please ensure that no pod is using it and then delete the
883 _PersistentVolumeClaim_ via `kubectl`. For example, to remove the volume that we
884 just made, run the following two commands, noting that the _Pod_ must be
885 unscheduled before the _PersistentVolumeClaim_ will be removed:
886
887 ----
888 kubectl delete pod fedora # unschedule the pod.
889
890 kubectl get pod -w # wait for pod to be unscheduled
891
892 kubectl delete pvc my-first-linstor-volume # remove the PersistentVolumeClaim, the PersistentVolume, and the LINSTOR Volume.
893 ----
894
895 [[s-kubernetes-sc-parameters]]
896 ==== Available parameters in a StorageClass
897
898 The following storage class contains all currently available parameters to configure the provisioned storage.
899
900 NOTE: `linstor.csi.linbit.com/` is an optional, but recommended prefix for LINSTOR CSI specific parameters.
901
902 [source,yaml]
903 ----
904 apiVersion: storage.k8s.io/v1
905 kind: StorageClass
906 metadata:
907 name: full-example
908 provisioner: linstor.csi.linbit.com
909 parameters:
910 # CSI related parameters
911 csi.storage.k8s.io/fstype: xfs
912 # LINSTOR parameters
913 linstor.csi.linbit.com/autoPlace: "2"
914 linstor.csi.linbit.com/placementCount: "2"
915 linstor.csi.linbit.com/resourceGroup: "full-example"
916 linstor.csi.linbit.com/storagePool: "my-storage-pool"
917 linstor.csi.linbit.com/disklessStoragePool: "DfltDisklessStorPool"
918 linstor.csi.linbit.com/layerList: "drbd storage"
919 linstor.csi.linbit.com/placementPolicy: "AutoPlaceTopology"
920 linstor.csi.linbit.com/allowRemoteVolumeAccess: "true"
921 linstor.csi.linbit.com/encryption: "true"
922 linstor.csi.linbit.com/nodeList: "diskful-a diskful-b"
923 linstor.csi.linbit.com/clientList: "diskless-a diskless-b"
924 linstor.csi.linbit.com/replicasOnSame: "zone=a"
925 linstor.csi.linbit.com/replicasOnDifferent: "rack"
926 linstor.csi.linbit.com/disklessOnRemaining: "false"
927 linstor.csi.linbit.com/doNotPlaceWithRegex: "tainted.*"
928 linstor.csi.linbit.com/fsOpts: "-E nodiscard"
929 linstor.csi.linbit.com/mountOpts: "noatime"
930 linstor.csi.linbit.com/postMountXfsOpts: "extsize 2m"
931 # Linstor properties
932 property.linstor.csi.linbit.com/*: <x>
933 # DRBD parameters
934 DrbdOptions/*: <x>
935 ----
936
937 [[s-kubernetes-filesystem]]
938 ==== csi.storage.k8s.io/fstype
939
940 Sets the file system type to create for `volumeMode: FileSystem` PVCs. Currently supported are:
941
942 * `ext4` (default)
943 * `xfs`
944
945 [[s-kubernetes-autoplace]]
946 ==== autoPlace
947
948 `autoPlace` is an integer that determines the amount of replicas a volume of
949 this _StorageClass_ will have. For instance, `autoPlace: "3"` will produce
950 volumes with three-way replication. If neither `autoPlace` nor `nodeList` are
951 set, volumes will be <<s-autoplace-linstor,automatically placed>> on one node.
952
953 IMPORTANT: If you use this option, you must not use <<s-kubernetes-nodelist,nodeList>>.
954
955 IMPORTANT: You have to use quotes, otherwise Kubernetes will complain about a malformed _StorageClass_.
956
957 TIP: This option (and all options which affect autoplacement behavior) modifies the
958 number of LINSTOR nodes on which the underlying storage for volumes will be
959 provisioned and is orthogonal to which _kubelets_ those volumes will be accessible
960 from.
961
962 ==== placementCount
963
964 `placementCount` is an alias for <<s-kubernetes-autoplace,`autoPlace`>>
965
966 ==== resourceGroup
967
968 The <<s-linstor-resource-groups, LINSTOR Resource Group (RG)>> to associate with this StorageClass. If not set,
969 a new RG will be created for each new PVC.
970
971 [[s-kubernetes-storagepool]]
972 ==== storagePool
973
974 `storagePool` is the name of the LINSTOR <<s-storage_pools,storage pool>> that
975 will be used to provide storage to the newly-created volumes.
976
977 CAUTION: Only nodes configured with this same _storage pool_ with be considered
978 for <<s-kubernetes-autoplace,autoplacement>>. Likewise, for _StorageClasses_ using
979 <<s-kubernetes-nodelist,nodeList>> all nodes specified in that list must have this
980 _storage pool_ configured on them.
981
982 [[s-kubernetes-disklessstoragepool]]
983 ==== disklessStoragePool
984
985 `disklessStoragePool` is an optional parameter that only effects LINSTOR volumes
986 assigned disklessly to _kubelets_ i.e., as clients. If you have a custom
987 _diskless storage pool_ defined in LINSTOR, you'll specify that here.
988
989 ==== layerList
990
991 A comma-separated list of layers to use for the created volumes. The available layers and their order are described
992 towards the end of <<s-linstor-without-drbd, this section>>. Defaults to `drbd,storage`
993
994 [[s-kubernetes-placementpolicy]]
995 ==== placementPolicy
996
997 Select from one of the available volume schedulers:
998
999 * `AutoPlaceTopology`, the default: Use topology information from Kubernetes together with user provided constraints (see <<s-kubernetes-replicasonsame>> and <<s-kubernetes-replicasondifferent>>).
1000 * `AutoPlace` Use LINSTOR autoplace, influenced by <<s-kubernetes-replicasonsame>> and <<s-kubernetes-replicasondifferent>>
1001 * `FollowTopology`: Use CSI Topology information to place at least one volume in each "preferred" zone. Only useable if CSI Topology is enabled.
1002 * `Manual`: Use only the nodes listed in `nodeList` and `clientList`.
1003 * `Balanced`: **EXPERIMENTAL** Place volumes across failure domains, using the least used storage pool on each selected node.
1004
1005 [[s-kubernetes-params-allow-remote-volume-access]]
1006 ==== allowRemoteVolumeAccess
1007
1008 Control on which nodes a volume is accessible. The value for this option can take two different forms:
1009
1010 - A simple `"true"` or `"false"` allows access from all nodes, or only those nodes with diskfull resources.
1011
1012 - Advanced rules, which allow more granular rules on which nodes can access the volume.
1013 +
1014 The current implementation can grant access to the volume for nodes that share the same labels. For example, if you want
1015 to allow access from all nodes in the same region and zone as a diskfull resource, you could use:
1016 +
1017 [source,yaml]
1018 ----
1019 parameters:
1020 linstor.csi.linbit.com/allowRemoteVolumeAccess: |
1021 - fromSame:
1022 - topology.kubernetes.io/region
1023 - topology.kubernetes.io/zone
1024 ----
1025 +
1026 You can specify multiple rules. The rules are additive, a node only need to match one rule to be assignable.
1027
1028 [[s-kubernetes-encryption]]
1029 ==== encryption
1030
1031 `encryption` is an optional parameter that determines whether to encrypt
1032 volumes. LINSTOR must be <<s-linstor-encrypted-volumes,configured for encryption>>
1033 for this to work properly.
1034
1035 [[s-kubernetes-nodelist]]
1036 ==== nodeList
1037
1038 `nodeList` is a list of nodes for volumes to be assigned to. This will assign
1039 the volume to each node and it will be replicated among all of them. This
1040 can also be used to select a single node by hostname, but it's more flexible to use
1041 <<s-kubernetes-replicasonsame,replicasOnSame>> to select a single node.
1042
1043 IMPORTANT: If you use this option, you must not use <<s-kubernetes-autoplace,autoPlace>>.
1044
1045 TIP: This option determines on which LINSTOR nodes the underlying storage for volumes
1046 will be provisioned and is orthogonal from which _kubelets_ these volumes will be
1047 accessible.
1048
1049 ==== clientList
1050
1051 `clientList` is a list of nodes for diskless volumes to be assigned to. Use in conjunction with <<s-kubernetes-nodelist>>.
1052
1053 [[s-kubernetes-replicasonsame]]
1054 ==== replicasOnSame
1055
1056 // These should link to the linstor documentation about node properties, but those
1057 // do not exist at the time of this commit.
1058 `replicasOnSame` is a list of `key` or `key=value` items used as autoplacement selection
1059 labels when <<s-kubernetes-autoplace,autoplace>> is used to determine where to
1060 provision storage. These labels correspond to LINSTOR node properties.
1061
1062 NOTE: The operator periodically synchronizes all labels from Kubernetes Nodes, so you can use them as keys for
1063 scheduling constraints.
1064
1065 Let's explore this behavior with examples assuming a LINSTOR cluster such that `node-a` is configured with the
1066 following auxiliary property `zone=z1` and `role=backups`, while `node-b` is configured with
1067 only `zone=z1`.
1068
1069 If we configure a _StorageClass_ with `autoPlace: "1"` and `replicasOnSame: "zone=z1 role=backups"`,
1070 then all volumes created from that _StorageClass_ will be provisioned on `node-a`,
1071 since that is the only node with all of the correct key=value pairs in the LINSTOR
1072 cluster. This is the most flexible way to select a single node for provisioning.
1073
1074 IMPORTANT: This guide assumes LINSTOR CSI version 0.10.0 or newer. All properties referenced in `replicasOnSame`
1075 and `replicasOnDifferent` are interpreted as auxiliary properties. If you are using an older version of LINSTOR CSI, you
1076 need to add the `Aux/` prefix to all property names. So `replicasOnSame: "zone=z1"` would be `replicasOnSame: "Aux/zone=z1"`
1077 Using `Aux/` manually will continue to work on newer LINSTOR CSI versions.
1078
1079 If we configure a _StorageClass_ with `autoPlace: "1"` and `replicasOnSame: "zone=z1"`,
1080 then volumes will be provisioned on either `node-a` or `node-b` as they both have
1081 the `zone=z1` aux prop.
1082
1083 If we configure a _StorageClass_ with `autoPlace: "2"` and `replicasOnSame: "zone=z1 role=backups"`,
1084 then provisioning will fail, as there are not two or more nodes that have
1085 the appropriate auxiliary properties.
1086
1087 If we configure a _StorageClass_ with `autoPlace: "2"` and `replicasOnSame: "zone=z1"`,
1088 then volumes will be provisioned on both `node-a` and `node-b` as they both have
1089 the `zone=z1` aux prop.
1090
1091 You can also use a property key without providing a value to ensure all replicas are placed on nodes with the same property value,
1092 with caring about the particular value. Assuming there are 4 nodes, `node-a1` and `node-a2` are configured with `zone=a`. `node-b1` and `node-b2`
1093 are configured with `zone=b`. Using `autoPlace: "2"` and `replicasOnSame: "zone"` will place on either `node-a1` and `node-a2` OR on `node-b1` and `node-b2`.
1094
1095 [[s-kubernetes-replicasondifferent]]
1096 ==== replicasOnDifferent
1097
1098 `replicasOnDifferent` takes a list of properties to consider, same as <<s-kubernetes-replicasonsame,replicasOnSame>>.
1099 There are two modes of using `replicasOnDifferent`:
1100
1101 * Preventing volume placement on specific nodes:
1102 +
1103 If a value is given for the property, the nodes which have that property-value pair assigned will be considered last.
1104 +
1105 Example: `replicasOnDifferent: "no-csi-volumes=true"` will place no volume on any node with property
1106 `no-csi-volumes=true` unless there are not enough other nodes to fulfill the `autoPlace` setting.
1107
1108 * Distribute volumes across nodes with different values for the same key:
1109 +
1110 If no property value is given, LINSTOR will place the volumes across nodes with different values for that property if
1111 possible.
1112 +
1113 Example: Assuming there are 4 nodes, `node-a1` and `node-a2` are configured with `zone=a`. `node-b1` and `node-b2`
1114 are configured with `zone=b`. Using a _StorageClass_ with `autoPlace: "2"` and `replicasOnDifferent: "zone"`,
1115 LINSTOR will create one replica on either `node-a1` or `node-a2` _and_ one replica on either `node-b1` or `node-b2`.
1116
1117 ==== disklessOnRemaining
1118
1119 Create a diskless resource on _all_ nodes that were not assigned a diskful resource.
1120
1121 ==== doNotPlaceWithRegex
1122
1123 Do not place the resource on a node which has a resource with a name matching the regex.
1124
1125 [[s-kubernetes-fsops]]
1126 ==== fsOpts
1127 `fsOpts` is an optional parameter that passes options to the volume's
1128 filesystem at creation time.
1129
1130 IMPORTANT: Please note these values are specific to your chosen
1131 <<s-kubernetes-filesystem, filesystem>>.
1132
1133 [[s-kubernetes-mountops]]
1134 ==== mountOpts
1135 `mountOpts` is an optional parameter that passes options to the volume's
1136 filesystem at mount time.
1137
1138 ==== postMountXfsOpts
1139
1140 Extra arguments to pass to `xfs_io`, which gets called before right before first use of the volume.
1141
1142 [[s-kubernetes-storage-class-properties]]
1143 ==== property.linstor.csi.linbit.com/*
1144
1145 Parameters starting with `property.linstor.csi.linbit.com/` are translated to Linstor properties that are set on the
1146 <<s-linstor-resource-groups,Resource Group>> associated with the StorageClass.
1147
1148 For example, to set `DrbdOptions/auto-quorum` to `disabled`, use:
1149
1150 ----
1151 property.linstor.csi.linbit.com/DrbdOptions/auto-quorum: disabled
1152 ----
1153
1154 The full list of options is available https://app.swaggerhub.com/apis-docs/Linstor/Linstor/1.7.0#/developers/resourceDefinitionModify[here]
1155
1156 ==== DrbdOptions/*: <x>
1157
1158 NOTE: This option is deprecated, use the more general <<s-kubernetes-storage-class-properties, `property.linstor.csi.linbit.com/*`>> form.
1159
1160 Advanced DRBD options to pass to LINSTOR. For example, to change the replication protocol, use
1161 `DrbdOptions/Net/protocol: "A"`.
1162
1163 [[s-kubernetes-snapshots]]
1164 === Snapshots
1165
1166 Creating <<s-linstor-snapshots, snapshots>> and creating new volumes from
1167 snapshots is done via the use of __VolumeSnapshot__s, __VolumeSnapshotClass__es,
1168 and __PVC__s.
1169
1170 [[s-kubernetes-add-snaphot-support]]
1171 ==== Adding snapshot support
1172
1173 LINSTOR supports the volume snapshot feature, which is configured in some, but not all Kubernetes distributions.
1174
1175 To check if your Kubernetes distribution supports snapshots out of the box, run the following command:
1176
1177 ----
1178 $ kubectl get --raw /apis/snapshot.storage.k8s.io/v1
1179 {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"snapshot.storage.k8s.io/v1"...
1180 $ # If your distribution does NOT support snapshots out of the box, you will get a different response:
1181 $ kubectl get --raw /apis/snapshot.storage.k8s.io/v1
1182 Error from server (NotFound): the server could not find the requested resource
1183 ----
1184
1185 :snapshot-controller-link: https://github.com/kubernetes-csi/external-snapshotter/
1186 :piraeus-charts-link: https://artifacthub.io/packages/helm/piraeus-charts/snapshot-controller
1187 :piraeus-org: https://github.com/piraeusdatastore
1188
1189 In case your Kubernetes distribution _does not_ support snapshots, you can manually add the
1190 {snapshot-controller-link}[required components] from the Kubernetes Storage SIG. For convenience, you can use
1191 {piraeus-charts-link}[Helm Charts] provided by the {piraeus-org}[Piraeus team] to add these components.
1192
1193 .Adding snapshot support using the Piraeus Charts
1194 ----
1195 $ kubectl create namespace snapshot-controller
1196 $ helm repo add piraeus-charts https://piraeus.io/helm-charts/
1197 $ helm install -n snapshot-controller snapshot-validation-webhook piraeus-charts/snapshot-validation-webhook
1198 $ helm install -n snapshot-controller snapshot-controller piraeus-charts/snapshot-controller --wait
1199 ----
1200
1201 [[s-kubernetes-use-snapshot]]
1202 ==== Using volume snapshots
1203 Then we can create our _VolumeSnapshotClass_:
1204
1205 .my-first-linstor-snapshot-class.yaml
1206 [source,yaml]
1207 ----
1208 apiVersion: snapshot.storage.k8s.io/v1
1209 kind: VolumeSnapshotClass
1210 metadata:
1211 name: my-first-linstor-snapshot-class
1212 driver: linstor.csi.linbit.com
1213 deletionPolicy: Delete
1214 ----
1215
1216 Create the _VolumeSnapshotClass_ with `kubectl`:
1217
1218 ----
1219 kubectl create -f my-first-linstor-snapshot-class.yaml
1220 ----
1221
1222 Now we will create a volume snapshot for the volume that we created above. This
1223 is done with a _VolumeSnapshot_:
1224
1225 .my-first-linstor-snapshot.yaml
1226 [source,yaml]
1227 ----
1228 apiVersion: snapshot.storage.k8s.io/v1
1229 kind: VolumeSnapshot
1230 metadata:
1231 name: my-first-linstor-snapshot
1232 spec:
1233 volumeSnapshotClassName: my-first-linstor-snapshot-class
1234 source:
1235 persistentVolumeClaimName: my-first-linstor-volume
1236 ----
1237
1238 Create the _VolumeSnapshot_ with `kubectl`:
1239
1240 ----
1241 kubectl create -f my-first-linstor-snapshot.yaml
1242 ----
1243
1244 You can check that the snapshot creation was successful
1245
1246 ----
1247 kubectl describe volumesnapshots.snapshot.storage.k8s.io my-first-linstor-snapshot
1248 ...
1249 Spec:
1250 Source:
1251 Persistent Volume Claim Name: my-first-linstor-snapshot
1252 Volume Snapshot Class Name: my-first-linstor-snapshot-class
1253 Status:
1254 Bound Volume Snapshot Content Name: snapcontent-b6072ab7-6ddf-482b-a4e3-693088136d2c
1255 Creation Time: 2020-06-04T13:02:28Z
1256 Ready To Use: true
1257 Restore Size: 500Mi
1258 ----
1259
1260 Finally, we'll create a new volume from the snapshot with a _PVC_.
1261
1262 .my-first-linstor-volume-from-snapshot.yaml
1263 [source,yaml]
1264 ----
1265 apiVersion: v1
1266 kind: PersistentVolumeClaim
1267 metadata:
1268 name: my-first-linstor-volume-from-snapshot
1269 spec:
1270 storageClassName: linstor-basic-storage-class
1271 dataSource:
1272 name: my-first-linstor-snapshot
1273 kind: VolumeSnapshot
1274 apiGroup: snapshot.storage.k8s.io
1275 accessModes:
1276 - ReadWriteOnce
1277 resources:
1278 requests:
1279 storage: 500Mi
1280 ----
1281
1282 Create the _PVC_ with `kubectl`:
1283
1284 ----
1285 kubectl create -f my-first-linstor-volume-from-snapshot.yaml
1286 ----
1287
1288
1289 [[s-kubernetes-volume-accessibility-and-locality]]
1290 === Volume Accessibility and Locality
1291 // This only covers DRBD volumes, section might change if linked docs are updated.
1292 LINSTOR volumes are typically accessible both locally and <<s-drbd_clients,over the network>>. The CSI driver will
1293 ensure that the volume is accessible on whatever node was selected for the consumer. The driver also provides options
1294 to ensure volume locality (the consumer is placed on the same node as the backing data) and restrict accessibility
1295 (only a subset of nodes can access the volume over the network).
1296
1297 Volume locality is achieved by setting `volumeBindingMode: WaitForFirstConsumer` in the storage class. This tell
1298 Kubernetes and the CSI driver to wait until the first consumer (Pod) referencing the PVC is scheduled. The CSI driver
1299 then provisions the volume with backing data on the same node as the consumer. In case a node without appropriate
1300 storage pool was selected, a replacement node in the set of accessible nodes is chosen (see below).
1301
1302 Volume accessibility is controlled by the
1303 <<s-kubernetes-params-allow-remote-volume-access,`allowRemoteVolumeAccess` parameter>>. Whenever the CSI plugin needs to
1304 place a volume, this parameter is consulted to get the set of "accessible" nodes. This means they can share volumes
1305 placed on them via the network. This information is also propagated to Kubernetes via label selectors on the PV.
1306
1307 ==== Volume Accessibility and Locality examples
1308
1309 The following example show common scenarios where you want to optimize volume accessibility and locality. It also
1310 includes examples of how to spread volume replicas across zones in a cluster.
1311
1312 ===== Single-zone homogeneous cluster
1313
1314 The cluster only spans a single zone, i.e. latency between nodes is low. The cluster is homogeneous, i.e. all nodes
1315 are configured similarly, all nodes have their own local storage pool.
1316
1317 .example-storage-class.yaml
1318 [source,yaml]
1319 ----
1320 apiVersion: storage.k8s.io/v1
1321 kind: StorageClass
1322 metadata:
1323 name: linstor-storage
1324 provisioner: linstor.csi.linbit.com
1325 volumeBindingMode: WaitForFirstConsumer <1>
1326 parameters:
1327 linstor.csi.linbit.com/storagePool: linstor-pool <2>
1328 linstor.csi.linbit.com/placementCount: "2" <3>
1329 linstor.csi.linbit.com/allowRemoteVolumeAccess: "true" <4>
1330 ----
1331
1332 <1> Enable late volume binding. This places one replica on the same node as the first consuming pod, if possible.
1333 <2> Set the storage pool(s) to use.
1334 <3> Ensure that the data is replicated, so that at least 2 nodes store the data.
1335 <4> Allow using the volume even on nodes without replica. Since all nodes are connected equally, performance impact
1336 should be manageable.
1337
1338 === Multi-zone homogenous cluster
1339
1340 As before, in our homogenous cluster all nodes are configured similarly with their own local storage pool. The cluster
1341 spans now multiple zones, with increased latency across nodes in different zones. To ensure low latency, we want
1342 to restrict access to the volume with a local replica to only those zones that do have a replica. At the same time,
1343 we want to spread our data across multiple zones.
1344
1345 .example-storage-class.yaml
1346 [source,yaml]
1347 ----
1348 apiVersion: storage.k8s.io/v1
1349 kind: StorageClass
1350 metadata:
1351 name: linstor-storage
1352 provisioner: linstor.csi.linbit.com
1353 volumeBindingMode: WaitForFirstConsumer <1>
1354 parameters:
1355 linstor.csi.linbit.com/storagePool: linstor-pool <2>
1356 linstor.csi.linbit.com/placementCount: "2" <3>
1357 linstor.csi.linbit.com/allowRemoteVolumeAccess: | <4>
1358 - fromSame:
1359 - topology.kubernetes.io/zone
1360 linstor.csi.linbit.com/replicasOnDifferent: topology.kubernetes.io/zone <5>
1361 ----
1362
1363 <1> Enable late volume binding. This places one replica on the same node as the first consuming pod, if possible.
1364 <2> Set the storage pool(s) to use.
1365 <3> Ensure that the data is replicated, so that at least 2 nodes store the data.
1366 <4> Allow using the volume on nodes in the same zone as a replica, under the assumption that zone internal networking
1367 is fast and low latency.
1368 <5> Spread the replicas across different zones.
1369
1370 === Multi-region cluster
1371
1372 Our cluster now spans multiple regions. We don't want to incur the latency penalty to replica our data across regions,
1373 we just want to replica in the same zone.
1374
1375 .example-storage-class.yaml
1376 [source,yaml]
1377 ----
1378 apiVersion: storage.k8s.io/v1
1379 kind: StorageClass
1380 metadata:
1381 name: linstor-storage
1382 provisioner: linstor.csi.linbit.com
1383 volumeBindingMode: WaitForFirstConsumer <1>
1384 parameters:
1385 linstor.csi.linbit.com/storagePool: linstor-pool <2>
1386 linstor.csi.linbit.com/placementCount: "2" <3>
1387 linstor.csi.linbit.com/allowRemoteVolumeAccess: | <4>
1388 - fromSame:
1389 - topology.kubernetes.io/zone
1390 linstor.csi.linbit.com/replicasOnSame: topology.kubernetes.io/region <5>
1391 ----
1392
1393 <1> Enable late volume binding. This places one replica on the same node as the first consuming pod, if possible.
1394 <2> Set the storage pool(s) to use.
1395 <3> Ensure that the data is replicated, so that at least 2 nodes store the data.
1396 <4> Allow using the volume on nodes in the same zone as a replica, under the assumption that zone internal networking
1397 is fast and low latency.
1398 <5> Restrict replicas to only a single region.
1399
1400 === Cluster with external storage
1401
1402 Our cluster is now only comprised of compute nodes without local storage. Any volume access has to occur via remote
1403 volume access.
1404
1405 .example-storage-class.yaml
1406 [source,yaml]
1407 ----
1408 apiVersion: storage.k8s.io/v1
1409 kind: StorageClass
1410 metadata:
1411 name: linstor-storage
1412 provisioner: linstor.csi.linbit.com
1413 parameters:
1414 linstor.csi.linbit.com/storagePool: linstor-pool <1>
1415 linstor.csi.linbit.com/placementCount: "1" <2>
1416 linstor.csi.linbit.com/allowRemoteVolumeAccess: "true" <3>
1417 ----
1418
1419 <1> Set the storage pool(s) to use.
1420 <2> Assuming we only have one storage host, we can only place a single volume without additional replicas
1421 <3> Our worker nodes need to be allowed to connect to the external storage host
1422
1423
1424 [[s-kubernetes-stork]]
1425 === Volume Locality Optimization using Stork
1426
1427 WARNING: The Stork integration is no longer actively maintained. Instead we recommend using
1428 `volumeBindingMode: WaitForFirstConsumer` in your storage class. For more details
1429 <<s-kubernetes-volume-accessibility-and-locality,see above>>.
1430
1431 Stork is a scheduler extender plugin for Kubernetes which allows a storage
1432 driver to give the Kubernetes scheduler hints about where to place a new pod
1433 so that it is optimally located for storage performance. You can learn more
1434 about the project on its https://portworx.com/stork-storage-orchestration-kubernetes/[GitHub page].
1435
1436 The next Stork release will include the LINSTOR driver by default.
1437 In the meantime, you can use a custom-built Stork container by LINBIT which includes a LINSTOR driver,
1438 https://hub.docker.com/repository/docker/linbit/stork[available on Docker Hub]
1439
1440 [[s-kubernetes-using-stork]]
1441 ==== Using Stork
1442
1443 By default, the chart will not install Stork. To install it and register a new scheduler called `stork`
1444 with Kubernetes, pass `--set stork.enabled=false` to the install command.
1445 The new scheduler can be used to place pods near to their volumes.
1446
1447 [source,yaml]
1448 ----
1449 apiVersion: v1
1450 kind: Pod
1451 metadata:
1452 name: busybox
1453 namespace: default
1454 spec:
1455 schedulerName: stork <1>
1456 containers:
1457 - name: busybox
1458 image: busybox
1459 command: ["tail", "-f", "/dev/null"]
1460 volumeMounts:
1461 - name: my-first-linstor-volume
1462 mountPath: /data
1463 ports:
1464 - containerPort: 80
1465 volumes:
1466 - name: my-first-linstor-volume
1467 persistentVolumeClaim:
1468 claimName: "test-volume"
1469 ----
1470
1471 <1> Add the name of the scheduler to your pod.
1472
1473 [[s-kubernetes-ha-controller]]
1474 === Fast workload fail over using the High Availability Controller
1475
1476 The LINSTOR High Availability Controller (HA Controller) will speed up the fail over process for stateful workloads using LINSTOR for storage.
1477 It is deployed by default, and can be scaled to multiple replicas:
1478
1479 [source]
1480 ----
1481 $ kubectl get pods -l app.kubernetes.io/name=linstor-op-ha-controller
1482 NAME READY STATUS RESTARTS AGE
1483 linstor-op-ha-controller-f496c5f77-fr76m 1/1 Running 0 89s
1484 linstor-op-ha-controller-f496c5f77-jnqtc 1/1 Running 0 89s
1485 linstor-op-ha-controller-f496c5f77-zcrqg 1/1 Running 0 89s
1486 ----
1487
1488 In the event of node failures, Kubernetes is very conservative in rescheduling stateful workloads. This means it can
1489 take more than 15 minutes for Pods to be moved from unreachable nodes. With the information available to DRBD and
1490 LINSTOR, this process can be sped up significantly.
1491
1492 The HA Controller enables fast fail over for:
1493
1494 * Pods using DRBD backed PersistentVolumes. The DRBD resources must make use of the quorum functionality
1495 LINSTOR will configure this automatically for volumes with 2 or more replicas in clusters with at least 3 nodes.
1496 * The workload does not use any external resources in a way that could lead to a conflicting state if two instances
1497 try to use the external resource at the same time. While DRBD can ensure that only one instance can have write access
1498 to the storage, it cannot provide the same guarantee for external resources.
1499 * The Pod is marked with the `linstor.csi.linbit.com/on-storage-lost: remove` label.
1500
1501 ==== Example
1502
1503 The following StatefulSet uses the HA Controller to manage fail over of a pod.
1504
1505 [source,yaml]
1506 ----
1507 apiVersion: apps/v1
1508 kind: StatefulSet
1509 metadata:
1510 name: my-stateful-app
1511 spec:
1512 serviceName: my-stateful-app
1513 selector:
1514 matchLabels:
1515 app.kubernetes.io/name: my-stateful-app
1516 template:
1517 metadata:
1518 labels:
1519 app.kubernetes.io/name: my-stateful-app
1520 linstor.csi.linbit.com/on-storage-lost: remove <1>
1521 ...
1522 ----
1523
1524 <1> The label is applied to Pod template, **not** the StatefulSet. The label was applied correctly, if your Pod appears
1525 in the output of `kubectl get pods -l linstor.csi.linbit.com/on-storage-lost=remove`.
1526
1527 Deploy the set and wait for the pod to start
1528
1529 [source]
1530 ----
1531 $ kubectl get pod -o wide
1532 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
1533 my-stateful-app-0 1/1 Running 0 5m 172.31.0.1 node01.ha.cluster <none> <none>
1534 ----
1535
1536 Then one of the nodes becomes unreachable. Shortly after, Kubernetes will mark the node as `NotReady`
1537
1538 [source]
1539 ----
1540 $ kubectl get nodes
1541 NAME STATUS ROLES AGE VERSION
1542 master01.ha.cluster Ready master 12d v1.19.4
1543 master02.ha.cluster Ready master 12d v1.19.4
1544 master03.ha.cluster Ready master 12d v1.19.4
1545 node01.ha.cluster NotReady compute 12d v1.19.4
1546 node02.ha.cluster Ready compute 12d v1.19.4
1547 node03.ha.cluster Ready compute 12d v1.19.4
1548 ----
1549
1550 After about 45 seconds, the Pod will be removed by the HA Controller and re-created by the StatefulSet
1551
1552 ----
1553 $ kubectl get pod -o wide
1554 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
1555 my-stateful-app-0 0/1 ContainerCreating 0 3s 172.31.0.1 node02.ha.cluster <none> <none>
1556 $ kubectl get events --sort-by=.metadata.creationTimestamp -w
1557 ...
1558 0s Warning ForceDeleted pod/my-stateful-app-0 pod deleted because a used volume is marked as failing
1559 0s Warning ForceDetached volumeattachment/csi-d2b994ff19d526ace7059a2d8dea45146552ed078d00ed843ac8a8433c1b5f6f volume detached because it is marked as failing
1560 ...
1561 ----