| 1 |
[[ch-kubernetes]] |
| 2 |
== LINSTOR Volumes in Kubernetes |
| 3 |
|
| 4 |
indexterm:[Kubernetes]This chapter describes the usage of LINSTOR in Kubernetes |
| 5 |
as managed by the operator and with volumes provisioned using the |
| 6 |
https://github.com/LINBIT/linstor-csi[LINSTOR CSI plugin]. |
| 7 |
|
| 8 |
This Chapter goes into great detail regarding all the install time |
| 9 |
options and various configurations possible with LINSTOR and |
| 10 |
Kubernetes. For those more interested in a "quick-start" for testing, |
| 11 |
or those looking for some examples for reference. We have some |
| 12 |
complete <<Helm Install Examples>> of a few common uses near the end |
| 13 |
of the chapter. |
| 14 |
|
| 15 |
[[s-kubernetes-overview]] |
| 16 |
=== Kubernetes Overview |
| 17 |
|
| 18 |
_Kubernetes_ is a container orchestrator. Kubernetes defines the behavior of |
| 19 |
containers and related services via declarative specifications. In this guide, |
| 20 |
we'll focus on using `kubectl` to manipulate `.yaml` files that define the |
| 21 |
specifications of Kubernetes objects. |
| 22 |
|
| 23 |
[[s-kubernetes-deploy]] |
| 24 |
=== Deploying LINSTOR on Kubernetes |
| 25 |
|
| 26 |
[[s-kubernetes-deploy-linstor-operator]] |
| 27 |
==== Deploying with the LINSTOR Operator |
| 28 |
|
| 29 |
LINBIT provides a LINSTOR operator to commercial support customers. |
| 30 |
The operator eases deployment of LINSTOR on Kubernetes by installing DRBD, |
| 31 |
managing Satellite and Controller pods, and other related functions. |
| 32 |
|
| 33 |
The operator itself is installed using a Helm v3 chart as follows: |
| 34 |
|
| 35 |
* Create a kubernetes secret containing your my.linbit.com credentials: |
| 36 |
+ |
| 37 |
---- |
| 38 |
kubectl create secret docker-registry drbdiocred --docker-server=drbd.io --docker-username=<YOUR_LOGIN> --docker-email=<YOUR_EMAIL> --docker-password=<YOUR_PASSWORD> |
| 39 |
---- |
| 40 |
+ |
| 41 |
The name of this secret must match the one specified in the Helm values, |
| 42 |
by default `drbdiocred`. |
| 43 |
|
| 44 |
* Configure the LINSTOR database backend. By default, the chart configures Etcd as database backend. Starting with |
| 45 |
1.7.0, the operator can also configure LINSTOR to use <<s-kubernetes-linstor-k8s-backend,Kubernetes as datastore>> |
| 46 |
directly. If you go the Etcd route, you should configure persistent storage for it: |
| 47 |
** Use an existing storage provisioner with a default `StorageClass`. |
| 48 |
** <<s-kubernetes-etcd-hostpath-persistence,Use `hostPath` volumes>>. |
| 49 |
** Disable persistence, **for basic testing only**. This can be done by adding |
| 50 |
`--set etcd.persistentVolume.enabled=false` to the `helm install` command below. |
| 51 |
|
| 52 |
* Read <<s-kubernetes-storage, the storage guide>> and configure a basic storage setup for LINSTOR |
| 53 |
|
| 54 |
* Read the <<s-kubernetes-securing-deployment,section on securing the deployment>> and configure as needed. |
| 55 |
|
| 56 |
* Select the appropriate kernel module injector using `--set` with the `helm install` command in the final step. |
| 57 |
|
| 58 |
** Choose the injector according to the distribution you are using. |
| 59 |
Select the latest version from one of `drbd9-rhel7`, `drbd9-rhel8`,... from http://drbd.io/ as appropriate. |
| 60 |
The drbd9-rhel8 image should also be used for RHCOS (OpenShift). For the SUSE CaaS Platform use the SLES injector |
| 61 |
that matches the base system of the CaaS Platform you are using (e.g., `drbd9-sles15sp1`). For example: |
| 62 |
+ |
| 63 |
---- |
| 64 |
operator.satelliteSet.kernelModuleInjectionImage=drbd.io/drbd9-rhel8:v9.0.30 |
| 65 |
---- |
| 66 |
|
| 67 |
** Only inject modules that are already present on the host machine. If a module is not found, it will be skipped. |
| 68 |
+ |
| 69 |
---- |
| 70 |
operator.satelliteSet.kernelModuleInjectionMode=DepsOnly |
| 71 |
---- |
| 72 |
|
| 73 |
** Disable kernel module injection if you are installing DRBD by other means. Deprecated by `DepsOnly` |
| 74 |
+ |
| 75 |
---- |
| 76 |
operator.satelliteSet.kernelModuleInjectionMode=None |
| 77 |
---- |
| 78 |
|
| 79 |
* Finally create a Helm deployment named `linstor-op` that will set up everything. |
| 80 |
+ |
| 81 |
---- |
| 82 |
helm repo add linstor https://charts.linstor.io |
| 83 |
helm install linstor-op linstor/linstor |
| 84 |
---- |
| 85 |
Further deployment customization is discussed in the <<s-kubernetes-advanced-deployments,advanced deployment section>> |
| 86 |
|
| 87 |
[[s-kubernetes-linstor-k8s-backend]] |
| 88 |
===== LINSTOR k8s backend |
| 89 |
|
| 90 |
WARNING: The Kubernetes backend for LINSTOR is still new, you might encounter issues, especially on LINSTOR version |
| 91 |
upgrade. Please read <<s-kubernetes-upgrade-k8s-backend, the upgrade guide>> for more information. |
| 92 |
|
| 93 |
Starting with LINSTOR 1.16.0, the Controller can use the Kubernetes API directly to persist its cluster state. To enable |
| 94 |
this backend, use the following override file during the chart installation: |
| 95 |
|
| 96 |
.k8s-backend.yaml |
| 97 |
[source,yaml] |
| 98 |
---- |
| 99 |
etcd: |
| 100 |
enabled: false |
| 101 |
operator: |
| 102 |
controller: |
| 103 |
dbConnectionURL: k8s |
| 104 |
---- |
| 105 |
|
| 106 |
NOTE: It is currently **NOT** possible to migrate from an existing cluster with Etcd backend to the Kubernetes |
| 107 |
backend. |
| 108 |
|
| 109 |
[[s-kubernetes-etcd-hostpath-persistence]] |
| 110 |
===== LINSTOR etcd `hostPath` persistence |
| 111 |
|
| 112 |
You can use the `pv-hostpath` Helm templates to create `hostPath` persistent |
| 113 |
volumes. Create as many PVs as needed to satisfy your configured etcd |
| 114 |
`replicas` (default 1). |
| 115 |
|
| 116 |
Create the `hostPath` persistent volumes, substituting cluster node |
| 117 |
names accordingly in the `nodes=` option: |
| 118 |
|
| 119 |
---- |
| 120 |
helm repo add linstor https://charts.linstor.io |
| 121 |
helm install linstor-etcd linstor/pv-hostpath |
| 122 |
---- |
| 123 |
|
| 124 |
By default, a PV is created on every `control-plane` node. You can manually select the storage nodes by |
| 125 |
passing `--set "nodes={<NODE0>,<NODE1>,<NODE2>}"` to the install command. |
| 126 |
|
| 127 |
[[s-kubernetes-existing-database]] |
| 128 |
===== Using an existing database |
| 129 |
|
| 130 |
LINSTOR can connect to an existing PostgreSQL, MariaDB or etcd database. For |
| 131 |
instance, for a PostgreSQL instance with the following configuration: |
| 132 |
|
| 133 |
---- |
| 134 |
POSTGRES_DB: postgresdb |
| 135 |
POSTGRES_USER: postgresadmin |
| 136 |
POSTGRES_PASSWORD: admin123 |
| 137 |
---- |
| 138 |
|
| 139 |
The Helm chart can be configured to use this database instead of deploying an |
| 140 |
etcd cluster by adding the following to the Helm install command: |
| 141 |
|
| 142 |
---- |
| 143 |
--set etcd.enabled=false --set "operator.controller.dbConnectionURL=jdbc:postgresql://postgres/postgresdb?user=postgresadmin&password=admin123" |
| 144 |
---- |
| 145 |
|
| 146 |
[[s-kubernetes-storage]] |
| 147 |
==== Configuring storage |
| 148 |
|
| 149 |
The LINSTOR operator can automate some basic storage set up for LINSTOR. |
| 150 |
|
| 151 |
===== Configuring storage pool creation |
| 152 |
|
| 153 |
The LINSTOR operator can be used to create LINSTOR storage pools. Creation is under control of the |
| 154 |
LinstorSatelliteSet resource: |
| 155 |
|
| 156 |
[source] |
| 157 |
---- |
| 158 |
$ kubectl get LinstorSatelliteSet.linstor.linbit.com linstor-op-ns -o yaml |
| 159 |
kind: LinstorSatelliteSet |
| 160 |
metadata: |
| 161 |
.. |
| 162 |
spec: |
| 163 |
.. |
| 164 |
storagePools: |
| 165 |
lvmPools: |
| 166 |
- name: lvm-thick |
| 167 |
volumeGroup: drbdpool |
| 168 |
lvmThinPools: |
| 169 |
- name: lvm-thin |
| 170 |
thinVolume: thinpool |
| 171 |
volumeGroup: "" |
| 172 |
zfsPools: |
| 173 |
- name: my-linstor-zpool |
| 174 |
zPool: for-linstor |
| 175 |
thin: true |
| 176 |
---- |
| 177 |
|
| 178 |
====== At install time |
| 179 |
|
| 180 |
At install time, by setting the value of `operator.satelliteSet.storagePools` when running helm install. |
| 181 |
|
| 182 |
First create a file with the storage configuration like: |
| 183 |
|
| 184 |
[source,yaml] |
| 185 |
---- |
| 186 |
operator: |
| 187 |
satelliteSet: |
| 188 |
storagePools: |
| 189 |
lvmPools: |
| 190 |
- name: lvm-thick |
| 191 |
volumeGroup: drbdpool |
| 192 |
---- |
| 193 |
|
| 194 |
This file can be passed to the helm installation like this: |
| 195 |
|
| 196 |
[source] |
| 197 |
---- |
| 198 |
helm install -f <file> linstor-op linstor/linstor |
| 199 |
---- |
| 200 |
|
| 201 |
====== After install |
| 202 |
|
| 203 |
On a cluster with the operator already configured (i.e. after `helm install`), |
| 204 |
you can edit the LinstorSatelliteSet configuration like this: |
| 205 |
|
| 206 |
[source] |
| 207 |
---- |
| 208 |
$ kubectl edit LinstorSatelliteSet.linstor.linbit.com <satellitesetname> |
| 209 |
---- |
| 210 |
|
| 211 |
The storage pool configuration can be updated like in the example above. |
| 212 |
|
| 213 |
===== Preparing physical devices |
| 214 |
|
| 215 |
By default, LINSTOR expects the referenced VolumeGroups, ThinPools and so on to be present. You can use the |
| 216 |
`devicePaths: []` option to let LINSTOR automatically prepare devices for the pool. Eligible for automatic configuration |
| 217 |
are block devices that: |
| 218 |
|
| 219 |
* Are a root device (no partition) |
| 220 |
* do not contain partition information |
| 221 |
* have more than 1 GiB |
| 222 |
|
| 223 |
To enable automatic configuration of devices, set the `devicePaths` key on `storagePools` entries: |
| 224 |
|
| 225 |
[source,yaml] |
| 226 |
---- |
| 227 |
storagePools: |
| 228 |
lvmPools: |
| 229 |
- name: lvm-thick |
| 230 |
volumeGroup: drbdpool |
| 231 |
devicePaths: |
| 232 |
- /dev/vdb |
| 233 |
lvmThinPools: |
| 234 |
- name: lvm-thin |
| 235 |
thinVolume: thinpool |
| 236 |
volumeGroup: linstor_thinpool |
| 237 |
devicePaths: |
| 238 |
- /dev/vdc |
| 239 |
- /dev/vdd |
| 240 |
---- |
| 241 |
|
| 242 |
Currently, this method supports creation of LVM and LVMTHIN storage pools. |
| 243 |
|
| 244 |
===== `lvmPools` configuration |
| 245 |
* `name` name of the LINSTOR storage pool.Required |
| 246 |
* `volumeGroup` name of the VG to create.Required |
| 247 |
* `devicePaths` devices to configure for this pool.Must be empty and >= 1GiB to be recognized.Optional |
| 248 |
* `raidLevel` LVM raid level.Optional |
| 249 |
* `vdo` Enable [VDO] (requires VDO tools in the satellite).Optional |
| 250 |
* `vdoLogicalSizeKib` Size of the created VG (expected to be bigger than the backing devices by using VDO).Optional |
| 251 |
* `vdoSlabSizeKib` Slab size for VDO. Optional |
| 252 |
|
| 253 |
[VDO]: https://www.redhat.com/en/blog/look-vdo-new-linux-compression-layer |
| 254 |
|
| 255 |
===== `lvmThinPools` configuration |
| 256 |
* `name` name of the LINSTOR storage pool. Required |
| 257 |
* `volumeGroup` VG to use for the thin pool. If you want to use `devicePaths`, you must set this to `""`. |
| 258 |
This is required because LINSTOR does not allow configuration of the VG name when preparing devices. |
| 259 |
* `thinVolume` name of the thinpool. Required |
| 260 |
* `devicePaths` devices to configure for this pool. Must be empty and >= 1GiB to be recognized. Optional |
| 261 |
* `raidLevel` LVM raid level. Optional |
| 262 |
|
| 263 |
NOTE: The volume group created by LINSTOR for LVMTHIN pools will always follow the scheme "linstor_$THINPOOL". |
| 264 |
|
| 265 |
===== `zfsPools` configuration |
| 266 |
* `name` name of the LINSTOR storage pool. Required |
| 267 |
* `zPool` name of the zpool to use. Must already be present on all machines. Required |
| 268 |
* `thin` `true` to use thin provisioning, `false` otherwise. Required |
| 269 |
|
| 270 |
===== Using `automaticStorageType` (DEPRECATED) |
| 271 |
|
| 272 |
_ALL_ eligible devices will be prepared according to the value of `operator.satelliteSet.automaticStorageType`, unless |
| 273 |
they are already prepared using the `storagePools` section. Devices are added to a storage pool based on the device |
| 274 |
name (i.e. all `/dev/nvme1` devices will be part of the pool `autopool-nvme1`) |
| 275 |
|
| 276 |
The possible values for `operator.satelliteSet.automaticStorageType`: |
| 277 |
|
| 278 |
* `None` no automatic set up (default) |
| 279 |
* `LVM` create a LVM (thick) storage pool |
| 280 |
* `LVMTHIN` create a LVM thin storage pool |
| 281 |
* `ZFS` create a ZFS based storage pool (**UNTESTED**) |
| 282 |
|
| 283 |
[[s-kubernetes-securing-deployment]] |
| 284 |
==== Securing deployment |
| 285 |
This section describes the different options for enabling security features available when |
| 286 |
using this operator. The following guides assume the operator is installed <<s-kubernetes-deploy-linstor-operator,using Helm>> |
| 287 |
|
| 288 |
===== Secure communication with an existing etcd instance |
| 289 |
|
| 290 |
Secure communication to an `etcd` instance can be enabled by providing a CA certificate to the operator in form of a |
| 291 |
kubernetes secret. The secret has to contain the key `ca.pem` with the PEM encoded CA certificate as value. |
| 292 |
|
| 293 |
The secret can then be passed to the controller by passing the following argument to `helm install` |
| 294 |
|
| 295 |
---- |
| 296 |
--set operator.controller.dbCertSecret=<secret name> |
| 297 |
---- |
| 298 |
|
| 299 |
====== Authentication with `etcd` using certificates |
| 300 |
|
| 301 |
If you want to use TLS certificates to authenticate with an `etcd` database, you need to set the following option on |
| 302 |
helm install: |
| 303 |
|
| 304 |
---- |
| 305 |
--set operator.controller.dbUseClientCert=true |
| 306 |
---- |
| 307 |
|
| 308 |
If this option is active, the secret specified in the above section must contain two additional keys: |
| 309 |
|
| 310 |
* `client.cert` PEM formatted certificate presented to `etcd` for authentication |
| 311 |
* `client.key` private key **in PKCS8 format**, matching the above client certificate. |
| 312 |
|
| 313 |
Keys can be converted into PKCS8 format using `openssl`: |
| 314 |
|
| 315 |
---- |
| 316 |
openssl pkcs8 -topk8 -nocrypt -in client-key.pem -out client-key.pkcs8 |
| 317 |
---- |
| 318 |
|
| 319 |
===== Configuring secure communication between LINSTOR components |
| 320 |
|
| 321 |
The default communication between LINSTOR components is not secured by TLS. If this is needed for your setup, |
| 322 |
follow these steps: |
| 323 |
|
| 324 |
* Create private keys in the java keystore format, one for the controller, one for all satellites: |
| 325 |
---- |
| 326 |
keytool -keyalg rsa -keysize 2048 -genkey -keystore satellite-keys.jks -storepass linstor -alias satellite -dname "CN=XX, OU=satellite, O=Example, L=XX, ST=XX, C=X" |
| 327 |
keytool -keyalg rsa -keysize 2048 -genkey -keystore control-keys.jks -storepass linstor -alias control -dname "CN=XX, OU=control, O=Example, L=XX, ST=XX, C=XX" |
| 328 |
---- |
| 329 |
* Create a trust store with the public keys that each component needs to trust: |
| 330 |
* Controller needs to trust the satellites |
| 331 |
* Nodes need to trust the controller |
| 332 |
+ |
| 333 |
---- |
| 334 |
keytool -importkeystore -srcstorepass linstor -deststorepass linstor -srckeystore control-keys.jks -destkeystore satellite-trust.jks |
| 335 |
keytool -importkeystore -srcstorepass linstor -deststorepass linstor -srckeystore satellite-keys.jks -destkeystore control-trust.jks |
| 336 |
---- |
| 337 |
|
| 338 |
* Create kubernetes secrets that can be passed to the controller and satellite pods |
| 339 |
+ |
| 340 |
---- |
| 341 |
kubectl create secret generic control-secret --from-file=keystore.jks=control-keys.jks --from-file=certificates.jks=control-trust.jks |
| 342 |
kubectl create secret generic satellite-secret --from-file=keystore.jks=satellite-keys.jks --from-file=certificates.jks=satellite-trust.jks |
| 343 |
---- |
| 344 |
* Pass the names of the created secrets to `helm install` |
| 345 |
+ |
| 346 |
---- |
| 347 |
--set operator.satelliteSet.sslSecret=satellite-secret --set operator.controller.sslSecret=control-secret |
| 348 |
---- |
| 349 |
|
| 350 |
IMPORTANT: It is currently **NOT** possible to change the keystore password. LINSTOR expects the passwords to be |
| 351 |
`linstor`. This is a current limitation of LINSTOR. |
| 352 |
|
| 353 |
===== Configuring secure communications for the LINSTOR API |
| 354 |
|
| 355 |
Various components need to talk to the LINSTOR controller via its REST interface. This interface can be |
| 356 |
secured via HTTPS, which automatically includes authentication. For HTTPS+authentication to work, each component |
| 357 |
needs access to: |
| 358 |
|
| 359 |
* A private key |
| 360 |
* A certificate based on the key |
| 361 |
* A trusted certificate, used to verify that other components are trustworthy |
| 362 |
|
| 363 |
The next sections will guide you through creating all required components. |
| 364 |
|
| 365 |
====== Creating the private keys |
| 366 |
|
| 367 |
Private keys can be created using java's keytool |
| 368 |
|
| 369 |
---- |
| 370 |
keytool -keyalg rsa -keysize 2048 -genkey -keystore controller.pkcs12 -storetype pkcs12 -storepass linstor -ext san=dns:linstor-op-cs.default.svc -dname "CN=XX, OU=controller, O=Example, L=XX, ST=XX, C=X" -validity 5000 |
| 371 |
keytool -keyalg rsa -keysize 2048 -genkey -keystore client.pkcs12 -storetype pkcs12 -storepass linstor -dname "CN=XX, OU=client, O=Example, L=XX, ST=XX, C=XX" -validity 5000 |
| 372 |
---- |
| 373 |
|
| 374 |
The clients need private keys and certificate in a different format, so we need to convert it |
| 375 |
|
| 376 |
---- |
| 377 |
openssl pkcs12 -in client.pkcs12 -passin pass:linstor -out client.cert -clcerts -nokeys |
| 378 |
openssl pkcs12 -in client.pkcs12 -passin pass:linstor -out client.key -nocerts -nodes |
| 379 |
---- |
| 380 |
|
| 381 |
NOTE: The alias specified for the controller key (i.e. `-ext san=dns:linstor-op-cs.default.svc`) has to exactly match the |
| 382 |
service name created by the operator. When using `helm`, this is always of the form `<release-name>-cs.<release-namespace>.svc`. |
| 383 |
|
| 384 |
IMPORTANT: It is currently NOT possible to change the keystore password. LINSTOR expects the passwords to be linstor. This is a current limitation of LINSTOR |
| 385 |
|
| 386 |
====== Create the trusted certificates |
| 387 |
|
| 388 |
For the controller to trust the clients, we can use the following command to create a truststore, importing the client certificate |
| 389 |
|
| 390 |
---- |
| 391 |
keytool -importkeystore -srcstorepass linstor -srckeystore client.pkcs12 -deststorepass linstor -deststoretype pkcs12 -destkeystore controller-trust.pkcs12 |
| 392 |
---- |
| 393 |
|
| 394 |
For the client, we have to convert the controller certificate into a different format |
| 395 |
|
| 396 |
---- |
| 397 |
openssl pkcs12 -in controller.pkcs12 -passin pass:linstor -out ca.pem -clcerts -nokeys |
| 398 |
---- |
| 399 |
|
| 400 |
====== Create Kubernetes secrets |
| 401 |
|
| 402 |
Now you can create secrets for the controller and for clients: |
| 403 |
|
| 404 |
---- |
| 405 |
kubectl create secret generic http-controller --from-file=keystore.jks=controller.pkcs12 --from-file=truststore.jks=controller-trust.pkcs12 |
| 406 |
kubectl create secret generic http-client --from-file=ca.pem=ca.pem --from-file=client.cert=client.cert --from-file=client.key=client.key |
| 407 |
---- |
| 408 |
|
| 409 |
The names of the secrets can be passed to `helm install` to configure all clients to use https. |
| 410 |
|
| 411 |
---- |
| 412 |
--set linstorHttpsControllerSecret=http-controller --set linstorHttpsClientSecret=http-client |
| 413 |
---- |
| 414 |
|
| 415 |
===== Automatically set the passphrase for LINSTOR |
| 416 |
|
| 417 |
LINSTOR needs to store confidential data to support encrypted information. This data is protected by a master |
| 418 |
passphrase. A passphrase is automatically generated on the first chart install. |
| 419 |
|
| 420 |
If you want to use a custom passphrase, store it in a secret: |
| 421 |
|
| 422 |
---- |
| 423 |
kubectl create secret generic linstor-pass --from-literal=MASTER_PASSPHRASE=<password> |
| 424 |
---- |
| 425 |
|
| 426 |
On install, add the following arguments to the helm command: |
| 427 |
|
| 428 |
---- |
| 429 |
--set operator.controller.luksSecret=linstor-pass |
| 430 |
---- |
| 431 |
|
| 432 |
===== Helm Install Examples |
| 433 |
|
| 434 |
All the below examples use the following `sp-values.yaml` file. Feel |
| 435 |
free to adjust this for your uses and environment. See <<Configuring storage pool creation>> |
| 436 |
for further details. |
| 437 |
|
| 438 |
----- |
| 439 |
operator: |
| 440 |
satelliteSet: |
| 441 |
storagePools: |
| 442 |
lvmThinPools: |
| 443 |
- name: lvm-thin |
| 444 |
thinVolume: thinpool |
| 445 |
volumeGroup: "" |
| 446 |
devicePaths: |
| 447 |
- /dev/sdb |
| 448 |
----- |
| 449 |
|
| 450 |
Default install. Please note this does not setup any persistence for |
| 451 |
the backing etcd key-value store. |
| 452 |
|
| 453 |
WARNING: This is not suggested for any use |
| 454 |
outside of testing. |
| 455 |
|
| 456 |
----- |
| 457 |
kubectl create secret docker-registry drbdiocred --docker-server=drbd.io --docker-username=<YOUR_LOGIN> --docker-password=<YOUR_PASSWORD> |
| 458 |
helm repo add linstor https://charts.linstor.io |
| 459 |
helm install linstor-op linstor/linstor |
| 460 |
----- |
| 461 |
|
| 462 |
Install with LINSTOR storage-pools defined at install via |
| 463 |
`sp-values.yaml`, persistent hostPath volumes, 3 etcd replicas, and by |
| 464 |
compiling the DRBD kernel modules for the host kernels. |
| 465 |
|
| 466 |
This should be adequate for most basic deployments. Please note that |
| 467 |
this deployment is not using the pre-compiled DRBD kernel modules just |
| 468 |
to make this command more portable. Using the pre-compiled binaries |
| 469 |
will make for a much faster install and deployment. Using the |
| 470 |
`Compile` option would not be suggested for use in a large Kubernetes clusters. |
| 471 |
|
| 472 |
----- |
| 473 |
kubectl create secret docker-registry drbdiocred --docker-server=drbd.io --docker-username=<YOUR_LOGIN> --docker-password=<YOUR_PASSWORD> |
| 474 |
helm repo add linstor https://charts.linstor.io |
| 475 |
helm install linstor-etcd linstor/pv-hostpath --set "nodes={<NODE0>,<NODE1>,<NODE2>}" |
| 476 |
helm install -f sp-values.yaml linstor-op linstor/linstor --set etcd.replicas=3 --set operator.satelliteSet.kernelModuleInjectionMode=Compile |
| 477 |
----- |
| 478 |
|
| 479 |
Install with LINSTOR storage-pools defined at install via |
| 480 |
`sp-values.yaml`, use an already created PostgreSQL DB (preferably |
| 481 |
clustered), instead of etcd, and use already compiled kernel modules for |
| 482 |
DRBD. Additionally, we'll disable the Stork scheduler in this example. |
| 483 |
|
| 484 |
The PostgreSQL database in this particular example is reachable via a |
| 485 |
service endpoint named `postgres`. PostgreSQL itself is configured with |
| 486 |
`POSTGRES_DB=postgresdb`, `POSTGRES_USER=postgresadmin`, and |
| 487 |
`POSTGRES_PASSWORD=admin123` |
| 488 |
|
| 489 |
----- |
| 490 |
kubectl create secret docker-registry drbdiocred --docker-server=drbd.io --docker-username=<YOUR_LOGIN> --docker-email=<YOUR_EMAIL> --docker-password=<YOUR_PASSWORD> |
| 491 |
helm repo add linstor https://charts.linstor.io |
| 492 |
helm install -f sp-values.yaml linstor-op linstor/linstor --set etcd.enabled=false --set "operator.controller.dbConnectionURL=jdbc:postgresql://postgres/postgresdb?user=postgresadmin&password=admin123" --set stork.enabled=false |
| 493 |
----- |
| 494 |
|
| 495 |
[[s-kubernetes-helm-terminate]] |
| 496 |
===== Terminating Helm deployment |
| 497 |
|
| 498 |
To protect the storage infrastructure of the cluster from accidentally deleting vital components, it is necessary to perform some manual steps before deleting a Helm deployment. |
| 499 |
|
| 500 |
1. Delete all volume claims managed by LINSTOR components. You can use the following command to get a list of volume claims managed by LINSTOR. After checking that none of the listed volumes still hold needed data, you can delete them using the generated kubectl delete command. |
| 501 |
+ |
| 502 |
---- |
| 503 |
$ kubectl get pvc --all-namespaces -o=jsonpath='{range .items[?(@.metadata.annotations.volume\.beta\.kubernetes\.io/storage-provisioner=="linstor.csi.linbit.com")]}kubectl delete pvc --namespace {.metadata.namespace} {.metadata.name}{"\n"}{end}' |
| 504 |
kubectl delete pvc --namespace default data-mysql-0 |
| 505 |
kubectl delete pvc --namespace default data-mysql-1 |
| 506 |
kubectl delete pvc --namespace default data-mysql-2 |
| 507 |
---- |
| 508 |
+ |
| 509 |
WARNING: These volumes, once deleted, cannot be recovered. |
| 510 |
|
| 511 |
2. Delete the LINSTOR controller and satellite resources. |
| 512 |
+ |
| 513 |
Deployment of LINSTOR satellite and controller is controlled by the LinstorSatelliteSet and LinstorController resources. You can delete the resources associated with your deployment using kubectl |
| 514 |
+ |
| 515 |
---- |
| 516 |
kubectl delete linstorcontroller <helm-deploy-name>-cs |
| 517 |
kubectl delete linstorsatelliteset <helm-deploy-name>-ns |
| 518 |
---- |
| 519 |
+ |
| 520 |
After a short wait, the controller and satellite pods should terminate. If they continue to run, you can check the above resources for errors (they are only removed after all associated pods terminate) |
| 521 |
|
| 522 |
3. Delete the Helm deployment. |
| 523 |
+ |
| 524 |
If you removed all PVCs and all LINSTOR pods have terminated, you can uninstall the helm deployment |
| 525 |
+ |
| 526 |
---- |
| 527 |
helm uninstall linstor-op |
| 528 |
---- |
| 529 |
+ |
| 530 |
NOTE: Due to the Helm's current policy, the Custom Resource Definitions named LinstorController and LinstorSatelliteSet will not be deleted by the command. |
| 531 |
More information regarding Helm's current position on CRDs can be found |
| 532 |
https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#method-1-let-helm-do-it-for-you[here]. |
| 533 |
|
| 534 |
[[s-kubernetes-advanced-deployments]] |
| 535 |
==== Advanced deployment options |
| 536 |
|
| 537 |
The helm charts provide a set of further customization options for advanced use cases. |
| 538 |
|
| 539 |
[source,yaml] |
| 540 |
---- |
| 541 |
global: |
| 542 |
imagePullPolicy: IfNotPresent # empty pull policy means k8s default is used ("always" if tag == ":latest", "ifnotpresent" else) <1> |
| 543 |
setSecurityContext: true # Force non-privileged containers to run as non-root users |
| 544 |
# Dependency charts |
| 545 |
etcd: |
| 546 |
enabled: true |
| 547 |
persistentVolume: |
| 548 |
enabled: true |
| 549 |
storage: 1Gi |
| 550 |
replicas: 1 # How many instances of etcd will be added to the initial cluster. <2> |
| 551 |
resources: {} # resource requirements for etcd containers <3> |
| 552 |
image: |
| 553 |
repository: gcr.io/etcd-development/etcd |
| 554 |
tag: v3.4.15 |
| 555 |
stork: |
| 556 |
enabled: false |
| 557 |
storkImage: docker.io/openstorage/stork:2.6.5 |
| 558 |
schedulerImage: k8s.gcr.io/kube-scheduler-amd64 |
| 559 |
schedulerTag: "" |
| 560 |
replicas: 1 <2> |
| 561 |
storkResources: {} # resources requirements for the stork plugin containers <3> |
| 562 |
schedulerResources: {} # resource requirements for the kube-scheduler containers <3> |
| 563 |
podsecuritycontext: {} |
| 564 |
csi: |
| 565 |
enabled: true |
| 566 |
pluginImage: "drbd.io/linstor-csi:v0.17.0" |
| 567 |
csiAttacherImage: k8s.gcr.io/sig-storage/csi-attacher:v3.3.0 |
| 568 |
csiLivenessProbeImage: k8s.gcr.io/sig-storage/livenessprobe:v2.5.0 |
| 569 |
csiNodeDriverRegistrarImage: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0 |
| 570 |
csiProvisionerImage: k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0 |
| 571 |
csiSnapshotterImage: k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1 |
| 572 |
csiResizerImage: k8s.gcr.io/sig-storage/csi-resizer:v1.3.0 |
| 573 |
controllerReplicas: 1 <2> |
| 574 |
nodeAffinity: {} <4> |
| 575 |
nodeTolerations: [] <4> |
| 576 |
controllerAffinity: {} <4> |
| 577 |
controllerTolerations: [] <4> |
| 578 |
enableTopology: true |
| 579 |
resources: {} <3> |
| 580 |
kubeletPath: /var/lib/kubelet <7> |
| 581 |
priorityClassName: "" |
| 582 |
drbdRepoCred: drbdiocred |
| 583 |
linstorHttpsControllerSecret: "" # <- name of secret containing linstor server certificates+key. |
| 584 |
linstorHttpsClientSecret: "" # <- name of secret containing linstor client certificates+key. |
| 585 |
controllerEndpoint: "" # <- override to the generated controller endpoint. use if controller is not deployed via operator |
| 586 |
psp: |
| 587 |
privilegedRole: "" |
| 588 |
unprivilegedRole: "" |
| 589 |
operator: |
| 590 |
replicas: 1 # <- number of replicas for the operator deployment <2> |
| 591 |
image: "drbd.io/linstor-operator:v1.7.0" |
| 592 |
affinity: {} <4> |
| 593 |
tolerations: [] <4> |
| 594 |
resources: {} <3> |
| 595 |
podsecuritycontext: {} |
| 596 |
controller: |
| 597 |
enabled: true |
| 598 |
controllerImage: "drbd.io/linstor-controller:v1.17.0" |
| 599 |
dbConnectionURL: "" |
| 600 |
luksSecret: "" |
| 601 |
dbCertSecret: "" |
| 602 |
dbUseClientCert: false |
| 603 |
sslSecret: "" |
| 604 |
affinity: {} <4> |
| 605 |
tolerations: <4> |
| 606 |
- key: node-role.kubernetes.io/master |
| 607 |
operator: "Exists" |
| 608 |
effect: "NoSchedule" |
| 609 |
resources: {} <3> |
| 610 |
replicas: 1 <2> |
| 611 |
additionalEnv: [] <5> |
| 612 |
additionalProperties: {} <6> |
| 613 |
satelliteSet: |
| 614 |
enabled: true |
| 615 |
satelliteImage: "drbd.io/linstor-satellite:v1.17.0" |
| 616 |
storagePools: {} |
| 617 |
sslSecret: "" |
| 618 |
automaticStorageType: None |
| 619 |
affinity: {} <4> |
| 620 |
tolerations: [] <4> |
| 621 |
resources: {} <3> |
| 622 |
monitoringImage: "drbd.io/drbd-reactor:v0.5.1" |
| 623 |
kernelModuleInjectionImage: "drbd.io/drbd9-rhel7:v9.1.4" |
| 624 |
kernelModuleInjectionMode: ShippedModules |
| 625 |
kernelModuleInjectionResources: {} <3> |
| 626 |
additionalEnv: [] <5> |
| 627 |
haController: |
| 628 |
enabled: true |
| 629 |
image: drbd.io/linstor-k8s-ha-controller:v0.2.1 |
| 630 |
affinity: {} <4> |
| 631 |
tolerations: [] <4> |
| 632 |
resources: {} <3> |
| 633 |
replicas: 1 <2> |
| 634 |
---- |
| 635 |
<1> Sets the pull policy for all images. |
| 636 |
<2> Controls the number of replicas for each component. |
| 637 |
<3> Set container resource requests and limits. See https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/[the kubernetes docs]. |
| 638 |
Most containers need a minimal amount of resources, except for: |
| 639 |
* `etcd.resources` See the https://etcd.io/docs/v3.4.0/op-guide/hardware/[etcd docs] |
| 640 |
* `operator.controller.resources` Around `700MiB` memory is required |
| 641 |
* `operater.satelliteSet.resources` Around `700MiB` memory is required |
| 642 |
* `operator.satelliteSet.kernelModuleInjectionResources` If kernel modules are compiled, 1GiB of memory is required. |
| 643 |
<4> Affinity and toleration determine where pods are scheduled on the cluster. See the |
| 644 |
https://kubernetes.io/docs/concepts/scheduling-eviction/[kubernetes docs on affinity and toleration]. |
| 645 |
This may be especially important for the `operator.satelliteSet` and `csi.node*` values. To schedule a pod using |
| 646 |
a LINSTOR persistent volume, the node requires a running LINSTOR satellite and LINSTOR CSI pod. |
| 647 |
<5> Sets additional environments variables to pass to the Linstor Controller and Satellites. Uses the same format as |
| 648 |
https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/[the `env` value of a container] |
| 649 |
<6> Sets additional properties on the Linstor Controller. Expects a simple mapping of `<property-key>: <value>`. |
| 650 |
<7> Kubelet expects every CSI plugin to mount volumes under a specific subdirectory of it's own state directory. By |
| 651 |
default, this state directory is `/var/lib/kubelet`. Some Kubernetes distributions use a different directory: |
| 652 |
* microk8s: `/var/snap/microk8s/common/var/lib/kubelet` |
| 653 |
|
| 654 |
[[s-kubernetes-ha-deployment]] |
| 655 |
===== High Availability Deployment |
| 656 |
To create a High Availability deployment of all components, take a look at the https://github.com/piraeusdatastore/piraeus-operator/blob/b00fd34/doc/scheduling.md[upstream guide] |
| 657 |
The default values are chosen so that scaling the components to multiple replicas ensures that the replicas are placed on different nodes. This ensures |
| 658 |
that a single node failures will not interrupt the service. |
| 659 |
|
| 660 |
[[s-kubernetes-monitoring]] |
| 661 |
===== Monitoring with Prometheus |
| 662 |
Starting with Linstor Operator v1.5.0, you can use https://prometheus.io/[Prometheus] to monitor Linstor components. |
| 663 |
The operator will set up monitoring containers along the existing components and make them available as a `Service`. |
| 664 |
|
| 665 |
If you use the https://prometheus-operator.dev/[Prometheus Operator], the Linstor Operator will also set up the `ServiceMonitor` |
| 666 |
instances. The metrics will automatically be collected by the Prometheus instance associated to the operator, assuming |
| 667 |
https://prometheus-operator.dev/docs/kube/monitoring-other-namespaces/[watching the Piraeus namespace is enabled]. |
| 668 |
|
| 669 |
To disable exporting of metrics, set `operator.satelliteSet.monitoringImage` to an empty value. |
| 670 |
|
| 671 |
====== Linstor Controller Monitoring |
| 672 |
|
| 673 |
The Linstor Controller exports cluster-wide metrics. Metrics are exported on the existing controller service, using the |
| 674 |
path https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-linstor-monitoring[`/metrics`]. |
| 675 |
|
| 676 |
====== DRBD Resource Monitoring |
| 677 |
|
| 678 |
All satellites are bundled with a secondary container that uses https://github.com/LINBIT/drbd-reactor/[`drbd-reactor`] |
| 679 |
to export metrics directly from DRBD. The metrics are available on port 9942, for convenience a headless service named |
| 680 |
`<linstorsatelliteset-name>-monitoring` is provided. |
| 681 |
|
| 682 |
If you want to disable the monitoring container, set `monitoringImage` to `""` in your LinstorSatelliteSet resource. |
| 683 |
|
| 684 |
[[s-kubernetes-deploy-external-controller]] |
| 685 |
==== Deploying with an external LINSTOR controller |
| 686 |
|
| 687 |
The operator can configure the satellites and CSI plugin to use an existing LINSTOR setup. This can be useful in cases |
| 688 |
where the storage infrastructure is separate from the Kubernetes cluster. Volumes can be provisioned in diskless mode |
| 689 |
on the Kubernetes nodes while the storage nodes will provide the backing disk storage. |
| 690 |
|
| 691 |
To skip the creation of a LINSTOR Controller deployment and configure the other components to use your existing LINSTOR |
| 692 |
Controller, use the following options when running `helm install`: |
| 693 |
|
| 694 |
* `operator.controller.enabled=false` This disables creation of the `LinstorController` resource |
| 695 |
* `operator.etcd.enabled=false` Since no LINSTOR Controller will run on Kubernetes, no database is required. |
| 696 |
* `controllerEndpoint=<url-of-linstor-controller>` The HTTP endpoint of the existing LINSTOR Controller. For example: `http://linstor.storage.cluster:3370/` |
| 697 |
|
| 698 |
After all pods are ready, you should see the Kubernetes cluster nodes as satellites in your LINSTOR setup. |
| 699 |
|
| 700 |
IMPORTANT: Your kubernetes nodes must be reachable using their IP by the controller and storage nodes. |
| 701 |
|
| 702 |
Create a storage class referencing an existing storage pool on your storage nodes. |
| 703 |
|
| 704 |
[source,yaml] |
| 705 |
---- |
| 706 |
apiVersion: storage.k8s.io/v1 |
| 707 |
kind: StorageClass |
| 708 |
metadata: |
| 709 |
name: linstor-on-k8s |
| 710 |
provisioner: linstor.csi.linbit.com |
| 711 |
parameters: |
| 712 |
autoPlace: "3" |
| 713 |
storagePool: existing-storage-pool |
| 714 |
resourceGroup: linstor-on-k8s |
| 715 |
---- |
| 716 |
|
| 717 |
You can provision new volumes by creating PVCs using your storage class. The volumes will first be placed only on nodes |
| 718 |
with the given storage pool, i.e. your storage infrastructure. Once you want to use the volume in a pod, LINSTOR CSI |
| 719 |
will create a diskless resource on the Kubernetes node and attach over the network to the diskful resource. |
| 720 |
|
| 721 |
[[s-kubernetes-deploy-piraeus-operator]] |
| 722 |
==== Deploying with the Piraeus Operator |
| 723 |
|
| 724 |
The community supported edition of the LINSTOR deployment in Kubernetes is |
| 725 |
called Piraeus. The Piraeus project provides |
| 726 |
https://github.com/piraeusdatastore/piraeus-operator[an operator] for |
| 727 |
deployment. |
| 728 |
|
| 729 |
[[s-kubernetes-linstor-interacting]] |
| 730 |
=== Interacting with LINSTOR in Kubernetes |
| 731 |
|
| 732 |
The Controller pod includes a LINSTOR Client, making it easy to interact directly with LINSTOR. |
| 733 |
For instance: |
| 734 |
|
| 735 |
---- |
| 736 |
kubectl exec deployment/linstor-op-cs-controller -- linstor storage-pool list |
| 737 |
---- |
| 738 |
|
| 739 |
For a convenient shortcut to the above command, download https://github.com/piraeusdatastore/kubectl-linstor/releases[`kubectl-linstor`] |
| 740 |
and install it alongside `kubectl`. Then you can use `kubectl linstor` to get access to the complete Linstor CLI |
| 741 |
|
| 742 |
---- |
| 743 |
$ kubectl linstor node list |
| 744 |
╭───────────────────────────────────────────────────────────────────────────────────────────────╮ |
| 745 |
┊ Node ┊ NodeType ┊ Addresses ┊ State ┊ |
| 746 |
╞═══════════════════════════════════════════════════════════════════════════════════════════════╡ |
| 747 |
┊ kube-node-01.test ┊ SATELLITE ┊ 10.43.224.26:3366 (PLAIN) ┊ Online ┊ |
| 748 |
┊ kube-node-02.test ┊ SATELLITE ┊ 10.43.224.27:3366 (PLAIN) ┊ Online ┊ |
| 749 |
┊ kube-node-03.test ┊ SATELLITE ┊ 10.43.224.28:3366 (PLAIN) ┊ Online ┊ |
| 750 |
┊ linstor-op-cs-controller-85b4f757f5-kxdvn ┊ CONTROLLER ┊ 172.24.116.114:3366 (PLAIN) ┊ Online ┊ |
| 751 |
╰───────────────────────────────────────────────────────────────────────────────────────────────╯ |
| 752 |
---- |
| 753 |
|
| 754 |
It also expands references to PVCs to the matching Linstor resource |
| 755 |
|
| 756 |
---- |
| 757 |
$ kubectl linstor resource list -r pvc:my-namespace/demo-pvc-1 --all |
| 758 |
pvc:my-namespace/demo-pvc-1 -> pvc-2f982fb4-bc05-4ee5-b15b-688b696c8526 |
| 759 |
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ |
| 760 |
┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊ |
| 761 |
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡ |
| 762 |
┊ pvc-2f982fb4-bc05-4ee5-b15b-688b696c8526 ┊ kube-node-01.test ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2021-02-05 09:16:09 ┊ |
| 763 |
┊ pvc-2f982fb4-bc05-4ee5-b15b-688b696c8526 ┊ kube-node-02.test ┊ 7000 ┊ Unused ┊ Ok ┊ TieBreaker ┊ 2021-02-05 09:16:08 ┊ |
| 764 |
┊ pvc-2f982fb4-bc05-4ee5-b15b-688b696c8526 ┊ kube-node-03.test ┊ 7000 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2021-02-05 09:16:09 ┊ |
| 765 |
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ |
| 766 |
---- |
| 767 |
|
| 768 |
It also expands references of the form `pod:[<namespace>/]<podname>` into a list resources in use by the pod. |
| 769 |
|
| 770 |
This should only be necessary for investigating problems and accessing advanced functionality. |
| 771 |
Regular operation such as creating volumes should be achieved via the |
| 772 |
<<s-kubernetes-basic-configuration-and-deployment,Kubernetes integration>>. |
| 773 |
|
| 774 |
[[s-kubernetes-basic-configuration-and-deployment]] |
| 775 |
=== Basic Configuration and Deployment |
| 776 |
|
| 777 |
Once all linstor-csi __Pod__s are up and running, we can provision volumes |
| 778 |
using the usual Kubernetes workflows. |
| 779 |
|
| 780 |
Configuring the behavior and properties of LINSTOR volumes deployed via Kubernetes |
| 781 |
is accomplished via the use of __StorageClass__es. |
| 782 |
|
| 783 |
IMPORTANT: the "resourceGroup" parameter is mandatory. Usually you want it to be unique and the same as the storage class name. |
| 784 |
|
| 785 |
Here below is the simplest practical _StorageClass_ that can be used to deploy volumes: |
| 786 |
|
| 787 |
.linstor-basic-sc.yaml |
| 788 |
[source,yaml] |
| 789 |
---- |
| 790 |
apiVersion: storage.k8s.io/v1 |
| 791 |
kind: StorageClass |
| 792 |
metadata: |
| 793 |
# The name used to identify this StorageClass. |
| 794 |
name: linstor-basic-storage-class |
| 795 |
# The name used to match this StorageClass with a provisioner. |
| 796 |
# linstor.csi.linbit.com is the name that the LINSTOR CSI plugin uses to identify itself |
| 797 |
provisioner: linstor.csi.linbit.com |
| 798 |
volumeBindingMode: WaitForFirstConsumer |
| 799 |
parameters: |
| 800 |
# LINSTOR will provision volumes from the drbdpool storage pool configured |
| 801 |
# On the satellite nodes in the LINSTOR cluster specified in the plugin's deployment |
| 802 |
storagePool: "drbdpool" |
| 803 |
resourceGroup: "linstor-basic-storage-class" |
| 804 |
# Setting a fstype is required for "fsGroup" permissions to work correctly. |
| 805 |
# Currently supported: xfs/ext4 |
| 806 |
csi.storage.k8s.io/fstype: xfs |
| 807 |
---- |
| 808 |
|
| 809 |
We can create the _StorageClass_ with the following command: |
| 810 |
|
| 811 |
---- |
| 812 |
kubectl create -f linstor-basic-sc.yaml |
| 813 |
---- |
| 814 |
|
| 815 |
Now that our _StorageClass_ is created, we can now create a _PersistentVolumeClaim_ |
| 816 |
which can be used to provision volumes known both to Kubernetes and LINSTOR: |
| 817 |
|
| 818 |
.my-first-linstor-volume-pvc.yaml |
| 819 |
[source,yaml] |
| 820 |
---- |
| 821 |
kind: PersistentVolumeClaim |
| 822 |
apiVersion: v1 |
| 823 |
metadata: |
| 824 |
name: my-first-linstor-volume |
| 825 |
spec: |
| 826 |
storageClassName: linstor-basic-storage-class |
| 827 |
accessModes: |
| 828 |
- ReadWriteOnce |
| 829 |
resources: |
| 830 |
requests: |
| 831 |
storage: 500Mi |
| 832 |
---- |
| 833 |
|
| 834 |
We can create the _PersistentVolumeClaim_ with the following command: |
| 835 |
|
| 836 |
---- |
| 837 |
kubectl create -f my-first-linstor-volume-pvc.yaml |
| 838 |
---- |
| 839 |
|
| 840 |
This will create a _PersistentVolumeClaim_, but no volume will be created just yet. |
| 841 |
The storage class we used specified `volumeBindingMode: WaitForFirstConsumer`, which |
| 842 |
means that the volume is only created once a workload starts using it. This ensures |
| 843 |
that the volume is placed on the same node as the workload. |
| 844 |
|
| 845 |
For our example, we create a simple Pod, which mounts or volume by referencing the |
| 846 |
_PersistentVolumeClaim_. |
| 847 |
.my-first-linstor-volume-pod.yaml |
| 848 |
[source,yaml] |
| 849 |
---- |
| 850 |
apiVersion: v1 |
| 851 |
kind: Pod |
| 852 |
metadata: |
| 853 |
name: fedora |
| 854 |
namespace: default |
| 855 |
spec: |
| 856 |
containers: |
| 857 |
- name: fedora |
| 858 |
image: fedora |
| 859 |
command: [/bin/bash] |
| 860 |
args: ["-c", "while true; do sleep 10; done"] |
| 861 |
volumeMounts: |
| 862 |
- name: my-first-linstor-volume |
| 863 |
mountPath: /data |
| 864 |
ports: |
| 865 |
- containerPort: 80 |
| 866 |
volumes: |
| 867 |
- name: my-first-linstor-volume |
| 868 |
persistentVolumeClaim: |
| 869 |
claimName: "my-first-linstor-volume" |
| 870 |
---- |
| 871 |
|
| 872 |
We can create the _Pod_ with the following command: |
| 873 |
|
| 874 |
---- |
| 875 |
kubectl create -f my-first-linstor-volume-pod.yaml |
| 876 |
---- |
| 877 |
|
| 878 |
Running `kubectl describe pod fedora` can be used to confirm that _Pod_ |
| 879 |
scheduling and volume attachment succeeded. Taking a look at the _PersistentVolumeClaim_, |
| 880 |
we can see that it is now bound to a volume. |
| 881 |
|
| 882 |
To remove a volume, please ensure that no pod is using it and then delete the |
| 883 |
_PersistentVolumeClaim_ via `kubectl`. For example, to remove the volume that we |
| 884 |
just made, run the following two commands, noting that the _Pod_ must be |
| 885 |
unscheduled before the _PersistentVolumeClaim_ will be removed: |
| 886 |
|
| 887 |
---- |
| 888 |
kubectl delete pod fedora # unschedule the pod. |
| 889 |
|
| 890 |
kubectl get pod -w # wait for pod to be unscheduled |
| 891 |
|
| 892 |
kubectl delete pvc my-first-linstor-volume # remove the PersistentVolumeClaim, the PersistentVolume, and the LINSTOR Volume. |
| 893 |
---- |
| 894 |
|
| 895 |
[[s-kubernetes-sc-parameters]] |
| 896 |
==== Available parameters in a StorageClass |
| 897 |
|
| 898 |
The following storage class contains all currently available parameters to configure the provisioned storage. |
| 899 |
|
| 900 |
NOTE: `linstor.csi.linbit.com/` is an optional, but recommended prefix for LINSTOR CSI specific parameters. |
| 901 |
|
| 902 |
[source,yaml] |
| 903 |
---- |
| 904 |
apiVersion: storage.k8s.io/v1 |
| 905 |
kind: StorageClass |
| 906 |
metadata: |
| 907 |
name: full-example |
| 908 |
provisioner: linstor.csi.linbit.com |
| 909 |
parameters: |
| 910 |
# CSI related parameters |
| 911 |
csi.storage.k8s.io/fstype: xfs |
| 912 |
# LINSTOR parameters |
| 913 |
linstor.csi.linbit.com/autoPlace: "2" |
| 914 |
linstor.csi.linbit.com/placementCount: "2" |
| 915 |
linstor.csi.linbit.com/resourceGroup: "full-example" |
| 916 |
linstor.csi.linbit.com/storagePool: "my-storage-pool" |
| 917 |
linstor.csi.linbit.com/disklessStoragePool: "DfltDisklessStorPool" |
| 918 |
linstor.csi.linbit.com/layerList: "drbd storage" |
| 919 |
linstor.csi.linbit.com/placementPolicy: "AutoPlaceTopology" |
| 920 |
linstor.csi.linbit.com/allowRemoteVolumeAccess: "true" |
| 921 |
linstor.csi.linbit.com/encryption: "true" |
| 922 |
linstor.csi.linbit.com/nodeList: "diskful-a diskful-b" |
| 923 |
linstor.csi.linbit.com/clientList: "diskless-a diskless-b" |
| 924 |
linstor.csi.linbit.com/replicasOnSame: "zone=a" |
| 925 |
linstor.csi.linbit.com/replicasOnDifferent: "rack" |
| 926 |
linstor.csi.linbit.com/disklessOnRemaining: "false" |
| 927 |
linstor.csi.linbit.com/doNotPlaceWithRegex: "tainted.*" |
| 928 |
linstor.csi.linbit.com/fsOpts: "-E nodiscard" |
| 929 |
linstor.csi.linbit.com/mountOpts: "noatime" |
| 930 |
linstor.csi.linbit.com/postMountXfsOpts: "extsize 2m" |
| 931 |
# Linstor properties |
| 932 |
property.linstor.csi.linbit.com/*: <x> |
| 933 |
# DRBD parameters |
| 934 |
DrbdOptions/*: <x> |
| 935 |
---- |
| 936 |
|
| 937 |
[[s-kubernetes-filesystem]] |
| 938 |
==== csi.storage.k8s.io/fstype |
| 939 |
|
| 940 |
Sets the file system type to create for `volumeMode: FileSystem` PVCs. Currently supported are: |
| 941 |
|
| 942 |
* `ext4` (default) |
| 943 |
* `xfs` |
| 944 |
|
| 945 |
[[s-kubernetes-autoplace]] |
| 946 |
==== autoPlace |
| 947 |
|
| 948 |
`autoPlace` is an integer that determines the amount of replicas a volume of |
| 949 |
this _StorageClass_ will have. For instance, `autoPlace: "3"` will produce |
| 950 |
volumes with three-way replication. If neither `autoPlace` nor `nodeList` are |
| 951 |
set, volumes will be <<s-autoplace-linstor,automatically placed>> on one node. |
| 952 |
|
| 953 |
IMPORTANT: If you use this option, you must not use <<s-kubernetes-nodelist,nodeList>>. |
| 954 |
|
| 955 |
IMPORTANT: You have to use quotes, otherwise Kubernetes will complain about a malformed _StorageClass_. |
| 956 |
|
| 957 |
TIP: This option (and all options which affect autoplacement behavior) modifies the |
| 958 |
number of LINSTOR nodes on which the underlying storage for volumes will be |
| 959 |
provisioned and is orthogonal to which _kubelets_ those volumes will be accessible |
| 960 |
from. |
| 961 |
|
| 962 |
==== placementCount |
| 963 |
|
| 964 |
`placementCount` is an alias for <<s-kubernetes-autoplace,`autoPlace`>> |
| 965 |
|
| 966 |
==== resourceGroup |
| 967 |
|
| 968 |
The <<s-linstor-resource-groups, LINSTOR Resource Group (RG)>> to associate with this StorageClass. If not set, |
| 969 |
a new RG will be created for each new PVC. |
| 970 |
|
| 971 |
[[s-kubernetes-storagepool]] |
| 972 |
==== storagePool |
| 973 |
|
| 974 |
`storagePool` is the name of the LINSTOR <<s-storage_pools,storage pool>> that |
| 975 |
will be used to provide storage to the newly-created volumes. |
| 976 |
|
| 977 |
CAUTION: Only nodes configured with this same _storage pool_ with be considered |
| 978 |
for <<s-kubernetes-autoplace,autoplacement>>. Likewise, for _StorageClasses_ using |
| 979 |
<<s-kubernetes-nodelist,nodeList>> all nodes specified in that list must have this |
| 980 |
_storage pool_ configured on them. |
| 981 |
|
| 982 |
[[s-kubernetes-disklessstoragepool]] |
| 983 |
==== disklessStoragePool |
| 984 |
|
| 985 |
`disklessStoragePool` is an optional parameter that only effects LINSTOR volumes |
| 986 |
assigned disklessly to _kubelets_ i.e., as clients. If you have a custom |
| 987 |
_diskless storage pool_ defined in LINSTOR, you'll specify that here. |
| 988 |
|
| 989 |
==== layerList |
| 990 |
|
| 991 |
A comma-separated list of layers to use for the created volumes. The available layers and their order are described |
| 992 |
towards the end of <<s-linstor-without-drbd, this section>>. Defaults to `drbd,storage` |
| 993 |
|
| 994 |
[[s-kubernetes-placementpolicy]] |
| 995 |
==== placementPolicy |
| 996 |
|
| 997 |
Select from one of the available volume schedulers: |
| 998 |
|
| 999 |
* `AutoPlaceTopology`, the default: Use topology information from Kubernetes together with user provided constraints (see <<s-kubernetes-replicasonsame>> and <<s-kubernetes-replicasondifferent>>). |
| 1000 |
* `AutoPlace` Use LINSTOR autoplace, influenced by <<s-kubernetes-replicasonsame>> and <<s-kubernetes-replicasondifferent>> |
| 1001 |
* `FollowTopology`: Use CSI Topology information to place at least one volume in each "preferred" zone. Only useable if CSI Topology is enabled. |
| 1002 |
* `Manual`: Use only the nodes listed in `nodeList` and `clientList`. |
| 1003 |
* `Balanced`: **EXPERIMENTAL** Place volumes across failure domains, using the least used storage pool on each selected node. |
| 1004 |
|
| 1005 |
[[s-kubernetes-params-allow-remote-volume-access]] |
| 1006 |
==== allowRemoteVolumeAccess |
| 1007 |
|
| 1008 |
Control on which nodes a volume is accessible. The value for this option can take two different forms: |
| 1009 |
|
| 1010 |
- A simple `"true"` or `"false"` allows access from all nodes, or only those nodes with diskfull resources. |
| 1011 |
|
| 1012 |
- Advanced rules, which allow more granular rules on which nodes can access the volume. |
| 1013 |
+ |
| 1014 |
The current implementation can grant access to the volume for nodes that share the same labels. For example, if you want |
| 1015 |
to allow access from all nodes in the same region and zone as a diskfull resource, you could use: |
| 1016 |
+ |
| 1017 |
[source,yaml] |
| 1018 |
---- |
| 1019 |
parameters: |
| 1020 |
linstor.csi.linbit.com/allowRemoteVolumeAccess: | |
| 1021 |
- fromSame: |
| 1022 |
- topology.kubernetes.io/region |
| 1023 |
- topology.kubernetes.io/zone |
| 1024 |
---- |
| 1025 |
+ |
| 1026 |
You can specify multiple rules. The rules are additive, a node only need to match one rule to be assignable. |
| 1027 |
|
| 1028 |
[[s-kubernetes-encryption]] |
| 1029 |
==== encryption |
| 1030 |
|
| 1031 |
`encryption` is an optional parameter that determines whether to encrypt |
| 1032 |
volumes. LINSTOR must be <<s-linstor-encrypted-volumes,configured for encryption>> |
| 1033 |
for this to work properly. |
| 1034 |
|
| 1035 |
[[s-kubernetes-nodelist]] |
| 1036 |
==== nodeList |
| 1037 |
|
| 1038 |
`nodeList` is a list of nodes for volumes to be assigned to. This will assign |
| 1039 |
the volume to each node and it will be replicated among all of them. This |
| 1040 |
can also be used to select a single node by hostname, but it's more flexible to use |
| 1041 |
<<s-kubernetes-replicasonsame,replicasOnSame>> to select a single node. |
| 1042 |
|
| 1043 |
IMPORTANT: If you use this option, you must not use <<s-kubernetes-autoplace,autoPlace>>. |
| 1044 |
|
| 1045 |
TIP: This option determines on which LINSTOR nodes the underlying storage for volumes |
| 1046 |
will be provisioned and is orthogonal from which _kubelets_ these volumes will be |
| 1047 |
accessible. |
| 1048 |
|
| 1049 |
==== clientList |
| 1050 |
|
| 1051 |
`clientList` is a list of nodes for diskless volumes to be assigned to. Use in conjunction with <<s-kubernetes-nodelist>>. |
| 1052 |
|
| 1053 |
[[s-kubernetes-replicasonsame]] |
| 1054 |
==== replicasOnSame |
| 1055 |
|
| 1056 |
// These should link to the linstor documentation about node properties, but those |
| 1057 |
// do not exist at the time of this commit. |
| 1058 |
`replicasOnSame` is a list of `key` or `key=value` items used as autoplacement selection |
| 1059 |
labels when <<s-kubernetes-autoplace,autoplace>> is used to determine where to |
| 1060 |
provision storage. These labels correspond to LINSTOR node properties. |
| 1061 |
|
| 1062 |
NOTE: The operator periodically synchronizes all labels from Kubernetes Nodes, so you can use them as keys for |
| 1063 |
scheduling constraints. |
| 1064 |
|
| 1065 |
Let's explore this behavior with examples assuming a LINSTOR cluster such that `node-a` is configured with the |
| 1066 |
following auxiliary property `zone=z1` and `role=backups`, while `node-b` is configured with |
| 1067 |
only `zone=z1`. |
| 1068 |
|
| 1069 |
If we configure a _StorageClass_ with `autoPlace: "1"` and `replicasOnSame: "zone=z1 role=backups"`, |
| 1070 |
then all volumes created from that _StorageClass_ will be provisioned on `node-a`, |
| 1071 |
since that is the only node with all of the correct key=value pairs in the LINSTOR |
| 1072 |
cluster. This is the most flexible way to select a single node for provisioning. |
| 1073 |
|
| 1074 |
IMPORTANT: This guide assumes LINSTOR CSI version 0.10.0 or newer. All properties referenced in `replicasOnSame` |
| 1075 |
and `replicasOnDifferent` are interpreted as auxiliary properties. If you are using an older version of LINSTOR CSI, you |
| 1076 |
need to add the `Aux/` prefix to all property names. So `replicasOnSame: "zone=z1"` would be `replicasOnSame: "Aux/zone=z1"` |
| 1077 |
Using `Aux/` manually will continue to work on newer LINSTOR CSI versions. |
| 1078 |
|
| 1079 |
If we configure a _StorageClass_ with `autoPlace: "1"` and `replicasOnSame: "zone=z1"`, |
| 1080 |
then volumes will be provisioned on either `node-a` or `node-b` as they both have |
| 1081 |
the `zone=z1` aux prop. |
| 1082 |
|
| 1083 |
If we configure a _StorageClass_ with `autoPlace: "2"` and `replicasOnSame: "zone=z1 role=backups"`, |
| 1084 |
then provisioning will fail, as there are not two or more nodes that have |
| 1085 |
the appropriate auxiliary properties. |
| 1086 |
|
| 1087 |
If we configure a _StorageClass_ with `autoPlace: "2"` and `replicasOnSame: "zone=z1"`, |
| 1088 |
then volumes will be provisioned on both `node-a` and `node-b` as they both have |
| 1089 |
the `zone=z1` aux prop. |
| 1090 |
|
| 1091 |
You can also use a property key without providing a value to ensure all replicas are placed on nodes with the same property value, |
| 1092 |
with caring about the particular value. Assuming there are 4 nodes, `node-a1` and `node-a2` are configured with `zone=a`. `node-b1` and `node-b2` |
| 1093 |
are configured with `zone=b`. Using `autoPlace: "2"` and `replicasOnSame: "zone"` will place on either `node-a1` and `node-a2` OR on `node-b1` and `node-b2`. |
| 1094 |
|
| 1095 |
[[s-kubernetes-replicasondifferent]] |
| 1096 |
==== replicasOnDifferent |
| 1097 |
|
| 1098 |
`replicasOnDifferent` takes a list of properties to consider, same as <<s-kubernetes-replicasonsame,replicasOnSame>>. |
| 1099 |
There are two modes of using `replicasOnDifferent`: |
| 1100 |
|
| 1101 |
* Preventing volume placement on specific nodes: |
| 1102 |
+ |
| 1103 |
If a value is given for the property, the nodes which have that property-value pair assigned will be considered last. |
| 1104 |
+ |
| 1105 |
Example: `replicasOnDifferent: "no-csi-volumes=true"` will place no volume on any node with property |
| 1106 |
`no-csi-volumes=true` unless there are not enough other nodes to fulfill the `autoPlace` setting. |
| 1107 |
|
| 1108 |
* Distribute volumes across nodes with different values for the same key: |
| 1109 |
+ |
| 1110 |
If no property value is given, LINSTOR will place the volumes across nodes with different values for that property if |
| 1111 |
possible. |
| 1112 |
+ |
| 1113 |
Example: Assuming there are 4 nodes, `node-a1` and `node-a2` are configured with `zone=a`. `node-b1` and `node-b2` |
| 1114 |
are configured with `zone=b`. Using a _StorageClass_ with `autoPlace: "2"` and `replicasOnDifferent: "zone"`, |
| 1115 |
LINSTOR will create one replica on either `node-a1` or `node-a2` _and_ one replica on either `node-b1` or `node-b2`. |
| 1116 |
|
| 1117 |
==== disklessOnRemaining |
| 1118 |
|
| 1119 |
Create a diskless resource on _all_ nodes that were not assigned a diskful resource. |
| 1120 |
|
| 1121 |
==== doNotPlaceWithRegex |
| 1122 |
|
| 1123 |
Do not place the resource on a node which has a resource with a name matching the regex. |
| 1124 |
|
| 1125 |
[[s-kubernetes-fsops]] |
| 1126 |
==== fsOpts |
| 1127 |
`fsOpts` is an optional parameter that passes options to the volume's |
| 1128 |
filesystem at creation time. |
| 1129 |
|
| 1130 |
IMPORTANT: Please note these values are specific to your chosen |
| 1131 |
<<s-kubernetes-filesystem, filesystem>>. |
| 1132 |
|
| 1133 |
[[s-kubernetes-mountops]] |
| 1134 |
==== mountOpts |
| 1135 |
`mountOpts` is an optional parameter that passes options to the volume's |
| 1136 |
filesystem at mount time. |
| 1137 |
|
| 1138 |
==== postMountXfsOpts |
| 1139 |
|
| 1140 |
Extra arguments to pass to `xfs_io`, which gets called before right before first use of the volume. |
| 1141 |
|
| 1142 |
[[s-kubernetes-storage-class-properties]] |
| 1143 |
==== property.linstor.csi.linbit.com/* |
| 1144 |
|
| 1145 |
Parameters starting with `property.linstor.csi.linbit.com/` are translated to Linstor properties that are set on the |
| 1146 |
<<s-linstor-resource-groups,Resource Group>> associated with the StorageClass. |
| 1147 |
|
| 1148 |
For example, to set `DrbdOptions/auto-quorum` to `disabled`, use: |
| 1149 |
|
| 1150 |
---- |
| 1151 |
property.linstor.csi.linbit.com/DrbdOptions/auto-quorum: disabled |
| 1152 |
---- |
| 1153 |
|
| 1154 |
The full list of options is available https://app.swaggerhub.com/apis-docs/Linstor/Linstor/1.7.0#/developers/resourceDefinitionModify[here] |
| 1155 |
|
| 1156 |
==== DrbdOptions/*: <x> |
| 1157 |
|
| 1158 |
NOTE: This option is deprecated, use the more general <<s-kubernetes-storage-class-properties, `property.linstor.csi.linbit.com/*`>> form. |
| 1159 |
|
| 1160 |
Advanced DRBD options to pass to LINSTOR. For example, to change the replication protocol, use |
| 1161 |
`DrbdOptions/Net/protocol: "A"`. |
| 1162 |
|
| 1163 |
[[s-kubernetes-snapshots]] |
| 1164 |
=== Snapshots |
| 1165 |
|
| 1166 |
Creating <<s-linstor-snapshots, snapshots>> and creating new volumes from |
| 1167 |
snapshots is done via the use of __VolumeSnapshot__s, __VolumeSnapshotClass__es, |
| 1168 |
and __PVC__s. |
| 1169 |
|
| 1170 |
[[s-kubernetes-add-snaphot-support]] |
| 1171 |
==== Adding snapshot support |
| 1172 |
|
| 1173 |
LINSTOR supports the volume snapshot feature, which is configured in some, but not all Kubernetes distributions. |
| 1174 |
|
| 1175 |
To check if your Kubernetes distribution supports snapshots out of the box, run the following command: |
| 1176 |
|
| 1177 |
---- |
| 1178 |
$ kubectl get --raw /apis/snapshot.storage.k8s.io/v1 |
| 1179 |
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"snapshot.storage.k8s.io/v1"... |
| 1180 |
$ # If your distribution does NOT support snapshots out of the box, you will get a different response: |
| 1181 |
$ kubectl get --raw /apis/snapshot.storage.k8s.io/v1 |
| 1182 |
Error from server (NotFound): the server could not find the requested resource |
| 1183 |
---- |
| 1184 |
|
| 1185 |
:snapshot-controller-link: https://github.com/kubernetes-csi/external-snapshotter/ |
| 1186 |
:piraeus-charts-link: https://artifacthub.io/packages/helm/piraeus-charts/snapshot-controller |
| 1187 |
:piraeus-org: https://github.com/piraeusdatastore |
| 1188 |
|
| 1189 |
In case your Kubernetes distribution _does not_ support snapshots, you can manually add the |
| 1190 |
{snapshot-controller-link}[required components] from the Kubernetes Storage SIG. For convenience, you can use |
| 1191 |
{piraeus-charts-link}[Helm Charts] provided by the {piraeus-org}[Piraeus team] to add these components. |
| 1192 |
|
| 1193 |
.Adding snapshot support using the Piraeus Charts |
| 1194 |
---- |
| 1195 |
$ kubectl create namespace snapshot-controller |
| 1196 |
$ helm repo add piraeus-charts https://piraeus.io/helm-charts/ |
| 1197 |
$ helm install -n snapshot-controller snapshot-validation-webhook piraeus-charts/snapshot-validation-webhook |
| 1198 |
$ helm install -n snapshot-controller snapshot-controller piraeus-charts/snapshot-controller --wait |
| 1199 |
---- |
| 1200 |
|
| 1201 |
[[s-kubernetes-use-snapshot]] |
| 1202 |
==== Using volume snapshots |
| 1203 |
Then we can create our _VolumeSnapshotClass_: |
| 1204 |
|
| 1205 |
.my-first-linstor-snapshot-class.yaml |
| 1206 |
[source,yaml] |
| 1207 |
---- |
| 1208 |
apiVersion: snapshot.storage.k8s.io/v1 |
| 1209 |
kind: VolumeSnapshotClass |
| 1210 |
metadata: |
| 1211 |
name: my-first-linstor-snapshot-class |
| 1212 |
driver: linstor.csi.linbit.com |
| 1213 |
deletionPolicy: Delete |
| 1214 |
---- |
| 1215 |
|
| 1216 |
Create the _VolumeSnapshotClass_ with `kubectl`: |
| 1217 |
|
| 1218 |
---- |
| 1219 |
kubectl create -f my-first-linstor-snapshot-class.yaml |
| 1220 |
---- |
| 1221 |
|
| 1222 |
Now we will create a volume snapshot for the volume that we created above. This |
| 1223 |
is done with a _VolumeSnapshot_: |
| 1224 |
|
| 1225 |
.my-first-linstor-snapshot.yaml |
| 1226 |
[source,yaml] |
| 1227 |
---- |
| 1228 |
apiVersion: snapshot.storage.k8s.io/v1 |
| 1229 |
kind: VolumeSnapshot |
| 1230 |
metadata: |
| 1231 |
name: my-first-linstor-snapshot |
| 1232 |
spec: |
| 1233 |
volumeSnapshotClassName: my-first-linstor-snapshot-class |
| 1234 |
source: |
| 1235 |
persistentVolumeClaimName: my-first-linstor-volume |
| 1236 |
---- |
| 1237 |
|
| 1238 |
Create the _VolumeSnapshot_ with `kubectl`: |
| 1239 |
|
| 1240 |
---- |
| 1241 |
kubectl create -f my-first-linstor-snapshot.yaml |
| 1242 |
---- |
| 1243 |
|
| 1244 |
You can check that the snapshot creation was successful |
| 1245 |
|
| 1246 |
---- |
| 1247 |
kubectl describe volumesnapshots.snapshot.storage.k8s.io my-first-linstor-snapshot |
| 1248 |
... |
| 1249 |
Spec: |
| 1250 |
Source: |
| 1251 |
Persistent Volume Claim Name: my-first-linstor-snapshot |
| 1252 |
Volume Snapshot Class Name: my-first-linstor-snapshot-class |
| 1253 |
Status: |
| 1254 |
Bound Volume Snapshot Content Name: snapcontent-b6072ab7-6ddf-482b-a4e3-693088136d2c |
| 1255 |
Creation Time: 2020-06-04T13:02:28Z |
| 1256 |
Ready To Use: true |
| 1257 |
Restore Size: 500Mi |
| 1258 |
---- |
| 1259 |
|
| 1260 |
Finally, we'll create a new volume from the snapshot with a _PVC_. |
| 1261 |
|
| 1262 |
.my-first-linstor-volume-from-snapshot.yaml |
| 1263 |
[source,yaml] |
| 1264 |
---- |
| 1265 |
apiVersion: v1 |
| 1266 |
kind: PersistentVolumeClaim |
| 1267 |
metadata: |
| 1268 |
name: my-first-linstor-volume-from-snapshot |
| 1269 |
spec: |
| 1270 |
storageClassName: linstor-basic-storage-class |
| 1271 |
dataSource: |
| 1272 |
name: my-first-linstor-snapshot |
| 1273 |
kind: VolumeSnapshot |
| 1274 |
apiGroup: snapshot.storage.k8s.io |
| 1275 |
accessModes: |
| 1276 |
- ReadWriteOnce |
| 1277 |
resources: |
| 1278 |
requests: |
| 1279 |
storage: 500Mi |
| 1280 |
---- |
| 1281 |
|
| 1282 |
Create the _PVC_ with `kubectl`: |
| 1283 |
|
| 1284 |
---- |
| 1285 |
kubectl create -f my-first-linstor-volume-from-snapshot.yaml |
| 1286 |
---- |
| 1287 |
|
| 1288 |
|
| 1289 |
[[s-kubernetes-volume-accessibility-and-locality]] |
| 1290 |
=== Volume Accessibility and Locality |
| 1291 |
// This only covers DRBD volumes, section might change if linked docs are updated. |
| 1292 |
LINSTOR volumes are typically accessible both locally and <<s-drbd_clients,over the network>>. The CSI driver will |
| 1293 |
ensure that the volume is accessible on whatever node was selected for the consumer. The driver also provides options |
| 1294 |
to ensure volume locality (the consumer is placed on the same node as the backing data) and restrict accessibility |
| 1295 |
(only a subset of nodes can access the volume over the network). |
| 1296 |
|
| 1297 |
Volume locality is achieved by setting `volumeBindingMode: WaitForFirstConsumer` in the storage class. This tell |
| 1298 |
Kubernetes and the CSI driver to wait until the first consumer (Pod) referencing the PVC is scheduled. The CSI driver |
| 1299 |
then provisions the volume with backing data on the same node as the consumer. In case a node without appropriate |
| 1300 |
storage pool was selected, a replacement node in the set of accessible nodes is chosen (see below). |
| 1301 |
|
| 1302 |
Volume accessibility is controlled by the |
| 1303 |
<<s-kubernetes-params-allow-remote-volume-access,`allowRemoteVolumeAccess` parameter>>. Whenever the CSI plugin needs to |
| 1304 |
place a volume, this parameter is consulted to get the set of "accessible" nodes. This means they can share volumes |
| 1305 |
placed on them via the network. This information is also propagated to Kubernetes via label selectors on the PV. |
| 1306 |
|
| 1307 |
==== Volume Accessibility and Locality examples |
| 1308 |
|
| 1309 |
The following example show common scenarios where you want to optimize volume accessibility and locality. It also |
| 1310 |
includes examples of how to spread volume replicas across zones in a cluster. |
| 1311 |
|
| 1312 |
===== Single-zone homogeneous cluster |
| 1313 |
|
| 1314 |
The cluster only spans a single zone, i.e. latency between nodes is low. The cluster is homogeneous, i.e. all nodes |
| 1315 |
are configured similarly, all nodes have their own local storage pool. |
| 1316 |
|
| 1317 |
.example-storage-class.yaml |
| 1318 |
[source,yaml] |
| 1319 |
---- |
| 1320 |
apiVersion: storage.k8s.io/v1 |
| 1321 |
kind: StorageClass |
| 1322 |
metadata: |
| 1323 |
name: linstor-storage |
| 1324 |
provisioner: linstor.csi.linbit.com |
| 1325 |
volumeBindingMode: WaitForFirstConsumer <1> |
| 1326 |
parameters: |
| 1327 |
linstor.csi.linbit.com/storagePool: linstor-pool <2> |
| 1328 |
linstor.csi.linbit.com/placementCount: "2" <3> |
| 1329 |
linstor.csi.linbit.com/allowRemoteVolumeAccess: "true" <4> |
| 1330 |
---- |
| 1331 |
|
| 1332 |
<1> Enable late volume binding. This places one replica on the same node as the first consuming pod, if possible. |
| 1333 |
<2> Set the storage pool(s) to use. |
| 1334 |
<3> Ensure that the data is replicated, so that at least 2 nodes store the data. |
| 1335 |
<4> Allow using the volume even on nodes without replica. Since all nodes are connected equally, performance impact |
| 1336 |
should be manageable. |
| 1337 |
|
| 1338 |
=== Multi-zone homogenous cluster |
| 1339 |
|
| 1340 |
As before, in our homogenous cluster all nodes are configured similarly with their own local storage pool. The cluster |
| 1341 |
spans now multiple zones, with increased latency across nodes in different zones. To ensure low latency, we want |
| 1342 |
to restrict access to the volume with a local replica to only those zones that do have a replica. At the same time, |
| 1343 |
we want to spread our data across multiple zones. |
| 1344 |
|
| 1345 |
.example-storage-class.yaml |
| 1346 |
[source,yaml] |
| 1347 |
---- |
| 1348 |
apiVersion: storage.k8s.io/v1 |
| 1349 |
kind: StorageClass |
| 1350 |
metadata: |
| 1351 |
name: linstor-storage |
| 1352 |
provisioner: linstor.csi.linbit.com |
| 1353 |
volumeBindingMode: WaitForFirstConsumer <1> |
| 1354 |
parameters: |
| 1355 |
linstor.csi.linbit.com/storagePool: linstor-pool <2> |
| 1356 |
linstor.csi.linbit.com/placementCount: "2" <3> |
| 1357 |
linstor.csi.linbit.com/allowRemoteVolumeAccess: | <4> |
| 1358 |
- fromSame: |
| 1359 |
- topology.kubernetes.io/zone |
| 1360 |
linstor.csi.linbit.com/replicasOnDifferent: topology.kubernetes.io/zone <5> |
| 1361 |
---- |
| 1362 |
|
| 1363 |
<1> Enable late volume binding. This places one replica on the same node as the first consuming pod, if possible. |
| 1364 |
<2> Set the storage pool(s) to use. |
| 1365 |
<3> Ensure that the data is replicated, so that at least 2 nodes store the data. |
| 1366 |
<4> Allow using the volume on nodes in the same zone as a replica, under the assumption that zone internal networking |
| 1367 |
is fast and low latency. |
| 1368 |
<5> Spread the replicas across different zones. |
| 1369 |
|
| 1370 |
=== Multi-region cluster |
| 1371 |
|
| 1372 |
Our cluster now spans multiple regions. We don't want to incur the latency penalty to replica our data across regions, |
| 1373 |
we just want to replica in the same zone. |
| 1374 |
|
| 1375 |
.example-storage-class.yaml |
| 1376 |
[source,yaml] |
| 1377 |
---- |
| 1378 |
apiVersion: storage.k8s.io/v1 |
| 1379 |
kind: StorageClass |
| 1380 |
metadata: |
| 1381 |
name: linstor-storage |
| 1382 |
provisioner: linstor.csi.linbit.com |
| 1383 |
volumeBindingMode: WaitForFirstConsumer <1> |
| 1384 |
parameters: |
| 1385 |
linstor.csi.linbit.com/storagePool: linstor-pool <2> |
| 1386 |
linstor.csi.linbit.com/placementCount: "2" <3> |
| 1387 |
linstor.csi.linbit.com/allowRemoteVolumeAccess: | <4> |
| 1388 |
- fromSame: |
| 1389 |
- topology.kubernetes.io/zone |
| 1390 |
linstor.csi.linbit.com/replicasOnSame: topology.kubernetes.io/region <5> |
| 1391 |
---- |
| 1392 |
|
| 1393 |
<1> Enable late volume binding. This places one replica on the same node as the first consuming pod, if possible. |
| 1394 |
<2> Set the storage pool(s) to use. |
| 1395 |
<3> Ensure that the data is replicated, so that at least 2 nodes store the data. |
| 1396 |
<4> Allow using the volume on nodes in the same zone as a replica, under the assumption that zone internal networking |
| 1397 |
is fast and low latency. |
| 1398 |
<5> Restrict replicas to only a single region. |
| 1399 |
|
| 1400 |
=== Cluster with external storage |
| 1401 |
|
| 1402 |
Our cluster is now only comprised of compute nodes without local storage. Any volume access has to occur via remote |
| 1403 |
volume access. |
| 1404 |
|
| 1405 |
.example-storage-class.yaml |
| 1406 |
[source,yaml] |
| 1407 |
---- |
| 1408 |
apiVersion: storage.k8s.io/v1 |
| 1409 |
kind: StorageClass |
| 1410 |
metadata: |
| 1411 |
name: linstor-storage |
| 1412 |
provisioner: linstor.csi.linbit.com |
| 1413 |
parameters: |
| 1414 |
linstor.csi.linbit.com/storagePool: linstor-pool <1> |
| 1415 |
linstor.csi.linbit.com/placementCount: "1" <2> |
| 1416 |
linstor.csi.linbit.com/allowRemoteVolumeAccess: "true" <3> |
| 1417 |
---- |
| 1418 |
|
| 1419 |
<1> Set the storage pool(s) to use. |
| 1420 |
<2> Assuming we only have one storage host, we can only place a single volume without additional replicas |
| 1421 |
<3> Our worker nodes need to be allowed to connect to the external storage host |
| 1422 |
|
| 1423 |
|
| 1424 |
[[s-kubernetes-stork]] |
| 1425 |
=== Volume Locality Optimization using Stork |
| 1426 |
|
| 1427 |
WARNING: The Stork integration is no longer actively maintained. Instead we recommend using |
| 1428 |
`volumeBindingMode: WaitForFirstConsumer` in your storage class. For more details |
| 1429 |
<<s-kubernetes-volume-accessibility-and-locality,see above>>. |
| 1430 |
|
| 1431 |
Stork is a scheduler extender plugin for Kubernetes which allows a storage |
| 1432 |
driver to give the Kubernetes scheduler hints about where to place a new pod |
| 1433 |
so that it is optimally located for storage performance. You can learn more |
| 1434 |
about the project on its https://portworx.com/stork-storage-orchestration-kubernetes/[GitHub page]. |
| 1435 |
|
| 1436 |
The next Stork release will include the LINSTOR driver by default. |
| 1437 |
In the meantime, you can use a custom-built Stork container by LINBIT which includes a LINSTOR driver, |
| 1438 |
https://hub.docker.com/repository/docker/linbit/stork[available on Docker Hub] |
| 1439 |
|
| 1440 |
[[s-kubernetes-using-stork]] |
| 1441 |
==== Using Stork |
| 1442 |
|
| 1443 |
By default, the chart will not install Stork. To install it and register a new scheduler called `stork` |
| 1444 |
with Kubernetes, pass `--set stork.enabled=false` to the install command. |
| 1445 |
The new scheduler can be used to place pods near to their volumes. |
| 1446 |
|
| 1447 |
[source,yaml] |
| 1448 |
---- |
| 1449 |
apiVersion: v1 |
| 1450 |
kind: Pod |
| 1451 |
metadata: |
| 1452 |
name: busybox |
| 1453 |
namespace: default |
| 1454 |
spec: |
| 1455 |
schedulerName: stork <1> |
| 1456 |
containers: |
| 1457 |
- name: busybox |
| 1458 |
image: busybox |
| 1459 |
command: ["tail", "-f", "/dev/null"] |
| 1460 |
volumeMounts: |
| 1461 |
- name: my-first-linstor-volume |
| 1462 |
mountPath: /data |
| 1463 |
ports: |
| 1464 |
- containerPort: 80 |
| 1465 |
volumes: |
| 1466 |
- name: my-first-linstor-volume |
| 1467 |
persistentVolumeClaim: |
| 1468 |
claimName: "test-volume" |
| 1469 |
---- |
| 1470 |
|
| 1471 |
<1> Add the name of the scheduler to your pod. |
| 1472 |
|
| 1473 |
[[s-kubernetes-ha-controller]] |
| 1474 |
=== Fast workload fail over using the High Availability Controller |
| 1475 |
|
| 1476 |
The LINSTOR High Availability Controller (HA Controller) will speed up the fail over process for stateful workloads using LINSTOR for storage. |
| 1477 |
It is deployed by default, and can be scaled to multiple replicas: |
| 1478 |
|
| 1479 |
[source] |
| 1480 |
---- |
| 1481 |
$ kubectl get pods -l app.kubernetes.io/name=linstor-op-ha-controller |
| 1482 |
NAME READY STATUS RESTARTS AGE |
| 1483 |
linstor-op-ha-controller-f496c5f77-fr76m 1/1 Running 0 89s |
| 1484 |
linstor-op-ha-controller-f496c5f77-jnqtc 1/1 Running 0 89s |
| 1485 |
linstor-op-ha-controller-f496c5f77-zcrqg 1/1 Running 0 89s |
| 1486 |
---- |
| 1487 |
|
| 1488 |
In the event of node failures, Kubernetes is very conservative in rescheduling stateful workloads. This means it can |
| 1489 |
take more than 15 minutes for Pods to be moved from unreachable nodes. With the information available to DRBD and |
| 1490 |
LINSTOR, this process can be sped up significantly. |
| 1491 |
|
| 1492 |
The HA Controller enables fast fail over for: |
| 1493 |
|
| 1494 |
* Pods using DRBD backed PersistentVolumes. The DRBD resources must make use of the quorum functionality |
| 1495 |
LINSTOR will configure this automatically for volumes with 2 or more replicas in clusters with at least 3 nodes. |
| 1496 |
* The workload does not use any external resources in a way that could lead to a conflicting state if two instances |
| 1497 |
try to use the external resource at the same time. While DRBD can ensure that only one instance can have write access |
| 1498 |
to the storage, it cannot provide the same guarantee for external resources. |
| 1499 |
* The Pod is marked with the `linstor.csi.linbit.com/on-storage-lost: remove` label. |
| 1500 |
|
| 1501 |
==== Example |
| 1502 |
|
| 1503 |
The following StatefulSet uses the HA Controller to manage fail over of a pod. |
| 1504 |
|
| 1505 |
[source,yaml] |
| 1506 |
---- |
| 1507 |
apiVersion: apps/v1 |
| 1508 |
kind: StatefulSet |
| 1509 |
metadata: |
| 1510 |
name: my-stateful-app |
| 1511 |
spec: |
| 1512 |
serviceName: my-stateful-app |
| 1513 |
selector: |
| 1514 |
matchLabels: |
| 1515 |
app.kubernetes.io/name: my-stateful-app |
| 1516 |
template: |
| 1517 |
metadata: |
| 1518 |
labels: |
| 1519 |
app.kubernetes.io/name: my-stateful-app |
| 1520 |
linstor.csi.linbit.com/on-storage-lost: remove <1> |
| 1521 |
... |
| 1522 |
---- |
| 1523 |
|
| 1524 |
<1> The label is applied to Pod template, **not** the StatefulSet. The label was applied correctly, if your Pod appears |
| 1525 |
in the output of `kubectl get pods -l linstor.csi.linbit.com/on-storage-lost=remove`. |
| 1526 |
|
| 1527 |
Deploy the set and wait for the pod to start |
| 1528 |
|
| 1529 |
[source] |
| 1530 |
---- |
| 1531 |
$ kubectl get pod -o wide |
| 1532 |
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES |
| 1533 |
my-stateful-app-0 1/1 Running 0 5m 172.31.0.1 node01.ha.cluster <none> <none> |
| 1534 |
---- |
| 1535 |
|
| 1536 |
Then one of the nodes becomes unreachable. Shortly after, Kubernetes will mark the node as `NotReady` |
| 1537 |
|
| 1538 |
[source] |
| 1539 |
---- |
| 1540 |
$ kubectl get nodes |
| 1541 |
NAME STATUS ROLES AGE VERSION |
| 1542 |
master01.ha.cluster Ready master 12d v1.19.4 |
| 1543 |
master02.ha.cluster Ready master 12d v1.19.4 |
| 1544 |
master03.ha.cluster Ready master 12d v1.19.4 |
| 1545 |
node01.ha.cluster NotReady compute 12d v1.19.4 |
| 1546 |
node02.ha.cluster Ready compute 12d v1.19.4 |
| 1547 |
node03.ha.cluster Ready compute 12d v1.19.4 |
| 1548 |
---- |
| 1549 |
|
| 1550 |
After about 45 seconds, the Pod will be removed by the HA Controller and re-created by the StatefulSet |
| 1551 |
|
| 1552 |
---- |
| 1553 |
$ kubectl get pod -o wide |
| 1554 |
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES |
| 1555 |
my-stateful-app-0 0/1 ContainerCreating 0 3s 172.31.0.1 node02.ha.cluster <none> <none> |
| 1556 |
$ kubectl get events --sort-by=.metadata.creationTimestamp -w |
| 1557 |
... |
| 1558 |
0s Warning ForceDeleted pod/my-stateful-app-0 pod deleted because a used volume is marked as failing |
| 1559 |
0s Warning ForceDetached volumeattachment/csi-d2b994ff19d526ace7059a2d8dea45146552ed078d00ed843ac8a8433c1b5f6f volume detached because it is marked as failing |
| 1560 |
... |
| 1561 |
---- |