Debian Codegraph @

UG9/en/administration-linstor.adoc

raw
1 [[s-administrative-tasks-setup]]
2 == Basic administrative tasks / Setup
3
4 LINSTOR is a configuration management system for storage on Linux systems.
5 It manages LVM logical volumes and/or ZFS ZVOLs on a cluster of nodes. It
6 leverages DRBD for replication between different nodes and to provide
7 block storage devices to users and applications. It manages snapshots,
8 encryption and caching of HDD backed data in SSDs via bcache.
9
10 // Troubleshooting for LINSTOR guide?
11 /////
12 This chapter outlines typical administrative tasks encountered during
13 day-to-day operations. It does not cover troubleshooting tasks, these
14 are covered in detail in <<ch-troubleshooting>>.
15 /////
16
17
18 [[s-concepts_and_terms]]
19 === Concepts and Terms
20 This section goes over core concepts and terms that you will need to familiarize yourself
21 with to understand how LINSTOR works and deploys storage. The section is laid out in a
22 "ground up" approach.
23
24
25 ==== Installable Components
26 ===== linstor-controller
27 A LINSTOR setup requires at least one active controller and one or more satellites.
28
29 // Once the chapter on making your controller HA is done we need to link that here
30
31 The _linstor-controller_ relies on a database that holds all configuration
32 information for the whole cluster. It makes all decisions that need to have a
33 view of the whole cluster. Multiple controllers can be used for LINSTOR but
34 only one can be active.
35
36 ===== linstor-satellite
37 The _linstor-satellite_ runs on each node where LINSTOR consumes local
38 storage or provides storage to services. It is stateless; it receives
39 all the information it needs from the controller. It runs programs
40 like `lvcreate` and `drbdadm`. It acts like a node agent.
41
42 ===== linstor-client
43 The _linstor-client_ is a command line utility that you use to issue
44 commands to the system and to investigate the status of the system.
45
46 ==== Objects
47 Objects are the end result which LINSTOR presents to the end-user or application,
48 such as; Kubernetes/OpenShift, a replicated block device (DRBD), NVMeOF target, etc.
49
50 ===== Node
51 Node's are a server or container that participate in a LINSTOR cluster. The _Node_
52 attribute defines:
53
54 * Determines which LINSTOR cluster the node participates in
55 * Sets the role of the node: Controller, Satellite, Auxiliary
56 * *NetInterface* objects define the node's connectivity
57
58 ===== NetInterface
59 As the name implies, this is how you define the interface/address of a node's network interface.
60
61 ===== Definitions
62 Definitions define attributes of an object, they can be thought of as
63 profile or template. Objects created will inherit the configuration
64 defined in the definitions. A definition must be defined prior to creating
65 the associated object. For example; you must create a _ResourceDefinition_
66 prior to creating the _Resource_
67
68 StoragePoolDefinition :::
69 * Defines the name of a storage pool
70
71 ResourceDefinition :::
72 Resource definitions define the following attributes of a resource:
73 * The name of a DRBD resource
74 * The TCP port for DRBD to use for the resource's connection
75
76 VolumeDefinition :::
77 Volume definitions define the following:
78
79 * A volume of a DRBD resource
80 * The size of the volume
81 * The volume number of the DRBD resource's volume
82 * The meta data properties of the volume
83 * The minor number to use for the DRBD device associated with the DRBD volume
84
85 ===== StoragePool
86 The _StoragePool_ identifies storage in the context of LINSTOR. It defines:
87
88 * The configuration of a storage pool on a specific node
89 * The storage back-end driver to use for the storage pool on the cluster node (LVM, ZFS, etc)
90 * The parameters and configuration to pass to the storage backed driver
91
92 ===== Resource
93 LINSTOR has now has expanded its capabilities to manage a broader set of storage technologies
94 outside of just DRBD. A _Resource_:
95
96 * Represents the placement of a DRBD resource, as defined within the _ResourceDefinition_
97 * Places a resource on a node in the cluster
98 * Defines the placement of a _ResourceDefinition_ on a node
99
100 ===== Volume
101 Volumes are a subset of a _Resource_. A _Resource_ could have multiple volumes, for example
102 you may wish to have your database stored on slower storage than your logs in your MySQL cluster.
103 By keeping the _volumes_ under a single _resource_ you are essentially creating a consistency group.
104 The _Volume_ attribute can define also define attributes on a more granular level.
105
106 [[s-broader_context]]
107 === Broader Context
108
109 While LINSTOR might be used to make the management of DRBD more
110 convenient, it is often integrated with software stacks higher up.
111 Such integration exist already for Kubernetes, OpenStack, OpenNebula
112 and Proxmox. Chapters specific to deploying LINSTOR in these
113 environments are included in this guide.
114
115 The southbound drivers used by LINSTOR are LVM, thinLVM and ZFS.
116
117 [[s-packages]]
118 === Packages
119
120 LINSTOR is packaged in both the .rpm and the .deb variants:
121
122 . _linstor-client_ contains the command line client program. It depends
123 on python which is usually already installed. In RHEL 8 systems you will need to symlink
124 python
125 . _linstor-controller_ and _linstor-satellite_ Both contain systemd unit files
126 for the services. They depend on Java runtime environment (JRE) version 1.8
127 (headless) or higher.
128
129 For further detail about these packages see the
130 <<Installable Components,Installable Components>> section above.
131
132 NOTE: If you have a support subscription to LINBIT, you will have access to
133 our certified binaries via our repositories.
134
135 [[s-installation]]
136 === Installation
137
138 IMPORTANT: If you want to use LINSTOR in containers skip this Topic and use the "Containers" section below for the installation.
139
140
141
142 [[s-admin-linstor-ubuntu_linux]]
143 ==== Ubuntu Linux
144
145 If you want to have the option of creating replicated storage using DRBD, you will need to install _drbd-dkms_ and _drbd-utils_. These packages will need to be installed on all nodes. You will also need
146 to choose a volume manager, either ZFS or LVM, in this instance we're using LVM.
147
148 --------------------------------------------
149 # apt install -y drbd-dkms drbd-utils lvm2
150 --------------------------------------------
151
152 Depending on whether your node is a LINSTOR controller, satellite, or both (Combined) will determine what packages are required on that node. For combined type nodes, we'll need both the controller and satellite LINSTOR package.
153
154 Combined node:
155
156 --------------------------------------------------------------------
157 # apt install linstor-controller linstor-satellite linstor-client
158 --------------------------------------------------------------------
159
160 That will make our remaining nodes our Satellites, so we'll need to install the following packages on them:
161
162 -------------------------------------------------
163 # apt install linstor-satellite linstor-client
164 -------------------------------------------------
165
166
167 [[s-admin-suse_linux_enterprise_server]]
168 ==== SUSE Linux Enterprise Server
169
170
171 SLES High Availability Extension (HAE) includes DRBD.
172
173 On SLES, DRBD is normally installed via the software installation component of YaST2. It comes bundled with the High Availability
174 package selection.
175
176 As we download DRBD's newest module we can check if the LVM-tools are up to date as well. User who prefer a command line install
177 may simply issue the following command to get the newest DRBD and LVM version:
178
179 ----------------------
180 # zypper install drbd lvm2
181 ----------------------
182
183
184
185 Depending on whether your node is a LINSTOR controller, satellite, or both (Combined) will determine what packages
186 are required on that node. For combined type nodes, we'll need both the controller and satellite LINSTOR package.
187
188 Combined node:
189
190 ------------------------------------------------------------------------
191 # zypper install linstor-controller linstor-satellite linstor-client
192 ------------------------------------------------------------------------
193
194 That will make our remaining nodes our Satellites, so we'll need to install the following packages on them:
195
196 ----------------------------------------------------
197 # zypper install linstor-satellite linstor-client
198 ----------------------------------------------------
199
200
201 [[s-admin-centos]]
202 ==== CentOS
203
204 CentOS has had DRBD 8 since release 5. For DRBD 9 you'll need to look at EPEL and similar sources.
205 Alternatively, if you have an active support contract with LINBIT you can utilize our RHEL 8 repositories.
206 DRBD can be installed using `yum`. We can also check for the newest version of the LVM-tools as well.
207
208 IMPORTANT: LINSTOR *requires* DRBD 9 if you wish to have replicated storage. This requires an external
209 repository to be configured, either LINBIT's or a 3rd parties.
210
211 -----------------------------
212 # yum install drbd kmod-drbd lvm2
213 -----------------------------
214
215 Depending on whether your node is a LINSTOR controller, satellite, or both (Combined) will determine what packages
216 are required on that node. For combined type nodes, we'll need both the controller and satellite LINSTOR package.
217
218 NOTE: On RHEL 8 systems you will need to install python2 for the linstor-client to work.
219
220 Combined node:
221
222 -------------------------------------------------------------------
223 # yum install linstor-controller linstor-satellite linstor-client
224 -------------------------------------------------------------------
225
226 That will make our remaining nodes our Satellites, so we'll need to install the following packages on them:
227
228 ------------------------------------------------
229 # yum install linstor-satellite linstor-client
230 ------------------------------------------------
231
232
233 [[s-upgrading]]
234 === Upgrading
235
236 LINSTOR doesn't support rolling upgrade, controller and satellites must have the same version, otherwise
237 the controller with discard the satellite with a `VERSION_MISMATCH`.
238 But this isn't a problem, as the satellite won't do any actions as long it isn't connected to a controller
239 and DRBD will not be disrupted by any means.
240
241 If you are using the embedded default H2 database and the linstor-controller package is upgraded an automatic
242 backup file of the database will be created in the default `/var/lib/linstor` directory.
243 This file is a good restore point if for any reason a linstor-controller database migration should fail,
244 than it is recommended to report the error to LINBIT and restore the old database file and downgrade to your previous
245 controller version.
246
247 If you use any external database or etcd, it is recommended to do a manually backup of your current database to have
248 a restore point.
249
250 So first upgrade the `linstor-controller`, `linstor-client` package on you controller host and restart the `linstor-controller`,
251 the controller should start and all of it's client should show `OFFLINE(VERSION_MISMATCH)`.
252 After that you can continue upgrading `linstor-satellite` on all satellite nodes and restart them, after a short reconnection time
253 they should all show `ONLINE` again and your upgrade is finished.
254
255
256 [[s-containers]]
257 === Containers
258
259 LINSTOR is also available as containers. The base images are available
260 in LINBIT's container registry, `drbd.io`.
261
262 In order to access the images, you first have to login to the
263 registry (reach out to [email protected] for credentials):
264
265 ----------------------------
266 # docker login drbd.io
267 ----------------------------
268
269 The containers available in this repository are:
270
271 * drbd.io/drbd9-rhel8
272 * drbd.io/drbd9-rhel7
273 * drbd.io/drbd9-sles15sp1
274 * drbd.io/drbd9-bionic
275 * drbd.io/drbd9-focal
276 * drbd.io/linstor-csi
277 * drbd.io/linstor-controller
278 * drbd.io/linstor-satellite
279 * drbd.io/linstor-client
280
281 An up to date list of available images with versions can be retrieved by opening http://drbd.io in your
282 browser. Make sure to access the host via "http", as the registry's images themselves are served via "https".
283
284 To load the kernel module, needed only for LINSTOR satellites, you'll need to
285 run a `drbd9-$dist` container in privileged mode. The kernel module containers
286 either retrieve an official LINBIT package from a customer repository, use
287 shipped packages, or they try to build the kernel modules from source. If you
288 intend to build from source, you need to have the according kernel headers
289 (e.g., `kernel-devel`) installed on the host. There are 4 ways to execute such
290 a module load container:
291
292 * Building from shipped source
293 * Using a shipped/pre-built kernel module
294 * Specifying a LINBIT node hash and a distribution.
295 * Bind-mounting an existing repository configuration.
296
297 Example building from shipped source (RHEL based):
298
299 ----------------------------
300 # docker run -it --rm --privileged -v /lib/modules:/lib/modules \
301 -v /usr/src:/usr/src:ro \
302 drbd.io/drbd9-rhel7
303 ----------------------------
304
305 Example using a module shipped with the container, which is enabled by *not* bind-mounting `/usr/src`:
306
307 ----------------------------
308 # docker run -it --rm --privileged -v /lib/modules:/lib/modules \
309 drbd.io/drbd9-rhel8
310 ----------------------------
311
312 Example using a hash and a distribution (rarely used):
313
314 ----------------------------
315 # docker run -it --rm --privileged -v /lib/modules:/lib/modules \
316 -e LB_DIST=rhel7.7 -e LB_HASH=ThisIsMyNodeHash \
317 drbd.io/drbd9-rhel7
318 ----------------------------
319
320 Example using an existing repo config (rarely used):
321
322 ----------------------------
323 # docker run -it --rm --privileged -v /lib/modules:/lib/modules \
324 -v /etc/yum.repos.d/linbit.repo:/etc/yum.repos.d/linbit.repo:ro \
325 drbd.io/drbd9-rhel7
326 ----------------------------
327
328 IMPORTANT: In both cases (hash + distribution, as well as bind-mounting a repo)
329 the hash or config has to be from a node that has a special property set. Feel
330 free to contact our support, and we set this property.
331
332 IMPORTANT: For now (i.e., pre DRBD 9 version "9.0.17"), you must use the containerized DRBD kernel module,
333 as opposed to loading a kernel module onto the host system. If you
334 intend to use the containers you should not install the DRBD kernel
335 module on your host systems. For DRBD version 9.0.17 or greater, you can install the kernel module as usual on
336 the host system, but you need to make sure to load the module with the `usermode_helper=disabled` parameter
337 (e.g., `modprobe drbd usermode_helper=disabled`).
338
339 Then run the LINSTOR satellite container, also privileged, as a daemon:
340
341 ----------------------------
342 # docker run -d --name=linstor-satellite --net=host -v /dev:/dev --privileged drbd.io/linstor-satellite
343 ----------------------------
344
345 NOTE: `net=host` is required for the containerized `drbd-utils` to be
346 able to communicate with the host-kernel via netlink.
347
348 To run the LINSTOR controller container as a daemon, mapping ports
349 `3370`, `3376` and `3377` on the host to the container:
350
351 ----------------------------
352 # docker run -d --name=linstor-controller -p 3370:3370 -p 3376:3376 -p 3377:3377 drbd.io/linstor-controller
353 ----------------------------
354
355 To interact with the containerized LINSTOR cluster, you can either use
356 a LINSTOR client installed on a system via packages, or via the
357 containerized LINSTOR client. To use the LINSTOR client container:
358
359 ----------------------------
360 # docker run -it --rm -e LS_CONTROLLERS=<controller-host-IP-address> drbd.io/linstor-client node list
361 ----------------------------
362
363 From this point you would use the LINSTOR client to initialize your
364 cluster and begin creating resources using the typical LINSTOR
365 patterns.
366
367 To stop and remove a daemonized container and image:
368
369 ----------------------------
370 # docker stop linstor-controller
371 # docker rm linstor-controller
372 ----------------------------
373
374 [[s-linstor-init-cluster]]
375 === Initializing your cluster
376 We assume that the following steps are accomplished on *all* cluster nodes:
377
378 . The DRBD9 kernel module is installed and loaded
379 . `drbd-utils` are installed
380 . `LVM` tools are installed
381 . `linstor-controller` and/or `linstor-satellite` its dependencies are installed
382 . The `linstor-client` is installed on the `linstor-controller` node
383
384 Start and enable the linstor-controller service on the host where it has been installed:
385
386 ----------------------------
387 # systemctl enable --now linstor-controller
388 ----------------------------
389
390 If you are sure the linstor-controller service gets automatically enabled on installation you can use the
391 following command as well:
392
393 ------------------------------------------
394 # systemctl start linstor-controller
395 ------------------------------------------
396
397 [[s-using_the_linstor_client]]
398 === Using the LINSTOR client
399 Whenever you run the LINSTOR command line client, it needs to know where your
400 linstor-controller runs. If you do not specify it, it will try to reach a locally
401 running linstor-controller listening on IP `127.0.0.1` port `3376`. Therefore we
402 will use the `linstor-client` on the same host as the `linstor-controller`.
403
404 IMPORTANT: The `linstor-satellite` requires ports 3366 and 3367. The `linstor-controller`
405 requires ports 3376 and 3377. Make sure you have these ports allowed on your firewall.
406
407 ----------------------------
408 # linstor node list
409 ----------------------------
410 should give you an empty list and not an error message.
411
412 You can use the `linstor` command on any other machine, but then you need
413 to tell the client how to find the linstor-controller. As shown, this can be
414 specified as a command line option, an environment variable, or in a global
415 file:
416
417 ----------------------------
418 # linstor --controllers=alice node list
419 # LS_CONTROLLERS=alice linstor node list
420 ----------------------------
421
422 Alternatively you can create the `/etc/linstor/linstor-client.conf`
423 file and populate it like below.
424
425
426 -----
427 [global]
428 controllers=alice
429 -----
430
431 If you have multiple linstor-controllers configured you can simply
432 specify them all in a comma separated list. The linstor-client will
433 simply try them in the order listed.
434
435
436 NOTE: The linstor-client commands can also be used in a much faster
437 and convenient way by only writing the starting letters of the parameters
438 e.g.: `linstor node list` -> `linstor n l`
439
440 [[s-adding_nodes_to_your_cluster]]
441 === Adding nodes to your cluster
442 The next step is to add nodes to your LINSTOR cluster.
443
444 ----------------------------
445 # linstor node create bravo 10.43.70.3
446 ----------------------------
447
448 If the IP is omitted, the client will try to resolve the given node-name as
449 host-name by itself.
450
451 Linstor will automatically detect the node's local `uname -n` which is
452 later used for the DRBD-resource.
453
454 When you use `linstor node list` you will see that the new node
455 is marked as offline. Now start and enable the linstor-satellite on that node
456 so that the service comes up on reboot as well:
457 ----------------------------
458 # systemctl enable --now linstor-satellite
459 ----------------------------
460
461 You can also use `systemctl start linstor-satellite`
462 if you are sure that the service is already enabled as default and comes up on
463 reboot.
464
465 About 10 seconds later you will see the status in `linstor node list`
466 becoming online. Of course the satellite process may be started before
467 the controller knows about the existence of the satellite node.
468
469 NOTE: In case the node which hosts your controller should also contribute
470 storage to the LINSTOR cluster, you have to add it as a node and start
471 the linstor-satellite as well.
472
473 If you want to have other services wait until the linstor-satellite had a chance
474 to create the necessary devices (i.e. after a boot), you can update the
475 corresponding .service file and change `Type=simple` to `Type=notify`.
476
477 This will cause the satellite to delay sending the `READY=1` message to systemd
478 until the controller connects, sends all required data to the satellite and the
479 satellite at least tried once to get the devices up and running.
480
481 [[s-storage_pools]]
482 === Storage pools
483
484 <<StoragePool,StoragePools>> identify storage in the context of LINSTOR.
485 To group storage pools from multiple nodes, simply use the same name
486 on each node.
487 For example, one valid approach is to give all SSDs one name and
488 all HDDs another.
489
490 On each host contributing storage, you need to create
491 either an LVM VG or a ZFS zPool. The VGs and zPools identified with one
492 LINSTOR storage pool name may have different VG or zPool names on the
493 hosts, but do yourself a favor and use the same VG or zPool name on all
494 nodes.
495
496 ----------------------------
497 # vgcreate vg_ssd /dev/nvme0n1 /dev/nvme1n1 [...]
498 ----------------------------
499
500 These then need to be registered with LINSTOR:
501
502 ----------------------------
503 # linstor storage-pool create lvm alpha pool_ssd vg_ssd
504 # linstor storage-pool create lvm bravo pool_ssd vg_ssd
505 ----------------------------
506
507 NOTE: The storage pool name and common metadata is referred to as a
508 _storage pool definition_.
509 The listed commands create a storage pool definition implicitly.
510 You can see that by using `linstor storage-pool-definition list`.
511 Creating storage pool definitions explicitly is possible but
512 not necessary.
513
514 To list your storage-pools you can use:
515
516 ------
517 # linstor storage-pool list
518 ------
519
520 or using the short version
521
522 -----
523 # linstor sp l
524 -----
525
526 ////
527 In case anything goes wrong with the storage pool's VG/zPool, e.g. the VG having been renamed or somehow
528 became invalid you can delete the storage pool in LINSTOR with the following command, given that only
529 resources with all their volumes in the so-called 'lost' storage pool are attached. This feature is available
530 since LINSTOR v0.9.13.
531
532 ------
533 # linstor storage-pool lost alpha pool_ssd
534 ------
535
536 or using the short version
537
538 -----
539 # linstor sp lo alpha pool_ssd
540 -----
541 ////
542
543 Should the deletion of the storage pool be prevented due to attached resources or snapshots with some of its
544 volumes in another still functional storage pool, hints will be given in the 'status' column of the
545 corresponding list-command (e.g. `linstor resource list`). After deleting the LINSTOR-objects in the lost storage pool
546 manually, the lost-command can be executed again to ensure a complete deletion of the storage pool and its
547 remaining objects.
548
549 [[s-a_storage_pool_per_backend_device]]
550 ==== A storage pool per backend device
551
552 In clusters where you have only one kind of storage and the capability
553 to hot-repair storage devices, you may choose a model where you create
554 one storage pool per physical backing device. The advantage of this
555 model is to confine failure domains to a single storage device.
556
557 [[s-physical-storage-command]]
558 ==== Physical storage command
559
560 Since linstor-server 1.5.2 and a recent linstor-client, LINSTOR can create LVM/ZFS pools on a satellite for you.
561 The linstor-client has the following commands to list possible disks and create storage pools, but such LVM/ZFS pools
562 are not managed by LINSTOR and there is no delete command, so such action must be done manually on the nodes.
563
564 -----
565 # linstor physical-storage list
566 -----
567
568 Will give you a list of available disks grouped by size and rotational(SSD/Magnetic Disk).
569
570 It will only show disks that pass the following filters:
571
572 * The device size must be greater than 1GiB
573 * The device is a root device (not having children) e.g.: /dev/vda, /dev/sda
574 * The device does not have any file-system or other `blkid` marker (`wipefs -a` might be needed)
575 * The device is no DRBD device
576
577
578 With the `create-device-pool` command you can create a LVM pool on a disk and also directly
579 add it as a storage-pool in LINSTOR.
580
581 ----
582 # linstor physical-storage create-device-pool --pool-name lv_my_pool LVMTHIN node_alpha /dev/vdc --storage-pool newpool
583 ----
584
585 If the `--storage-pool` option was provided, LINSTOR will create a storage-pool with the given name.
586
587 For more options and exact command usage please check the linstor-client help.
588
589 [[s-linstor-resource-groups]]
590 === Resource groups
591
592 A resource group is a parent object of a resource definition where all
593 property changes made on a resource group will be inherited by it's
594 resource definition children. The resource group also stores settings
595 for automatic placement rules and can spawn a resource definition
596 depending on the stored rules.
597
598 In simpler terms, resource groups are like templates that define
599 characteristics of resources created from them. Changes to these
600 pseudo templates will be applied to all resources that were created
601 from the resource group, retroactively.
602
603 TIP: Using resource groups to define how you'd like your resources
604 provisioned should be considered the de facto method for deploying
605 volumes provisioned by LINSTOR. Chapters that follow which describe
606 creating each _resource_ from a _resource-definition_ and
607 _volume-definition_ should only be used in special scenarios.
608
609 NOTE: Even if you choose not to create and use _resource-groups_ in
610 your LINSTOR cluster, all resources created from
611 _resource-definitions_ and _volume-definitions_ will exist in the
612 'DfltRscGrp' _resource-group_.
613
614 A simple pattern for deploying resources using resource groups would
615 look like this:
616
617 ----
618 # linstor resource-group create my_ssd_group --storage-pool pool_ssd --place-count 2
619 # linstor volume-group create my_ssd_group
620 # linstor resource-group spawn-resources my_ssd_group my_ssd_res 20G
621 ----
622
623 The commands above would result in a resource named 'my_ssd_res' with a
624 20GB volume replicated twice being automatically provisioned from nodes who
625 participate in the storage pool named 'pool_ssd'.
626
627 A more useful pattern could be to create a resource group with
628 settings you've determined are optimal for your use case. Perhaps
629 you have to run nightly online verifications of your volumes'
630 consistency, in that case, you could create a resource group with the
631 'verify-alg' of your choice already set so that resources spawned from
632 the group are pre-configured with 'verify-alg' set:
633
634 ----
635 # linstor resource-group create my_verify_group --storage-pool pool_ssd --place-count 2
636 # linstor resource-group drbd-options --verify-alg crc32c my_verify_group
637 # linstor volume-group create my_verify_group
638 # for i in {00..19}; do
639 linstor resource-group spawn-resources my_verify_group res$i 10G
640 done
641 ----
642
643 The commands above result in twenty 10GiB resources being created each
644 with the 'crc32c' 'verify-alg' pre-configured.
645
646 You can tune the settings of individual resources or volumes spawned
647 from resource groups by setting options on the respective
648 _resource-definition_ or _volume-definition_. For example, if 'res11'
649 from the example above is used by a very active database receiving
650 lots of small random writes, you might want to increase the
651 'al-extents' for that specific resource:
652
653 ----
654 # linstor resource-definition drbd-options --al-extents 6007 res11
655 ----
656
657 If you configure a setting in a _resource-definition_ that is already
658 configured on the _resource-group_ it was spawned from, the value set
659 in the _resource-definition_ will override the value set on the parent
660 _resource-group_. For example, if the same 'res11' was required to use
661 the slower but more secure 'sha256' hash algorithm in its
662 verifications, setting the 'verify-alg' on the _resource-definition_
663 for 'res11' would override the value set on the _resource-group_:
664
665 ----
666 # linstor resource-definition drbd-options --verify-alg sha256 res11
667 ----
668
669 TIP: A rule of thumb for the hierarchy in which settings are inherited
670 is the value "closer" to the resource or volume wins: _volume-definition_
671 settings take precedence over _volume-group_ settings, and
672 _resource-definition_ settings take precedence over _resource-group_
673 settings.
674
675
676 [[s-linstor-set-config]]
677 === Cluster configuration
678
679 [[s-available_storage_plugins]]
680 ==== Available storage plugins
681
682 indexterm:[linstor, storage plugins]
683
684 LINSTOR has the following supported storage plugins as of writing:
685
686 * Thick LVM
687
688 * Thin LVM with a single thin pool
689
690 * Thick ZFS
691
692 * Thin ZFS
693
694 [[s-linstor-new-volume]]
695 === Creating and deploying resources/volumes
696 In the following scenario we assume that the goal is to create a resource
697 'backups' with a size of '500 GB' that is replicated among three cluster nodes.
698
699 First, we create a new resource definition:
700
701 ----------------------------
702 # linstor resource-definition create backups
703 ----------------------------
704
705 Second, we create a new volume definition within that resource definition:
706
707 ----------------------------
708 # linstor volume-definition create backups 500G
709 ----------------------------
710
711 If you want to change the size of the volume-definition you can simply do that by:
712
713 -------
714 # linstor volume-definition set-size backups 0 100G
715 -------
716
717 The parameter `0` is the number of the volume in the resource `backups`. You have to provide this parameter
718 , because resources can have multiple volumes and they are identified by a so called volume-number. This number
719 can be found by listing the volume-definitions.
720
721 IMPORTANT: The size of a volume-definition can only be decreased if it has no resource. Despite
722 of that the size can be increased even with an deployed resource.
723
724 So far we have only created objects in LINSTOR's database, not a single LV was
725 created on the storage nodes. Now you have the choice of delegating the
726 task of placement to LINSTOR or doing it yourself.
727
728 [[s-manual_placement]]
729 ==== Manual placement
730
731 With the `resource create` command you may assign a resource definition
732 to named nodes explicitly.
733
734 ----------------------------
735 # linstor resource create alpha backups --storage-pool pool_hdd
736 # linstor resource create bravo backups --storage-pool pool_hdd
737 # linstor resource create charlie backups --storage-pool pool_hdd
738 ----------------------------
739
740 [[s-autoplace-linstor]]
741 ==== Autoplace
742
743 The value after autoplace tells LINSTOR how many replicas you want to have.
744 The storage-pool option should be obvious.
745 ----------------------------
746 # linstor resource create backups --auto-place 3 --storage-pool pool_hdd
747 ----------------------------
748 Maybe not so obvious is that you may omit the `--storage-pool` option, then
749 LINSTOR may select a storage pool on its own. The selection follows these rules:
750
751 * Ignore all nodes and storage pools the current user has no access to
752 * Ignore all diskless storage pools
753 * Ignore all storage pools not having enough free space
754
755 The remaining storage pools will be rated by different strategies.
756 LINSTOR has currently three strategies:
757
758 * `MaxFreeSpace`: This strategy maps the rating 1:1 to the remaining free
759 space of the storage pool. However, this strategy only considers the actually
760 allocated space (in case of thinly provisioned storage pool this might grow
761 with time without creating new resources)
762 * `MinReservedSpace`: Unlike the "MaxFreeSpace", this strategy considers the
763 reserved spaced. That is the space that a thin volume can grow to before reaching
764 its limit. The sum of reserved spaces might exceed the storage pools capacity,
765 which is as overprovisioning.
766 * `MinRscCount`: Simply the count of resources already deployed in a given
767 storage pool
768 * `MaxThroughput`: For this strategy, the storage pool's
769 `Autoplacer/MaxThroughput` property is the base of the score, or 0 if the property
770 is not present. Every Volume deployed in the given storage pool will subtract
771 its defined `sys/fs/blkio_throttle_read` and `sys/fs/blkio_throttle_write` property-
772 value from the storage pool's max throughput. The resulting score might be negative.
773
774 The scores of the strategies will be normalized, weighted and summed up, where
775 the scores of minimizing strategies will be converted first to allow an overall
776 maximization of the resulting score.
777
778 The weights of the strategies can be configured with
779
780 ----
781 linstor controller set-property Autoplacer/Weights/$name_of_the_strategy $weight
782 ----
783 whereas the strategy-names are listed above and the weight can be an arbitrary
784 decimal.
785
786 NOTE: To keep the behaviour of the autoplacer similar to the old one (due to
787 compatibility), all strategies have a default-weight of 0, except the `MaxFreeSpace`
788 which has a weight of 1.
789
790 NOTE: Neither 0 nor a negative score will prevent a storage pool from getting
791 selected, just making them to be considered later.
792
793 Finally LINSTOR tries to find the best matching group of storage pools meeting
794 all requirements. This step also considers other autoplacement restrictions
795 as `--replicas-on-same`, `--replicas-on-different` and others.
796
797 These two arguments, `--replicas-on-same` and `--replicas-on-different` expect the
798 name of a property within the `Aux/` namespace. The following example shows that the
799 client automatically prefixes the `testProperty` with the `Aux/` namespace.
800
801 ----
802 linstor resource-group create testRscGrp --replicas-on-same testProperty
803 SUCCESS:
804 Description:
805 New resource group 'testRscGrp' created.
806 Details:
807 Resource group 'testRscGrp' UUID is: 35e043cb-65ab-49ac-9920-ccbf48f7e27d
808
809 linstor resource-group list
810 +-----------------------------------------------------------------------------+
811 | ResourceGroup | SelectFilter | VlmNrs | Description |
812 |-============================================================================|
813 | DfltRscGrp | PlaceCount: 2 | | |
814 |-----------------------------------------------------------------------------|
815 | testRscGrp | PlaceCount: 2 | | |
816 | | ReplicasOnSame: ['Aux/testProperty'] | | |
817 +-----------------------------------------------------------------------------+
818 ----
819
820 NOTE: If everything went right the DRBD-resource has now been created by LINSTOR.
821 This can be checked by looking for the DRBD block device with the `lsblk`
822 command which should look like `drbd0000` or similar.
823
824
825 Now we should be able to mount the block device of our resource and start using
826 LINSTOR.
827
828 [[s-more-about-linstor]]
829 == Further LINSTOR tasks
830
831 [[s-linstor_ha]]
832 === LINSTOR high availability
833 By default a LINSTOR cluster consists of exactly one LINSTOR controller. Making LINSTOR highly-available
834 involves providing replicated storage for the controller database, multiple LINSTOR controllers where only one
835 is active at a time, and a service manager that takes care of mounting/unmounting the highly-available storage
836 and starting/stopping LINSTOR controllers.
837
838 ==== Highly-available Storage
839 For configuring the highly-available storage we use LINSTOR itself. This has the advantage that the storage is
840 under LINSTOR control and can for example be easily extended to new cluster nodes. Just create a new
841 resource with 200MB in size. This could look like this, you certainly need to adapt the storage pool name:
842
843 ----------------------------
844 # linstor resource-definition create linstor_db
845 # linstor resource-definition drbd-options --on-no-quorum=io-error linstor_db
846 # linstor resource-definition drbd-options --auto-promote=no linstor_db
847 # linstor volume-definition create linstor_db 200M
848 # linstor resource create linstor_db -s pool1 --auto-place 3
849 ----------------------------
850
851 From now on we assume the resource's name is "linstor_db". It is crucial that your
852 cluster qualifies for auto-quorum and uses the `io-error` policy (see Section <<s-linstor-auto-quorum>>), and
853 that `auto-promote` is disabled.
854
855 After the resource is created, it is time to move the LINSTOR DB to the new storage and to create a `systemd`
856 mount service. First we stop the current controller and disable it, as it will be managed by `drbd-reactor` later.
857
858 ----------------------------
859 # systemctl disable --now linstor-controller
860
861 # cat << EOF > /etc/systemd/system/var-lib-linstor.mount
862 [Unit]
863 Description=Filesystem for the LINSTOR controller
864
865 [Mount]
866 # you can use the minor like /dev/drbdX or the udev symlink
867 What=/dev/drbd/by-res/linstor_db/0
868 Where=/var/lib/linstor
869 EOF
870
871 # mv /var/lib/linstor{,.orig}
872 # mkdir /var/lib/linstor
873 # chattr +i /var/lib/linstor # only if on LINSTOR >= 1.14.0
874 # drbdadm primary linstor_db
875 # mkfs.ext4 /dev/drbd/by-res/linstor_db/0
876 # systemctl start var-lib-linstor.mount
877 # cp -r /var/lib/linstor.orig/* /var/lib/linstor
878 # systemctl start linstor-controller
879 ----------------------------
880
881 copy the `/etc/systemd/system/var-lib-linstor.mount` mount file to all the standby nodes for the linstor controller.
882 Again, do not `systemctl enable` any of these services, they get managed by `drbd-reactor`.
883
884 ==== Multiple LINSTOR controllers
885 The next step is to install LINSTOR controllers on all nodes that have access
886 to the `linstor_db` DRBD resource (as they need to mount the DRBD volume) and
887 which you want to become a possible LINSTOR controller. It is important that the controllers
888 are manged by `drbd-reactor`, so make sure the `linstor-controller.service` is
889 disabled on all nodes! To be sure, execute `systemctl disable linstor-controller`
890 on all cluster nodes and `systemctl stop linstor-controller` on all nodes except the one it is currently
891 running from the previous step. Also make sure to set `chattr +i /var/lib/linstor` on all potential controller
892 nodes if you use LINSTOR version equal or greater to 1.14.0.
893
894 ==== Managing the services
895 For starting and stopping the mount service and the linstor-controller service we use `drbd-reactor`. Install this
896 component on all nodes that could become a LINSTOR controller and edit their `/etc/drbd-reactor.d/linstor_db.toml` configuration
897 file. It should contain an enabled promoter plugin section like this:
898
899 ----------------------------
900 [[promoter]]
901 id = "linstor_db"
902 [promoter.resources.linstor_db]
903 start = ["var-lib-linstor.mount", "linstor-controller.service"]
904 ----------------------------
905
906 Depending on your requirements you might also want to set an `on-stop-failure` action and set `stop-services-on-exit`.
907
908 After that restart `drbd-reactor` and enable it on all the nodes you configured it.
909 ----------------------------
910 # systemctl restart drbd-reactor
911 # systemctl enable drbd-reactor
912 ----------------------------
913
914 Check that there are no warnings from drbd-reactor in the logs by running `systemctl status drbd-reactor`.
915 As there is already an active LINSTOR controller things will just stay the way they are.
916 Run `drbd-reactorctl status linstor_db` to check the health of the linstor_db target unit.
917
918 The last but never the less important step is to configure the LINSTOR
919 satellite services to not delete (and then regenerate) the resource file for the
920 LINSTOR controller DB at its startup. Do not edit the service files directly, but use `systemctl edit`. Edit
921 the service file on all nodes that could become a LINSTOR controller and that are also LINSTOR satellites.
922
923 --------------
924 # systemctl edit linstor-satellite
925 [Service]
926 Environment=LS_KEEP_RES=linstor_db
927 --------------
928
929 After this change you should execute `systemctl restart linstor-satellite` on all satellite nodes.
930
931 CAUTION: Be sure to configure your LINSTOR client for use with multiple controllers as described in
932 the section titled, <<s-using_the_linstor_client>> and make sure you also
933 configured your integration plugins (e.g., the Proxmox plugin) to be ready for
934 multiple LINSTOR controllers.
935
936 [[s-drbd_clients]]
937 === DRBD clients
938 By using the `--drbd-diskless` option instead of `--storage-pool` you can
939 have a permanently diskless DRBD device on a node. This means that
940 the resource will appear as block device and can be mounted to the
941 filesystem without an existing storage-device. The data of the
942 resource is accessed over the network on another nodes with the
943 same resource.
944
945 ----------------------------
946 # linstor resource create delta backups --drbd-diskless
947 ----------------------------
948
949 NOTE: The option `--diskless` was deprecated. Please use `--drbd-diskless`
950 or `--nvme-initiator` instead.
951
952 [[s-linstor-drbd-consistency-group-multiple-volumes]]
953 === LINSTOR - DRBD consistency group/multiple volumes
954
955 The so called consistency group is a feature from DRBD. It is mentioned in this user-guide, due to the
956 fact that one of LINSTOR's main functions is to manage storage-clusters with DRBD. Multiple volumes in
957 one resource are a consistency group.
958
959 This means that changes on different volumes from one resource are getting replicated in
960 the same chronological order on the other Satellites.
961
962 Therefore you don't have to worry about the timing if you have interdependent data on different volumes in a
963 resource.
964
965 To deploy more than one volume in a LINSTOR-resource you have to create two volume-definitions with the same name.
966
967 ----
968 # linstor volume-definition create backups 500G
969 # linstor volume-definition create backups 100G
970 ----
971
972
973 [[s-volumes_of_one_resource_to_different_storage_pools]]
974 === Volumes of one resource to different Storage-Pools
975 This can be achieved by setting the `StorPoolName` property to the volume
976 definitions before the resource is deployed to the nodes:
977
978
979 ----------------------------
980 # linstor resource-definition create backups
981 # linstor volume-definition create backups 500G
982 # linstor volume-definition create backups 100G
983 # linstor volume-definition set-property backups 0 StorPoolName pool_hdd
984 # linstor volume-definition set-property backups 1 StorPoolName pool_ssd
985 # linstor resource create alpha backups
986 # linstor resource create bravo backups
987 # linstor resource create charlie backups
988 ----------------------------
989
990 NOTE: Since the `volume-definition create` command is used without the `--vlmnr` option
991 LINSTOR assigned the volume numbers starting at 0. In the following two
992 lines the 0 and 1 refer to these automatically assigned volume numbers.
993
994 Here the 'resource create' commands do not need a `--storage-pool` option.
995 In this case LINSTOR uses a 'fallback' storage pool. Finding that
996 storage pool, LINSTOR queries the properties of the following objects
997 in the following order:
998
999 * Volume definition
1000 * Resource
1001 * Resource definition
1002 * Node
1003
1004 If none of those objects contain a `StorPoolName` property, the controller
1005 falls back to a hard-coded 'DfltStorPool' string as a storage pool.
1006
1007 This also means that if you forgot to define a storage pool prior deploying a
1008 resource, you will get an error message that LINSTOR could not find the
1009 storage pool named 'DfltStorPool'.
1010
1011
1012 [[s-linstor-without-drbd]]
1013 === LINSTOR without DRBD
1014
1015 LINSTOR can be used without DRBD as well. Without DRBD, LINSTOR is
1016 able to provision volumes from LVM and ZFS backed storage pools, and
1017 create those volumes on individual nodes in your LINSTOR cluster.
1018
1019 Currently LINSTOR supports the creation of LVM and ZFS
1020 volumes with the option of layering some combinations of LUKS,
1021 DRBD, and/or NVMe-oF/NVMe-TCP on top of those volumes.
1022
1023 For example, assume we have a Thin LVM backed storage pool defined in
1024 our LINSTOR cluster named, `thin-lvm`:
1025
1026 ----
1027 # linstor --no-utf8 storage-pool list
1028 +--------------------------------------------------------------+
1029 | StoragePool | Node | Driver | PoolName | ... |
1030 |--------------------------------------------------------------|
1031 | thin-lvm | linstor-a | LVM_THIN | drbdpool/thinpool | ... |
1032 | thin-lvm | linstor-b | LVM_THIN | drbdpool/thinpool | ... |
1033 | thin-lvm | linstor-c | LVM_THIN | drbdpool/thinpool | ... |
1034 | thin-lvm | linstor-d | LVM_THIN | drbdpool/thinpool | ... |
1035 +--------------------------------------------------------------+
1036 ----
1037
1038 We could use LINSTOR to create a Thin LVM on `linstor-d` that's 100GiB
1039 in size using the following commands:
1040
1041 ----
1042 # linstor resource-definition create rsc-1
1043 # linstor volume-definition create rsc-1 100GiB
1044 # linstor resource create --layer-list storage \
1045 --storage-pool thin-lvm linstor-d rsc-1
1046 ----
1047
1048 You should then see you have a new Thin LVM on `linstor-d`. You can
1049 extract the device path from LINSTOR by listing your linstor resources
1050 with the `--machine-readable` flag set:
1051
1052 ----
1053 # linstor --machine-readable resource list | grep device_path
1054 "device_path": "/dev/drbdpool/rsc-1_00000",
1055 ----
1056
1057 If you wanted to layer DRBD on top of this volume, which is the default
1058 `--layer-list` option in LINSTOR for ZFS or LVM backed volumes, you
1059 would use the following resource creation pattern instead:
1060
1061 ----
1062 # linstor resource-definition create rsc-1
1063 # linstor volume-definition create rsc-1 100GiB
1064 # linstor resource create --layer-list drbd,storage \
1065 --storage-pool thin-lvm linstor-d rsc-1
1066 ----
1067
1068 You would then see that you have a new Thin LVM backing a DRBD volume
1069 on `linstor-d`:
1070
1071 ----
1072 # linstor --machine-readable resource list | grep -e device_path -e backing_disk
1073 "device_path": "/dev/drbd1000",
1074 "backing_disk": "/dev/drbdpool/rsc-1_00000",
1075 ----
1076
1077 The following table shows which layer can be followed by which child-layer:
1078
1079 [cols=">1,<5"]
1080 |===
1081 | Layer | Child layer
1082
1083 | DRBD | CACHE, WRITECACHE, NVME, LUKS, STORAGE
1084 | CACHE | WRITECACHE, NVME, LUKS, STORAGE
1085 | WRITECACHE | CACHE, NVME, LUKS, STORAGE
1086 | NVME | CACHE, WRITECACHE, LUKS, STORAGE
1087 | LUKS | STORAGE
1088 | STORAGE | -
1089 |===
1090
1091 NOTE: One layer can only occur once in the layer-list
1092
1093 TIP: For information about the prerequisites for the `LUKS` layer,
1094 refer to the Encrypted Volumes section of this User's Guide.
1095
1096 [[s-nvme-layer]]
1097 ==== NVMe-oF/NVMe-TCP LINSTOR Layer
1098
1099 NVMe-oF/NVMe-TCP allows LINSTOR to connect diskless resources to a
1100 node with the same resource where the data is stored over NVMe
1101 fabrics. This leads to the advantage that resources can be mounted
1102 without using local storage by accessing the data over the network.
1103 LINSTOR is not using DRBD in this case, and therefore NVMe resources
1104 provisioned by LINSTOR are not replicated, the data is stored on one
1105 node.
1106
1107 NOTE: NVMe-oF only works on RDMA-capable networks and NVMe-TCP on
1108 every network that can carry IP traffic. If you want to know more
1109 about NVMe-oF/NVMe-TCP visit
1110 https://www.linbit.com/en/nvme-linstor-swordfish/ for more
1111 information.
1112
1113 To use NVMe-oF/NVMe-TCP with LINSTOR the package `nvme-cli` needs to
1114 be installed on every Node which acts as a Satellite and will use
1115 NVMe-oF/NVMe-TCP for a resource:
1116
1117 IMPORTANT: If you are not using Ubuntu use the suitable command for
1118 installing packages on your OS - SLES: zypper - CentOS: yum
1119
1120 ------
1121 # apt install nvme-cli
1122 ------
1123
1124 To make a resource which uses NVMe-oF/NVMe-TCP an additional parameter
1125 has to be given as you create the resource-definition:
1126
1127 ------
1128 # linstor resource-definition create nvmedata -l nvme,storage
1129 ------
1130
1131 NOTE: As default the -l (layer-stack) parameter is set to `drbd,
1132 storage` when DRBD is used. If you want to create LINSTOR resources
1133 with neither NVMe nor DRBD you have to set the `-l` parameter to only
1134 `storage`.
1135
1136 In order to use NVMe-TCP instead of the default NVMe-oF the following
1137 property needs to be set:
1138
1139 ------
1140 # linstor resource-definition set-property nvmedata NVMe/TRType tcp
1141 ------
1142
1143 The property `NVMe/TRType` can alternatively be set on resource-group
1144 or controller level.
1145
1146 Next, create the volume-definition for our resource:
1147
1148 ------
1149 # linstor volume-definition create nvmedata 500G
1150 ------
1151
1152 Before you create the resource on your nodes you have to know where
1153 the data will be stored locally and which node accesses it over the
1154 network.
1155
1156 First we create the resource on the node where our data will be stored:
1157
1158 ------
1159 # linstor resource create alpha nvmedata --storage-pool pool_ssd
1160 ------
1161
1162 On the nodes where the resource-data will be accessed over the
1163 network, the resource has to be defined as diskless:
1164
1165 -----
1166 # linstor resource create beta nvmedata --nvme-initiator
1167 -----
1168
1169 Now you can mount the resource `nvmedata` on one of your nodes.
1170
1171 IMPORTANT: If your nodes have more than one NIC you should force the
1172 route between them for NVMe-of/NVME-TCP, otherwise multiple NICs
1173 could cause troubles.
1174
1175 [[s-openflex]]
1176 ==== OpenFlex(TM) Layer
1177
1178 Since version 1.5.0 the additional Layer `openflex` can be used in LINSTOR.
1179 From LINSTOR's perspective, the
1180 https://www.westerndigital.com/products/storage-platforms/openflex-composable-infrastructure[OpenFlex
1181 Composable Infrastructure] takes the role of a combined layer acting as a
1182 storage layer (like LVM) and also providing the allocated space as an NVMe target.
1183 OpenFlex has a REST API which is also used by LINSTOR to operate with.
1184
1185 As OpenFlex combines concepts of LINSTOR's storage as well as NVMe-layer, LINSTOR was
1186 added both, a new storage driver for the storage pools as well as a dedicated `openflex`
1187 layer which uses the mentioned REST API.
1188
1189 In order for LINSTOR to communicate with the OpenFlex-API, LINSTOR needs some additional
1190 properties, which can be set once on `controller` level to take LINSTOR-cluster wide effect:
1191
1192 * `StorDriver/Openflex/ApiHost` specifies the host or IP of the API entry-point
1193 * `StorDriver/Openflex/ApiPort` this property is glued with a colon to the previous to form the
1194 basic `http://ip:port` part used by the REST calls
1195 * `StorDriver/Openflex/UserName` the REST username
1196 * `StorDriver/Openflex/UserPassword` the password for the REST user
1197
1198 Once that is configured, we can now create LINSTOR objects to represent the OpenFlex architecture.
1199 The theoretical mapping of LINSTOR objects to OpenFlex objects are as follows:
1200 Obviously an OpenFlex storage pool is represented by a LINSTOR storage pool. As the next thing above
1201 a LINSTOR storage pool is already the node, a LINSTOR node represents an OpenFlex storage device.
1202 The OpenFlex objects above storage device are not mapped by LINSTOR.
1203
1204 When using NVMe, LINSTOR was designed to run on both sides, the NVMe target as well as on the
1205 NVMe initiator side. In the case of OpenFlex, LINSTOR cannot (or even should not) run on the NVMe
1206 target side as that is completely managed by OpenFlex. As LINSTOR still needs nodes and storage pools
1207 to represent the OpenFlex counterparts, the LINSTOR client was extended with special node create commands
1208 since 1.0.14. These commands not only accept additionally needed configuration data, but also
1209 starts a "special satellite" besides the already running controller instance. This special satellites
1210 are completely LINSTOR managed, they will shutdown when the controller shuts down and will be started
1211 again when the controller starts.
1212 The new client command for creating a "special satellite" representing an OpenFlex storage device is:
1213
1214 ----
1215 $ linstor node create-openflex-target ofNode1 192.168.166.7 000af795789d
1216 ----
1217 The arguments are as follows:
1218
1219 * `ofNode1` is the node name which is also used by the standard `linstor node create` command
1220 * `192.168.166.7` is the address on which the provided NVMe devices can be accessed.
1221 As the NVMe devices are accessed by a dedicated network interface, this address differs from the
1222 address specified with the property `StorDriver/Openflex/ApiHost`. The latter is used for the
1223 management / REST API.
1224 * `000af795789d` is the identifier for the OpenFlex storage device.
1225
1226 The last step of the configuration is the creation of LINSTOR storage pools:
1227
1228 ----
1229 $ linstor storage-pool create openflex ofNode1 sp0 0
1230 ----
1231
1232 * `ofNode1` and `sp0` are the node name and storage pool name, respectively, just as usual for the
1233 LINSTOR's create storage pool command
1234 * The last `0` is the identifier of the OpenFlex storage pool within the previously defined storage device
1235
1236 Once all necessary storage pools are created in LINSTOR, the next steps are similar to
1237 the usage of using an NVMe resource with LINSTOR. Here is a complete example:
1238
1239 ----
1240 # set the properties once
1241 linstor controller set-property StorDriver/Openflex/ApiHost 10.43.7.185
1242 linstor controller set-property StorDriver/Openflex/ApiPort 80
1243 linstor controller set-property StorDriver/Openflex/UserName myusername
1244 linstor controller set-property StorDriver/Openflex/UserPassword mypassword
1245
1246 # create a node for openflex storage device "000af795789d"
1247 linstor node create-openflex-target ofNode1 192.168.166.7 000af795789d
1248
1249 # create a usual linstor satellite. later used as nvme initiator
1250 linstor node create bravo
1251
1252 # create a storage pool for openflex storage pool "0" within storage device "000af795789d"
1253 linstor storage-pool create openflex ofNode1 sp0 0
1254
1255 # create resource- and volume-definition
1256 linstor resource-definition create backupRsc
1257 linstor volume-definition create backupRsc 10G
1258
1259 # create openflex-based nvme target
1260 linstor resource create ofNode1 backupRsc --storage-pool sp0 --layer-list openflex
1261
1262 # create openflex-based nvme initiator
1263 linstor resource create bravo backupRsc --nvme-initiator --layer-list openflex
1264 ----
1265
1266 NOTE: In case a node should access the OpenFlex REST API through a different host than specified with +
1267 `linstor controller set-property StorDriver/Openflex/ApiHost 10.43.7.185` you can always use LINSTOR's
1268 inheritance mechanism for properties. That means simply define the same property on the node-level
1269 you need it, i.e. +
1270 `linstor node set-property ofNode1 StorDriver/Openflex/ApiHost 10.43.8.185`
1271
1272 [[s-writecache-layer]]
1273 ==== Writecache Layer
1274
1275 A https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/writecache.html[DM-Writecache]
1276 device is composed by two devices, one storage device and one cache device.
1277 LINSTOR can setup such a writecache device, but needs some additional information, like
1278 the storage pool and the size of the cache device.
1279
1280 ------
1281 # linstor storage-pool create lvm node1 lvmpool drbdpool
1282 # linstor storage-pool create lvm node1 pmempool pmempool
1283
1284 # linstor resource-definition create r1
1285 # linstor volume-definition create r1 100G
1286
1287 # linstor volume-definition set-property r1 0 Writecache/PoolName pmempool
1288 # linstor volume-definition set-property r1 0 Writecache/Size 1%
1289
1290 # linstor resource create node1 r1 --storage-pool lvmpool --layer-list WRITECACHE,STORAGE
1291 ------
1292
1293 The two properties set in the examples are mandatory, but can also be set on
1294 controller level which would act as a default for all resources with `WRITECACHE` in their
1295 `--layer-list`. However, please note that the `Writecache/PoolName` refers to
1296 the corresponding node. If the node does not have a storage-pool named `pmempool` you will
1297 get an error message.
1298
1299 The 4 mandatory parameters required by
1300 https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/writecache.html[DM-Writecache]
1301 are either configured via property or figured out by LINSTOR.
1302 The optional properties listed in the mentioned link can also be set via property.
1303 Please see `linstor controller set-property --help` for a list of `Writecache/*`
1304 property-keys.
1305
1306 Using `--layer-list DRBD,WRITECACHE,STORAGE` while having DRBD configured to use
1307 external metadata, only the backing device will use a writecache, not the
1308 device holding the external metadata.
1309
1310 [[s-cache-layer]]
1311 ==== Cache Layer
1312
1313 LINSTOR can also setup a https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/cache.html[DM-Cache]
1314 device, which is very similar to the DM-Writecache from the previous section. The major difference
1315 is that a cache device is composed by three devices: one storage device, one cache device and one
1316 meta device. The LINSTOR properties are quite similar to those of the writecache but are located
1317 in the `Cache` namespace:
1318
1319 ------
1320 # linstor storage-pool create lvm node1 lvmpool drbdpool
1321 # linstor storage-pool create lvm node1 pmempool pmempool
1322
1323 # linstor resource-definition create r1
1324 # linstor volume-definition create r1 100G
1325
1326 # linstor volume-definition set-property r1 0 Cache/CachePool pmempool
1327 # linstor volume-definition set-property r1 0 Cache/Cachesize 1%
1328
1329 # linstor resource create node1 r1 --storage-pool lvmpool --layer-list CACHE,STORAGE
1330 ------
1331
1332 NOTE: Instead of `Writecache/PoolName` (as when configuring the Writecache layer) the
1333 Cache layer's only required property is called `Cache/CachePool`. The reason for this
1334 is that the Cache layer also has a `Cache/MetaPool` which can be configured separately
1335 or it defaults to the value of `Cache/CachePool`.
1336
1337 Please see `linstor controller set-property --help` for a list of `Cache/*`
1338 property-keys and default values for omitted properties.
1339
1340 Using `--layer-list DRBD,CACHE,STORAGE` while having DRBD configured to use external
1341 metadata, only the backing device will use a cache, not the device holding the external
1342 metadata.
1343
1344 [[s-storage-layer]]
1345 ==== Storage Layer
1346
1347 The storage layer will provide new devices from well known volume managers like LVM, ZFS or
1348 others. Every layer combination needs to be based on a storage layer, even if the resource
1349 should be diskless - for that type there is a dedicated `diskless` provider type.
1350
1351 For a list of providers with their properties please see <<s-storage-proviers, Storage Providers>>.
1352
1353 For some storage providers LINSTOR has special properties:
1354
1355 * `StorDriver/WaitTimeoutAfterCreate`: If LINSTOR expects a device to appear after creation
1356 (for example after calls of `lvcreate`, `zfs create`,...), LINSTOR waits per default 500ms
1357 for the device to appear. These 500ms can be overridden by this property.
1358 * `StorDriver/dm_stats`: If set to `true` LINSTOR calls `dmstats create $device` after
1359 creation and `dmstats delete $device --allregions` after deletion of a volume.
1360 Currently only enabled for LVM and LVM_THIN storage providers.
1361
1362 [[s-storage-providers]]
1363 === Storage Providers
1364
1365 LINSTOR has a few storage providers. The most used ones are LVM and ZFS. But also for those
1366 two providers there are already sub-types for their thinly provisioned variants.
1367
1368 * Diskless: This provider type is mostly required to have a storage pool that can be configured
1369 with LINSTOR properties like `PrefNic` as described in <<s-managing_network_interface_cards,
1370 Managing Network Interface Cards>>.
1371 * LVM / LVM-Thin: The adminstrator is expected to specify the LVM volume group or the thin-pool
1372 (in form of "LV/thinpool") in order to use the corresponding storage type. These drivers support
1373 following properties for fine-tuning:
1374 ** `StorDriver/LvcreateOptions`: The value of this property is appended to every
1375 `lvcreate ...` call LINSTOR executes.
1376 * ZFS / ZFS-Thin: The administrator is expected to specify the ZPool that LINSTOR should use.
1377 These drivers support following properties for fine-tuning:
1378 ** `StorDriver/ZfscreateOptions`: The value of this property is appended to every
1379 `zfs create ...` call LINSTOR executes.
1380 * File / FileThin: Mostly used for demonstration / experiments. LINSTOR will basically reserve a
1381 file in a given directory and will configure a https://man7.org/linux/man-pages/man4/loop.4.html[loop device]
1382 on top of that file.
1383 * OpenFlex: This special storage provider currently requires to be run on a "special satellite".
1384 Please see <<s-openflex, OpenFlex(TM) Layer>> for more details.
1385 * EXOS: This special storage provider currently requires to be run on a "special satellite".
1386 Please see the <<ch-exos, EXOS Integration>> chapter
1387 * SPDK: The administrator is expected to speicify the logical volume store which LINSTOR should use.
1388 The usage of this storage provider implies the usage of the <<s-nvme-layer,NVME Layer>>.
1389 ** Remote-SPDK: This special storage provider currently requires to be run on a "special satellite".
1390 Please see <<s-remote-spdk, Remote SPDK Provider>> for more details.
1391
1392 [[s-remote-spdk]]
1393 ==== Remote SPDK Provider
1394 A storage pool with the type remote SPDK can only be created on a "special satellite". For this
1395 you first need to start a new satellite using the command:
1396
1397 ----
1398 $ linstor node create-remote-spdk-target nodeName 192.168.1.110
1399 ----
1400
1401 This will start a new satellite instance running on the same machine as the controller. This special
1402 satellite will do all the REST based RPC communication towards the remote SPDK proxy. As the help message
1403 of the LINSTOR command shows, the administrator might want to use additional settings when creating
1404 this special satellite:
1405
1406 ----
1407 $ linstor node create-remote-spdk-target -h
1408 usage: linstor node create-remote-spdk-target [-h] [--api-port API_PORT]
1409 [--api-user API_USER]
1410 [--api-user-env API_USER_ENV]
1411 [--api-pw [API_PW]]
1412 [--api-pw-env API_PW_ENV]
1413 node_name api_host
1414 ----
1415
1416 The difference between the `--api-\*` and their corresponding `--api-\*-env` versions is that the
1417 version with the `-env` ending will look for an environment variable containing the actual value to use
1418 whereas the `--api-\*` version directly take the value which is stored in the LINSTOR property.
1419 Administrators might not want to save the `--api-pw` in plaintext, which would be clearly visible
1420 using commands like `linstor node list-property <nodeName>`.
1421
1422 Once that special satellite is up and running the actual storage pool can be created:
1423
1424 ----
1425 $ linstor storage-pool create remotespdk -h
1426 usage: linstor storage-pool create remotespdk [-h]
1427 [--shared-space SHARED_SPACE]
1428 [--external-locking]
1429 node_name name driver_pool_name
1430 ----
1431
1432 Whereas `node_name` is self-explanatory, `name` is the name of the LINSTOR storage pool and
1433 `driver_pool_name` refers to the SPDK logical volume store.
1434
1435 Once this _remotespdk_ storage pool is created the remaining procedure is quite similar as
1436 using NVMe: First the target has to be created by creating a simple "diskful" resource followed
1437 by a second resource having the `--nvme-initiator` option enabled.
1438
1439
1440 [[s-managing_network_interface_cards]]
1441 === Managing Network Interface Cards
1442
1443 LINSTOR can deal with multiple network interface cards (NICs) in a machine,
1444 they are called `netif` in LINSTOR speak.
1445
1446 NOTE: When a satellite node is created a first `netif` gets created implicitly
1447 with the name `default`. Using the `--interface-name` option of the `node create`
1448 command you can give it a different name.
1449
1450 Additional NICs are created like this:
1451 ----------------------------
1452 # linstor node interface create alpha 100G_nic 192.168.43.221
1453 # linstor node interface create alpha 10G_nic 192.168.43.231
1454 ----------------------------
1455
1456 NICs are identified by the IP address only, the name is arbitrary and is
1457 *not* related to the interface name used by Linux. The NICs can be assigned
1458 to storage pools so that whenever a resource is created in such a storage
1459 pool, the DRBD traffic will be routed through the specified NIC.
1460
1461 ----------------------------
1462 # linstor storage-pool set-property alpha pool_hdd PrefNic 10G_nic
1463 # linstor storage-pool set-property alpha pool_ssd PrefNic 100G_nic
1464 ----------------------------
1465
1466 FIXME describe how to route the controller +<->+ client communication through
1467 a specific `netif`.
1468
1469 [[s-linstor-encrypted-volumes]]
1470 === Encrypted volumes
1471 LINSTOR can handle transparent encryption of drbd volumes. dm-crypt is used to
1472 encrypt the provided storage from the storage device.
1473
1474 NOTE: In order to use dm-crypt please make sure to have `cryptsetup` installed before
1475 you start the satellite
1476
1477 Basic steps to use encryption:
1478
1479 1. Disable user security on the controller (this will be obsolete once authentication works)
1480 2. Create a master passphrase
1481 3. Add `luks` to the layer-list. Note that all plugins (e.g., Proxmox) require a DRBD layer as the top most layer if they do not explicitly state otherwise.
1482 4. Don't forget to re-enter the master passphrase after a controller restart.
1483
1484 [[s-disable_user_security]]
1485 ==== Disable user security
1486 Disabling the user security on the `Linstor` controller is a one time operation and is
1487 afterwards persisted.
1488
1489 1. Stop the running linstor-controller via systemd: `systemctl stop linstor-controller`
1490 2. Start a linstor-controller in debug mode: `/usr/share/linstor-server/bin/Controller -c /etc/linstor -d`
1491 3. In the debug console enter: `setSecLvl secLvl(NO_SECURITY)`
1492 4. Stop linstor-controller with the debug shutdown command: `shutdown`
1493 5. Start the controller again with systemd: `systemctl start linstor-controller`
1494
1495 [[s-encrypt_commands]]
1496 ==== Encrypt commands
1497 Below are details about the commands.
1498
1499 Before LINSTOR can encrypt any volume a master passphrase needs to be created.
1500 This can be done with the linstor-client.
1501
1502 ----
1503 # linstor encryption create-passphrase
1504 ----
1505
1506 `crypt-create-passphrase` will wait for the user to input the initial master passphrase
1507 (as all other crypt commands will with no arguments).
1508
1509 If you ever want to change the master passphrase this can be done with:
1510
1511 ----
1512 # linstor encryption modify-passphrase
1513 ----
1514
1515 The `luks` layer can be added when creating the resource-definition or the resource
1516 itself, whereas the former method is recommended since it will be automatically applied
1517 to all resource created from that resource-definition.
1518
1519 ----
1520 # linstor resource-definition create crypt_rsc --layer-list luks,storage
1521 ----
1522
1523 To enter the master passphrase (after controller restart) use the following command:
1524
1525 ----
1526 # linstor encryption enter-passphrase
1527 ----
1528
1529 NOTE: Whenever the linstor-controller is restarted, the user has to send
1530 the master passphrase to the controller, otherwise LINSTOR is unable to reopen or
1531 create encrypted volumes.
1532
1533 [[s-automatic_passphrase]]
1534 ==== Automatic Passphrase
1535 It is possible to automate the process of creating and re-entering the master passphrase.
1536
1537 To use this, either an environment variable called `MASTER_PASSPHRASE` or an entry in
1538 `/etc/linstor/linstor.toml` containing the master passphrase has to be created.
1539
1540 The required `linstor.toml` looks like this:
1541
1542 ----
1543 [encrypt]
1544 passphrase="example"
1545 ----
1546
1547 If either one of these is set, then every time the controller starts it will check whether
1548 a master passphrase already exists. If there is none, it will create a new master passphrase as specified.
1549 Otherwise, the controller enters the passphrase.
1550
1551 WARNING: If a master passphrase is already configured, and it is not the same one as specified
1552 in the environment variable or `linstor.toml`, the controller will be unable to re-enter the
1553 master passphrase and react as if the user had entered a wrong passphrase.
1554 This can only be resolved through manual input from the user, using the same commands as if
1555 the controller was started without the automatic passphrase.
1556
1557 NOTE: In case the master passphrase is set in both an environment variable and the `linstor.toml`,
1558 only the master passphrase from the `linstor.toml` will be used.
1559
1560 [[s-linstor-status]]
1561 === Checking the state of your cluster
1562 LINSTOR provides various commands to check the state of your cluster.
1563 These commands start with a 'list-' prefix and provide various filtering and
1564 sorting options. The '--groupby' option can be used to group and sort the
1565 output in multiple dimensions.
1566
1567 ----------------------------
1568 # linstor node list
1569 # linstor storage-pool list --groupby Size
1570 ----------------------------
1571
1572 [[s-linstor-snapshots]]
1573 === Managing snapshots
1574 Snapshots are supported with thin LVM and ZFS storage pools.
1575
1576 [[s-creating_a_snapshot-linstor]]
1577 ==== Creating a snapshot
1578 Assuming a resource definition named 'resource1' which has been placed on some
1579 nodes, a snapshot can be created as follows:
1580
1581 ----------------------------
1582 # linstor snapshot create resource1 snap1
1583 ----------------------------
1584
1585 This will create snapshots on all nodes where the resource is present.
1586 LINSTOR will ensure that consistent snapshots are taken even when the
1587 resource is in active use.
1588
1589 Setting the resource-definition property `AutoSnapshot/RunEvery`
1590 LINSTOR will automatically create snapshots every X minute.
1591 The optional property `AutoSnapshot/Keep` can be used to clean-up old snapshots
1592 which were created automatically. No manually created snapshot will be cleaned-up / deleted.
1593 If `AutoSnapshot/Keep` is omitted (or <= 0), LINSTOR will keep the last 10 snapshots
1594 by default.
1595
1596 ----------------------------
1597 # linstor resource-definition set-property AutoSnapshot/RunEvery 15
1598 # linstor resource-definition set-property AutoSnapshot/Keep 5
1599 ----------------------------
1600
1601
1602 [[s-restoring_a_snapshot-linstor]]
1603 ==== Restoring a snapshot
1604 The following steps restore a snapshot to a new resource.
1605 This is possible even when the original resource has been removed
1606 from the nodes where the snapshots were taken.
1607
1608 First define the new resource with volumes matching those from the snapshot:
1609
1610 ----------------------------
1611 # linstor resource-definition create resource2
1612 # linstor snapshot volume-definition restore --from-resource resource1 --from-snapshot snap1 --to-resource resource2
1613 ----------------------------
1614
1615 At this point, additional configuration can be applied if necessary.
1616 Then, when ready, create resources based on the snapshots:
1617
1618 ----------------------------
1619 # linstor snapshot resource restore --from-resource resource1 --from-snapshot snap1 --to-resource resource2
1620 ----------------------------
1621
1622 This will place the new resource on all nodes where the snapshot is present.
1623 The nodes on which to place the resource can also be selected explicitly;
1624 see the help (`linstor snapshot resource restore -h`).
1625
1626 [[s-rolling_back_snapshot-linstor]]
1627 ==== Rolling back to a snapshot
1628 LINSTOR can roll a resource back to a snapshot state.
1629 The resource must not be in use.
1630 That is, it may not be mounted on any nodes.
1631 If the resource is in use, consider whether you can achieve your goal by
1632 <<s-restoring_a_snapshot-linstor,restoring the snapshot>> instead.
1633
1634 Rollback is performed as follows:
1635
1636 ----------------------------
1637 # linstor snapshot rollback resource1 snap1
1638 ----------------------------
1639
1640 A resource can only be rolled back to the most recent snapshot.
1641 To roll back to an older snapshot, first delete the intermediate snapshots.
1642
1643 [[s-removing_a_snapshot-linstor]]
1644 ==== Removing a snapshot
1645 An existing snapshot can be removed as follows:
1646
1647 ----------------------------
1648 # linstor snapshot delete resource1 snap1
1649 ----------------------------
1650
1651 [[s-shipping_snapshots-linstor]]
1652 ==== Shipping a snapshot
1653
1654 Snapshots can be shipped between LINSTOR nodes or between different LINSTOR clusters,
1655 as well as to an S3 storage such as https://aws.amazon.com/s3/[Amazon S3] or https://min.io/[min.io].
1656
1657 [[s-shipping_snapshots-remotes]]
1658 ===== Remotes
1659
1660 In a LINSTOR cluster, the definition of a shipping target is called a remote.
1661 Currently, there are two different types of remotes: LINSTOR remotes and S3 remotes.
1662 LINSTOR remotes are used to ship snapshots to a different LINSTOR cluster, while
1663 S3 remotes are needed to ship snapshots to AWS S3, min.io or any other service using S3
1664 compatible object storage.
1665
1666 IMPORTANT: Since a remote needs to store sensitive data, such as passwords, it is
1667 neccessary to have encryption enabled whenever you wish to use a remote in any way.
1668 How to set up LINSTOR's encryption is described <<s-encrypt_commands,here>>.
1669
1670 To create an S3 remote, LINSTOR will need to know the endpoint
1671 (that is, the URL of the target S3 server), the name of the target bucket, the region the S3
1672 server is in, as well as the access-key and secret-key used to access the bucket. If the command
1673 is sent without adding the secret-key, a prompt will pop up to enter it in. The command
1674 should look like this:
1675
1676 --------------------------
1677 # linstor remote create s3 myRemote s3.us-west-2.amazonaws.com my-bucket us-west-2 admin password
1678 --------------------------
1679
1680 TIP: Usually, LINSTOR uses the endpoint and bucket to create an url using the virtual-hosted-style
1681 for its access to the given bucket (e.g. my-bucket.s3.us-west-2.amazonaws.com). Should your setup not
1682 allow access this way, change the remote to path-style access (e.g. s3.us-west-2.amazonaws.com/my-bucket)
1683 by adding the `--use-path-style` argument to make LINSTOR combine the parameters accordingly.
1684
1685 To create a LINSTOR remote, only the URL or IP address of the controller of the target cluster
1686 is needed. The command goes as follows:
1687
1688 ----------------------------
1689 # linstor remote create linstor myRemote 192.168.0.15
1690 ----------------------------
1691
1692 Additionally, to ship LUKS-based (encrypted) backups, it is necessary to add the `--passphrase`
1693 and `--cluster-id` arguments to the command. This is used to save the passphrase and cluster-id of
1694 the target cluster to the remote respectively. For more details on shipping LUKS-based backups
1695 between two LINSTOR clusters, see <<s-shipping_snapshots-l2l,this chapter>>.
1696
1697 To see all the remotes known to the local cluster, use `linstor remote list`. To delete a remote, use
1698 `linstor remote delete myRemoteName`. Should an existing remote need altering, use `linstor remote
1699 modify` to change it.
1700
1701 [[s-shipping_snapshots-s3]]
1702 ===== Shipping snapshots to S3
1703
1704 All that is needed to ship a snapshot to S3 is to create an S3-remote that the current cluster can reach
1705 as well as the resource that should be shipped. Then, simply use the following command to ship it there:
1706
1707 --------------------------
1708 # linstor backup create myRemote myRsc
1709 --------------------------
1710
1711 This command will create a snapshot of your resource and ship it to the given remote. If this
1712 isn't the first time you shipped a backup of this resource (to that remote) and the snapshot
1713 of the previous backup hasn't been deleted yet, an incremental backup will be shipped.
1714 To force the creation of a full backup, add the `--full` argument to the command. Getting a
1715 specific node to ship the backup is also possible by using `--node myNode`, but if the specified
1716 node is not available or only has the resource diskless, a different node will be chosen.
1717
1718 To see which backups exist in a specific remote, use `linstor backup list myRemote`. A resource-name
1719 can be added to the command as a filter to only show backups of that specific resource by using the
1720 argument `--resource myRsc`. If you use the `--other` argument, only entries in the bucket that LINSTOR
1721 does not recognize as a backup will be shown. LINSTOR always names backups in a certain way, and
1722 as long as an item in the remote is named according to this schema, it is assumed that it is a backup
1723 created by LINSTOR - so this list will show everything else.
1724
1725 There are several options when it comes to deleting backups:
1726
1727 * `linstor backup delete all myRemote`: This command deletes ALL S3-objects on the given
1728 remote, as long as they are recognized to be backups, that is, fit the expected naming schema.
1729 There is the option `--cluster` to only delete backups that were created by the current cluster.
1730
1731 * `linstor backup delete id myRemote my-rsc_back_20210824_072543`: This command deletes
1732 a single backup from the given remote - namely the one with the given id, which consists
1733 of the resource-name, the automatically generated snapshot-name (back_timestamp) and, if
1734 set, the backup-suffix. The option `--prefix` lets you delete all backups starting with
1735 the given id. The option `--cascade` deletes not only the specified backup, but all other
1736 incremental backups depending on it.
1737
1738 * `linstor backup delete filter myRemote ...`: This command has a few different arguments
1739 to specify a selection of backups to delete. `-t 20210914_120000` will delete all backups
1740 made before 12 o'clock on the 14th of September, 2021. `-n myNode` will delete all backups
1741 uploaded by the given node. `-r myRsc` will delete all backups with the given resource name.
1742 These filters can be combined as needed. Finally, `--cascade` deletes not only the selected
1743 backup(s), but all other incremental backups depending on any of the selected backups.
1744
1745 * `linstor backup delete s3key myRemote randomPictureInWrongBucket`: This command will find
1746 the object with the given S3-key and delete it - without considering anything else. This should
1747 only be used to either delete non-backup items from the remote, or to clean up a broken backup
1748 that is no longer deleteable by other means. Using this command to delete a regular, working
1749 backup will break that backup, so beware!
1750
1751 WARNING: All commands that have the `--cascade` option will NOT delete a backup that has
1752 incremental backups depending on it unless you explicitly add that option.
1753
1754 TIP: All `linstor backup delete ...` commands have the `--dry-run` option, which will
1755 give you a list of all the S3-objects that will be deleted. This can be used to ensure
1756 nothing that should not be deleted is accidentally deleted.
1757
1758 Maybe the most important task after creating a backup is restoring it. To do so, only the remote
1759 is needed - but it is also possible to restore into an existing resource definition with no existing
1760 snapshots nor resources. There are two options for the command:
1761
1762 -----------------------
1763 # linstor backup restore myRemote myNode targetRsc --resource sourceRsc
1764 # linstor backup restore myRemote myNode targetRsc --id sourceRsc_back_20210824_072543
1765 -----------------------
1766
1767 Either `--resource (-r)` or `--id` must be used, but you cannot use both of them together. `-r` is used to
1768 restore the latest backup of the resource specified with this option, while `--id` restores the
1769 exact backup specified by the given id, and can therefore be used to restore backups other than
1770 the most recent.
1771
1772 If the backup to be restored includes a LUKS layer, the `--passphrase` argument is required. With
1773 it, the passphrase of the original cluster of the backup needs to be set so that LINSTOR can decrypt
1774 the volumes after download and re-encrypt them with the local passphrase.
1775
1776 The backup restore will download all the snapshots from the last full backup up to the specified
1777 backup. Afterwards, it restores the snapshots into a new resource. If that last step should be skipped,
1778 the `--download-only` option needs to be added to the command.
1779
1780 Backups can be downloaded from any cluster, not just the one that uploaded them, as long as the setup
1781 is correct. Specifically, the target resource cannot have any existing resources or snapshots, and the
1782 storage pool(s) used need to have the same storage providers. If the storage pool(s) on the target
1783 node have the exact same names as on the cluster the backup was created on, no extra action is
1784 necessary. Should they have different names, the option `--storpool-rename` needs to be used. It
1785 expects at least one `oldname=newname` pair. For every storage pool of the original backup that
1786 is not named in that list, it will be assumed that its name is exactly the same on the target node.
1787
1788 To find out exactly which storage pools need to be renamed, as well as how big the download and the
1789 restored resource will be, the command `linstor backup info myRemote ...` can be used. Similar to the restore
1790 command, either `-r` or `--id` need to be given, which add the same restrictions as with that command.
1791 To see how much space will be left over in the local storage pools after a restore, the argument `-n myNode`
1792 needs to be added. Just like with a restore, it assumes the storage pool names are exactly the same
1793 on the given node as with the backup. Should that not be the case, again, just like with the restore
1794 command, `--storpool-rename` should be used.
1795
1796 [[s-shipping_snapshots-l2l]]
1797 ===== Shipping snapshots whithin LINSTOR
1798
1799 Shipping a snapshot directly between two LINSTOR clusters can be done with a linstor-remote as well
1800 as a resource definition with at least one diskful resource on the source side (where the command
1801 is issued). On the target side, you only need a linstor-remote with the cluster-id of the source-cluster set.
1802 If you wish to ship a snapshot inside the same cluster just use a remote pointing to the local controller.
1803 The command looks like this:
1804
1805 --------------------------
1806 # linstor backup ship myRemote localRsc targetRsc
1807 --------------------------
1808
1809 Additionally, you can use `--source-node` and `--target-node` to specify which node should send and
1810 receive the backup respectively. In case those nodes are not available, a different one will be chosen
1811 automatically.
1812
1813 If the target resource already has resources deployed or the `--download-only` option was specified,
1814 the snapshots will only be shipped to the target cluster, but not restored.
1815
1816 When the snapshot you wish to ship contains a LUKS layer, the remote on the target cluster also needs
1817 the passphrase of the source cluster set.
1818
1819 [[s-shipping_snapshots-old]]
1820 ===== Shipping a snapshot in the same cluster
1821
1822 WARNING: The `snapshot ship` command is considered deprecated and any bugs found
1823 with it will not be fixed. Instead, use the `backup ship` command with a remote pointing
1824 to your local controller. For more details, see <<s-shipping_snapshots-l2l,the previous section>>.
1825
1826 Both, the source as well as the target node have to have the resource for
1827 snapshot shipping deployed. Additionally, the target resource has to be
1828 deactivated.
1829
1830 ----------------------------
1831 # linstor resource deactivate nodeTarget resource1
1832 ----------------------------
1833
1834 WARNING: Deactivating a resource with DRBD in its layer-list can NOT be
1835 reactivated again. However, a successfully shipped snapshot of a DRBD resource
1836 can still be <<s-restoring_a_snapshot-linstor,restored into a new resource>>.
1837
1838 To manually start the snapshot-shipping, use:
1839
1840 ----------------------------
1841 # linstor snapshot ship --from-node nodeSource --to-node nodeTarget --resource resource1
1842 ----------------------------
1843
1844 By default, the snapshot-shipping uses tcp ports from the range 12000-12999. To change
1845 this range, the property `SnapshotShipping/TcpPortRange`, which accepts a to-from range,
1846 can be set on the controller:
1847
1848 ----------------------------
1849 # linstor controller set-property SnapshotShipping/TcpPortRange 10000-12000
1850 ----------------------------
1851
1852 A resource can also be periodically shipped. To accomplish this, it is mandatory to
1853 set the properties `SnapshotShipping/TargetNode` as well as `SnapshotShipping/RunEvery`
1854 on the resource-definition.
1855 `SnapshotShipping/SourceNode` can also be set, but if omitted LINSTOR will choose
1856 an active resource of the same resource-definition.
1857
1858 To allow incremental snapshot-shipping, LINSTOR has to keep at least the last shipped
1859 snapshot on the target node. The property `SnapshotShipping/Keep` can be used to specify
1860 how many snapshots LINSTOR should keep. If the property is not set (or <= 0) LINSTOR
1861 will keep the last 10 shipped snapshots by default.
1862
1863 ----------------------------
1864 # linstor resource-definition set-property resource1 SnapshotShipping/TargetNode nodeTarget
1865 # linstor resource-definition set-property resource1 SnapshotShipping/SourceNode nodeSource
1866 # linstor resource-definition set-property resource1 SnapshotShipping/RunEvery 15
1867 # linstor resource-definition set-property resource1 SnapshotShipping/Keep 5
1868 ----------------------------
1869
1870 [[s-linstor-setupopts]]
1871 === Setting options for resources
1872
1873 DRBD options are set using LINSTOR commands.
1874 Configuration in files such as `/etc/drbd.d/global_common.conf` that are not
1875 managed by LINSTOR will be ignored.
1876 The following commands show the usage and available options:
1877
1878 ----------------------------
1879 # linstor controller drbd-options -h
1880 # linstor resource-definition drbd-options -h
1881 # linstor volume-definition drbd-options -h
1882 # linstor resource drbd-peer-options -h
1883 ----------------------------
1884
1885 For instance, it is easy to set the DRBD protocol for a resource named
1886 `backups`:
1887
1888 ----------------------------
1889 # linstor resource-definition drbd-options --protocol C backups
1890 ----------------------------
1891
1892 [[s-linstor-toggle-disk]]
1893 === Adding and removing disks
1894 LINSTOR can convert resources between diskless and having a disk.
1895 This is achieved with the `resource toggle-disk` command,
1896 which has syntax similar to `resource create`.
1897
1898 For instance, add a disk to the diskless resource `backups` on 'alpha':
1899
1900 ----------------------------
1901 # linstor resource toggle-disk alpha backups --storage-pool pool_ssd
1902 ----------------------------
1903
1904 Remove this disk again:
1905
1906 ----------------------------
1907 # linstor resource toggle-disk alpha backups --diskless
1908 ----------------------------
1909
1910 [[s-linstor-migrate-disk]]
1911 ==== Migrating disks
1912 In order to move a resource between nodes without reducing redundancy at any point,
1913 LINSTOR's disk migrate feature can be used.
1914 First create a diskless resource on the target node,
1915 and then add a disk using the `--migrate-from` option.
1916 This will wait until the data has been synced to the new disk and then remove
1917 the source disk.
1918
1919 For example, to migrate a resource `backups` from 'alpha' to 'bravo':
1920
1921 ----------------------------
1922 # linstor resource create bravo backups --drbd-diskless
1923 # linstor resource toggle-disk bravo backups --storage-pool pool_ssd --migrate-from alpha
1924 ----------------------------
1925
1926 [[s-linstor-proxy]]
1927 === DRBD Proxy with LINSTOR
1928
1929 LINSTOR expects DRBD Proxy to be running on the nodes which are involved in the
1930 relevant connections. It does not currently support connections via DRBD Proxy
1931 on a separate node.
1932
1933 Suppose our cluster consists of nodes 'alpha' and 'bravo' in a local network
1934 and 'charlie' at a remote site, with a resource definition named `backups`
1935 deployed to each of the nodes. Then DRBD Proxy can be enabled for the
1936 connections to 'charlie' as follows:
1937
1938 ----------------------------
1939 # linstor drbd-proxy enable alpha charlie backups
1940 # linstor drbd-proxy enable bravo charlie backups
1941 ----------------------------
1942
1943 The DRBD Proxy configuration can be tailored with commands such as:
1944
1945 ----------------------------
1946 # linstor drbd-proxy options backups --memlimit 100000000
1947 # linstor drbd-proxy compression zlib backups --level 9
1948 ----------------------------
1949
1950 LINSTOR does not automatically optimize the DRBD configuration for
1951 long-distance replication, so you will probably want to set some configuration
1952 options such as the protocol:
1953
1954 ----------------------------
1955 # linstor resource-connection drbd-options alpha charlie backups --protocol A
1956 # linstor resource-connection drbd-options bravo charlie backups --protocol A
1957 ----------------------------
1958
1959 Please contact LINBIT for assistance optimizing your configuration.
1960
1961 ==== Automatically enable DRBD Proxy
1962
1963 LINSTOR can also be configured to automatically enable the above mentioned Proxy
1964 connection between two nodes. For this automation, LINSTOR first needs to know
1965 on which site each node is.
1966
1967 ----------------------------
1968 # linstor node set-property alpha Site A
1969 # linstor node set-property bravo Site A
1970 # linstor node set-property charlie Site B
1971 ----------------------------
1972
1973 As the `Site` property might also be used for other site-based decisions in
1974 future features, the `DrbdProxy/AutoEnable` also has to be set to `true`:
1975
1976 ----------------------------
1977 # linstor controller set-property DrbdProxy/AutoEnable true
1978 ----------------------------
1979
1980 This property can also be set on node, resource-definition, resource and
1981 resource-connection level (from left to right in increasing priority, whereas
1982 the controller is the left-most, i.e. least prioritized level)
1983
1984 Once this initialization steps are completed, every newly created resource
1985 will automatically check if it has to enable DRBD proxy to any of its
1986 peer-resources.
1987
1988
1989 [[s-linstor-external-database]]
1990 === External database
1991
1992 It is possible to have LINSTOR working with an external database provider
1993 like PostgreSQL, MariaDB and since version 1.1.0 even ETCD key value store is supported.
1994
1995 To use an external database there are a few additional steps to configure.
1996 You have to create a DB/Schema and user to use for linstor, and configure this in the
1997 `/etc/linstor/linstor.toml`.
1998
1999
2000 [[s-postgresql]]
2001 ==== PostgreSQL
2002
2003 A sample PostgreSQL `linstor.toml` looks like this:
2004
2005 ------------------------------------------------------
2006 [db]
2007 user = "linstor"
2008 password = "linstor"
2009 connection_url = "jdbc:postgresql://localhost/linstor"
2010 ------------------------------------------------------
2011
2012 [[s-mariadb_mysql]]
2013 ==== MariaDB/MySQL
2014
2015 A sample MariaDB `linstor.toml` looks like this:
2016
2017 ------------------------------------------------------
2018 [db]
2019 user = "linstor"
2020 password = "linstor"
2021 connection_url = "jdbc:mariadb://localhost/LINSTOR?createDatabaseIfNotExist=true"
2022 ------------------------------------------------------
2023
2024 NOTE: The LINSTOR schema/database is created as `LINSTOR` so make sure the MariaDB connection string
2025 refers to the `LINSTOR` schema, as in the example above.
2026
2027
2028 [[s-etcd]]
2029 ==== ETCD
2030
2031 ETCD is a distributed key-value store that makes it easy to keep your LINSTOR database distributed in a HA-setup.
2032 The ETCD driver is already included in the LINSTOR-controller package and only needs to be configured in the `linstor.toml`.
2033
2034 More information on how to install and configure ETCD can be found here: https://etcd.io/docs[ETCD docs]
2035
2036 And here is a sample [db] section from the `linstor.toml`:
2037
2038 ------------------------------------------------------
2039 [db]
2040 ## only set user/password if you want to use authentication, only since LINSTOR 1.2.1
2041 # user = "linstor"
2042 # password = "linstor"
2043
2044 ## for etcd
2045 ## do not set user field if no authentication required
2046 connection_url = "etcd://etcdhost1:2379,etcdhost2:2379,etcdhost3:2379"
2047
2048 ## if you want to use TLS, only since LINSTOR 1.2.1
2049 # ca_certificate = "ca.pem"
2050 # client_certificate = "client.pem"
2051
2052 ## if you want to use client TLS authentication too, only since LINSTOR 1.2.1
2053 # client_key_pkcs8_pem = "client-key.pkcs8"
2054 ## set client_key_password if private key has a password
2055 # client_key_password = "mysecret"
2056 ------------------------------------------------------
2057
2058 [[s-linstor-rest-api]]
2059 === LINSTOR REST-API
2060
2061 To make LINSTOR's administrative tasks more accessible and also available for web-frontends a
2062 REST-API has been created. The REST-API is embedded in the linstor-controller
2063 and since LINSTOR 0.9.13 configured via the `linstor.toml` configuration file.
2064
2065 ---------
2066 [http]
2067 enabled = true
2068 port = 3370
2069 listen_addr = "127.0.0.1" # to disable remote access
2070 ---------
2071
2072
2073 If you want to use the REST-API the current documentation can be found on the following link:
2074 https://app.swaggerhub.com/apis-docs/Linstor/Linstor/
2075
2076 [[s-linstor-rest-api-https]]
2077 ==== LINSTOR REST-API HTTPS
2078
2079 The HTTP REST-API can also run secured by HTTPS and is highly recommended if you use any features that
2080 require authorization. Todo so you have to create a java keystore file with a valid certificate
2081 that will be used to encrypt all HTTPS traffic.
2082
2083 Here is a simple example on how you can create a self signed certificate with the `keytool` that is included
2084 in the java runtime:
2085
2086 ---------
2087 keytool -keyalg rsa -keysize 2048 -genkey -keystore ./keystore_linstor.jks\
2088 -alias linstor_controller\
2089 -dname "CN=localhost, OU=SecureUnit, O=ExampleOrg, L=Vienna, ST=Austria, C=AT"
2090 ---------
2091
2092 `keytool` will ask for a password to secure the generated keystore file and is needed for the
2093 LINSTOR-controller configuration.
2094 In your `linstor.toml` file you have to add the following section:
2095
2096 ---------
2097 [https]
2098 keystore = "/path/to/keystore_linstor.jks"
2099 keystore_password = "linstor"
2100 ---------
2101
2102 Now (re)start the `linstor-controller` and the HTTPS REST-API should be available on port 3371.
2103
2104 More information on how to import other certificates can be found here:
2105 https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html
2106
2107 NOTE: When HTTPS is enabled, all requests to the HTTP /v1/ REST-API will be redirected to the
2108 HTTPS redirect.
2109
2110
2111 [[s-rest-api-https-restricted-client]]
2112 ===== LINSTOR REST-API HTTPS restricted client access
2113
2114 Client access can be restricted by using a SSL truststore on the Controller.
2115 Basically you create a certificate for your client and add it to your truststore and the client
2116 then uses this certificate for authentication.
2117
2118 First create a client certificate:
2119
2120 ---------
2121 keytool -keyalg rsa -keysize 2048 -genkey -keystore client.jks\
2122 -storepass linstor -keypass linstor\
2123 -alias client1\
2124 -dname "CN=Client Cert, OU=client, O=Example, L=Vienna, ST=Austria, C=AT"
2125 ---------
2126
2127 Then we import this certificate to our controller truststore:
2128
2129 ---------
2130 keytool -importkeystore\
2131 -srcstorepass linstor -deststorepass linstor -keypass linstor\
2132 -srckeystore client.jks -destkeystore trustore_client.jks
2133 ---------
2134
2135 And enable the truststore in the `linstor.toml` configuration file:
2136
2137 ---------
2138 [https]
2139 keystore = "/path/to/keystore_linstor.jks"
2140 keystore_password = "linstor"
2141 truststore = "/path/to/trustore_client.jks"
2142 truststore_password = "linstor"
2143 ---------
2144
2145 Now restart the Controller and it will no longer be possible to access the controller API without a
2146 correct certificate.
2147
2148 The LINSTOR client needs the certificate in PEM format, so before we can use it we have to convert
2149 the java keystore certificate to the PEM format.
2150
2151 ---------
2152 # Convert to pkcs12
2153 keytool -importkeystore -srckeystore client.jks -destkeystore client.p12\
2154 -storepass linstor -keypass linstor\
2155 -srcalias client1 -srcstoretype jks -deststoretype pkcs12
2156
2157 # use openssl to convert to PEM
2158 openssl pkcs12 -in client.p12 -out client_with_pass.pem
2159 ---------
2160
2161 To avoid entering the PEM file password all the time it might be convenient to remove the password.
2162
2163 ---------
2164 openssl rsa -in client_with_pass.pem -out client1.pem
2165 openssl x509 -in client_with_pass.pem >> client1.pem
2166 ---------
2167
2168 Now this PEM file can easily be used in the client:
2169
2170 ---------
2171 linstor --certfile client1.pem node list
2172 ---------
2173
2174
2175 The `--certfile` parameter can also added to the client configuration file, see
2176 <<s-using_the_linstor_client>> for more details.
2177
2178
2179 [[s-linstor-logging]]
2180 === Logging
2181
2182 Linstor uses https://www.slf4j.org/[SLF4J] with https://logback.qos.ch/[Logback] as binding. This gives
2183 Linstor the possibility to distinguish between the log levels `ERROR`, `WARN`, `INFO`, `DEBUG` and `TRACE`
2184 (in order of increasing verbosity). In the current linstor version (1.1.2) the user has the following
2185 four methods to control the logging level, ordered by priority (first has highest priority):
2186
2187
2188 . `TRACE` mode can be `enabled` or `disabled` using the debug console:
2189 +
2190 ----
2191 Command ==> SetTrcMode MODE(enabled)
2192 SetTrcMode Set TRACE level logging mode
2193 New TRACE level logging mode: ENABLED
2194 ----
2195
2196 . When starting the controller or satellite a command line argument can be passed:
2197 +
2198 ----
2199 java ... com.linbit.linstor.core.Controller ... --log-level INFO
2200 java ... com.linbit.linstor.core.Satellite ... --log-level INFO
2201 ----
2202
2203 . The recommended place is the `logging` section in the configuration file.
2204 The default configuration file location is `/etc/linstor/linstor.toml` for the controller
2205 and `/etc/linstor/linstor_satellite.toml` for the satellite.
2206 Configure the logging level as follows:
2207 +
2208 ----
2209 [logging]
2210 level="INFO"
2211 ----
2212
2213 . As Linstor is using Logback as an implementation, `/usr/share/linstor-server/lib/logback.xml` can
2214 also be used. Currently only this approach supports different log levels for different components, like
2215 shown in the example below:
2216 +
2217 ----
2218 <?xml version="1.0" encoding="UTF-8"?>
2219 <configuration scan="false" scanPeriod="60 seconds">
2220 <!--
2221 Values for scanPeriod can be specified in units of milliseconds, seconds, minutes or hours
2222 https://logback.qos.ch/manual/configuration.html
2223 -->
2224 <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
2225 <!-- encoders are assigned the type
2226 ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
2227 <encoder>
2228 <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n</pattern>
2229 </encoder>
2230 </appender>
2231 <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
2232 <file>${log.directory}/linstor-${log.module}.log</file>
2233 <append>true</append>
2234 <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
2235 <Pattern>%d{yyyy_MM_dd HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n</Pattern>
2236 </encoder>
2237 <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
2238 <FileNamePattern>logs/linstor-${log.module}.%i.log.zip</FileNamePattern>
2239 <MinIndex>1</MinIndex>
2240 <MaxIndex>10</MaxIndex>
2241 </rollingPolicy>
2242 <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
2243 <MaxFileSize>2MB</MaxFileSize>
2244 </triggeringPolicy>
2245 </appender>
2246 <logger name="LINSTOR/Controller" level="INFO" additivity="false">
2247 <appender-ref ref="STDOUT" />
2248 <!-- <appender-ref ref="FILE" /> -->
2249 </logger>
2250 <logger name="LINSTOR/Satellite" level="INFO" additivity="false">
2251 <appender-ref ref="STDOUT" />
2252 <!-- <appender-ref ref="FILE" /> -->
2253 </logger>
2254 <root level="WARN">
2255 <appender-ref ref="STDOUT" />
2256 <!-- <appender-ref ref="FILE" /> -->
2257 </root>
2258 </configuration>
2259 ----
2260
2261 See the https://logback.qos.ch/manual/index.html[Logback Manual] to find more details about `logback.xml`.
2262
2263
2264 When none of the configuration methods above is used Linstor will default to `INFO` log level.
2265
2266 [[s-linstor-monitoring]]
2267 === Monitoring
2268
2269 Since LINSTOR 1.8.0, a https://prometheus.io/[Prometheus] `/metrics` HTTP path is provided with LINSTOR and JVM specific
2270 exports.
2271
2272 The `/metrics` path also supports 3 GET arguments to reduce LINSTOR's reported data:
2273
2274 * resource
2275 * storage_pools
2276 * error_reports
2277
2278 These are all default `true`, to disabled e.g. error-report data: `http://localhost:3370/metrics?error_reports=false`
2279
2280 ==== Health check
2281
2282 The LINSTOR-Controller also provides a `/health` HTTP path that will simply return HTTP-Status 200 if the controller can access its
2283 database and all services are up and running. Otherwise it will return HTTP error status code 500 `Internal Server Error`.
2284
2285
2286 [[s-linstor-secure-connections]]
2287 === Secure Satellite connections
2288
2289 It is possible to have the LINSTOR use SSL secure TCP connection between controller and satellite connections.
2290 Without going into further details on how java's SSL engine works we will give you
2291 command line snippets using the `keytool` from java's runtime environment on how to configure
2292 a 3 node setup using secure connections.
2293 The node setup looks like this:
2294
2295 Node `alpha` is the just the controller.
2296 Node `bravo` and node `charlie` are just satellites.
2297
2298
2299 Here are the commands to generate such a keystore setup,
2300 values should of course be edited for your environment.
2301
2302 ---------
2303 # create directories to hold the key files
2304 mkdir -p /tmp/linstor-ssl
2305 cd /tmp/linstor-ssl
2306 mkdir alpha bravo charlie
2307
2308
2309 # create private keys for all nodes
2310 keytool -keyalg rsa -keysize 2048 -genkey -keystore alpha/keystore.jks\
2311 -storepass linstor -keypass linstor\
2312 -alias alpha\
2313 -dname "CN=Max Mustermann, OU=alpha, O=Example, L=Vienna, ST=Austria, C=AT"
2314
2315 keytool -keyalg rsa -keysize 2048 -genkey -keystore bravo/keystore.jks\
2316 -storepass linstor -keypass linstor\
2317 -alias bravo\
2318 -dname "CN=Max Mustermann, OU=bravo, O=Example, L=Vienna, ST=Austria, C=AT"
2319
2320 keytool -keyalg rsa -keysize 2048 -genkey -keystore charlie/keystore.jks\
2321 -storepass linstor -keypass linstor\
2322 -alias charlie\
2323 -dname "CN=Max Mustermann, OU=charlie, O=Example, L=Vienna, ST=Austria, C=AT"
2324
2325 # import truststore certificates for alpha (needs all satellite certificates)
2326 keytool -importkeystore\
2327 -srcstorepass linstor -deststorepass linstor -keypass linstor\
2328 -srckeystore bravo/keystore.jks -destkeystore alpha/certificates.jks
2329
2330 keytool -importkeystore\
2331 -srcstorepass linstor -deststorepass linstor -keypass linstor\
2332 -srckeystore charlie/keystore.jks -destkeystore alpha/certificates.jks
2333
2334 # import controller certificate into satellite truststores
2335 keytool -importkeystore\
2336 -srcstorepass linstor -deststorepass linstor -keypass linstor\
2337 -srckeystore alpha/keystore.jks -destkeystore bravo/certificates.jks
2338
2339 keytool -importkeystore\
2340 -srcstorepass linstor -deststorepass linstor -keypass linstor\
2341 -srckeystore alpha/keystore.jks -destkeystore charlie/certificates.jks
2342
2343 # now copy the keystore files to their host destinations
2344 ssh root@alpha mkdir /etc/linstor/ssl
2345 scp alpha/* root@alpha:/etc/linstor/ssl/
2346 ssh root@bravo mkdir /etc/linstor/ssl
2347 scp bravo/* root@bravo:/etc/linstor/ssl/
2348 ssh root@charlie mkdir /etc/linstor/ssl
2349 scp charlie/* root@charlie:/etc/linstor/ssl/
2350
2351 # generate the satellite ssl config entry
2352 echo '[netcom]
2353 type="ssl"
2354 port=3367
2355 server_certificate="ssl/keystore.jks"
2356 trusted_certificates="ssl/certificates.jks"
2357 key_password="linstor"
2358 keystore_password="linstor"
2359 truststore_password="linstor"
2360 ssl_protocol="TLSv1.2"
2361 ' | ssh root@bravo "cat > /etc/linstor/linstor_satellite.toml"
2362
2363 echo '[netcom]
2364 type="ssl"
2365 port=3367
2366 server_certificate="ssl/keystore.jks"
2367 trusted_certificates="ssl/certificates.jks"
2368 key_password="linstor"
2369 keystore_password="linstor"
2370 truststore_password="linstor"
2371 ssl_protocol="TLSv1.2"
2372 ' | ssh root@charlie "cat > /etc/linstor/linstor_satellite.toml"
2373 ---------
2374
2375 Now just start controller and satellites and add the nodes with `--communication-type SSL`.
2376
2377 [[s-linstor-drbd-automatisms]]
2378 === Automatisms for DRBD-Resources
2379
2380 [[s-linstor-auto-quorum]]
2381 ==== AutoQuorum Policies
2382
2383 LINSTOR automatically configures quorum policies on resources *when
2384 quorum is achievable*. This means, whenever you have at least two diskful and one
2385 or more diskless resource assignments, or three or more diskful
2386 resource assignments, LINSTOR will enable quorum policies for your
2387 resources automatically.
2388
2389 Inversely, LINSTOR will automatically disable quorum policies whenever
2390 there are less than the minimum required resource assignments to
2391 achieve quorum.
2392
2393 This is controlled via the, `DrbdOptions/auto-quorum`, property which
2394 can be applied to the _linstor-controller_, _resource-group_, and
2395 _resource-definition_. Accepted values for the
2396 `DrbdOptions/auto-quorum` property are `disabled`, `suspend-io`, and
2397 `io-error`.
2398
2399 Setting the `DrbdOptions/auto-quorum` property to `disabled` will
2400 allow you to manually, or more granularly, control the quorum policies
2401 of your resources should you so desire.
2402
2403 TIP: The default policies for `DrbdOptions/auto-quorum` are `quorum
2404 majority`, and `on-no-quorum io-error`. For more information on DRBD's
2405 quorum features and their behavior, please refer to the
2406 {url-drbd-ug}#s-feature-quorum[quorum section of the DRBD user's guide].
2407
2408 IMPORTANT: The `DrbdOptions/auto-quorum` policies will override any
2409 manually configured properties if `DrbdOptions/auto-quorum` is not disabled.
2410
2411 For example, to manually set the quorum policies of a _resource-group_
2412 named `my_ssd_group`, you would use the following commands:
2413
2414 ----
2415 # linstor resource-group set-property my_ssd_group DrbdOptions/auto-quorum disabled
2416 # linstor resource-group set-property my_ssd_group DrbdOptions/Resource/quorum majority
2417 # linstor resource-group set-property my_ssd_group DrbdOptions/Resource/on-no-quorum suspend-io
2418 ----
2419
2420 You may wish to disable DRBD's quorum features completely. To do that,
2421 you would need to first disable `DrbdOptions/auto-quorum` on the
2422 appropriate LINSTOR object, and then set the DRBD quorum features
2423 accordingly. For example, use the following commands to disable quorum
2424 entirely on the `my_ssd_group` _resource-group_:
2425
2426 ----
2427 # linstor resource-group set-property my_ssd_group DrbdOptions/auto-quorum disabled
2428 # linstor resource-group set-property my_ssd_group DrbdOptions/Resource/quorum off
2429 # linstor resource-group set-property my_ssd_group DrbdOptions/Resource/on-no-quorum
2430 ----
2431
2432 NOTE: Setting `DrbdOptions/Resource/on-no-quorum` to an empty value
2433 in the commands above deletes the property from the object entirely.
2434
2435 [[s-linstor-auto-evict]]
2436 ==== Auto-Evict
2437
2438 If a satellite is offline for a prolonged period of time, LINSTOR can be configured to
2439 declare that node as evicted. This triggers an automated reassignment of the affected
2440 DRBD-resources to other nodes to ensure a minimum replica count is kept.
2441
2442 This feature uses the following properties to adapt the behaviour.
2443
2444 * `DrbdOptions/AutoEvictMinReplicaCount` sets the number of replicas that should always
2445 be present. You can set this property on the controller to change a global default, or on
2446 a specific resource-definition or resource-group to change it only for that
2447 resource-definition or resource-group. If this property is left empty, the place-count
2448 set for the auto-placer of the corresponding resource-group will be used.
2449
2450 * `DrbdOptions/AutoEvictAfterTime` describes how long a node can be offline in minutes
2451 before the eviction is triggered. You can set this property on the controller to change a
2452 global default, or on a single node to give it a different behavior. The default value
2453 for this property is 60 minutes.
2454
2455 * `DrbdOptions/AutoEvictMaxDisconnectedNodes` sets the percentage of nodes that can
2456 be not reachable (for whatever reason) at the same time. If more than the given percent
2457 of nodes are offline at the same time, the auto-evict will not be triggered for any node
2458 , since in this case LINSTOR assumes connection problems from the controller. This property
2459 can only be set for the controller, and only accepts a value between 0 and 100. The default
2460 value is 34. If you wish to turn the auto-evict-feature off, simply set this property
2461 to 0. If you want to always trigger the auto-evict, regardless of how many satellites
2462 are unreachable, set it to 100.
2463
2464 * `DrbdOptions/AutoEvictAllowEviction` is an additional property that can stop a node from
2465 being evicted. This can be useful for various cases, for example if you need to shut down
2466 a node for maintenance. You can set this property on the controller to change a global default,
2467 or on a single node to give it a different behavior. It accepts true and false as values and
2468 per default is set to true on the controller. You can use this property to turn the auto-evict
2469 feature off by setting it to false on the controller, although this might not work
2470 completely if you already set different values for individual nodes, since those values take precedence
2471 over the global default.
2472
2473 After the linstor-controller loses the connection to a satellite, aside from trying to
2474 reconnect, it starts a timer for that satellite. As soon as that timer exceeds
2475 `DrbdOptions/AutoEvictAfterTime` and all of the DRBD-connections to the DRBD-resources
2476 on that satellite are broken, the controller will check whether or not
2477 `DrbdOptions/AutoEvictMaxDisconnectedNodes` has been met. If it hasn't, and
2478 `DrbdOptions/AutoEvictAllowEviction` is true for the node in question, the satellite
2479 will be marked as EVICTED. At the same time, the controller will check for every DRBD-resource whether
2480 the number of resources is still above `DrbdOptions/AutoEvictMinReplicaCount`. If it is,
2481 the resource in question will be marked as DELETED. If it isn't, an auto-place with the settings
2482 from the corresponding resource-group will be started. Should the auto-place fail, the
2483 controller will try again later when changes that might allow a different result, such
2484 as adding a new node, have happened. Resources where an auto-place is necessary will only be
2485 marked as DELETED if the corresponding auto-place was successful.
2486
2487 The evicted satellite itself will not be able to reestablish connection with the controller.
2488 Even if the node is up and running, a manual reconnect will fail. It is also not possible to
2489 delete the satellite, even if it is working as it should be. The satellite can, however, be restored.
2490 This will remove the EVICTED-flag from the satellite and allow you to use it again. Previously
2491 configured network interfaces, storage pools, properties and similar entities as well as
2492 non-DRBD-related resources and resources that could not be autoplaced somewhere else will
2493 still be on the satellite. To restore a satellite, use
2494
2495 ------
2496 # linstor node restore [nodename]
2497 ------
2498
2499 Should you wish to instead throw everything that once was on that node, including the node
2500 itself, away, you need to use the `node lost` command instead.
2501
2502 [[s-linstor-qos]]
2503 === QoS Settings
2504 ==== Sysfs
2505
2506 LINSTOR is able to set the following Sysfs settings:
2507
2508 [cols="3,2"]
2509 |===
2510 | SysFs | Linstor property
2511
2512 | `/sys/fs/cgroup/blkio/blkio.throttle.read_bps_device` | `sys/fs/blkio_throttle_read`
2513 | `/sys/fs/cgroup/blkio/blkio.throttle.write_bps_device` | `sys/fs/blkio_throttle_write`
2514 | `/sys/fs/cgroup/blkio/blkio.throttle.read_iops_device` | `sys/fs/blkio_throttle_read_iops`
2515 | `/sys/fs/cgroup/blkio/blkio.throttle.write_iops_device` | `sys/fs/blkio_throttle_write_iops`
2516 |===
2517
2518 If a LINSTOR volume is composed of multiple "stacked" volume (for example DRBD with external
2519 metadata will have 3 devices: backing (storage) device, metadata device and the resulting
2520 DRBD device), setting a `sys/fs/\*` property for a Volume, only the bottom-most local
2521 "data"-device will receive the corresponding `/sys/fs/cgroup/...` setting.
2522 That means, in case of the example above only the backing device will receive the setting.
2523 In case a resource-definition has an nvme-target as well as an nvme-initiator resource,
2524 both bottom-most devices of each node will receive the setting. In case of the target the
2525 bottom-most device will be the volume of LVM or ZFS, whereas in case of the initiator the
2526 bottom-most device will be the connected nvme-device, regardless which other layers
2527 are stacked on top of that.
2528
2529
2530 [[s-linstor-getting-help]]
2531 === Getting help
2532 ==== From the command line
2533 A quick way to list available commands on the command line is to type
2534 `linstor`.
2535
2536 Further information on sub-commands (e.g., list-nodes) can be retrieved in
2537 two ways:
2538
2539 ----------------------------
2540 # linstor node list -h
2541 # linstor help node list
2542 ----------------------------
2543
2544 Using the 'help' sub-command is especially helpful when LINSTOR is executed
2545 in interactive mode (`linstor interactive`).
2546
2547 One of the most helpful features of LINSTOR is its rich tab-completion,
2548 which can be used to complete basically every object LINSTOR knows about
2549 (e.g., node names, IP addresses, resource names, ...).
2550 In the following examples, we show some possible completions, and their results:
2551
2552 ----------------------------
2553 # linstor node create alpha 1<tab> # completes the IP address if hostname can be resolved
2554 # linstor resource create b<tab> c<tab> # linstor assign-resource backups charlie
2555 ----------------------------
2556
2557 If tab-completion does not work out of the box, please try to source the
2558 appropriate file:
2559
2560 ----------------------------
2561 # source /etc/bash_completion.d/linstor # or
2562 # source /usr/share/bash_completion/completions/linstor
2563 ----------------------------
2564
2565 For zsh shell users linstor-client can generate a zsh compilation file,
2566 that has basic support for command and argument completion.
2567
2568 ----------------------------
2569 # linstor gen-zsh-completer > /usr/share/zsh/functions/Completion/Linux/_linstor
2570 ----------------------------
2571
2572 ==== SOS-Report
2573
2574 If something goes wrong and you need help finding the cause of the issue, you can use
2575
2576 ----------------------------
2577 # linstor sos-report create
2578 ----------------------------
2579
2580 The command above will create a new sos-report in `/var/log/linstor/controller/` on the
2581 controller node. Alternatively you can use
2582
2583 ----------------------------
2584 # linstor sos-report download
2585 ----------------------------
2586
2587 which will create a new sos-report and additionally downloads that report to the local machine
2588 into your current working directory.
2589
2590 This sos-report contains logs and useful debug-information from several sources
2591 (Linstor-logs, `dmesg`, versions of external tools used by Linstor, `ip a`, database dump
2592 and many more).
2593 These information are stored for each node in plaintext in the resulting `.tar.gz` file.
2594
2595
2596 ==== From the community
2597 For help from the community please subscribe to our mailing list located here: https://lists.linbit.com/listinfo/drbd-user
2598
2599 ==== GitHub
2600 To file bug or feature request please check out our GitHub page https://github.com/linbit
2601
2602 ==== Paid support and development
2603 Alternatively, if you wish to purchase remote installation services, 24/7 support, access to certified repositories, or feature development please contact us: +1-877-454-6248 (1-877-4LINBIT) , International: +43-1-8178292-0 | [email protected]