| 1 |
[[s-administrative-tasks-setup]] |
| 2 |
== Basic administrative tasks / Setup |
| 3 |
|
| 4 |
LINSTOR is a configuration management system for storage on Linux systems. |
| 5 |
It manages LVM logical volumes and/or ZFS ZVOLs on a cluster of nodes. It |
| 6 |
leverages DRBD for replication between different nodes and to provide |
| 7 |
block storage devices to users and applications. It manages snapshots, |
| 8 |
encryption and caching of HDD backed data in SSDs via bcache. |
| 9 |
|
| 10 |
// Troubleshooting for LINSTOR guide? |
| 11 |
///// |
| 12 |
This chapter outlines typical administrative tasks encountered during |
| 13 |
day-to-day operations. It does not cover troubleshooting tasks, these |
| 14 |
are covered in detail in <<ch-troubleshooting>>. |
| 15 |
///// |
| 16 |
|
| 17 |
|
| 18 |
[[s-concepts_and_terms]] |
| 19 |
=== Concepts and Terms |
| 20 |
This section goes over core concepts and terms that you will need to familiarize yourself |
| 21 |
with to understand how LINSTOR works and deploys storage. The section is laid out in a |
| 22 |
"ground up" approach. |
| 23 |
|
| 24 |
|
| 25 |
==== Installable Components |
| 26 |
===== linstor-controller |
| 27 |
A LINSTOR setup requires at least one active controller and one or more satellites. |
| 28 |
|
| 29 |
// Once the chapter on making your controller HA is done we need to link that here |
| 30 |
|
| 31 |
The _linstor-controller_ relies on a database that holds all configuration |
| 32 |
information for the whole cluster. It makes all decisions that need to have a |
| 33 |
view of the whole cluster. Multiple controllers can be used for LINSTOR but |
| 34 |
only one can be active. |
| 35 |
|
| 36 |
===== linstor-satellite |
| 37 |
The _linstor-satellite_ runs on each node where LINSTOR consumes local |
| 38 |
storage or provides storage to services. It is stateless; it receives |
| 39 |
all the information it needs from the controller. It runs programs |
| 40 |
like `lvcreate` and `drbdadm`. It acts like a node agent. |
| 41 |
|
| 42 |
===== linstor-client |
| 43 |
The _linstor-client_ is a command line utility that you use to issue |
| 44 |
commands to the system and to investigate the status of the system. |
| 45 |
|
| 46 |
==== Objects |
| 47 |
Objects are the end result which LINSTOR presents to the end-user or application, |
| 48 |
such as; Kubernetes/OpenShift, a replicated block device (DRBD), NVMeOF target, etc. |
| 49 |
|
| 50 |
===== Node |
| 51 |
Node's are a server or container that participate in a LINSTOR cluster. The _Node_ |
| 52 |
attribute defines: |
| 53 |
|
| 54 |
* Determines which LINSTOR cluster the node participates in |
| 55 |
* Sets the role of the node: Controller, Satellite, Auxiliary |
| 56 |
* *NetInterface* objects define the node's connectivity |
| 57 |
|
| 58 |
===== NetInterface |
| 59 |
As the name implies, this is how you define the interface/address of a node's network interface. |
| 60 |
|
| 61 |
===== Definitions |
| 62 |
Definitions define attributes of an object, they can be thought of as |
| 63 |
profile or template. Objects created will inherit the configuration |
| 64 |
defined in the definitions. A definition must be defined prior to creating |
| 65 |
the associated object. For example; you must create a _ResourceDefinition_ |
| 66 |
prior to creating the _Resource_ |
| 67 |
|
| 68 |
StoragePoolDefinition ::: |
| 69 |
* Defines the name of a storage pool |
| 70 |
|
| 71 |
ResourceDefinition ::: |
| 72 |
Resource definitions define the following attributes of a resource: |
| 73 |
* The name of a DRBD resource |
| 74 |
* The TCP port for DRBD to use for the resource's connection |
| 75 |
|
| 76 |
VolumeDefinition ::: |
| 77 |
Volume definitions define the following: |
| 78 |
|
| 79 |
* A volume of a DRBD resource |
| 80 |
* The size of the volume |
| 81 |
* The volume number of the DRBD resource's volume |
| 82 |
* The meta data properties of the volume |
| 83 |
* The minor number to use for the DRBD device associated with the DRBD volume |
| 84 |
|
| 85 |
===== StoragePool |
| 86 |
The _StoragePool_ identifies storage in the context of LINSTOR. It defines: |
| 87 |
|
| 88 |
* The configuration of a storage pool on a specific node |
| 89 |
* The storage back-end driver to use for the storage pool on the cluster node (LVM, ZFS, etc) |
| 90 |
* The parameters and configuration to pass to the storage backed driver |
| 91 |
|
| 92 |
===== Resource |
| 93 |
LINSTOR has now has expanded its capabilities to manage a broader set of storage technologies |
| 94 |
outside of just DRBD. A _Resource_: |
| 95 |
|
| 96 |
* Represents the placement of a DRBD resource, as defined within the _ResourceDefinition_ |
| 97 |
* Places a resource on a node in the cluster |
| 98 |
* Defines the placement of a _ResourceDefinition_ on a node |
| 99 |
|
| 100 |
===== Volume |
| 101 |
Volumes are a subset of a _Resource_. A _Resource_ could have multiple volumes, for example |
| 102 |
you may wish to have your database stored on slower storage than your logs in your MySQL cluster. |
| 103 |
By keeping the _volumes_ under a single _resource_ you are essentially creating a consistency group. |
| 104 |
The _Volume_ attribute can define also define attributes on a more granular level. |
| 105 |
|
| 106 |
[[s-broader_context]] |
| 107 |
=== Broader Context |
| 108 |
|
| 109 |
While LINSTOR might be used to make the management of DRBD more |
| 110 |
convenient, it is often integrated with software stacks higher up. |
| 111 |
Such integration exist already for Kubernetes, OpenStack, OpenNebula |
| 112 |
and Proxmox. Chapters specific to deploying LINSTOR in these |
| 113 |
environments are included in this guide. |
| 114 |
|
| 115 |
The southbound drivers used by LINSTOR are LVM, thinLVM and ZFS. |
| 116 |
|
| 117 |
[[s-packages]] |
| 118 |
=== Packages |
| 119 |
|
| 120 |
LINSTOR is packaged in both the .rpm and the .deb variants: |
| 121 |
|
| 122 |
. _linstor-client_ contains the command line client program. It depends |
| 123 |
on python which is usually already installed. In RHEL 8 systems you will need to symlink |
| 124 |
python |
| 125 |
. _linstor-controller_ and _linstor-satellite_ Both contain systemd unit files |
| 126 |
for the services. They depend on Java runtime environment (JRE) version 1.8 |
| 127 |
(headless) or higher. |
| 128 |
|
| 129 |
For further detail about these packages see the |
| 130 |
<<Installable Components,Installable Components>> section above. |
| 131 |
|
| 132 |
NOTE: If you have a support subscription to LINBIT, you will have access to |
| 133 |
our certified binaries via our repositories. |
| 134 |
|
| 135 |
[[s-installation]] |
| 136 |
=== Installation |
| 137 |
|
| 138 |
IMPORTANT: If you want to use LINSTOR in containers skip this Topic and use the "Containers" section below for the installation. |
| 139 |
|
| 140 |
|
| 141 |
|
| 142 |
[[s-admin-linstor-ubuntu_linux]] |
| 143 |
==== Ubuntu Linux |
| 144 |
|
| 145 |
If you want to have the option of creating replicated storage using DRBD, you will need to install _drbd-dkms_ and _drbd-utils_. These packages will need to be installed on all nodes. You will also need |
| 146 |
to choose a volume manager, either ZFS or LVM, in this instance we're using LVM. |
| 147 |
|
| 148 |
-------------------------------------------- |
| 149 |
# apt install -y drbd-dkms drbd-utils lvm2 |
| 150 |
-------------------------------------------- |
| 151 |
|
| 152 |
Depending on whether your node is a LINSTOR controller, satellite, or both (Combined) will determine what packages are required on that node. For combined type nodes, we'll need both the controller and satellite LINSTOR package. |
| 153 |
|
| 154 |
Combined node: |
| 155 |
|
| 156 |
-------------------------------------------------------------------- |
| 157 |
# apt install linstor-controller linstor-satellite linstor-client |
| 158 |
-------------------------------------------------------------------- |
| 159 |
|
| 160 |
That will make our remaining nodes our Satellites, so we'll need to install the following packages on them: |
| 161 |
|
| 162 |
------------------------------------------------- |
| 163 |
# apt install linstor-satellite linstor-client |
| 164 |
------------------------------------------------- |
| 165 |
|
| 166 |
|
| 167 |
[[s-admin-suse_linux_enterprise_server]] |
| 168 |
==== SUSE Linux Enterprise Server |
| 169 |
|
| 170 |
|
| 171 |
SLES High Availability Extension (HAE) includes DRBD. |
| 172 |
|
| 173 |
On SLES, DRBD is normally installed via the software installation component of YaST2. It comes bundled with the High Availability |
| 174 |
package selection. |
| 175 |
|
| 176 |
As we download DRBD's newest module we can check if the LVM-tools are up to date as well. User who prefer a command line install |
| 177 |
may simply issue the following command to get the newest DRBD and LVM version: |
| 178 |
|
| 179 |
---------------------- |
| 180 |
# zypper install drbd lvm2 |
| 181 |
---------------------- |
| 182 |
|
| 183 |
|
| 184 |
|
| 185 |
Depending on whether your node is a LINSTOR controller, satellite, or both (Combined) will determine what packages |
| 186 |
are required on that node. For combined type nodes, we'll need both the controller and satellite LINSTOR package. |
| 187 |
|
| 188 |
Combined node: |
| 189 |
|
| 190 |
------------------------------------------------------------------------ |
| 191 |
# zypper install linstor-controller linstor-satellite linstor-client |
| 192 |
------------------------------------------------------------------------ |
| 193 |
|
| 194 |
That will make our remaining nodes our Satellites, so we'll need to install the following packages on them: |
| 195 |
|
| 196 |
---------------------------------------------------- |
| 197 |
# zypper install linstor-satellite linstor-client |
| 198 |
---------------------------------------------------- |
| 199 |
|
| 200 |
|
| 201 |
[[s-admin-centos]] |
| 202 |
==== CentOS |
| 203 |
|
| 204 |
CentOS has had DRBD 8 since release 5. For DRBD 9 you'll need to look at EPEL and similar sources. |
| 205 |
Alternatively, if you have an active support contract with LINBIT you can utilize our RHEL 8 repositories. |
| 206 |
DRBD can be installed using `yum`. We can also check for the newest version of the LVM-tools as well. |
| 207 |
|
| 208 |
IMPORTANT: LINSTOR *requires* DRBD 9 if you wish to have replicated storage. This requires an external |
| 209 |
repository to be configured, either LINBIT's or a 3rd parties. |
| 210 |
|
| 211 |
----------------------------- |
| 212 |
# yum install drbd kmod-drbd lvm2 |
| 213 |
----------------------------- |
| 214 |
|
| 215 |
Depending on whether your node is a LINSTOR controller, satellite, or both (Combined) will determine what packages |
| 216 |
are required on that node. For combined type nodes, we'll need both the controller and satellite LINSTOR package. |
| 217 |
|
| 218 |
NOTE: On RHEL 8 systems you will need to install python2 for the linstor-client to work. |
| 219 |
|
| 220 |
Combined node: |
| 221 |
|
| 222 |
------------------------------------------------------------------- |
| 223 |
# yum install linstor-controller linstor-satellite linstor-client |
| 224 |
------------------------------------------------------------------- |
| 225 |
|
| 226 |
That will make our remaining nodes our Satellites, so we'll need to install the following packages on them: |
| 227 |
|
| 228 |
------------------------------------------------ |
| 229 |
# yum install linstor-satellite linstor-client |
| 230 |
------------------------------------------------ |
| 231 |
|
| 232 |
|
| 233 |
[[s-upgrading]] |
| 234 |
=== Upgrading |
| 235 |
|
| 236 |
LINSTOR doesn't support rolling upgrade, controller and satellites must have the same version, otherwise |
| 237 |
the controller with discard the satellite with a `VERSION_MISMATCH`. |
| 238 |
But this isn't a problem, as the satellite won't do any actions as long it isn't connected to a controller |
| 239 |
and DRBD will not be disrupted by any means. |
| 240 |
|
| 241 |
If you are using the embedded default H2 database and the linstor-controller package is upgraded an automatic |
| 242 |
backup file of the database will be created in the default `/var/lib/linstor` directory. |
| 243 |
This file is a good restore point if for any reason a linstor-controller database migration should fail, |
| 244 |
than it is recommended to report the error to LINBIT and restore the old database file and downgrade to your previous |
| 245 |
controller version. |
| 246 |
|
| 247 |
If you use any external database or etcd, it is recommended to do a manually backup of your current database to have |
| 248 |
a restore point. |
| 249 |
|
| 250 |
So first upgrade the `linstor-controller`, `linstor-client` package on you controller host and restart the `linstor-controller`, |
| 251 |
the controller should start and all of it's client should show `OFFLINE(VERSION_MISMATCH)`. |
| 252 |
After that you can continue upgrading `linstor-satellite` on all satellite nodes and restart them, after a short reconnection time |
| 253 |
they should all show `ONLINE` again and your upgrade is finished. |
| 254 |
|
| 255 |
|
| 256 |
[[s-containers]] |
| 257 |
=== Containers |
| 258 |
|
| 259 |
LINSTOR is also available as containers. The base images are available |
| 260 |
in LINBIT's container registry, `drbd.io`. |
| 261 |
|
| 262 |
In order to access the images, you first have to login to the |
| 263 |
registry (reach out to [email protected] for credentials): |
| 264 |
|
| 265 |
---------------------------- |
| 266 |
# docker login drbd.io |
| 267 |
---------------------------- |
| 268 |
|
| 269 |
The containers available in this repository are: |
| 270 |
|
| 271 |
* drbd.io/drbd9-rhel8 |
| 272 |
* drbd.io/drbd9-rhel7 |
| 273 |
* drbd.io/drbd9-sles15sp1 |
| 274 |
* drbd.io/drbd9-bionic |
| 275 |
* drbd.io/drbd9-focal |
| 276 |
* drbd.io/linstor-csi |
| 277 |
* drbd.io/linstor-controller |
| 278 |
* drbd.io/linstor-satellite |
| 279 |
* drbd.io/linstor-client |
| 280 |
|
| 281 |
An up to date list of available images with versions can be retrieved by opening http://drbd.io in your |
| 282 |
browser. Make sure to access the host via "http", as the registry's images themselves are served via "https". |
| 283 |
|
| 284 |
To load the kernel module, needed only for LINSTOR satellites, you'll need to |
| 285 |
run a `drbd9-$dist` container in privileged mode. The kernel module containers |
| 286 |
either retrieve an official LINBIT package from a customer repository, use |
| 287 |
shipped packages, or they try to build the kernel modules from source. If you |
| 288 |
intend to build from source, you need to have the according kernel headers |
| 289 |
(e.g., `kernel-devel`) installed on the host. There are 4 ways to execute such |
| 290 |
a module load container: |
| 291 |
|
| 292 |
* Building from shipped source |
| 293 |
* Using a shipped/pre-built kernel module |
| 294 |
* Specifying a LINBIT node hash and a distribution. |
| 295 |
* Bind-mounting an existing repository configuration. |
| 296 |
|
| 297 |
Example building from shipped source (RHEL based): |
| 298 |
|
| 299 |
---------------------------- |
| 300 |
# docker run -it --rm --privileged -v /lib/modules:/lib/modules \ |
| 301 |
-v /usr/src:/usr/src:ro \ |
| 302 |
drbd.io/drbd9-rhel7 |
| 303 |
---------------------------- |
| 304 |
|
| 305 |
Example using a module shipped with the container, which is enabled by *not* bind-mounting `/usr/src`: |
| 306 |
|
| 307 |
---------------------------- |
| 308 |
# docker run -it --rm --privileged -v /lib/modules:/lib/modules \ |
| 309 |
drbd.io/drbd9-rhel8 |
| 310 |
---------------------------- |
| 311 |
|
| 312 |
Example using a hash and a distribution (rarely used): |
| 313 |
|
| 314 |
---------------------------- |
| 315 |
# docker run -it --rm --privileged -v /lib/modules:/lib/modules \ |
| 316 |
-e LB_DIST=rhel7.7 -e LB_HASH=ThisIsMyNodeHash \ |
| 317 |
drbd.io/drbd9-rhel7 |
| 318 |
---------------------------- |
| 319 |
|
| 320 |
Example using an existing repo config (rarely used): |
| 321 |
|
| 322 |
---------------------------- |
| 323 |
# docker run -it --rm --privileged -v /lib/modules:/lib/modules \ |
| 324 |
-v /etc/yum.repos.d/linbit.repo:/etc/yum.repos.d/linbit.repo:ro \ |
| 325 |
drbd.io/drbd9-rhel7 |
| 326 |
---------------------------- |
| 327 |
|
| 328 |
IMPORTANT: In both cases (hash + distribution, as well as bind-mounting a repo) |
| 329 |
the hash or config has to be from a node that has a special property set. Feel |
| 330 |
free to contact our support, and we set this property. |
| 331 |
|
| 332 |
IMPORTANT: For now (i.e., pre DRBD 9 version "9.0.17"), you must use the containerized DRBD kernel module, |
| 333 |
as opposed to loading a kernel module onto the host system. If you |
| 334 |
intend to use the containers you should not install the DRBD kernel |
| 335 |
module on your host systems. For DRBD version 9.0.17 or greater, you can install the kernel module as usual on |
| 336 |
the host system, but you need to make sure to load the module with the `usermode_helper=disabled` parameter |
| 337 |
(e.g., `modprobe drbd usermode_helper=disabled`). |
| 338 |
|
| 339 |
Then run the LINSTOR satellite container, also privileged, as a daemon: |
| 340 |
|
| 341 |
---------------------------- |
| 342 |
# docker run -d --name=linstor-satellite --net=host -v /dev:/dev --privileged drbd.io/linstor-satellite |
| 343 |
---------------------------- |
| 344 |
|
| 345 |
NOTE: `net=host` is required for the containerized `drbd-utils` to be |
| 346 |
able to communicate with the host-kernel via netlink. |
| 347 |
|
| 348 |
To run the LINSTOR controller container as a daemon, mapping ports |
| 349 |
`3370`, `3376` and `3377` on the host to the container: |
| 350 |
|
| 351 |
---------------------------- |
| 352 |
# docker run -d --name=linstor-controller -p 3370:3370 -p 3376:3376 -p 3377:3377 drbd.io/linstor-controller |
| 353 |
---------------------------- |
| 354 |
|
| 355 |
To interact with the containerized LINSTOR cluster, you can either use |
| 356 |
a LINSTOR client installed on a system via packages, or via the |
| 357 |
containerized LINSTOR client. To use the LINSTOR client container: |
| 358 |
|
| 359 |
---------------------------- |
| 360 |
# docker run -it --rm -e LS_CONTROLLERS=<controller-host-IP-address> drbd.io/linstor-client node list |
| 361 |
---------------------------- |
| 362 |
|
| 363 |
From this point you would use the LINSTOR client to initialize your |
| 364 |
cluster and begin creating resources using the typical LINSTOR |
| 365 |
patterns. |
| 366 |
|
| 367 |
To stop and remove a daemonized container and image: |
| 368 |
|
| 369 |
---------------------------- |
| 370 |
# docker stop linstor-controller |
| 371 |
# docker rm linstor-controller |
| 372 |
---------------------------- |
| 373 |
|
| 374 |
[[s-linstor-init-cluster]] |
| 375 |
=== Initializing your cluster |
| 376 |
We assume that the following steps are accomplished on *all* cluster nodes: |
| 377 |
|
| 378 |
. The DRBD9 kernel module is installed and loaded |
| 379 |
. `drbd-utils` are installed |
| 380 |
. `LVM` tools are installed |
| 381 |
. `linstor-controller` and/or `linstor-satellite` its dependencies are installed |
| 382 |
. The `linstor-client` is installed on the `linstor-controller` node |
| 383 |
|
| 384 |
Start and enable the linstor-controller service on the host where it has been installed: |
| 385 |
|
| 386 |
---------------------------- |
| 387 |
# systemctl enable --now linstor-controller |
| 388 |
---------------------------- |
| 389 |
|
| 390 |
If you are sure the linstor-controller service gets automatically enabled on installation you can use the |
| 391 |
following command as well: |
| 392 |
|
| 393 |
------------------------------------------ |
| 394 |
# systemctl start linstor-controller |
| 395 |
------------------------------------------ |
| 396 |
|
| 397 |
[[s-using_the_linstor_client]] |
| 398 |
=== Using the LINSTOR client |
| 399 |
Whenever you run the LINSTOR command line client, it needs to know where your |
| 400 |
linstor-controller runs. If you do not specify it, it will try to reach a locally |
| 401 |
running linstor-controller listening on IP `127.0.0.1` port `3376`. Therefore we |
| 402 |
will use the `linstor-client` on the same host as the `linstor-controller`. |
| 403 |
|
| 404 |
IMPORTANT: The `linstor-satellite` requires ports 3366 and 3367. The `linstor-controller` |
| 405 |
requires ports 3376 and 3377. Make sure you have these ports allowed on your firewall. |
| 406 |
|
| 407 |
---------------------------- |
| 408 |
# linstor node list |
| 409 |
---------------------------- |
| 410 |
should give you an empty list and not an error message. |
| 411 |
|
| 412 |
You can use the `linstor` command on any other machine, but then you need |
| 413 |
to tell the client how to find the linstor-controller. As shown, this can be |
| 414 |
specified as a command line option, an environment variable, or in a global |
| 415 |
file: |
| 416 |
|
| 417 |
---------------------------- |
| 418 |
# linstor --controllers=alice node list |
| 419 |
# LS_CONTROLLERS=alice linstor node list |
| 420 |
---------------------------- |
| 421 |
|
| 422 |
Alternatively you can create the `/etc/linstor/linstor-client.conf` |
| 423 |
file and populate it like below. |
| 424 |
|
| 425 |
|
| 426 |
----- |
| 427 |
[global] |
| 428 |
controllers=alice |
| 429 |
----- |
| 430 |
|
| 431 |
If you have multiple linstor-controllers configured you can simply |
| 432 |
specify them all in a comma separated list. The linstor-client will |
| 433 |
simply try them in the order listed. |
| 434 |
|
| 435 |
|
| 436 |
NOTE: The linstor-client commands can also be used in a much faster |
| 437 |
and convenient way by only writing the starting letters of the parameters |
| 438 |
e.g.: `linstor node list` -> `linstor n l` |
| 439 |
|
| 440 |
[[s-adding_nodes_to_your_cluster]] |
| 441 |
=== Adding nodes to your cluster |
| 442 |
The next step is to add nodes to your LINSTOR cluster. |
| 443 |
|
| 444 |
---------------------------- |
| 445 |
# linstor node create bravo 10.43.70.3 |
| 446 |
---------------------------- |
| 447 |
|
| 448 |
If the IP is omitted, the client will try to resolve the given node-name as |
| 449 |
host-name by itself. |
| 450 |
|
| 451 |
Linstor will automatically detect the node's local `uname -n` which is |
| 452 |
later used for the DRBD-resource. |
| 453 |
|
| 454 |
When you use `linstor node list` you will see that the new node |
| 455 |
is marked as offline. Now start and enable the linstor-satellite on that node |
| 456 |
so that the service comes up on reboot as well: |
| 457 |
---------------------------- |
| 458 |
# systemctl enable --now linstor-satellite |
| 459 |
---------------------------- |
| 460 |
|
| 461 |
You can also use `systemctl start linstor-satellite` |
| 462 |
if you are sure that the service is already enabled as default and comes up on |
| 463 |
reboot. |
| 464 |
|
| 465 |
About 10 seconds later you will see the status in `linstor node list` |
| 466 |
becoming online. Of course the satellite process may be started before |
| 467 |
the controller knows about the existence of the satellite node. |
| 468 |
|
| 469 |
NOTE: In case the node which hosts your controller should also contribute |
| 470 |
storage to the LINSTOR cluster, you have to add it as a node and start |
| 471 |
the linstor-satellite as well. |
| 472 |
|
| 473 |
If you want to have other services wait until the linstor-satellite had a chance |
| 474 |
to create the necessary devices (i.e. after a boot), you can update the |
| 475 |
corresponding .service file and change `Type=simple` to `Type=notify`. |
| 476 |
|
| 477 |
This will cause the satellite to delay sending the `READY=1` message to systemd |
| 478 |
until the controller connects, sends all required data to the satellite and the |
| 479 |
satellite at least tried once to get the devices up and running. |
| 480 |
|
| 481 |
[[s-storage_pools]] |
| 482 |
=== Storage pools |
| 483 |
|
| 484 |
<<StoragePool,StoragePools>> identify storage in the context of LINSTOR. |
| 485 |
To group storage pools from multiple nodes, simply use the same name |
| 486 |
on each node. |
| 487 |
For example, one valid approach is to give all SSDs one name and |
| 488 |
all HDDs another. |
| 489 |
|
| 490 |
On each host contributing storage, you need to create |
| 491 |
either an LVM VG or a ZFS zPool. The VGs and zPools identified with one |
| 492 |
LINSTOR storage pool name may have different VG or zPool names on the |
| 493 |
hosts, but do yourself a favor and use the same VG or zPool name on all |
| 494 |
nodes. |
| 495 |
|
| 496 |
---------------------------- |
| 497 |
# vgcreate vg_ssd /dev/nvme0n1 /dev/nvme1n1 [...] |
| 498 |
---------------------------- |
| 499 |
|
| 500 |
These then need to be registered with LINSTOR: |
| 501 |
|
| 502 |
---------------------------- |
| 503 |
# linstor storage-pool create lvm alpha pool_ssd vg_ssd |
| 504 |
# linstor storage-pool create lvm bravo pool_ssd vg_ssd |
| 505 |
---------------------------- |
| 506 |
|
| 507 |
NOTE: The storage pool name and common metadata is referred to as a |
| 508 |
_storage pool definition_. |
| 509 |
The listed commands create a storage pool definition implicitly. |
| 510 |
You can see that by using `linstor storage-pool-definition list`. |
| 511 |
Creating storage pool definitions explicitly is possible but |
| 512 |
not necessary. |
| 513 |
|
| 514 |
To list your storage-pools you can use: |
| 515 |
|
| 516 |
------ |
| 517 |
# linstor storage-pool list |
| 518 |
------ |
| 519 |
|
| 520 |
or using the short version |
| 521 |
|
| 522 |
----- |
| 523 |
# linstor sp l |
| 524 |
----- |
| 525 |
|
| 526 |
//// |
| 527 |
In case anything goes wrong with the storage pool's VG/zPool, e.g. the VG having been renamed or somehow |
| 528 |
became invalid you can delete the storage pool in LINSTOR with the following command, given that only |
| 529 |
resources with all their volumes in the so-called 'lost' storage pool are attached. This feature is available |
| 530 |
since LINSTOR v0.9.13. |
| 531 |
|
| 532 |
------ |
| 533 |
# linstor storage-pool lost alpha pool_ssd |
| 534 |
------ |
| 535 |
|
| 536 |
or using the short version |
| 537 |
|
| 538 |
----- |
| 539 |
# linstor sp lo alpha pool_ssd |
| 540 |
----- |
| 541 |
//// |
| 542 |
|
| 543 |
Should the deletion of the storage pool be prevented due to attached resources or snapshots with some of its |
| 544 |
volumes in another still functional storage pool, hints will be given in the 'status' column of the |
| 545 |
corresponding list-command (e.g. `linstor resource list`). After deleting the LINSTOR-objects in the lost storage pool |
| 546 |
manually, the lost-command can be executed again to ensure a complete deletion of the storage pool and its |
| 547 |
remaining objects. |
| 548 |
|
| 549 |
[[s-a_storage_pool_per_backend_device]] |
| 550 |
==== A storage pool per backend device |
| 551 |
|
| 552 |
In clusters where you have only one kind of storage and the capability |
| 553 |
to hot-repair storage devices, you may choose a model where you create |
| 554 |
one storage pool per physical backing device. The advantage of this |
| 555 |
model is to confine failure domains to a single storage device. |
| 556 |
|
| 557 |
[[s-physical-storage-command]] |
| 558 |
==== Physical storage command |
| 559 |
|
| 560 |
Since linstor-server 1.5.2 and a recent linstor-client, LINSTOR can create LVM/ZFS pools on a satellite for you. |
| 561 |
The linstor-client has the following commands to list possible disks and create storage pools, but such LVM/ZFS pools |
| 562 |
are not managed by LINSTOR and there is no delete command, so such action must be done manually on the nodes. |
| 563 |
|
| 564 |
----- |
| 565 |
# linstor physical-storage list |
| 566 |
----- |
| 567 |
|
| 568 |
Will give you a list of available disks grouped by size and rotational(SSD/Magnetic Disk). |
| 569 |
|
| 570 |
It will only show disks that pass the following filters: |
| 571 |
|
| 572 |
* The device size must be greater than 1GiB |
| 573 |
* The device is a root device (not having children) e.g.: /dev/vda, /dev/sda |
| 574 |
* The device does not have any file-system or other `blkid` marker (`wipefs -a` might be needed) |
| 575 |
* The device is no DRBD device |
| 576 |
|
| 577 |
|
| 578 |
With the `create-device-pool` command you can create a LVM pool on a disk and also directly |
| 579 |
add it as a storage-pool in LINSTOR. |
| 580 |
|
| 581 |
---- |
| 582 |
# linstor physical-storage create-device-pool --pool-name lv_my_pool LVMTHIN node_alpha /dev/vdc --storage-pool newpool |
| 583 |
---- |
| 584 |
|
| 585 |
If the `--storage-pool` option was provided, LINSTOR will create a storage-pool with the given name. |
| 586 |
|
| 587 |
For more options and exact command usage please check the linstor-client help. |
| 588 |
|
| 589 |
[[s-linstor-resource-groups]] |
| 590 |
=== Resource groups |
| 591 |
|
| 592 |
A resource group is a parent object of a resource definition where all |
| 593 |
property changes made on a resource group will be inherited by it's |
| 594 |
resource definition children. The resource group also stores settings |
| 595 |
for automatic placement rules and can spawn a resource definition |
| 596 |
depending on the stored rules. |
| 597 |
|
| 598 |
In simpler terms, resource groups are like templates that define |
| 599 |
characteristics of resources created from them. Changes to these |
| 600 |
pseudo templates will be applied to all resources that were created |
| 601 |
from the resource group, retroactively. |
| 602 |
|
| 603 |
TIP: Using resource groups to define how you'd like your resources |
| 604 |
provisioned should be considered the de facto method for deploying |
| 605 |
volumes provisioned by LINSTOR. Chapters that follow which describe |
| 606 |
creating each _resource_ from a _resource-definition_ and |
| 607 |
_volume-definition_ should only be used in special scenarios. |
| 608 |
|
| 609 |
NOTE: Even if you choose not to create and use _resource-groups_ in |
| 610 |
your LINSTOR cluster, all resources created from |
| 611 |
_resource-definitions_ and _volume-definitions_ will exist in the |
| 612 |
'DfltRscGrp' _resource-group_. |
| 613 |
|
| 614 |
A simple pattern for deploying resources using resource groups would |
| 615 |
look like this: |
| 616 |
|
| 617 |
---- |
| 618 |
# linstor resource-group create my_ssd_group --storage-pool pool_ssd --place-count 2 |
| 619 |
# linstor volume-group create my_ssd_group |
| 620 |
# linstor resource-group spawn-resources my_ssd_group my_ssd_res 20G |
| 621 |
---- |
| 622 |
|
| 623 |
The commands above would result in a resource named 'my_ssd_res' with a |
| 624 |
20GB volume replicated twice being automatically provisioned from nodes who |
| 625 |
participate in the storage pool named 'pool_ssd'. |
| 626 |
|
| 627 |
A more useful pattern could be to create a resource group with |
| 628 |
settings you've determined are optimal for your use case. Perhaps |
| 629 |
you have to run nightly online verifications of your volumes' |
| 630 |
consistency, in that case, you could create a resource group with the |
| 631 |
'verify-alg' of your choice already set so that resources spawned from |
| 632 |
the group are pre-configured with 'verify-alg' set: |
| 633 |
|
| 634 |
---- |
| 635 |
# linstor resource-group create my_verify_group --storage-pool pool_ssd --place-count 2 |
| 636 |
# linstor resource-group drbd-options --verify-alg crc32c my_verify_group |
| 637 |
# linstor volume-group create my_verify_group |
| 638 |
# for i in {00..19}; do |
| 639 |
linstor resource-group spawn-resources my_verify_group res$i 10G |
| 640 |
done |
| 641 |
---- |
| 642 |
|
| 643 |
The commands above result in twenty 10GiB resources being created each |
| 644 |
with the 'crc32c' 'verify-alg' pre-configured. |
| 645 |
|
| 646 |
You can tune the settings of individual resources or volumes spawned |
| 647 |
from resource groups by setting options on the respective |
| 648 |
_resource-definition_ or _volume-definition_. For example, if 'res11' |
| 649 |
from the example above is used by a very active database receiving |
| 650 |
lots of small random writes, you might want to increase the |
| 651 |
'al-extents' for that specific resource: |
| 652 |
|
| 653 |
---- |
| 654 |
# linstor resource-definition drbd-options --al-extents 6007 res11 |
| 655 |
---- |
| 656 |
|
| 657 |
If you configure a setting in a _resource-definition_ that is already |
| 658 |
configured on the _resource-group_ it was spawned from, the value set |
| 659 |
in the _resource-definition_ will override the value set on the parent |
| 660 |
_resource-group_. For example, if the same 'res11' was required to use |
| 661 |
the slower but more secure 'sha256' hash algorithm in its |
| 662 |
verifications, setting the 'verify-alg' on the _resource-definition_ |
| 663 |
for 'res11' would override the value set on the _resource-group_: |
| 664 |
|
| 665 |
---- |
| 666 |
# linstor resource-definition drbd-options --verify-alg sha256 res11 |
| 667 |
---- |
| 668 |
|
| 669 |
TIP: A rule of thumb for the hierarchy in which settings are inherited |
| 670 |
is the value "closer" to the resource or volume wins: _volume-definition_ |
| 671 |
settings take precedence over _volume-group_ settings, and |
| 672 |
_resource-definition_ settings take precedence over _resource-group_ |
| 673 |
settings. |
| 674 |
|
| 675 |
|
| 676 |
[[s-linstor-set-config]] |
| 677 |
=== Cluster configuration |
| 678 |
|
| 679 |
[[s-available_storage_plugins]] |
| 680 |
==== Available storage plugins |
| 681 |
|
| 682 |
indexterm:[linstor, storage plugins] |
| 683 |
|
| 684 |
LINSTOR has the following supported storage plugins as of writing: |
| 685 |
|
| 686 |
* Thick LVM |
| 687 |
|
| 688 |
* Thin LVM with a single thin pool |
| 689 |
|
| 690 |
* Thick ZFS |
| 691 |
|
| 692 |
* Thin ZFS |
| 693 |
|
| 694 |
[[s-linstor-new-volume]] |
| 695 |
=== Creating and deploying resources/volumes |
| 696 |
In the following scenario we assume that the goal is to create a resource |
| 697 |
'backups' with a size of '500 GB' that is replicated among three cluster nodes. |
| 698 |
|
| 699 |
First, we create a new resource definition: |
| 700 |
|
| 701 |
---------------------------- |
| 702 |
# linstor resource-definition create backups |
| 703 |
---------------------------- |
| 704 |
|
| 705 |
Second, we create a new volume definition within that resource definition: |
| 706 |
|
| 707 |
---------------------------- |
| 708 |
# linstor volume-definition create backups 500G |
| 709 |
---------------------------- |
| 710 |
|
| 711 |
If you want to change the size of the volume-definition you can simply do that by: |
| 712 |
|
| 713 |
------- |
| 714 |
# linstor volume-definition set-size backups 0 100G |
| 715 |
------- |
| 716 |
|
| 717 |
The parameter `0` is the number of the volume in the resource `backups`. You have to provide this parameter |
| 718 |
, because resources can have multiple volumes and they are identified by a so called volume-number. This number |
| 719 |
can be found by listing the volume-definitions. |
| 720 |
|
| 721 |
IMPORTANT: The size of a volume-definition can only be decreased if it has no resource. Despite |
| 722 |
of that the size can be increased even with an deployed resource. |
| 723 |
|
| 724 |
So far we have only created objects in LINSTOR's database, not a single LV was |
| 725 |
created on the storage nodes. Now you have the choice of delegating the |
| 726 |
task of placement to LINSTOR or doing it yourself. |
| 727 |
|
| 728 |
[[s-manual_placement]] |
| 729 |
==== Manual placement |
| 730 |
|
| 731 |
With the `resource create` command you may assign a resource definition |
| 732 |
to named nodes explicitly. |
| 733 |
|
| 734 |
---------------------------- |
| 735 |
# linstor resource create alpha backups --storage-pool pool_hdd |
| 736 |
# linstor resource create bravo backups --storage-pool pool_hdd |
| 737 |
# linstor resource create charlie backups --storage-pool pool_hdd |
| 738 |
---------------------------- |
| 739 |
|
| 740 |
[[s-autoplace-linstor]] |
| 741 |
==== Autoplace |
| 742 |
|
| 743 |
The value after autoplace tells LINSTOR how many replicas you want to have. |
| 744 |
The storage-pool option should be obvious. |
| 745 |
---------------------------- |
| 746 |
# linstor resource create backups --auto-place 3 --storage-pool pool_hdd |
| 747 |
---------------------------- |
| 748 |
Maybe not so obvious is that you may omit the `--storage-pool` option, then |
| 749 |
LINSTOR may select a storage pool on its own. The selection follows these rules: |
| 750 |
|
| 751 |
* Ignore all nodes and storage pools the current user has no access to |
| 752 |
* Ignore all diskless storage pools |
| 753 |
* Ignore all storage pools not having enough free space |
| 754 |
|
| 755 |
The remaining storage pools will be rated by different strategies. |
| 756 |
LINSTOR has currently three strategies: |
| 757 |
|
| 758 |
* `MaxFreeSpace`: This strategy maps the rating 1:1 to the remaining free |
| 759 |
space of the storage pool. However, this strategy only considers the actually |
| 760 |
allocated space (in case of thinly provisioned storage pool this might grow |
| 761 |
with time without creating new resources) |
| 762 |
* `MinReservedSpace`: Unlike the "MaxFreeSpace", this strategy considers the |
| 763 |
reserved spaced. That is the space that a thin volume can grow to before reaching |
| 764 |
its limit. The sum of reserved spaces might exceed the storage pools capacity, |
| 765 |
which is as overprovisioning. |
| 766 |
* `MinRscCount`: Simply the count of resources already deployed in a given |
| 767 |
storage pool |
| 768 |
* `MaxThroughput`: For this strategy, the storage pool's |
| 769 |
`Autoplacer/MaxThroughput` property is the base of the score, or 0 if the property |
| 770 |
is not present. Every Volume deployed in the given storage pool will subtract |
| 771 |
its defined `sys/fs/blkio_throttle_read` and `sys/fs/blkio_throttle_write` property- |
| 772 |
value from the storage pool's max throughput. The resulting score might be negative. |
| 773 |
|
| 774 |
The scores of the strategies will be normalized, weighted and summed up, where |
| 775 |
the scores of minimizing strategies will be converted first to allow an overall |
| 776 |
maximization of the resulting score. |
| 777 |
|
| 778 |
The weights of the strategies can be configured with |
| 779 |
|
| 780 |
---- |
| 781 |
linstor controller set-property Autoplacer/Weights/$name_of_the_strategy $weight |
| 782 |
---- |
| 783 |
whereas the strategy-names are listed above and the weight can be an arbitrary |
| 784 |
decimal. |
| 785 |
|
| 786 |
NOTE: To keep the behaviour of the autoplacer similar to the old one (due to |
| 787 |
compatibility), all strategies have a default-weight of 0, except the `MaxFreeSpace` |
| 788 |
which has a weight of 1. |
| 789 |
|
| 790 |
NOTE: Neither 0 nor a negative score will prevent a storage pool from getting |
| 791 |
selected, just making them to be considered later. |
| 792 |
|
| 793 |
Finally LINSTOR tries to find the best matching group of storage pools meeting |
| 794 |
all requirements. This step also considers other autoplacement restrictions |
| 795 |
as `--replicas-on-same`, `--replicas-on-different` and others. |
| 796 |
|
| 797 |
These two arguments, `--replicas-on-same` and `--replicas-on-different` expect the |
| 798 |
name of a property within the `Aux/` namespace. The following example shows that the |
| 799 |
client automatically prefixes the `testProperty` with the `Aux/` namespace. |
| 800 |
|
| 801 |
---- |
| 802 |
linstor resource-group create testRscGrp --replicas-on-same testProperty |
| 803 |
SUCCESS: |
| 804 |
Description: |
| 805 |
New resource group 'testRscGrp' created. |
| 806 |
Details: |
| 807 |
Resource group 'testRscGrp' UUID is: 35e043cb-65ab-49ac-9920-ccbf48f7e27d |
| 808 |
|
| 809 |
linstor resource-group list |
| 810 |
+-----------------------------------------------------------------------------+ |
| 811 |
| ResourceGroup | SelectFilter | VlmNrs | Description | |
| 812 |
|-============================================================================| |
| 813 |
| DfltRscGrp | PlaceCount: 2 | | | |
| 814 |
|-----------------------------------------------------------------------------| |
| 815 |
| testRscGrp | PlaceCount: 2 | | | |
| 816 |
| | ReplicasOnSame: ['Aux/testProperty'] | | | |
| 817 |
+-----------------------------------------------------------------------------+ |
| 818 |
---- |
| 819 |
|
| 820 |
NOTE: If everything went right the DRBD-resource has now been created by LINSTOR. |
| 821 |
This can be checked by looking for the DRBD block device with the `lsblk` |
| 822 |
command which should look like `drbd0000` or similar. |
| 823 |
|
| 824 |
|
| 825 |
Now we should be able to mount the block device of our resource and start using |
| 826 |
LINSTOR. |
| 827 |
|
| 828 |
[[s-more-about-linstor]] |
| 829 |
== Further LINSTOR tasks |
| 830 |
|
| 831 |
[[s-linstor_ha]] |
| 832 |
=== LINSTOR high availability |
| 833 |
By default a LINSTOR cluster consists of exactly one LINSTOR controller. Making LINSTOR highly-available |
| 834 |
involves providing replicated storage for the controller database, multiple LINSTOR controllers where only one |
| 835 |
is active at a time, and a service manager that takes care of mounting/unmounting the highly-available storage |
| 836 |
and starting/stopping LINSTOR controllers. |
| 837 |
|
| 838 |
==== Highly-available Storage |
| 839 |
For configuring the highly-available storage we use LINSTOR itself. This has the advantage that the storage is |
| 840 |
under LINSTOR control and can for example be easily extended to new cluster nodes. Just create a new |
| 841 |
resource with 200MB in size. This could look like this, you certainly need to adapt the storage pool name: |
| 842 |
|
| 843 |
---------------------------- |
| 844 |
# linstor resource-definition create linstor_db |
| 845 |
# linstor resource-definition drbd-options --on-no-quorum=io-error linstor_db |
| 846 |
# linstor resource-definition drbd-options --auto-promote=no linstor_db |
| 847 |
# linstor volume-definition create linstor_db 200M |
| 848 |
# linstor resource create linstor_db -s pool1 --auto-place 3 |
| 849 |
---------------------------- |
| 850 |
|
| 851 |
From now on we assume the resource's name is "linstor_db". It is crucial that your |
| 852 |
cluster qualifies for auto-quorum and uses the `io-error` policy (see Section <<s-linstor-auto-quorum>>), and |
| 853 |
that `auto-promote` is disabled. |
| 854 |
|
| 855 |
After the resource is created, it is time to move the LINSTOR DB to the new storage and to create a `systemd` |
| 856 |
mount service. First we stop the current controller and disable it, as it will be managed by `drbd-reactor` later. |
| 857 |
|
| 858 |
---------------------------- |
| 859 |
# systemctl disable --now linstor-controller |
| 860 |
|
| 861 |
# cat << EOF > /etc/systemd/system/var-lib-linstor.mount |
| 862 |
[Unit] |
| 863 |
Description=Filesystem for the LINSTOR controller |
| 864 |
|
| 865 |
[Mount] |
| 866 |
# you can use the minor like /dev/drbdX or the udev symlink |
| 867 |
What=/dev/drbd/by-res/linstor_db/0 |
| 868 |
Where=/var/lib/linstor |
| 869 |
EOF |
| 870 |
|
| 871 |
# mv /var/lib/linstor{,.orig} |
| 872 |
# mkdir /var/lib/linstor |
| 873 |
# chattr +i /var/lib/linstor # only if on LINSTOR >= 1.14.0 |
| 874 |
# drbdadm primary linstor_db |
| 875 |
# mkfs.ext4 /dev/drbd/by-res/linstor_db/0 |
| 876 |
# systemctl start var-lib-linstor.mount |
| 877 |
# cp -r /var/lib/linstor.orig/* /var/lib/linstor |
| 878 |
# systemctl start linstor-controller |
| 879 |
---------------------------- |
| 880 |
|
| 881 |
copy the `/etc/systemd/system/var-lib-linstor.mount` mount file to all the standby nodes for the linstor controller. |
| 882 |
Again, do not `systemctl enable` any of these services, they get managed by `drbd-reactor`. |
| 883 |
|
| 884 |
==== Multiple LINSTOR controllers |
| 885 |
The next step is to install LINSTOR controllers on all nodes that have access |
| 886 |
to the `linstor_db` DRBD resource (as they need to mount the DRBD volume) and |
| 887 |
which you want to become a possible LINSTOR controller. It is important that the controllers |
| 888 |
are manged by `drbd-reactor`, so make sure the `linstor-controller.service` is |
| 889 |
disabled on all nodes! To be sure, execute `systemctl disable linstor-controller` |
| 890 |
on all cluster nodes and `systemctl stop linstor-controller` on all nodes except the one it is currently |
| 891 |
running from the previous step. Also make sure to set `chattr +i /var/lib/linstor` on all potential controller |
| 892 |
nodes if you use LINSTOR version equal or greater to 1.14.0. |
| 893 |
|
| 894 |
==== Managing the services |
| 895 |
For starting and stopping the mount service and the linstor-controller service we use `drbd-reactor`. Install this |
| 896 |
component on all nodes that could become a LINSTOR controller and edit their `/etc/drbd-reactor.d/linstor_db.toml` configuration |
| 897 |
file. It should contain an enabled promoter plugin section like this: |
| 898 |
|
| 899 |
---------------------------- |
| 900 |
[[promoter]] |
| 901 |
id = "linstor_db" |
| 902 |
[promoter.resources.linstor_db] |
| 903 |
start = ["var-lib-linstor.mount", "linstor-controller.service"] |
| 904 |
---------------------------- |
| 905 |
|
| 906 |
Depending on your requirements you might also want to set an `on-stop-failure` action and set `stop-services-on-exit`. |
| 907 |
|
| 908 |
After that restart `drbd-reactor` and enable it on all the nodes you configured it. |
| 909 |
---------------------------- |
| 910 |
# systemctl restart drbd-reactor |
| 911 |
# systemctl enable drbd-reactor |
| 912 |
---------------------------- |
| 913 |
|
| 914 |
Check that there are no warnings from drbd-reactor in the logs by running `systemctl status drbd-reactor`. |
| 915 |
As there is already an active LINSTOR controller things will just stay the way they are. |
| 916 |
Run `drbd-reactorctl status linstor_db` to check the health of the linstor_db target unit. |
| 917 |
|
| 918 |
The last but never the less important step is to configure the LINSTOR |
| 919 |
satellite services to not delete (and then regenerate) the resource file for the |
| 920 |
LINSTOR controller DB at its startup. Do not edit the service files directly, but use `systemctl edit`. Edit |
| 921 |
the service file on all nodes that could become a LINSTOR controller and that are also LINSTOR satellites. |
| 922 |
|
| 923 |
-------------- |
| 924 |
# systemctl edit linstor-satellite |
| 925 |
[Service] |
| 926 |
Environment=LS_KEEP_RES=linstor_db |
| 927 |
-------------- |
| 928 |
|
| 929 |
After this change you should execute `systemctl restart linstor-satellite` on all satellite nodes. |
| 930 |
|
| 931 |
CAUTION: Be sure to configure your LINSTOR client for use with multiple controllers as described in |
| 932 |
the section titled, <<s-using_the_linstor_client>> and make sure you also |
| 933 |
configured your integration plugins (e.g., the Proxmox plugin) to be ready for |
| 934 |
multiple LINSTOR controllers. |
| 935 |
|
| 936 |
[[s-drbd_clients]] |
| 937 |
=== DRBD clients |
| 938 |
By using the `--drbd-diskless` option instead of `--storage-pool` you can |
| 939 |
have a permanently diskless DRBD device on a node. This means that |
| 940 |
the resource will appear as block device and can be mounted to the |
| 941 |
filesystem without an existing storage-device. The data of the |
| 942 |
resource is accessed over the network on another nodes with the |
| 943 |
same resource. |
| 944 |
|
| 945 |
---------------------------- |
| 946 |
# linstor resource create delta backups --drbd-diskless |
| 947 |
---------------------------- |
| 948 |
|
| 949 |
NOTE: The option `--diskless` was deprecated. Please use `--drbd-diskless` |
| 950 |
or `--nvme-initiator` instead. |
| 951 |
|
| 952 |
[[s-linstor-drbd-consistency-group-multiple-volumes]] |
| 953 |
=== LINSTOR - DRBD consistency group/multiple volumes |
| 954 |
|
| 955 |
The so called consistency group is a feature from DRBD. It is mentioned in this user-guide, due to the |
| 956 |
fact that one of LINSTOR's main functions is to manage storage-clusters with DRBD. Multiple volumes in |
| 957 |
one resource are a consistency group. |
| 958 |
|
| 959 |
This means that changes on different volumes from one resource are getting replicated in |
| 960 |
the same chronological order on the other Satellites. |
| 961 |
|
| 962 |
Therefore you don't have to worry about the timing if you have interdependent data on different volumes in a |
| 963 |
resource. |
| 964 |
|
| 965 |
To deploy more than one volume in a LINSTOR-resource you have to create two volume-definitions with the same name. |
| 966 |
|
| 967 |
---- |
| 968 |
# linstor volume-definition create backups 500G |
| 969 |
# linstor volume-definition create backups 100G |
| 970 |
---- |
| 971 |
|
| 972 |
|
| 973 |
[[s-volumes_of_one_resource_to_different_storage_pools]] |
| 974 |
=== Volumes of one resource to different Storage-Pools |
| 975 |
This can be achieved by setting the `StorPoolName` property to the volume |
| 976 |
definitions before the resource is deployed to the nodes: |
| 977 |
|
| 978 |
|
| 979 |
---------------------------- |
| 980 |
# linstor resource-definition create backups |
| 981 |
# linstor volume-definition create backups 500G |
| 982 |
# linstor volume-definition create backups 100G |
| 983 |
# linstor volume-definition set-property backups 0 StorPoolName pool_hdd |
| 984 |
# linstor volume-definition set-property backups 1 StorPoolName pool_ssd |
| 985 |
# linstor resource create alpha backups |
| 986 |
# linstor resource create bravo backups |
| 987 |
# linstor resource create charlie backups |
| 988 |
---------------------------- |
| 989 |
|
| 990 |
NOTE: Since the `volume-definition create` command is used without the `--vlmnr` option |
| 991 |
LINSTOR assigned the volume numbers starting at 0. In the following two |
| 992 |
lines the 0 and 1 refer to these automatically assigned volume numbers. |
| 993 |
|
| 994 |
Here the 'resource create' commands do not need a `--storage-pool` option. |
| 995 |
In this case LINSTOR uses a 'fallback' storage pool. Finding that |
| 996 |
storage pool, LINSTOR queries the properties of the following objects |
| 997 |
in the following order: |
| 998 |
|
| 999 |
* Volume definition |
| 1000 |
* Resource |
| 1001 |
* Resource definition |
| 1002 |
* Node |
| 1003 |
|
| 1004 |
If none of those objects contain a `StorPoolName` property, the controller |
| 1005 |
falls back to a hard-coded 'DfltStorPool' string as a storage pool. |
| 1006 |
|
| 1007 |
This also means that if you forgot to define a storage pool prior deploying a |
| 1008 |
resource, you will get an error message that LINSTOR could not find the |
| 1009 |
storage pool named 'DfltStorPool'. |
| 1010 |
|
| 1011 |
|
| 1012 |
[[s-linstor-without-drbd]] |
| 1013 |
=== LINSTOR without DRBD |
| 1014 |
|
| 1015 |
LINSTOR can be used without DRBD as well. Without DRBD, LINSTOR is |
| 1016 |
able to provision volumes from LVM and ZFS backed storage pools, and |
| 1017 |
create those volumes on individual nodes in your LINSTOR cluster. |
| 1018 |
|
| 1019 |
Currently LINSTOR supports the creation of LVM and ZFS |
| 1020 |
volumes with the option of layering some combinations of LUKS, |
| 1021 |
DRBD, and/or NVMe-oF/NVMe-TCP on top of those volumes. |
| 1022 |
|
| 1023 |
For example, assume we have a Thin LVM backed storage pool defined in |
| 1024 |
our LINSTOR cluster named, `thin-lvm`: |
| 1025 |
|
| 1026 |
---- |
| 1027 |
# linstor --no-utf8 storage-pool list |
| 1028 |
+--------------------------------------------------------------+ |
| 1029 |
| StoragePool | Node | Driver | PoolName | ... | |
| 1030 |
|--------------------------------------------------------------| |
| 1031 |
| thin-lvm | linstor-a | LVM_THIN | drbdpool/thinpool | ... | |
| 1032 |
| thin-lvm | linstor-b | LVM_THIN | drbdpool/thinpool | ... | |
| 1033 |
| thin-lvm | linstor-c | LVM_THIN | drbdpool/thinpool | ... | |
| 1034 |
| thin-lvm | linstor-d | LVM_THIN | drbdpool/thinpool | ... | |
| 1035 |
+--------------------------------------------------------------+ |
| 1036 |
---- |
| 1037 |
|
| 1038 |
We could use LINSTOR to create a Thin LVM on `linstor-d` that's 100GiB |
| 1039 |
in size using the following commands: |
| 1040 |
|
| 1041 |
---- |
| 1042 |
# linstor resource-definition create rsc-1 |
| 1043 |
# linstor volume-definition create rsc-1 100GiB |
| 1044 |
# linstor resource create --layer-list storage \ |
| 1045 |
--storage-pool thin-lvm linstor-d rsc-1 |
| 1046 |
---- |
| 1047 |
|
| 1048 |
You should then see you have a new Thin LVM on `linstor-d`. You can |
| 1049 |
extract the device path from LINSTOR by listing your linstor resources |
| 1050 |
with the `--machine-readable` flag set: |
| 1051 |
|
| 1052 |
---- |
| 1053 |
# linstor --machine-readable resource list | grep device_path |
| 1054 |
"device_path": "/dev/drbdpool/rsc-1_00000", |
| 1055 |
---- |
| 1056 |
|
| 1057 |
If you wanted to layer DRBD on top of this volume, which is the default |
| 1058 |
`--layer-list` option in LINSTOR for ZFS or LVM backed volumes, you |
| 1059 |
would use the following resource creation pattern instead: |
| 1060 |
|
| 1061 |
---- |
| 1062 |
# linstor resource-definition create rsc-1 |
| 1063 |
# linstor volume-definition create rsc-1 100GiB |
| 1064 |
# linstor resource create --layer-list drbd,storage \ |
| 1065 |
--storage-pool thin-lvm linstor-d rsc-1 |
| 1066 |
---- |
| 1067 |
|
| 1068 |
You would then see that you have a new Thin LVM backing a DRBD volume |
| 1069 |
on `linstor-d`: |
| 1070 |
|
| 1071 |
---- |
| 1072 |
# linstor --machine-readable resource list | grep -e device_path -e backing_disk |
| 1073 |
"device_path": "/dev/drbd1000", |
| 1074 |
"backing_disk": "/dev/drbdpool/rsc-1_00000", |
| 1075 |
---- |
| 1076 |
|
| 1077 |
The following table shows which layer can be followed by which child-layer: |
| 1078 |
|
| 1079 |
[cols=">1,<5"] |
| 1080 |
|=== |
| 1081 |
| Layer | Child layer |
| 1082 |
|
| 1083 |
| DRBD | CACHE, WRITECACHE, NVME, LUKS, STORAGE |
| 1084 |
| CACHE | WRITECACHE, NVME, LUKS, STORAGE |
| 1085 |
| WRITECACHE | CACHE, NVME, LUKS, STORAGE |
| 1086 |
| NVME | CACHE, WRITECACHE, LUKS, STORAGE |
| 1087 |
| LUKS | STORAGE |
| 1088 |
| STORAGE | - |
| 1089 |
|=== |
| 1090 |
|
| 1091 |
NOTE: One layer can only occur once in the layer-list |
| 1092 |
|
| 1093 |
TIP: For information about the prerequisites for the `LUKS` layer, |
| 1094 |
refer to the Encrypted Volumes section of this User's Guide. |
| 1095 |
|
| 1096 |
[[s-nvme-layer]] |
| 1097 |
==== NVMe-oF/NVMe-TCP LINSTOR Layer |
| 1098 |
|
| 1099 |
NVMe-oF/NVMe-TCP allows LINSTOR to connect diskless resources to a |
| 1100 |
node with the same resource where the data is stored over NVMe |
| 1101 |
fabrics. This leads to the advantage that resources can be mounted |
| 1102 |
without using local storage by accessing the data over the network. |
| 1103 |
LINSTOR is not using DRBD in this case, and therefore NVMe resources |
| 1104 |
provisioned by LINSTOR are not replicated, the data is stored on one |
| 1105 |
node. |
| 1106 |
|
| 1107 |
NOTE: NVMe-oF only works on RDMA-capable networks and NVMe-TCP on |
| 1108 |
every network that can carry IP traffic. If you want to know more |
| 1109 |
about NVMe-oF/NVMe-TCP visit |
| 1110 |
https://www.linbit.com/en/nvme-linstor-swordfish/ for more |
| 1111 |
information. |
| 1112 |
|
| 1113 |
To use NVMe-oF/NVMe-TCP with LINSTOR the package `nvme-cli` needs to |
| 1114 |
be installed on every Node which acts as a Satellite and will use |
| 1115 |
NVMe-oF/NVMe-TCP for a resource: |
| 1116 |
|
| 1117 |
IMPORTANT: If you are not using Ubuntu use the suitable command for |
| 1118 |
installing packages on your OS - SLES: zypper - CentOS: yum |
| 1119 |
|
| 1120 |
------ |
| 1121 |
# apt install nvme-cli |
| 1122 |
------ |
| 1123 |
|
| 1124 |
To make a resource which uses NVMe-oF/NVMe-TCP an additional parameter |
| 1125 |
has to be given as you create the resource-definition: |
| 1126 |
|
| 1127 |
------ |
| 1128 |
# linstor resource-definition create nvmedata -l nvme,storage |
| 1129 |
------ |
| 1130 |
|
| 1131 |
NOTE: As default the -l (layer-stack) parameter is set to `drbd, |
| 1132 |
storage` when DRBD is used. If you want to create LINSTOR resources |
| 1133 |
with neither NVMe nor DRBD you have to set the `-l` parameter to only |
| 1134 |
`storage`. |
| 1135 |
|
| 1136 |
In order to use NVMe-TCP instead of the default NVMe-oF the following |
| 1137 |
property needs to be set: |
| 1138 |
|
| 1139 |
------ |
| 1140 |
# linstor resource-definition set-property nvmedata NVMe/TRType tcp |
| 1141 |
------ |
| 1142 |
|
| 1143 |
The property `NVMe/TRType` can alternatively be set on resource-group |
| 1144 |
or controller level. |
| 1145 |
|
| 1146 |
Next, create the volume-definition for our resource: |
| 1147 |
|
| 1148 |
------ |
| 1149 |
# linstor volume-definition create nvmedata 500G |
| 1150 |
------ |
| 1151 |
|
| 1152 |
Before you create the resource on your nodes you have to know where |
| 1153 |
the data will be stored locally and which node accesses it over the |
| 1154 |
network. |
| 1155 |
|
| 1156 |
First we create the resource on the node where our data will be stored: |
| 1157 |
|
| 1158 |
------ |
| 1159 |
# linstor resource create alpha nvmedata --storage-pool pool_ssd |
| 1160 |
------ |
| 1161 |
|
| 1162 |
On the nodes where the resource-data will be accessed over the |
| 1163 |
network, the resource has to be defined as diskless: |
| 1164 |
|
| 1165 |
----- |
| 1166 |
# linstor resource create beta nvmedata --nvme-initiator |
| 1167 |
----- |
| 1168 |
|
| 1169 |
Now you can mount the resource `nvmedata` on one of your nodes. |
| 1170 |
|
| 1171 |
IMPORTANT: If your nodes have more than one NIC you should force the |
| 1172 |
route between them for NVMe-of/NVME-TCP, otherwise multiple NICs |
| 1173 |
could cause troubles. |
| 1174 |
|
| 1175 |
[[s-openflex]] |
| 1176 |
==== OpenFlex(TM) Layer |
| 1177 |
|
| 1178 |
Since version 1.5.0 the additional Layer `openflex` can be used in LINSTOR. |
| 1179 |
From LINSTOR's perspective, the |
| 1180 |
https://www.westerndigital.com/products/storage-platforms/openflex-composable-infrastructure[OpenFlex |
| 1181 |
Composable Infrastructure] takes the role of a combined layer acting as a |
| 1182 |
storage layer (like LVM) and also providing the allocated space as an NVMe target. |
| 1183 |
OpenFlex has a REST API which is also used by LINSTOR to operate with. |
| 1184 |
|
| 1185 |
As OpenFlex combines concepts of LINSTOR's storage as well as NVMe-layer, LINSTOR was |
| 1186 |
added both, a new storage driver for the storage pools as well as a dedicated `openflex` |
| 1187 |
layer which uses the mentioned REST API. |
| 1188 |
|
| 1189 |
In order for LINSTOR to communicate with the OpenFlex-API, LINSTOR needs some additional |
| 1190 |
properties, which can be set once on `controller` level to take LINSTOR-cluster wide effect: |
| 1191 |
|
| 1192 |
* `StorDriver/Openflex/ApiHost` specifies the host or IP of the API entry-point |
| 1193 |
* `StorDriver/Openflex/ApiPort` this property is glued with a colon to the previous to form the |
| 1194 |
basic `http://ip:port` part used by the REST calls |
| 1195 |
* `StorDriver/Openflex/UserName` the REST username |
| 1196 |
* `StorDriver/Openflex/UserPassword` the password for the REST user |
| 1197 |
|
| 1198 |
Once that is configured, we can now create LINSTOR objects to represent the OpenFlex architecture. |
| 1199 |
The theoretical mapping of LINSTOR objects to OpenFlex objects are as follows: |
| 1200 |
Obviously an OpenFlex storage pool is represented by a LINSTOR storage pool. As the next thing above |
| 1201 |
a LINSTOR storage pool is already the node, a LINSTOR node represents an OpenFlex storage device. |
| 1202 |
The OpenFlex objects above storage device are not mapped by LINSTOR. |
| 1203 |
|
| 1204 |
When using NVMe, LINSTOR was designed to run on both sides, the NVMe target as well as on the |
| 1205 |
NVMe initiator side. In the case of OpenFlex, LINSTOR cannot (or even should not) run on the NVMe |
| 1206 |
target side as that is completely managed by OpenFlex. As LINSTOR still needs nodes and storage pools |
| 1207 |
to represent the OpenFlex counterparts, the LINSTOR client was extended with special node create commands |
| 1208 |
since 1.0.14. These commands not only accept additionally needed configuration data, but also |
| 1209 |
starts a "special satellite" besides the already running controller instance. This special satellites |
| 1210 |
are completely LINSTOR managed, they will shutdown when the controller shuts down and will be started |
| 1211 |
again when the controller starts. |
| 1212 |
The new client command for creating a "special satellite" representing an OpenFlex storage device is: |
| 1213 |
|
| 1214 |
---- |
| 1215 |
$ linstor node create-openflex-target ofNode1 192.168.166.7 000af795789d |
| 1216 |
---- |
| 1217 |
The arguments are as follows: |
| 1218 |
|
| 1219 |
* `ofNode1` is the node name which is also used by the standard `linstor node create` command |
| 1220 |
* `192.168.166.7` is the address on which the provided NVMe devices can be accessed. |
| 1221 |
As the NVMe devices are accessed by a dedicated network interface, this address differs from the |
| 1222 |
address specified with the property `StorDriver/Openflex/ApiHost`. The latter is used for the |
| 1223 |
management / REST API. |
| 1224 |
* `000af795789d` is the identifier for the OpenFlex storage device. |
| 1225 |
|
| 1226 |
The last step of the configuration is the creation of LINSTOR storage pools: |
| 1227 |
|
| 1228 |
---- |
| 1229 |
$ linstor storage-pool create openflex ofNode1 sp0 0 |
| 1230 |
---- |
| 1231 |
|
| 1232 |
* `ofNode1` and `sp0` are the node name and storage pool name, respectively, just as usual for the |
| 1233 |
LINSTOR's create storage pool command |
| 1234 |
* The last `0` is the identifier of the OpenFlex storage pool within the previously defined storage device |
| 1235 |
|
| 1236 |
Once all necessary storage pools are created in LINSTOR, the next steps are similar to |
| 1237 |
the usage of using an NVMe resource with LINSTOR. Here is a complete example: |
| 1238 |
|
| 1239 |
---- |
| 1240 |
# set the properties once |
| 1241 |
linstor controller set-property StorDriver/Openflex/ApiHost 10.43.7.185 |
| 1242 |
linstor controller set-property StorDriver/Openflex/ApiPort 80 |
| 1243 |
linstor controller set-property StorDriver/Openflex/UserName myusername |
| 1244 |
linstor controller set-property StorDriver/Openflex/UserPassword mypassword |
| 1245 |
|
| 1246 |
# create a node for openflex storage device "000af795789d" |
| 1247 |
linstor node create-openflex-target ofNode1 192.168.166.7 000af795789d |
| 1248 |
|
| 1249 |
# create a usual linstor satellite. later used as nvme initiator |
| 1250 |
linstor node create bravo |
| 1251 |
|
| 1252 |
# create a storage pool for openflex storage pool "0" within storage device "000af795789d" |
| 1253 |
linstor storage-pool create openflex ofNode1 sp0 0 |
| 1254 |
|
| 1255 |
# create resource- and volume-definition |
| 1256 |
linstor resource-definition create backupRsc |
| 1257 |
linstor volume-definition create backupRsc 10G |
| 1258 |
|
| 1259 |
# create openflex-based nvme target |
| 1260 |
linstor resource create ofNode1 backupRsc --storage-pool sp0 --layer-list openflex |
| 1261 |
|
| 1262 |
# create openflex-based nvme initiator |
| 1263 |
linstor resource create bravo backupRsc --nvme-initiator --layer-list openflex |
| 1264 |
---- |
| 1265 |
|
| 1266 |
NOTE: In case a node should access the OpenFlex REST API through a different host than specified with + |
| 1267 |
`linstor controller set-property StorDriver/Openflex/ApiHost 10.43.7.185` you can always use LINSTOR's |
| 1268 |
inheritance mechanism for properties. That means simply define the same property on the node-level |
| 1269 |
you need it, i.e. + |
| 1270 |
`linstor node set-property ofNode1 StorDriver/Openflex/ApiHost 10.43.8.185` |
| 1271 |
|
| 1272 |
[[s-writecache-layer]] |
| 1273 |
==== Writecache Layer |
| 1274 |
|
| 1275 |
A https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/writecache.html[DM-Writecache] |
| 1276 |
device is composed by two devices, one storage device and one cache device. |
| 1277 |
LINSTOR can setup such a writecache device, but needs some additional information, like |
| 1278 |
the storage pool and the size of the cache device. |
| 1279 |
|
| 1280 |
------ |
| 1281 |
# linstor storage-pool create lvm node1 lvmpool drbdpool |
| 1282 |
# linstor storage-pool create lvm node1 pmempool pmempool |
| 1283 |
|
| 1284 |
# linstor resource-definition create r1 |
| 1285 |
# linstor volume-definition create r1 100G |
| 1286 |
|
| 1287 |
# linstor volume-definition set-property r1 0 Writecache/PoolName pmempool |
| 1288 |
# linstor volume-definition set-property r1 0 Writecache/Size 1% |
| 1289 |
|
| 1290 |
# linstor resource create node1 r1 --storage-pool lvmpool --layer-list WRITECACHE,STORAGE |
| 1291 |
------ |
| 1292 |
|
| 1293 |
The two properties set in the examples are mandatory, but can also be set on |
| 1294 |
controller level which would act as a default for all resources with `WRITECACHE` in their |
| 1295 |
`--layer-list`. However, please note that the `Writecache/PoolName` refers to |
| 1296 |
the corresponding node. If the node does not have a storage-pool named `pmempool` you will |
| 1297 |
get an error message. |
| 1298 |
|
| 1299 |
The 4 mandatory parameters required by |
| 1300 |
https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/writecache.html[DM-Writecache] |
| 1301 |
are either configured via property or figured out by LINSTOR. |
| 1302 |
The optional properties listed in the mentioned link can also be set via property. |
| 1303 |
Please see `linstor controller set-property --help` for a list of `Writecache/*` |
| 1304 |
property-keys. |
| 1305 |
|
| 1306 |
Using `--layer-list DRBD,WRITECACHE,STORAGE` while having DRBD configured to use |
| 1307 |
external metadata, only the backing device will use a writecache, not the |
| 1308 |
device holding the external metadata. |
| 1309 |
|
| 1310 |
[[s-cache-layer]] |
| 1311 |
==== Cache Layer |
| 1312 |
|
| 1313 |
LINSTOR can also setup a https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/cache.html[DM-Cache] |
| 1314 |
device, which is very similar to the DM-Writecache from the previous section. The major difference |
| 1315 |
is that a cache device is composed by three devices: one storage device, one cache device and one |
| 1316 |
meta device. The LINSTOR properties are quite similar to those of the writecache but are located |
| 1317 |
in the `Cache` namespace: |
| 1318 |
|
| 1319 |
------ |
| 1320 |
# linstor storage-pool create lvm node1 lvmpool drbdpool |
| 1321 |
# linstor storage-pool create lvm node1 pmempool pmempool |
| 1322 |
|
| 1323 |
# linstor resource-definition create r1 |
| 1324 |
# linstor volume-definition create r1 100G |
| 1325 |
|
| 1326 |
# linstor volume-definition set-property r1 0 Cache/CachePool pmempool |
| 1327 |
# linstor volume-definition set-property r1 0 Cache/Cachesize 1% |
| 1328 |
|
| 1329 |
# linstor resource create node1 r1 --storage-pool lvmpool --layer-list CACHE,STORAGE |
| 1330 |
------ |
| 1331 |
|
| 1332 |
NOTE: Instead of `Writecache/PoolName` (as when configuring the Writecache layer) the |
| 1333 |
Cache layer's only required property is called `Cache/CachePool`. The reason for this |
| 1334 |
is that the Cache layer also has a `Cache/MetaPool` which can be configured separately |
| 1335 |
or it defaults to the value of `Cache/CachePool`. |
| 1336 |
|
| 1337 |
Please see `linstor controller set-property --help` for a list of `Cache/*` |
| 1338 |
property-keys and default values for omitted properties. |
| 1339 |
|
| 1340 |
Using `--layer-list DRBD,CACHE,STORAGE` while having DRBD configured to use external |
| 1341 |
metadata, only the backing device will use a cache, not the device holding the external |
| 1342 |
metadata. |
| 1343 |
|
| 1344 |
[[s-storage-layer]] |
| 1345 |
==== Storage Layer |
| 1346 |
|
| 1347 |
The storage layer will provide new devices from well known volume managers like LVM, ZFS or |
| 1348 |
others. Every layer combination needs to be based on a storage layer, even if the resource |
| 1349 |
should be diskless - for that type there is a dedicated `diskless` provider type. |
| 1350 |
|
| 1351 |
For a list of providers with their properties please see <<s-storage-proviers, Storage Providers>>. |
| 1352 |
|
| 1353 |
For some storage providers LINSTOR has special properties: |
| 1354 |
|
| 1355 |
* `StorDriver/WaitTimeoutAfterCreate`: If LINSTOR expects a device to appear after creation |
| 1356 |
(for example after calls of `lvcreate`, `zfs create`,...), LINSTOR waits per default 500ms |
| 1357 |
for the device to appear. These 500ms can be overridden by this property. |
| 1358 |
* `StorDriver/dm_stats`: If set to `true` LINSTOR calls `dmstats create $device` after |
| 1359 |
creation and `dmstats delete $device --allregions` after deletion of a volume. |
| 1360 |
Currently only enabled for LVM and LVM_THIN storage providers. |
| 1361 |
|
| 1362 |
[[s-storage-providers]] |
| 1363 |
=== Storage Providers |
| 1364 |
|
| 1365 |
LINSTOR has a few storage providers. The most used ones are LVM and ZFS. But also for those |
| 1366 |
two providers there are already sub-types for their thinly provisioned variants. |
| 1367 |
|
| 1368 |
* Diskless: This provider type is mostly required to have a storage pool that can be configured |
| 1369 |
with LINSTOR properties like `PrefNic` as described in <<s-managing_network_interface_cards, |
| 1370 |
Managing Network Interface Cards>>. |
| 1371 |
* LVM / LVM-Thin: The adminstrator is expected to specify the LVM volume group or the thin-pool |
| 1372 |
(in form of "LV/thinpool") in order to use the corresponding storage type. These drivers support |
| 1373 |
following properties for fine-tuning: |
| 1374 |
** `StorDriver/LvcreateOptions`: The value of this property is appended to every |
| 1375 |
`lvcreate ...` call LINSTOR executes. |
| 1376 |
* ZFS / ZFS-Thin: The administrator is expected to specify the ZPool that LINSTOR should use. |
| 1377 |
These drivers support following properties for fine-tuning: |
| 1378 |
** `StorDriver/ZfscreateOptions`: The value of this property is appended to every |
| 1379 |
`zfs create ...` call LINSTOR executes. |
| 1380 |
* File / FileThin: Mostly used for demonstration / experiments. LINSTOR will basically reserve a |
| 1381 |
file in a given directory and will configure a https://man7.org/linux/man-pages/man4/loop.4.html[loop device] |
| 1382 |
on top of that file. |
| 1383 |
* OpenFlex: This special storage provider currently requires to be run on a "special satellite". |
| 1384 |
Please see <<s-openflex, OpenFlex(TM) Layer>> for more details. |
| 1385 |
* EXOS: This special storage provider currently requires to be run on a "special satellite". |
| 1386 |
Please see the <<ch-exos, EXOS Integration>> chapter |
| 1387 |
* SPDK: The administrator is expected to speicify the logical volume store which LINSTOR should use. |
| 1388 |
The usage of this storage provider implies the usage of the <<s-nvme-layer,NVME Layer>>. |
| 1389 |
** Remote-SPDK: This special storage provider currently requires to be run on a "special satellite". |
| 1390 |
Please see <<s-remote-spdk, Remote SPDK Provider>> for more details. |
| 1391 |
|
| 1392 |
[[s-remote-spdk]] |
| 1393 |
==== Remote SPDK Provider |
| 1394 |
A storage pool with the type remote SPDK can only be created on a "special satellite". For this |
| 1395 |
you first need to start a new satellite using the command: |
| 1396 |
|
| 1397 |
---- |
| 1398 |
$ linstor node create-remote-spdk-target nodeName 192.168.1.110 |
| 1399 |
---- |
| 1400 |
|
| 1401 |
This will start a new satellite instance running on the same machine as the controller. This special |
| 1402 |
satellite will do all the REST based RPC communication towards the remote SPDK proxy. As the help message |
| 1403 |
of the LINSTOR command shows, the administrator might want to use additional settings when creating |
| 1404 |
this special satellite: |
| 1405 |
|
| 1406 |
---- |
| 1407 |
$ linstor node create-remote-spdk-target -h |
| 1408 |
usage: linstor node create-remote-spdk-target [-h] [--api-port API_PORT] |
| 1409 |
[--api-user API_USER] |
| 1410 |
[--api-user-env API_USER_ENV] |
| 1411 |
[--api-pw [API_PW]] |
| 1412 |
[--api-pw-env API_PW_ENV] |
| 1413 |
node_name api_host |
| 1414 |
---- |
| 1415 |
|
| 1416 |
The difference between the `--api-\*` and their corresponding `--api-\*-env` versions is that the |
| 1417 |
version with the `-env` ending will look for an environment variable containing the actual value to use |
| 1418 |
whereas the `--api-\*` version directly take the value which is stored in the LINSTOR property. |
| 1419 |
Administrators might not want to save the `--api-pw` in plaintext, which would be clearly visible |
| 1420 |
using commands like `linstor node list-property <nodeName>`. |
| 1421 |
|
| 1422 |
Once that special satellite is up and running the actual storage pool can be created: |
| 1423 |
|
| 1424 |
---- |
| 1425 |
$ linstor storage-pool create remotespdk -h |
| 1426 |
usage: linstor storage-pool create remotespdk [-h] |
| 1427 |
[--shared-space SHARED_SPACE] |
| 1428 |
[--external-locking] |
| 1429 |
node_name name driver_pool_name |
| 1430 |
---- |
| 1431 |
|
| 1432 |
Whereas `node_name` is self-explanatory, `name` is the name of the LINSTOR storage pool and |
| 1433 |
`driver_pool_name` refers to the SPDK logical volume store. |
| 1434 |
|
| 1435 |
Once this _remotespdk_ storage pool is created the remaining procedure is quite similar as |
| 1436 |
using NVMe: First the target has to be created by creating a simple "diskful" resource followed |
| 1437 |
by a second resource having the `--nvme-initiator` option enabled. |
| 1438 |
|
| 1439 |
|
| 1440 |
[[s-managing_network_interface_cards]] |
| 1441 |
=== Managing Network Interface Cards |
| 1442 |
|
| 1443 |
LINSTOR can deal with multiple network interface cards (NICs) in a machine, |
| 1444 |
they are called `netif` in LINSTOR speak. |
| 1445 |
|
| 1446 |
NOTE: When a satellite node is created a first `netif` gets created implicitly |
| 1447 |
with the name `default`. Using the `--interface-name` option of the `node create` |
| 1448 |
command you can give it a different name. |
| 1449 |
|
| 1450 |
Additional NICs are created like this: |
| 1451 |
---------------------------- |
| 1452 |
# linstor node interface create alpha 100G_nic 192.168.43.221 |
| 1453 |
# linstor node interface create alpha 10G_nic 192.168.43.231 |
| 1454 |
---------------------------- |
| 1455 |
|
| 1456 |
NICs are identified by the IP address only, the name is arbitrary and is |
| 1457 |
*not* related to the interface name used by Linux. The NICs can be assigned |
| 1458 |
to storage pools so that whenever a resource is created in such a storage |
| 1459 |
pool, the DRBD traffic will be routed through the specified NIC. |
| 1460 |
|
| 1461 |
---------------------------- |
| 1462 |
# linstor storage-pool set-property alpha pool_hdd PrefNic 10G_nic |
| 1463 |
# linstor storage-pool set-property alpha pool_ssd PrefNic 100G_nic |
| 1464 |
---------------------------- |
| 1465 |
|
| 1466 |
FIXME describe how to route the controller +<->+ client communication through |
| 1467 |
a specific `netif`. |
| 1468 |
|
| 1469 |
[[s-linstor-encrypted-volumes]] |
| 1470 |
=== Encrypted volumes |
| 1471 |
LINSTOR can handle transparent encryption of drbd volumes. dm-crypt is used to |
| 1472 |
encrypt the provided storage from the storage device. |
| 1473 |
|
| 1474 |
NOTE: In order to use dm-crypt please make sure to have `cryptsetup` installed before |
| 1475 |
you start the satellite |
| 1476 |
|
| 1477 |
Basic steps to use encryption: |
| 1478 |
|
| 1479 |
1. Disable user security on the controller (this will be obsolete once authentication works) |
| 1480 |
2. Create a master passphrase |
| 1481 |
3. Add `luks` to the layer-list. Note that all plugins (e.g., Proxmox) require a DRBD layer as the top most layer if they do not explicitly state otherwise. |
| 1482 |
4. Don't forget to re-enter the master passphrase after a controller restart. |
| 1483 |
|
| 1484 |
[[s-disable_user_security]] |
| 1485 |
==== Disable user security |
| 1486 |
Disabling the user security on the `Linstor` controller is a one time operation and is |
| 1487 |
afterwards persisted. |
| 1488 |
|
| 1489 |
1. Stop the running linstor-controller via systemd: `systemctl stop linstor-controller` |
| 1490 |
2. Start a linstor-controller in debug mode: `/usr/share/linstor-server/bin/Controller -c /etc/linstor -d` |
| 1491 |
3. In the debug console enter: `setSecLvl secLvl(NO_SECURITY)` |
| 1492 |
4. Stop linstor-controller with the debug shutdown command: `shutdown` |
| 1493 |
5. Start the controller again with systemd: `systemctl start linstor-controller` |
| 1494 |
|
| 1495 |
[[s-encrypt_commands]] |
| 1496 |
==== Encrypt commands |
| 1497 |
Below are details about the commands. |
| 1498 |
|
| 1499 |
Before LINSTOR can encrypt any volume a master passphrase needs to be created. |
| 1500 |
This can be done with the linstor-client. |
| 1501 |
|
| 1502 |
---- |
| 1503 |
# linstor encryption create-passphrase |
| 1504 |
---- |
| 1505 |
|
| 1506 |
`crypt-create-passphrase` will wait for the user to input the initial master passphrase |
| 1507 |
(as all other crypt commands will with no arguments). |
| 1508 |
|
| 1509 |
If you ever want to change the master passphrase this can be done with: |
| 1510 |
|
| 1511 |
---- |
| 1512 |
# linstor encryption modify-passphrase |
| 1513 |
---- |
| 1514 |
|
| 1515 |
The `luks` layer can be added when creating the resource-definition or the resource |
| 1516 |
itself, whereas the former method is recommended since it will be automatically applied |
| 1517 |
to all resource created from that resource-definition. |
| 1518 |
|
| 1519 |
---- |
| 1520 |
# linstor resource-definition create crypt_rsc --layer-list luks,storage |
| 1521 |
---- |
| 1522 |
|
| 1523 |
To enter the master passphrase (after controller restart) use the following command: |
| 1524 |
|
| 1525 |
---- |
| 1526 |
# linstor encryption enter-passphrase |
| 1527 |
---- |
| 1528 |
|
| 1529 |
NOTE: Whenever the linstor-controller is restarted, the user has to send |
| 1530 |
the master passphrase to the controller, otherwise LINSTOR is unable to reopen or |
| 1531 |
create encrypted volumes. |
| 1532 |
|
| 1533 |
[[s-automatic_passphrase]] |
| 1534 |
==== Automatic Passphrase |
| 1535 |
It is possible to automate the process of creating and re-entering the master passphrase. |
| 1536 |
|
| 1537 |
To use this, either an environment variable called `MASTER_PASSPHRASE` or an entry in |
| 1538 |
`/etc/linstor/linstor.toml` containing the master passphrase has to be created. |
| 1539 |
|
| 1540 |
The required `linstor.toml` looks like this: |
| 1541 |
|
| 1542 |
---- |
| 1543 |
[encrypt] |
| 1544 |
passphrase="example" |
| 1545 |
---- |
| 1546 |
|
| 1547 |
If either one of these is set, then every time the controller starts it will check whether |
| 1548 |
a master passphrase already exists. If there is none, it will create a new master passphrase as specified. |
| 1549 |
Otherwise, the controller enters the passphrase. |
| 1550 |
|
| 1551 |
WARNING: If a master passphrase is already configured, and it is not the same one as specified |
| 1552 |
in the environment variable or `linstor.toml`, the controller will be unable to re-enter the |
| 1553 |
master passphrase and react as if the user had entered a wrong passphrase. |
| 1554 |
This can only be resolved through manual input from the user, using the same commands as if |
| 1555 |
the controller was started without the automatic passphrase. |
| 1556 |
|
| 1557 |
NOTE: In case the master passphrase is set in both an environment variable and the `linstor.toml`, |
| 1558 |
only the master passphrase from the `linstor.toml` will be used. |
| 1559 |
|
| 1560 |
[[s-linstor-status]] |
| 1561 |
=== Checking the state of your cluster |
| 1562 |
LINSTOR provides various commands to check the state of your cluster. |
| 1563 |
These commands start with a 'list-' prefix and provide various filtering and |
| 1564 |
sorting options. The '--groupby' option can be used to group and sort the |
| 1565 |
output in multiple dimensions. |
| 1566 |
|
| 1567 |
---------------------------- |
| 1568 |
# linstor node list |
| 1569 |
# linstor storage-pool list --groupby Size |
| 1570 |
---------------------------- |
| 1571 |
|
| 1572 |
[[s-linstor-snapshots]] |
| 1573 |
=== Managing snapshots |
| 1574 |
Snapshots are supported with thin LVM and ZFS storage pools. |
| 1575 |
|
| 1576 |
[[s-creating_a_snapshot-linstor]] |
| 1577 |
==== Creating a snapshot |
| 1578 |
Assuming a resource definition named 'resource1' which has been placed on some |
| 1579 |
nodes, a snapshot can be created as follows: |
| 1580 |
|
| 1581 |
---------------------------- |
| 1582 |
# linstor snapshot create resource1 snap1 |
| 1583 |
---------------------------- |
| 1584 |
|
| 1585 |
This will create snapshots on all nodes where the resource is present. |
| 1586 |
LINSTOR will ensure that consistent snapshots are taken even when the |
| 1587 |
resource is in active use. |
| 1588 |
|
| 1589 |
Setting the resource-definition property `AutoSnapshot/RunEvery` |
| 1590 |
LINSTOR will automatically create snapshots every X minute. |
| 1591 |
The optional property `AutoSnapshot/Keep` can be used to clean-up old snapshots |
| 1592 |
which were created automatically. No manually created snapshot will be cleaned-up / deleted. |
| 1593 |
If `AutoSnapshot/Keep` is omitted (or <= 0), LINSTOR will keep the last 10 snapshots |
| 1594 |
by default. |
| 1595 |
|
| 1596 |
---------------------------- |
| 1597 |
# linstor resource-definition set-property AutoSnapshot/RunEvery 15 |
| 1598 |
# linstor resource-definition set-property AutoSnapshot/Keep 5 |
| 1599 |
---------------------------- |
| 1600 |
|
| 1601 |
|
| 1602 |
[[s-restoring_a_snapshot-linstor]] |
| 1603 |
==== Restoring a snapshot |
| 1604 |
The following steps restore a snapshot to a new resource. |
| 1605 |
This is possible even when the original resource has been removed |
| 1606 |
from the nodes where the snapshots were taken. |
| 1607 |
|
| 1608 |
First define the new resource with volumes matching those from the snapshot: |
| 1609 |
|
| 1610 |
---------------------------- |
| 1611 |
# linstor resource-definition create resource2 |
| 1612 |
# linstor snapshot volume-definition restore --from-resource resource1 --from-snapshot snap1 --to-resource resource2 |
| 1613 |
---------------------------- |
| 1614 |
|
| 1615 |
At this point, additional configuration can be applied if necessary. |
| 1616 |
Then, when ready, create resources based on the snapshots: |
| 1617 |
|
| 1618 |
---------------------------- |
| 1619 |
# linstor snapshot resource restore --from-resource resource1 --from-snapshot snap1 --to-resource resource2 |
| 1620 |
---------------------------- |
| 1621 |
|
| 1622 |
This will place the new resource on all nodes where the snapshot is present. |
| 1623 |
The nodes on which to place the resource can also be selected explicitly; |
| 1624 |
see the help (`linstor snapshot resource restore -h`). |
| 1625 |
|
| 1626 |
[[s-rolling_back_snapshot-linstor]] |
| 1627 |
==== Rolling back to a snapshot |
| 1628 |
LINSTOR can roll a resource back to a snapshot state. |
| 1629 |
The resource must not be in use. |
| 1630 |
That is, it may not be mounted on any nodes. |
| 1631 |
If the resource is in use, consider whether you can achieve your goal by |
| 1632 |
<<s-restoring_a_snapshot-linstor,restoring the snapshot>> instead. |
| 1633 |
|
| 1634 |
Rollback is performed as follows: |
| 1635 |
|
| 1636 |
---------------------------- |
| 1637 |
# linstor snapshot rollback resource1 snap1 |
| 1638 |
---------------------------- |
| 1639 |
|
| 1640 |
A resource can only be rolled back to the most recent snapshot. |
| 1641 |
To roll back to an older snapshot, first delete the intermediate snapshots. |
| 1642 |
|
| 1643 |
[[s-removing_a_snapshot-linstor]] |
| 1644 |
==== Removing a snapshot |
| 1645 |
An existing snapshot can be removed as follows: |
| 1646 |
|
| 1647 |
---------------------------- |
| 1648 |
# linstor snapshot delete resource1 snap1 |
| 1649 |
---------------------------- |
| 1650 |
|
| 1651 |
[[s-shipping_snapshots-linstor]] |
| 1652 |
==== Shipping a snapshot |
| 1653 |
|
| 1654 |
Snapshots can be shipped between LINSTOR nodes or between different LINSTOR clusters, |
| 1655 |
as well as to an S3 storage such as https://aws.amazon.com/s3/[Amazon S3] or https://min.io/[min.io]. |
| 1656 |
|
| 1657 |
[[s-shipping_snapshots-remotes]] |
| 1658 |
===== Remotes |
| 1659 |
|
| 1660 |
In a LINSTOR cluster, the definition of a shipping target is called a remote. |
| 1661 |
Currently, there are two different types of remotes: LINSTOR remotes and S3 remotes. |
| 1662 |
LINSTOR remotes are used to ship snapshots to a different LINSTOR cluster, while |
| 1663 |
S3 remotes are needed to ship snapshots to AWS S3, min.io or any other service using S3 |
| 1664 |
compatible object storage. |
| 1665 |
|
| 1666 |
IMPORTANT: Since a remote needs to store sensitive data, such as passwords, it is |
| 1667 |
neccessary to have encryption enabled whenever you wish to use a remote in any way. |
| 1668 |
How to set up LINSTOR's encryption is described <<s-encrypt_commands,here>>. |
| 1669 |
|
| 1670 |
To create an S3 remote, LINSTOR will need to know the endpoint |
| 1671 |
(that is, the URL of the target S3 server), the name of the target bucket, the region the S3 |
| 1672 |
server is in, as well as the access-key and secret-key used to access the bucket. If the command |
| 1673 |
is sent without adding the secret-key, a prompt will pop up to enter it in. The command |
| 1674 |
should look like this: |
| 1675 |
|
| 1676 |
-------------------------- |
| 1677 |
# linstor remote create s3 myRemote s3.us-west-2.amazonaws.com my-bucket us-west-2 admin password |
| 1678 |
-------------------------- |
| 1679 |
|
| 1680 |
TIP: Usually, LINSTOR uses the endpoint and bucket to create an url using the virtual-hosted-style |
| 1681 |
for its access to the given bucket (e.g. my-bucket.s3.us-west-2.amazonaws.com). Should your setup not |
| 1682 |
allow access this way, change the remote to path-style access (e.g. s3.us-west-2.amazonaws.com/my-bucket) |
| 1683 |
by adding the `--use-path-style` argument to make LINSTOR combine the parameters accordingly. |
| 1684 |
|
| 1685 |
To create a LINSTOR remote, only the URL or IP address of the controller of the target cluster |
| 1686 |
is needed. The command goes as follows: |
| 1687 |
|
| 1688 |
---------------------------- |
| 1689 |
# linstor remote create linstor myRemote 192.168.0.15 |
| 1690 |
---------------------------- |
| 1691 |
|
| 1692 |
Additionally, to ship LUKS-based (encrypted) backups, it is necessary to add the `--passphrase` |
| 1693 |
and `--cluster-id` arguments to the command. This is used to save the passphrase and cluster-id of |
| 1694 |
the target cluster to the remote respectively. For more details on shipping LUKS-based backups |
| 1695 |
between two LINSTOR clusters, see <<s-shipping_snapshots-l2l,this chapter>>. |
| 1696 |
|
| 1697 |
To see all the remotes known to the local cluster, use `linstor remote list`. To delete a remote, use |
| 1698 |
`linstor remote delete myRemoteName`. Should an existing remote need altering, use `linstor remote |
| 1699 |
modify` to change it. |
| 1700 |
|
| 1701 |
[[s-shipping_snapshots-s3]] |
| 1702 |
===== Shipping snapshots to S3 |
| 1703 |
|
| 1704 |
All that is needed to ship a snapshot to S3 is to create an S3-remote that the current cluster can reach |
| 1705 |
as well as the resource that should be shipped. Then, simply use the following command to ship it there: |
| 1706 |
|
| 1707 |
-------------------------- |
| 1708 |
# linstor backup create myRemote myRsc |
| 1709 |
-------------------------- |
| 1710 |
|
| 1711 |
This command will create a snapshot of your resource and ship it to the given remote. If this |
| 1712 |
isn't the first time you shipped a backup of this resource (to that remote) and the snapshot |
| 1713 |
of the previous backup hasn't been deleted yet, an incremental backup will be shipped. |
| 1714 |
To force the creation of a full backup, add the `--full` argument to the command. Getting a |
| 1715 |
specific node to ship the backup is also possible by using `--node myNode`, but if the specified |
| 1716 |
node is not available or only has the resource diskless, a different node will be chosen. |
| 1717 |
|
| 1718 |
To see which backups exist in a specific remote, use `linstor backup list myRemote`. A resource-name |
| 1719 |
can be added to the command as a filter to only show backups of that specific resource by using the |
| 1720 |
argument `--resource myRsc`. If you use the `--other` argument, only entries in the bucket that LINSTOR |
| 1721 |
does not recognize as a backup will be shown. LINSTOR always names backups in a certain way, and |
| 1722 |
as long as an item in the remote is named according to this schema, it is assumed that it is a backup |
| 1723 |
created by LINSTOR - so this list will show everything else. |
| 1724 |
|
| 1725 |
There are several options when it comes to deleting backups: |
| 1726 |
|
| 1727 |
* `linstor backup delete all myRemote`: This command deletes ALL S3-objects on the given |
| 1728 |
remote, as long as they are recognized to be backups, that is, fit the expected naming schema. |
| 1729 |
There is the option `--cluster` to only delete backups that were created by the current cluster. |
| 1730 |
|
| 1731 |
* `linstor backup delete id myRemote my-rsc_back_20210824_072543`: This command deletes |
| 1732 |
a single backup from the given remote - namely the one with the given id, which consists |
| 1733 |
of the resource-name, the automatically generated snapshot-name (back_timestamp) and, if |
| 1734 |
set, the backup-suffix. The option `--prefix` lets you delete all backups starting with |
| 1735 |
the given id. The option `--cascade` deletes not only the specified backup, but all other |
| 1736 |
incremental backups depending on it. |
| 1737 |
|
| 1738 |
* `linstor backup delete filter myRemote ...`: This command has a few different arguments |
| 1739 |
to specify a selection of backups to delete. `-t 20210914_120000` will delete all backups |
| 1740 |
made before 12 o'clock on the 14th of September, 2021. `-n myNode` will delete all backups |
| 1741 |
uploaded by the given node. `-r myRsc` will delete all backups with the given resource name. |
| 1742 |
These filters can be combined as needed. Finally, `--cascade` deletes not only the selected |
| 1743 |
backup(s), but all other incremental backups depending on any of the selected backups. |
| 1744 |
|
| 1745 |
* `linstor backup delete s3key myRemote randomPictureInWrongBucket`: This command will find |
| 1746 |
the object with the given S3-key and delete it - without considering anything else. This should |
| 1747 |
only be used to either delete non-backup items from the remote, or to clean up a broken backup |
| 1748 |
that is no longer deleteable by other means. Using this command to delete a regular, working |
| 1749 |
backup will break that backup, so beware! |
| 1750 |
|
| 1751 |
WARNING: All commands that have the `--cascade` option will NOT delete a backup that has |
| 1752 |
incremental backups depending on it unless you explicitly add that option. |
| 1753 |
|
| 1754 |
TIP: All `linstor backup delete ...` commands have the `--dry-run` option, which will |
| 1755 |
give you a list of all the S3-objects that will be deleted. This can be used to ensure |
| 1756 |
nothing that should not be deleted is accidentally deleted. |
| 1757 |
|
| 1758 |
Maybe the most important task after creating a backup is restoring it. To do so, only the remote |
| 1759 |
is needed - but it is also possible to restore into an existing resource definition with no existing |
| 1760 |
snapshots nor resources. There are two options for the command: |
| 1761 |
|
| 1762 |
----------------------- |
| 1763 |
# linstor backup restore myRemote myNode targetRsc --resource sourceRsc |
| 1764 |
# linstor backup restore myRemote myNode targetRsc --id sourceRsc_back_20210824_072543 |
| 1765 |
----------------------- |
| 1766 |
|
| 1767 |
Either `--resource (-r)` or `--id` must be used, but you cannot use both of them together. `-r` is used to |
| 1768 |
restore the latest backup of the resource specified with this option, while `--id` restores the |
| 1769 |
exact backup specified by the given id, and can therefore be used to restore backups other than |
| 1770 |
the most recent. |
| 1771 |
|
| 1772 |
If the backup to be restored includes a LUKS layer, the `--passphrase` argument is required. With |
| 1773 |
it, the passphrase of the original cluster of the backup needs to be set so that LINSTOR can decrypt |
| 1774 |
the volumes after download and re-encrypt them with the local passphrase. |
| 1775 |
|
| 1776 |
The backup restore will download all the snapshots from the last full backup up to the specified |
| 1777 |
backup. Afterwards, it restores the snapshots into a new resource. If that last step should be skipped, |
| 1778 |
the `--download-only` option needs to be added to the command. |
| 1779 |
|
| 1780 |
Backups can be downloaded from any cluster, not just the one that uploaded them, as long as the setup |
| 1781 |
is correct. Specifically, the target resource cannot have any existing resources or snapshots, and the |
| 1782 |
storage pool(s) used need to have the same storage providers. If the storage pool(s) on the target |
| 1783 |
node have the exact same names as on the cluster the backup was created on, no extra action is |
| 1784 |
necessary. Should they have different names, the option `--storpool-rename` needs to be used. It |
| 1785 |
expects at least one `oldname=newname` pair. For every storage pool of the original backup that |
| 1786 |
is not named in that list, it will be assumed that its name is exactly the same on the target node. |
| 1787 |
|
| 1788 |
To find out exactly which storage pools need to be renamed, as well as how big the download and the |
| 1789 |
restored resource will be, the command `linstor backup info myRemote ...` can be used. Similar to the restore |
| 1790 |
command, either `-r` or `--id` need to be given, which add the same restrictions as with that command. |
| 1791 |
To see how much space will be left over in the local storage pools after a restore, the argument `-n myNode` |
| 1792 |
needs to be added. Just like with a restore, it assumes the storage pool names are exactly the same |
| 1793 |
on the given node as with the backup. Should that not be the case, again, just like with the restore |
| 1794 |
command, `--storpool-rename` should be used. |
| 1795 |
|
| 1796 |
[[s-shipping_snapshots-l2l]] |
| 1797 |
===== Shipping snapshots whithin LINSTOR |
| 1798 |
|
| 1799 |
Shipping a snapshot directly between two LINSTOR clusters can be done with a linstor-remote as well |
| 1800 |
as a resource definition with at least one diskful resource on the source side (where the command |
| 1801 |
is issued). On the target side, you only need a linstor-remote with the cluster-id of the source-cluster set. |
| 1802 |
If you wish to ship a snapshot inside the same cluster just use a remote pointing to the local controller. |
| 1803 |
The command looks like this: |
| 1804 |
|
| 1805 |
-------------------------- |
| 1806 |
# linstor backup ship myRemote localRsc targetRsc |
| 1807 |
-------------------------- |
| 1808 |
|
| 1809 |
Additionally, you can use `--source-node` and `--target-node` to specify which node should send and |
| 1810 |
receive the backup respectively. In case those nodes are not available, a different one will be chosen |
| 1811 |
automatically. |
| 1812 |
|
| 1813 |
If the target resource already has resources deployed or the `--download-only` option was specified, |
| 1814 |
the snapshots will only be shipped to the target cluster, but not restored. |
| 1815 |
|
| 1816 |
When the snapshot you wish to ship contains a LUKS layer, the remote on the target cluster also needs |
| 1817 |
the passphrase of the source cluster set. |
| 1818 |
|
| 1819 |
[[s-shipping_snapshots-old]] |
| 1820 |
===== Shipping a snapshot in the same cluster |
| 1821 |
|
| 1822 |
WARNING: The `snapshot ship` command is considered deprecated and any bugs found |
| 1823 |
with it will not be fixed. Instead, use the `backup ship` command with a remote pointing |
| 1824 |
to your local controller. For more details, see <<s-shipping_snapshots-l2l,the previous section>>. |
| 1825 |
|
| 1826 |
Both, the source as well as the target node have to have the resource for |
| 1827 |
snapshot shipping deployed. Additionally, the target resource has to be |
| 1828 |
deactivated. |
| 1829 |
|
| 1830 |
---------------------------- |
| 1831 |
# linstor resource deactivate nodeTarget resource1 |
| 1832 |
---------------------------- |
| 1833 |
|
| 1834 |
WARNING: Deactivating a resource with DRBD in its layer-list can NOT be |
| 1835 |
reactivated again. However, a successfully shipped snapshot of a DRBD resource |
| 1836 |
can still be <<s-restoring_a_snapshot-linstor,restored into a new resource>>. |
| 1837 |
|
| 1838 |
To manually start the snapshot-shipping, use: |
| 1839 |
|
| 1840 |
---------------------------- |
| 1841 |
# linstor snapshot ship --from-node nodeSource --to-node nodeTarget --resource resource1 |
| 1842 |
---------------------------- |
| 1843 |
|
| 1844 |
By default, the snapshot-shipping uses tcp ports from the range 12000-12999. To change |
| 1845 |
this range, the property `SnapshotShipping/TcpPortRange`, which accepts a to-from range, |
| 1846 |
can be set on the controller: |
| 1847 |
|
| 1848 |
---------------------------- |
| 1849 |
# linstor controller set-property SnapshotShipping/TcpPortRange 10000-12000 |
| 1850 |
---------------------------- |
| 1851 |
|
| 1852 |
A resource can also be periodically shipped. To accomplish this, it is mandatory to |
| 1853 |
set the properties `SnapshotShipping/TargetNode` as well as `SnapshotShipping/RunEvery` |
| 1854 |
on the resource-definition. |
| 1855 |
`SnapshotShipping/SourceNode` can also be set, but if omitted LINSTOR will choose |
| 1856 |
an active resource of the same resource-definition. |
| 1857 |
|
| 1858 |
To allow incremental snapshot-shipping, LINSTOR has to keep at least the last shipped |
| 1859 |
snapshot on the target node. The property `SnapshotShipping/Keep` can be used to specify |
| 1860 |
how many snapshots LINSTOR should keep. If the property is not set (or <= 0) LINSTOR |
| 1861 |
will keep the last 10 shipped snapshots by default. |
| 1862 |
|
| 1863 |
---------------------------- |
| 1864 |
# linstor resource-definition set-property resource1 SnapshotShipping/TargetNode nodeTarget |
| 1865 |
# linstor resource-definition set-property resource1 SnapshotShipping/SourceNode nodeSource |
| 1866 |
# linstor resource-definition set-property resource1 SnapshotShipping/RunEvery 15 |
| 1867 |
# linstor resource-definition set-property resource1 SnapshotShipping/Keep 5 |
| 1868 |
---------------------------- |
| 1869 |
|
| 1870 |
[[s-linstor-setupopts]] |
| 1871 |
=== Setting options for resources |
| 1872 |
|
| 1873 |
DRBD options are set using LINSTOR commands. |
| 1874 |
Configuration in files such as `/etc/drbd.d/global_common.conf` that are not |
| 1875 |
managed by LINSTOR will be ignored. |
| 1876 |
The following commands show the usage and available options: |
| 1877 |
|
| 1878 |
---------------------------- |
| 1879 |
# linstor controller drbd-options -h |
| 1880 |
# linstor resource-definition drbd-options -h |
| 1881 |
# linstor volume-definition drbd-options -h |
| 1882 |
# linstor resource drbd-peer-options -h |
| 1883 |
---------------------------- |
| 1884 |
|
| 1885 |
For instance, it is easy to set the DRBD protocol for a resource named |
| 1886 |
`backups`: |
| 1887 |
|
| 1888 |
---------------------------- |
| 1889 |
# linstor resource-definition drbd-options --protocol C backups |
| 1890 |
---------------------------- |
| 1891 |
|
| 1892 |
[[s-linstor-toggle-disk]] |
| 1893 |
=== Adding and removing disks |
| 1894 |
LINSTOR can convert resources between diskless and having a disk. |
| 1895 |
This is achieved with the `resource toggle-disk` command, |
| 1896 |
which has syntax similar to `resource create`. |
| 1897 |
|
| 1898 |
For instance, add a disk to the diskless resource `backups` on 'alpha': |
| 1899 |
|
| 1900 |
---------------------------- |
| 1901 |
# linstor resource toggle-disk alpha backups --storage-pool pool_ssd |
| 1902 |
---------------------------- |
| 1903 |
|
| 1904 |
Remove this disk again: |
| 1905 |
|
| 1906 |
---------------------------- |
| 1907 |
# linstor resource toggle-disk alpha backups --diskless |
| 1908 |
---------------------------- |
| 1909 |
|
| 1910 |
[[s-linstor-migrate-disk]] |
| 1911 |
==== Migrating disks |
| 1912 |
In order to move a resource between nodes without reducing redundancy at any point, |
| 1913 |
LINSTOR's disk migrate feature can be used. |
| 1914 |
First create a diskless resource on the target node, |
| 1915 |
and then add a disk using the `--migrate-from` option. |
| 1916 |
This will wait until the data has been synced to the new disk and then remove |
| 1917 |
the source disk. |
| 1918 |
|
| 1919 |
For example, to migrate a resource `backups` from 'alpha' to 'bravo': |
| 1920 |
|
| 1921 |
---------------------------- |
| 1922 |
# linstor resource create bravo backups --drbd-diskless |
| 1923 |
# linstor resource toggle-disk bravo backups --storage-pool pool_ssd --migrate-from alpha |
| 1924 |
---------------------------- |
| 1925 |
|
| 1926 |
[[s-linstor-proxy]] |
| 1927 |
=== DRBD Proxy with LINSTOR |
| 1928 |
|
| 1929 |
LINSTOR expects DRBD Proxy to be running on the nodes which are involved in the |
| 1930 |
relevant connections. It does not currently support connections via DRBD Proxy |
| 1931 |
on a separate node. |
| 1932 |
|
| 1933 |
Suppose our cluster consists of nodes 'alpha' and 'bravo' in a local network |
| 1934 |
and 'charlie' at a remote site, with a resource definition named `backups` |
| 1935 |
deployed to each of the nodes. Then DRBD Proxy can be enabled for the |
| 1936 |
connections to 'charlie' as follows: |
| 1937 |
|
| 1938 |
---------------------------- |
| 1939 |
# linstor drbd-proxy enable alpha charlie backups |
| 1940 |
# linstor drbd-proxy enable bravo charlie backups |
| 1941 |
---------------------------- |
| 1942 |
|
| 1943 |
The DRBD Proxy configuration can be tailored with commands such as: |
| 1944 |
|
| 1945 |
---------------------------- |
| 1946 |
# linstor drbd-proxy options backups --memlimit 100000000 |
| 1947 |
# linstor drbd-proxy compression zlib backups --level 9 |
| 1948 |
---------------------------- |
| 1949 |
|
| 1950 |
LINSTOR does not automatically optimize the DRBD configuration for |
| 1951 |
long-distance replication, so you will probably want to set some configuration |
| 1952 |
options such as the protocol: |
| 1953 |
|
| 1954 |
---------------------------- |
| 1955 |
# linstor resource-connection drbd-options alpha charlie backups --protocol A |
| 1956 |
# linstor resource-connection drbd-options bravo charlie backups --protocol A |
| 1957 |
---------------------------- |
| 1958 |
|
| 1959 |
Please contact LINBIT for assistance optimizing your configuration. |
| 1960 |
|
| 1961 |
==== Automatically enable DRBD Proxy |
| 1962 |
|
| 1963 |
LINSTOR can also be configured to automatically enable the above mentioned Proxy |
| 1964 |
connection between two nodes. For this automation, LINSTOR first needs to know |
| 1965 |
on which site each node is. |
| 1966 |
|
| 1967 |
---------------------------- |
| 1968 |
# linstor node set-property alpha Site A |
| 1969 |
# linstor node set-property bravo Site A |
| 1970 |
# linstor node set-property charlie Site B |
| 1971 |
---------------------------- |
| 1972 |
|
| 1973 |
As the `Site` property might also be used for other site-based decisions in |
| 1974 |
future features, the `DrbdProxy/AutoEnable` also has to be set to `true`: |
| 1975 |
|
| 1976 |
---------------------------- |
| 1977 |
# linstor controller set-property DrbdProxy/AutoEnable true |
| 1978 |
---------------------------- |
| 1979 |
|
| 1980 |
This property can also be set on node, resource-definition, resource and |
| 1981 |
resource-connection level (from left to right in increasing priority, whereas |
| 1982 |
the controller is the left-most, i.e. least prioritized level) |
| 1983 |
|
| 1984 |
Once this initialization steps are completed, every newly created resource |
| 1985 |
will automatically check if it has to enable DRBD proxy to any of its |
| 1986 |
peer-resources. |
| 1987 |
|
| 1988 |
|
| 1989 |
[[s-linstor-external-database]] |
| 1990 |
=== External database |
| 1991 |
|
| 1992 |
It is possible to have LINSTOR working with an external database provider |
| 1993 |
like PostgreSQL, MariaDB and since version 1.1.0 even ETCD key value store is supported. |
| 1994 |
|
| 1995 |
To use an external database there are a few additional steps to configure. |
| 1996 |
You have to create a DB/Schema and user to use for linstor, and configure this in the |
| 1997 |
`/etc/linstor/linstor.toml`. |
| 1998 |
|
| 1999 |
|
| 2000 |
[[s-postgresql]] |
| 2001 |
==== PostgreSQL |
| 2002 |
|
| 2003 |
A sample PostgreSQL `linstor.toml` looks like this: |
| 2004 |
|
| 2005 |
------------------------------------------------------ |
| 2006 |
[db] |
| 2007 |
user = "linstor" |
| 2008 |
password = "linstor" |
| 2009 |
connection_url = "jdbc:postgresql://localhost/linstor" |
| 2010 |
------------------------------------------------------ |
| 2011 |
|
| 2012 |
[[s-mariadb_mysql]] |
| 2013 |
==== MariaDB/MySQL |
| 2014 |
|
| 2015 |
A sample MariaDB `linstor.toml` looks like this: |
| 2016 |
|
| 2017 |
------------------------------------------------------ |
| 2018 |
[db] |
| 2019 |
user = "linstor" |
| 2020 |
password = "linstor" |
| 2021 |
connection_url = "jdbc:mariadb://localhost/LINSTOR?createDatabaseIfNotExist=true" |
| 2022 |
------------------------------------------------------ |
| 2023 |
|
| 2024 |
NOTE: The LINSTOR schema/database is created as `LINSTOR` so make sure the MariaDB connection string |
| 2025 |
refers to the `LINSTOR` schema, as in the example above. |
| 2026 |
|
| 2027 |
|
| 2028 |
[[s-etcd]] |
| 2029 |
==== ETCD |
| 2030 |
|
| 2031 |
ETCD is a distributed key-value store that makes it easy to keep your LINSTOR database distributed in a HA-setup. |
| 2032 |
The ETCD driver is already included in the LINSTOR-controller package and only needs to be configured in the `linstor.toml`. |
| 2033 |
|
| 2034 |
More information on how to install and configure ETCD can be found here: https://etcd.io/docs[ETCD docs] |
| 2035 |
|
| 2036 |
And here is a sample [db] section from the `linstor.toml`: |
| 2037 |
|
| 2038 |
------------------------------------------------------ |
| 2039 |
[db] |
| 2040 |
## only set user/password if you want to use authentication, only since LINSTOR 1.2.1 |
| 2041 |
# user = "linstor" |
| 2042 |
# password = "linstor" |
| 2043 |
|
| 2044 |
## for etcd |
| 2045 |
## do not set user field if no authentication required |
| 2046 |
connection_url = "etcd://etcdhost1:2379,etcdhost2:2379,etcdhost3:2379" |
| 2047 |
|
| 2048 |
## if you want to use TLS, only since LINSTOR 1.2.1 |
| 2049 |
# ca_certificate = "ca.pem" |
| 2050 |
# client_certificate = "client.pem" |
| 2051 |
|
| 2052 |
## if you want to use client TLS authentication too, only since LINSTOR 1.2.1 |
| 2053 |
# client_key_pkcs8_pem = "client-key.pkcs8" |
| 2054 |
## set client_key_password if private key has a password |
| 2055 |
# client_key_password = "mysecret" |
| 2056 |
------------------------------------------------------ |
| 2057 |
|
| 2058 |
[[s-linstor-rest-api]] |
| 2059 |
=== LINSTOR REST-API |
| 2060 |
|
| 2061 |
To make LINSTOR's administrative tasks more accessible and also available for web-frontends a |
| 2062 |
REST-API has been created. The REST-API is embedded in the linstor-controller |
| 2063 |
and since LINSTOR 0.9.13 configured via the `linstor.toml` configuration file. |
| 2064 |
|
| 2065 |
--------- |
| 2066 |
[http] |
| 2067 |
enabled = true |
| 2068 |
port = 3370 |
| 2069 |
listen_addr = "127.0.0.1" # to disable remote access |
| 2070 |
--------- |
| 2071 |
|
| 2072 |
|
| 2073 |
If you want to use the REST-API the current documentation can be found on the following link: |
| 2074 |
https://app.swaggerhub.com/apis-docs/Linstor/Linstor/ |
| 2075 |
|
| 2076 |
[[s-linstor-rest-api-https]] |
| 2077 |
==== LINSTOR REST-API HTTPS |
| 2078 |
|
| 2079 |
The HTTP REST-API can also run secured by HTTPS and is highly recommended if you use any features that |
| 2080 |
require authorization. Todo so you have to create a java keystore file with a valid certificate |
| 2081 |
that will be used to encrypt all HTTPS traffic. |
| 2082 |
|
| 2083 |
Here is a simple example on how you can create a self signed certificate with the `keytool` that is included |
| 2084 |
in the java runtime: |
| 2085 |
|
| 2086 |
--------- |
| 2087 |
keytool -keyalg rsa -keysize 2048 -genkey -keystore ./keystore_linstor.jks\ |
| 2088 |
-alias linstor_controller\ |
| 2089 |
-dname "CN=localhost, OU=SecureUnit, O=ExampleOrg, L=Vienna, ST=Austria, C=AT" |
| 2090 |
--------- |
| 2091 |
|
| 2092 |
`keytool` will ask for a password to secure the generated keystore file and is needed for the |
| 2093 |
LINSTOR-controller configuration. |
| 2094 |
In your `linstor.toml` file you have to add the following section: |
| 2095 |
|
| 2096 |
--------- |
| 2097 |
[https] |
| 2098 |
keystore = "/path/to/keystore_linstor.jks" |
| 2099 |
keystore_password = "linstor" |
| 2100 |
--------- |
| 2101 |
|
| 2102 |
Now (re)start the `linstor-controller` and the HTTPS REST-API should be available on port 3371. |
| 2103 |
|
| 2104 |
More information on how to import other certificates can be found here: |
| 2105 |
https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html |
| 2106 |
|
| 2107 |
NOTE: When HTTPS is enabled, all requests to the HTTP /v1/ REST-API will be redirected to the |
| 2108 |
HTTPS redirect. |
| 2109 |
|
| 2110 |
|
| 2111 |
[[s-rest-api-https-restricted-client]] |
| 2112 |
===== LINSTOR REST-API HTTPS restricted client access |
| 2113 |
|
| 2114 |
Client access can be restricted by using a SSL truststore on the Controller. |
| 2115 |
Basically you create a certificate for your client and add it to your truststore and the client |
| 2116 |
then uses this certificate for authentication. |
| 2117 |
|
| 2118 |
First create a client certificate: |
| 2119 |
|
| 2120 |
--------- |
| 2121 |
keytool -keyalg rsa -keysize 2048 -genkey -keystore client.jks\ |
| 2122 |
-storepass linstor -keypass linstor\ |
| 2123 |
-alias client1\ |
| 2124 |
-dname "CN=Client Cert, OU=client, O=Example, L=Vienna, ST=Austria, C=AT" |
| 2125 |
--------- |
| 2126 |
|
| 2127 |
Then we import this certificate to our controller truststore: |
| 2128 |
|
| 2129 |
--------- |
| 2130 |
keytool -importkeystore\ |
| 2131 |
-srcstorepass linstor -deststorepass linstor -keypass linstor\ |
| 2132 |
-srckeystore client.jks -destkeystore trustore_client.jks |
| 2133 |
--------- |
| 2134 |
|
| 2135 |
And enable the truststore in the `linstor.toml` configuration file: |
| 2136 |
|
| 2137 |
--------- |
| 2138 |
[https] |
| 2139 |
keystore = "/path/to/keystore_linstor.jks" |
| 2140 |
keystore_password = "linstor" |
| 2141 |
truststore = "/path/to/trustore_client.jks" |
| 2142 |
truststore_password = "linstor" |
| 2143 |
--------- |
| 2144 |
|
| 2145 |
Now restart the Controller and it will no longer be possible to access the controller API without a |
| 2146 |
correct certificate. |
| 2147 |
|
| 2148 |
The LINSTOR client needs the certificate in PEM format, so before we can use it we have to convert |
| 2149 |
the java keystore certificate to the PEM format. |
| 2150 |
|
| 2151 |
--------- |
| 2152 |
# Convert to pkcs12 |
| 2153 |
keytool -importkeystore -srckeystore client.jks -destkeystore client.p12\ |
| 2154 |
-storepass linstor -keypass linstor\ |
| 2155 |
-srcalias client1 -srcstoretype jks -deststoretype pkcs12 |
| 2156 |
|
| 2157 |
# use openssl to convert to PEM |
| 2158 |
openssl pkcs12 -in client.p12 -out client_with_pass.pem |
| 2159 |
--------- |
| 2160 |
|
| 2161 |
To avoid entering the PEM file password all the time it might be convenient to remove the password. |
| 2162 |
|
| 2163 |
--------- |
| 2164 |
openssl rsa -in client_with_pass.pem -out client1.pem |
| 2165 |
openssl x509 -in client_with_pass.pem >> client1.pem |
| 2166 |
--------- |
| 2167 |
|
| 2168 |
Now this PEM file can easily be used in the client: |
| 2169 |
|
| 2170 |
--------- |
| 2171 |
linstor --certfile client1.pem node list |
| 2172 |
--------- |
| 2173 |
|
| 2174 |
|
| 2175 |
The `--certfile` parameter can also added to the client configuration file, see |
| 2176 |
<<s-using_the_linstor_client>> for more details. |
| 2177 |
|
| 2178 |
|
| 2179 |
[[s-linstor-logging]] |
| 2180 |
=== Logging |
| 2181 |
|
| 2182 |
Linstor uses https://www.slf4j.org/[SLF4J] with https://logback.qos.ch/[Logback] as binding. This gives |
| 2183 |
Linstor the possibility to distinguish between the log levels `ERROR`, `WARN`, `INFO`, `DEBUG` and `TRACE` |
| 2184 |
(in order of increasing verbosity). In the current linstor version (1.1.2) the user has the following |
| 2185 |
four methods to control the logging level, ordered by priority (first has highest priority): |
| 2186 |
|
| 2187 |
|
| 2188 |
. `TRACE` mode can be `enabled` or `disabled` using the debug console: |
| 2189 |
+ |
| 2190 |
---- |
| 2191 |
Command ==> SetTrcMode MODE(enabled) |
| 2192 |
SetTrcMode Set TRACE level logging mode |
| 2193 |
New TRACE level logging mode: ENABLED |
| 2194 |
---- |
| 2195 |
|
| 2196 |
. When starting the controller or satellite a command line argument can be passed: |
| 2197 |
+ |
| 2198 |
---- |
| 2199 |
java ... com.linbit.linstor.core.Controller ... --log-level INFO |
| 2200 |
java ... com.linbit.linstor.core.Satellite ... --log-level INFO |
| 2201 |
---- |
| 2202 |
|
| 2203 |
. The recommended place is the `logging` section in the configuration file. |
| 2204 |
The default configuration file location is `/etc/linstor/linstor.toml` for the controller |
| 2205 |
and `/etc/linstor/linstor_satellite.toml` for the satellite. |
| 2206 |
Configure the logging level as follows: |
| 2207 |
+ |
| 2208 |
---- |
| 2209 |
[logging] |
| 2210 |
level="INFO" |
| 2211 |
---- |
| 2212 |
|
| 2213 |
. As Linstor is using Logback as an implementation, `/usr/share/linstor-server/lib/logback.xml` can |
| 2214 |
also be used. Currently only this approach supports different log levels for different components, like |
| 2215 |
shown in the example below: |
| 2216 |
+ |
| 2217 |
---- |
| 2218 |
<?xml version="1.0" encoding="UTF-8"?> |
| 2219 |
<configuration scan="false" scanPeriod="60 seconds"> |
| 2220 |
<!-- |
| 2221 |
Values for scanPeriod can be specified in units of milliseconds, seconds, minutes or hours |
| 2222 |
https://logback.qos.ch/manual/configuration.html |
| 2223 |
--> |
| 2224 |
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"> |
| 2225 |
<!-- encoders are assigned the type |
| 2226 |
ch.qos.logback.classic.encoder.PatternLayoutEncoder by default --> |
| 2227 |
<encoder> |
| 2228 |
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n</pattern> |
| 2229 |
</encoder> |
| 2230 |
</appender> |
| 2231 |
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> |
| 2232 |
<file>${log.directory}/linstor-${log.module}.log</file> |
| 2233 |
<append>true</append> |
| 2234 |
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> |
| 2235 |
<Pattern>%d{yyyy_MM_dd HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n</Pattern> |
| 2236 |
</encoder> |
| 2237 |
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy"> |
| 2238 |
<FileNamePattern>logs/linstor-${log.module}.%i.log.zip</FileNamePattern> |
| 2239 |
<MinIndex>1</MinIndex> |
| 2240 |
<MaxIndex>10</MaxIndex> |
| 2241 |
</rollingPolicy> |
| 2242 |
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy"> |
| 2243 |
<MaxFileSize>2MB</MaxFileSize> |
| 2244 |
</triggeringPolicy> |
| 2245 |
</appender> |
| 2246 |
<logger name="LINSTOR/Controller" level="INFO" additivity="false"> |
| 2247 |
<appender-ref ref="STDOUT" /> |
| 2248 |
<!-- <appender-ref ref="FILE" /> --> |
| 2249 |
</logger> |
| 2250 |
<logger name="LINSTOR/Satellite" level="INFO" additivity="false"> |
| 2251 |
<appender-ref ref="STDOUT" /> |
| 2252 |
<!-- <appender-ref ref="FILE" /> --> |
| 2253 |
</logger> |
| 2254 |
<root level="WARN"> |
| 2255 |
<appender-ref ref="STDOUT" /> |
| 2256 |
<!-- <appender-ref ref="FILE" /> --> |
| 2257 |
</root> |
| 2258 |
</configuration> |
| 2259 |
---- |
| 2260 |
|
| 2261 |
See the https://logback.qos.ch/manual/index.html[Logback Manual] to find more details about `logback.xml`. |
| 2262 |
|
| 2263 |
|
| 2264 |
When none of the configuration methods above is used Linstor will default to `INFO` log level. |
| 2265 |
|
| 2266 |
[[s-linstor-monitoring]] |
| 2267 |
=== Monitoring |
| 2268 |
|
| 2269 |
Since LINSTOR 1.8.0, a https://prometheus.io/[Prometheus] `/metrics` HTTP path is provided with LINSTOR and JVM specific |
| 2270 |
exports. |
| 2271 |
|
| 2272 |
The `/metrics` path also supports 3 GET arguments to reduce LINSTOR's reported data: |
| 2273 |
|
| 2274 |
* resource |
| 2275 |
* storage_pools |
| 2276 |
* error_reports |
| 2277 |
|
| 2278 |
These are all default `true`, to disabled e.g. error-report data: `http://localhost:3370/metrics?error_reports=false` |
| 2279 |
|
| 2280 |
==== Health check |
| 2281 |
|
| 2282 |
The LINSTOR-Controller also provides a `/health` HTTP path that will simply return HTTP-Status 200 if the controller can access its |
| 2283 |
database and all services are up and running. Otherwise it will return HTTP error status code 500 `Internal Server Error`. |
| 2284 |
|
| 2285 |
|
| 2286 |
[[s-linstor-secure-connections]] |
| 2287 |
=== Secure Satellite connections |
| 2288 |
|
| 2289 |
It is possible to have the LINSTOR use SSL secure TCP connection between controller and satellite connections. |
| 2290 |
Without going into further details on how java's SSL engine works we will give you |
| 2291 |
command line snippets using the `keytool` from java's runtime environment on how to configure |
| 2292 |
a 3 node setup using secure connections. |
| 2293 |
The node setup looks like this: |
| 2294 |
|
| 2295 |
Node `alpha` is the just the controller. |
| 2296 |
Node `bravo` and node `charlie` are just satellites. |
| 2297 |
|
| 2298 |
|
| 2299 |
Here are the commands to generate such a keystore setup, |
| 2300 |
values should of course be edited for your environment. |
| 2301 |
|
| 2302 |
--------- |
| 2303 |
# create directories to hold the key files |
| 2304 |
mkdir -p /tmp/linstor-ssl |
| 2305 |
cd /tmp/linstor-ssl |
| 2306 |
mkdir alpha bravo charlie |
| 2307 |
|
| 2308 |
|
| 2309 |
# create private keys for all nodes |
| 2310 |
keytool -keyalg rsa -keysize 2048 -genkey -keystore alpha/keystore.jks\ |
| 2311 |
-storepass linstor -keypass linstor\ |
| 2312 |
-alias alpha\ |
| 2313 |
-dname "CN=Max Mustermann, OU=alpha, O=Example, L=Vienna, ST=Austria, C=AT" |
| 2314 |
|
| 2315 |
keytool -keyalg rsa -keysize 2048 -genkey -keystore bravo/keystore.jks\ |
| 2316 |
-storepass linstor -keypass linstor\ |
| 2317 |
-alias bravo\ |
| 2318 |
-dname "CN=Max Mustermann, OU=bravo, O=Example, L=Vienna, ST=Austria, C=AT" |
| 2319 |
|
| 2320 |
keytool -keyalg rsa -keysize 2048 -genkey -keystore charlie/keystore.jks\ |
| 2321 |
-storepass linstor -keypass linstor\ |
| 2322 |
-alias charlie\ |
| 2323 |
-dname "CN=Max Mustermann, OU=charlie, O=Example, L=Vienna, ST=Austria, C=AT" |
| 2324 |
|
| 2325 |
# import truststore certificates for alpha (needs all satellite certificates) |
| 2326 |
keytool -importkeystore\ |
| 2327 |
-srcstorepass linstor -deststorepass linstor -keypass linstor\ |
| 2328 |
-srckeystore bravo/keystore.jks -destkeystore alpha/certificates.jks |
| 2329 |
|
| 2330 |
keytool -importkeystore\ |
| 2331 |
-srcstorepass linstor -deststorepass linstor -keypass linstor\ |
| 2332 |
-srckeystore charlie/keystore.jks -destkeystore alpha/certificates.jks |
| 2333 |
|
| 2334 |
# import controller certificate into satellite truststores |
| 2335 |
keytool -importkeystore\ |
| 2336 |
-srcstorepass linstor -deststorepass linstor -keypass linstor\ |
| 2337 |
-srckeystore alpha/keystore.jks -destkeystore bravo/certificates.jks |
| 2338 |
|
| 2339 |
keytool -importkeystore\ |
| 2340 |
-srcstorepass linstor -deststorepass linstor -keypass linstor\ |
| 2341 |
-srckeystore alpha/keystore.jks -destkeystore charlie/certificates.jks |
| 2342 |
|
| 2343 |
# now copy the keystore files to their host destinations |
| 2344 |
ssh root@alpha mkdir /etc/linstor/ssl |
| 2345 |
scp alpha/* root@alpha:/etc/linstor/ssl/ |
| 2346 |
ssh root@bravo mkdir /etc/linstor/ssl |
| 2347 |
scp bravo/* root@bravo:/etc/linstor/ssl/ |
| 2348 |
ssh root@charlie mkdir /etc/linstor/ssl |
| 2349 |
scp charlie/* root@charlie:/etc/linstor/ssl/ |
| 2350 |
|
| 2351 |
# generate the satellite ssl config entry |
| 2352 |
echo '[netcom] |
| 2353 |
type="ssl" |
| 2354 |
port=3367 |
| 2355 |
server_certificate="ssl/keystore.jks" |
| 2356 |
trusted_certificates="ssl/certificates.jks" |
| 2357 |
key_password="linstor" |
| 2358 |
keystore_password="linstor" |
| 2359 |
truststore_password="linstor" |
| 2360 |
ssl_protocol="TLSv1.2" |
| 2361 |
' | ssh root@bravo "cat > /etc/linstor/linstor_satellite.toml" |
| 2362 |
|
| 2363 |
echo '[netcom] |
| 2364 |
type="ssl" |
| 2365 |
port=3367 |
| 2366 |
server_certificate="ssl/keystore.jks" |
| 2367 |
trusted_certificates="ssl/certificates.jks" |
| 2368 |
key_password="linstor" |
| 2369 |
keystore_password="linstor" |
| 2370 |
truststore_password="linstor" |
| 2371 |
ssl_protocol="TLSv1.2" |
| 2372 |
' | ssh root@charlie "cat > /etc/linstor/linstor_satellite.toml" |
| 2373 |
--------- |
| 2374 |
|
| 2375 |
Now just start controller and satellites and add the nodes with `--communication-type SSL`. |
| 2376 |
|
| 2377 |
[[s-linstor-drbd-automatisms]] |
| 2378 |
=== Automatisms for DRBD-Resources |
| 2379 |
|
| 2380 |
[[s-linstor-auto-quorum]] |
| 2381 |
==== AutoQuorum Policies |
| 2382 |
|
| 2383 |
LINSTOR automatically configures quorum policies on resources *when |
| 2384 |
quorum is achievable*. This means, whenever you have at least two diskful and one |
| 2385 |
or more diskless resource assignments, or three or more diskful |
| 2386 |
resource assignments, LINSTOR will enable quorum policies for your |
| 2387 |
resources automatically. |
| 2388 |
|
| 2389 |
Inversely, LINSTOR will automatically disable quorum policies whenever |
| 2390 |
there are less than the minimum required resource assignments to |
| 2391 |
achieve quorum. |
| 2392 |
|
| 2393 |
This is controlled via the, `DrbdOptions/auto-quorum`, property which |
| 2394 |
can be applied to the _linstor-controller_, _resource-group_, and |
| 2395 |
_resource-definition_. Accepted values for the |
| 2396 |
`DrbdOptions/auto-quorum` property are `disabled`, `suspend-io`, and |
| 2397 |
`io-error`. |
| 2398 |
|
| 2399 |
Setting the `DrbdOptions/auto-quorum` property to `disabled` will |
| 2400 |
allow you to manually, or more granularly, control the quorum policies |
| 2401 |
of your resources should you so desire. |
| 2402 |
|
| 2403 |
TIP: The default policies for `DrbdOptions/auto-quorum` are `quorum |
| 2404 |
majority`, and `on-no-quorum io-error`. For more information on DRBD's |
| 2405 |
quorum features and their behavior, please refer to the |
| 2406 |
{url-drbd-ug}#s-feature-quorum[quorum section of the DRBD user's guide]. |
| 2407 |
|
| 2408 |
IMPORTANT: The `DrbdOptions/auto-quorum` policies will override any |
| 2409 |
manually configured properties if `DrbdOptions/auto-quorum` is not disabled. |
| 2410 |
|
| 2411 |
For example, to manually set the quorum policies of a _resource-group_ |
| 2412 |
named `my_ssd_group`, you would use the following commands: |
| 2413 |
|
| 2414 |
---- |
| 2415 |
# linstor resource-group set-property my_ssd_group DrbdOptions/auto-quorum disabled |
| 2416 |
# linstor resource-group set-property my_ssd_group DrbdOptions/Resource/quorum majority |
| 2417 |
# linstor resource-group set-property my_ssd_group DrbdOptions/Resource/on-no-quorum suspend-io |
| 2418 |
---- |
| 2419 |
|
| 2420 |
You may wish to disable DRBD's quorum features completely. To do that, |
| 2421 |
you would need to first disable `DrbdOptions/auto-quorum` on the |
| 2422 |
appropriate LINSTOR object, and then set the DRBD quorum features |
| 2423 |
accordingly. For example, use the following commands to disable quorum |
| 2424 |
entirely on the `my_ssd_group` _resource-group_: |
| 2425 |
|
| 2426 |
---- |
| 2427 |
# linstor resource-group set-property my_ssd_group DrbdOptions/auto-quorum disabled |
| 2428 |
# linstor resource-group set-property my_ssd_group DrbdOptions/Resource/quorum off |
| 2429 |
# linstor resource-group set-property my_ssd_group DrbdOptions/Resource/on-no-quorum |
| 2430 |
---- |
| 2431 |
|
| 2432 |
NOTE: Setting `DrbdOptions/Resource/on-no-quorum` to an empty value |
| 2433 |
in the commands above deletes the property from the object entirely. |
| 2434 |
|
| 2435 |
[[s-linstor-auto-evict]] |
| 2436 |
==== Auto-Evict |
| 2437 |
|
| 2438 |
If a satellite is offline for a prolonged period of time, LINSTOR can be configured to |
| 2439 |
declare that node as evicted. This triggers an automated reassignment of the affected |
| 2440 |
DRBD-resources to other nodes to ensure a minimum replica count is kept. |
| 2441 |
|
| 2442 |
This feature uses the following properties to adapt the behaviour. |
| 2443 |
|
| 2444 |
* `DrbdOptions/AutoEvictMinReplicaCount` sets the number of replicas that should always |
| 2445 |
be present. You can set this property on the controller to change a global default, or on |
| 2446 |
a specific resource-definition or resource-group to change it only for that |
| 2447 |
resource-definition or resource-group. If this property is left empty, the place-count |
| 2448 |
set for the auto-placer of the corresponding resource-group will be used. |
| 2449 |
|
| 2450 |
* `DrbdOptions/AutoEvictAfterTime` describes how long a node can be offline in minutes |
| 2451 |
before the eviction is triggered. You can set this property on the controller to change a |
| 2452 |
global default, or on a single node to give it a different behavior. The default value |
| 2453 |
for this property is 60 minutes. |
| 2454 |
|
| 2455 |
* `DrbdOptions/AutoEvictMaxDisconnectedNodes` sets the percentage of nodes that can |
| 2456 |
be not reachable (for whatever reason) at the same time. If more than the given percent |
| 2457 |
of nodes are offline at the same time, the auto-evict will not be triggered for any node |
| 2458 |
, since in this case LINSTOR assumes connection problems from the controller. This property |
| 2459 |
can only be set for the controller, and only accepts a value between 0 and 100. The default |
| 2460 |
value is 34. If you wish to turn the auto-evict-feature off, simply set this property |
| 2461 |
to 0. If you want to always trigger the auto-evict, regardless of how many satellites |
| 2462 |
are unreachable, set it to 100. |
| 2463 |
|
| 2464 |
* `DrbdOptions/AutoEvictAllowEviction` is an additional property that can stop a node from |
| 2465 |
being evicted. This can be useful for various cases, for example if you need to shut down |
| 2466 |
a node for maintenance. You can set this property on the controller to change a global default, |
| 2467 |
or on a single node to give it a different behavior. It accepts true and false as values and |
| 2468 |
per default is set to true on the controller. You can use this property to turn the auto-evict |
| 2469 |
feature off by setting it to false on the controller, although this might not work |
| 2470 |
completely if you already set different values for individual nodes, since those values take precedence |
| 2471 |
over the global default. |
| 2472 |
|
| 2473 |
After the linstor-controller loses the connection to a satellite, aside from trying to |
| 2474 |
reconnect, it starts a timer for that satellite. As soon as that timer exceeds |
| 2475 |
`DrbdOptions/AutoEvictAfterTime` and all of the DRBD-connections to the DRBD-resources |
| 2476 |
on that satellite are broken, the controller will check whether or not |
| 2477 |
`DrbdOptions/AutoEvictMaxDisconnectedNodes` has been met. If it hasn't, and |
| 2478 |
`DrbdOptions/AutoEvictAllowEviction` is true for the node in question, the satellite |
| 2479 |
will be marked as EVICTED. At the same time, the controller will check for every DRBD-resource whether |
| 2480 |
the number of resources is still above `DrbdOptions/AutoEvictMinReplicaCount`. If it is, |
| 2481 |
the resource in question will be marked as DELETED. If it isn't, an auto-place with the settings |
| 2482 |
from the corresponding resource-group will be started. Should the auto-place fail, the |
| 2483 |
controller will try again later when changes that might allow a different result, such |
| 2484 |
as adding a new node, have happened. Resources where an auto-place is necessary will only be |
| 2485 |
marked as DELETED if the corresponding auto-place was successful. |
| 2486 |
|
| 2487 |
The evicted satellite itself will not be able to reestablish connection with the controller. |
| 2488 |
Even if the node is up and running, a manual reconnect will fail. It is also not possible to |
| 2489 |
delete the satellite, even if it is working as it should be. The satellite can, however, be restored. |
| 2490 |
This will remove the EVICTED-flag from the satellite and allow you to use it again. Previously |
| 2491 |
configured network interfaces, storage pools, properties and similar entities as well as |
| 2492 |
non-DRBD-related resources and resources that could not be autoplaced somewhere else will |
| 2493 |
still be on the satellite. To restore a satellite, use |
| 2494 |
|
| 2495 |
------ |
| 2496 |
# linstor node restore [nodename] |
| 2497 |
------ |
| 2498 |
|
| 2499 |
Should you wish to instead throw everything that once was on that node, including the node |
| 2500 |
itself, away, you need to use the `node lost` command instead. |
| 2501 |
|
| 2502 |
[[s-linstor-qos]] |
| 2503 |
=== QoS Settings |
| 2504 |
==== Sysfs |
| 2505 |
|
| 2506 |
LINSTOR is able to set the following Sysfs settings: |
| 2507 |
|
| 2508 |
[cols="3,2"] |
| 2509 |
|=== |
| 2510 |
| SysFs | Linstor property |
| 2511 |
|
| 2512 |
| `/sys/fs/cgroup/blkio/blkio.throttle.read_bps_device` | `sys/fs/blkio_throttle_read` |
| 2513 |
| `/sys/fs/cgroup/blkio/blkio.throttle.write_bps_device` | `sys/fs/blkio_throttle_write` |
| 2514 |
| `/sys/fs/cgroup/blkio/blkio.throttle.read_iops_device` | `sys/fs/blkio_throttle_read_iops` |
| 2515 |
| `/sys/fs/cgroup/blkio/blkio.throttle.write_iops_device` | `sys/fs/blkio_throttle_write_iops` |
| 2516 |
|=== |
| 2517 |
|
| 2518 |
If a LINSTOR volume is composed of multiple "stacked" volume (for example DRBD with external |
| 2519 |
metadata will have 3 devices: backing (storage) device, metadata device and the resulting |
| 2520 |
DRBD device), setting a `sys/fs/\*` property for a Volume, only the bottom-most local |
| 2521 |
"data"-device will receive the corresponding `/sys/fs/cgroup/...` setting. |
| 2522 |
That means, in case of the example above only the backing device will receive the setting. |
| 2523 |
In case a resource-definition has an nvme-target as well as an nvme-initiator resource, |
| 2524 |
both bottom-most devices of each node will receive the setting. In case of the target the |
| 2525 |
bottom-most device will be the volume of LVM or ZFS, whereas in case of the initiator the |
| 2526 |
bottom-most device will be the connected nvme-device, regardless which other layers |
| 2527 |
are stacked on top of that. |
| 2528 |
|
| 2529 |
|
| 2530 |
[[s-linstor-getting-help]] |
| 2531 |
=== Getting help |
| 2532 |
==== From the command line |
| 2533 |
A quick way to list available commands on the command line is to type |
| 2534 |
`linstor`. |
| 2535 |
|
| 2536 |
Further information on sub-commands (e.g., list-nodes) can be retrieved in |
| 2537 |
two ways: |
| 2538 |
|
| 2539 |
---------------------------- |
| 2540 |
# linstor node list -h |
| 2541 |
# linstor help node list |
| 2542 |
---------------------------- |
| 2543 |
|
| 2544 |
Using the 'help' sub-command is especially helpful when LINSTOR is executed |
| 2545 |
in interactive mode (`linstor interactive`). |
| 2546 |
|
| 2547 |
One of the most helpful features of LINSTOR is its rich tab-completion, |
| 2548 |
which can be used to complete basically every object LINSTOR knows about |
| 2549 |
(e.g., node names, IP addresses, resource names, ...). |
| 2550 |
In the following examples, we show some possible completions, and their results: |
| 2551 |
|
| 2552 |
---------------------------- |
| 2553 |
# linstor node create alpha 1<tab> # completes the IP address if hostname can be resolved |
| 2554 |
# linstor resource create b<tab> c<tab> # linstor assign-resource backups charlie |
| 2555 |
---------------------------- |
| 2556 |
|
| 2557 |
If tab-completion does not work out of the box, please try to source the |
| 2558 |
appropriate file: |
| 2559 |
|
| 2560 |
---------------------------- |
| 2561 |
# source /etc/bash_completion.d/linstor # or |
| 2562 |
# source /usr/share/bash_completion/completions/linstor |
| 2563 |
---------------------------- |
| 2564 |
|
| 2565 |
For zsh shell users linstor-client can generate a zsh compilation file, |
| 2566 |
that has basic support for command and argument completion. |
| 2567 |
|
| 2568 |
---------------------------- |
| 2569 |
# linstor gen-zsh-completer > /usr/share/zsh/functions/Completion/Linux/_linstor |
| 2570 |
---------------------------- |
| 2571 |
|
| 2572 |
==== SOS-Report |
| 2573 |
|
| 2574 |
If something goes wrong and you need help finding the cause of the issue, you can use |
| 2575 |
|
| 2576 |
---------------------------- |
| 2577 |
# linstor sos-report create |
| 2578 |
---------------------------- |
| 2579 |
|
| 2580 |
The command above will create a new sos-report in `/var/log/linstor/controller/` on the |
| 2581 |
controller node. Alternatively you can use |
| 2582 |
|
| 2583 |
---------------------------- |
| 2584 |
# linstor sos-report download |
| 2585 |
---------------------------- |
| 2586 |
|
| 2587 |
which will create a new sos-report and additionally downloads that report to the local machine |
| 2588 |
into your current working directory. |
| 2589 |
|
| 2590 |
This sos-report contains logs and useful debug-information from several sources |
| 2591 |
(Linstor-logs, `dmesg`, versions of external tools used by Linstor, `ip a`, database dump |
| 2592 |
and many more). |
| 2593 |
These information are stored for each node in plaintext in the resulting `.tar.gz` file. |
| 2594 |
|
| 2595 |
|
| 2596 |
==== From the community |
| 2597 |
For help from the community please subscribe to our mailing list located here: https://lists.linbit.com/listinfo/drbd-user |
| 2598 |
|
| 2599 |
==== GitHub |
| 2600 |
To file bug or feature request please check out our GitHub page https://github.com/linbit |
| 2601 |
|
| 2602 |
==== Paid support and development |
| 2603 |
Alternatively, if you wish to purchase remote installation services, 24/7 support, access to certified repositories, or feature development please contact us: +1-877-454-6248 (1-877-4LINBIT) , International: +43-1-8178292-0 | [email protected] |