[[s-administrative-tasks-setup]] == Basic administrative tasks / Setup LINSTOR is a configuration management system for storage on Linux systems. It manages LVM logical volumes and/or ZFS ZVOLs on a cluster of nodes. It leverages DRBD for replication between different nodes and to provide block storage devices to users and applications. It manages snapshots, encryption and caching of HDD backed data in SSDs via bcache. // Troubleshooting for LINSTOR guide? ///// This chapter outlines typical administrative tasks encountered during day-to-day operations. It does not cover troubleshooting tasks, these are covered in detail in <>. ///// [[s-concepts_and_terms]] === Concepts and Terms This section goes over core concepts and terms that you will need to familiarize yourself with to understand how LINSTOR works and deploys storage. The section is laid out in a "ground up" approach. ==== Installable Components ===== linstor-controller A LINSTOR setup requires at least one active controller and one or more satellites. // Once the chapter on making your controller HA is done we need to link that here The _linstor-controller_ relies on a database that holds all configuration information for the whole cluster. It makes all decisions that need to have a view of the whole cluster. Multiple controllers can be used for LINSTOR but only one can be active. ===== linstor-satellite The _linstor-satellite_ runs on each node where LINSTOR consumes local storage or provides storage to services. It is stateless; it receives all the information it needs from the controller. It runs programs like `lvcreate` and `drbdadm`. It acts like a node agent. ===== linstor-client The _linstor-client_ is a command line utility that you use to issue commands to the system and to investigate the status of the system. ==== Objects Objects are the end result which LINSTOR presents to the end-user or application, such as; Kubernetes/OpenShift, a replicated block device (DRBD), NVMeOF target, etc. ===== Node Node's are a server or container that participate in a LINSTOR cluster. The _Node_ attribute defines: * Determines which LINSTOR cluster the node participates in * Sets the role of the node: Controller, Satellite, Auxiliary * *NetInterface* objects define the node's connectivity ===== NetInterface As the name implies, this is how you define the interface/address of a node's network interface. ===== Definitions Definitions define attributes of an object, they can be thought of as profile or template. Objects created will inherit the configuration defined in the definitions. A definition must be defined prior to creating the associated object. For example; you must create a _ResourceDefinition_ prior to creating the _Resource_ StoragePoolDefinition ::: * Defines the name of a storage pool ResourceDefinition ::: Resource definitions define the following attributes of a resource: * The name of a DRBD resource * The TCP port for DRBD to use for the resource's connection VolumeDefinition ::: Volume definitions define the following: * A volume of a DRBD resource * The size of the volume * The volume number of the DRBD resource's volume * The meta data properties of the volume * The minor number to use for the DRBD device associated with the DRBD volume ===== StoragePool The _StoragePool_ identifies storage in the context of LINSTOR. It defines: * The configuration of a storage pool on a specific node * The storage back-end driver to use for the storage pool on the cluster node (LVM, ZFS, etc) * The parameters and configuration to pass to the storage backed driver ===== Resource LINSTOR has now has expanded its capabilities to manage a broader set of storage technologies outside of just DRBD. A _Resource_: * Represents the placement of a DRBD resource, as defined within the _ResourceDefinition_ * Places a resource on a node in the cluster * Defines the placement of a _ResourceDefinition_ on a node ===== Volume Volumes are a subset of a _Resource_. A _Resource_ could have multiple volumes, for example you may wish to have your database stored on slower storage than your logs in your MySQL cluster. By keeping the _volumes_ under a single _resource_ you are essentially creating a consistency group. The _Volume_ attribute can define also define attributes on a more granular level. [[s-broader_context]] === Broader Context While LINSTOR might be used to make the management of DRBD more convenient, it is often integrated with software stacks higher up. Such integration exist already for Kubernetes, OpenStack, OpenNebula and Proxmox. Chapters specific to deploying LINSTOR in these environments are included in this guide. The southbound drivers used by LINSTOR are LVM, thinLVM and ZFS. [[s-packages]] === Packages LINSTOR is packaged in both the .rpm and the .deb variants: . _linstor-client_ contains the command line client program. It depends on python which is usually already installed. In RHEL 8 systems you will need to symlink python . _linstor-controller_ and _linstor-satellite_ Both contain systemd unit files for the services. They depend on Java runtime environment (JRE) version 1.8 (headless) or higher. For further detail about these packages see the <> section above. NOTE: If you have a support subscription to LINBIT, you will have access to our certified binaries via our repositories. [[s-installation]] === Installation IMPORTANT: If you want to use LINSTOR in containers skip this Topic and use the "Containers" section below for the installation. [[s-admin-linstor-ubuntu_linux]] ==== Ubuntu Linux If you want to have the option of creating replicated storage using DRBD, you will need to install _drbd-dkms_ and _drbd-utils_. These packages will need to be installed on all nodes. You will also need to choose a volume manager, either ZFS or LVM, in this instance we're using LVM. -------------------------------------------- # apt install -y drbd-dkms drbd-utils lvm2 -------------------------------------------- Depending on whether your node is a LINSTOR controller, satellite, or both (Combined) will determine what packages are required on that node. For combined type nodes, we'll need both the controller and satellite LINSTOR package. Combined node: -------------------------------------------------------------------- # apt install linstor-controller linstor-satellite linstor-client -------------------------------------------------------------------- That will make our remaining nodes our Satellites, so we'll need to install the following packages on them: ------------------------------------------------- # apt install linstor-satellite linstor-client ------------------------------------------------- [[s-admin-suse_linux_enterprise_server]] ==== SUSE Linux Enterprise Server SLES High Availability Extension (HAE) includes DRBD. On SLES, DRBD is normally installed via the software installation component of YaST2. It comes bundled with the High Availability package selection. As we download DRBD's newest module we can check if the LVM-tools are up to date as well. User who prefer a command line install may simply issue the following command to get the newest DRBD and LVM version: ---------------------- # zypper install drbd lvm2 ---------------------- Depending on whether your node is a LINSTOR controller, satellite, or both (Combined) will determine what packages are required on that node. For combined type nodes, we'll need both the controller and satellite LINSTOR package. Combined node: ------------------------------------------------------------------------ # zypper install linstor-controller linstor-satellite linstor-client ------------------------------------------------------------------------ That will make our remaining nodes our Satellites, so we'll need to install the following packages on them: ---------------------------------------------------- # zypper install linstor-satellite linstor-client ---------------------------------------------------- [[s-admin-centos]] ==== CentOS CentOS has had DRBD 8 since release 5. For DRBD 9 you'll need to look at EPEL and similar sources. Alternatively, if you have an active support contract with LINBIT you can utilize our RHEL 8 repositories. DRBD can be installed using `yum`. We can also check for the newest version of the LVM-tools as well. IMPORTANT: LINSTOR *requires* DRBD 9 if you wish to have replicated storage. This requires an external repository to be configured, either LINBIT's or a 3rd parties. ----------------------------- # yum install drbd kmod-drbd lvm2 ----------------------------- Depending on whether your node is a LINSTOR controller, satellite, or both (Combined) will determine what packages are required on that node. For combined type nodes, we'll need both the controller and satellite LINSTOR package. NOTE: On RHEL 8 systems you will need to install python2 for the linstor-client to work. Combined node: ------------------------------------------------------------------- # yum install linstor-controller linstor-satellite linstor-client ------------------------------------------------------------------- That will make our remaining nodes our Satellites, so we'll need to install the following packages on them: ------------------------------------------------ # yum install linstor-satellite linstor-client ------------------------------------------------ [[s-upgrading]] === Upgrading LINSTOR doesn't support rolling upgrade, controller and satellites must have the same version, otherwise the controller with discard the satellite with a `VERSION_MISMATCH`. But this isn't a problem, as the satellite won't do any actions as long it isn't connected to a controller and DRBD will not be disrupted by any means. If you are using the embedded default H2 database and the linstor-controller package is upgraded an automatic backup file of the database will be created in the default `/var/lib/linstor` directory. This file is a good restore point if for any reason a linstor-controller database migration should fail, than it is recommended to report the error to LINBIT and restore the old database file and downgrade to your previous controller version. If you use any external database or etcd, it is recommended to do a manually backup of your current database to have a restore point. So first upgrade the `linstor-controller`, `linstor-client` package on you controller host and restart the `linstor-controller`, the controller should start and all of it's client should show `OFFLINE(VERSION_MISMATCH)`. After that you can continue upgrading `linstor-satellite` on all satellite nodes and restart them, after a short reconnection time they should all show `ONLINE` again and your upgrade is finished. [[s-containers]] === Containers LINSTOR is also available as containers. The base images are available in LINBIT's container registry, `drbd.io`. In order to access the images, you first have to login to the registry (reach out to sales@linbit.com for credentials): ---------------------------- # docker login drbd.io ---------------------------- The containers available in this repository are: * drbd.io/drbd9-rhel8 * drbd.io/drbd9-rhel7 * drbd.io/drbd9-sles15sp1 * drbd.io/drbd9-bionic * drbd.io/drbd9-focal * drbd.io/linstor-csi * drbd.io/linstor-controller * drbd.io/linstor-satellite * drbd.io/linstor-client An up to date list of available images with versions can be retrieved by opening http://drbd.io in your browser. Make sure to access the host via "http", as the registry's images themselves are served via "https". To load the kernel module, needed only for LINSTOR satellites, you'll need to run a `drbd9-$dist` container in privileged mode. The kernel module containers either retrieve an official LINBIT package from a customer repository, use shipped packages, or they try to build the kernel modules from source. If you intend to build from source, you need to have the according kernel headers (e.g., `kernel-devel`) installed on the host. There are 4 ways to execute such a module load container: * Building from shipped source * Using a shipped/pre-built kernel module * Specifying a LINBIT node hash and a distribution. * Bind-mounting an existing repository configuration. Example building from shipped source (RHEL based): ---------------------------- # docker run -it --rm --privileged -v /lib/modules:/lib/modules \ -v /usr/src:/usr/src:ro \ drbd.io/drbd9-rhel7 ---------------------------- Example using a module shipped with the container, which is enabled by *not* bind-mounting `/usr/src`: ---------------------------- # docker run -it --rm --privileged -v /lib/modules:/lib/modules \ drbd.io/drbd9-rhel8 ---------------------------- Example using a hash and a distribution (rarely used): ---------------------------- # docker run -it --rm --privileged -v /lib/modules:/lib/modules \ -e LB_DIST=rhel7.7 -e LB_HASH=ThisIsMyNodeHash \ drbd.io/drbd9-rhel7 ---------------------------- Example using an existing repo config (rarely used): ---------------------------- # docker run -it --rm --privileged -v /lib/modules:/lib/modules \ -v /etc/yum.repos.d/linbit.repo:/etc/yum.repos.d/linbit.repo:ro \ drbd.io/drbd9-rhel7 ---------------------------- IMPORTANT: In both cases (hash + distribution, as well as bind-mounting a repo) the hash or config has to be from a node that has a special property set. Feel free to contact our support, and we set this property. IMPORTANT: For now (i.e., pre DRBD 9 version "9.0.17"), you must use the containerized DRBD kernel module, as opposed to loading a kernel module onto the host system. If you intend to use the containers you should not install the DRBD kernel module on your host systems. For DRBD version 9.0.17 or greater, you can install the kernel module as usual on the host system, but you need to make sure to load the module with the `usermode_helper=disabled` parameter (e.g., `modprobe drbd usermode_helper=disabled`). Then run the LINSTOR satellite container, also privileged, as a daemon: ---------------------------- # docker run -d --name=linstor-satellite --net=host -v /dev:/dev --privileged drbd.io/linstor-satellite ---------------------------- NOTE: `net=host` is required for the containerized `drbd-utils` to be able to communicate with the host-kernel via netlink. To run the LINSTOR controller container as a daemon, mapping ports `3370`, `3376` and `3377` on the host to the container: ---------------------------- # docker run -d --name=linstor-controller -p 3370:3370 -p 3376:3376 -p 3377:3377 drbd.io/linstor-controller ---------------------------- To interact with the containerized LINSTOR cluster, you can either use a LINSTOR client installed on a system via packages, or via the containerized LINSTOR client. To use the LINSTOR client container: ---------------------------- # docker run -it --rm -e LS_CONTROLLERS= drbd.io/linstor-client node list ---------------------------- From this point you would use the LINSTOR client to initialize your cluster and begin creating resources using the typical LINSTOR patterns. To stop and remove a daemonized container and image: ---------------------------- # docker stop linstor-controller # docker rm linstor-controller ---------------------------- [[s-linstor-init-cluster]] === Initializing your cluster We assume that the following steps are accomplished on *all* cluster nodes: . The DRBD9 kernel module is installed and loaded . `drbd-utils` are installed . `LVM` tools are installed . `linstor-controller` and/or `linstor-satellite` its dependencies are installed . The `linstor-client` is installed on the `linstor-controller` node Start and enable the linstor-controller service on the host where it has been installed: ---------------------------- # systemctl enable --now linstor-controller ---------------------------- If you are sure the linstor-controller service gets automatically enabled on installation you can use the following command as well: ------------------------------------------ # systemctl start linstor-controller ------------------------------------------ [[s-using_the_linstor_client]] === Using the LINSTOR client Whenever you run the LINSTOR command line client, it needs to know where your linstor-controller runs. If you do not specify it, it will try to reach a locally running linstor-controller listening on IP `127.0.0.1` port `3376`. Therefore we will use the `linstor-client` on the same host as the `linstor-controller`. IMPORTANT: The `linstor-satellite` requires ports 3366 and 3367. The `linstor-controller` requires ports 3376 and 3377. Make sure you have these ports allowed on your firewall. ---------------------------- # linstor node list ---------------------------- should give you an empty list and not an error message. You can use the `linstor` command on any other machine, but then you need to tell the client how to find the linstor-controller. As shown, this can be specified as a command line option, an environment variable, or in a global file: ---------------------------- # linstor --controllers=alice node list # LS_CONTROLLERS=alice linstor node list ---------------------------- Alternatively you can create the `/etc/linstor/linstor-client.conf` file and populate it like below. ----- [global] controllers=alice ----- If you have multiple linstor-controllers configured you can simply specify them all in a comma separated list. The linstor-client will simply try them in the order listed. NOTE: The linstor-client commands can also be used in a much faster and convenient way by only writing the starting letters of the parameters e.g.: `linstor node list` -> `linstor n l` [[s-adding_nodes_to_your_cluster]] === Adding nodes to your cluster The next step is to add nodes to your LINSTOR cluster. ---------------------------- # linstor node create bravo 10.43.70.3 ---------------------------- If the IP is omitted, the client will try to resolve the given node-name as host-name by itself. Linstor will automatically detect the node's local `uname -n` which is later used for the DRBD-resource. When you use `linstor node list` you will see that the new node is marked as offline. Now start and enable the linstor-satellite on that node so that the service comes up on reboot as well: ---------------------------- # systemctl enable --now linstor-satellite ---------------------------- You can also use `systemctl start linstor-satellite` if you are sure that the service is already enabled as default and comes up on reboot. About 10 seconds later you will see the status in `linstor node list` becoming online. Of course the satellite process may be started before the controller knows about the existence of the satellite node. NOTE: In case the node which hosts your controller should also contribute storage to the LINSTOR cluster, you have to add it as a node and start the linstor-satellite as well. If you want to have other services wait until the linstor-satellite had a chance to create the necessary devices (i.e. after a boot), you can update the corresponding .service file and change `Type=simple` to `Type=notify`. This will cause the satellite to delay sending the `READY=1` message to systemd until the controller connects, sends all required data to the satellite and the satellite at least tried once to get the devices up and running. [[s-storage_pools]] === Storage pools <> identify storage in the context of LINSTOR. To group storage pools from multiple nodes, simply use the same name on each node. For example, one valid approach is to give all SSDs one name and all HDDs another. On each host contributing storage, you need to create either an LVM VG or a ZFS zPool. The VGs and zPools identified with one LINSTOR storage pool name may have different VG or zPool names on the hosts, but do yourself a favor and use the same VG or zPool name on all nodes. ---------------------------- # vgcreate vg_ssd /dev/nvme0n1 /dev/nvme1n1 [...] ---------------------------- These then need to be registered with LINSTOR: ---------------------------- # linstor storage-pool create lvm alpha pool_ssd vg_ssd # linstor storage-pool create lvm bravo pool_ssd vg_ssd ---------------------------- NOTE: The storage pool name and common metadata is referred to as a _storage pool definition_. The listed commands create a storage pool definition implicitly. You can see that by using `linstor storage-pool-definition list`. Creating storage pool definitions explicitly is possible but not necessary. To list your storage-pools you can use: ------ # linstor storage-pool list ------ or using the short version ----- # linstor sp l ----- //// In case anything goes wrong with the storage pool's VG/zPool, e.g. the VG having been renamed or somehow became invalid you can delete the storage pool in LINSTOR with the following command, given that only resources with all their volumes in the so-called 'lost' storage pool are attached. This feature is available since LINSTOR v0.9.13. ------ # linstor storage-pool lost alpha pool_ssd ------ or using the short version ----- # linstor sp lo alpha pool_ssd ----- //// Should the deletion of the storage pool be prevented due to attached resources or snapshots with some of its volumes in another still functional storage pool, hints will be given in the 'status' column of the corresponding list-command (e.g. `linstor resource list`). After deleting the LINSTOR-objects in the lost storage pool manually, the lost-command can be executed again to ensure a complete deletion of the storage pool and its remaining objects. [[s-a_storage_pool_per_backend_device]] ==== A storage pool per backend device In clusters where you have only one kind of storage and the capability to hot-repair storage devices, you may choose a model where you create one storage pool per physical backing device. The advantage of this model is to confine failure domains to a single storage device. [[s-physical-storage-command]] ==== Physical storage command Since linstor-server 1.5.2 and a recent linstor-client, LINSTOR can create LVM/ZFS pools on a satellite for you. The linstor-client has the following commands to list possible disks and create storage pools, but such LVM/ZFS pools are not managed by LINSTOR and there is no delete command, so such action must be done manually on the nodes. ----- # linstor physical-storage list ----- Will give you a list of available disks grouped by size and rotational(SSD/Magnetic Disk). It will only show disks that pass the following filters: * The device size must be greater than 1GiB * The device is a root device (not having children) e.g.: /dev/vda, /dev/sda * The device does not have any file-system or other `blkid` marker (`wipefs -a` might be needed) * The device is no DRBD device With the `create-device-pool` command you can create a LVM pool on a disk and also directly add it as a storage-pool in LINSTOR. ---- # linstor physical-storage create-device-pool --pool-name lv_my_pool LVMTHIN node_alpha /dev/vdc --storage-pool newpool ---- If the `--storage-pool` option was provided, LINSTOR will create a storage-pool with the given name. For more options and exact command usage please check the linstor-client help. [[s-linstor-resource-groups]] === Resource groups A resource group is a parent object of a resource definition where all property changes made on a resource group will be inherited by it's resource definition children. The resource group also stores settings for automatic placement rules and can spawn a resource definition depending on the stored rules. In simpler terms, resource groups are like templates that define characteristics of resources created from them. Changes to these pseudo templates will be applied to all resources that were created from the resource group, retroactively. TIP: Using resource groups to define how you'd like your resources provisioned should be considered the de facto method for deploying volumes provisioned by LINSTOR. Chapters that follow which describe creating each _resource_ from a _resource-definition_ and _volume-definition_ should only be used in special scenarios. NOTE: Even if you choose not to create and use _resource-groups_ in your LINSTOR cluster, all resources created from _resource-definitions_ and _volume-definitions_ will exist in the 'DfltRscGrp' _resource-group_. A simple pattern for deploying resources using resource groups would look like this: ---- # linstor resource-group create my_ssd_group --storage-pool pool_ssd --place-count 2 # linstor volume-group create my_ssd_group # linstor resource-group spawn-resources my_ssd_group my_ssd_res 20G ---- The commands above would result in a resource named 'my_ssd_res' with a 20GB volume replicated twice being automatically provisioned from nodes who participate in the storage pool named 'pool_ssd'. A more useful pattern could be to create a resource group with settings you've determined are optimal for your use case. Perhaps you have to run nightly online verifications of your volumes' consistency, in that case, you could create a resource group with the 'verify-alg' of your choice already set so that resources spawned from the group are pre-configured with 'verify-alg' set: ---- # linstor resource-group create my_verify_group --storage-pool pool_ssd --place-count 2 # linstor resource-group drbd-options --verify-alg crc32c my_verify_group # linstor volume-group create my_verify_group # for i in {00..19}; do linstor resource-group spawn-resources my_verify_group res$i 10G done ---- The commands above result in twenty 10GiB resources being created each with the 'crc32c' 'verify-alg' pre-configured. You can tune the settings of individual resources or volumes spawned from resource groups by setting options on the respective _resource-definition_ or _volume-definition_. For example, if 'res11' from the example above is used by a very active database receiving lots of small random writes, you might want to increase the 'al-extents' for that specific resource: ---- # linstor resource-definition drbd-options --al-extents 6007 res11 ---- If you configure a setting in a _resource-definition_ that is already configured on the _resource-group_ it was spawned from, the value set in the _resource-definition_ will override the value set on the parent _resource-group_. For example, if the same 'res11' was required to use the slower but more secure 'sha256' hash algorithm in its verifications, setting the 'verify-alg' on the _resource-definition_ for 'res11' would override the value set on the _resource-group_: ---- # linstor resource-definition drbd-options --verify-alg sha256 res11 ---- TIP: A rule of thumb for the hierarchy in which settings are inherited is the value "closer" to the resource or volume wins: _volume-definition_ settings take precedence over _volume-group_ settings, and _resource-definition_ settings take precedence over _resource-group_ settings. [[s-linstor-set-config]] === Cluster configuration [[s-available_storage_plugins]] ==== Available storage plugins indexterm:[linstor, storage plugins] LINSTOR has the following supported storage plugins as of writing: * Thick LVM * Thin LVM with a single thin pool * Thick ZFS * Thin ZFS [[s-linstor-new-volume]] === Creating and deploying resources/volumes In the following scenario we assume that the goal is to create a resource 'backups' with a size of '500 GB' that is replicated among three cluster nodes. First, we create a new resource definition: ---------------------------- # linstor resource-definition create backups ---------------------------- Second, we create a new volume definition within that resource definition: ---------------------------- # linstor volume-definition create backups 500G ---------------------------- If you want to change the size of the volume-definition you can simply do that by: ------- # linstor volume-definition set-size backups 0 100G ------- The parameter `0` is the number of the volume in the resource `backups`. You have to provide this parameter , because resources can have multiple volumes and they are identified by a so called volume-number. This number can be found by listing the volume-definitions. IMPORTANT: The size of a volume-definition can only be decreased if it has no resource. Despite of that the size can be increased even with an deployed resource. So far we have only created objects in LINSTOR's database, not a single LV was created on the storage nodes. Now you have the choice of delegating the task of placement to LINSTOR or doing it yourself. [[s-manual_placement]] ==== Manual placement With the `resource create` command you may assign a resource definition to named nodes explicitly. ---------------------------- # linstor resource create alpha backups --storage-pool pool_hdd # linstor resource create bravo backups --storage-pool pool_hdd # linstor resource create charlie backups --storage-pool pool_hdd ---------------------------- [[s-autoplace-linstor]] ==== Autoplace The value after autoplace tells LINSTOR how many replicas you want to have. The storage-pool option should be obvious. ---------------------------- # linstor resource create backups --auto-place 3 --storage-pool pool_hdd ---------------------------- Maybe not so obvious is that you may omit the `--storage-pool` option, then LINSTOR may select a storage pool on its own. The selection follows these rules: * Ignore all nodes and storage pools the current user has no access to * Ignore all diskless storage pools * Ignore all storage pools not having enough free space The remaining storage pools will be rated by different strategies. LINSTOR has currently three strategies: * `MaxFreeSpace`: This strategy maps the rating 1:1 to the remaining free space of the storage pool. However, this strategy only considers the actually allocated space (in case of thinly provisioned storage pool this might grow with time without creating new resources) * `MinReservedSpace`: Unlike the "MaxFreeSpace", this strategy considers the reserved spaced. That is the space that a thin volume can grow to before reaching its limit. The sum of reserved spaces might exceed the storage pools capacity, which is as overprovisioning. * `MinRscCount`: Simply the count of resources already deployed in a given storage pool * `MaxThroughput`: For this strategy, the storage pool's `Autoplacer/MaxThroughput` property is the base of the score, or 0 if the property is not present. Every Volume deployed in the given storage pool will subtract its defined `sys/fs/blkio_throttle_read` and `sys/fs/blkio_throttle_write` property- value from the storage pool's max throughput. The resulting score might be negative. The scores of the strategies will be normalized, weighted and summed up, where the scores of minimizing strategies will be converted first to allow an overall maximization of the resulting score. The weights of the strategies can be configured with ---- linstor controller set-property Autoplacer/Weights/$name_of_the_strategy $weight ---- whereas the strategy-names are listed above and the weight can be an arbitrary decimal. NOTE: To keep the behaviour of the autoplacer similar to the old one (due to compatibility), all strategies have a default-weight of 0, except the `MaxFreeSpace` which has a weight of 1. NOTE: Neither 0 nor a negative score will prevent a storage pool from getting selected, just making them to be considered later. Finally LINSTOR tries to find the best matching group of storage pools meeting all requirements. This step also considers other autoplacement restrictions as `--replicas-on-same`, `--replicas-on-different` and others. These two arguments, `--replicas-on-same` and `--replicas-on-different` expect the name of a property within the `Aux/` namespace. The following example shows that the client automatically prefixes the `testProperty` with the `Aux/` namespace. ---- linstor resource-group create testRscGrp --replicas-on-same testProperty SUCCESS: Description: New resource group 'testRscGrp' created. Details: Resource group 'testRscGrp' UUID is: 35e043cb-65ab-49ac-9920-ccbf48f7e27d linstor resource-group list +-----------------------------------------------------------------------------+ | ResourceGroup | SelectFilter | VlmNrs | Description | |-============================================================================| | DfltRscGrp | PlaceCount: 2 | | | |-----------------------------------------------------------------------------| | testRscGrp | PlaceCount: 2 | | | | | ReplicasOnSame: ['Aux/testProperty'] | | | +-----------------------------------------------------------------------------+ ---- NOTE: If everything went right the DRBD-resource has now been created by LINSTOR. This can be checked by looking for the DRBD block device with the `lsblk` command which should look like `drbd0000` or similar. Now we should be able to mount the block device of our resource and start using LINSTOR. [[s-more-about-linstor]] == Further LINSTOR tasks [[s-linstor_ha]] === LINSTOR high availability By default a LINSTOR cluster consists of exactly one LINSTOR controller. Making LINSTOR highly-available involves providing replicated storage for the controller database, multiple LINSTOR controllers where only one is active at a time, and a service manager that takes care of mounting/unmounting the highly-available storage and starting/stopping LINSTOR controllers. ==== Highly-available Storage For configuring the highly-available storage we use LINSTOR itself. This has the advantage that the storage is under LINSTOR control and can for example be easily extended to new cluster nodes. Just create a new resource with 200MB in size. This could look like this, you certainly need to adapt the storage pool name: ---------------------------- # linstor resource-definition create linstor_db # linstor resource-definition drbd-options --on-no-quorum=io-error linstor_db # linstor resource-definition drbd-options --auto-promote=no linstor_db # linstor volume-definition create linstor_db 200M # linstor resource create linstor_db -s pool1 --auto-place 3 ---------------------------- From now on we assume the resource's name is "linstor_db". It is crucial that your cluster qualifies for auto-quorum and uses the `io-error` policy (see Section <>), and that `auto-promote` is disabled. After the resource is created, it is time to move the LINSTOR DB to the new storage and to create a `systemd` mount service. First we stop the current controller and disable it, as it will be managed by `drbd-reactor` later. ---------------------------- # systemctl disable --now linstor-controller # cat << EOF > /etc/systemd/system/var-lib-linstor.mount [Unit] Description=Filesystem for the LINSTOR controller [Mount] # you can use the minor like /dev/drbdX or the udev symlink What=/dev/drbd/by-res/linstor_db/0 Where=/var/lib/linstor EOF # mv /var/lib/linstor{,.orig} # mkdir /var/lib/linstor # chattr +i /var/lib/linstor # only if on LINSTOR >= 1.14.0 # drbdadm primary linstor_db # mkfs.ext4 /dev/drbd/by-res/linstor_db/0 # systemctl start var-lib-linstor.mount # cp -r /var/lib/linstor.orig/* /var/lib/linstor # systemctl start linstor-controller ---------------------------- copy the `/etc/systemd/system/var-lib-linstor.mount` mount file to all the standby nodes for the linstor controller. Again, do not `systemctl enable` any of these services, they get managed by `drbd-reactor`. ==== Multiple LINSTOR controllers The next step is to install LINSTOR controllers on all nodes that have access to the `linstor_db` DRBD resource (as they need to mount the DRBD volume) and which you want to become a possible LINSTOR controller. It is important that the controllers are manged by `drbd-reactor`, so make sure the `linstor-controller.service` is disabled on all nodes! To be sure, execute `systemctl disable linstor-controller` on all cluster nodes and `systemctl stop linstor-controller` on all nodes except the one it is currently running from the previous step. Also make sure to set `chattr +i /var/lib/linstor` on all potential controller nodes if you use LINSTOR version equal or greater to 1.14.0. ==== Managing the services For starting and stopping the mount service and the linstor-controller service we use `drbd-reactor`. Install this component on all nodes that could become a LINSTOR controller and edit their `/etc/drbd-reactor.d/linstor_db.toml` configuration file. It should contain an enabled promoter plugin section like this: ---------------------------- [[promoter]] id = "linstor_db" [promoter.resources.linstor_db] start = ["var-lib-linstor.mount", "linstor-controller.service"] ---------------------------- Depending on your requirements you might also want to set an `on-stop-failure` action and set `stop-services-on-exit`. After that restart `drbd-reactor` and enable it on all the nodes you configured it. ---------------------------- # systemctl restart drbd-reactor # systemctl enable drbd-reactor ---------------------------- Check that there are no warnings from drbd-reactor in the logs by running `systemctl status drbd-reactor`. As there is already an active LINSTOR controller things will just stay the way they are. Run `drbd-reactorctl status linstor_db` to check the health of the linstor_db target unit. The last but never the less important step is to configure the LINSTOR satellite services to not delete (and then regenerate) the resource file for the LINSTOR controller DB at its startup. Do not edit the service files directly, but use `systemctl edit`. Edit the service file on all nodes that could become a LINSTOR controller and that are also LINSTOR satellites. -------------- # systemctl edit linstor-satellite [Service] Environment=LS_KEEP_RES=linstor_db -------------- After this change you should execute `systemctl restart linstor-satellite` on all satellite nodes. CAUTION: Be sure to configure your LINSTOR client for use with multiple controllers as described in the section titled, <> and make sure you also configured your integration plugins (e.g., the Proxmox plugin) to be ready for multiple LINSTOR controllers. [[s-drbd_clients]] === DRBD clients By using the `--drbd-diskless` option instead of `--storage-pool` you can have a permanently diskless DRBD device on a node. This means that the resource will appear as block device and can be mounted to the filesystem without an existing storage-device. The data of the resource is accessed over the network on another nodes with the same resource. ---------------------------- # linstor resource create delta backups --drbd-diskless ---------------------------- NOTE: The option `--diskless` was deprecated. Please use `--drbd-diskless` or `--nvme-initiator` instead. [[s-linstor-drbd-consistency-group-multiple-volumes]] === LINSTOR - DRBD consistency group/multiple volumes The so called consistency group is a feature from DRBD. It is mentioned in this user-guide, due to the fact that one of LINSTOR's main functions is to manage storage-clusters with DRBD. Multiple volumes in one resource are a consistency group. This means that changes on different volumes from one resource are getting replicated in the same chronological order on the other Satellites. Therefore you don't have to worry about the timing if you have interdependent data on different volumes in a resource. To deploy more than one volume in a LINSTOR-resource you have to create two volume-definitions with the same name. ---- # linstor volume-definition create backups 500G # linstor volume-definition create backups 100G ---- [[s-volumes_of_one_resource_to_different_storage_pools]] === Volumes of one resource to different Storage-Pools This can be achieved by setting the `StorPoolName` property to the volume definitions before the resource is deployed to the nodes: ---------------------------- # linstor resource-definition create backups # linstor volume-definition create backups 500G # linstor volume-definition create backups 100G # linstor volume-definition set-property backups 0 StorPoolName pool_hdd # linstor volume-definition set-property backups 1 StorPoolName pool_ssd # linstor resource create alpha backups # linstor resource create bravo backups # linstor resource create charlie backups ---------------------------- NOTE: Since the `volume-definition create` command is used without the `--vlmnr` option LINSTOR assigned the volume numbers starting at 0. In the following two lines the 0 and 1 refer to these automatically assigned volume numbers. Here the 'resource create' commands do not need a `--storage-pool` option. In this case LINSTOR uses a 'fallback' storage pool. Finding that storage pool, LINSTOR queries the properties of the following objects in the following order: * Volume definition * Resource * Resource definition * Node If none of those objects contain a `StorPoolName` property, the controller falls back to a hard-coded 'DfltStorPool' string as a storage pool. This also means that if you forgot to define a storage pool prior deploying a resource, you will get an error message that LINSTOR could not find the storage pool named 'DfltStorPool'. [[s-linstor-without-drbd]] === LINSTOR without DRBD LINSTOR can be used without DRBD as well. Without DRBD, LINSTOR is able to provision volumes from LVM and ZFS backed storage pools, and create those volumes on individual nodes in your LINSTOR cluster. Currently LINSTOR supports the creation of LVM and ZFS volumes with the option of layering some combinations of LUKS, DRBD, and/or NVMe-oF/NVMe-TCP on top of those volumes. For example, assume we have a Thin LVM backed storage pool defined in our LINSTOR cluster named, `thin-lvm`: ---- # linstor --no-utf8 storage-pool list +--------------------------------------------------------------+ | StoragePool | Node | Driver | PoolName | ... | |--------------------------------------------------------------| | thin-lvm | linstor-a | LVM_THIN | drbdpool/thinpool | ... | | thin-lvm | linstor-b | LVM_THIN | drbdpool/thinpool | ... | | thin-lvm | linstor-c | LVM_THIN | drbdpool/thinpool | ... | | thin-lvm | linstor-d | LVM_THIN | drbdpool/thinpool | ... | +--------------------------------------------------------------+ ---- We could use LINSTOR to create a Thin LVM on `linstor-d` that's 100GiB in size using the following commands: ---- # linstor resource-definition create rsc-1 # linstor volume-definition create rsc-1 100GiB # linstor resource create --layer-list storage \ --storage-pool thin-lvm linstor-d rsc-1 ---- You should then see you have a new Thin LVM on `linstor-d`. You can extract the device path from LINSTOR by listing your linstor resources with the `--machine-readable` flag set: ---- # linstor --machine-readable resource list | grep device_path "device_path": "/dev/drbdpool/rsc-1_00000", ---- If you wanted to layer DRBD on top of this volume, which is the default `--layer-list` option in LINSTOR for ZFS or LVM backed volumes, you would use the following resource creation pattern instead: ---- # linstor resource-definition create rsc-1 # linstor volume-definition create rsc-1 100GiB # linstor resource create --layer-list drbd,storage \ --storage-pool thin-lvm linstor-d rsc-1 ---- You would then see that you have a new Thin LVM backing a DRBD volume on `linstor-d`: ---- # linstor --machine-readable resource list | grep -e device_path -e backing_disk "device_path": "/dev/drbd1000", "backing_disk": "/dev/drbdpool/rsc-1_00000", ---- The following table shows which layer can be followed by which child-layer: [cols=">1,<5"] |=== | Layer | Child layer | DRBD | CACHE, WRITECACHE, NVME, LUKS, STORAGE | CACHE | WRITECACHE, NVME, LUKS, STORAGE | WRITECACHE | CACHE, NVME, LUKS, STORAGE | NVME | CACHE, WRITECACHE, LUKS, STORAGE | LUKS | STORAGE | STORAGE | - |=== NOTE: One layer can only occur once in the layer-list TIP: For information about the prerequisites for the `LUKS` layer, refer to the Encrypted Volumes section of this User's Guide. [[s-nvme-layer]] ==== NVMe-oF/NVMe-TCP LINSTOR Layer NVMe-oF/NVMe-TCP allows LINSTOR to connect diskless resources to a node with the same resource where the data is stored over NVMe fabrics. This leads to the advantage that resources can be mounted without using local storage by accessing the data over the network. LINSTOR is not using DRBD in this case, and therefore NVMe resources provisioned by LINSTOR are not replicated, the data is stored on one node. NOTE: NVMe-oF only works on RDMA-capable networks and NVMe-TCP on every network that can carry IP traffic. If you want to know more about NVMe-oF/NVMe-TCP visit https://www.linbit.com/en/nvme-linstor-swordfish/ for more information. To use NVMe-oF/NVMe-TCP with LINSTOR the package `nvme-cli` needs to be installed on every Node which acts as a Satellite and will use NVMe-oF/NVMe-TCP for a resource: IMPORTANT: If you are not using Ubuntu use the suitable command for installing packages on your OS - SLES: zypper - CentOS: yum ------ # apt install nvme-cli ------ To make a resource which uses NVMe-oF/NVMe-TCP an additional parameter has to be given as you create the resource-definition: ------ # linstor resource-definition create nvmedata -l nvme,storage ------ NOTE: As default the -l (layer-stack) parameter is set to `drbd, storage` when DRBD is used. If you want to create LINSTOR resources with neither NVMe nor DRBD you have to set the `-l` parameter to only `storage`. In order to use NVMe-TCP instead of the default NVMe-oF the following property needs to be set: ------ # linstor resource-definition set-property nvmedata NVMe/TRType tcp ------ The property `NVMe/TRType` can alternatively be set on resource-group or controller level. Next, create the volume-definition for our resource: ------ # linstor volume-definition create nvmedata 500G ------ Before you create the resource on your nodes you have to know where the data will be stored locally and which node accesses it over the network. First we create the resource on the node where our data will be stored: ------ # linstor resource create alpha nvmedata --storage-pool pool_ssd ------ On the nodes where the resource-data will be accessed over the network, the resource has to be defined as diskless: ----- # linstor resource create beta nvmedata --nvme-initiator ----- Now you can mount the resource `nvmedata` on one of your nodes. IMPORTANT: If your nodes have more than one NIC you should force the route between them for NVMe-of/NVME-TCP, otherwise multiple NICs could cause troubles. [[s-openflex]] ==== OpenFlex(TM) Layer Since version 1.5.0 the additional Layer `openflex` can be used in LINSTOR. From LINSTOR's perspective, the https://www.westerndigital.com/products/storage-platforms/openflex-composable-infrastructure[OpenFlex Composable Infrastructure] takes the role of a combined layer acting as a storage layer (like LVM) and also providing the allocated space as an NVMe target. OpenFlex has a REST API which is also used by LINSTOR to operate with. As OpenFlex combines concepts of LINSTOR's storage as well as NVMe-layer, LINSTOR was added both, a new storage driver for the storage pools as well as a dedicated `openflex` layer which uses the mentioned REST API. In order for LINSTOR to communicate with the OpenFlex-API, LINSTOR needs some additional properties, which can be set once on `controller` level to take LINSTOR-cluster wide effect: * `StorDriver/Openflex/ApiHost` specifies the host or IP of the API entry-point * `StorDriver/Openflex/ApiPort` this property is glued with a colon to the previous to form the basic `http://ip:port` part used by the REST calls * `StorDriver/Openflex/UserName` the REST username * `StorDriver/Openflex/UserPassword` the password for the REST user Once that is configured, we can now create LINSTOR objects to represent the OpenFlex architecture. The theoretical mapping of LINSTOR objects to OpenFlex objects are as follows: Obviously an OpenFlex storage pool is represented by a LINSTOR storage pool. As the next thing above a LINSTOR storage pool is already the node, a LINSTOR node represents an OpenFlex storage device. The OpenFlex objects above storage device are not mapped by LINSTOR. When using NVMe, LINSTOR was designed to run on both sides, the NVMe target as well as on the NVMe initiator side. In the case of OpenFlex, LINSTOR cannot (or even should not) run on the NVMe target side as that is completely managed by OpenFlex. As LINSTOR still needs nodes and storage pools to represent the OpenFlex counterparts, the LINSTOR client was extended with special node create commands since 1.0.14. These commands not only accept additionally needed configuration data, but also starts a "special satellite" besides the already running controller instance. This special satellites are completely LINSTOR managed, they will shutdown when the controller shuts down and will be started again when the controller starts. The new client command for creating a "special satellite" representing an OpenFlex storage device is: ---- $ linstor node create-openflex-target ofNode1 192.168.166.7 000af795789d ---- The arguments are as follows: * `ofNode1` is the node name which is also used by the standard `linstor node create` command * `192.168.166.7` is the address on which the provided NVMe devices can be accessed. As the NVMe devices are accessed by a dedicated network interface, this address differs from the address specified with the property `StorDriver/Openflex/ApiHost`. The latter is used for the management / REST API. * `000af795789d` is the identifier for the OpenFlex storage device. The last step of the configuration is the creation of LINSTOR storage pools: ---- $ linstor storage-pool create openflex ofNode1 sp0 0 ---- * `ofNode1` and `sp0` are the node name and storage pool name, respectively, just as usual for the LINSTOR's create storage pool command * The last `0` is the identifier of the OpenFlex storage pool within the previously defined storage device Once all necessary storage pools are created in LINSTOR, the next steps are similar to the usage of using an NVMe resource with LINSTOR. Here is a complete example: ---- # set the properties once linstor controller set-property StorDriver/Openflex/ApiHost 10.43.7.185 linstor controller set-property StorDriver/Openflex/ApiPort 80 linstor controller set-property StorDriver/Openflex/UserName myusername linstor controller set-property StorDriver/Openflex/UserPassword mypassword # create a node for openflex storage device "000af795789d" linstor node create-openflex-target ofNode1 192.168.166.7 000af795789d # create a usual linstor satellite. later used as nvme initiator linstor node create bravo # create a storage pool for openflex storage pool "0" within storage device "000af795789d" linstor storage-pool create openflex ofNode1 sp0 0 # create resource- and volume-definition linstor resource-definition create backupRsc linstor volume-definition create backupRsc 10G # create openflex-based nvme target linstor resource create ofNode1 backupRsc --storage-pool sp0 --layer-list openflex # create openflex-based nvme initiator linstor resource create bravo backupRsc --nvme-initiator --layer-list openflex ---- NOTE: In case a node should access the OpenFlex REST API through a different host than specified with + `linstor controller set-property StorDriver/Openflex/ApiHost 10.43.7.185` you can always use LINSTOR's inheritance mechanism for properties. That means simply define the same property on the node-level you need it, i.e. + `linstor node set-property ofNode1 StorDriver/Openflex/ApiHost 10.43.8.185` [[s-writecache-layer]] ==== Writecache Layer A https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/writecache.html[DM-Writecache] device is composed by two devices, one storage device and one cache device. LINSTOR can setup such a writecache device, but needs some additional information, like the storage pool and the size of the cache device. ------ # linstor storage-pool create lvm node1 lvmpool drbdpool # linstor storage-pool create lvm node1 pmempool pmempool # linstor resource-definition create r1 # linstor volume-definition create r1 100G # linstor volume-definition set-property r1 0 Writecache/PoolName pmempool # linstor volume-definition set-property r1 0 Writecache/Size 1% # linstor resource create node1 r1 --storage-pool lvmpool --layer-list WRITECACHE,STORAGE ------ The two properties set in the examples are mandatory, but can also be set on controller level which would act as a default for all resources with `WRITECACHE` in their `--layer-list`. However, please note that the `Writecache/PoolName` refers to the corresponding node. If the node does not have a storage-pool named `pmempool` you will get an error message. The 4 mandatory parameters required by https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/writecache.html[DM-Writecache] are either configured via property or figured out by LINSTOR. The optional properties listed in the mentioned link can also be set via property. Please see `linstor controller set-property --help` for a list of `Writecache/*` property-keys. Using `--layer-list DRBD,WRITECACHE,STORAGE` while having DRBD configured to use external metadata, only the backing device will use a writecache, not the device holding the external metadata. [[s-cache-layer]] ==== Cache Layer LINSTOR can also setup a https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/cache.html[DM-Cache] device, which is very similar to the DM-Writecache from the previous section. The major difference is that a cache device is composed by three devices: one storage device, one cache device and one meta device. The LINSTOR properties are quite similar to those of the writecache but are located in the `Cache` namespace: ------ # linstor storage-pool create lvm node1 lvmpool drbdpool # linstor storage-pool create lvm node1 pmempool pmempool # linstor resource-definition create r1 # linstor volume-definition create r1 100G # linstor volume-definition set-property r1 0 Cache/CachePool pmempool # linstor volume-definition set-property r1 0 Cache/Cachesize 1% # linstor resource create node1 r1 --storage-pool lvmpool --layer-list CACHE,STORAGE ------ NOTE: Instead of `Writecache/PoolName` (as when configuring the Writecache layer) the Cache layer's only required property is called `Cache/CachePool`. The reason for this is that the Cache layer also has a `Cache/MetaPool` which can be configured separately or it defaults to the value of `Cache/CachePool`. Please see `linstor controller set-property --help` for a list of `Cache/*` property-keys and default values for omitted properties. Using `--layer-list DRBD,CACHE,STORAGE` while having DRBD configured to use external metadata, only the backing device will use a cache, not the device holding the external metadata. [[s-storage-layer]] ==== Storage Layer The storage layer will provide new devices from well known volume managers like LVM, ZFS or others. Every layer combination needs to be based on a storage layer, even if the resource should be diskless - for that type there is a dedicated `diskless` provider type. For a list of providers with their properties please see <>. For some storage providers LINSTOR has special properties: * `StorDriver/WaitTimeoutAfterCreate`: If LINSTOR expects a device to appear after creation (for example after calls of `lvcreate`, `zfs create`,...), LINSTOR waits per default 500ms for the device to appear. These 500ms can be overridden by this property. * `StorDriver/dm_stats`: If set to `true` LINSTOR calls `dmstats create $device` after creation and `dmstats delete $device --allregions` after deletion of a volume. Currently only enabled for LVM and LVM_THIN storage providers. [[s-storage-providers]] === Storage Providers LINSTOR has a few storage providers. The most used ones are LVM and ZFS. But also for those two providers there are already sub-types for their thinly provisioned variants. * Diskless: This provider type is mostly required to have a storage pool that can be configured with LINSTOR properties like `PrefNic` as described in <>. * LVM / LVM-Thin: The adminstrator is expected to specify the LVM volume group or the thin-pool (in form of "LV/thinpool") in order to use the corresponding storage type. These drivers support following properties for fine-tuning: ** `StorDriver/LvcreateOptions`: The value of this property is appended to every `lvcreate ...` call LINSTOR executes. * ZFS / ZFS-Thin: The administrator is expected to specify the ZPool that LINSTOR should use. These drivers support following properties for fine-tuning: ** `StorDriver/ZfscreateOptions`: The value of this property is appended to every `zfs create ...` call LINSTOR executes. * File / FileThin: Mostly used for demonstration / experiments. LINSTOR will basically reserve a file in a given directory and will configure a https://man7.org/linux/man-pages/man4/loop.4.html[loop device] on top of that file. * OpenFlex: This special storage provider currently requires to be run on a "special satellite". Please see <> for more details. * EXOS: This special storage provider currently requires to be run on a "special satellite". Please see the <> chapter * SPDK: The administrator is expected to speicify the logical volume store which LINSTOR should use. The usage of this storage provider implies the usage of the <>. ** Remote-SPDK: This special storage provider currently requires to be run on a "special satellite". Please see <> for more details. [[s-remote-spdk]] ==== Remote SPDK Provider A storage pool with the type remote SPDK can only be created on a "special satellite". For this you first need to start a new satellite using the command: ---- $ linstor node create-remote-spdk-target nodeName 192.168.1.110 ---- This will start a new satellite instance running on the same machine as the controller. This special satellite will do all the REST based RPC communication towards the remote SPDK proxy. As the help message of the LINSTOR command shows, the administrator might want to use additional settings when creating this special satellite: ---- $ linstor node create-remote-spdk-target -h usage: linstor node create-remote-spdk-target [-h] [--api-port API_PORT] [--api-user API_USER] [--api-user-env API_USER_ENV] [--api-pw [API_PW]] [--api-pw-env API_PW_ENV] node_name api_host ---- The difference between the `--api-\*` and their corresponding `--api-\*-env` versions is that the version with the `-env` ending will look for an environment variable containing the actual value to use whereas the `--api-\*` version directly take the value which is stored in the LINSTOR property. Administrators might not want to save the `--api-pw` in plaintext, which would be clearly visible using commands like `linstor node list-property `. Once that special satellite is up and running the actual storage pool can be created: ---- $ linstor storage-pool create remotespdk -h usage: linstor storage-pool create remotespdk [-h] [--shared-space SHARED_SPACE] [--external-locking] node_name name driver_pool_name ---- Whereas `node_name` is self-explanatory, `name` is the name of the LINSTOR storage pool and `driver_pool_name` refers to the SPDK logical volume store. Once this _remotespdk_ storage pool is created the remaining procedure is quite similar as using NVMe: First the target has to be created by creating a simple "diskful" resource followed by a second resource having the `--nvme-initiator` option enabled. [[s-managing_network_interface_cards]] === Managing Network Interface Cards LINSTOR can deal with multiple network interface cards (NICs) in a machine, they are called `netif` in LINSTOR speak. NOTE: When a satellite node is created a first `netif` gets created implicitly with the name `default`. Using the `--interface-name` option of the `node create` command you can give it a different name. Additional NICs are created like this: ---------------------------- # linstor node interface create alpha 100G_nic 192.168.43.221 # linstor node interface create alpha 10G_nic 192.168.43.231 ---------------------------- NICs are identified by the IP address only, the name is arbitrary and is *not* related to the interface name used by Linux. The NICs can be assigned to storage pools so that whenever a resource is created in such a storage pool, the DRBD traffic will be routed through the specified NIC. ---------------------------- # linstor storage-pool set-property alpha pool_hdd PrefNic 10G_nic # linstor storage-pool set-property alpha pool_ssd PrefNic 100G_nic ---------------------------- FIXME describe how to route the controller +<->+ client communication through a specific `netif`. [[s-linstor-encrypted-volumes]] === Encrypted volumes LINSTOR can handle transparent encryption of drbd volumes. dm-crypt is used to encrypt the provided storage from the storage device. NOTE: In order to use dm-crypt please make sure to have `cryptsetup` installed before you start the satellite Basic steps to use encryption: 1. Disable user security on the controller (this will be obsolete once authentication works) 2. Create a master passphrase 3. Add `luks` to the layer-list. Note that all plugins (e.g., Proxmox) require a DRBD layer as the top most layer if they do not explicitly state otherwise. 4. Don't forget to re-enter the master passphrase after a controller restart. [[s-disable_user_security]] ==== Disable user security Disabling the user security on the `Linstor` controller is a one time operation and is afterwards persisted. 1. Stop the running linstor-controller via systemd: `systemctl stop linstor-controller` 2. Start a linstor-controller in debug mode: `/usr/share/linstor-server/bin/Controller -c /etc/linstor -d` 3. In the debug console enter: `setSecLvl secLvl(NO_SECURITY)` 4. Stop linstor-controller with the debug shutdown command: `shutdown` 5. Start the controller again with systemd: `systemctl start linstor-controller` [[s-encrypt_commands]] ==== Encrypt commands Below are details about the commands. Before LINSTOR can encrypt any volume a master passphrase needs to be created. This can be done with the linstor-client. ---- # linstor encryption create-passphrase ---- `crypt-create-passphrase` will wait for the user to input the initial master passphrase (as all other crypt commands will with no arguments). If you ever want to change the master passphrase this can be done with: ---- # linstor encryption modify-passphrase ---- The `luks` layer can be added when creating the resource-definition or the resource itself, whereas the former method is recommended since it will be automatically applied to all resource created from that resource-definition. ---- # linstor resource-definition create crypt_rsc --layer-list luks,storage ---- To enter the master passphrase (after controller restart) use the following command: ---- # linstor encryption enter-passphrase ---- NOTE: Whenever the linstor-controller is restarted, the user has to send the master passphrase to the controller, otherwise LINSTOR is unable to reopen or create encrypted volumes. [[s-automatic_passphrase]] ==== Automatic Passphrase It is possible to automate the process of creating and re-entering the master passphrase. To use this, either an environment variable called `MASTER_PASSPHRASE` or an entry in `/etc/linstor/linstor.toml` containing the master passphrase has to be created. The required `linstor.toml` looks like this: ---- [encrypt] passphrase="example" ---- If either one of these is set, then every time the controller starts it will check whether a master passphrase already exists. If there is none, it will create a new master passphrase as specified. Otherwise, the controller enters the passphrase. WARNING: If a master passphrase is already configured, and it is not the same one as specified in the environment variable or `linstor.toml`, the controller will be unable to re-enter the master passphrase and react as if the user had entered a wrong passphrase. This can only be resolved through manual input from the user, using the same commands as if the controller was started without the automatic passphrase. NOTE: In case the master passphrase is set in both an environment variable and the `linstor.toml`, only the master passphrase from the `linstor.toml` will be used. [[s-linstor-status]] === Checking the state of your cluster LINSTOR provides various commands to check the state of your cluster. These commands start with a 'list-' prefix and provide various filtering and sorting options. The '--groupby' option can be used to group and sort the output in multiple dimensions. ---------------------------- # linstor node list # linstor storage-pool list --groupby Size ---------------------------- [[s-linstor-snapshots]] === Managing snapshots Snapshots are supported with thin LVM and ZFS storage pools. [[s-creating_a_snapshot-linstor]] ==== Creating a snapshot Assuming a resource definition named 'resource1' which has been placed on some nodes, a snapshot can be created as follows: ---------------------------- # linstor snapshot create resource1 snap1 ---------------------------- This will create snapshots on all nodes where the resource is present. LINSTOR will ensure that consistent snapshots are taken even when the resource is in active use. Setting the resource-definition property `AutoSnapshot/RunEvery` LINSTOR will automatically create snapshots every X minute. The optional property `AutoSnapshot/Keep` can be used to clean-up old snapshots which were created automatically. No manually created snapshot will be cleaned-up / deleted. If `AutoSnapshot/Keep` is omitted (or <= 0), LINSTOR will keep the last 10 snapshots by default. ---------------------------- # linstor resource-definition set-property AutoSnapshot/RunEvery 15 # linstor resource-definition set-property AutoSnapshot/Keep 5 ---------------------------- [[s-restoring_a_snapshot-linstor]] ==== Restoring a snapshot The following steps restore a snapshot to a new resource. This is possible even when the original resource has been removed from the nodes where the snapshots were taken. First define the new resource with volumes matching those from the snapshot: ---------------------------- # linstor resource-definition create resource2 # linstor snapshot volume-definition restore --from-resource resource1 --from-snapshot snap1 --to-resource resource2 ---------------------------- At this point, additional configuration can be applied if necessary. Then, when ready, create resources based on the snapshots: ---------------------------- # linstor snapshot resource restore --from-resource resource1 --from-snapshot snap1 --to-resource resource2 ---------------------------- This will place the new resource on all nodes where the snapshot is present. The nodes on which to place the resource can also be selected explicitly; see the help (`linstor snapshot resource restore -h`). [[s-rolling_back_snapshot-linstor]] ==== Rolling back to a snapshot LINSTOR can roll a resource back to a snapshot state. The resource must not be in use. That is, it may not be mounted on any nodes. If the resource is in use, consider whether you can achieve your goal by <> instead. Rollback is performed as follows: ---------------------------- # linstor snapshot rollback resource1 snap1 ---------------------------- A resource can only be rolled back to the most recent snapshot. To roll back to an older snapshot, first delete the intermediate snapshots. [[s-removing_a_snapshot-linstor]] ==== Removing a snapshot An existing snapshot can be removed as follows: ---------------------------- # linstor snapshot delete resource1 snap1 ---------------------------- [[s-shipping_snapshots-linstor]] ==== Shipping a snapshot Snapshots can be shipped between LINSTOR nodes or between different LINSTOR clusters, as well as to an S3 storage such as https://aws.amazon.com/s3/[Amazon S3] or https://min.io/[min.io]. [[s-shipping_snapshots-remotes]] ===== Remotes In a LINSTOR cluster, the definition of a shipping target is called a remote. Currently, there are two different types of remotes: LINSTOR remotes and S3 remotes. LINSTOR remotes are used to ship snapshots to a different LINSTOR cluster, while S3 remotes are needed to ship snapshots to AWS S3, min.io or any other service using S3 compatible object storage. IMPORTANT: Since a remote needs to store sensitive data, such as passwords, it is neccessary to have encryption enabled whenever you wish to use a remote in any way. How to set up LINSTOR's encryption is described <>. To create an S3 remote, LINSTOR will need to know the endpoint (that is, the URL of the target S3 server), the name of the target bucket, the region the S3 server is in, as well as the access-key and secret-key used to access the bucket. If the command is sent without adding the secret-key, a prompt will pop up to enter it in. The command should look like this: -------------------------- # linstor remote create s3 myRemote s3.us-west-2.amazonaws.com my-bucket us-west-2 admin password -------------------------- TIP: Usually, LINSTOR uses the endpoint and bucket to create an url using the virtual-hosted-style for its access to the given bucket (e.g. my-bucket.s3.us-west-2.amazonaws.com). Should your setup not allow access this way, change the remote to path-style access (e.g. s3.us-west-2.amazonaws.com/my-bucket) by adding the `--use-path-style` argument to make LINSTOR combine the parameters accordingly. To create a LINSTOR remote, only the URL or IP address of the controller of the target cluster is needed. The command goes as follows: ---------------------------- # linstor remote create linstor myRemote 192.168.0.15 ---------------------------- Additionally, to ship LUKS-based (encrypted) backups, it is necessary to add the `--passphrase` and `--cluster-id` arguments to the command. This is used to save the passphrase and cluster-id of the target cluster to the remote respectively. For more details on shipping LUKS-based backups between two LINSTOR clusters, see <>. To see all the remotes known to the local cluster, use `linstor remote list`. To delete a remote, use `linstor remote delete myRemoteName`. Should an existing remote need altering, use `linstor remote modify` to change it. [[s-shipping_snapshots-s3]] ===== Shipping snapshots to S3 All that is needed to ship a snapshot to S3 is to create an S3-remote that the current cluster can reach as well as the resource that should be shipped. Then, simply use the following command to ship it there: -------------------------- # linstor backup create myRemote myRsc -------------------------- This command will create a snapshot of your resource and ship it to the given remote. If this isn't the first time you shipped a backup of this resource (to that remote) and the snapshot of the previous backup hasn't been deleted yet, an incremental backup will be shipped. To force the creation of a full backup, add the `--full` argument to the command. Getting a specific node to ship the backup is also possible by using `--node myNode`, but if the specified node is not available or only has the resource diskless, a different node will be chosen. To see which backups exist in a specific remote, use `linstor backup list myRemote`. A resource-name can be added to the command as a filter to only show backups of that specific resource by using the argument `--resource myRsc`. If you use the `--other` argument, only entries in the bucket that LINSTOR does not recognize as a backup will be shown. LINSTOR always names backups in a certain way, and as long as an item in the remote is named according to this schema, it is assumed that it is a backup created by LINSTOR - so this list will show everything else. There are several options when it comes to deleting backups: * `linstor backup delete all myRemote`: This command deletes ALL S3-objects on the given remote, as long as they are recognized to be backups, that is, fit the expected naming schema. There is the option `--cluster` to only delete backups that were created by the current cluster. * `linstor backup delete id myRemote my-rsc_back_20210824_072543`: This command deletes a single backup from the given remote - namely the one with the given id, which consists of the resource-name, the automatically generated snapshot-name (back_timestamp) and, if set, the backup-suffix. The option `--prefix` lets you delete all backups starting with the given id. The option `--cascade` deletes not only the specified backup, but all other incremental backups depending on it. * `linstor backup delete filter myRemote ...`: This command has a few different arguments to specify a selection of backups to delete. `-t 20210914_120000` will delete all backups made before 12 o'clock on the 14th of September, 2021. `-n myNode` will delete all backups uploaded by the given node. `-r myRsc` will delete all backups with the given resource name. These filters can be combined as needed. Finally, `--cascade` deletes not only the selected backup(s), but all other incremental backups depending on any of the selected backups. * `linstor backup delete s3key myRemote randomPictureInWrongBucket`: This command will find the object with the given S3-key and delete it - without considering anything else. This should only be used to either delete non-backup items from the remote, or to clean up a broken backup that is no longer deleteable by other means. Using this command to delete a regular, working backup will break that backup, so beware! WARNING: All commands that have the `--cascade` option will NOT delete a backup that has incremental backups depending on it unless you explicitly add that option. TIP: All `linstor backup delete ...` commands have the `--dry-run` option, which will give you a list of all the S3-objects that will be deleted. This can be used to ensure nothing that should not be deleted is accidentally deleted. Maybe the most important task after creating a backup is restoring it. To do so, only the remote is needed - but it is also possible to restore into an existing resource definition with no existing snapshots nor resources. There are two options for the command: ----------------------- # linstor backup restore myRemote myNode targetRsc --resource sourceRsc # linstor backup restore myRemote myNode targetRsc --id sourceRsc_back_20210824_072543 ----------------------- Either `--resource (-r)` or `--id` must be used, but you cannot use both of them together. `-r` is used to restore the latest backup of the resource specified with this option, while `--id` restores the exact backup specified by the given id, and can therefore be used to restore backups other than the most recent. If the backup to be restored includes a LUKS layer, the `--passphrase` argument is required. With it, the passphrase of the original cluster of the backup needs to be set so that LINSTOR can decrypt the volumes after download and re-encrypt them with the local passphrase. The backup restore will download all the snapshots from the last full backup up to the specified backup. Afterwards, it restores the snapshots into a new resource. If that last step should be skipped, the `--download-only` option needs to be added to the command. Backups can be downloaded from any cluster, not just the one that uploaded them, as long as the setup is correct. Specifically, the target resource cannot have any existing resources or snapshots, and the storage pool(s) used need to have the same storage providers. If the storage pool(s) on the target node have the exact same names as on the cluster the backup was created on, no extra action is necessary. Should they have different names, the option `--storpool-rename` needs to be used. It expects at least one `oldname=newname` pair. For every storage pool of the original backup that is not named in that list, it will be assumed that its name is exactly the same on the target node. To find out exactly which storage pools need to be renamed, as well as how big the download and the restored resource will be, the command `linstor backup info myRemote ...` can be used. Similar to the restore command, either `-r` or `--id` need to be given, which add the same restrictions as with that command. To see how much space will be left over in the local storage pools after a restore, the argument `-n myNode` needs to be added. Just like with a restore, it assumes the storage pool names are exactly the same on the given node as with the backup. Should that not be the case, again, just like with the restore command, `--storpool-rename` should be used. [[s-shipping_snapshots-l2l]] ===== Shipping snapshots whithin LINSTOR Shipping a snapshot directly between two LINSTOR clusters can be done with a linstor-remote as well as a resource definition with at least one diskful resource on the source side (where the command is issued). On the target side, you only need a linstor-remote with the cluster-id of the source-cluster set. If you wish to ship a snapshot inside the same cluster just use a remote pointing to the local controller. The command looks like this: -------------------------- # linstor backup ship myRemote localRsc targetRsc -------------------------- Additionally, you can use `--source-node` and `--target-node` to specify which node should send and receive the backup respectively. In case those nodes are not available, a different one will be chosen automatically. If the target resource already has resources deployed or the `--download-only` option was specified, the snapshots will only be shipped to the target cluster, but not restored. When the snapshot you wish to ship contains a LUKS layer, the remote on the target cluster also needs the passphrase of the source cluster set. [[s-shipping_snapshots-old]] ===== Shipping a snapshot in the same cluster WARNING: The `snapshot ship` command is considered deprecated and any bugs found with it will not be fixed. Instead, use the `backup ship` command with a remote pointing to your local controller. For more details, see <>. Both, the source as well as the target node have to have the resource for snapshot shipping deployed. Additionally, the target resource has to be deactivated. ---------------------------- # linstor resource deactivate nodeTarget resource1 ---------------------------- WARNING: Deactivating a resource with DRBD in its layer-list can NOT be reactivated again. However, a successfully shipped snapshot of a DRBD resource can still be <>. To manually start the snapshot-shipping, use: ---------------------------- # linstor snapshot ship --from-node nodeSource --to-node nodeTarget --resource resource1 ---------------------------- By default, the snapshot-shipping uses tcp ports from the range 12000-12999. To change this range, the property `SnapshotShipping/TcpPortRange`, which accepts a to-from range, can be set on the controller: ---------------------------- # linstor controller set-property SnapshotShipping/TcpPortRange 10000-12000 ---------------------------- A resource can also be periodically shipped. To accomplish this, it is mandatory to set the properties `SnapshotShipping/TargetNode` as well as `SnapshotShipping/RunEvery` on the resource-definition. `SnapshotShipping/SourceNode` can also be set, but if omitted LINSTOR will choose an active resource of the same resource-definition. To allow incremental snapshot-shipping, LINSTOR has to keep at least the last shipped snapshot on the target node. The property `SnapshotShipping/Keep` can be used to specify how many snapshots LINSTOR should keep. If the property is not set (or <= 0) LINSTOR will keep the last 10 shipped snapshots by default. ---------------------------- # linstor resource-definition set-property resource1 SnapshotShipping/TargetNode nodeTarget # linstor resource-definition set-property resource1 SnapshotShipping/SourceNode nodeSource # linstor resource-definition set-property resource1 SnapshotShipping/RunEvery 15 # linstor resource-definition set-property resource1 SnapshotShipping/Keep 5 ---------------------------- [[s-linstor-setupopts]] === Setting options for resources DRBD options are set using LINSTOR commands. Configuration in files such as `/etc/drbd.d/global_common.conf` that are not managed by LINSTOR will be ignored. The following commands show the usage and available options: ---------------------------- # linstor controller drbd-options -h # linstor resource-definition drbd-options -h # linstor volume-definition drbd-options -h # linstor resource drbd-peer-options -h ---------------------------- For instance, it is easy to set the DRBD protocol for a resource named `backups`: ---------------------------- # linstor resource-definition drbd-options --protocol C backups ---------------------------- [[s-linstor-toggle-disk]] === Adding and removing disks LINSTOR can convert resources between diskless and having a disk. This is achieved with the `resource toggle-disk` command, which has syntax similar to `resource create`. For instance, add a disk to the diskless resource `backups` on 'alpha': ---------------------------- # linstor resource toggle-disk alpha backups --storage-pool pool_ssd ---------------------------- Remove this disk again: ---------------------------- # linstor resource toggle-disk alpha backups --diskless ---------------------------- [[s-linstor-migrate-disk]] ==== Migrating disks In order to move a resource between nodes without reducing redundancy at any point, LINSTOR's disk migrate feature can be used. First create a diskless resource on the target node, and then add a disk using the `--migrate-from` option. This will wait until the data has been synced to the new disk and then remove the source disk. For example, to migrate a resource `backups` from 'alpha' to 'bravo': ---------------------------- # linstor resource create bravo backups --drbd-diskless # linstor resource toggle-disk bravo backups --storage-pool pool_ssd --migrate-from alpha ---------------------------- [[s-linstor-proxy]] === DRBD Proxy with LINSTOR LINSTOR expects DRBD Proxy to be running on the nodes which are involved in the relevant connections. It does not currently support connections via DRBD Proxy on a separate node. Suppose our cluster consists of nodes 'alpha' and 'bravo' in a local network and 'charlie' at a remote site, with a resource definition named `backups` deployed to each of the nodes. Then DRBD Proxy can be enabled for the connections to 'charlie' as follows: ---------------------------- # linstor drbd-proxy enable alpha charlie backups # linstor drbd-proxy enable bravo charlie backups ---------------------------- The DRBD Proxy configuration can be tailored with commands such as: ---------------------------- # linstor drbd-proxy options backups --memlimit 100000000 # linstor drbd-proxy compression zlib backups --level 9 ---------------------------- LINSTOR does not automatically optimize the DRBD configuration for long-distance replication, so you will probably want to set some configuration options such as the protocol: ---------------------------- # linstor resource-connection drbd-options alpha charlie backups --protocol A # linstor resource-connection drbd-options bravo charlie backups --protocol A ---------------------------- Please contact LINBIT for assistance optimizing your configuration. ==== Automatically enable DRBD Proxy LINSTOR can also be configured to automatically enable the above mentioned Proxy connection between two nodes. For this automation, LINSTOR first needs to know on which site each node is. ---------------------------- # linstor node set-property alpha Site A # linstor node set-property bravo Site A # linstor node set-property charlie Site B ---------------------------- As the `Site` property might also be used for other site-based decisions in future features, the `DrbdProxy/AutoEnable` also has to be set to `true`: ---------------------------- # linstor controller set-property DrbdProxy/AutoEnable true ---------------------------- This property can also be set on node, resource-definition, resource and resource-connection level (from left to right in increasing priority, whereas the controller is the left-most, i.e. least prioritized level) Once this initialization steps are completed, every newly created resource will automatically check if it has to enable DRBD proxy to any of its peer-resources. [[s-linstor-external-database]] === External database It is possible to have LINSTOR working with an external database provider like PostgreSQL, MariaDB and since version 1.1.0 even ETCD key value store is supported. To use an external database there are a few additional steps to configure. You have to create a DB/Schema and user to use for linstor, and configure this in the `/etc/linstor/linstor.toml`. [[s-postgresql]] ==== PostgreSQL A sample PostgreSQL `linstor.toml` looks like this: ------------------------------------------------------ [db] user = "linstor" password = "linstor" connection_url = "jdbc:postgresql://localhost/linstor" ------------------------------------------------------ [[s-mariadb_mysql]] ==== MariaDB/MySQL A sample MariaDB `linstor.toml` looks like this: ------------------------------------------------------ [db] user = "linstor" password = "linstor" connection_url = "jdbc:mariadb://localhost/LINSTOR?createDatabaseIfNotExist=true" ------------------------------------------------------ NOTE: The LINSTOR schema/database is created as `LINSTOR` so make sure the MariaDB connection string refers to the `LINSTOR` schema, as in the example above. [[s-etcd]] ==== ETCD ETCD is a distributed key-value store that makes it easy to keep your LINSTOR database distributed in a HA-setup. The ETCD driver is already included in the LINSTOR-controller package and only needs to be configured in the `linstor.toml`. More information on how to install and configure ETCD can be found here: https://etcd.io/docs[ETCD docs] And here is a sample [db] section from the `linstor.toml`: ------------------------------------------------------ [db] ## only set user/password if you want to use authentication, only since LINSTOR 1.2.1 # user = "linstor" # password = "linstor" ## for etcd ## do not set user field if no authentication required connection_url = "etcd://etcdhost1:2379,etcdhost2:2379,etcdhost3:2379" ## if you want to use TLS, only since LINSTOR 1.2.1 # ca_certificate = "ca.pem" # client_certificate = "client.pem" ## if you want to use client TLS authentication too, only since LINSTOR 1.2.1 # client_key_pkcs8_pem = "client-key.pkcs8" ## set client_key_password if private key has a password # client_key_password = "mysecret" ------------------------------------------------------ [[s-linstor-rest-api]] === LINSTOR REST-API To make LINSTOR's administrative tasks more accessible and also available for web-frontends a REST-API has been created. The REST-API is embedded in the linstor-controller and since LINSTOR 0.9.13 configured via the `linstor.toml` configuration file. --------- [http] enabled = true port = 3370 listen_addr = "127.0.0.1" # to disable remote access --------- If you want to use the REST-API the current documentation can be found on the following link: https://app.swaggerhub.com/apis-docs/Linstor/Linstor/ [[s-linstor-rest-api-https]] ==== LINSTOR REST-API HTTPS The HTTP REST-API can also run secured by HTTPS and is highly recommended if you use any features that require authorization. Todo so you have to create a java keystore file with a valid certificate that will be used to encrypt all HTTPS traffic. Here is a simple example on how you can create a self signed certificate with the `keytool` that is included in the java runtime: --------- keytool -keyalg rsa -keysize 2048 -genkey -keystore ./keystore_linstor.jks\ -alias linstor_controller\ -dname "CN=localhost, OU=SecureUnit, O=ExampleOrg, L=Vienna, ST=Austria, C=AT" --------- `keytool` will ask for a password to secure the generated keystore file and is needed for the LINSTOR-controller configuration. In your `linstor.toml` file you have to add the following section: --------- [https] keystore = "/path/to/keystore_linstor.jks" keystore_password = "linstor" --------- Now (re)start the `linstor-controller` and the HTTPS REST-API should be available on port 3371. More information on how to import other certificates can be found here: https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html NOTE: When HTTPS is enabled, all requests to the HTTP /v1/ REST-API will be redirected to the HTTPS redirect. [[s-rest-api-https-restricted-client]] ===== LINSTOR REST-API HTTPS restricted client access Client access can be restricted by using a SSL truststore on the Controller. Basically you create a certificate for your client and add it to your truststore and the client then uses this certificate for authentication. First create a client certificate: --------- keytool -keyalg rsa -keysize 2048 -genkey -keystore client.jks\ -storepass linstor -keypass linstor\ -alias client1\ -dname "CN=Client Cert, OU=client, O=Example, L=Vienna, ST=Austria, C=AT" --------- Then we import this certificate to our controller truststore: --------- keytool -importkeystore\ -srcstorepass linstor -deststorepass linstor -keypass linstor\ -srckeystore client.jks -destkeystore trustore_client.jks --------- And enable the truststore in the `linstor.toml` configuration file: --------- [https] keystore = "/path/to/keystore_linstor.jks" keystore_password = "linstor" truststore = "/path/to/trustore_client.jks" truststore_password = "linstor" --------- Now restart the Controller and it will no longer be possible to access the controller API without a correct certificate. The LINSTOR client needs the certificate in PEM format, so before we can use it we have to convert the java keystore certificate to the PEM format. --------- # Convert to pkcs12 keytool -importkeystore -srckeystore client.jks -destkeystore client.p12\ -storepass linstor -keypass linstor\ -srcalias client1 -srcstoretype jks -deststoretype pkcs12 # use openssl to convert to PEM openssl pkcs12 -in client.p12 -out client_with_pass.pem --------- To avoid entering the PEM file password all the time it might be convenient to remove the password. --------- openssl rsa -in client_with_pass.pem -out client1.pem openssl x509 -in client_with_pass.pem >> client1.pem --------- Now this PEM file can easily be used in the client: --------- linstor --certfile client1.pem node list --------- The `--certfile` parameter can also added to the client configuration file, see <> for more details. [[s-linstor-logging]] === Logging Linstor uses https://www.slf4j.org/[SLF4J] with https://logback.qos.ch/[Logback] as binding. This gives Linstor the possibility to distinguish between the log levels `ERROR`, `WARN`, `INFO`, `DEBUG` and `TRACE` (in order of increasing verbosity). In the current linstor version (1.1.2) the user has the following four methods to control the logging level, ordered by priority (first has highest priority): . `TRACE` mode can be `enabled` or `disabled` using the debug console: + ---- Command ==> SetTrcMode MODE(enabled) SetTrcMode Set TRACE level logging mode New TRACE level logging mode: ENABLED ---- . When starting the controller or satellite a command line argument can be passed: + ---- java ... com.linbit.linstor.core.Controller ... --log-level INFO java ... com.linbit.linstor.core.Satellite ... --log-level INFO ---- . The recommended place is the `logging` section in the configuration file. The default configuration file location is `/etc/linstor/linstor.toml` for the controller and `/etc/linstor/linstor_satellite.toml` for the satellite. Configure the logging level as follows: + ---- [logging] level="INFO" ---- . As Linstor is using Logback as an implementation, `/usr/share/linstor-server/lib/logback.xml` can also be used. Currently only this approach supports different log levels for different components, like shown in the example below: + ---- %d{HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n ${log.directory}/linstor-${log.module}.log true %d{yyyy_MM_dd HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n logs/linstor-${log.module}.%i.log.zip 1 10 2MB ---- See the https://logback.qos.ch/manual/index.html[Logback Manual] to find more details about `logback.xml`. When none of the configuration methods above is used Linstor will default to `INFO` log level. [[s-linstor-monitoring]] === Monitoring Since LINSTOR 1.8.0, a https://prometheus.io/[Prometheus] `/metrics` HTTP path is provided with LINSTOR and JVM specific exports. The `/metrics` path also supports 3 GET arguments to reduce LINSTOR's reported data: * resource * storage_pools * error_reports These are all default `true`, to disabled e.g. error-report data: `http://localhost:3370/metrics?error_reports=false` ==== Health check The LINSTOR-Controller also provides a `/health` HTTP path that will simply return HTTP-Status 200 if the controller can access its database and all services are up and running. Otherwise it will return HTTP error status code 500 `Internal Server Error`. [[s-linstor-secure-connections]] === Secure Satellite connections It is possible to have the LINSTOR use SSL secure TCP connection between controller and satellite connections. Without going into further details on how java's SSL engine works we will give you command line snippets using the `keytool` from java's runtime environment on how to configure a 3 node setup using secure connections. The node setup looks like this: Node `alpha` is the just the controller. Node `bravo` and node `charlie` are just satellites. Here are the commands to generate such a keystore setup, values should of course be edited for your environment. --------- # create directories to hold the key files mkdir -p /tmp/linstor-ssl cd /tmp/linstor-ssl mkdir alpha bravo charlie # create private keys for all nodes keytool -keyalg rsa -keysize 2048 -genkey -keystore alpha/keystore.jks\ -storepass linstor -keypass linstor\ -alias alpha\ -dname "CN=Max Mustermann, OU=alpha, O=Example, L=Vienna, ST=Austria, C=AT" keytool -keyalg rsa -keysize 2048 -genkey -keystore bravo/keystore.jks\ -storepass linstor -keypass linstor\ -alias bravo\ -dname "CN=Max Mustermann, OU=bravo, O=Example, L=Vienna, ST=Austria, C=AT" keytool -keyalg rsa -keysize 2048 -genkey -keystore charlie/keystore.jks\ -storepass linstor -keypass linstor\ -alias charlie\ -dname "CN=Max Mustermann, OU=charlie, O=Example, L=Vienna, ST=Austria, C=AT" # import truststore certificates for alpha (needs all satellite certificates) keytool -importkeystore\ -srcstorepass linstor -deststorepass linstor -keypass linstor\ -srckeystore bravo/keystore.jks -destkeystore alpha/certificates.jks keytool -importkeystore\ -srcstorepass linstor -deststorepass linstor -keypass linstor\ -srckeystore charlie/keystore.jks -destkeystore alpha/certificates.jks # import controller certificate into satellite truststores keytool -importkeystore\ -srcstorepass linstor -deststorepass linstor -keypass linstor\ -srckeystore alpha/keystore.jks -destkeystore bravo/certificates.jks keytool -importkeystore\ -srcstorepass linstor -deststorepass linstor -keypass linstor\ -srckeystore alpha/keystore.jks -destkeystore charlie/certificates.jks # now copy the keystore files to their host destinations ssh root@alpha mkdir /etc/linstor/ssl scp alpha/* root@alpha:/etc/linstor/ssl/ ssh root@bravo mkdir /etc/linstor/ssl scp bravo/* root@bravo:/etc/linstor/ssl/ ssh root@charlie mkdir /etc/linstor/ssl scp charlie/* root@charlie:/etc/linstor/ssl/ # generate the satellite ssl config entry echo '[netcom] type="ssl" port=3367 server_certificate="ssl/keystore.jks" trusted_certificates="ssl/certificates.jks" key_password="linstor" keystore_password="linstor" truststore_password="linstor" ssl_protocol="TLSv1.2" ' | ssh root@bravo "cat > /etc/linstor/linstor_satellite.toml" echo '[netcom] type="ssl" port=3367 server_certificate="ssl/keystore.jks" trusted_certificates="ssl/certificates.jks" key_password="linstor" keystore_password="linstor" truststore_password="linstor" ssl_protocol="TLSv1.2" ' | ssh root@charlie "cat > /etc/linstor/linstor_satellite.toml" --------- Now just start controller and satellites and add the nodes with `--communication-type SSL`. [[s-linstor-drbd-automatisms]] === Automatisms for DRBD-Resources [[s-linstor-auto-quorum]] ==== AutoQuorum Policies LINSTOR automatically configures quorum policies on resources *when quorum is achievable*. This means, whenever you have at least two diskful and one or more diskless resource assignments, or three or more diskful resource assignments, LINSTOR will enable quorum policies for your resources automatically. Inversely, LINSTOR will automatically disable quorum policies whenever there are less than the minimum required resource assignments to achieve quorum. This is controlled via the, `DrbdOptions/auto-quorum`, property which can be applied to the _linstor-controller_, _resource-group_, and _resource-definition_. Accepted values for the `DrbdOptions/auto-quorum` property are `disabled`, `suspend-io`, and `io-error`. Setting the `DrbdOptions/auto-quorum` property to `disabled` will allow you to manually, or more granularly, control the quorum policies of your resources should you so desire. TIP: The default policies for `DrbdOptions/auto-quorum` are `quorum majority`, and `on-no-quorum io-error`. For more information on DRBD's quorum features and their behavior, please refer to the {url-drbd-ug}#s-feature-quorum[quorum section of the DRBD user's guide]. IMPORTANT: The `DrbdOptions/auto-quorum` policies will override any manually configured properties if `DrbdOptions/auto-quorum` is not disabled. For example, to manually set the quorum policies of a _resource-group_ named `my_ssd_group`, you would use the following commands: ---- # linstor resource-group set-property my_ssd_group DrbdOptions/auto-quorum disabled # linstor resource-group set-property my_ssd_group DrbdOptions/Resource/quorum majority # linstor resource-group set-property my_ssd_group DrbdOptions/Resource/on-no-quorum suspend-io ---- You may wish to disable DRBD's quorum features completely. To do that, you would need to first disable `DrbdOptions/auto-quorum` on the appropriate LINSTOR object, and then set the DRBD quorum features accordingly. For example, use the following commands to disable quorum entirely on the `my_ssd_group` _resource-group_: ---- # linstor resource-group set-property my_ssd_group DrbdOptions/auto-quorum disabled # linstor resource-group set-property my_ssd_group DrbdOptions/Resource/quorum off # linstor resource-group set-property my_ssd_group DrbdOptions/Resource/on-no-quorum ---- NOTE: Setting `DrbdOptions/Resource/on-no-quorum` to an empty value in the commands above deletes the property from the object entirely. [[s-linstor-auto-evict]] ==== Auto-Evict If a satellite is offline for a prolonged period of time, LINSTOR can be configured to declare that node as evicted. This triggers an automated reassignment of the affected DRBD-resources to other nodes to ensure a minimum replica count is kept. This feature uses the following properties to adapt the behaviour. * `DrbdOptions/AutoEvictMinReplicaCount` sets the number of replicas that should always be present. You can set this property on the controller to change a global default, or on a specific resource-definition or resource-group to change it only for that resource-definition or resource-group. If this property is left empty, the place-count set for the auto-placer of the corresponding resource-group will be used. * `DrbdOptions/AutoEvictAfterTime` describes how long a node can be offline in minutes before the eviction is triggered. You can set this property on the controller to change a global default, or on a single node to give it a different behavior. The default value for this property is 60 minutes. * `DrbdOptions/AutoEvictMaxDisconnectedNodes` sets the percentage of nodes that can be not reachable (for whatever reason) at the same time. If more than the given percent of nodes are offline at the same time, the auto-evict will not be triggered for any node , since in this case LINSTOR assumes connection problems from the controller. This property can only be set for the controller, and only accepts a value between 0 and 100. The default value is 34. If you wish to turn the auto-evict-feature off, simply set this property to 0. If you want to always trigger the auto-evict, regardless of how many satellites are unreachable, set it to 100. * `DrbdOptions/AutoEvictAllowEviction` is an additional property that can stop a node from being evicted. This can be useful for various cases, for example if you need to shut down a node for maintenance. You can set this property on the controller to change a global default, or on a single node to give it a different behavior. It accepts true and false as values and per default is set to true on the controller. You can use this property to turn the auto-evict feature off by setting it to false on the controller, although this might not work completely if you already set different values for individual nodes, since those values take precedence over the global default. After the linstor-controller loses the connection to a satellite, aside from trying to reconnect, it starts a timer for that satellite. As soon as that timer exceeds `DrbdOptions/AutoEvictAfterTime` and all of the DRBD-connections to the DRBD-resources on that satellite are broken, the controller will check whether or not `DrbdOptions/AutoEvictMaxDisconnectedNodes` has been met. If it hasn't, and `DrbdOptions/AutoEvictAllowEviction` is true for the node in question, the satellite will be marked as EVICTED. At the same time, the controller will check for every DRBD-resource whether the number of resources is still above `DrbdOptions/AutoEvictMinReplicaCount`. If it is, the resource in question will be marked as DELETED. If it isn't, an auto-place with the settings from the corresponding resource-group will be started. Should the auto-place fail, the controller will try again later when changes that might allow a different result, such as adding a new node, have happened. Resources where an auto-place is necessary will only be marked as DELETED if the corresponding auto-place was successful. The evicted satellite itself will not be able to reestablish connection with the controller. Even if the node is up and running, a manual reconnect will fail. It is also not possible to delete the satellite, even if it is working as it should be. The satellite can, however, be restored. This will remove the EVICTED-flag from the satellite and allow you to use it again. Previously configured network interfaces, storage pools, properties and similar entities as well as non-DRBD-related resources and resources that could not be autoplaced somewhere else will still be on the satellite. To restore a satellite, use ------ # linstor node restore [nodename] ------ Should you wish to instead throw everything that once was on that node, including the node itself, away, you need to use the `node lost` command instead. [[s-linstor-qos]] === QoS Settings ==== Sysfs LINSTOR is able to set the following Sysfs settings: [cols="3,2"] |=== | SysFs | Linstor property | `/sys/fs/cgroup/blkio/blkio.throttle.read_bps_device` | `sys/fs/blkio_throttle_read` | `/sys/fs/cgroup/blkio/blkio.throttle.write_bps_device` | `sys/fs/blkio_throttle_write` | `/sys/fs/cgroup/blkio/blkio.throttle.read_iops_device` | `sys/fs/blkio_throttle_read_iops` | `/sys/fs/cgroup/blkio/blkio.throttle.write_iops_device` | `sys/fs/blkio_throttle_write_iops` |=== If a LINSTOR volume is composed of multiple "stacked" volume (for example DRBD with external metadata will have 3 devices: backing (storage) device, metadata device and the resulting DRBD device), setting a `sys/fs/\*` property for a Volume, only the bottom-most local "data"-device will receive the corresponding `/sys/fs/cgroup/...` setting. That means, in case of the example above only the backing device will receive the setting. In case a resource-definition has an nvme-target as well as an nvme-initiator resource, both bottom-most devices of each node will receive the setting. In case of the target the bottom-most device will be the volume of LVM or ZFS, whereas in case of the initiator the bottom-most device will be the connected nvme-device, regardless which other layers are stacked on top of that. [[s-linstor-getting-help]] === Getting help ==== From the command line A quick way to list available commands on the command line is to type `linstor`. Further information on sub-commands (e.g., list-nodes) can be retrieved in two ways: ---------------------------- # linstor node list -h # linstor help node list ---------------------------- Using the 'help' sub-command is especially helpful when LINSTOR is executed in interactive mode (`linstor interactive`). One of the most helpful features of LINSTOR is its rich tab-completion, which can be used to complete basically every object LINSTOR knows about (e.g., node names, IP addresses, resource names, ...). In the following examples, we show some possible completions, and their results: ---------------------------- # linstor node create alpha 1 # completes the IP address if hostname can be resolved # linstor resource create b c # linstor assign-resource backups charlie ---------------------------- If tab-completion does not work out of the box, please try to source the appropriate file: ---------------------------- # source /etc/bash_completion.d/linstor # or # source /usr/share/bash_completion/completions/linstor ---------------------------- For zsh shell users linstor-client can generate a zsh compilation file, that has basic support for command and argument completion. ---------------------------- # linstor gen-zsh-completer > /usr/share/zsh/functions/Completion/Linux/_linstor ---------------------------- ==== SOS-Report If something goes wrong and you need help finding the cause of the issue, you can use ---------------------------- # linstor sos-report create ---------------------------- The command above will create a new sos-report in `/var/log/linstor/controller/` on the controller node. Alternatively you can use ---------------------------- # linstor sos-report download ---------------------------- which will create a new sos-report and additionally downloads that report to the local machine into your current working directory. This sos-report contains logs and useful debug-information from several sources (Linstor-logs, `dmesg`, versions of external tools used by Linstor, `ip a`, database dump and many more). These information are stored for each node in plaintext in the resulting `.tar.gz` file. ==== From the community For help from the community please subscribe to our mailing list located here: https://lists.linbit.com/listinfo/drbd-user ==== GitHub To file bug or feature request please check out our GitHub page https://github.com/linbit ==== Paid support and development Alternatively, if you wish to purchase remote installation services, 24/7 support, access to certified repositories, or feature development please contact us: +1-877-454-6248 (1-877-4LINBIT) , International: +43-1-8178292-0 | sales@linbit.com