[[ch-openstack-linstor]] == LINSTOR Volumes in Openstack indexterm:[Openstack]indexterm:[Cinder] This chapter describes using LINSTOR to provision persistent, replicated, and high-performance block storage for Openstack. [[s-openstack-linstor-overview]] === Openstack Overview Openstack consists of a wide range of individual services; The service responsible for provisioning and managing block storage is called https://docs.openstack.org/cinder/latest/[Cinder]. Other openstack services such as the compute instance service https://docs.openstack.org/nova/latest[Nova] can request volumes from Cinder. Cinder will then make a volume accessible to the requesting service. LINSTOR can integrate with Cinder using a *volume driver*. The volume driver translates calls to the Cinder API to LINSTOR commands. For example: requesting a volume from Cinder will create new resources in LINSTOR, Cinder Volume snapshots translate to snapshots in LINSTOR and so on. [[s-openstack-linstor-install]] === LINSTOR for Openstack Installation An initial installation and configuration of DRBD and LINSTOR must be completed prior to using the OpenStack driver. * A detailed installation guide can be found <>. * <> your cluster * Learn how to use the <> * <> to the LINSTOR cluster At this point you should be able to list your storage cluster nodes using the LINSTOR client: [source] ---- $ linstor node info ╭────────────────────────────────────────────────────────────────────────────╮ ┊ Node ┊ NodeType ┊ Addresses ┊ State ┊ ╞════════════════════════════════════════════════════════════════════════════╡ ┊ cinder-01.openstack.test ┊ COMBINED ┊ 10.43.224.21:3366 (PLAIN) ┊ Online ┊ ┊ cinder-02.openstack.test ┊ COMBINED ┊ 10.43.224.22:3366 (PLAIN) ┊ Online ┊ ┊ storage-01.openstack.test ┊ SATELLITE ┊ 10.43.224.11:3366 (PLAIN) ┊ Online ┊ ┊ storage-02.openstack.test ┊ SATELLITE ┊ 10.43.224.12:3366 (PLAIN) ┊ Online ┊ ┊ storage-03.openstack.test ┊ SATELLITE ┊ 10.43.224.13:3366 (PLAIN) ┊ Online ┊ ╰────────────────────────────────────────────────────────────────────────────╯ ---- You should <> per node. This guide assumes the storage pool is named `cinderpool`. LINSTOR should list the storage pool for each node, including the diskless storage pool created by default. [source] ---- $ linstor storage-pool list ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ ┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ ╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡ ┊ DfltDisklessStorPool ┊ cinder-01.openstack.test ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ ┊ DfltDisklessStorPool ┊ cinder-02.openstack.test ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ ┊ DfltDisklessStorPool ┊ storage-01.openstack.test ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ ┊ DfltDisklessStorPool ┊ storage-02.openstack.test ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ ┊ DfltDisklessStorPool ┊ storage-03.openstack.test ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ ┊ cinderpool ┊ storage-01.openstack.test ┊ LVM_THIN ┊ ssds/cinderpool ┊ 100 GiB ┊ 100 GiB ┊ True ┊ Ok ┊ ┊ cinderpool ┊ storage-02.openstack.test ┊ LVM_THIN ┊ ssds/cinderpool ┊ 100 GiB ┊ 100 GiB ┊ True ┊ Ok ┊ ┊ cinderpool ┊ storage-03.openstack.test ┊ LVM_THIN ┊ ssds/cinderpool ┊ 100 GiB ┊ 100 GiB ┊ True ┊ Ok ┊ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ---- [[s-update_the_linstor_driver]] ==== Upgrades of LINSTOR driver If this is a fresh installation, skip this section and continue with <>. ===== From 1.x to 2.x Driver version 2 dropped some static configuration options in favour of managing these options at runtime via <>. [frame="topbot",options="header"] |==== | Option in `cinder.conf` | Status | Replace with | `linstor_autoplace_count` | removed | Use the `linstor:redundancy` property on the volume type. Using a value of 0 for full cluster replication is not supported, use the advanced options of <> | `linstor_controller_diskless` | removed | No replacement needed, the driver will create a diskless resource on the cinder host when required | `linstor_default_blocksize` | removed | This setting had no effect. | `linstor_default_storage_pool_name` | deprecated | This setting is deprecated for removal in a future version. Use the `linstor:storage_pool` property on the volume type instead. | `linstor_default_uri` | deprecated | Replaced by the more aptly named `linstor_uris`. | `linstor_default_volume_group_name` | removed | Creating nodes and storage pools was completely removed in Driver version 2. See <> | `linstor_volume_downsize_factor` | removed | This setting served no purpose, it was removed without replacement. |==== [[s-install_the_linstor_driver]] ==== Install the LINSTOR driver Starting with OpenStack Stein, the LINSTOR driver is part of the Cinder project. While the driver can be used as is, it might be missing features or fixes available in newer version. Due to OpenStacks update policy for stable versions, most improvements to the driver will not get back-ported to older stable releases. https://github.com/LINBIT/openstack-cinder[LINBIT maintains a fork] of the Cinder repository with all improvements to the LINSTOR driver backported to the supported stable versions. Currently, these are: [frame="topbot",options="header"] |==== | OpenStack Release | Included Version | LINBIT Version | LINBIT Branch | `wallaby` | 1.0.1 | 2.0.0 | https://github.com/LINBIT/openstack-cinder/tree/linstor%2Fstable%2Fwallaby[`linstor/stable/wallaby`] | `victoria` | 1.0.1 | 2.0.0 | https://github.com/LINBIT/openstack-cinder/tree/linstor%2Fstable%2Fvictoria[`linstor/stable/victoria`] | `ussuri` | 1.0.1 | 2.0.0 | https://github.com/LINBIT/openstack-cinder/tree/linstor%2Fstable%2Fussuri[`linstor/stable/ussuri`] | `train` | 1.0.0 | 2.0.0 | https://github.com/LINBIT/openstack-cinder/tree/linstor%2Fstable%2Ftrain[`linstor/stable/train`] | `stein` | 1.0.0 | 2.0.0 | https://github.com/LINBIT/openstack-cinder/tree/linstor%2Fstable%2Fstein[`linstor/stable/stein`] | `rocky` | n/a | n/a | n/a | `queens` | n/a | n/a | n/a | `pike` | n/a | n/a | n/a |==== The exact steps to enable the Linstor Driver depend on your OpenStack distribution. In general, the `python-linstor` package needs to be installed on all hosts running the Cinder volume service. The next section will cover the installation process for common OpenStack distributions. ===== DevStack https://docs.openstack.org/devstack/latest/[DevStack] is a great way to try out OpenStack in a lab environment. To use the most recent driver use the following DevStack configuration: .local.conf ---- # This ensures the Linstor Driver has access to the 'python-linstor' package. # # This is needed even if using the included driver! USE_VENV=True ADDITIONAL_VENV_PACKAGES=python-linstor # This is required to select the LINBIT version of the driver CINDER_REPO=https://github.com/LINBIT/openstack-cinder.git # Replace linstor/stable/victoria with the reference matching your Openstack release. CINDER_BRANCH=linstor/stable/victoria ---- ===== Kolla https://docs.openstack.org/kolla/latest/[Kolla] packages OpenStack components in containers. They can then be deployed, for example using https://docs.openstack.org/kolla-ansible/latest/[Kolla Ansible] You can take advantage of the available customisation options for kolla containers to set up the Linstor driver. To ensure that the required `python-linstor` package is installed, use the following override file: .template-override.j2 [source] ---- {% extends parent_template %} # Cinder {% set cinder_base_pip_packages_append = ['python-linstor'] %} ---- To install the LINBIT version of the driver, update your `kolla-build.conf` ./etc/kolla/kolla-build.conf ---- [cinder-base] type = git location = https://github.com/LINBIT/openstack-cinder.git # Replace linstor/stable/victoria with the reference matching your Openstack release. reference = linstor/stable/victoria ---- To rebuild the Cinder containers, run: [source,shell] ---- # A private registry used to store the kolla container images REGISTRY=deployment-registry.example.com # The image namespace in the registry NAMESPACE=kolla # The tag to apply to all images. Use the release name for compatibility with kolla-ansible TAG=victoria kolla-build -t source --template-override template-override.j2 cinder --registry $REGISTRY --namespace $NAMESPACE --tag $TAG ---- ====== Kolla Ansible When deploying OpenStack using Kolla Ansible, you need to make sure that: * the custom Cinder images, created in the section above, are used * deployment of Cinder services is enabled ./etc/kolla/globals.yml [source,yaml] ---- # use "source" images kolla_install_type: source # use the same registry as for running kolla-build above docker_registry: deployment-registry.example.com # use the same namespace as for running kolla-build above docker_namespace: kolla # deploy cinder block storage service enable_cinder: "yes" # disable verification of cinder backends, kolla-ansible only supports a small subset of available backends for this skip_cinder_backend_check: True # add the LINSTOR backend to the enabled backends. For backend configuration see below cinder_enabled_backends: - name: linstor-drbd ---- You can place the Linstor driver configuration in one of the override directories for kolla-ansible. For more details on the available configuration options, see the section below. ./etc/kolla/config/cinder/cinder-volume.conf ---- [linstor-drbd] volume_backend_name = linstor-drbd volume_driver = cinder.volume.drivers.linstordrv.LinstorDrbdDriver linstor_uris = linstor://cinder-01.openstack.test,linstor://cinder-02.openstack.test ---- ===== OpenStack Ansible https://docs.openstack.org/openstack-ansible/[OpenStack Ansible] provides Ansible playbooks to configure and deploy of OpenStack environments. It allows for fine-grained customization of the deployment, letting you set up the Linstor driver directly. ./etc/openstack_ansile/user_variables.yml ---- cinder_git_repo: https://github.com/LINBIT/openstack-cinder.git cinder_git_install_branch: linstor/stable/victoria cinder_user_pip_packages: - python-linstor cinder_backends: <1> linstor-drbd: volume_backend_name: linstor-drbd volume_driver: cinder.volume.drivers.linstordrv.LinstorDrbdDriver linstor_uris: linstor://cinder-01.openstack.test,linstor://cinder-02.openstack.test ---- <1> A detailed description of the available backend parameters can be found in the section below ===== Generic Cinder deployment For other forms of OpenStack deployments, this guide can only provide non-specific hints. To update the Linstor driver version, find your cinder installation. Some likely paths are: ---- /usr/lib/python3.6/dist-packages/cinder/ /usr/lib/python3.6/site-packages/cinder/ /usr/lib/python2.7/dist-packages/cinder/ /usr/lib/python2.7/site-packages/cinder/ ---- The Linstor driver consists of a single file called `linstordrv.py`, located in the Cinder directory: ---- $CINDER_PATH/volume/drivers/linstordrv.py ---- To update the driver, replace the file with one from the LINBIT repository ---- RELEASE=linstor/stable/victoria curl -fL "https://raw.githubusercontent.com/LINBIT/openstack-cinder/$RELEASE/cinder/volume/drivers/linstordrv.py" > $CINDER_PATH/volume/drivers/linstordrv.py ---- You might also need to remove the Python cache for the update to be registered: ---- rm -rf $CINDER_PATH/volume/drivers/__pycache__ ---- === Configure a Linstor Backend for Cinder To use the Linstor driver, configure the Cinder volume service. This is done by editing the Cinder configuration file and then restarting the Cinder Volume service. Most of the time, the Cinder configuration file is located at `/etc/cinder/cinder.conf`. Some deployment options allow manipulating this file in advance. See the section above for specifics. To configure a new volume backend using Linstor, add the following section to `cinder.conf` ---- [linstor-drbd] volume_backend_name = linstor-drbd <1> volume_driver = cinder.volume.drivers.linstordrv.LinstorDrbdDriver <2> linstor_uris = linstor://cinder-01.openstack.test,linstor://cinder-02.openstack.test <3> linstor_trusted_ca = /path/to/trusted/ca.cert <4> linstor_client_key = /path/to/client.key <5> linstor_client_cert = /path/to/client.cert <5> # Deprecated or removed in 2.0.0 linstor_default_storage_pool_name = cinderpool <6> linstor_autoplace_count = 2 <7> linstor_controller_diskless = true <8> # non-linstor-specific options ... <9> ---- NOTE: The parameters described here are based on the latest release provided by Linbit. The driver included in OpenStack might not support all of these parameters. Take a look at the https://docs.openstack.org/cinder/latest/configuration/block-storage/drivers/linstor-driver.html[OpenStack driver documentation] to find out more. <1> The name of the volume backend. Needs to be unique in the Cinder configuration. The whole section should share the same name. This name is referenced again in `cinder.conf` in the `enabled_backends` setting and when creating a new volume type. <2> The version of the Linstor driver to use. There are two options: * `cinder.volume.drivers.linstordrv.LinstorDrbdDriver` * `cinder.volume.drivers.linstordrv.LinstorIscsiDriver` + Which driver you should use depends on your Linstor set up and requirements. Details on each choice are documented in <> <3> The URL(s) of the Linstor Controller(s). Multiple Controllers can be specified to make use of <>. If not set, defaults to `linstor://localhost`. + NOTE: In driver versions prior to 2.0.0, this option is called `linstor_default_uri` <4> If <> the referenced certificate is used to verify the Linstor Controller authenticity. <5> If <> the referenced key and certificate will be presented to the Linstor Controller for authentication. <6> *Deprecated in 2.0.0, use <> instead.* The storage pools to use when placing resources. Applies to all diskfull resources created. Defaults to `DfltStorPool`. <7> *Removed in 2.0.0, use <> instead.* The number of replicas to create for the given volume. A value of `0` will create a replica on all nodes. Defaults to `0`. <8> *Removed in 2.0.0, volumes are created on demand by the driver.* If set to true, ensures that at least one (diskless) replica is deployed on the Cinder Controller host. This is useful for ISCSI transports. Defaults to `true`. <9> You can specify more generic Cinder options here, for example `target_helper = tgtadm` for the ISCSI connector. NOTE: You can also configure multiple Linstor backends, choosing a different name and configuration options for each. After configuring the Linstor backend, it should also be enabled. Add it to the list of enabled backends in `cinder.conf`, and optionally set is as the default backend: ---- [DEFAULT] ... default_volume_type = linstor-drbd-volume enabled_backends = lvm,linstor-drbd ... ---- As a last step, if you changed the Cinder configuration or updated the driver itself, make sure to restart the Cinder service(s). Please check the documentation for your OpenStack Distribution on how to restart services. [[s-openstack-linstor-transport-options]] ==== Choice of Transport Protocol The Transport Protocol in Cinder is how clients (for example nova-compute) access the actual volumes. With Linstor, you can choose between two different drivers that use different transports. * `cinder.volume.drivers.linstordrv.LinstorDrbdDriver`, which uses DRBD as transport * `cinder.volume.drivers.linstordrv.LinstorIscsiDriver`, which uses ISCSI as transport ===== Using DRBD as Transport Protocol The `LinstorDrbdDriver` works by ensuring a replica of the volume is available locally on the node where a client (i.e. nova-compute) issued a request. This only works if _all_ compute nodes are also running Linstor Satellites that are part of the same Linstor cluster. The advantages of this option are: * Once set up, the Cinder host is no longer involved in the data path. All read and write to the volume are handled by the local DRBD module, which will handle replication across its configured peers. * Since the Cinder host is not involved in the data path, any disruptions to the Cinder service do not affect volumes that are already attached. ====== Known limitations: * Not all hosts and hypervisors support using DRBD volumes. This restricts deployment to Linux hosts and `kvm` hypervisors. * Resizing of attached and in-use volumes does not fully work. While the resize itself is successful, the compute service will not propagate it to the VM until after a restart. * Multi-attach (attaching the same volume on multiple VMs) is not supported. * https://docs.openstack.org/cinder/latest/configuration/block-storage/volume-encryption.html#create-an-encrypted-volume-type[Encrypted volumes] only work if udev rules for DRBD devices are in place. + NOTE: `udev` rules are either part of the `drbd-utils` package or have their own `drbd-udev` package. ===== Using ISCSI as Transport Protocol The default way to export Cinder volumes is via iSCSI. This brings the advantage of maximum compatibility - iSCSI can be used with every hypervisor, be it VMWare, Xen, HyperV, or KVM. The drawback is that all data has to be sent to a Cinder node, to be processed by an (userspace) iSCSI daemon; that means that the data needs to pass the kernel/userspace border, and these transitions will cost some performance. Another drawback is the introduction of a single point of failure. If a Cinder node running the iSCSI daemon crashes, other nodes lose access to their volumes. There are ways to configure Cinder for automatic fail-over to mitigate this, but it requires considerable effort. NOTE: In driver versions prior to 2.0.0, the Cinder host needs access to a local replica of every volume. This can be achieved by either setting `linstor_controller_diskless=True` or using `linstor_autoplace_count=0`. Newer driver versions will create such a volume on demand. [[s-openstack-linstor-backend-status]] ==== Verify status of Linstor backends To verify that all backends are up and running, you can use the OpenStack command line client: [source,shell] ---- $ openstack volume service list +------------------+----------------------------------------+------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At | +------------------+----------------------------------------+------+---------+-------+----------------------------+ | cinder-scheduler | cinder-01.openstack.test | nova | enabled | up | 2021-03-10T12:24:37.000000 | | cinder-volume | cinder-01.openstack.test@linstor-drbd | nova | enabled | up | 2021-03-10T12:24:34.000000 | | cinder-volume | cinder-01.openstack.test@linstor-iscsi | nova | enabled | up | 2021-03-10T12:24:35.000000 | +------------------+----------------------------------------+------+---------+-------+----------------------------+ ---- If you have the Horizon GUI deployed, check `Admin > System Information > Block Storage Service` instead. In the above example all configured services are `enabled` and `up`. If there are any issues, please check the logs of the Cinder Volume service. [[s-openstack-volume-types]] === Create a new volume type for Linstor Before creating volumes using Cinder, you have to create a volume type. This can either be done via the command line: [source,shell] ---- # Create a volume using the default backend $ openstack volume type create default +-------------+--------------------------------------+ | Field | Value | +-------------+--------------------------------------+ | description | None | | id | 58365ffb-959a-4d91-8821-5d72e5c39c26 | | is_public | True | | name | default | +-------------+--------------------------------------+ # Create a volume using a specific backend $ openstack volume type create --property volume_backend_name=linstor-drbd linstor-drbd-volume +-------------+--------------------------------------+ | Field | Value | +-------------+--------------------------------------+ | description | None | | id | 08562ea8-e90b-4f95-87c8-821ac64630a5 | | is_public | True | | name | linstor-drbd-volume | | properties | volume_backend_name='linstor-drbd' | +-------------+--------------------------------------+ ---- Alternatively, you can create volume types via the Horizon GUI. Navigate to `Admin > Volume > Volume Types` and click "Create Volume Type". You can assign it a backend by adding the `volume_backend_name` as "Extra Specs" to it. ==== Advanced Configuration of volume types Each volume type can be customized by adding properties or "Extra Specs" as they are called in the Horizon GUI. To add a property to a volume type on the command line use: ---- openstack volume type set linstor_drbd_b --property linstor:redundancy=5 ---- Alternatively, you can set the property via the GUI by navigating tp `Admin > Volume > Volume Types`. In the `Actions` column, open the dropdown menu and click the `View Extra Specs` button. This opens a dialog you can use to create, edit and delete properties. ===== Available volume type properties linstor:diskless_on_remaining:: Create diskless replicas on non-selected nodes after auto-placing. linstor:do_not_place_with_regex:: Do not place the resource on a node which has a resource with a name matching the regex. linstor:layer_list:: Comma-separated list of layers to apply for resources. If empty, defaults to DRBD,Storage. linstor:provider_list:: Comma-separated list of providers to use. If empty, Linstor will automatically choose a suitable provider. linstor:redundancy:: Number of replicas to create. Defaults to 2. linstor:replicas_on_different:: A comma-separated list of key or key=value items used as autoplacement selection labels when autoplace is used to determine where to provision storage. linstor:replicas_on_same:: A comma-separated list of key or key=value items used as autoplacement selection labels when autoplace is used to determine where to provision storage. linstor:storage_pool:: Comma-separated list of storage pools to use when auto-placing. linstor:property:*:: If a key is prefixed by `linstor:property:`, it is interpreted as a LINSTOR property. The property gets set on the <> created for the volume type. + For example: To change the <> `DrbdOptions/auto-quorum` needs to be set. This can be done by setting the `linstor:property:DrbdOptions/auto-quorum` property. === Using volumes Once you have a volume type configured, you can start using it to provision new volumes. For example, to create a simple 1Gb volume on the command line you can use: [source,shell] ---- openstack volume create --type linstor-drbd-volume --size 1 --availability-zone nova linstor-test-vol openstack volume list ---- NOTE: If you set `default_volume_type = linstor-drbd-volume` in your `/etc/cinder/cinder.conf`, you may omit the `--type linstor-drbd-volume` from the `openstack volume create ...` command above. === Troubleshooting This section describes what to do in case you encounter problems with using Linstor volumes and snapshots. ==== Checking for error messages in Horizon Every volume and snapshot has a Messages tab in the Horizon dashboard. In case of errors, the list of messages can be used as a starting point for further investigation. Some common messages in case of errors: create volume from backend storage:Driver failed to create the volume. There was an error creating a new volume. Check the Cinder Volume service logs for more details schedule allocate volume:Could not find any available weighted backend. If this is the only error message, this means the Cinder Scheduler could not find a volume backend suitable for creating the volume. This is most likely because: * The volume backend is offline. See <> * The volume backend has not enough free capacity to fulfil the request. Check the output of `cinder get-pools --detail` and `linstor storage-pool list` to ensure that the requested capacity is available. ==== Checking the Cinder Volume Service The Linstor driver is called as part of the Cinder Volume service. [frame="topbot",options="header"] |==== | Distribution | Log location or command | DevStack | `journalctl -u devstack@c-vol` |==== ==== Checking the compute srvice logs Some issues will not be logged in the Cinder Service but in the actual consumer of the volumes, most likely the compute service (Nova). As with the volume service, the exact host and location to check depends on your Openstack distribution: [frame="topbot",options="header"] |==== | Distribution | Log location or command | DevStack | `journalctl -u devstack@n-cpu` |==== // Keep the empty line before this comment, otherwise the next chapter is folded into this