[[ch-proxmox-linstor]] == LINSTOR Volumes in Proxmox VE indexterm:[Proxmox]This chapter describes DRBD in Proxmox VE via the http://git.linbit.com/linstor-proxmox.git[LINSTOR Proxmox Plugin]. [[s-proxmox-ls-overview]] === Proxmox VE Overview http://www.proxmox.com/en/[Proxmox VE] is an easy to use, complete server virtualization environment with KVM, Linux Containers and HA. 'linstor-proxmox' is a Perl plugin for Proxmox that, in combination with LINSTOR, allows to replicate VM //(LVM volumes on DRBD) disks on several Proxmox VE nodes. This allows to live-migrate active VMs within a few seconds and with no downtime without needing a central SAN, as the data is already replicated to multiple nodes. [[s-proxmox-ls-upgrades]] === Upgrades If this is a fresh installation, skip this section and continue with <>. [[s-proxmox-ls-upgrades-4-5]] ==== From 4.x to 5.x Version 5 of the plugin drops compatibility with the legacy configuration options "storagepool" and "redundancy". Version 5 *requires* a "resourcegroup" option, and obviously a LINSTOR resource group. The old options should be removed from the config. Configuring LINSTOR is described in Section <>, a typical example follows: Let's assume the pool was set to "mypool", and redundancy to 3. ---------------------------- # linstor resource-group create --storage-pool=mypool --place-count=3 drbdMypoolThree # linstor volume-group create drbdMypoolThree # vi /etc/pve/storage.cfg drbd: drbdstorage content images,rootdir controller 10.11.12.13 resourcegroup drbdMypoolThree ---------------------------- [[s-proxmox-ls-upgrades-5-6]] ==== From 5.x to 6.x With version 6 PVE added additional parameters to some functions and rightfully reset their "APIAGE". This means that old plugins, while actually usable as they don't use any of these changed functions do not work anymore. Please upgrade to plugin version 5.2.1 at least. [[s-proxmox-ls-install]] === Proxmox Plugin Installation LINBIT provides a dedicated public repository for Proxmox VE users. This repository not only contains the Proxmox plugin, but the whole DRBD SDS stack including a DRBD SDS kernel module and user space utilities. The DRBD9 kernel module is installed as a `dkms` package (i.e., `drbd-dkms`), therefore you'll have to install `pve-headers` package, before you set up/install the software packages from LINBIT's repositories. Following that order, ensures that the kernel module will build properly for your kernel. If you don't plan to install the latest Proxmox kernel, you have to install kernel headers matching your current running kernel (e.g., `pve-headers-$(uname -r)`). If you missed this step, then still you can rebuild the dkms package against your current kernel, (kernel headers have to be installed in advance), by issuing `apt install --reinstall drbd-dkms` command. LINBIT's repository can be enabled as follows, where "$PVERS" should be set to your Proxmox VE *major version* (e.g., "7", not "7.1"): ---------------------------- # wget -O- https://packages.linbit.com/package-signing-pubkey.asc | apt-key add - # PVERS=7 && echo "deb http://packages.linbit.com/proxmox/ proxmox-$PVERS drbd-9" > \ /etc/apt/sources.list.d/linbit.list # apt update && apt install linstor-proxmox ---------------------------- [[s-proxmox-ls-ls-configuration]] === LINSTOR Configuration For the rest of this guide we assume that you have a LINSTOR cluster configured as described in <>. Start the "linstor-controller" on one node, and the "linstor-satellite" on all nodes. The preferred way to use the plugin, starting from version 4.1.0, is via LINSTOR resource groups and a single volume group within every resource group. LINSTOR resource groups are described in <>. All the required LINSTOR configuration (e.g., redundancy count) has to be set on the resource group. [[s-proxmox-ls-configuration]] === Proxmox Plugin Configuration The final step is to provide a configuration for Proxmox itself. This can be done by adding an entry in the `/etc/pve/storage.cfg` file, with a content similar to the following. ---------------------------- drbd: drbdstorage content images,rootdir controller 10.11.12.13 resourcegroup defaultpool ---------------------------- The "drbd" entry is fixed and you are not allowed to modify it, as it tells to Proxmox to use DRBD as storage backend. The "drbdstorage" entry can be modified and is used as a friendly name that will be shown in the PVE web GUI to locate the DRBD storage. The "content" entry is also fixed, so do not change it. The redundancy (specified in the resource group) specifies how many replicas of the data will be stored in the cluster. The recommendation is to set it to 2 or 3 depending on your setup. The data is accessible from all nodes, even if some of them do not have local copies of the data. For example, in a 5 node cluster, all nodes will be able to access 3 copies of the data, no matter where they are stored in. The "controller" parameter must be set to the IP of the node that runs the LINSTOR controller service. Only one node can be set to run as LINSTOR controller at the same time. If that node fails, start the LINSTOR controller on another node and change that value to its IP address. A configuration using different storage pools in different resource groups would look like this: ---------------------------- drbd: drbdstorage content images,rootdir controller 10.11.12.13 resourcegroup defaultpool drbd: fastdrbd content images,rootdir controller 10.11.12.13 resourcegroup ssd drbd: slowdrbd content images,rootdir controller 10.11.12.13 resourcegroup backup ---------------------------- By now, you should be able to create VMs via Proxmox's web GUI by selecting "__drbdstorage__", or any other of the defined pools as storage location. Starting from version 5 of the plugin one can set the option "preferlocal yes". If it is set, the plugin tries to create a diskful assignment on the node that issued the storage create command. With this option one can make sure the VM gets local storage if possible. Without that option LINSTOR might place the storage on nodes 'B' and 'C', while the VM is initially started on node 'A'. This would still work as node 'A' then would get a diskless assignment, but having local storage might be preferred. .NOTE: DRBD supports only the **raw** disk format at the moment. At this point you can try to live migrate the VM - as all data is accessible on all nodes (even on Diskless nodes) - it will take just a few seconds. The overall process might take a bit longer if the VM is under load and if there is a lot of RAM being dirtied all the time. But in any case, the downtime should be minimal and you will see no interruption at all. .Table Configuration Options |=== | Option | Meaning | `controller` | The IP of the LINSTOR controller (',' separated list allowed) | `resourcegroup` | The name of a LINSTOR resource group which defines the deployment of new VMs. As described above | `preferlocal` | Prefer to create local storage (yes/no). As decribed above | `statuscache` | Time in seconds status information is cached, 0 means no extra cache. Relevant on huge clusters with hundreds of resources. This has to be set on *all* `drbd` storages in `/etc/pve/storage.cfg` to take effect. | `apicrt` | Path to the client certificate | `apikey` | Path to the client private key | `apica` | Path to the CA certificate |=== [[s-proxmox-ls-HA]] === Making the Controller Highly-Available (optional) Making LINSTOR highly-available is a matter of making the LINSTOR controller highly-available. This step is described in Section <>. The last -- but crucial -- step is to configure the Proxmox plugin to be able to connect to multiple LINSTOR controllers. It will use the first one it receives an answer from. This is done by adding a comma-separated list of controllers in the `controller` section of the plugin like this: ---------------------------- drbd: drbdstorage content images,rootdir controller 10.11.12.13,10.11.12.14,10.11.12.15 resourcegroup defaultpool ----------------------------