| 1 |
[[ch-proxmox-linstor]] |
| 2 |
== LINSTOR Volumes in Proxmox VE |
| 3 |
|
| 4 |
indexterm:[Proxmox]This chapter describes DRBD in Proxmox VE via |
| 5 |
the http://git.linbit.com/linstor-proxmox.git[LINSTOR Proxmox Plugin]. |
| 6 |
|
| 7 |
[[s-proxmox-ls-overview]] |
| 8 |
=== Proxmox VE Overview |
| 9 |
|
| 10 |
http://www.proxmox.com/en/[Proxmox VE] is an easy to use, complete server |
| 11 |
virtualization environment with KVM, Linux Containers and HA. |
| 12 |
|
| 13 |
'linstor-proxmox' is a Perl plugin for Proxmox that, in combination with LINSTOR, allows to replicate VM |
| 14 |
//(LVM volumes on DRBD) |
| 15 |
disks on several Proxmox VE nodes. This allows to live-migrate |
| 16 |
active VMs within a few seconds and with no downtime without needing a central SAN, as the data is already |
| 17 |
replicated to multiple nodes. |
| 18 |
|
| 19 |
[[s-proxmox-ls-upgrades]] |
| 20 |
=== Upgrades |
| 21 |
If this is a fresh installation, skip this section and continue with <<s-proxmox-ls-install>>. |
| 22 |
|
| 23 |
[[s-proxmox-ls-upgrades-4-5]] |
| 24 |
==== From 4.x to 5.x |
| 25 |
Version 5 of the plugin drops compatibility with the legacy configuration options "storagepool" and |
| 26 |
"redundancy". Version 5 *requires* a "resourcegroup" option, and obviously a LINSTOR resource group. The old |
| 27 |
options should be removed from the config. |
| 28 |
|
| 29 |
Configuring LINSTOR is described in Section <<s-proxmox-ls-ls-configuration>>, a typical example follows: |
| 30 |
Let's assume the pool was set to "mypool", and redundancy to 3. |
| 31 |
---------------------------- |
| 32 |
# linstor resource-group create --storage-pool=mypool --place-count=3 drbdMypoolThree |
| 33 |
# linstor volume-group create drbdMypoolThree |
| 34 |
# vi /etc/pve/storage.cfg |
| 35 |
drbd: drbdstorage |
| 36 |
content images,rootdir |
| 37 |
controller 10.11.12.13 |
| 38 |
resourcegroup drbdMypoolThree |
| 39 |
---------------------------- |
| 40 |
|
| 41 |
[[s-proxmox-ls-upgrades-5-6]] |
| 42 |
==== From 5.x to 6.x |
| 43 |
With version 6 PVE added additional parameters to some functions and rightfully reset their "APIAGE". This |
| 44 |
means that old plugins, while actually usable as they don't use any of these changed functions do not work |
| 45 |
anymore. Please upgrade to plugin version 5.2.1 at least. |
| 46 |
|
| 47 |
[[s-proxmox-ls-install]] |
| 48 |
=== Proxmox Plugin Installation |
| 49 |
|
| 50 |
LINBIT provides a dedicated public repository for Proxmox VE users. This repository not only contains the |
| 51 |
Proxmox plugin, but the whole DRBD SDS stack including a DRBD SDS kernel |
| 52 |
module and user space utilities. |
| 53 |
|
| 54 |
The DRBD9 kernel module is installed as a `dkms` package (i.e., `drbd-dkms`), therefore you'll have to install |
| 55 |
`pve-headers` package, before you set up/install the software packages from LINBIT's repositories. Following |
| 56 |
that order, ensures that the kernel module will build properly for your kernel. If you don't plan to install |
| 57 |
the latest Proxmox kernel, you have to install kernel headers matching your current running kernel (e.g., |
| 58 |
`pve-headers-$(uname -r)`). If you missed this step, then still you can rebuild the dkms package against |
| 59 |
your current kernel, (kernel headers have to be installed in advance), by issuing `apt install --reinstall |
| 60 |
drbd-dkms` command. |
| 61 |
|
| 62 |
LINBIT's repository can be enabled as follows, where "$PVERS" should be set to your Proxmox VE *major version* |
| 63 |
(e.g., "7", not "7.1"): |
| 64 |
|
| 65 |
---------------------------- |
| 66 |
# wget -O- https://packages.linbit.com/package-signing-pubkey.asc | apt-key add - |
| 67 |
# PVERS=7 && echo "deb http://packages.linbit.com/proxmox/ proxmox-$PVERS drbd-9" > \ |
| 68 |
/etc/apt/sources.list.d/linbit.list |
| 69 |
# apt update && apt install linstor-proxmox |
| 70 |
---------------------------- |
| 71 |
|
| 72 |
[[s-proxmox-ls-ls-configuration]] |
| 73 |
=== LINSTOR Configuration |
| 74 |
For the rest of this guide we assume that you have a LINSTOR cluster configured as described in |
| 75 |
<<s-linstor-init-cluster>>. Start the "linstor-controller" on one node, and the "linstor-satellite" on all |
| 76 |
nodes. The preferred way to use the plugin, starting from version 4.1.0, is via LINSTOR resource groups and a |
| 77 |
single volume group within every resource group. LINSTOR resource groups are described in |
| 78 |
<<s-linstor-resource-groups>>. All the required LINSTOR configuration (e.g., redundancy count) has to be set |
| 79 |
on the resource group. |
| 80 |
|
| 81 |
[[s-proxmox-ls-configuration]] |
| 82 |
=== Proxmox Plugin Configuration |
| 83 |
The final step is to provide a configuration for Proxmox itself. This can be done by adding an entry in the |
| 84 |
`/etc/pve/storage.cfg` file, with a content similar to the following. |
| 85 |
|
| 86 |
---------------------------- |
| 87 |
drbd: drbdstorage |
| 88 |
content images,rootdir |
| 89 |
controller 10.11.12.13 |
| 90 |
resourcegroup defaultpool |
| 91 |
---------------------------- |
| 92 |
|
| 93 |
The "drbd" entry is fixed and you are not allowed to modify it, as it tells to Proxmox to use DRBD as storage |
| 94 |
backend. The "drbdstorage" entry can be modified and is used as a friendly name that will be shown in the PVE |
| 95 |
web GUI to locate the DRBD storage. The "content" entry is also fixed, so do not change it. The redundancy |
| 96 |
(specified in the resource group) specifies how many replicas of the data will be stored in the cluster. The recommendation is to set it |
| 97 |
to 2 or 3 depending on your setup. The data is accessible from all nodes, even |
| 98 |
if some of them do not have local copies of the data. For example, in a 5 node cluster, all nodes will be |
| 99 |
able to access 3 copies of the data, no matter where they are stored in. The "controller" parameter must be |
| 100 |
set to the IP of the node that runs the LINSTOR controller service. Only one node can be set to run as LINSTOR |
| 101 |
controller at the same time. If that node fails, start the LINSTOR controller on another node and change that |
| 102 |
value to its IP address. |
| 103 |
|
| 104 |
A configuration using different storage pools in different resource groups would look like this: |
| 105 |
|
| 106 |
---------------------------- |
| 107 |
drbd: drbdstorage |
| 108 |
content images,rootdir |
| 109 |
controller 10.11.12.13 |
| 110 |
resourcegroup defaultpool |
| 111 |
|
| 112 |
drbd: fastdrbd |
| 113 |
content images,rootdir |
| 114 |
controller 10.11.12.13 |
| 115 |
resourcegroup ssd |
| 116 |
|
| 117 |
drbd: slowdrbd |
| 118 |
content images,rootdir |
| 119 |
controller 10.11.12.13 |
| 120 |
resourcegroup backup |
| 121 |
---------------------------- |
| 122 |
|
| 123 |
By now, you should be able to create VMs via Proxmox's web GUI by selecting "__drbdstorage__", or any other of |
| 124 |
the defined pools as storage location. |
| 125 |
|
| 126 |
Starting from version 5 of the plugin one can set the option "preferlocal yes". If it is set, the plugin tries |
| 127 |
to create a diskful assignment on the node that issued the storage create command. With this option one can |
| 128 |
make sure the VM gets local storage if possible. Without that option LINSTOR might place the storage on nodes |
| 129 |
'B' and 'C', while the VM is initially started on node 'A'. This would still work as node 'A' then would get a |
| 130 |
diskless assignment, but having local storage might be preferred. |
| 131 |
|
| 132 |
.NOTE: DRBD supports only the **raw** disk format at the moment. |
| 133 |
|
| 134 |
At this point you can try to live migrate the VM - as all data is accessible on all nodes (even on Diskless |
| 135 |
nodes) - it will take just a few seconds. The overall process might take a bit longer if the VM is under |
| 136 |
load and if there is a lot of RAM being dirtied all the time. But in any case, the downtime should be minimal |
| 137 |
and you will see no interruption at all. |
| 138 |
|
| 139 |
.Table Configuration Options |
| 140 |
|=== |
| 141 |
| Option | Meaning |
| 142 |
|
| 143 |
| `controller` | The IP of the LINSTOR controller (',' separated list allowed) |
| 144 |
| `resourcegroup` | The name of a LINSTOR resource group which defines the deployment of new VMs. As described above |
| 145 |
| `preferlocal` | Prefer to create local storage (yes/no). As decribed above |
| 146 |
| `statuscache` | Time in seconds status information is cached, 0 means no extra cache. Relevant on huge clusters with hundreds of resources. This has to be set on *all* `drbd` storages in `/etc/pve/storage.cfg` to take effect. |
| 147 |
| `apicrt` | Path to the client certificate |
| 148 |
| `apikey` | Path to the client private key |
| 149 |
| `apica` | Path to the CA certificate |
| 150 |
|=== |
| 151 |
|
| 152 |
[[s-proxmox-ls-HA]] |
| 153 |
=== Making the Controller Highly-Available (optional) |
| 154 |
Making LINSTOR highly-available is a matter of making the LINSTOR controller |
| 155 |
highly-available. This step is described in Section <<s-linstor_ha>>. |
| 156 |
|
| 157 |
The last -- but crucial -- step is to configure the Proxmox plugin to be |
| 158 |
able to connect to multiple LINSTOR controllers. It will use the first one it |
| 159 |
receives an answer from. This is done by adding a comma-separated list of |
| 160 |
controllers in the `controller` section of the plugin like this: |
| 161 |
|
| 162 |
---------------------------- |
| 163 |
drbd: drbdstorage |
| 164 |
content images,rootdir |
| 165 |
controller 10.11.12.13,10.11.12.14,10.11.12.15 |
| 166 |
resourcegroup defaultpool |
| 167 |
---------------------------- |
| 168 |
|