Debian Codegraph @

UG9/en/proxmox-linstor.adoc

raw
1 [[ch-proxmox-linstor]]
2 == LINSTOR Volumes in Proxmox VE
3
4 indexterm:[Proxmox]This chapter describes DRBD in Proxmox VE via
5 the http://git.linbit.com/linstor-proxmox.git[LINSTOR Proxmox Plugin].
6
7 [[s-proxmox-ls-overview]]
8 === Proxmox VE Overview
9
10 http://www.proxmox.com/en/[Proxmox VE] is an easy to use, complete server
11 virtualization environment with KVM, Linux Containers and HA.
12
13 'linstor-proxmox' is a Perl plugin for Proxmox that, in combination with LINSTOR, allows to replicate VM
14 //(LVM volumes on DRBD)
15 disks on several Proxmox VE nodes. This allows to live-migrate
16 active VMs within a few seconds and with no downtime without needing a central SAN, as the data is already
17 replicated to multiple nodes.
18
19 [[s-proxmox-ls-upgrades]]
20 === Upgrades
21 If this is a fresh installation, skip this section and continue with <<s-proxmox-ls-install>>.
22
23 [[s-proxmox-ls-upgrades-4-5]]
24 ==== From 4.x to 5.x
25 Version 5 of the plugin drops compatibility with the legacy configuration options "storagepool" and
26 "redundancy". Version 5 *requires* a "resourcegroup" option, and obviously a LINSTOR resource group. The old
27 options should be removed from the config.
28
29 Configuring LINSTOR is described in Section <<s-proxmox-ls-ls-configuration>>, a typical example follows:
30 Let's assume the pool was set to "mypool", and redundancy to 3.
31 ----------------------------
32 # linstor resource-group create --storage-pool=mypool --place-count=3 drbdMypoolThree
33 # linstor volume-group create drbdMypoolThree
34 # vi /etc/pve/storage.cfg
35 drbd: drbdstorage
36 content images,rootdir
37 controller 10.11.12.13
38 resourcegroup drbdMypoolThree
39 ----------------------------
40
41 [[s-proxmox-ls-upgrades-5-6]]
42 ==== From 5.x to 6.x
43 With version 6 PVE added additional parameters to some functions and rightfully reset their "APIAGE". This
44 means that old plugins, while actually usable as they don't use any of these changed functions do not work
45 anymore. Please upgrade to plugin version 5.2.1 at least.
46
47 [[s-proxmox-ls-install]]
48 === Proxmox Plugin Installation
49
50 LINBIT provides a dedicated public repository for Proxmox VE users. This repository not only contains the
51 Proxmox plugin, but the whole DRBD SDS stack including a DRBD SDS kernel
52 module and user space utilities.
53
54 The DRBD9 kernel module is installed as a `dkms` package (i.e., `drbd-dkms`), therefore you'll have to install
55 `pve-headers` package, before you set up/install the software packages from LINBIT's repositories. Following
56 that order, ensures that the kernel module will build properly for your kernel. If you don't plan to install
57 the latest Proxmox kernel, you have to install kernel headers matching your current running kernel (e.g.,
58 `pve-headers-$(uname -r)`). If you missed this step, then still you can rebuild the dkms package against
59 your current kernel, (kernel headers have to be installed in advance), by issuing `apt install --reinstall
60 drbd-dkms` command.
61
62 LINBIT's repository can be enabled as follows, where "$PVERS" should be set to your Proxmox VE *major version*
63 (e.g., "7", not "7.1"):
64
65 ----------------------------
66 # wget -O- https://packages.linbit.com/package-signing-pubkey.asc | apt-key add -
67 # PVERS=7 && echo "deb http://packages.linbit.com/proxmox/ proxmox-$PVERS drbd-9" > \
68 /etc/apt/sources.list.d/linbit.list
69 # apt update && apt install linstor-proxmox
70 ----------------------------
71
72 [[s-proxmox-ls-ls-configuration]]
73 === LINSTOR Configuration
74 For the rest of this guide we assume that you have a LINSTOR cluster configured as described in
75 <<s-linstor-init-cluster>>. Start the "linstor-controller" on one node, and the "linstor-satellite" on all
76 nodes. The preferred way to use the plugin, starting from version 4.1.0, is via LINSTOR resource groups and a
77 single volume group within every resource group. LINSTOR resource groups are described in
78 <<s-linstor-resource-groups>>. All the required LINSTOR configuration (e.g., redundancy count) has to be set
79 on the resource group.
80
81 [[s-proxmox-ls-configuration]]
82 === Proxmox Plugin Configuration
83 The final step is to provide a configuration for Proxmox itself. This can be done by adding an entry in the
84 `/etc/pve/storage.cfg` file, with a content similar to the following.
85
86 ----------------------------
87 drbd: drbdstorage
88 content images,rootdir
89 controller 10.11.12.13
90 resourcegroup defaultpool
91 ----------------------------
92
93 The "drbd" entry is fixed and you are not allowed to modify it, as it tells to Proxmox to use DRBD as storage
94 backend. The "drbdstorage" entry can be modified and is used as a friendly name that will be shown in the PVE
95 web GUI to locate the DRBD storage. The "content" entry is also fixed, so do not change it. The redundancy
96 (specified in the resource group) specifies how many replicas of the data will be stored in the cluster. The recommendation is to set it
97 to 2 or 3 depending on your setup. The data is accessible from all nodes, even
98 if some of them do not have local copies of the data. For example, in a 5 node cluster, all nodes will be
99 able to access 3 copies of the data, no matter where they are stored in. The "controller" parameter must be
100 set to the IP of the node that runs the LINSTOR controller service. Only one node can be set to run as LINSTOR
101 controller at the same time. If that node fails, start the LINSTOR controller on another node and change that
102 value to its IP address.
103
104 A configuration using different storage pools in different resource groups would look like this:
105
106 ----------------------------
107 drbd: drbdstorage
108 content images,rootdir
109 controller 10.11.12.13
110 resourcegroup defaultpool
111
112 drbd: fastdrbd
113 content images,rootdir
114 controller 10.11.12.13
115 resourcegroup ssd
116
117 drbd: slowdrbd
118 content images,rootdir
119 controller 10.11.12.13
120 resourcegroup backup
121 ----------------------------
122
123 By now, you should be able to create VMs via Proxmox's web GUI by selecting "__drbdstorage__", or any other of
124 the defined pools as storage location.
125
126 Starting from version 5 of the plugin one can set the option "preferlocal yes". If it is set, the plugin tries
127 to create a diskful assignment on the node that issued the storage create command. With this option one can
128 make sure the VM gets local storage if possible. Without that option LINSTOR might place the storage on nodes
129 'B' and 'C', while the VM is initially started on node 'A'. This would still work as node 'A' then would get a
130 diskless assignment, but having local storage might be preferred.
131
132 .NOTE: DRBD supports only the **raw** disk format at the moment.
133
134 At this point you can try to live migrate the VM - as all data is accessible on all nodes (even on Diskless
135 nodes) - it will take just a few seconds. The overall process might take a bit longer if the VM is under
136 load and if there is a lot of RAM being dirtied all the time. But in any case, the downtime should be minimal
137 and you will see no interruption at all.
138
139 .Table Configuration Options
140 |===
141 | Option | Meaning
142
143 | `controller` | The IP of the LINSTOR controller (',' separated list allowed)
144 | `resourcegroup` | The name of a LINSTOR resource group which defines the deployment of new VMs. As described above
145 | `preferlocal` | Prefer to create local storage (yes/no). As decribed above
146 | `statuscache` | Time in seconds status information is cached, 0 means no extra cache. Relevant on huge clusters with hundreds of resources. This has to be set on *all* `drbd` storages in `/etc/pve/storage.cfg` to take effect.
147 | `apicrt` | Path to the client certificate
148 | `apikey` | Path to the client private key
149 | `apica` | Path to the CA certificate
150 |===
151
152 [[s-proxmox-ls-HA]]
153 === Making the Controller Highly-Available (optional)
154 Making LINSTOR highly-available is a matter of making the LINSTOR controller
155 highly-available. This step is described in Section <<s-linstor_ha>>.
156
157 The last -- but crucial -- step is to configure the Proxmox plugin to be
158 able to connect to multiple LINSTOR controllers. It will use the first one it
159 receives an answer from. This is done by adding a comma-separated list of
160 controllers in the `controller` section of the plugin like this:
161
162 ----------------------------
163 drbd: drbdstorage
164 content images,rootdir
165 controller 10.11.12.13,10.11.12.14,10.11.12.15
166 resourcegroup defaultpool
167 ----------------------------
168