| 1 |
[[ch-linstor-gateway]] |
| 2 |
== Exporting Highly Available Storage using LINSTOR Gateway |
| 3 |
|
| 4 |
*LINSTOR Gateway* manages highly available iSCSI targets and NFS exports by leveraging on LINSTOR and |
| 5 |
Pacemaker. Setting up LINSTOR - including storage pools and resource groups - as well as Corosync and |
| 6 |
Pacemaker's properties are a prerequisite to use this tool. |
| 7 |
|
| 8 |
[[s-linstor-gateway-requirements]] |
| 9 |
=== Requirements |
| 10 |
|
| 11 |
==== LINSTOR |
| 12 |
|
| 13 |
A LINSTOR cluster is required to operate LINSTOR Gateway. It is *highly recommended* to run the LINSTOR |
| 14 |
controller as a Pacemaker resource. This needs to be configured manually. Such a resource could look like the |
| 15 |
following: (optional) |
| 16 |
|
| 17 |
----------------- |
| 18 |
primitive p_linstor-controller systemd:linstor-controller \ |
| 19 |
op start interval=0 timeout=100s \ |
| 20 |
op stop interval=0 timeout=100s \ |
| 21 |
op monitor interval=30s timeout=100s |
| 22 |
----------------- |
| 23 |
|
| 24 |
For both iSCSI and NFS, `storage pool` , `resource group` and a `volume group` for LINSTOR Gateway needs to be |
| 25 |
present. Lets do it; |
| 26 |
|
| 27 |
Creating the storage pool in 3 nodes using the phsical device `/dev/sdb`; |
| 28 |
|
| 29 |
----------------- |
| 30 |
linstor physical-storage create-device-pool --pool-name lvpool LVM LINSTOR1 /dev/sdb --storage-pool lvmpool |
| 31 |
linstor physical-storage create-device-pool --pool-name lvpool LVM LINSTOR2 /dev/sdb --storage-pool lvmpool |
| 32 |
linstor physical-storage create-device-pool --pool-name lvpool LVM LINSTOR3 /dev/sdb --storage-pool lvmpool |
| 33 |
----------------- |
| 34 |
|
| 35 |
We also need resource groups and volume groups; |
| 36 |
|
| 37 |
----------------- |
| 38 |
linstor rg c iSCSI_group --storage-pool lvmpool --place-count 2 |
| 39 |
linstor rg c nfs_group --storage-pool lvmpool --place-count 3 |
| 40 |
----------------- |
| 41 |
|
| 42 |
----------------- |
| 43 |
linstor vg c iSCSI_group |
| 44 |
linstor vg c nfs_group |
| 45 |
----------------- |
| 46 |
|
| 47 |
For a more detailed explanation of the storage pool, resource group and volume group creation, check the |
| 48 |
https://linbit.com/drbd-user-guide/LINSTOR-guide-1_0-en/#s-storage_pools[LINSTOR user guide] |
| 49 |
|
| 50 |
==== Pacemaker |
| 51 |
|
| 52 |
A working Corosync/Pacemaker cluster is expected on the machine where LINSTOR Gateway is running. |
| 53 |
The https://github.com/LINBIT/drbd-utils/blob/master/scripts/drbd-attr[drbd-attr] resource agent is required |
| 54 |
to run LINSTOR Gateway. This is included in LINBIT's `drbd-utils` package for Ubuntu based distributions, or |
| 55 |
the drbd-pacemaker package on RHEL/CentOS. LINSTOR Gateway sets up all required Pacemaker resource and |
| 56 |
constraints by itself, except for the LINSTOR controller resource. |
| 57 |
|
| 58 |
==== iSCSI & NFS |
| 59 |
|
| 60 |
LINSTOR Gateway uses Pacemaker's `ocf::heartbeat:iSCSITarget` resource agent for its iSCSI integration, which |
| 61 |
requires an iSCSI implementation to be installed. Using `targetcli` is recommended. |
| 62 |
|
| 63 |
For iSCSI, please install targetcli. |
| 64 |
|
| 65 |
----------------- |
| 66 |
yum install targetcli |
| 67 |
----------------- |
| 68 |
|
| 69 |
For nfs, nfs-server needs to be enabled and ready; |
| 70 |
|
| 71 |
----------------- |
| 72 |
systemctl enable --now nfs-server |
| 73 |
----------------- |
| 74 |
|
| 75 |
[[s-linstor-gateway-preparation]] |
| 76 |
=== Preparation |
| 77 |
First, let's check that all the components are available. This guide assumes you already installed and |
| 78 |
configured a LINSTOR cluster. Volume Group, Storage Pool and Resource Group should be defined before using |
| 79 |
linstor-iscsi or linstor-nfs. |
| 80 |
|
| 81 |
Tools need to be present in server; |
| 82 |
|
| 83 |
* linstor-client (managing the LINSTOR cluster) |
| 84 |
* drbd-attr resource agent (part of drbd-utils in Debian/Ubuntu, and part of drbd-pacemaker for other |
| 85 |
Linux distrubutions) |
| 86 |
* targetcli (for iSCSI) |
| 87 |
* nfs-utils, nfs-server |
| 88 |
* pcs or crmsh for pacemaker client. (checking the status of the iSCSI or nfs targets) |
| 89 |
|
| 90 |
[[s-linstor-gateway-checking]] |
| 91 |
=== Checking the Cluster |
| 92 |
|
| 93 |
Check the LINSTOR cluster status with; |
| 94 |
|
| 95 |
----------------- |
| 96 |
[root@LINSTOR1 ~]# linstor n l |
| 97 |
╭────────────────────────────────────────────────────────────╮ |
| 98 |
┊ Node ┊ NodeType ┊ Addresses ┊ State ┊ |
| 99 |
╞════════════════════════════════════════════════════════════╡ |
| 100 |
┊ LINSTOR1 ┊ COMBINED ┊ 172.16.16.111:3366 (PLAIN) ┊ Online ┊ |
| 101 |
┊ LINSTOR2 ┊ SATELLITE ┊ 172.16.16.112:3366 (PLAIN) ┊ Online ┊ |
| 102 |
┊ LINSTOR3 ┊ SATELLITE ┊ 172.16.16.113:3366 (PLAIN) ┊ Online ┊ |
| 103 |
╰────────────────────────────────────────────────────────────╯ |
| 104 |
----------------- |
| 105 |
Check the LINSTOR storage pool list with; |
| 106 |
|
| 107 |
----------------- |
| 108 |
root@LINSTOR1 ~]# linstor sp l |
| 109 |
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ |
| 110 |
┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ |
| 111 |
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════╡ |
| 112 |
┊ DfltDisklessStorPool ┊ LINSTOR1 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ |
| 113 |
┊ DfltDisklessStorPool ┊ LINSTOR2 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ |
| 114 |
┊ DfltDisklessStorPool ┊ LINSTOR3 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ |
| 115 |
┊ lvmpool ┊ LINSTOR1 ┊ LVM ┊ lvpool ┊ 10.00 GiB ┊ 10.00 GiB ┊ False ┊ Ok ┊ |
| 116 |
┊ lvmpool ┊ LINSTOR2 ┊ LVM ┊ lvpool ┊ 10.00 GiB ┊ 10.00 GiB ┊ False ┊ Ok ┊ |
| 117 |
┊ lvmpool ┊ LINSTOR3 ┊ LVM ┊ lvpool ┊ 10.00 GiB ┊ 10.00 GiB ┊ False ┊ Ok ┊ |
| 118 |
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ |
| 119 |
----------------- |
| 120 |
|
| 121 |
LINSTOR resource group list (please do not forget to create volume group for the resource group with: |
| 122 |
`linstor vg c iscsi_group`) |
| 123 |
|
| 124 |
----------------- |
| 125 |
[root@LINSTOR1 ~]# linstor rg l |
| 126 |
╭────────────────────────────────────────────────────────────────╮ |
| 127 |
┊ ResourceGroup ┊ SelectFilter ┊ VlmNrs ┊ Description ┊ |
| 128 |
╞════════════════════════════════════════════════════════════════╡ |
| 129 |
┊ DfltRscGrp ┊ PlaceCount: 2 ┊ ┊ ┊ |
| 130 |
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡ |
| 131 |
┊ iscsi_group ┊ PlaceCount: 2 ┊ 0 ┊ ┊ |
| 132 |
┊ ┊ StoragePool(s): lvmpool ┊ ┊ ┊ |
| 133 |
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡ |
| 134 |
┊ nfs_group ┊ PlaceCount: 3 ┊ 0 ┊ ┊ |
| 135 |
┊ ┊ StoragePool(s): lvmpool ┊ ┊ ┊ |
| 136 |
╰────────────────────────────────────────────────────────────────╯ |
| 137 |
|
| 138 |
----------------- |
| 139 |
|
| 140 |
Check and disable stonith. Stonith is a technique for fencing node in clusters. We have 3 nodes here so quorum |
| 141 |
will be used instead of fencing. |
| 142 |
|
| 143 |
----------------- |
| 144 |
pcs property set stonith-enabled=false |
| 145 |
pcs property show |
| 146 |
----------------- |
| 147 |
|
| 148 |
Check the pacemaker cluster health |
| 149 |
|
| 150 |
----------------- |
| 151 |
[root@LINSTOR1 ~]# pcs status |
| 152 |
Cluster name: LINSTOR |
| 153 |
Cluster Summary: |
| 154 |
* Stack: corosync |
| 155 |
* Current DC: LINSTOR1 (version 2.0.5.linbit-1.0.el7-ba59be712) - partition with quorum |
| 156 |
* Last updated: Wed Mar 24 21:24:10 2021 |
| 157 |
* Last change: Wed Mar 24 21:24:01 2021 by root via cibadmin on LINSTOR1 |
| 158 |
* 3 nodes configured |
| 159 |
* 0 resource instances configured |
| 160 |
|
| 161 |
Node List: |
| 162 |
* Online: [ LINSTOR1 LINSTOR2 LINSTOR3 ] |
| 163 |
|
| 164 |
Full List of Resources: |
| 165 |
* No resources |
| 166 |
|
| 167 |
Daemon Status: |
| 168 |
corosync: active/enabled |
| 169 |
pacemaker: active/enabled |
| 170 |
pcsd: active/enabled |
| 171 |
----------------- |
| 172 |
|
| 173 |
[[s-linstor-gateway-setup-iscsi]] |
| 174 |
=== Setting up iSCSI target |
| 175 |
|
| 176 |
|
| 177 |
Now, everything looks good, let's start creating our first iSCSI lun. linstor-iscsi tool will be used for all |
| 178 |
iSCSI related actions. Please check "linstor-iscsi help" for detailed usage. At first it creates a new |
| 179 |
resource within the LINSTOR system under the specified name and using the specified resource group. After that |
| 180 |
it creates resource primitives in the Pacemaker cluster including all necessary order and location |
| 181 |
constraints. The Pacemaker primitives are prefixed with `p_`, contain the resource name and a resource type |
| 182 |
postfix. |
| 183 |
|
| 184 |
------------------------ |
| 185 |
linstor-iscsi create --iqn=iqn.2021-04.com.linbit:lun4 --ip=172.16.16.101/24 --username=foo --lun=4 --password=bar --resource-group=iSCSI_group --size=1G |
| 186 |
------------------------ |
| 187 |
|
| 188 |
This command will create a 1G iSCSI disk with the provided username and password in the resource group defined |
| 189 |
`iSCSI_group` DRBD and pacemaker resources will be automatically created by linstor-iscsi. You can check the |
| 190 |
pacemaker resources with "pcs status" command. |
| 191 |
|
| 192 |
------------------------ |
| 193 |
[root@LINSTOR1 ~]# linstor-iscsi list |
| 194 |
+-----------------------------+-----+---------------+-----------+--------------+---------+ |
| 195 |
| IQN | LUN | Pacemaker LUN | Pacemaker | Pacemaker IP | LINSTOR | |
| 196 |
+-----------------------------+-----+---------------+-----------+--------------+---------+ |
| 197 |
| iqn.2020-06.com.linbit:lun4 | 4 | ✓ | ✓ | ✓ | ✓ | |
| 198 |
+-----------------------------+-----+---------------+-----------+--------------+---------+ |
| 199 |
------------------------ |
| 200 |
|
| 201 |
[[s-linstor-gateway-delete-iscsi]] |
| 202 |
=== Deleting iSCSI target |
| 203 |
|
| 204 |
The following command will delete the iSCSI target from pacemaker as well as the LINSTOR cluster; |
| 205 |
|
| 206 |
------------------------ |
| 207 |
linstor-iscsi delete -i iqn.2021-04.com.linbit:lun4 -l 4 |
| 208 |
------------------------ |
| 209 |
|
| 210 |
[[s-linstor-gateway-setup-nfs]] |
| 211 |
=== Setting up NFS export |
| 212 |
|
| 213 |
Before creating the nfs exports you need to tell LINSTOR that filesystem will be used for NFS exports will be |
| 214 |
EXT4. in order to do that, we'll apply a property to the resource group of NFS resources simply by typing; |
| 215 |
|
| 216 |
------------------------ |
| 217 |
linstor rg set-property nfs_group FileSystem/Type ext4 |
| 218 |
------------------------ |
| 219 |
|
| 220 |
The following command will create a NFS export in the cluster. At first it creates a new resource within the |
| 221 |
LINSTOR system under the specified name and using the specified resource group. After that it creates resource |
| 222 |
primitives in the Pacemaker cluster including all necessary order and location constraints. The Pacemaker |
| 223 |
primites are prefixed with p_, contain the resource name and a resource type postfix. |
| 224 |
|
| 225 |
------------------------ |
| 226 |
linstor-nfs create --resource=nfstest --service-ip=172.16.16.102/32 --allowed-ips=172.16.16.0/24 --resource-group=nfs_group --size=1G |
| 227 |
------------------------ |
| 228 |
|
| 229 |
You can simply list the nfs exports with the command below; |
| 230 |
|
| 231 |
------------------------ |
| 232 |
[root@LINSTOR1 ~]# LINSTOR-nfs list |
| 233 |
+---------------+------------------+-----------------------+------------+------------+ |
| 234 |
| Resource name | LINSTOR resource | Filesystem mountpoint | NFS export | Service IP | |
| 235 |
+---------------+------------------+-----------------------+------------+------------+ |
| 236 |
| nfstest | ✓ | ✓ | ✓ | ✓ | |
| 237 |
+---------------+------------------+-----------------------+------------+------------+ |
| 238 |
------------------------ |
| 239 |
|
| 240 |
[[s-linstor-gateway-delete-nfs]] |
| 241 |
=== Deleting NFS Export |
| 242 |
|
| 243 |
The following command will delete the nfs export from pacemaker as well as the LINSTOR cluster; |
| 244 |
|
| 245 |
------------------------ |
| 246 |
[root@LINSTOR1 ~]# linstor-nfs delete -r nfstest |
| 247 |
------------------------ |