Debian Codegraph @

UG9/en/linstor-gateway.adoc

raw
1 [[ch-linstor-gateway]]
2 == Exporting Highly Available Storage using LINSTOR Gateway
3
4 *LINSTOR Gateway* manages highly available iSCSI targets and NFS exports by leveraging on LINSTOR and
5 Pacemaker. Setting up LINSTOR - including storage pools and resource groups - as well as Corosync and
6 Pacemaker's properties are a prerequisite to use this tool.
7
8 [[s-linstor-gateway-requirements]]
9 === Requirements
10
11 ==== LINSTOR
12
13 A LINSTOR cluster is required to operate LINSTOR Gateway. It is *highly recommended* to run the LINSTOR
14 controller as a Pacemaker resource. This needs to be configured manually. Such a resource could look like the
15 following: (optional)
16
17 -----------------
18 primitive p_linstor-controller systemd:linstor-controller \
19 op start interval=0 timeout=100s \
20 op stop interval=0 timeout=100s \
21 op monitor interval=30s timeout=100s
22 -----------------
23
24 For both iSCSI and NFS, `storage pool` , `resource group` and a `volume group` for LINSTOR Gateway needs to be
25 present. Lets do it;
26
27 Creating the storage pool in 3 nodes using the phsical device `/dev/sdb`;
28
29 -----------------
30 linstor physical-storage create-device-pool --pool-name lvpool LVM LINSTOR1 /dev/sdb --storage-pool lvmpool
31 linstor physical-storage create-device-pool --pool-name lvpool LVM LINSTOR2 /dev/sdb --storage-pool lvmpool
32 linstor physical-storage create-device-pool --pool-name lvpool LVM LINSTOR3 /dev/sdb --storage-pool lvmpool
33 -----------------
34
35 We also need resource groups and volume groups;
36
37 -----------------
38 linstor rg c iSCSI_group --storage-pool lvmpool --place-count 2
39 linstor rg c nfs_group --storage-pool lvmpool --place-count 3
40 -----------------
41
42 -----------------
43 linstor vg c iSCSI_group
44 linstor vg c nfs_group
45 -----------------
46
47 For a more detailed explanation of the storage pool, resource group and volume group creation, check the
48 https://linbit.com/drbd-user-guide/LINSTOR-guide-1_0-en/#s-storage_pools[LINSTOR user guide]
49
50 ==== Pacemaker
51
52 A working Corosync/Pacemaker cluster is expected on the machine where LINSTOR Gateway is running.
53 The https://github.com/LINBIT/drbd-utils/blob/master/scripts/drbd-attr[drbd-attr] resource agent is required
54 to run LINSTOR Gateway. This is included in LINBIT's `drbd-utils` package for Ubuntu based distributions, or
55 the drbd-pacemaker package on RHEL/CentOS. LINSTOR Gateway sets up all required Pacemaker resource and
56 constraints by itself, except for the LINSTOR controller resource.
57
58 ==== iSCSI & NFS
59
60 LINSTOR Gateway uses Pacemaker's `ocf::heartbeat:iSCSITarget` resource agent for its iSCSI integration, which
61 requires an iSCSI implementation to be installed. Using `targetcli` is recommended.
62
63 For iSCSI, please install targetcli.
64
65 -----------------
66 yum install targetcli
67 -----------------
68
69 For nfs, nfs-server needs to be enabled and ready;
70
71 -----------------
72 systemctl enable --now nfs-server
73 -----------------
74
75 [[s-linstor-gateway-preparation]]
76 === Preparation
77 First, let's check that all the components are available. This guide assumes you already installed and
78 configured a LINSTOR cluster. Volume Group, Storage Pool and Resource Group should be defined before using
79 linstor-iscsi or linstor-nfs.
80
81 Tools need to be present in server;
82
83 * linstor-client (managing the LINSTOR cluster)
84 * drbd-attr resource agent (part of drbd-utils in Debian/Ubuntu, and part of drbd-pacemaker for other
85 Linux distrubutions)
86 * targetcli (for iSCSI)
87 * nfs-utils, nfs-server
88 * pcs or crmsh for pacemaker client. (checking the status of the iSCSI or nfs targets)
89
90 [[s-linstor-gateway-checking]]
91 === Checking the Cluster
92
93 Check the LINSTOR cluster status with;
94
95 -----------------
96 [root@LINSTOR1 ~]# linstor n l
97 ╭────────────────────────────────────────────────────────────╮
98 ┊ Node ┊ NodeType ┊ Addresses ┊ State ┊
99 ╞════════════════════════════════════════════════════════════╡
100 ┊ LINSTOR1 ┊ COMBINED ┊ 172.16.16.111:3366 (PLAIN) ┊ Online ┊
101 ┊ LINSTOR2 ┊ SATELLITE ┊ 172.16.16.112:3366 (PLAIN) ┊ Online ┊
102 ┊ LINSTOR3 ┊ SATELLITE ┊ 172.16.16.113:3366 (PLAIN) ┊ Online ┊
103 ╰────────────────────────────────────────────────────────────╯
104 -----------------
105 Check the LINSTOR storage pool list with;
106
107 -----------------
108 root@LINSTOR1 ~]# linstor sp l
109 ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
110 ┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊
111 ╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
112 ┊ DfltDisklessStorPool ┊ LINSTOR1 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
113 ┊ DfltDisklessStorPool ┊ LINSTOR2 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
114 ┊ DfltDisklessStorPool ┊ LINSTOR3 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
115 ┊ lvmpool ┊ LINSTOR1 ┊ LVM ┊ lvpool ┊ 10.00 GiB ┊ 10.00 GiB ┊ False ┊ Ok ┊
116 ┊ lvmpool ┊ LINSTOR2 ┊ LVM ┊ lvpool ┊ 10.00 GiB ┊ 10.00 GiB ┊ False ┊ Ok ┊
117 ┊ lvmpool ┊ LINSTOR3 ┊ LVM ┊ lvpool ┊ 10.00 GiB ┊ 10.00 GiB ┊ False ┊ Ok ┊
118 ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
119 -----------------
120
121 LINSTOR resource group list (please do not forget to create volume group for the resource group with:
122 `linstor vg c iscsi_group`)
123
124 -----------------
125 [root@LINSTOR1 ~]# linstor rg l
126 ╭────────────────────────────────────────────────────────────────╮
127 ┊ ResourceGroup ┊ SelectFilter ┊ VlmNrs ┊ Description ┊
128 ╞════════════════════════════════════════════════════════════════╡
129 ┊ DfltRscGrp ┊ PlaceCount: 2 ┊ ┊ ┊
130 ╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
131 ┊ iscsi_group ┊ PlaceCount: 2 ┊ 0 ┊ ┊
132 ┊ ┊ StoragePool(s): lvmpool ┊ ┊ ┊
133 ╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
134 ┊ nfs_group ┊ PlaceCount: 3 ┊ 0 ┊ ┊
135 ┊ ┊ StoragePool(s): lvmpool ┊ ┊ ┊
136 ╰────────────────────────────────────────────────────────────────╯
137
138 -----------------
139
140 Check and disable stonith. Stonith is a technique for fencing node in clusters. We have 3 nodes here so quorum
141 will be used instead of fencing.
142
143 -----------------
144 pcs property set stonith-enabled=false
145 pcs property show
146 -----------------
147
148 Check the pacemaker cluster health
149
150 -----------------
151 [root@LINSTOR1 ~]# pcs status
152 Cluster name: LINSTOR
153 Cluster Summary:
154 * Stack: corosync
155 * Current DC: LINSTOR1 (version 2.0.5.linbit-1.0.el7-ba59be712) - partition with quorum
156 * Last updated: Wed Mar 24 21:24:10 2021
157 * Last change: Wed Mar 24 21:24:01 2021 by root via cibadmin on LINSTOR1
158 * 3 nodes configured
159 * 0 resource instances configured
160
161 Node List:
162 * Online: [ LINSTOR1 LINSTOR2 LINSTOR3 ]
163
164 Full List of Resources:
165 * No resources
166
167 Daemon Status:
168 corosync: active/enabled
169 pacemaker: active/enabled
170 pcsd: active/enabled
171 -----------------
172
173 [[s-linstor-gateway-setup-iscsi]]
174 === Setting up iSCSI target
175
176
177 Now, everything looks good, let's start creating our first iSCSI lun. linstor-iscsi tool will be used for all
178 iSCSI related actions. Please check "linstor-iscsi help" for detailed usage. At first it creates a new
179 resource within the LINSTOR system under the specified name and using the specified resource group. After that
180 it creates resource primitives in the Pacemaker cluster including all necessary order and location
181 constraints. The Pacemaker primitives are prefixed with `p_`, contain the resource name and a resource type
182 postfix.
183
184 ------------------------
185 linstor-iscsi create --iqn=iqn.2021-04.com.linbit:lun4 --ip=172.16.16.101/24 --username=foo --lun=4 --password=bar --resource-group=iSCSI_group --size=1G
186 ------------------------
187
188 This command will create a 1G iSCSI disk with the provided username and password in the resource group defined
189 `iSCSI_group` DRBD and pacemaker resources will be automatically created by linstor-iscsi. You can check the
190 pacemaker resources with "pcs status" command.
191
192 ------------------------
193 [root@LINSTOR1 ~]# linstor-iscsi list
194 +-----------------------------+-----+---------------+-----------+--------------+---------+
195 | IQN | LUN | Pacemaker LUN | Pacemaker | Pacemaker IP | LINSTOR |
196 +-----------------------------+-----+---------------+-----------+--------------+---------+
197 | iqn.2020-06.com.linbit:lun4 | 4 | ✓ | ✓ | ✓ | ✓ |
198 +-----------------------------+-----+---------------+-----------+--------------+---------+
199 ------------------------
200
201 [[s-linstor-gateway-delete-iscsi]]
202 === Deleting iSCSI target
203
204 The following command will delete the iSCSI target from pacemaker as well as the LINSTOR cluster;
205
206 ------------------------
207 linstor-iscsi delete -i iqn.2021-04.com.linbit:lun4 -l 4
208 ------------------------
209
210 [[s-linstor-gateway-setup-nfs]]
211 === Setting up NFS export
212
213 Before creating the nfs exports you need to tell LINSTOR that filesystem will be used for NFS exports will be
214 EXT4. in order to do that, we'll apply a property to the resource group of NFS resources simply by typing;
215
216 ------------------------
217 linstor rg set-property nfs_group FileSystem/Type ext4
218 ------------------------
219
220 The following command will create a NFS export in the cluster. At first it creates a new resource within the
221 LINSTOR system under the specified name and using the specified resource group. After that it creates resource
222 primitives in the Pacemaker cluster including all necessary order and location constraints. The Pacemaker
223 primites are prefixed with p_, contain the resource name and a resource type postfix.
224
225 ------------------------
226 linstor-nfs create --resource=nfstest --service-ip=172.16.16.102/32 --allowed-ips=172.16.16.0/24 --resource-group=nfs_group --size=1G
227 ------------------------
228
229 You can simply list the nfs exports with the command below;
230
231 ------------------------
232 [root@LINSTOR1 ~]# LINSTOR-nfs list
233 +---------------+------------------+-----------------------+------------+------------+
234 | Resource name | LINSTOR resource | Filesystem mountpoint | NFS export | Service IP |
235 +---------------+------------------+-----------------------+------------+------------+
236 | nfstest | ✓ | ✓ | ✓ | ✓ |
237 +---------------+------------------+-----------------------+------------+------------+
238 ------------------------
239
240 [[s-linstor-gateway-delete-nfs]]
241 === Deleting NFS Export
242
243 The following command will delete the nfs export from pacemaker as well as the LINSTOR cluster;
244
245 ------------------------
246 [root@LINSTOR1 ~]# linstor-nfs delete -r nfstest
247 ------------------------