Debian Codegraph @

UG9/en/nomad-linstor.adoc

raw
1 [[ch-nomad-linstor]]
2 == LINSTOR Volumes in Nomad
3
4 indexterm:[Nomad]
5
6 :consul-connect: https://www.nomadproject.io/docs/integrations/consul-integration#service-discovery
7
8 This chapter describes using LINSTOR and DRBD to provision volumes in Nomad.
9
10 [[s-nomad-linstor-overview]]
11 === Nomad Overview
12
13 https://www.nomadproject.io/[Nomad] is a simple and flexible workload orchestrator to deploy and manage containers
14 and non-containerized applications across on-prem and clouds.
15
16 Nomad supports provisioning storage volumes via https://www.nomadproject.io/docs/internals/plugins/csi[plugins]
17 conforming to the https://github.com/container-storage-interface/spec[Container Storage Interface (CSI)].
18
19 LINBIT distributes a CSI plugin in the form of container images from http://drbd.io[drbd.io]. The plugin can be
20 configured to work with a LINSTOR cluster that is deployed along or inside a Nomad cluster.
21
22 [[s-nomad-linstor-deployment]]
23 === Deploying LINSTOR on Nomad
24
25 This section describes how you can deploy and configure a new LINSTOR cluster in Nomad.
26
27 NOTE: If you want to install LINSTOR directly on your nodes, check out the
28 <<s-installation, guide on installing LINSTOR>>. You can skip this section and jump directly to
29 <<s-nomad-linstor-csi-deployment, deploying the CSI driver>>.
30
31 [[s-nomad-prepare]]
32 ==== Preparing Nomad
33
34 In order to run LINSTOR, every Nomad agent needs to be configured to:
35
36 - Support the https://www.nomadproject.io/docs/drivers/docker[docker driver] and allow executing privileged containers
37 +
38 To allow running privileged containers, add the following snippet to your Nomad agent configuration and restart Nomad
39 +
40 ./etc/nomad.d/docker-privileged.hcl
41 [source,hcl]
42 ----
43 plugin "docker" {
44 config {
45 allow_privileged = true
46 }
47 }
48 ----
49
50 - Support for container networking. If you don't have the https://www.cni.dev/[Container Network Interface] plugins
51 installed, you will only be able to use `mode = "host"` in your job networks. For most production setups, we recommend
52 installing the default plugins:
53 +
54 Head to the https://github.com/containernetworking/plugins/releases/[plugin release page], select the release archive
55 appropriate for your distribution and unpack them in `/opt/cni/bin`. You might need to create the directory before
56 unpacking.
57
58 - Provide a host volume, allowing a container access to the hosts `/dev` directory
59 +
60 To create a host volume, add the following snippet to your Nomad agent configuration and restart Nomad
61 +
62 ./etc/nomad.d/host-volume-dev.hcl
63 [source,hcl]
64 ----
65 client {
66 host_volume "dev" {
67 path = "/dev"
68 }
69 }
70 ----
71
72 ==== Create a LINSTOR Controller Job
73
74 The LINSTOR Controller is deployed as a service with no replicas. At any point in time, there can only be one LINSTOR
75 Controller running in a cluster. It is possible to restart the controller on a new node, provided it still has
76 access to the database. See <<s-linstor_ha>> for more information.
77
78 The following example will create a Nomad job starting a single LINSTOR Controller in datacenter `dc1` and connect
79 to an external database.
80
81 .linstor-controller.hcl
82 [source,hcl]
83 ----
84 job "linstor-controller" {
85 datacenters = ["dc1"] <1>
86 type = "service"
87
88 group "linstor-controller" {
89 network {
90 mode = "bridge"
91 # port "linstor-api" { <2>
92 # static = 3370
93 # to = 3370
94 # }
95 }
96
97 service { <3>
98 name = "linstor-api"
99 port = "3370"
100
101 connect {
102 sidecar_service {}
103 }
104
105 check {
106 expose = true
107 type = "http"
108 name = "api-health"
109 path = "/health"
110 interval = "30s"
111 timeout = "5s"
112 }
113 }
114
115 task "linstor-controller" {
116 driver = "docker"
117 config {
118 image = "drbd.io/linstor-controller:v1.13.0" <4>
119
120 auth { <5>
121 username = "example"
122 password = "example"
123 server_address = "drbd.io"
124 }
125
126 mount {
127 type = "bind"
128 source = "local"
129 target = "/etc/linstor"
130 }
131 }
132
133 # template { <6>
134 # destination = "local/linstor.toml"
135 # data = <<EOH
136 # [db]
137 # user = "example"
138 # password = "example"
139 # connection_url = "jdbc:postgresql://postgres.internal.example.com/linstor"
140 # EOH
141 # }
142
143 resources {
144 cpu = 500 # 500 MHz
145 memory = 700 # 700MB
146 }
147 }
148 }
149 }
150 ----
151
152 <1> Replace `dc1` with your own datacenter name
153
154 <2> This exposes the LINSTOR API on the host on port `3370`.
155 +
156 NOTE: Uncomment this section if your cluster is not configured with {consul-connect}[Consul Connect].
157
158 <3> The `service` block is used to expose the LINSTOR API to other jobs via the service mesh.
159 +
160 NOTE: If your cluster is not configured for {consul-connect}[Consul Connect] you can remove this section.
161
162 <4> This sets the LINSTOR Controller image to run. The latest images are available from http://drbd.io[drbd.io].
163 +
164 IMPORTANT: The use of the `:latest` tag is strongly discouraged, as it can quickly lead to version mismatches and
165 unintended upgrades.
166
167 <5> Sets the authentication to use when pulling the image. If pulling from `drbd.io`, you need to use your LINBIT
168 customer login here. Read more about pulling from a private repo
169 https://www.nomadproject.io/docs/drivers/docker#authentication[here].
170
171 <6> This template can be used to set arbitrary configuration options for LINSTOR. This example configures an external
172 database for LINSTOR. You can find a more detailed explanation of LINSTORs database options
173 <<s-linstor-external-database,here>> and more on Nomad templates
174 https://www.nomadproject.io/docs/job-specification/template#template-examples[here].
175
176 Apply the job by running:
177
178 [source,shell]
179 ----
180 $ nomad job run linstor-controller.hcl
181 ==> Monitoring evaluation "7d8911a7"
182 Evaluation triggered by job "linstor-controller"
183 ==> Monitoring evaluation "7d8911a7"
184 Evaluation within deployment: "436f4b2d"
185 Allocation "0b564c73" created: node "07754224", group "controller"
186 Evaluation status changed: "pending" -> "complete"
187 ==> Evaluation "7d8911a7" finished with status "complete"
188 ----
189
190 ===== Using a host volume for Linstors database
191
192 If you want to try LINSTOR without setting up an external database, you can make use of Linstors built-in filesystem
193 based database. To make the database persistent, you need to ensure it is placed on a host volume.
194
195 IMPORTANT: Using a host volume means that only a single node is able to run the LINSTOR Controller. If the node is
196 unavailable, the LINSTOR Cluster will also be unavailable. For alternatives, use an external (highly available) database
197 or deploy the <<s-linstor_ha,LINSTOR cluster directly on the hosts>>.
198
199 To create a host volume for the LINSTOR database, first create the directory on the host with the expected permissions
200
201 [source,shell]
202 ----
203 $ mkdir -p /var/lib/linstor
204 $ chown -R 1000:0 /var/lib/linstor
205 ----
206
207 Then add the following snippet to your Nomad agent configuration and restart Nomad
208
209 ./etc/nomad.d/host-volume-linstor-db.hcl
210 [source,hcl]
211 ----
212 client {
213 host_volume "linstor-db" {
214 path = "/var/lib/linstor"
215 }
216 }
217 ----
218
219 Then, add the following snippets to the `linstor-controller.hcl` example from above and adapt the `connection_url`
220 option from the configuration template.
221
222 .`job > group`
223 [source,hcl]
224 ----
225 volume "linstor-db" {
226 type = "host"
227 source = "linstor-db"
228 }
229 ----
230
231 .`job > group > task`
232 [source,hcl]
233 ----
234 volume_mount {
235 volume = "linstor-db"
236 destination = "/var/lib/linstor"
237 }
238
239 template {
240 destination = "local/linstor.toml"
241 data = <<EOH
242 [db]
243 user = "linstor"
244 password = "linstor"
245 connection_url = "jdbc:h2:/var/lib/linstor/linstordb"
246 EOH
247 }
248 ----
249
250 ==== Create a LINSTOR Satellite job
251
252 The LINSTOR Satellites are deployed as a system job in Nomad, running in a privileged container. In addition to the
253 satellites, the job will also load the DRBD module along with other kernel modules used by LINSTOR.
254
255 The following example will create a Nomad job starting a LINSTOR satellite on every node in datacenter `dc1`.
256
257 .linstor-satellite.hcl
258 [source,hcl]
259 ----
260 job "linstor-satellite" {
261 datacenters = ["dc1"] <1>
262 type = "system"
263
264 group "satellite" {
265 network {
266 mode = "host"
267 }
268
269 volume "dev" { <2>
270 type = "host"
271 source = "dev"
272 }
273
274 task "linstor-satellite" {
275 driver = "docker"
276
277 config {
278 image = "drbd.io/linstor-satellite:v1.13.0" <3>
279
280 auth { <4>
281 username = "example"
282 password = "example"
283 server_address = "drbd.io"
284 }
285
286 privileged = true <5>
287 network_mode = "host" <6>
288 }
289
290 volume_mount { <2>
291 volume = "dev"
292 destination = "/dev"
293 }
294
295 resources {
296 cpu = 500 # 500 MHz
297 memory = 500 # 500MB
298 }
299 }
300
301 task "drbd-loader" {
302 driver = "docker"
303 lifecycle {
304 hook = "prestart" <7>
305 }
306
307 config {
308 image = "drbd.io/drbd9-rhel8:v9.0.29" <8>
309
310 privileged = true <5>
311 auth { <4>
312 username = "example"
313 password = "example"
314 server_address = "drbd.io"
315 }
316 }
317
318 env {
319 LB_HOW = "shipped_modules" <9>
320 }
321
322 volume_mount { <10>
323 volume = "kernel-src"
324 destination = "/usr/src"
325 }
326 volume_mount { <10>
327 volume = "modules"
328 destination = "/lib/modules"
329 }
330 }
331
332 volume "modules" { <10>
333 type = "host"
334 source = "modules"
335 read_only = true
336 }
337
338 volume "kernel-src" { <10>
339 type = "host"
340 source = "kernel-src"
341 read_only = true
342 }
343 }
344 }
345 ----
346
347 <1> Replace `dc1` with your own datacenter name
348
349 <2> The `dev` host volume is the volume created in <<s-nomad-prepare>>, which allows the satellite to manage the hosts
350 block devices.
351
352 <3> This sets the LINSTOR Satellite image to run. The latest images are available from http://drbd.io[drbd.io]. The
353 satellite image version has to match the version of the controller image.
354 +
355 IMPORTANT: The use of the `:latest` tag is strongly discouraged, as it can quickly lead to version mismatches and
356 unintended upgrades.
357
358 <4> Sets the authentication to use when pulling the image. If pulling from `drbd.io`, you need to use your LINBIT
359 customer login here. Read more about pulling from a private repo
360 https://www.nomadproject.io/docs/drivers/docker#authentication[here].
361
362 <5> In order to configure storage devices, DRBD and load kernel modules, the containers need to be running in privileged
363 mode.
364
365 <6> The satellite needs to communicate with DRBD, which requires access to the netlink interface running in the hosts
366 network.
367
368 <7> The `drbd-loader` task will be executed once at the start of the satellite and load DRBD and other useful kernel
369 modules.
370
371 <8> The `drbd-loader` is specific to the distribution you are using. Available options are:
372 * `drbd.io/drbd9-bionic` for Ubuntu 18.04 (Bionic Beaver)
373 * `drbd.io/drbd9-focal` for Ubuntu 20.04 (Focal Fossa)
374 * `drbd.io/drbd9-rhel8` for RHEL 8
375 * `drbd.io/drbd9-rhel7` for RHEL 7
376
377 <9> The `drbd-loader` container can be configured via environment variables. `LB_HOW` tells the container how to insert
378 the DRBD kernel module. Available options are:
379 `shipped_modules`:: uses the prepackaged RPMs or DEBs delivered with the container.
380 `compile`:: Compile DRBD from source. Requires access to the kernel headers (see below).
381 `deps_only`:: Only try to load existing modules used by the LINSTOR satellite (for example `dm_thin_pool` and `dm_cache`).
382
383 <10> In order for the `drbd-loader` container to build DRBD or load existing modules, it needs access to a hosts
384 `/usr/src` and `/lib/modules` respectively.
385 +
386 This requires setting up additional host volumes on every node. The following snippet needs to be added to every Nomad
387 agent confiration, then Nomad needs to be restarted.
388 +
389 ./etc/nomad.d/drbd-loader-volumes.hcl
390 [source,hcl]
391 ----
392 client {
393 host_volume "modules" {
394 path = "/lib/modules"
395 read_only = true
396 }
397 host_volume "kernel-src" {
398 path = "/usr/src"
399 read_only = true
400 }
401 }
402 ----
403
404 Apply the job by running:
405
406 [source,shell]
407 ----
408 $ nomad job run linstor-satellite.hcl
409 ==> Monitoring evaluation "0c07469d"
410 Evaluation triggered by job "linstor-satellite"
411 ==> Monitoring evaluation "0c07469d"
412 Evaluation status changed: "pending" -> "complete"
413 ==> Evaluation "0c07469d" finished with status "complete"
414 ----
415
416 ==== Configuring LINSTOR in Nomad
417
418 Once the `linstor-controller` and `linstor-satellite` jobs are running, you can start configuring the cluster using
419 the `linstor` command line tool.
420
421 This can done:
422
423 * directly by `nomad exec`-ing into the `linstor-controller` container
424 * using the `drbd.io/linstor-client` container on the host running the `linstor-controller`
425 +
426 ----
427 docker run -it --rm --net=host drbd.io/linstor-client node create
428 ----
429 * by installing the `linstor-client` package on the host running the `linstor-controller`.
430
431 In all cases, you need to <<s-adding_nodes_to_your_cluster,add the satellites to your cluster>> and
432 <<s-storage_pools,create some storage pools>>. For example, to add the node `nomad-01.example.com` and configure
433 a LVM Thin storage pool, you would run:
434
435 [source,shell]
436 ----
437 $ linstor node create nomad-01.example.com
438 $ linstor storage-pool create lvmthin nomad-01.example.com thinpool linstor_vg/thinpool
439 ----
440
441 IMPORTANT: The CSI driver requires your satellites to be named after their hostname. To be precise, the satellite name
442 needs to match Nomads `attr.unique.hostname` attribute on the node.
443
444 [[s-nomad-linstor-csi-deployment]]
445 === Deploying the LINSTOR CSI Driver on Nomad
446
447 The CSI driver is deployed as a system job, meaning it runs on every node in the cluster.
448
449 The following example will create a Nomad job starting a LINSTOR CSI Driver on every node in datacenter `dc1`.
450
451 .linstor-csi.hcl
452 [source,hcl]
453 ----
454 job "linstor-csi" {
455 datacenters = ["dc1"] <1>
456 type = "system"
457
458 group "csi" {
459 network {
460 mode = "bridge"
461 }
462
463 service {
464 connect {
465 sidecar_service { <2>
466 proxy {
467 upstreams {
468 destination_name = "linstor-api"
469 local_bind_port = 8080
470 }
471 }
472 }
473 }
474 }
475
476 task "csi-plugin" {
477 driver = "docker"
478 config {
479 image = "drbd.io/linstor-csi:v0.13.1" <3>
480
481 auth { <4>
482 username = "example"
483 password = "example"
484 server_address = "drbd.io"
485 }
486
487 args = [
488 "--csi-endpoint=unix://csi/csi.sock",
489 "--node=${attr.unique.hostname}", <5>
490 "--linstor-endpoint=http://${NOMAD_UPSTREAM_ADDR_linstor_api}", <6>
491 "--log-level=info"
492 ]
493
494 privileged = true <7>
495 }
496
497 csi_plugin { <8>
498 id = "linstor.csi.linbit.com"
499 type = "monolith"
500 mount_dir = "/csi"
501 }
502
503 resources {
504 cpu = 100 # 100 MHz
505 memory = 200 # 200MB
506 }
507 }
508 }
509 }
510 ----
511
512 <1> Replace `dc1` with your own datacenter name
513
514 <2> The `sidecar_service` stanza enables use of the service mesh generated by using {consul-connect}[Consul Connect].
515 If you have not configured this feature in Nomad, or you are using an external LINSTOR Controller,
516 you can skip this configuration.
517
518 <3> This sets the LINSTOR CSI Driver image to run. The latest images are available from http://drbd.io[drbd.io].
519 +
520 IMPORTANT: The use of the `:latest` tag is strongly discouraged, as it can quickly lead to version mismatches and
521 unintended upgrades.
522
523 <4> Sets the authentication to use when pulling the image. If pulling from `drbd.io`, you need to use your LINBIT
524 customer login here. Read more about pulling from a private repo
525 https://www.nomadproject.io/docs/drivers/docker#authentication[here].
526
527 <5> This argument sets the node name used by the CSI driver to identify itself in the LINSTOR API. By default, this
528 is set to the nodes hostname.
529
530 <6> This argument sets the LINSTOR API endpoint. If you are not using the consul service mesh (see Nr. 2 above), this
531 needs to be set to the Controllers API endpoint. The endpoint needs to be reachable from every node this is deployed on.
532
533 <7> The CSI driver needs to execute mount commands, requiring privileged containers.
534
535 <8> The `csi_plugin` stanza informs Nomad that this task is a CSI plugin. The Nomad agent will forward requests for
536 volumes to one of the jobs containers.
537
538 Apply the job by running:
539
540 [source,shell]
541 ----
542 $ nomad job run linstor-csi.hcl
543 ==> Monitoring evaluation "0119f19c"
544 Evaluation triggered by job "linstor-csi"
545 ==> Monitoring evaluation "0119f19c"
546 Evaluation status changed: "pending" -> "complete"
547 ==> Evaluation "0119f19c" finished with status "complete"
548 ----
549
550 === Using LINSTOR volumes in Nomad
551
552 Volumes in Nomad are created using a https://www.nomadproject.io/docs/commands/volume/create#volume-specification[volume-specification].
553
554 As an example, the following specification requests a 1GiB volume with 2 replicas from the LINSTOR storage pool `thinpool`.
555
556 .vol1.hcl
557 [source,hcl]
558 ----
559 id = "vol1" <1>
560 name = "vol1" <2>
561
562 type = "csi"
563 plugin_id = "linstor.csi.linbit.com"
564
565 capacity_min = "1GiB"
566 capacity_max = "1GiB"
567
568 capability {
569 access_mode = "single-node-writer" <3>
570 attachment_mode = "file-system" <4>
571 }
572
573 mount_options {
574 fs_type = "ext4" <5>
575 }
576
577 parameters { <6>
578 "resourceGroup" = "default-resource",
579 "storagePool" = "thinpool",
580 "autoPlace" = "2"
581 }
582 ----
583
584 <1> The `id` is used to reference this volume in Nomad. Used in the `volume.source` field of a job specification.
585
586 <2> The `name` is used when creating the volume in the backend (i.e. LINSTOR). Ideally this matches the `id` and is a
587 valid LINSTOR resource name. If the name would not be valid, LINSTOR CSI will generate a random compatible name.
588
589 <3> What kind of access the volume should support. LINSTOR CSI supports:
590 `single-node-reader-only`::Allow read only access on one node at a time
591 `single-node-writer`:: Allow read and write access on one node at a time
592 `multi-node-reader-only`:: Allow read only access from multiple nodes.
593
594 <4> Can be `file-system` or `block-device`.
595
596 <5> Specify the file system to use. LINSTOR CSI supports `ext4` and `xfs`.
597
598 <6> Additional parameters to pass to LINSTOR CSI. The example above requests the resource be part of the
599 `default-resource` <<s-linstor-resource-groups,resource group>> and should deploy 2 replicas.
600 +
601 For a complete list of available parameters, you can check out the
602 <<s-kubernetes-sc-parameters,guide on Kubernetes storage classes>>. Kubernetes, like Nomad, makes use of the CSI plugin.
603
604 To create the volume, run the following command:
605
606 [source,shell]
607 ----
608 $ nomad volume create vol1.hcl
609 Created external volume vol1 with ID vol1
610 $ nomad volume status
611 Container Storage Interface
612 ID Name Plugin ID Schedulable Access Mode
613 vol1 vol1 linstor.csi.linbit.com true <none>
614 $ linstor resource list
615 ╭──────────────────────────────────────────────────────────────────────────────────────────────╮
616 ┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
617 ╞══════════════════════════════════════════════════════════════════════════════════════════════╡
618 ┊ vol1 ┊ nomad-01.example.com ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2021-06-15 14:56:32 ┊
619 ┊ vol1 ┊ nomad-02.example.com ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2021-06-15 14:56:32 ┊
620 ╰──────────────────────────────────────────────────────────────────────────────────────────────╯
621 ----
622
623 ==== Using volumes in jobs
624
625 To use the volume in a job, add the `volume` and `volume_mount` stanzas to the job specification:
626
627 [source,hcl]
628 ----
629 job "example" {
630 ...
631
632 group "example" {
633 volume "example-vol" {
634 type = "csi"
635 source = "vol1"
636 attachment_mode = "file-system"
637 access_mode = "single-node-writer"
638 }
639
640 task "mount-example" {
641 volume_mount {
642 volume = "example-vol"
643 destination = "/data"
644 }
645
646 ...
647 }
648 }
649 }
650 ----
651
652 ==== Creating snapshots of volumes
653
654 LINSTOR can create snapshots of existing volumes, provided the underlying storage pool driver supports snapshots.
655
656 The following command creates a snapshot named `snap1` of the volume `vol1`.
657
658 [source,shell]
659 ----
660 $ nomad volume snapshot create vol1 snap1
661 Snapshot ID Volume ID Size Create Time Ready?
662 snap1 vol1 1.0 GiB None true
663 $ linstor s l
664 ╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
665 ┊ ResourceName ┊ SnapshotName ┊ NodeNames ┊ Volumes ┊ CreatedOn ┊ State ┊
666 ╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
667 ┊ vol1 ┊ snap1 ┊ nomad-01.example.com, nomad-02.example.com ┊ 0: 1 GiB ┊ 2021-06-15 15:04:10 ┊ Successful ┊
668 ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
669 ----
670
671 You can use a snapshot to pre-populate an existing volume with data from the snapshot
672
673 [source,shell]
674 ----
675 $ cat vol2.hcl
676 id = "vol2"
677 name = "vol2"
678 snapshot_id = "snap1"
679
680 type = "csi"
681 plugin_id = "linstor.csi.linbit.com"
682 ...
683
684 $ nomad volume create vol2.hcl
685 Created external volume vol2 with ID vol2
686 ----