[[ch-nomad-linstor]] == LINSTOR Volumes in Nomad indexterm:[Nomad] :consul-connect: https://www.nomadproject.io/docs/integrations/consul-integration#service-discovery This chapter describes using LINSTOR and DRBD to provision volumes in Nomad. [[s-nomad-linstor-overview]] === Nomad Overview https://www.nomadproject.io/[Nomad] is a simple and flexible workload orchestrator to deploy and manage containers and non-containerized applications across on-prem and clouds. Nomad supports provisioning storage volumes via https://www.nomadproject.io/docs/internals/plugins/csi[plugins] conforming to the https://github.com/container-storage-interface/spec[Container Storage Interface (CSI)]. LINBIT distributes a CSI plugin in the form of container images from http://drbd.io[drbd.io]. The plugin can be configured to work with a LINSTOR cluster that is deployed along or inside a Nomad cluster. [[s-nomad-linstor-deployment]] === Deploying LINSTOR on Nomad This section describes how you can deploy and configure a new LINSTOR cluster in Nomad. NOTE: If you want to install LINSTOR directly on your nodes, check out the <>. You can skip this section and jump directly to <>. [[s-nomad-prepare]] ==== Preparing Nomad In order to run LINSTOR, every Nomad agent needs to be configured to: - Support the https://www.nomadproject.io/docs/drivers/docker[docker driver] and allow executing privileged containers + To allow running privileged containers, add the following snippet to your Nomad agent configuration and restart Nomad + ./etc/nomad.d/docker-privileged.hcl [source,hcl] ---- plugin "docker" { config { allow_privileged = true } } ---- - Support for container networking. If you don't have the https://www.cni.dev/[Container Network Interface] plugins installed, you will only be able to use `mode = "host"` in your job networks. For most production setups, we recommend installing the default plugins: + Head to the https://github.com/containernetworking/plugins/releases/[plugin release page], select the release archive appropriate for your distribution and unpack them in `/opt/cni/bin`. You might need to create the directory before unpacking. - Provide a host volume, allowing a container access to the hosts `/dev` directory + To create a host volume, add the following snippet to your Nomad agent configuration and restart Nomad + ./etc/nomad.d/host-volume-dev.hcl [source,hcl] ---- client { host_volume "dev" { path = "/dev" } } ---- ==== Create a LINSTOR Controller Job The LINSTOR Controller is deployed as a service with no replicas. At any point in time, there can only be one LINSTOR Controller running in a cluster. It is possible to restart the controller on a new node, provided it still has access to the database. See <> for more information. The following example will create a Nomad job starting a single LINSTOR Controller in datacenter `dc1` and connect to an external database. .linstor-controller.hcl [source,hcl] ---- job "linstor-controller" { datacenters = ["dc1"] <1> type = "service" group "linstor-controller" { network { mode = "bridge" # port "linstor-api" { <2> # static = 3370 # to = 3370 # } } service { <3> name = "linstor-api" port = "3370" connect { sidecar_service {} } check { expose = true type = "http" name = "api-health" path = "/health" interval = "30s" timeout = "5s" } } task "linstor-controller" { driver = "docker" config { image = "drbd.io/linstor-controller:v1.13.0" <4> auth { <5> username = "example" password = "example" server_address = "drbd.io" } mount { type = "bind" source = "local" target = "/etc/linstor" } } # template { <6> # destination = "local/linstor.toml" # data = < Replace `dc1` with your own datacenter name <2> This exposes the LINSTOR API on the host on port `3370`. + NOTE: Uncomment this section if your cluster is not configured with {consul-connect}[Consul Connect]. <3> The `service` block is used to expose the LINSTOR API to other jobs via the service mesh. + NOTE: If your cluster is not configured for {consul-connect}[Consul Connect] you can remove this section. <4> This sets the LINSTOR Controller image to run. The latest images are available from http://drbd.io[drbd.io]. + IMPORTANT: The use of the `:latest` tag is strongly discouraged, as it can quickly lead to version mismatches and unintended upgrades. <5> Sets the authentication to use when pulling the image. If pulling from `drbd.io`, you need to use your LINBIT customer login here. Read more about pulling from a private repo https://www.nomadproject.io/docs/drivers/docker#authentication[here]. <6> This template can be used to set arbitrary configuration options for LINSTOR. This example configures an external database for LINSTOR. You can find a more detailed explanation of LINSTORs database options <> and more on Nomad templates https://www.nomadproject.io/docs/job-specification/template#template-examples[here]. Apply the job by running: [source,shell] ---- $ nomad job run linstor-controller.hcl ==> Monitoring evaluation "7d8911a7" Evaluation triggered by job "linstor-controller" ==> Monitoring evaluation "7d8911a7" Evaluation within deployment: "436f4b2d" Allocation "0b564c73" created: node "07754224", group "controller" Evaluation status changed: "pending" -> "complete" ==> Evaluation "7d8911a7" finished with status "complete" ---- ===== Using a host volume for Linstors database If you want to try LINSTOR without setting up an external database, you can make use of Linstors built-in filesystem based database. To make the database persistent, you need to ensure it is placed on a host volume. IMPORTANT: Using a host volume means that only a single node is able to run the LINSTOR Controller. If the node is unavailable, the LINSTOR Cluster will also be unavailable. For alternatives, use an external (highly available) database or deploy the <>. To create a host volume for the LINSTOR database, first create the directory on the host with the expected permissions [source,shell] ---- $ mkdir -p /var/lib/linstor $ chown -R 1000:0 /var/lib/linstor ---- Then add the following snippet to your Nomad agent configuration and restart Nomad ./etc/nomad.d/host-volume-linstor-db.hcl [source,hcl] ---- client { host_volume "linstor-db" { path = "/var/lib/linstor" } } ---- Then, add the following snippets to the `linstor-controller.hcl` example from above and adapt the `connection_url` option from the configuration template. .`job > group` [source,hcl] ---- volume "linstor-db" { type = "host" source = "linstor-db" } ---- .`job > group > task` [source,hcl] ---- volume_mount { volume = "linstor-db" destination = "/var/lib/linstor" } template { destination = "local/linstor.toml" data = < type = "system" group "satellite" { network { mode = "host" } volume "dev" { <2> type = "host" source = "dev" } task "linstor-satellite" { driver = "docker" config { image = "drbd.io/linstor-satellite:v1.13.0" <3> auth { <4> username = "example" password = "example" server_address = "drbd.io" } privileged = true <5> network_mode = "host" <6> } volume_mount { <2> volume = "dev" destination = "/dev" } resources { cpu = 500 # 500 MHz memory = 500 # 500MB } } task "drbd-loader" { driver = "docker" lifecycle { hook = "prestart" <7> } config { image = "drbd.io/drbd9-rhel8:v9.0.29" <8> privileged = true <5> auth { <4> username = "example" password = "example" server_address = "drbd.io" } } env { LB_HOW = "shipped_modules" <9> } volume_mount { <10> volume = "kernel-src" destination = "/usr/src" } volume_mount { <10> volume = "modules" destination = "/lib/modules" } } volume "modules" { <10> type = "host" source = "modules" read_only = true } volume "kernel-src" { <10> type = "host" source = "kernel-src" read_only = true } } } ---- <1> Replace `dc1` with your own datacenter name <2> The `dev` host volume is the volume created in <>, which allows the satellite to manage the hosts block devices. <3> This sets the LINSTOR Satellite image to run. The latest images are available from http://drbd.io[drbd.io]. The satellite image version has to match the version of the controller image. + IMPORTANT: The use of the `:latest` tag is strongly discouraged, as it can quickly lead to version mismatches and unintended upgrades. <4> Sets the authentication to use when pulling the image. If pulling from `drbd.io`, you need to use your LINBIT customer login here. Read more about pulling from a private repo https://www.nomadproject.io/docs/drivers/docker#authentication[here]. <5> In order to configure storage devices, DRBD and load kernel modules, the containers need to be running in privileged mode. <6> The satellite needs to communicate with DRBD, which requires access to the netlink interface running in the hosts network. <7> The `drbd-loader` task will be executed once at the start of the satellite and load DRBD and other useful kernel modules. <8> The `drbd-loader` is specific to the distribution you are using. Available options are: * `drbd.io/drbd9-bionic` for Ubuntu 18.04 (Bionic Beaver) * `drbd.io/drbd9-focal` for Ubuntu 20.04 (Focal Fossa) * `drbd.io/drbd9-rhel8` for RHEL 8 * `drbd.io/drbd9-rhel7` for RHEL 7 <9> The `drbd-loader` container can be configured via environment variables. `LB_HOW` tells the container how to insert the DRBD kernel module. Available options are: `shipped_modules`:: uses the prepackaged RPMs or DEBs delivered with the container. `compile`:: Compile DRBD from source. Requires access to the kernel headers (see below). `deps_only`:: Only try to load existing modules used by the LINSTOR satellite (for example `dm_thin_pool` and `dm_cache`). <10> In order for the `drbd-loader` container to build DRBD or load existing modules, it needs access to a hosts `/usr/src` and `/lib/modules` respectively. + This requires setting up additional host volumes on every node. The following snippet needs to be added to every Nomad agent confiration, then Nomad needs to be restarted. + ./etc/nomad.d/drbd-loader-volumes.hcl [source,hcl] ---- client { host_volume "modules" { path = "/lib/modules" read_only = true } host_volume "kernel-src" { path = "/usr/src" read_only = true } } ---- Apply the job by running: [source,shell] ---- $ nomad job run linstor-satellite.hcl ==> Monitoring evaluation "0c07469d" Evaluation triggered by job "linstor-satellite" ==> Monitoring evaluation "0c07469d" Evaluation status changed: "pending" -> "complete" ==> Evaluation "0c07469d" finished with status "complete" ---- ==== Configuring LINSTOR in Nomad Once the `linstor-controller` and `linstor-satellite` jobs are running, you can start configuring the cluster using the `linstor` command line tool. This can done: * directly by `nomad exec`-ing into the `linstor-controller` container * using the `drbd.io/linstor-client` container on the host running the `linstor-controller` + ---- docker run -it --rm --net=host drbd.io/linstor-client node create ---- * by installing the `linstor-client` package on the host running the `linstor-controller`. In all cases, you need to <> and <>. For example, to add the node `nomad-01.example.com` and configure a LVM Thin storage pool, you would run: [source,shell] ---- $ linstor node create nomad-01.example.com $ linstor storage-pool create lvmthin nomad-01.example.com thinpool linstor_vg/thinpool ---- IMPORTANT: The CSI driver requires your satellites to be named after their hostname. To be precise, the satellite name needs to match Nomads `attr.unique.hostname` attribute on the node. [[s-nomad-linstor-csi-deployment]] === Deploying the LINSTOR CSI Driver on Nomad The CSI driver is deployed as a system job, meaning it runs on every node in the cluster. The following example will create a Nomad job starting a LINSTOR CSI Driver on every node in datacenter `dc1`. .linstor-csi.hcl [source,hcl] ---- job "linstor-csi" { datacenters = ["dc1"] <1> type = "system" group "csi" { network { mode = "bridge" } service { connect { sidecar_service { <2> proxy { upstreams { destination_name = "linstor-api" local_bind_port = 8080 } } } } } task "csi-plugin" { driver = "docker" config { image = "drbd.io/linstor-csi:v0.13.1" <3> auth { <4> username = "example" password = "example" server_address = "drbd.io" } args = [ "--csi-endpoint=unix://csi/csi.sock", "--node=${attr.unique.hostname}", <5> "--linstor-endpoint=http://${NOMAD_UPSTREAM_ADDR_linstor_api}", <6> "--log-level=info" ] privileged = true <7> } csi_plugin { <8> id = "linstor.csi.linbit.com" type = "monolith" mount_dir = "/csi" } resources { cpu = 100 # 100 MHz memory = 200 # 200MB } } } } ---- <1> Replace `dc1` with your own datacenter name <2> The `sidecar_service` stanza enables use of the service mesh generated by using {consul-connect}[Consul Connect]. If you have not configured this feature in Nomad, or you are using an external LINSTOR Controller, you can skip this configuration. <3> This sets the LINSTOR CSI Driver image to run. The latest images are available from http://drbd.io[drbd.io]. + IMPORTANT: The use of the `:latest` tag is strongly discouraged, as it can quickly lead to version mismatches and unintended upgrades. <4> Sets the authentication to use when pulling the image. If pulling from `drbd.io`, you need to use your LINBIT customer login here. Read more about pulling from a private repo https://www.nomadproject.io/docs/drivers/docker#authentication[here]. <5> This argument sets the node name used by the CSI driver to identify itself in the LINSTOR API. By default, this is set to the nodes hostname. <6> This argument sets the LINSTOR API endpoint. If you are not using the consul service mesh (see Nr. 2 above), this needs to be set to the Controllers API endpoint. The endpoint needs to be reachable from every node this is deployed on. <7> The CSI driver needs to execute mount commands, requiring privileged containers. <8> The `csi_plugin` stanza informs Nomad that this task is a CSI plugin. The Nomad agent will forward requests for volumes to one of the jobs containers. Apply the job by running: [source,shell] ---- $ nomad job run linstor-csi.hcl ==> Monitoring evaluation "0119f19c" Evaluation triggered by job "linstor-csi" ==> Monitoring evaluation "0119f19c" Evaluation status changed: "pending" -> "complete" ==> Evaluation "0119f19c" finished with status "complete" ---- === Using LINSTOR volumes in Nomad Volumes in Nomad are created using a https://www.nomadproject.io/docs/commands/volume/create#volume-specification[volume-specification]. As an example, the following specification requests a 1GiB volume with 2 replicas from the LINSTOR storage pool `thinpool`. .vol1.hcl [source,hcl] ---- id = "vol1" <1> name = "vol1" <2> type = "csi" plugin_id = "linstor.csi.linbit.com" capacity_min = "1GiB" capacity_max = "1GiB" capability { access_mode = "single-node-writer" <3> attachment_mode = "file-system" <4> } mount_options { fs_type = "ext4" <5> } parameters { <6> "resourceGroup" = "default-resource", "storagePool" = "thinpool", "autoPlace" = "2" } ---- <1> The `id` is used to reference this volume in Nomad. Used in the `volume.source` field of a job specification. <2> The `name` is used when creating the volume in the backend (i.e. LINSTOR). Ideally this matches the `id` and is a valid LINSTOR resource name. If the name would not be valid, LINSTOR CSI will generate a random compatible name. <3> What kind of access the volume should support. LINSTOR CSI supports: `single-node-reader-only`::Allow read only access on one node at a time `single-node-writer`:: Allow read and write access on one node at a time `multi-node-reader-only`:: Allow read only access from multiple nodes. <4> Can be `file-system` or `block-device`. <5> Specify the file system to use. LINSTOR CSI supports `ext4` and `xfs`. <6> Additional parameters to pass to LINSTOR CSI. The example above requests the resource be part of the `default-resource` <> and should deploy 2 replicas. + For a complete list of available parameters, you can check out the <>. Kubernetes, like Nomad, makes use of the CSI plugin. To create the volume, run the following command: [source,shell] ---- $ nomad volume create vol1.hcl Created external volume vol1 with ID vol1 $ nomad volume status Container Storage Interface ID Name Plugin ID Schedulable Access Mode vol1 vol1 linstor.csi.linbit.com true $ linstor resource list ╭──────────────────────────────────────────────────────────────────────────────────────────────╮ ┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊ ╞══════════════════════════════════════════════════════════════════════════════════════════════╡ ┊ vol1 ┊ nomad-01.example.com ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2021-06-15 14:56:32 ┊ ┊ vol1 ┊ nomad-02.example.com ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2021-06-15 14:56:32 ┊ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ ---- ==== Using volumes in jobs To use the volume in a job, add the `volume` and `volume_mount` stanzas to the job specification: [source,hcl] ---- job "example" { ... group "example" { volume "example-vol" { type = "csi" source = "vol1" attachment_mode = "file-system" access_mode = "single-node-writer" } task "mount-example" { volume_mount { volume = "example-vol" destination = "/data" } ... } } } ---- ==== Creating snapshots of volumes LINSTOR can create snapshots of existing volumes, provided the underlying storage pool driver supports snapshots. The following command creates a snapshot named `snap1` of the volume `vol1`. [source,shell] ---- $ nomad volume snapshot create vol1 snap1 Snapshot ID Volume ID Size Create Time Ready? snap1 vol1 1.0 GiB None true $ linstor s l ╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ ┊ ResourceName ┊ SnapshotName ┊ NodeNames ┊ Volumes ┊ CreatedOn ┊ State ┊ ╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡ ┊ vol1 ┊ snap1 ┊ nomad-01.example.com, nomad-02.example.com ┊ 0: 1 GiB ┊ 2021-06-15 15:04:10 ┊ Successful ┊ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ---- You can use a snapshot to pre-populate an existing volume with data from the snapshot [source,shell] ---- $ cat vol2.hcl id = "vol2" name = "vol2" snapshot_id = "snap1" type = "csi" plugin_id = "linstor.csi.linbit.com" ... $ nomad volume create vol2.hcl Created external volume vol2 with ID vol2 ----