| 1 |
=============================== |
| 2 |
LIBNVDIMM: Non-Volatile Devices |
| 3 |
=============================== |
| 4 |
|
| 5 |
libnvdimm - kernel / libndctl - userspace helper library |
| 6 |
|
| 7 |
[email protected] |
| 8 |
|
| 9 |
Version 13 |
| 10 |
|
| 11 |
.. contents: |
| 12 |
|
| 13 |
Glossary |
| 14 |
Overview |
| 15 |
Supporting Documents |
| 16 |
Git Trees |
| 17 |
LIBNVDIMM PMEM |
| 18 |
PMEM-REGIONs, Atomic Sectors, and DAX |
| 19 |
Example NVDIMM Platform |
| 20 |
LIBNVDIMM Kernel Device Model and LIBNDCTL Userspace API |
| 21 |
LIBNDCTL: Context |
| 22 |
libndctl: instantiate a new library context example |
| 23 |
LIBNVDIMM/LIBNDCTL: Bus |
| 24 |
libnvdimm: control class device in /sys/class |
| 25 |
libnvdimm: bus |
| 26 |
libndctl: bus enumeration example |
| 27 |
LIBNVDIMM/LIBNDCTL: DIMM (NMEM) |
| 28 |
libnvdimm: DIMM (NMEM) |
| 29 |
libndctl: DIMM enumeration example |
| 30 |
LIBNVDIMM/LIBNDCTL: Region |
| 31 |
libnvdimm: region |
| 32 |
libndctl: region enumeration example |
| 33 |
Why Not Encode the Region Type into the Region Name? |
| 34 |
How Do I Determine the Major Type of a Region? |
| 35 |
LIBNVDIMM/LIBNDCTL: Namespace |
| 36 |
libnvdimm: namespace |
| 37 |
libndctl: namespace enumeration example |
| 38 |
libndctl: namespace creation example |
| 39 |
Why the Term "namespace"? |
| 40 |
LIBNVDIMM/LIBNDCTL: Block Translation Table "btt" |
| 41 |
libnvdimm: btt layout |
| 42 |
libndctl: btt creation example |
| 43 |
Summary LIBNDCTL Diagram |
| 44 |
|
| 45 |
|
| 46 |
Glossary |
| 47 |
======== |
| 48 |
|
| 49 |
PMEM: |
| 50 |
A system-physical-address range where writes are persistent. A |
| 51 |
block device composed of PMEM is capable of DAX. A PMEM address range |
| 52 |
may span an interleave of several DIMMs. |
| 53 |
|
| 54 |
DPA: |
| 55 |
DIMM Physical Address, is a DIMM-relative offset. With one DIMM in |
| 56 |
the system there would be a 1:1 system-physical-address:DPA association. |
| 57 |
Once more DIMMs are added a memory controller interleave must be |
| 58 |
decoded to determine the DPA associated with a given |
| 59 |
system-physical-address. |
| 60 |
|
| 61 |
DAX: |
| 62 |
File system extensions to bypass the page cache and block layer to |
| 63 |
mmap persistent memory, from a PMEM block device, directly into a |
| 64 |
process address space. |
| 65 |
|
| 66 |
DSM: |
| 67 |
Device Specific Method: ACPI method to control specific |
| 68 |
device - in this case the firmware. |
| 69 |
|
| 70 |
DCR: |
| 71 |
NVDIMM Control Region Structure defined in ACPI 6 Section 5.2.25.5. |
| 72 |
It defines a vendor-id, device-id, and interface format for a given DIMM. |
| 73 |
|
| 74 |
BTT: |
| 75 |
Block Translation Table: Persistent memory is byte addressable. |
| 76 |
Existing software may have an expectation that the power-fail-atomicity |
| 77 |
of writes is at least one sector, 512 bytes. The BTT is an indirection |
| 78 |
table with atomic update semantics to front a PMEM block device |
| 79 |
driver and present arbitrary atomic sector sizes. |
| 80 |
|
| 81 |
LABEL: |
| 82 |
Metadata stored on a DIMM device that partitions and identifies |
| 83 |
(persistently names) capacity allocated to different PMEM namespaces. It |
| 84 |
also indicates whether an address abstraction like a BTT is applied to |
| 85 |
the namespace. Note that traditional partition tables, GPT/MBR, are |
| 86 |
layered on top of a PMEM namespace, or an address abstraction like BTT |
| 87 |
if present, but partition support is deprecated going forward. |
| 88 |
|
| 89 |
|
| 90 |
Overview |
| 91 |
======== |
| 92 |
|
| 93 |
The LIBNVDIMM subsystem provides support for PMEM described by platform |
| 94 |
firmware or a device driver. On ACPI based systems the platform firmware |
| 95 |
conveys persistent memory resource via the ACPI NFIT "NVDIMM Firmware |
| 96 |
Interface Table" in ACPI 6. While the LIBNVDIMM subsystem implementation |
| 97 |
is generic and supports pre-NFIT platforms, it was guided by the |
| 98 |
superset of capabilities need to support this ACPI 6 definition for |
| 99 |
NVDIMM resources. The original implementation supported the |
| 100 |
block-window-aperture capability described in the NFIT, but that support |
| 101 |
has since been abandoned and never shipped in a product. |
| 102 |
|
| 103 |
Supporting Documents |
| 104 |
-------------------- |
| 105 |
|
| 106 |
ACPI 6: |
| 107 |
https://www.uefi.org/sites/default/files/resources/ACPI_6.0.pdf |
| 108 |
NVDIMM Namespace: |
| 109 |
https://pmem.io/documents/NVDIMM_Namespace_Spec.pdf |
| 110 |
DSM Interface Example: |
| 111 |
https://pmem.io/documents/NVDIMM_DSM_Interface_Example.pdf |
| 112 |
Driver Writer's Guide: |
| 113 |
https://pmem.io/documents/NVDIMM_Driver_Writers_Guide.pdf |
| 114 |
|
| 115 |
Git Trees |
| 116 |
--------- |
| 117 |
|
| 118 |
LIBNVDIMM: |
| 119 |
https://git.kernel.org/cgit/linux/kernel/git/nvdimm/nvdimm.git |
| 120 |
LIBNDCTL: |
| 121 |
https://github.com/pmem/ndctl.git |
| 122 |
|
| 123 |
|
| 124 |
LIBNVDIMM PMEM |
| 125 |
============== |
| 126 |
|
| 127 |
Prior to the arrival of the NFIT, non-volatile memory was described to a |
| 128 |
system in various ad-hoc ways. Usually only the bare minimum was |
| 129 |
provided, namely, a single system-physical-address range where writes |
| 130 |
are expected to be durable after a system power loss. Now, the NFIT |
| 131 |
specification standardizes not only the description of PMEM, but also |
| 132 |
platform message-passing entry points for control and configuration. |
| 133 |
|
| 134 |
PMEM (nd_pmem.ko): Drives a system-physical-address range. This range is |
| 135 |
contiguous in system memory and may be interleaved (hardware memory controller |
| 136 |
striped) across multiple DIMMs. When interleaved the platform may optionally |
| 137 |
provide details of which DIMMs are participating in the interleave. |
| 138 |
|
| 139 |
It is worth noting that when the labeling capability is detected (a EFI |
| 140 |
namespace label index block is found), then no block device is created |
| 141 |
by default as userspace needs to do at least one allocation of DPA to |
| 142 |
the PMEM range. In contrast ND_NAMESPACE_IO ranges, once registered, |
| 143 |
can be immediately attached to nd_pmem. This latter mode is called |
| 144 |
label-less or "legacy". |
| 145 |
|
| 146 |
PMEM-REGIONs, Atomic Sectors, and DAX |
| 147 |
------------------------------------- |
| 148 |
|
| 149 |
For the cases where an application or filesystem still needs atomic sector |
| 150 |
update guarantees it can register a BTT on a PMEM device or partition. See |
| 151 |
LIBNVDIMM/NDCTL: Block Translation Table "btt" |
| 152 |
|
| 153 |
|
| 154 |
Example NVDIMM Platform |
| 155 |
======================= |
| 156 |
|
| 157 |
For the remainder of this document the following diagram will be |
| 158 |
referenced for any example sysfs layouts:: |
| 159 |
|
| 160 |
|
| 161 |
(a) (b) DIMM |
| 162 |
+-------------------+--------+--------+--------+ |
| 163 |
+------+ | pm0.0 | free | pm1.0 | free | 0 |
| 164 |
| imc0 +--+- - - region0- - - +--------+ +--------+ |
| 165 |
+--+---+ | pm0.0 | free | pm1.0 | free | 1 |
| 166 |
| +-------------------+--------v v--------+ |
| 167 |
+--+---+ | | |
| 168 |
| cpu0 | region1 |
| 169 |
+--+---+ | | |
| 170 |
| +----------------------------^ ^--------+ |
| 171 |
+--+---+ | free | pm1.0 | free | 2 |
| 172 |
| imc1 +--+----------------------------| +--------+ |
| 173 |
+------+ | free | pm1.0 | free | 3 |
| 174 |
+----------------------------+--------+--------+ |
| 175 |
|
| 176 |
In this platform we have four DIMMs and two memory controllers in one |
| 177 |
socket. Each PMEM interleave set is identified by a region device with |
| 178 |
a dynamically assigned id. |
| 179 |
|
| 180 |
1. The first portion of DIMM0 and DIMM1 are interleaved as REGION0. A |
| 181 |
single PMEM namespace is created in the REGION0-SPA-range that spans most |
| 182 |
of DIMM0 and DIMM1 with a user-specified name of "pm0.0". Some of that |
| 183 |
interleaved system-physical-address range is left free for |
| 184 |
another PMEM namespace to be defined. |
| 185 |
|
| 186 |
2. In the last portion of DIMM0 and DIMM1 we have an interleaved |
| 187 |
system-physical-address range, REGION1, that spans those two DIMMs as |
| 188 |
well as DIMM2 and DIMM3. Some of REGION1 is allocated to a PMEM namespace |
| 189 |
named "pm1.0". |
| 190 |
|
| 191 |
This bus is provided by the kernel under the device |
| 192 |
/sys/devices/platform/nfit_test.0 when the nfit_test.ko module from |
| 193 |
tools/testing/nvdimm is loaded. This module is a unit test for |
| 194 |
LIBNVDIMM and the acpi_nfit.ko driver. |
| 195 |
|
| 196 |
|
| 197 |
LIBNVDIMM Kernel Device Model and LIBNDCTL Userspace API |
| 198 |
======================================================== |
| 199 |
|
| 200 |
What follows is a description of the LIBNVDIMM sysfs layout and a |
| 201 |
corresponding object hierarchy diagram as viewed through the LIBNDCTL |
| 202 |
API. The example sysfs paths and diagrams are relative to the Example |
| 203 |
NVDIMM Platform which is also the LIBNVDIMM bus used in the LIBNDCTL unit |
| 204 |
test. |
| 205 |
|
| 206 |
LIBNDCTL: Context |
| 207 |
----------------- |
| 208 |
|
| 209 |
Every API call in the LIBNDCTL library requires a context that holds the |
| 210 |
logging parameters and other library instance state. The library is |
| 211 |
based on the libabc template: |
| 212 |
|
| 213 |
https://git.kernel.org/cgit/linux/kernel/git/kay/libabc.git |
| 214 |
|
| 215 |
LIBNDCTL: instantiate a new library context example |
| 216 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 217 |
|
| 218 |
:: |
| 219 |
|
| 220 |
struct ndctl_ctx *ctx; |
| 221 |
|
| 222 |
if (ndctl_new(&ctx) == 0) |
| 223 |
return ctx; |
| 224 |
else |
| 225 |
return NULL; |
| 226 |
|
| 227 |
LIBNVDIMM/LIBNDCTL: Bus |
| 228 |
----------------------- |
| 229 |
|
| 230 |
A bus has a 1:1 relationship with an NFIT. The current expectation for |
| 231 |
ACPI based systems is that there is only ever one platform-global NFIT. |
| 232 |
That said, it is trivial to register multiple NFITs, the specification |
| 233 |
does not preclude it. The infrastructure supports multiple buses and |
| 234 |
we use this capability to test multiple NFIT configurations in the unit |
| 235 |
test. |
| 236 |
|
| 237 |
LIBNVDIMM: control class device in /sys/class |
| 238 |
--------------------------------------------- |
| 239 |
|
| 240 |
This character device accepts DSM messages to be passed to DIMM |
| 241 |
identified by its NFIT handle:: |
| 242 |
|
| 243 |
/sys/class/nd/ndctl0 |
| 244 |
|-- dev |
| 245 |
|-- device -> ../../../ndbus0 |
| 246 |
|-- subsystem -> ../../../../../../../class/nd |
| 247 |
|
| 248 |
|
| 249 |
|
| 250 |
LIBNVDIMM: bus |
| 251 |
-------------- |
| 252 |
|
| 253 |
:: |
| 254 |
|
| 255 |
struct nvdimm_bus *nvdimm_bus_register(struct device *parent, |
| 256 |
struct nvdimm_bus_descriptor *nfit_desc); |
| 257 |
|
| 258 |
:: |
| 259 |
|
| 260 |
/sys/devices/platform/nfit_test.0/ndbus0 |
| 261 |
|-- commands |
| 262 |
|-- nd |
| 263 |
|-- nfit |
| 264 |
|-- nmem0 |
| 265 |
|-- nmem1 |
| 266 |
|-- nmem2 |
| 267 |
|-- nmem3 |
| 268 |
|-- power |
| 269 |
|-- provider |
| 270 |
|-- region0 |
| 271 |
|-- region1 |
| 272 |
|-- region2 |
| 273 |
|-- region3 |
| 274 |
|-- region4 |
| 275 |
|-- region5 |
| 276 |
|-- uevent |
| 277 |
`-- wait_probe |
| 278 |
|
| 279 |
LIBNDCTL: bus enumeration example |
| 280 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 281 |
|
| 282 |
Find the bus handle that describes the bus from Example NVDIMM Platform:: |
| 283 |
|
| 284 |
static struct ndctl_bus *get_bus_by_provider(struct ndctl_ctx *ctx, |
| 285 |
const char *provider) |
| 286 |
{ |
| 287 |
struct ndctl_bus *bus; |
| 288 |
|
| 289 |
ndctl_bus_foreach(ctx, bus) |
| 290 |
if (strcmp(provider, ndctl_bus_get_provider(bus)) == 0) |
| 291 |
return bus; |
| 292 |
|
| 293 |
return NULL; |
| 294 |
} |
| 295 |
|
| 296 |
bus = get_bus_by_provider(ctx, "nfit_test.0"); |
| 297 |
|
| 298 |
|
| 299 |
LIBNVDIMM/LIBNDCTL: DIMM (NMEM) |
| 300 |
------------------------------- |
| 301 |
|
| 302 |
The DIMM device provides a character device for sending commands to |
| 303 |
hardware, and it is a container for LABELs. If the DIMM is defined by |
| 304 |
NFIT then an optional 'nfit' attribute sub-directory is available to add |
| 305 |
NFIT-specifics. |
| 306 |
|
| 307 |
Note that the kernel device name for "DIMMs" is "nmemX". The NFIT |
| 308 |
describes these devices via "Memory Device to System Physical Address |
| 309 |
Range Mapping Structure", and there is no requirement that they actually |
| 310 |
be physical DIMMs, so we use a more generic name. |
| 311 |
|
| 312 |
LIBNVDIMM: DIMM (NMEM) |
| 313 |
^^^^^^^^^^^^^^^^^^^^^^ |
| 314 |
|
| 315 |
:: |
| 316 |
|
| 317 |
struct nvdimm *nvdimm_create(struct nvdimm_bus *nvdimm_bus, void *provider_data, |
| 318 |
const struct attribute_group **groups, unsigned long flags, |
| 319 |
unsigned long *dsm_mask); |
| 320 |
|
| 321 |
:: |
| 322 |
|
| 323 |
/sys/devices/platform/nfit_test.0/ndbus0 |
| 324 |
|-- nmem0 |
| 325 |
| |-- available_slots |
| 326 |
| |-- commands |
| 327 |
| |-- dev |
| 328 |
| |-- devtype |
| 329 |
| |-- driver -> ../../../../../bus/nd/drivers/nvdimm |
| 330 |
| |-- modalias |
| 331 |
| |-- nfit |
| 332 |
| | |-- device |
| 333 |
| | |-- format |
| 334 |
| | |-- handle |
| 335 |
| | |-- phys_id |
| 336 |
| | |-- rev_id |
| 337 |
| | |-- serial |
| 338 |
| | `-- vendor |
| 339 |
| |-- state |
| 340 |
| |-- subsystem -> ../../../../../bus/nd |
| 341 |
| `-- uevent |
| 342 |
|-- nmem1 |
| 343 |
[..] |
| 344 |
|
| 345 |
|
| 346 |
LIBNDCTL: DIMM enumeration example |
| 347 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 348 |
|
| 349 |
Note, in this example we are assuming NFIT-defined DIMMs which are |
| 350 |
identified by an "nfit_handle" a 32-bit value where: |
| 351 |
|
| 352 |
- Bit 3:0 DIMM number within the memory channel |
| 353 |
- Bit 7:4 memory channel number |
| 354 |
- Bit 11:8 memory controller ID |
| 355 |
- Bit 15:12 socket ID (within scope of a Node controller if node |
| 356 |
controller is present) |
| 357 |
- Bit 27:16 Node Controller ID |
| 358 |
- Bit 31:28 Reserved |
| 359 |
|
| 360 |
:: |
| 361 |
|
| 362 |
static struct ndctl_dimm *get_dimm_by_handle(struct ndctl_bus *bus, |
| 363 |
unsigned int handle) |
| 364 |
{ |
| 365 |
struct ndctl_dimm *dimm; |
| 366 |
|
| 367 |
ndctl_dimm_foreach(bus, dimm) |
| 368 |
if (ndctl_dimm_get_handle(dimm) == handle) |
| 369 |
return dimm; |
| 370 |
|
| 371 |
return NULL; |
| 372 |
} |
| 373 |
|
| 374 |
#define DIMM_HANDLE(n, s, i, c, d) \ |
| 375 |
(((n & 0xfff) << 16) | ((s & 0xf) << 12) | ((i & 0xf) << 8) \ |
| 376 |
| ((c & 0xf) << 4) | (d & 0xf)) |
| 377 |
|
| 378 |
dimm = get_dimm_by_handle(bus, DIMM_HANDLE(0, 0, 0, 0, 0)); |
| 379 |
|
| 380 |
LIBNVDIMM/LIBNDCTL: Region |
| 381 |
-------------------------- |
| 382 |
|
| 383 |
A generic REGION device is registered for each PMEM interleave-set / |
| 384 |
range. Per the example there are 2 PMEM regions on the "nfit_test.0" |
| 385 |
bus. The primary role of regions are to be a container of "mappings". A |
| 386 |
mapping is a tuple of <DIMM, DPA-start-offset, length>. |
| 387 |
|
| 388 |
LIBNVDIMM provides a built-in driver for REGION devices. This driver |
| 389 |
is responsible for all parsing LABELs, if present, and then emitting NAMESPACE |
| 390 |
devices for the nd_pmem driver to consume. |
| 391 |
|
| 392 |
In addition to the generic attributes of "mapping"s, "interleave_ways" |
| 393 |
and "size" the REGION device also exports some convenience attributes. |
| 394 |
"nstype" indicates the integer type of namespace-device this region |
| 395 |
emits, "devtype" duplicates the DEVTYPE variable stored by udev at the |
| 396 |
'add' event, "modalias" duplicates the MODALIAS variable stored by udev |
| 397 |
at the 'add' event, and finally, the optional "spa_index" is provided in |
| 398 |
the case where the region is defined by a SPA. |
| 399 |
|
| 400 |
LIBNVDIMM: region:: |
| 401 |
|
| 402 |
struct nd_region *nvdimm_pmem_region_create(struct nvdimm_bus *nvdimm_bus, |
| 403 |
struct nd_region_desc *ndr_desc); |
| 404 |
|
| 405 |
:: |
| 406 |
|
| 407 |
/sys/devices/platform/nfit_test.0/ndbus0 |
| 408 |
|-- region0 |
| 409 |
| |-- available_size |
| 410 |
| |-- btt0 |
| 411 |
| |-- btt_seed |
| 412 |
| |-- devtype |
| 413 |
| |-- driver -> ../../../../../bus/nd/drivers/nd_region |
| 414 |
| |-- init_namespaces |
| 415 |
| |-- mapping0 |
| 416 |
| |-- mapping1 |
| 417 |
| |-- mappings |
| 418 |
| |-- modalias |
| 419 |
| |-- namespace0.0 |
| 420 |
| |-- namespace_seed |
| 421 |
| |-- numa_node |
| 422 |
| |-- nfit |
| 423 |
| | `-- spa_index |
| 424 |
| |-- nstype |
| 425 |
| |-- set_cookie |
| 426 |
| |-- size |
| 427 |
| |-- subsystem -> ../../../../../bus/nd |
| 428 |
| `-- uevent |
| 429 |
|-- region1 |
| 430 |
[..] |
| 431 |
|
| 432 |
LIBNDCTL: region enumeration example |
| 433 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 434 |
|
| 435 |
Sample region retrieval routines based on NFIT-unique data like |
| 436 |
"spa_index" (interleave set id). |
| 437 |
|
| 438 |
:: |
| 439 |
|
| 440 |
static struct ndctl_region *get_pmem_region_by_spa_index(struct ndctl_bus *bus, |
| 441 |
unsigned int spa_index) |
| 442 |
{ |
| 443 |
struct ndctl_region *region; |
| 444 |
|
| 445 |
ndctl_region_foreach(bus, region) { |
| 446 |
if (ndctl_region_get_type(region) != ND_DEVICE_REGION_PMEM) |
| 447 |
continue; |
| 448 |
if (ndctl_region_get_spa_index(region) == spa_index) |
| 449 |
return region; |
| 450 |
} |
| 451 |
return NULL; |
| 452 |
} |
| 453 |
|
| 454 |
|
| 455 |
LIBNVDIMM/LIBNDCTL: Namespace |
| 456 |
----------------------------- |
| 457 |
|
| 458 |
A REGION, after resolving DPA aliasing and LABEL specified boundaries, surfaces |
| 459 |
one or more "namespace" devices. The arrival of a "namespace" device currently |
| 460 |
triggers the nd_pmem driver to load and register a disk/block device. |
| 461 |
|
| 462 |
LIBNVDIMM: namespace |
| 463 |
^^^^^^^^^^^^^^^^^^^^ |
| 464 |
|
| 465 |
Here is a sample layout from the 2 major types of NAMESPACE where namespace0.0 |
| 466 |
represents DIMM-info-backed PMEM (note that it has a 'uuid' attribute), and |
| 467 |
namespace1.0 represents an anonymous PMEM namespace (note that has no 'uuid' |
| 468 |
attribute due to not support a LABEL) |
| 469 |
|
| 470 |
:: |
| 471 |
|
| 472 |
/sys/devices/platform/nfit_test.0/ndbus0/region0/namespace0.0 |
| 473 |
|-- alt_name |
| 474 |
|-- devtype |
| 475 |
|-- dpa_extents |
| 476 |
|-- force_raw |
| 477 |
|-- modalias |
| 478 |
|-- numa_node |
| 479 |
|-- resource |
| 480 |
|-- size |
| 481 |
|-- subsystem -> ../../../../../../bus/nd |
| 482 |
|-- type |
| 483 |
|-- uevent |
| 484 |
`-- uuid |
| 485 |
/sys/devices/platform/nfit_test.1/ndbus1/region1/namespace1.0 |
| 486 |
|-- block |
| 487 |
| `-- pmem0 |
| 488 |
|-- devtype |
| 489 |
|-- driver -> ../../../../../../bus/nd/drivers/pmem |
| 490 |
|-- force_raw |
| 491 |
|-- modalias |
| 492 |
|-- numa_node |
| 493 |
|-- resource |
| 494 |
|-- size |
| 495 |
|-- subsystem -> ../../../../../../bus/nd |
| 496 |
|-- type |
| 497 |
`-- uevent |
| 498 |
|
| 499 |
LIBNDCTL: namespace enumeration example |
| 500 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 501 |
Namespaces are indexed relative to their parent region, example below. |
| 502 |
These indexes are mostly static from boot to boot, but subsystem makes |
| 503 |
no guarantees in this regard. For a static namespace identifier use its |
| 504 |
'uuid' attribute. |
| 505 |
|
| 506 |
:: |
| 507 |
|
| 508 |
static struct ndctl_namespace |
| 509 |
*get_namespace_by_id(struct ndctl_region *region, unsigned int id) |
| 510 |
{ |
| 511 |
struct ndctl_namespace *ndns; |
| 512 |
|
| 513 |
ndctl_namespace_foreach(region, ndns) |
| 514 |
if (ndctl_namespace_get_id(ndns) == id) |
| 515 |
return ndns; |
| 516 |
|
| 517 |
return NULL; |
| 518 |
} |
| 519 |
|
| 520 |
LIBNDCTL: namespace creation example |
| 521 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 522 |
|
| 523 |
Idle namespaces are automatically created by the kernel if a given |
| 524 |
region has enough available capacity to create a new namespace. |
| 525 |
Namespace instantiation involves finding an idle namespace and |
| 526 |
configuring it. For the most part the setting of namespace attributes |
| 527 |
can occur in any order, the only constraint is that 'uuid' must be set |
| 528 |
before 'size'. This enables the kernel to track DPA allocations |
| 529 |
internally with a static identifier:: |
| 530 |
|
| 531 |
static int configure_namespace(struct ndctl_region *region, |
| 532 |
struct ndctl_namespace *ndns, |
| 533 |
struct namespace_parameters *parameters) |
| 534 |
{ |
| 535 |
char devname[50]; |
| 536 |
|
| 537 |
snprintf(devname, sizeof(devname), "namespace%d.%d", |
| 538 |
ndctl_region_get_id(region), parameters->id); |
| 539 |
|
| 540 |
ndctl_namespace_set_alt_name(ndns, devname); |
| 541 |
/* 'uuid' must be set prior to setting size! */ |
| 542 |
ndctl_namespace_set_uuid(ndns, parameters->uuid); |
| 543 |
ndctl_namespace_set_size(ndns, parameters->size); |
| 544 |
/* unlike pmem namespaces, blk namespaces have a sector size */ |
| 545 |
if (parameters->lbasize) |
| 546 |
ndctl_namespace_set_sector_size(ndns, parameters->lbasize); |
| 547 |
ndctl_namespace_enable(ndns); |
| 548 |
} |
| 549 |
|
| 550 |
|
| 551 |
Why the Term "namespace"? |
| 552 |
^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 553 |
|
| 554 |
1. Why not "volume" for instance? "volume" ran the risk of confusing |
| 555 |
ND (libnvdimm subsystem) to a volume manager like device-mapper. |
| 556 |
|
| 557 |
2. The term originated to describe the sub-devices that can be created |
| 558 |
within a NVME controller (see the nvme specification: |
| 559 |
https://www.nvmexpress.org/specifications/), and NFIT namespaces are |
| 560 |
meant to parallel the capabilities and configurability of |
| 561 |
NVME-namespaces. |
| 562 |
|
| 563 |
|
| 564 |
LIBNVDIMM/LIBNDCTL: Block Translation Table "btt" |
| 565 |
------------------------------------------------- |
| 566 |
|
| 567 |
A BTT (design document: https://pmem.io/2014/09/23/btt.html) is a |
| 568 |
personality driver for a namespace that fronts entire namespace as an |
| 569 |
'address abstraction'. |
| 570 |
|
| 571 |
LIBNVDIMM: btt layout |
| 572 |
^^^^^^^^^^^^^^^^^^^^^ |
| 573 |
|
| 574 |
Every region will start out with at least one BTT device which is the |
| 575 |
seed device. To activate it set the "namespace", "uuid", and |
| 576 |
"sector_size" attributes and then bind the device to the nd_pmem or |
| 577 |
nd_blk driver depending on the region type:: |
| 578 |
|
| 579 |
/sys/devices/platform/nfit_test.1/ndbus0/region0/btt0/ |
| 580 |
|-- namespace |
| 581 |
|-- delete |
| 582 |
|-- devtype |
| 583 |
|-- modalias |
| 584 |
|-- numa_node |
| 585 |
|-- sector_size |
| 586 |
|-- subsystem -> ../../../../../bus/nd |
| 587 |
|-- uevent |
| 588 |
`-- uuid |
| 589 |
|
| 590 |
LIBNDCTL: btt creation example |
| 591 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 592 |
|
| 593 |
Similar to namespaces an idle BTT device is automatically created per |
| 594 |
region. Each time this "seed" btt device is configured and enabled a new |
| 595 |
seed is created. Creating a BTT configuration involves two steps of |
| 596 |
finding and idle BTT and assigning it to consume a namespace. |
| 597 |
|
| 598 |
:: |
| 599 |
|
| 600 |
static struct ndctl_btt *get_idle_btt(struct ndctl_region *region) |
| 601 |
{ |
| 602 |
struct ndctl_btt *btt; |
| 603 |
|
| 604 |
ndctl_btt_foreach(region, btt) |
| 605 |
if (!ndctl_btt_is_enabled(btt) |
| 606 |
&& !ndctl_btt_is_configured(btt)) |
| 607 |
return btt; |
| 608 |
|
| 609 |
return NULL; |
| 610 |
} |
| 611 |
|
| 612 |
static int configure_btt(struct ndctl_region *region, |
| 613 |
struct btt_parameters *parameters) |
| 614 |
{ |
| 615 |
btt = get_idle_btt(region); |
| 616 |
|
| 617 |
ndctl_btt_set_uuid(btt, parameters->uuid); |
| 618 |
ndctl_btt_set_sector_size(btt, parameters->sector_size); |
| 619 |
ndctl_btt_set_namespace(btt, parameters->ndns); |
| 620 |
/* turn off raw mode device */ |
| 621 |
ndctl_namespace_disable(parameters->ndns); |
| 622 |
/* turn on btt access */ |
| 623 |
ndctl_btt_enable(btt); |
| 624 |
} |
| 625 |
|
| 626 |
Once instantiated a new inactive btt seed device will appear underneath |
| 627 |
the region. |
| 628 |
|
| 629 |
Once a "namespace" is removed from a BTT that instance of the BTT device |
| 630 |
will be deleted or otherwise reset to default values. This deletion is |
| 631 |
only at the device model level. In order to destroy a BTT the "info |
| 632 |
block" needs to be destroyed. Note, that to destroy a BTT the media |
| 633 |
needs to be written in raw mode. By default, the kernel will autodetect |
| 634 |
the presence of a BTT and disable raw mode. This autodetect behavior |
| 635 |
can be suppressed by enabling raw mode for the namespace via the |
| 636 |
ndctl_namespace_set_raw_mode() API. |
| 637 |
|
| 638 |
|
| 639 |
Summary LIBNDCTL Diagram |
| 640 |
------------------------ |
| 641 |
|
| 642 |
For the given example above, here is the view of the objects as seen by the |
| 643 |
LIBNDCTL API:: |
| 644 |
|
| 645 |
+---+ |
| 646 |
|CTX| |
| 647 |
+-+-+ |
| 648 |
| |
| 649 |
+-------+ | |
| 650 |
| DIMM0 <-+ | +---------+ +--------------+ +---------------+ |
| 651 |
+-------+ | | +-> REGION0 +---> NAMESPACE0.0 +--> PMEM8 "pm0.0" | |
| 652 |
| DIMM1 <-+ +-v--+ | +---------+ +--------------+ +---------------+ |
| 653 |
+-------+ +-+BUS0+-| +---------+ +--------------+ +----------------------+ |
| 654 |
| DIMM2 <-+ +----+ +-> REGION1 +---> NAMESPACE1.0 +--> PMEM6 "pm1.0" | BTT1 | |
| 655 |
+-------+ | | +---------+ +--------------+ +---------------+------+ |
| 656 |
| DIMM3 <-+ |
| 657 |
+-------+ |