| 1 |
.. role:: math(raw) |
| 2 |
:format: html latex |
| 3 |
.. |
| 4 |
|
| 5 |
============ |
| 6 |
Introduction |
| 7 |
============ |
| 8 |
|
| 9 |
Root Level Structure |
| 10 |
==================== |
| 11 |
|
| 12 |
While HDF5 gives great flexibility in data storage, straightforward file |
| 13 |
readability and exchange requires adhering to an agreed-upon naming and |
| 14 |
organizational convention. To achieve this goal, Data Exchange adopts a |
| 15 |
layered approach by defining a set of **mandatory** and optional fields. |
| 16 |
|
| 17 |
The general structure of a Data Exchange file is shown in |
| 18 |
table [tab:genrules]. The most basic file must have an **implements** |
| 19 |
string, and an **exchange** group at the root level/group of the HDF5 |
| 20 |
file. Optional *measurement* and *process* groups are also defined. |
| 21 |
Beyond this, additional groups may be added to meet individual needs, |
| 22 |
with guidelines suggesting the best structure. |
| 23 |
|
| 24 |
+---------------+----------------+-----------------------------------------+ |
| 25 |
| Member | Type | Example | |
| 26 |
+===============+================+=========================================+ |
| 27 |
|**implements** | string dataset | **exchange**:*measurement*:*process* | |
| 28 |
+---------------+----------------+-----------------------------------------+ |
| 29 |
| **exchange** | group | | |
| 30 |
+---------------+----------------+-----------------------------------------+ |
| 31 |
| *measurement* | group | | |
| 32 |
+---------------+----------------+-----------------------------------------+ |
| 33 |
| *process* | group | | |
| 34 |
+---------------+----------------+-----------------------------------------+ |
| 35 |
|
| 36 |
**implements** |
| 37 |
| |
| 38 |
| Mandatory scalar string dataset in the root of the HDF5 file whose |
| 39 |
value is a colon separated list that shows which components are |
| 40 |
present in the file. All components listed in the **implements** |
| 41 |
string are to be groups placed in the HDF5 file at the root |
| 42 |
level/group. In a minimal Data Exchange file, the only **mandatory** |
| 43 |
item in this list is **exchange**. A more general Data Exchange file |
| 44 |
also contain *measurement* and possibly *process*, in which case |
| 45 |
the **implements** string would be: **exchange**:*measurement*:*process*. |
| 46 |
|
| 47 |
**exchange** |
| 48 |
| |
| 49 |
| Mandatory group containing one or more arrays that represent the |
| 50 |
most basic version of the data, such as raw or normalized optical |
| 51 |
density maps or a elemental signal map. **Exchange_N** is |
| 52 |
used when more than one core dataset or derived datasets are saved |
| 53 |
in the file. The **exchange** implementation for specific techniques |
| 54 |
are defined in separate sections in the Reference Guide. |
| 55 |
|
| 56 |
*measurement* |
| 57 |
| |
| 58 |
| Optional group containing the *measurement* made on the sample; |
| 59 |
*measurement* contains information about the sample and the |
| 60 |
instrument; *measurement_N* is used when more than one |
| 61 |
*measurement* is stored in the same file. |
| 62 |
|
| 63 |
*process* |
| 64 |
| |
| 65 |
| The Process group describes all the “work” that has been done. |
| 66 |
This includes data processing steps that have been applied to the |
| 67 |
data as well as experimental steps (e.g. data collection strategy etc.) |
| 68 |
and sample preparation ahead of the experiment and during the measurement |
| 69 |
(e.g. environment conditions etc.). |
| 70 |
|
| 71 |
In a Data Exchange file, each dataset has a unit defined using the units |
| 72 |
attribute. units is not **mandatory** - if omitted, the default unit as |
| 73 |
defined in Appendix [appendix:units] is used. |
| 74 |
|
| 75 |
The detailed rules about how to store datasets within the exchange group |
| 76 |
are best shown through examples in the next section. Detailed reference |
| 77 |
information can be found in the section. |
| 78 |
|
| 79 |
Definitions |
| 80 |
=========== |
| 81 |
|
| 82 |
Color code |
| 83 |
---------- |
| 84 |
|
| 85 |
All the diagrams in this section follow the color conventions shown in |
| 86 |
:ref:`DiagramColorCode`. The basic elements are HDF5 datasets, |
| 87 |
attributes, and groups. We also support internal references to elements |
| 88 |
in the file by a simple scalar string that holds the path of the dataset |
| 89 |
within the file. On the diagram, this is shown as a reference dataset |
| 90 |
that points to the referred-to dataset. Note that we use this mechanism |
| 91 |
rather than HDF5 hard or soft links |
| 92 |
|
| 93 |
.. _DiagramColorCode: |
| 94 |
|
| 95 |
.. figure:: figures/dx_DiagramColorCode.png |
| 96 |
:align: center |
| 97 |
:alt: AExplanation of the color code used in the diagrams |
| 98 |
:width: 50.0% |
| 99 |
|
| 100 |
Color Code |
| 101 |
|
| 102 |
Explanation of the color code used in the diagrams |
| 103 |
|
| 104 |
Multidimensional data |
| 105 |
--------------------- |
| 106 |
|
| 107 |
A multidimensional dataset should be described as fully as possible, |
| 108 |
with units for the dataset as well as dimension descriptors (that also |
| 109 |
have units defined). There are also additional descriptive fields |
| 110 |
available such as title and description. The order of dimensions in the |
| 111 |
dataset should put the slowest changing dimension first, and the fastest |
| 112 |
changing dimension last. |
| 113 |
|
| 114 |
It is strongly encouraged that all datasets have a units attribute. The |
| 115 |
string value for units should preferably be an SI unit, however well |
| 116 |
understood non-SI units are acceptable, in particular *degrees*. The |
| 117 |
units strings should conform to those defined by UDUNITS :cite:`UNIDATA:01`. |
| 118 |
While UDUNITS is a software package, it contains simple XML files that |
| 119 |
describe units strings and acceptable aliases. |
| 120 |
|
| 121 |
The axes of a multidimensional dataset are described through the use of |
| 122 |
additional one-dimensional datasets (dimension descriptors), one for |
| 123 |
each axis in the main dataset. Take for example a 3-dimensional cube of |
| 124 |
images, with axes of x, y, and z where z represents the angle of the |
| 125 |
sample when each image was taken. There should be 3 additional |
| 126 |
one-dimensional datasets called x, y, and z where x and y contain an |
| 127 |
integer sequence, and z contains a list of angles. X and y have units of |
| 128 |
*counts* and z has units of *degree*. To simplify, it is acceptable to |
| 129 |
omit x and y, since the default interpretation will always be an integer |
| 130 |
sequence. |
| 131 |
|
| 132 |
The dimension descriptors (x, y, z) can be associated with the main |
| 133 |
dataset through two mechanisms. The HDF5 libraries contain a function |
| 134 |
call H5DSattach_scale to *attach* a dimension descriptor dataset to a |
| 135 |
given dimension of the main dataset. HDF5 takes care of entering several |
| 136 |
attributes in the file that serve to keep track of this association. If |
| 137 |
the particular programming language you work in does not support this |
| 138 |
HDF5 function, then you can instead add a string attribute to your main |
| 139 |
dataset called axes. The axes attribute is simply a colon separated |
| 140 |
string naming the dimension descriptor datasets in order, so *z:y:x* in |
| 141 |
this case. Additional examples below show this in action. |
| 142 |
|
| 143 |
Data Structure |
| 144 |
============== |
| 145 |
|
| 146 |
A tomographic data set consists of a series of projections, dark and white field images. The dark and white fields must have the same |
| 147 |
projection image dimensions and can be collected at any time before, after or during the projection data collection. The angular position of |
| 148 |
the tomographic rotation axis, theta, can be used to keep track of when the dark and white images are collected. |
| 149 |
These examples show projection, dark, and white images saved in three 3D arrays as shown in :ref:`MinimalTomo0` and :ref:`MinimalTomo1` using, by default, the natural HDF5 order of the a multidimensional array (rotation axis, ccd y, ccd x), i.e. with the fastest changing dimension being the last dimension, and the slowest changing dimension being the first dimension. If using the default dimension order, the axes attribute *theta:y:x* can be |
| 150 |
omitted. The attribute is **mandatory** if the 3D arrays use a different axes order. This could be the case when, for example, the arrays are |
| 151 |
optimized for sinogram read *y:theta:x*. As no units are specified the data is assumed to be in *counts" with the axes (x, y) in pixels. If the positions of the rotation axis for each projection, dark, and white images are not specified via theta dimension scale datasets, it is assumed that the raw projections are taken at equally spaced angular intervals between 0 and 180 degree, with white and dark field collected at the same time before or after the projection data collection. |
| 152 |
|
| 153 |
.. _MinimalTomo0: |
| 154 |
|
| 155 |
.. figure:: figures/dx_MinimalTomo0.png |
| 156 |
:align: center |
| 157 |
:alt: Diagram of a minimal Data Exchange file for a single tomographic data set including raw projections, dark, and white fields. |
| 158 |
:width: 50.0% |
| 159 |
|
| 160 |
Basic Tomo A |
| 161 |
|
| 162 |
Diagram of a minimal Data Exchange file for a single tomographic data set including raw projections, dark, and white fields |
| 163 |
|
| 164 |
.. _MinimalTomo1: |
| 165 |
|
| 166 |
.. figure:: figures/dx_MinimalTomo1.png |
| 167 |
:align: center |
| 168 |
:alt: Diagram of a single tomographic data set including raw projections, dark and white fields. In this case, there are additional dimension descriptor datasets theta, theta_dark, and theta_white that contain the positions of the rotation axis for each projection, dark, and white image. The lefthand example shows this as it would appear using the HDF5 H5DSattach_scale function. The righthand example shows this as it would appear by manually adding an axes attribute (for cases where H5DSattach_scale is unavailable). |
| 169 |
:width: 80.0% |
| 170 |
|
| 171 |
Basic Tomo B |
| 172 |
|
| 173 |
Diagram of a single tomographic data set including raw projections, |
| 174 |
dark and white fields. In this case, there are additional dimension |
| 175 |
descriptor datasets theta, theta_dark, and theta_white that contain |
| 176 |
the positions of the rotation axis for each projection, dark, and |
| 177 |
white image. The lefthand example shows this as it would appear using |
| 178 |
the HDF5 H5DSattach_scale function. The righthand example shows this |
| 179 |
as it would appear by manually adding an axes attribute (for cases |
| 180 |
where H5DSattach_scale is unavailable) |
| 181 |
|
| 182 |
Imaging |
| 183 |
------- |
| 184 |
The examples in this section show how one can store data for imaging |
| 185 |
experiments using the Data Exchange format. It is general enough, |
| 186 |
however, to show how Data Exchange can be extended or adapted to other |
| 187 |
techniques. These examples are meant to give a flavor for our approach. |
| 188 |
A complete reference to the core structure can be found in Section |
| 189 |
*Reference*. Technique specific extensions to the core structure |
| 190 |
can be found at the end of the Reference Guide. |
| 191 |
|
| 192 |
:ref:`Minimal1` shows a diagram of a minimal Data Exchange file |
| 193 |
to store a single projection image. It is strongly encouraged that all |
| 194 |
datasets shall have a units attribute. The axes of the dataset are not |
| 195 |
specified in this minimal case, and can be assumed to be x and y with a |
| 196 |
zero-based integer sequence, or more simply, pixels. |
| 197 |
|
| 198 |
.. _Minimal1: |
| 199 |
|
| 200 |
.. figure:: figures/dx_Minimal1.png |
| 201 |
:align: center |
| 202 |
:alt: Diagram of a minimal Data Exchange file for a single image. |
| 203 |
:width: 50.0% |
| 204 |
|
| 205 |
Minimal DXfile |
| 206 |
|
| 207 |
Diagram of a minimal Data Exchange file for a single image. |
| 208 |
|
| 209 |
Series |
| 210 |
------ |
| 211 |
|
| 212 |
A series of tomographic measurements, when relevant, can be stored in |
| 213 |
the same file appending _N to the measurement tag. |
| 214 |
A series of tomographic data sets are typically collected changing the |
| 215 |
instrument status (energy, detector or optics position); changing the |
| 216 |
sample status (position, environment etc.). Figure :ref:`MinimalTomo2`, |
| 217 |
:ref:`MinimalTomo3` and :ref:`MinimalTomo4` show the content of files |
| 218 |
changing the sample temperature, the X-ray source energy and |
| 219 |
detector-sample distance. |
| 220 |
In nano tomography experiments, for example, the detector field of view is |
| 221 |
often smaller than the sample. To collect a complete tomographic data set, |
| 222 |
it is necessary to raster the sample across the field of view moving its x |
| 223 |
and y location. Figure :ref:`NanoTomo1` shows a file from a nano |
| 224 |
tomography experiment when the sample rasters through the field of view. |
| 225 |
|
| 226 |
There are limits to this approach, as one clearly does not want to have |
| 227 |
hundreds of measurement groups in a file (or multiple files) where most |
| 228 |
of the metadata is the same. For measurements where there are many |
| 229 |
“positioner” values (aka a “scan”), it is more sensible to add |
| 230 |
dimension(s) to the exchange dataset, and describe the “positioner” |
| 231 |
values as dimension scales. This is a judgement left to the user. |
| 232 |
|
| 233 |
Temperature |
| 234 |
~~~~~~~~~~~ |
| 235 |
|
| 236 |
.. _MinimalTomo2: |
| 237 |
|
| 238 |
.. figure:: figures/dx_MinimalTomo2.png |
| 239 |
:align: center |
| 240 |
:alt: Diagram of two tomographic data sets taken at two different sample temperatures (100 and 200 Celsius). |
| 241 |
:width: 100.0% |
| 242 |
|
| 243 |
Temperature |
| 244 |
|
| 245 |
Diagram of two tomographic data sets taken at two different sample |
| 246 |
temperatures (100 and 200 Celsius) |
| 247 |
|
| 248 |
Energy |
| 249 |
~~~~~~ |
| 250 |
.. _MinimalTomo3: |
| 251 |
|
| 252 |
.. figure:: figures/dx_MinimalTomo3.png |
| 253 |
:align: center |
| 254 |
:alt: Diagram of two tomographic data sets taken at two different energy (10 and 20 keV). |
| 255 |
:width: 80.0% |
| 256 |
|
| 257 |
Energy |
| 258 |
|
| 259 |
Diagram of two tomographic data sets taken at two different energy |
| 260 |
(10 and 20 keV) |
| 261 |
|
| 262 |
Detector-sample distance |
| 263 |
~~~~~~~~~~~~~~~~~~~~~~~~ |
| 264 |
|
| 265 |
.. _MinimalTomo4: |
| 266 |
|
| 267 |
.. figure:: figures/dx_MinimalTomo4.png |
| 268 |
:align: center |
| 269 |
:alt: Diagram of two tomographic data sets collected with two different detector-sample distances (5 and 9 mm). Note the use of output_data dataset to associate the detector with the exchange group generated from the acquisition. |
| 270 |
:width: 80.0% |
| 271 |
|
| 272 |
Distance |
| 273 |
|
| 274 |
Diagram of two tomographic data sets collected with two different |
| 275 |
detector-sample distances (5 and 9 mm). Note the use of output_data |
| 276 |
dataset to associate the detector with the exchange group generated |
| 277 |
from the acquisition |
| 278 |
|
| 279 |
Raster |
| 280 |
~~~~~~ |
| 281 |
|
| 282 |
.. _NanoTomo1: |
| 283 |
|
| 284 |
.. figure:: figures/dx_NanoTomo1.png |
| 285 |
:align: center |
| 286 |
:alt: Diagram of a file with 4 tomographic data sets from a nano tomography experiment. |
| 287 |
:width: 90.0% |
| 288 |
|
| 289 |
Raster |
| 290 |
|
| 291 |
Diagram of a file with 4 tomographic data sets from a nano tomography |
| 292 |
experiment |
| 293 |
|