Debian Codegraph @

doc/source/introduction.rst

raw
1 .. role:: math(raw)
2 :format: html latex
3 ..
4
5 ============
6 Introduction
7 ============
8
9 Root Level Structure
10 ====================
11
12 While HDF5 gives great flexibility in data storage, straightforward file
13 readability and exchange requires adhering to an agreed-upon naming and
14 organizational convention. To achieve this goal, Data Exchange adopts a
15 layered approach by defining a set of **mandatory** and optional fields.
16
17 The general structure of a Data Exchange file is shown in
18 table [tab:genrules]. The most basic file must have an **implements**
19 string, and an **exchange** group at the root level/group of the HDF5
20 file. Optional *measurement* and *process* groups are also defined.
21 Beyond this, additional groups may be added to meet individual needs,
22 with guidelines suggesting the best structure.
23
24 +---------------+----------------+-----------------------------------------+
25 | Member | Type | Example |
26 +===============+================+=========================================+
27 |**implements** | string dataset | **exchange**:*measurement*:*process* |
28 +---------------+----------------+-----------------------------------------+
29 | **exchange** | group | |
30 +---------------+----------------+-----------------------------------------+
31 | *measurement* | group | |
32 +---------------+----------------+-----------------------------------------+
33 | *process* | group | |
34 +---------------+----------------+-----------------------------------------+
35
36 **implements**
37 |
38 | Mandatory scalar string dataset in the root of the HDF5 file whose
39 value is a colon separated list that shows which components are
40 present in the file. All components listed in the **implements**
41 string are to be groups placed in the HDF5 file at the root
42 level/group. In a minimal Data Exchange file, the only **mandatory**
43 item in this list is **exchange**. A more general Data Exchange file
44 also contain *measurement* and possibly *process*, in which case
45 the **implements** string would be: **exchange**:*measurement*:*process*.
46
47 **exchange**
48 |
49 | Mandatory group containing one or more arrays that represent the
50 most basic version of the data, such as raw or normalized optical
51 density maps or a elemental signal map. **Exchange_N** is
52 used when more than one core dataset or derived datasets are saved
53 in the file. The **exchange** implementation for specific techniques
54 are defined in separate sections in the Reference Guide.
55
56 *measurement*
57 |
58 | Optional group containing the *measurement* made on the sample;
59 *measurement* contains information about the sample and the
60 instrument; *measurement_N* is used when more than one
61 *measurement* is stored in the same file.
62
63 *process*
64 |
65 | The Process group describes all the “work” that has been done.
66 This includes data processing steps that have been applied to the
67 data as well as experimental steps (e.g. data collection strategy etc.)
68 and sample preparation ahead of the experiment and during the measurement
69 (e.g. environment conditions etc.).
70
71 In a Data Exchange file, each dataset has a unit defined using the units
72 attribute. units is not **mandatory** - if omitted, the default unit as
73 defined in Appendix [appendix:units] is used.
74
75 The detailed rules about how to store datasets within the exchange group
76 are best shown through examples in the next section. Detailed reference
77 information can be found in the section.
78
79 Definitions
80 ===========
81
82 Color code
83 ----------
84
85 All the diagrams in this section follow the color conventions shown in
86 :ref:`DiagramColorCode`. The basic elements are HDF5 datasets,
87 attributes, and groups. We also support internal references to elements
88 in the file by a simple scalar string that holds the path of the dataset
89 within the file. On the diagram, this is shown as a reference dataset
90 that points to the referred-to dataset. Note that we use this mechanism
91 rather than HDF5 hard or soft links
92
93 .. _DiagramColorCode:
94
95 .. figure:: figures/dx_DiagramColorCode.png
96 :align: center
97 :alt: AExplanation of the color code used in the diagrams
98 :width: 50.0%
99
100 Color Code
101
102 Explanation of the color code used in the diagrams
103
104 Multidimensional data
105 ---------------------
106
107 A multidimensional dataset should be described as fully as possible,
108 with units for the dataset as well as dimension descriptors (that also
109 have units defined). There are also additional descriptive fields
110 available such as title and description. The order of dimensions in the
111 dataset should put the slowest changing dimension first, and the fastest
112 changing dimension last.
113
114 It is strongly encouraged that all datasets have a units attribute. The
115 string value for units should preferably be an SI unit, however well
116 understood non-SI units are acceptable, in particular *degrees*. The
117 units strings should conform to those defined by UDUNITS :cite:`UNIDATA:01`.
118 While UDUNITS is a software package, it contains simple XML files that
119 describe units strings and acceptable aliases.
120
121 The axes of a multidimensional dataset are described through the use of
122 additional one-dimensional datasets (dimension descriptors), one for
123 each axis in the main dataset. Take for example a 3-dimensional cube of
124 images, with axes of x, y, and z where z represents the angle of the
125 sample when each image was taken. There should be 3 additional
126 one-dimensional datasets called x, y, and z where x and y contain an
127 integer sequence, and z contains a list of angles. X and y have units of
128 *counts* and z has units of *degree*. To simplify, it is acceptable to
129 omit x and y, since the default interpretation will always be an integer
130 sequence.
131
132 The dimension descriptors (x, y, z) can be associated with the main
133 dataset through two mechanisms. The HDF5 libraries contain a function
134 call H5DSattach_scale to *attach* a dimension descriptor dataset to a
135 given dimension of the main dataset. HDF5 takes care of entering several
136 attributes in the file that serve to keep track of this association. If
137 the particular programming language you work in does not support this
138 HDF5 function, then you can instead add a string attribute to your main
139 dataset called axes. The axes attribute is simply a colon separated
140 string naming the dimension descriptor datasets in order, so *z:y:x* in
141 this case. Additional examples below show this in action.
142
143 Data Structure
144 ==============
145
146 A tomographic data set consists of a series of projections, dark and white field images. The dark and white fields must have the same
147 projection image dimensions and can be collected at any time before, after or during the projection data collection. The angular position of
148 the tomographic rotation axis, theta, can be used to keep track of when the dark and white images are collected.
149 These examples show projection, dark, and white images saved in three 3D arrays as shown in :ref:`MinimalTomo0` and :ref:`MinimalTomo1` using, by default, the natural HDF5 order of the a multidimensional array (rotation axis, ccd y, ccd x), i.e. with the fastest changing dimension being the last dimension, and the slowest changing dimension being the first dimension. If using the default dimension order, the axes attribute *theta:y:x* can be
150 omitted. The attribute is **mandatory** if the 3D arrays use a different axes order. This could be the case when, for example, the arrays are
151 optimized for sinogram read *y:theta:x*. As no units are specified the data is assumed to be in *counts" with the axes (x, y) in pixels. If the positions of the rotation axis for each projection, dark, and white images are not specified via theta dimension scale datasets, it is assumed that the raw projections are taken at equally spaced angular intervals between 0 and 180 degree, with white and dark field collected at the same time before or after the projection data collection.
152
153 .. _MinimalTomo0:
154
155 .. figure:: figures/dx_MinimalTomo0.png
156 :align: center
157 :alt: Diagram of a minimal Data Exchange file for a single tomographic data set including raw projections, dark, and white fields.
158 :width: 50.0%
159
160 Basic Tomo A
161
162 Diagram of a minimal Data Exchange file for a single tomographic data set including raw projections, dark, and white fields
163
164 .. _MinimalTomo1:
165
166 .. figure:: figures/dx_MinimalTomo1.png
167 :align: center
168 :alt: Diagram of a single tomographic data set including raw projections, dark and white fields. In this case, there are additional dimension descriptor datasets theta, theta_dark, and theta_white that contain the positions of the rotation axis for each projection, dark, and white image. The lefthand example shows this as it would appear using the HDF5 H5DSattach_scale function. The righthand example shows this as it would appear by manually adding an axes attribute (for cases where H5DSattach_scale is unavailable).
169 :width: 80.0%
170
171 Basic Tomo B
172
173 Diagram of a single tomographic data set including raw projections,
174 dark and white fields. In this case, there are additional dimension
175 descriptor datasets theta, theta_dark, and theta_white that contain
176 the positions of the rotation axis for each projection, dark, and
177 white image. The lefthand example shows this as it would appear using
178 the HDF5 H5DSattach_scale function. The righthand example shows this
179 as it would appear by manually adding an axes attribute (for cases
180 where H5DSattach_scale is unavailable)
181
182 Imaging
183 -------
184 The examples in this section show how one can store data for imaging
185 experiments using the Data Exchange format. It is general enough,
186 however, to show how Data Exchange can be extended or adapted to other
187 techniques. These examples are meant to give a flavor for our approach.
188 A complete reference to the core structure can be found in Section
189 *Reference*. Technique specific extensions to the core structure
190 can be found at the end of the Reference Guide.
191
192 :ref:`Minimal1` shows a diagram of a minimal Data Exchange file
193 to store a single projection image. It is strongly encouraged that all
194 datasets shall have a units attribute. The axes of the dataset are not
195 specified in this minimal case, and can be assumed to be x and y with a
196 zero-based integer sequence, or more simply, pixels.
197
198 .. _Minimal1:
199
200 .. figure:: figures/dx_Minimal1.png
201 :align: center
202 :alt: Diagram of a minimal Data Exchange file for a single image.
203 :width: 50.0%
204
205 Minimal DXfile
206
207 Diagram of a minimal Data Exchange file for a single image.
208
209 Series
210 ------
211
212 A series of tomographic measurements, when relevant, can be stored in
213 the same file appending _N to the measurement tag.
214 A series of tomographic data sets are typically collected changing the
215 instrument status (energy, detector or optics position); changing the
216 sample status (position, environment etc.). Figure :ref:`MinimalTomo2`,
217 :ref:`MinimalTomo3` and :ref:`MinimalTomo4` show the content of files
218 changing the sample temperature, the X-ray source energy and
219 detector-sample distance.
220 In nano tomography experiments, for example, the detector field of view is
221 often smaller than the sample. To collect a complete tomographic data set,
222 it is necessary to raster the sample across the field of view moving its x
223 and y location. Figure :ref:`NanoTomo1` shows a file from a nano
224 tomography experiment when the sample rasters through the field of view.
225
226 There are limits to this approach, as one clearly does not want to have
227 hundreds of measurement groups in a file (or multiple files) where most
228 of the metadata is the same. For measurements where there are many
229 “positioner” values (aka a “scan”), it is more sensible to add
230 dimension(s) to the exchange dataset, and describe the “positioner”
231 values as dimension scales. This is a judgement left to the user.
232
233 Temperature
234 ~~~~~~~~~~~
235
236 .. _MinimalTomo2:
237
238 .. figure:: figures/dx_MinimalTomo2.png
239 :align: center
240 :alt: Diagram of two tomographic data sets taken at two different sample temperatures (100 and 200 Celsius).
241 :width: 100.0%
242
243 Temperature
244
245 Diagram of two tomographic data sets taken at two different sample
246 temperatures (100 and 200 Celsius)
247
248 Energy
249 ~~~~~~
250 .. _MinimalTomo3:
251
252 .. figure:: figures/dx_MinimalTomo3.png
253 :align: center
254 :alt: Diagram of two tomographic data sets taken at two different energy (10 and 20 keV).
255 :width: 80.0%
256
257 Energy
258
259 Diagram of two tomographic data sets taken at two different energy
260 (10 and 20 keV)
261
262 Detector-sample distance
263 ~~~~~~~~~~~~~~~~~~~~~~~~
264
265 .. _MinimalTomo4:
266
267 .. figure:: figures/dx_MinimalTomo4.png
268 :align: center
269 :alt: Diagram of two tomographic data sets collected with two different detector-sample distances (5 and 9 mm). Note the use of output_data dataset to associate the detector with the exchange group generated from the acquisition.
270 :width: 80.0%
271
272 Distance
273
274 Diagram of two tomographic data sets collected with two different
275 detector-sample distances (5 and 9 mm). Note the use of output_data
276 dataset to associate the detector with the exchange group generated
277 from the acquisition
278
279 Raster
280 ~~~~~~
281
282 .. _NanoTomo1:
283
284 .. figure:: figures/dx_NanoTomo1.png
285 :align: center
286 :alt: Diagram of a file with 4 tomographic data sets from a nano tomography experiment.
287 :width: 90.0%
288
289 Raster
290
291 Diagram of a file with 4 tomographic data sets from a nano tomography
292 experiment
293