Debian Codegraph @

doc/source/reference.rst

raw
1 .. role:: math(raw) :format: html latex ..
2
3 ============== Core Reference ============== Top level (root) ================ This node represents the top level of the HDF5 file and holds some general information about the file. +---------------+----------------+-----------------------------------------+ | Member | Type | Example |
4 +===============+================+=========================================+ |**implements** | string dataset | **exchange**:*measurement*:*process* | +---------------+----------------+-----------------------------------------+ |**exchange** | group | |
5 +---------------+----------------+-----------------------------------------+ |*measurement* | group | | +---------------+----------------+-----------------------------------------+ | *process* | group | | +---------------+----------------+-----------------------------------------+ implements | | A colon separated list that shows which components are present in the file. The only **mandatory** component is **exchange**. A more general Data Exchange file also contains *measurement* and *process* information, if so these will be declared in **implements** as **exchange**:*measurement*:*process* exchange or exchange_N
6 | | The data taken from measurements or processing. Dimension descriptors within the group may also serve to describe “positioner” values involved in a scan.
7
8 measurement or measurement_N | | Description of the sample and instrument as configured for the measurement. This group is appropriate for relatively static metadata. For measurements where there are many “positioner” values (aka a “scan”), it is more sensible to add dimension(s) to the exchange dataset, and describe the “positioner” values as dimension scales rather than record the data via multiple matching *measurement* and **exchange** groups. This is a judgement left to the user.
9
10 process | | The Process group describes all the "work" that has been done. This
11 includes data processing steps that have been applied to the data as
12 well as experimental steps (e.g. data collection strategy etc.)
13 and sample preparation ahead of the experiment and during the
14 measurement (e.g. environment conditions etc.). **exchange** ============ The exchange group is where scientific datasets reside. This group contains one or more array datasets containing n-dimensional data and optional descriptions of the axes (dimension scale datasets). Exactly how this group is used is dependent on the application, however the general idea is that one exchange group contains one cohesive dataset. If, for example, the dataset is processed into some other form, then another exchange group is used to store the derived data. Multiple exchange groups are numbered consecutively as **exchange_N**. At a minimum, each exchange group should have a primary dataset named **data**. The *title* is optional.
15 +---------------+----------------+-----------------------------------------+ | Member | Type | Example |
16 +===============+================+=========================================+ | *name* | string dataset | “absorption_tomography” | +---------------+----------------+-----------------------------------------+ | *description* | string dataset | "raw absorption tomo" | +---------------+----------------+-----------------------------------------+ | **data** | array dataset | n-dimensional dataset |
17 +---------------+----------------+-----------------------------------------+ Table: Exchange Group Members
18
19
20 name | | Descriptive *name* for **data** dataset. Current types include: absorption_tomography, phase_tomography, dpc_tomography description
21 |
22 | Description.
23
24 data | | The primary scientific dataset. Additional related datasets may have any arbitrary name. Each dataset should have a units and description attribute. Discussion of dimension descriptors and optional axes attribute is covered in Section [sec:multidims]. Attribute
25 ---------
26
27 Description and units can be added as attribute to any data, both array or values,
28 inside a data exchange file. If units is omitted default is SI.
29 +---------------+------------------------+------------------------+ | Member | Type | Example |
30 +===============+========================+========================+ | *description* | string attribute | “transmission” |
31 +---------------+------------------------+------------------------+ | *units* | string attribute | *counts* | +---------------+------------------------+------------------------+ Table: data attributes *measurement* ============= This group holds sample and instrument information. These groups are designed to hold relatively static data about the sample and instrument configuration at the time of the measurement. Rapidly changing *positioner* values (aka scan) are better represented in the exchange group dataset. +---------------+----------------------+------------------------+ | Member | Type | Example |
32 +===============+======================+========================+ | instrument_ | group | | +---------------+----------------------+------------------------+ | sample_ | group | |
33 +---------------+----------------------+------------------------+ Table: Measurement Group Members
34
35 instrument | | The instrument used to collect this data.
36
37 sample | | The sample measured.
38
39 .. _instrument:
40
41 *instrument* ------------ The instrument group stores all relevant beamline components status at the beginning of a measurement. While all these fields are optional, if you do intend to include them they should appear within this parentage of groups.
42
43 +---------------------------------------------+-------------------------+-------------------------+ | Member | Type | Example |
44 +=============================================+=========================+=========================+
45 | *name* | string dataset | "XSD/2-BM" | +---------------------------------------------+-------------------------+-------------------------+ | component_1_ | group | | +---------------------------------------------+-------------------------+-------------------------+ | *component_2* | group | | +---------------------------------------------+-------------------------+-------------------------+ | *component_n* | group | | +---------------------------------------------+-------------------------+-------------------------+ | setup_ | group | | +---------------------------------------------+-------------------------+-------------------------+
46
47 Table: Instrument
48
49 name | | Name of the instrument.
50
51 component | | List of components part of the instrument. Replace *component* with the actual item name, *source*, *mirror*, etc.
52
53 detector | | The detectors that compose the instrument.
54
55 .. _component_1:
56
57 *component* ~~~~~~~~~~~ Class describing the component being used.
58 +-----------------------------+--------------------------------+---------------------------+ | Member | Type | Example | +=============================+================================+===========================+
59 | *name* | string dataset | “APS” | +-----------------------------+--------------------------------+---------------------------+ | *description* | string dataset | “APS” | +-----------------------------+--------------------------------+---------------------------+ | *arbitrary_label_1* | string dataset | “what ever” | +-----------------------------+--------------------------------+---------------------------+ | *arbitrary_label_2* | string dataset | “what ever” | +-----------------------------+--------------------------------+---------------------------+ | *arbitrary_label_n* | string dataset | “what ever” | +-----------------------------+--------------------------------+---------------------------+ | setup_ | group | | +-----------------------------+--------------------------------+---------------------------+ | geometry_ | group | | +-----------------------------+--------------------------------+---------------------------+ Table: Component Description
60
61 name | | Name.
62
63 arbitrary_label(s) | | Date and time source was measured.
64
65
66 .. _setup:
67
68 *setup*
69 ~~~~~~~
70
71 Logging instrument and beamline component setup parameters (static setup values)
72 is not defined by Data Exchange because is specific and different for each instrument
73 and beamline. To capture this information Data Exchange requires to set a *setup*
74 group under each beamline component and leaves each facility free to store what
75 is relevant for each component (list of motor positions etc.).
76 Ideally each component in the instrument list (source, shutter, attenuator etc.) should have
77 included its setup group. For setup values not associated with a specific beamline component
78 a *setup* group in the instrument group should be created.
79 +----------------------------------------------+----------------------------------+----------------------------------+ | Member | Type | Example |
80 +==============================================+==================================+==================================+ | *positioner_x* | float | -10.107 | +----------------------------------------------+----------------------------------+----------------------------------+ | *positioner_y* | float | -17.900 | +----------------------------------------------+----------------------------------+----------------------------------+ | *positioner_z* | float | -5.950 | +----------------------------------------------+----------------------------------+----------------------------------+ Table: Setup Group Members
81
82
83 .. _geometry:
84
85 *geometry* ^^^^^^^^^^
86
87 The geometry group is common to many of the subgroups under measurement. The intent is to describe the translation and rotation (orientation) of the sample or instrument component relative to some coordinate system. Since we believe it is not possible to determine all possible uses at this time, we leave the precise definition of geometry up to the technique. We do encourage the use of separate translation and orientation subgroups within geometry. As such, we do not describe geometry further here. This class holds the general position and
88 orientation of a component.
89
90 +---------------+------------------------+------------------------+ | Member | Type | Example |
91 +===============+========================+========================+ | *translation* | group | | +---------------+------------------------+------------------------+ | *orientation* | group | |
92 +---------------+------------------------+------------------------+ translation | | The position of the object with respect to the origin of your coordinate system. orientation | | The rotation of the object with respect to your coordinate system.
93
94 .. _translation:
95
96 *translation*
97 `````````````
98
99 This is the description for the general spatial location of a component
100 for tomography.
101
102 +----------------------------+------------------------+-----------------+
103 | Member | Type | Example |
104 +============================+========================+=================+
105 | *distances* | 3 float array dataset | (0, 0.001, 0) |
106 +----------------------------+------------------------+-----------------+
107
108 distances
109 |
110 | The x, y and z components of the translation of the origin of the object
111 | relative to the origin of the global coordinate system (the place where
112 | the X-ray beam meets the sample when the sample is first aligned in the beam).
113 | If distances does not have the attribute units set then the units are in
114 | meters.
115
116 .. _orientation:
117
118 *orientation*
119 `````````````
120
121 This is the description for the orientation of a component for
122 tomography.
123
124 +----------------------------+------------------------+-----------------+
125 | Member | Type | Example |
126 +============================+========================+=================+
127 | *value* | 6 float array dataset | |
128 +----------------------------+------------------------+-----------------+
129
130 value
131 |
132 | Dot products between the local and the global unit vectors. Unitless
133
134
135 The orientation information is stored as direction cosines. The
136 direction cosines will be between the local coordinate directions and
137 the global coordinate directions. The unit vectors in both the local and
138 global coordinates are right-handed and orthonormal.
139
140 Calling the local unit vectors (x', y',z') and the reference unit
141 vectors (x, y, z) the six numbers will be
142
143
144 .. math:: [x \cdot x, x' \cdot y, x' \cdot z, y' \cdot x, y' \cdot y, y' \cdot z]
145
146 where
147
148 .. math:: `\cdot`
149
150 is the scalar dot product (cosine of the angle between the unit vectors).
151
152 Notice that this corresponds to the first two rows of the rotation
153 matrix that transforms from the global orientation to the local
154 orientation. The third row can be recovered by using the fact that the
155 basis vectors are orthonormal.
156
157
158 .. _sample:
159
160 *sample* -------- This group holds basic information about the sample, its geometry, properties, the sample owner (user) and sample proposal information. While all these fields are optional, if you do intend to include them they should appear within this parentage of groups.
161
162
163 +-------------------------------------+------------------------------------+-----------------------------+
164 | Member | Type | Example |
165 +=====================================+====================================+=============================+
166 | *name* | string dataset | "cells sample 1" |
167 +-------------------------------------+------------------------------------+-----------------------------+
168 | *description* | string dataset | "malaria cells" |
169 +-------------------------------------+------------------------------------+-----------------------------+
170 | *preparation_date* | string dataset (ISO 8601) | "2012-07-31T21:15:22+0600" |
171 +-------------------------------------+------------------------------------+-----------------------------+
172 | *chemical_formula* | string dataset (abbr. CIF format) | "(Cd 2+)3, 2(H2 O)" |
173 +-------------------------------------+------------------------------------+-----------------------------+
174 | *mass* | float dataset | 0.25 |
175 +-------------------------------------+------------------------------------+-----------------------------+
176 | *concentration* | float dataset | 0.4 |
177 +-------------------------------------+------------------------------------+-----------------------------+
178 | *environment* | string dataset | "air" |
179 +-------------------------------------+------------------------------------+-----------------------------+
180 | *temperature* | float dataset | 25.4 |
181 +-------------------------------------+------------------------------------+-----------------------------+
182 | *temperature_set* | float dataset | 26.0 |
183 +-------------------------------------+------------------------------------+-----------------------------+
184 | *pressure* | float dataset | 101325 |
185 +-------------------------------------+------------------------------------+-----------------------------+
186 | *thickness* | float dataset | 0.001 |
187 +-------------------------------------+------------------------------------+-----------------------------+
188 | *position* | string dataset | "2D" APS robot coord. |
189 +-------------------------------------+------------------------------------+-----------------------------+
190 | geometry_ | group | |
191 +-------------------------------------+------------------------------------+-----------------------------+
192 | setup_ | group | |
193 +-------------------------------------+------------------------------------+-----------------------------+
194 | experiment_ | group | |
195 +-------------------------------------+------------------------------------+-----------------------------+
196 | experimenter_ | group | |
197 +-------------------------------------+------------------------------------+-----------------------------+
198 Table: Sample Group Members
199
200 name | | Descriptive name of the sample.
201
202 description | | Description of the sample. preparation_date
203 | | Date and time the sample was prepared.
204
205 chemical_formula | | Sample chemical formula using the CIF format.
206
207 mass | | Mass of the sample. concentration
208 | | Mass/volume. environment
209 | | Sample environment. temperature
210 | | Sample temperature. temperature_set
211 | | Sample temperature set point. pressure
212 | | Sample pressure.
213
214 thickness | | Sample thickness. position
215 | | Sample position in the sample changer/robot.
216
217 geometry | | Sample center of mass position and orientation. experiment
218 | | Facility experiment identifiers. experimenter
219 | | Experimenter identifiers.
220 *experiment* ~~~~~~~~~~~~ This provides references to facility ids for the proposal, scheduled activity, and safety form. +---------------+-------------------------+----------------------+ | Member | Type | Example | +===============+=========================+======================+
221 | *proposal* | string dataset | “1234” | +---------------+-------------------------+----------------------+ | *activity* | string dataset | “9876” | +---------------+-------------------------+----------------------+ | *safety* | string dataset | “9876” | +---------------+-------------------------+----------------------+ Table: Experiment Group Members
222
223 proposal | | Proposal reference number. For the APS this is the General User | Proposal number.
224
225 activity | | Proposal scheduler id. For the APS this is the beamline scheduler activity id.
226
227 safety | | Safety reference document. For the APS this is the Experiment | Safety Approval Form number. *experimenter* ~~~~~~~~~~~~~~ Description of a single experimenter. Multiple experimenters can be represented through numbered entries such as experimenter_1, experimenter_2. +--------------------+-------------------------+--------------------------------------------+ | Member | Type | Example |
228 +====================+=========================+============================================+
229 | *name* | string dataset | “John Doe” | +--------------------+-------------------------+--------------------------------------------+ | *role* | string dataset | “Project PI” | +--------------------+-------------------------+--------------------------------------------+ | *affiliation* | string dataset | “University of California, Berkeley” | +--------------------+-------------------------+--------------------------------------------+ | *address* | string dataset | “EPS UC Berkeley CA 94720 4767 USA” | +--------------------+-------------------------+--------------------------------------------+ | *phone* | string dataset | “+1 123 456 0000” | +--------------------+-------------------------+--------------------------------------------+ | *email* | string dataset | “[email protected]” | +--------------------+-------------------------+--------------------------------------------+ | *facility_user_id* | string dataset | “a123456” | +--------------------+-------------------------+--------------------------------------------+ Table: Experimenter Group Members name: User name. role: User role. affiliation: User affiliation. address: User address. phoen: User phone number. email: User e-mail address facility_user_id: User badge number
230
231 *process*
232 ============
233
234 Process is the documentation of the data collection strategy
235 (*acquisition*) steps, all transformations, analyses and interpretations
236 of data performed by a sequence of process functions (*actor*) as well
237 as any sample preparation step done ahead of the experiment and during the
238 measurement (e.g. environment conditions etc.).
239
240 Maintaining this history, also called provenance, allows for reproducible
241 data. The Data Exchange format tracks process by allowing each actor
242 to append process information to a process table.
243
244 The process table tracks provenance in the execution order as a series
245 of processing steps by appending sequential actor entries in the process
246 table.
247
248 +-------------------------------------+------------------------------------+-----------------------------+
249 | Member | Type | Example |
250 +=====================================+====================================+=============================+
251 | *name* | string dataset | "name" |
252 +-------------------------------------+------------------------------------+-----------------------------+
253 | *description* | string dataset | "optional" |
254 +-------------------------------------+------------------------------------+-----------------------------+
255 | actor_1_ | group | |
256 +-------------------------------------+------------------------------------+-----------------------------+
257 | *actor_2* | group | |
258 +-------------------------------------+------------------------------------+-----------------------------+
259 | *actor_n* | group | |
260 +-------------------------------------+------------------------------------+-----------------------------+
261 | table_ | group | |
262 +-------------------------------------+------------------------------------+-----------------------------+
263 Table: Process Group Members
264
265 name | | Descriptive process task.
266
267 description | | Description of the process task.
268
269 .. _actor_1:
270
271 *actor*
272 -------
273
274 This is the actor description group. Each entry of the process table_ will refer to the correspondent
275 actor description.
276
277
278 +-------------------------------------+------------------------------------+---------------------------------------------+
279 | Member | Type | Example |
280 +=====================================+====================================+=============================================+
281 | *name* | string dataset | "test rec" |
282 +-------------------------------------+------------------------------------+---------------------------------------------+
283 | *description* | string dataset | "optional" |
284 +-------------------------------------+------------------------------------+---------------------------------------------+
285 | *version* | string dataset | https://github.com/tomopy_scripts/b9ad87e17 |
286 +-------------------------------------+------------------------------------+---------------------------------------------+
287 | *input_data* | string dataset | "/exchange" | +-------------------------------------+------------------------------------+---------------------------------------------+
288 | *output_data* | string dataset | "/exchange_1" | +-------------------------------------+------------------------------------+---------------------------------------------+
289 | set-up_ | group | |
290 +-------------------------------------+------------------------------------+---------------------------------------------+
291
292
293 Table: Actor Group Members
294
295 name | | Descriptive actor task.
296
297 description | | Description of the actor task.
298
299 version | | Version of the actor task.
300 | | If available this can be the repository link to the actor version used
301 | https://github.com/tomopy_scripts/b9ad87e17
302 input_data, output_data
303 | | Origin and destination of the data processed by the actor.
304
305 .. _set-up:
306
307
308 setup (actor)
309 ~~~~~~~~~~~~~
310
311 Here is where to log the actor setup parameters (static setup values).
312
313 +----------------------------------------------+------------------------------------+-----------------------------------------------+
314 | Member | Type | Example |
315 +==============================================+====================================+===============================================+
316 | *parameter_name_1* | float | 0.0 |
317 +----------------------------------------------+------------------------------------+-----------------------------------------------+
318 | *parameter_name_2* | string dataset | "Parzen" |
319 +----------------------------------------------+------------------------------------+-----------------------------------------------+
320 | *parameter_name_n* | float | 2.0 |
321 +----------------------------------------------+------------------------------------+-----------------------------------------------+
322 | *module__name_1* | string dataset | https://github.com/astra/b9ad87e17 |
323 +----------------------------------------------+------------------------------------+-----------------------------------------------+
324 | *module_name_2* | string dataset | https://github.com/tomopy/c9ad87e77 |
325 +----------------------------------------------+------------------------------------+-----------------------------------------------+
326
327 Table: Actor Setup Group
328
329 .. table:
330
331 *table*
332 -------
333
334 Scientific users will not generally be expected to maintain data in this
335 group. The expectation is that the data collection and analysis pipeline
336 tools will automatically record process steps using this group.
337 In addition, it is possible to re-run an analysis using the information
338 provided here.
339 +-----------+-------------------+-------------------+---------------+----------------------+--------------------------+-------------------------------------+ | actor | start_time | end_time | status | message | reference | description | +===========+===================+===================+===============+======================+==========================+=====================================+
340 | actor_1 | 21:15:22 | 21:15:23 | SUCCESS | OK | /process/actor_1 | raw data collection | +-----------+-------------------+-------------------+---------------+----------------------+--------------------------+-------------------------------------+ | actor_2 | 21:15:26 | 21:15:27 | RUNNING | OK | /process/actor_2 | reconstruct | +-----------+-------------------+-------------------+---------------+----------------------+--------------------------+-------------------------------------+ | actor_n | 21:17:28 | 22:15:22 | QUEUED | OK | /process/actor_n | transfer data to user | +-----------+-------------------+-------------------+---------------+----------------------+--------------------------+-------------------------------------+ Table: Process table to log actors activity
341
342 actor | | Name of the process in the pipeline stage that is executed at this step. *start_time* | | Time the process started. *end_time* | | TIme the process ended. *status* | | Current process status. May be one of the following: QUEUED, | RUNNING, FAILED, or SUCCESS. *message* | | A process specific message generated by the process. It may be a | confirmation that the process was successful, or a detailed error | message, for example. *reference* | | Path to the actor description group. The process description group | contains all metadata to perform the specific process. This | reference is simply the HDF5 path within this file of the | technique specific process description group. The process | description group should contain all parameters necessary to run | the process, including the name and version of any external | analysis tool used to process the data. It should also contain | input and output references that point to the | **exchange_N** groups that contain the input and output | datasets of the process. *description* | | Process description.