Debian Codegraph @

doc/contribute.rst

raw
1 ============
2 Contribute
3 ============
4
5 This project follows the standard open-source project github workflow, which is described in other projects like `scikit-image <https://scikit-image.org/docs/stable/development/contribute.html>`_.
6
7 Testing
8 =======
9
10 To run self-contained tests, from Python:
11
12 .. code-block:: python
13
14 import hdf5plugin.test
15 hdf5plugin.test.run_tests()
16
17 Or, from the command line::
18
19 python -m hdf5plugin.test
20
21 To also run tests relying on actual HDF5 files, run from the source directory::
22
23 python test/test.py
24
25 This tests the installed version of `hdf5plugin`.
26
27 Building documentation
28 ======================
29
30 Documentation relies on `Sphinx <https://www.sphinx-doc.org>`_.
31
32 To build documentation, run from the project root directory::
33
34 python setup.py build
35 PYTHONPATH=build/lib.<os>-<machine>-<pyver>/ sphinx-build -b html doc/ build/html
36
37 Guidelines to add a compression filter
38 ======================================
39
40 This briefly describes the steps to add a HDF5 compression filter to the zoo.
41
42 * Add the source of the HDF5 filter and compression algorithm code in a subdirectory in ``src/[filter]``.
43 Best is to use ``git subtree`` else copy the files there (including the license file).
44 A released version of the filter + compression library should be used.
45
46 ``git subtree`` command::
47
48 git subtree add --prefix=src/[filter] [git repository] [release tag] --squash
49
50 * Update ``setup.py`` to build the filter dynamic library by adding an extension using the ``HDF5PluginExtension`` class (a subclass of ``setuptools.Extension``) which adds extra files and compile options to enable dynamic loading of the filter.
51 The name of the extension should be ``hdf5plugin.plugins.libh5<filter_name>``.
52
53 * In case of import errors related to HDF5-related undefined symbols, add eventual missing functions under ``src/hdf5_dl.c``.
54
55 * Add a "CONSTANT" in ``src/hdf5plugin/_filters.py`` named with the ``FILTER_NAME_ID`` which value is the HDF5 filter ID
56 (See `HDF5 registered filters <https://github.com/HDFGroup/hdf5_plugins/blob/master/docs/RegisteredFilterPlugins.md#list-of-filters-registered-with-the-hdf-group>`_).
57
58 * Add a compression options helper class named ``FilterName`` in ``hdf5plugins/_filters.py`` which should inherit from ``_FilterRefClass``.
59 This is intended to ease the usage of ``h5py.Group.create_dataset`` ``compression_opts`` argument.
60 It must have a `filter_name` class attribute with the same name as in the extension defined in ``setup.py`` (without the ``libh5`` prefix) .
61 This name is used to find the filter library.
62
63 * Add ``FilterName`` to ``hdf5plugin._filters.FILTER_CLASSES``.
64
65 * Add to ``hdf5plugin/__init__.py`` the import of the filter ID and helper class:
66 ``from ._filters import FILTER_NAME_ID, FilterName # noqa``
67
68 * Add tests:
69
70 - In ``test/test.py`` for testing reading a compressed file that was produced with another software.
71 - In ``src/hdf5plugin/test.py`` for tests that writes data using the compression filter and the compression options helper function and reads back the data.
72
73 * Update the ``doc/information.rst`` file to document:
74
75 - The version of the HDF5 filter that is embedded in ``hdf5plugin``.
76 - The license of the filter (by adding a link to the license file).
77
78 * Update the ``doc/usage.rst`` file to document:
79
80 - The ``hdf5plugin.<FilterName>`` compression argument helper class.
81
82 * Update ``doc/contribute.rst`` to document the format of ``compression_opts`` expected by the filter
83 (see `h5py custom compression filters <https://docs.h5py.org/en/stable/high/dataset.html#custom-compression-filters>`_).
84
85 Low-level compression filter arguments
86 ======================================
87
88 Compression filters can be configured with the ``compression_opts`` argument of `h5py.Group.create_dataset <http://docs.h5py.org/en/stable/high/group.html#Group.create_dataset>`_ method by providing a tuple of integers.
89
90 The meaning of those integers is filter dependent and is described below.
91
92 bitshuffle
93 ..........
94
95 compression_opts: (**block_size**, **compression**, **level**)
96
97 - **block size**: Number of elements (not bytes) per block.
98 It MUST be a mulitple of 8.
99 Default: 0 for a block size of about 8 kB.
100 - **compression**:
101
102 * 0: No compression
103 * 2: LZ4
104 * 3: Zstd
105
106 - **level**: Compression level, only used with Zstd compression.
107
108 By default the filter uses bitshuffle, but does NOT compress with LZ4.
109
110 blosc
111 .....
112
113 compression_opts: (0, 0, 0, 0, **compression level**, **shuffle**, **compression**)
114
115 - First 4 values are reserved.
116 - **compression level**:
117 From 0 (no compression) to 9 (maximum compression).
118 Default: 5.
119 - **shuffle**: Shuffle filter:
120
121 * 0: no shuffle
122 * 1: byte shuffle
123 * 2: bit shuffle
124
125 - **compression**: The compressor blosc ID:
126
127 * 0: blosclz (default)
128 * 1: lz4
129 * 2: lz4hc
130 * 3: snappy
131 * 4: zlib
132 * 5: zstd
133
134 By default the filter uses byte shuffle and blosclz.
135
136 blosc2
137 ......
138
139 compression_opts: (0, 0, 0, 0, **compression level**, **filter**, **compression**)
140
141 - First 4 values are reserved.
142 - **compression level**:
143 From 0 (no compression) to 9 (maximum compression).
144 Default: 5.
145 - **filter**: Pre-compression filter:
146
147 * 0: no shuffle
148 * 1: byte shuffle
149 * 2: bit shuffle
150 * 3: delta: diff current block with first one
151 * 4: truncate precision: Truncate mantissa for floating point types
152
153 - **compression**: The compressor blosc ID:
154
155 * 0: blosclz (default)
156 * 1: lz4
157 * 2: lz4hc
158 * 3: unused
159 * 4: zlib
160 * 5: zstd
161
162 By default the filter uses byte shuffle and blosclz.
163
164 bzip2
165 .....
166
167 compression_opts: (**block size**,)
168
169 - **block_size**: Size of the blocks as a multiple of 100k.
170 It must be in the range [1, 9].
171
172 lz4
173 ...
174
175 compression_opts: (**block_size**,)
176
177 - **block size**: Number of bytes per block.
178 Default 0 for a block size of 1GB.
179 It MUST be < 1.9 GB.
180
181 sperr
182 .....
183
184 compression_opts: (**mode_quality_swap**,)
185
186 - **mode_quality_swap**: Store mode, quality and swap as a 32 bits unsigned integer:
187 For details see the implementation of the C function: `H5Z_SPERR_make_cd_values <https://github.com/NCAR/H5Z-SPERR/blob/v0.1.2/include/h5z-sperr.h#L21>`_
188
189 sz
190 ..
191
192 compression_opts:
193
194 - **error_bound_mode** (int32)
195 - **abs_error high** (big endian float64)
196 - **abs_error low**
197 - **rel_error high** (big endian float64)
198 - **rel_error low**
199 - **pw_rel_error high** (big endian float64)
200 - **pw_rel_error low**
201 - **psnr high** (big endian float64)
202 - **psnr low**
203
204 The `set_local` function prepends:
205
206 - For **dim size** from 2 to 5:
207
208 (**dim size**, **data type**, **r1**, **r2**, **r3** (if **dim size** >= 3), **r4** (if **dim size** >= 4), **r5** (if **dim size** == 5))
209
210 **rX** are set up to **dim size** (e.g., For **dim size** == 2 only **r1** and **r2** are used)
211
212 - For **dim size** == 1: **r1** is stored on 64 bits:
213
214 (**dim size**, **data type**, **r1 most-significant bytes**, **r1 least-significant bytes**)
215
216 sz3
217 ...
218
219 compression_opts:
220
221 - **mode**
222 - **abs_error high** (big endian float64)
223 - **abs_error low**
224 - **rel_error high** (big endian float64)
225 - **rel_error low**
226 - **norm2 high** (big endian float64)
227 - **norm2 low**
228 - **psnr high** (big endian float64)
229 - **psnr low**
230
231 zfp
232 ...
233
234 For more information, see `zfp modes <https://zfp.readthedocs.io/en/latest/modes.html>`_ and `hdf5-zfp generic interface <https://h5z-zfp.readthedocs.io/en/latest/interfaces.html#generic-interface>`_.
235
236 The first value of *compression_opts* is **mode**.
237 The following values depends on the value of **mode**:
238
239 - *Fixed-rate* mode: (1, 0, **rateHigh**, **rateLow**, 0, 0)
240 Rate, i.e., number of compressed bits per value, as a double stored as:
241
242 - **rateHigh**: High 32-bit word of the rate double.
243 - **rateLow**: Low 32-bit word of the rate double.
244
245 - *Fixed-precision* mode: (2, 0, **prec**, 0, 0, 0)
246
247 - **prec**: Number of uncompressed bits per value.
248
249 - *Fixed-accuracy* mode: (3, 0, **accHigh**, **accLow**, 0, 0)
250 Accuracy, i.e., absolute error tolerance, as a double stored as:
251
252 - **accHigh**: High 32-bit word of the accuracy double.
253 - **accLow**: Low 32-bit word of the accuracy double.
254
255 - *Expert* mode: (4, 0, **minbits**, **maxbits**, **maxprec**, **minexp**)
256
257 - **minbits**: Minimum number of compressed bits used to represent a block.
258 - **maxbits**: Maximum number of bits used to represent a block.
259 - **maxprec**: Maximum number of bit planes encoded.
260 - **minexp**: Smallest absolute bit plane number encoded.
261
262 - *Reversible* mode: (5, 0, 0, 0, 0, 0)
263
264 zstd
265 ....
266
267 compression_opts: (**clevel**,)
268
269 - **clevel**:
270 Compression level from 1 (lowest compression) to 22 (maximum compression).
271 Ultra compression extends from 20 through 22. Default: 3.