Debian Codegraph @

doc/usage.rst

raw
1 =======
2 Usage
3 =======
4
5 .. currentmodule:: hdf5plugin
6
7 ``hdf5plugin`` allows using additional HDF5 compression filters with `h5py`_ for reading and writing compressed datasets.
8
9 Read compressed datasets
10 ++++++++++++++++++++++++
11
12 In order to read compressed dataset with `h5py`_, use:
13
14 .. code-block:: python
15
16 import hdf5plugin
17
18 It registers ``hdf5plugin`` supported compression filters with the HDF5 library used by `h5py`_.
19 Hence, HDF5 compressed datasets can be read as any other dataset (see `h5py documentation <https://docs.h5py.org/en/stable/high/dataset.html#reading-writing-data>`_).
20
21 .. note::
22
23 HDF5 datasets compressed with `Blosc2`_ can require additional plugins to enable decompression, such as `blosc2-grok <https://pypi.org/project/blosc2-grok>`_ or `blosc2-openhtj2k <https://pypi.org/project/blosc2-openhtj2k>`_.
24 See list of Blosc2 `filters <https://www.blosc.org/c-blosc2/reference/utility_variables.html#codes-for-filters>`_ and `codecs <https://www.blosc.org/c-blosc2/reference/utility_variables.html#compressor-codecs>`_.
25
26 Write compressed datasets
27 +++++++++++++++++++++++++
28
29 As for reading compressed datasets, ``import hdf5plugin`` is required to enable the supported compression filters.
30
31 To create a compressed dataset use `h5py.Group.create_dataset`_ and set the ``compression`` and ``compression_opts`` arguments.
32
33 ``hdf5plugin`` provides helpers to prepare those compression options: `Bitshuffle`_, `Blosc`_, `Blosc2`_, `BZip2`_, `FciDecomp`_, `LZ4`_, `Sperr`_, `SZ`_, `SZ3`_, `Zfp`_, `Zstd`_.
34
35 Sample code:
36
37 .. code-block:: python
38
39 import numpy
40 import h5py
41 import hdf5plugin
42
43 # Compression
44 f = h5py.File('test.h5', 'w')
45 f.create_dataset('data', data=numpy.arange(100), compression=hdf5plugin.LZ4())
46 f.close()
47
48 # Decompression
49 f = h5py.File('test.h5', 'r')
50 data = f['data'][()]
51 f.close()
52
53 Relevant `h5py`_ documentation: `Filter pipeline <https://docs.h5py.org/en/stable/high/dataset.html#filter-pipeline>`_ and `Chunked Storage <https://docs.h5py.org/en/stable/high/dataset.html#chunked-storage>`_.
54
55
56 Bitshuffle
57 ==========
58
59 .. autoclass:: Bitshuffle
60 :members:
61 :undoc-members:
62
63 Blosc
64 =====
65
66 .. autoclass:: Blosc
67 :members:
68 :undoc-members:
69
70 Blosc2
71 ======
72
73 .. autoclass:: Blosc2
74 :members:
75 :undoc-members:
76
77 BZip2
78 =====
79
80 .. autoclass:: BZip2
81 :members:
82 :undoc-members:
83
84 FciDecomp
85 =========
86
87 .. autoclass:: FciDecomp
88 :members:
89 :undoc-members:
90
91 LZ4
92 ===
93
94 .. autoclass:: LZ4
95 :members:
96 :undoc-members:
97
98 Sperr
99 =====
100
101 .. autoclass:: Sperr
102 :members:
103 :undoc-members:
104
105 SZ
106 ==
107
108 .. autoclass:: SZ
109 :members:
110 :undoc-members:
111
112 SZ3
113 ===
114
115 .. autoclass:: SZ3
116 :members:
117 :undoc-members:
118
119 Zfp
120 ===
121
122 .. autoclass:: Zfp
123 :members:
124 :undoc-members:
125
126 Zstd
127 ====
128
129 .. autoclass:: Zstd
130 :members:
131 :undoc-members:
132
133 Get information about hdf5plugin
134 ++++++++++++++++++++++++++++++++
135
136 Constants:
137
138 .. py:data:: PLUGIN_PATH
139
140 Directory where the provided HDF5 filter plugins are stored.
141
142 Functions:
143
144 .. autofunction:: get_filters
145
146 .. autofunction:: get_config
147
148 Manage registered filters
149 +++++++++++++++++++++++++
150
151 When imported, `hdf5plugin` initialises and registers the filters it embeds if there is no already registered filters for the corresponding filter IDs.
152
153 `h5py`_ gives access to HDF5 functions handling registered filters in `h5py.h5z`_.
154 This module allows checking the filter availability and registering/unregistering filters.
155
156 `hdf5plugin` provides an extra `register` function to register the filters it provides, e.g., to override an already loaded filters.
157 Registering with this function is required to perform additional initialisation and enable writing compressed data with the given filter.
158
159 .. autofunction:: register
160
161 Use HDF5 filters in other applications
162 ++++++++++++++++++++++++++++++++++++++
163
164 Non `h5py`_ or non-Python users can also benefit from the supplied HDF5 compression filters for reading compressed datasets by setting the ``HDF5_PLUGIN_PATH`` environment variable the value of ``hdf5plugin.PLUGIN_PATH``, which can be retrieved from the command line with::
165
166 python -c "import hdf5plugin; print(hdf5plugin.PLUGIN_PATH)"
167
168 For instance::
169
170 export HDF5_PLUGIN_PATH=$(python -c "import hdf5plugin; print(hdf5plugin.PLUGIN_PATH)")
171
172 should allow MatLab or IDL users to read data compressed using the supported plugins.
173
174 Setting the ``HDF5_PLUGIN_PATH`` environment variable allows already existing programs or Python code to read compressed data without any modification.
175
176 .. _h5py: https://www.h5py.org
177 .. _h5py.h5z: https://github.com/h5py/h5py/blob/master/h5py/h5z.pyx
178 .. _h5py.Group.create_dataset: https://docs.h5py.org/en/stable/high/group.html#h5py.Group.create_dataset