| 1 |
======= |
| 2 |
Usage |
| 3 |
======= |
| 4 |
|
| 5 |
.. currentmodule:: hdf5plugin |
| 6 |
|
| 7 |
``hdf5plugin`` allows using additional HDF5 compression filters with `h5py`_ for reading and writing compressed datasets. |
| 8 |
|
| 9 |
Read compressed datasets |
| 10 |
++++++++++++++++++++++++ |
| 11 |
|
| 12 |
In order to read compressed dataset with `h5py`_, use: |
| 13 |
|
| 14 |
.. code-block:: python |
| 15 |
|
| 16 |
import hdf5plugin |
| 17 |
|
| 18 |
It registers ``hdf5plugin`` supported compression filters with the HDF5 library used by `h5py`_. |
| 19 |
Hence, HDF5 compressed datasets can be read as any other dataset (see `h5py documentation <https://docs.h5py.org/en/stable/high/dataset.html#reading-writing-data>`_). |
| 20 |
|
| 21 |
.. note:: |
| 22 |
|
| 23 |
HDF5 datasets compressed with `Blosc2`_ can require additional plugins to enable decompression, such as `blosc2-grok <https://pypi.org/project/blosc2-grok>`_ or `blosc2-openhtj2k <https://pypi.org/project/blosc2-openhtj2k>`_. |
| 24 |
See list of Blosc2 `filters <https://www.blosc.org/c-blosc2/reference/utility_variables.html#codes-for-filters>`_ and `codecs <https://www.blosc.org/c-blosc2/reference/utility_variables.html#compressor-codecs>`_. |
| 25 |
|
| 26 |
Write compressed datasets |
| 27 |
+++++++++++++++++++++++++ |
| 28 |
|
| 29 |
As for reading compressed datasets, ``import hdf5plugin`` is required to enable the supported compression filters. |
| 30 |
|
| 31 |
To create a compressed dataset use `h5py.Group.create_dataset`_ and set the ``compression`` and ``compression_opts`` arguments. |
| 32 |
|
| 33 |
``hdf5plugin`` provides helpers to prepare those compression options: `Bitshuffle`_, `Blosc`_, `Blosc2`_, `BZip2`_, `FciDecomp`_, `LZ4`_, `Sperr`_, `SZ`_, `SZ3`_, `Zfp`_, `Zstd`_. |
| 34 |
|
| 35 |
Sample code: |
| 36 |
|
| 37 |
.. code-block:: python |
| 38 |
|
| 39 |
import numpy |
| 40 |
import h5py |
| 41 |
import hdf5plugin |
| 42 |
|
| 43 |
# Compression |
| 44 |
f = h5py.File('test.h5', 'w') |
| 45 |
f.create_dataset('data', data=numpy.arange(100), compression=hdf5plugin.LZ4()) |
| 46 |
f.close() |
| 47 |
|
| 48 |
# Decompression |
| 49 |
f = h5py.File('test.h5', 'r') |
| 50 |
data = f['data'][()] |
| 51 |
f.close() |
| 52 |
|
| 53 |
Relevant `h5py`_ documentation: `Filter pipeline <https://docs.h5py.org/en/stable/high/dataset.html#filter-pipeline>`_ and `Chunked Storage <https://docs.h5py.org/en/stable/high/dataset.html#chunked-storage>`_. |
| 54 |
|
| 55 |
|
| 56 |
Bitshuffle |
| 57 |
========== |
| 58 |
|
| 59 |
.. autoclass:: Bitshuffle |
| 60 |
:members: |
| 61 |
:undoc-members: |
| 62 |
|
| 63 |
Blosc |
| 64 |
===== |
| 65 |
|
| 66 |
.. autoclass:: Blosc |
| 67 |
:members: |
| 68 |
:undoc-members: |
| 69 |
|
| 70 |
Blosc2 |
| 71 |
====== |
| 72 |
|
| 73 |
.. autoclass:: Blosc2 |
| 74 |
:members: |
| 75 |
:undoc-members: |
| 76 |
|
| 77 |
BZip2 |
| 78 |
===== |
| 79 |
|
| 80 |
.. autoclass:: BZip2 |
| 81 |
:members: |
| 82 |
:undoc-members: |
| 83 |
|
| 84 |
FciDecomp |
| 85 |
========= |
| 86 |
|
| 87 |
.. autoclass:: FciDecomp |
| 88 |
:members: |
| 89 |
:undoc-members: |
| 90 |
|
| 91 |
LZ4 |
| 92 |
=== |
| 93 |
|
| 94 |
.. autoclass:: LZ4 |
| 95 |
:members: |
| 96 |
:undoc-members: |
| 97 |
|
| 98 |
Sperr |
| 99 |
===== |
| 100 |
|
| 101 |
.. autoclass:: Sperr |
| 102 |
:members: |
| 103 |
:undoc-members: |
| 104 |
|
| 105 |
SZ |
| 106 |
== |
| 107 |
|
| 108 |
.. autoclass:: SZ |
| 109 |
:members: |
| 110 |
:undoc-members: |
| 111 |
|
| 112 |
SZ3 |
| 113 |
=== |
| 114 |
|
| 115 |
.. autoclass:: SZ3 |
| 116 |
:members: |
| 117 |
:undoc-members: |
| 118 |
|
| 119 |
Zfp |
| 120 |
=== |
| 121 |
|
| 122 |
.. autoclass:: Zfp |
| 123 |
:members: |
| 124 |
:undoc-members: |
| 125 |
|
| 126 |
Zstd |
| 127 |
==== |
| 128 |
|
| 129 |
.. autoclass:: Zstd |
| 130 |
:members: |
| 131 |
:undoc-members: |
| 132 |
|
| 133 |
Get information about hdf5plugin |
| 134 |
++++++++++++++++++++++++++++++++ |
| 135 |
|
| 136 |
Constants: |
| 137 |
|
| 138 |
.. py:data:: PLUGIN_PATH |
| 139 |
|
| 140 |
Directory where the provided HDF5 filter plugins are stored. |
| 141 |
|
| 142 |
Functions: |
| 143 |
|
| 144 |
.. autofunction:: get_filters |
| 145 |
|
| 146 |
.. autofunction:: get_config |
| 147 |
|
| 148 |
Manage registered filters |
| 149 |
+++++++++++++++++++++++++ |
| 150 |
|
| 151 |
When imported, `hdf5plugin` initialises and registers the filters it embeds if there is no already registered filters for the corresponding filter IDs. |
| 152 |
|
| 153 |
`h5py`_ gives access to HDF5 functions handling registered filters in `h5py.h5z`_. |
| 154 |
This module allows checking the filter availability and registering/unregistering filters. |
| 155 |
|
| 156 |
`hdf5plugin` provides an extra `register` function to register the filters it provides, e.g., to override an already loaded filters. |
| 157 |
Registering with this function is required to perform additional initialisation and enable writing compressed data with the given filter. |
| 158 |
|
| 159 |
.. autofunction:: register |
| 160 |
|
| 161 |
Use HDF5 filters in other applications |
| 162 |
++++++++++++++++++++++++++++++++++++++ |
| 163 |
|
| 164 |
Non `h5py`_ or non-Python users can also benefit from the supplied HDF5 compression filters for reading compressed datasets by setting the ``HDF5_PLUGIN_PATH`` environment variable the value of ``hdf5plugin.PLUGIN_PATH``, which can be retrieved from the command line with:: |
| 165 |
|
| 166 |
python -c "import hdf5plugin; print(hdf5plugin.PLUGIN_PATH)" |
| 167 |
|
| 168 |
For instance:: |
| 169 |
|
| 170 |
export HDF5_PLUGIN_PATH=$(python -c "import hdf5plugin; print(hdf5plugin.PLUGIN_PATH)") |
| 171 |
|
| 172 |
should allow MatLab or IDL users to read data compressed using the supported plugins. |
| 173 |
|
| 174 |
Setting the ``HDF5_PLUGIN_PATH`` environment variable allows already existing programs or Python code to read compressed data without any modification. |
| 175 |
|
| 176 |
.. _h5py: https://www.h5py.org |
| 177 |
.. _h5py.h5z: https://github.com/h5py/h5py/blob/master/h5py/h5z.pyx |
| 178 |
.. _h5py.Group.create_dataset: https://docs.h5py.org/en/stable/high/group.html#h5py.Group.create_dataset |