| 1 |
\documentclass{article} |
| 2 |
\usepackage{graphicx} % Required for inserting images |
| 3 |
\usepackage[colorlinks]{hyperref} |
| 4 |
\usepackage[margin=1in]{geometry} |
| 5 |
\usepackage[many]{tcolorbox} % for COLORED BOXES (tikz and xcolor included) |
| 6 |
\usepackage{minted} |
| 7 |
\usepackage{xspace} |
| 8 |
\setcounter{tocdepth}{1} |
| 9 |
\setminted{linenos} |
| 10 |
\setlength\parindent{0pt} % killing indentation for all the text |
| 11 |
\setlength\columnsep{0.25in} % setting length of column separator |
| 12 |
\pagestyle{empty} % setting pagestyle to be empty |
| 13 |
\definecolor{main}{HTML}{5989cf} % setting main color to be used |
| 14 |
\definecolor{sub}{HTML}{cde4ff} % setting sub color to be used |
| 15 |
\tcbset{ |
| 16 |
sharp corners, |
| 17 |
colback = white, |
| 18 |
before skip = 0.2cm, % add extra space before the box |
| 19 |
after skip = 0.5cm % add extra space after the box |
| 20 |
} % setting global options for tcolorbox |
| 21 |
\newtcolorbox{BlueBox}{ |
| 22 |
sharpish corners, % better drop shadow |
| 23 |
colback = sub, |
| 24 |
colframe = main, |
| 25 |
boxrule = 0pt, |
| 26 |
toprule = 4.5pt, % top rule weight |
| 27 |
enhanced, |
| 28 |
fuzzy shadow = {0pt}{-2pt}{-0.5pt}{0.5pt}{black!35} % {xshift}{yshift}{offset}{step}{options} |
| 29 |
} |
| 30 |
\newcommand{\callout}[1]{\begin{BlueBox}#1\end{BlueBox}} |
| 31 |
|
| 32 |
|
| 33 |
|
| 34 |
\title{SPERR Interface and Examples} |
| 35 |
\author{Samuel Li} |
| 36 |
\date{\today} |
| 37 |
|
| 38 |
\begin{document} |
| 39 |
|
| 40 |
\vspace{-1cm} |
| 41 |
\maketitle |
| 42 |
|
| 43 |
\vspace{-1cm} |
| 44 |
|
| 45 |
\section{Introduction} |
| 46 |
This document is the SPERR section of the handout for the |
| 47 |
\href{https://szcompressor.org/next.szcompressor.github.io/meetings/feb24fl/}{FZ workshop} |
| 48 |
hands-on session on February 15 2024, Sarasota, FL. |
| 49 |
It provides examples for the CLI interface, C++ interface, and C interface of SPERR |
| 50 |
(Section~\ref{sec:cli}, \ref{sec:cpp}, and \ref{sec:c}, respectively). |
| 51 |
|
| 52 |
\section{SPERR} |
| 53 |
Contact: \href{[email protected]}{Samuel Li ([email protected])} \\ |
| 54 |
Repo: \href{https://github.com/NCAR/SPERR/}{github.com/NCAR/SPERR/} \\ |
| 55 |
Wiki: \href{https://github.com/NCAR/SPERR/wiki}{github.com/NCAR/SPERR/wiki} |
| 56 |
|
| 57 |
\vspace{2mm} |
| 58 |
SPERR uses \textit{wavelet transforms} to decorrelate the data, encodes the quantized |
| 59 |
coefficients, and explicitly corrects any data point exceeding a prescribed point-wise error |
| 60 |
(PWE) tolerance. |
| 61 |
Most often, SPERR produces the smallest compressed bitstream |
| 62 |
honoring a PWE tolerance. |
| 63 |
|
| 64 |
A SPERR bitstream can be used to reconstruct the data fields |
| 65 |
in two additional fashions: \textit{flexible-rate} decoding and |
| 66 |
\textit{multi-resolution} decoding. |
| 67 |
|
| 68 |
\vspace{-2mm} |
| 69 |
\begin{itemize} |
| 70 |
\item \textit{Flexible-rate} decoding: any prefix of a SPERR bitstream |
| 71 |
(i.e., a sub-bitstream that starts from the very beginning) |
| 72 |
produced by a simple truncation is still valid to reconstruct the data |
| 73 |
field, though at a lower quality. |
| 74 |
This ability is useful for applications such as tiered storage and data sharing |
| 75 |
over slow connections, to name a few. |
| 76 |
\vspace{-2mm} |
| 77 |
\item \textit{Multi-resolution} decoding: a hierarchy of the data field |
| 78 |
with coarsened resolutions can be obtained together with the |
| 79 |
native resolution. |
| 80 |
This ability is useful for quick analyses with limited resources. |
| 81 |
\end{itemize} |
| 82 |
|
| 83 |
\callout{On a Unix-like system with a working C++ compiler and CMake, SPERR can be |
| 84 |
built from source and made available to users in just a few commands. |
| 85 |
See this \href{https://github.com/NCAR/SPERR?tab=readme-ov-file\#quick-build}{README} |
| 86 |
for detail.} |
| 87 |
|
| 88 |
|
| 89 |
\subsection{CLI Interface} |
| 90 |
\label{sec:cli} |
| 91 |
Upon a successful build, four CLI utility programs are placed in the \texttt{./bin/} |
| 92 |
directory; three of them are most relevant for end users: \texttt{sperr2d}, |
| 93 |
\texttt{sperr3d}, and \texttt{sperr3d\_trunc}; |
| 94 |
each of them can be invoked with the \texttt{-h} option to display a help message. |
| 95 |
|
| 96 |
\subsubsection{\texttt{sperr2d}} |
| 97 |
\texttt{sperr2d} is responsible for compressing and decompressing a 2D data slice. |
| 98 |
In compression mode (\texttt{-c}), it takes as input a raw binary file consisting of |
| 99 |
single- or double-precision floating point values, and outputs a compressed bitstream. |
| 100 |
In decompression mode (\texttt{-d}), it takes as input a compressed bitstream, and |
| 101 |
outputs the decompressed binary file of single- or double-precision floating point values. |
| 102 |
Its help message contains all the options \texttt{sperr2d} takes: |
| 103 |
|
| 104 |
\begin{minted}{bash} |
| 105 |
$ ./bin/sperr2d -h |
| 106 |
|
| 107 |
Usage: ./bin/sperr2d [OPTIONS] [filename] |
| 108 |
|
| 109 |
Positionals: |
| 110 |
filename TEXT:FILE A data slice to be compressed, or |
| 111 |
a bitstream to be decompressed. |
| 112 |
|
| 113 |
Options: |
| 114 |
-h,--help Print this help message and exit |
| 115 |
|
| 116 |
Execution settings: |
| 117 |
-c Excludes: -d Perform a compression task. |
| 118 |
-d Excludes: -c Perform a decompression task. |
| 119 |
|
| 120 |
Input properties: |
| 121 |
--ftype UINT Specify the input float type in bits. Must be 32 or 64. |
| 122 |
--dims [UINT,UINT] Dimensions of the input slice. E.g., `--dims 128 128` |
| 123 |
(The fastest-varying dimension appears first.) |
| 124 |
|
| 125 |
Output settings: |
| 126 |
--bitstream TEXT Output compressed bitstream. |
| 127 |
--decomp_f TEXT Output decompressed slice in f32 precision. |
| 128 |
--decomp_d TEXT Output decompressed slice in f64 precision. |
| 129 |
--decomp_lowres_f TEXT Output lower resolutions of the decompressed slice in f32 precision. |
| 130 |
--decomp_lowres_d TEXT Output lower resolutions of the decompressed slice in f64 precision. |
| 131 |
--print_stats Needs: -c Show statistics measuring the compression quality. |
| 132 |
|
| 133 |
Compression settings (choose one): |
| 134 |
--pwe FLOAT Excludes: --psnr --bpp |
| 135 |
Maximum point-wise error (PWE) tolerance. |
| 136 |
--psnr FLOAT Excludes: --pwe --bpp |
| 137 |
Target PSNR to achieve. |
| 138 |
--bpp FLOAT:FLOAT in [0 - 64] Excludes: --pwe --psnr |
| 139 |
Target bit-per-pixel (bpp) to achieve. |
| 140 |
\end{minted} |
| 141 |
|
| 142 |
Examples: |
| 143 |
\begin{enumerate} |
| 144 |
\item Compress a 2D slice in $512\times512$ dimension, single-precision floats |
| 145 |
with a PWE tolerance of $10^{-2}$: \\ |
| 146 |
\texttt{./bin/sperr2d -c --ftype 32 --dims 512 512 --pwe 1e-2 |
| 147 |
\textbackslash \\ |
| 148 |
--bitstream ./out.stream ./in.f32} |
| 149 |
|
| 150 |
\item Perform the compression task described above, plus also write out the |
| 151 |
compress-decompressed slice, and finally print statistics |
| 152 |
measuring the compression quality: \\ |
| 153 |
\texttt{./bin/sperr2d -c --ftype 32 --dims 512 512 --pwe 1e-2 |
| 154 |
\textbackslash \\ |
| 155 |
--decomp\_f ./out.decomp --print\_stats |
| 156 |
--bitstream ./out.stream ./in.f32} |
| 157 |
|
| 158 |
\item Decompress from a SPERR bitstream, and write out the slice in native |
| 159 |
and coarsened resolutions: \\ |
| 160 |
\texttt{./bin/sperr2d -d --decomp\_f ./out.decomp --decomp\_lowres\_f ./out.lowres |
| 161 |
./sperr.stream} |
| 162 |
In this example, the output file \texttt{out.decomp} will be of the native resolution, |
| 163 |
and six other files (\texttt{out.lowres.256x256, out.lowres.128x128, etc.}) |
| 164 |
will also be produced with their filenames indicating the coarsened resolution. |
| 165 |
\end{enumerate} |
| 166 |
|
| 167 |
\subsubsection{sperr3d} |
| 168 |
\label{sec:sperr3d} |
| 169 |
\texttt{sperr3d} is responsible for compressing and decompressing a 2D data volume. |
| 170 |
Similar to \texttt{sperr2d}, it operates in either compression (\texttt{-c}) or |
| 171 |
decompression (\texttt{-d}) mode, converting between raw binary floating-point values |
| 172 |
and compressed bitstreams. |
| 173 |
Different from \texttt{sperr2d} which compresses the input 2D slice as a whole, |
| 174 |
\texttt{sperr3d} divides an input 3D volume into smaller chunks, and then compresses |
| 175 |
each chunk individually. |
| 176 |
This chunking step allows for compressing and decompressing all the small chunks |
| 177 |
individually and in parallel. |
| 178 |
\texttt{sperr3d} uses $256^3$ as the default chunk size, but any number from dozens to |
| 179 |
low hundreds would work well. |
| 180 |
(Chunk dimensions that can divide the full volume are preferred, but not mandatory.) |
| 181 |
Command line options \texttt{--chunks} and \texttt{--omp} control the chunking and |
| 182 |
parallel execution behavior respectivelly. |
| 183 |
The help message of \texttt{sperr3d} details all the options it takes: |
| 184 |
|
| 185 |
\begin{minted}{bash} |
| 186 |
$ ./bin/sperr3d -h |
| 187 |
|
| 188 |
Usage: ./bin/sperr3d [OPTIONS] [filename] |
| 189 |
|
| 190 |
Positionals: |
| 191 |
filename TEXT:FILE A data volume to be compressed, or |
| 192 |
a bitstream to be decompressed. |
| 193 |
|
| 194 |
Options: |
| 195 |
-h,--help Print this help message and exit |
| 196 |
|
| 197 |
Execution settings: |
| 198 |
-c Excludes: -d Perform a compression task. |
| 199 |
-d Excludes: -c Perform a decompression task. |
| 200 |
--omp UINT Number of OpenMP threads to use. Default (or 0) to use all. |
| 201 |
|
| 202 |
Input properties (for compression): |
| 203 |
--ftype UINT Specify the input float type in bits. Must be 32 or 64. |
| 204 |
--dims [UINT,UINT,UINT] Dimensions of the input volume. E.g., `--dims 128 128 128` |
| 205 |
(The fastest-varying dimension appears first.) |
| 206 |
|
| 207 |
Output settings: |
| 208 |
--bitstream TEXT Output compressed bitstream. |
| 209 |
--decomp_f TEXT Output decompressed volume in f32 precision. |
| 210 |
--decomp_d TEXT Output decompressed volume in f64 precision. |
| 211 |
--decomp_lowres_f TEXT Output lower resolutions of the decompressed volume in f32 precision. |
| 212 |
--decomp_lowres_d TEXT Output lower resolutions of the decompressed volume in f64 precision. |
| 213 |
--print_stats Needs: -c Print statistics measuring the compression quality. |
| 214 |
|
| 215 |
Compression settings: |
| 216 |
--chunks [UINT,UINT,UINT] Dimensions of the preferred chunk size. Default: 256 256 256 |
| 217 |
(Volume dims don't need to be divisible by these chunk dims.) |
| 218 |
--pwe FLOAT Excludes: --psnr --bpp |
| 219 |
Maximum point-wise error (PWE) tolerance. |
| 220 |
--psnr FLOAT Excludes: --pwe --bpp |
| 221 |
Target PSNR to achieve. |
| 222 |
--bpp FLOAT:FLOAT in [0 - 64] Excludes: --pwe --psnr |
| 223 |
Target bit-per-pixel (bpp) to achieve. |
| 224 |
\end{minted} |
| 225 |
|
| 226 |
Examples: |
| 227 |
\begin{enumerate} |
| 228 |
\item Compress a 3D volume in $384\times384\times256$ dimension, double-precision floats, |
| 229 |
using a PWE tolerance of $10^{-9}$ and chunks of $192\times192\times256$: \\ |
| 230 |
\texttt{./bin/sperr3d -c --omp 4 --ftype 64 --dims 384 384 256 --chunks 192 192 256 |
| 231 |
\textbackslash \\ |
| 232 |
--pwe 1e-9 --bitstream ./out.stream ./in.f64} |
| 233 |
\item Perform the compression task described above, plus also write out the |
| 234 |
compress-decompressed volume, and finally print statistics |
| 235 |
measuring the compression quality: \\ |
| 236 |
\texttt{./bin/sperr3d -c --omp 4 --ftype 64 --dims 384 384 256 --chunks 192 192 256 |
| 237 |
\textbackslash \\ |
| 238 |
--pwe 1e-9 --decomp\_d ./out.decomp --print\_stats --bitstream ./out.stream ./in.f64} |
| 239 |
\item Decompress from a SPERR bitstream, and write out the volume in native |
| 240 |
and coarsened resolutions: \\ |
| 241 |
\texttt{./bin/sperr3d -d --decomp\_d ./out.decomp --decomp\_lowres\_d ./out.lowres |
| 242 |
./sperr.stream} |
| 243 |
In this example, the output file \texttt{out.decomp} will be of the native resolution, |
| 244 |
and five other files (\texttt{out.lowres.192x192x128, out.lowres.96x96x64, etc.}) |
| 245 |
will also be produced with their filenames indicating the coarsened resolution. |
| 246 |
\end{enumerate} |
| 247 |
|
| 248 |
\callout{To support multi-resolution decoding in 3D cases, the individual chunks |
| 249 |
(\texttt{--chunks}) need to 1) approximate a cube, so that there are the same number |
| 250 |
of wavelet transforms performed on each dimension, and 2) divide the full volume |
| 251 |
in each dimension.} |
| 252 |
|
| 253 |
\subsubsection{\texttt{sperr3d\_trunc} and Flexible-Rate Decoding} |
| 254 |
Compressed SPERR bitstreams support \textit{flexible-rate} decoding, meaning that a |
| 255 |
sub-bitstream of it from the beginning can still be used to reconstruct the data field. |
| 256 |
Equally important, the reconstruction will have the best possible quality (in terms of |
| 257 |
average error) under the the size constraint of the sub-bitstream, though lower quality |
| 258 |
than using the full bitstream. |
| 259 |
Given a compressed bitstream, a sub-bitstream can be obtained by a simple truncation, |
| 260 |
for example, a truncation that keeps the first 10\% of the full bitstream. |
| 261 |
%On Unix systems, utility tool \texttt{head} can easily perform this task. |
| 262 |
|
| 263 |
The chunking scheme used in 3D compression (see Section~\ref{sec:sperr3d}) |
| 264 |
brings some complication, |
| 265 |
because the bitstream representing each chunk needs to be truncated \textit{individually}. |
| 266 |
\texttt{sperr3d\_trunc} is thus introduced to properly truncate a multi-chunked SPERR bitstream. |
| 267 |
Specifically, it 1) locates the bitstream representing each chunk, |
| 268 |
2) truncates the bitstream, and 3) stitches all truncated bitstreams together so |
| 269 |
\texttt{sperr3d} can properly decompress it. |
| 270 |
|
| 271 |
The help message of \texttt{sperr3d\_trunc} details its options: |
| 272 |
|
| 273 |
\callout{SPERR bitstreams without using multi-chunks (i.e., \texttt{--dims} equals |
| 274 |
\texttt{--chunks} in 3D, and all 2D cases) can safely be truncated by any method. |
| 275 |
For example, the following command truncates a compressed bitstream \texttt{field.stream} |
| 276 |
to keep its first 5kB as \texttt{field2.stream}, which is also recognized by SPERR:\\ |
| 277 |
\texttt{head -c 5000 density.stream > density2.stream} |
| 278 |
} |
| 279 |
|
| 280 |
\begin{minted}{bash} |
| 281 |
$ ./bin/sperr3d_trunc -h |
| 282 |
|
| 283 |
Usage: ./bin/sperr3d_trunc [OPTIONS] [filename] |
| 284 |
|
| 285 |
Positionals: |
| 286 |
filename TEXT:FILE The original SPERR3D bitstream to be truncated. |
| 287 |
|
| 288 |
Options: |
| 289 |
-h,--help Print this help message and exit |
| 290 |
|
| 291 |
Truncation settings: |
| 292 |
--pct UINT REQUIRED Percentage (1--100) of the original bitstream to truncate. |
| 293 |
--omp UINT Number of OpenMP threads to use. Default (or 0) to use all. |
| 294 |
|
| 295 |
Output settings: |
| 296 |
-o TEXT Write out the truncated bitstream. |
| 297 |
|
| 298 |
Input settings: |
| 299 |
--orig32 TEXT Original raw data in 32-bit precision to calculate compression |
| 300 |
quality using the truncated bitstream. |
| 301 |
--orig64 TEXT Original raw data in 64-bit precision to calculate compression |
| 302 |
quality using the truncated bitstream. |
| 303 |
\end{minted} |
| 304 |
|
| 305 |
Examples: |
| 306 |
\begin{enumerate} |
| 307 |
\item Produce a truncated version of a bitstream using 10\% of the original length: \\ |
| 308 |
\texttt{./bin/sperr3d\_trunc --pct 10 -o ./stream.10 ./bitstream} |
| 309 |
\item Perform the task above, plus evaluate compression quality using the truncated |
| 310 |
bitstream:\\ |
| 311 |
\texttt{./bin/sperr3d\_trunc --pct 10 -o ./stream.10 --orig64 ./data.f64 ./bitstream} |
| 312 |
\end{enumerate} |
| 313 |
|
| 314 |
\subsection{C++ Interface} |
| 315 |
\label{sec:cpp} |
| 316 |
\subsubsection{2D Compression and Decompression} |
| 317 |
C++ class \texttt{sperr::SPECK2D\_FLT} is responsible for 2D compression and decompression. |
| 318 |
The sample code walks through necessary steps to perform a compression and decompression |
| 319 |
task, and a more concrete example can be found |
| 320 |
\href{https://github.com/NCAR/SPERR/blob/main/utilities/sperr2d.cpp}{here}. |
| 321 |
|
| 322 |
\begin{minted}{cpp} |
| 323 |
// |
| 324 |
// Example of using a sperr::SPECK2D_FLT() to compress a 2D slice. |
| 325 |
// This is a 6-step process. |
| 326 |
// |
| 327 |
#include "SPECK2D_FLT.h" |
| 328 |
|
| 329 |
// Step 1: create an encoder: |
| 330 |
auto encoder = sperr::SPECK2D_FLT(); |
| 331 |
|
| 332 |
// Step 2: specify the 2D slice dimension (the third dimension is left with 1): |
| 333 |
encoder.set_dims({128, 128, 1}); |
| 334 |
|
| 335 |
// Step 3: copy over the input data from a raw pointer (float* or double*): |
| 336 |
encoder.copy_data(ptr, 16'384); // 16,384 is the number of values. |
| 337 |
// Step 3 alternative: one can hand a memory buffer to the encoder to avoid a memory copy; |
| 338 |
// use either version is cool. |
| 339 |
encoder.take_data(std::move(input)); // input is of type std::vector<doubles>. |
| 340 |
|
| 341 |
// Step 4: specify the compression quality measured in one of three metrics; |
| 342 |
// only the last invoked quality metric is honored. |
| 343 |
encoder.set_tolerance(1e-9); // PWE tolerance = 1e-9 |
| 344 |
encoder.set_bitrate(2.2); // Target bitrate = 2.2 |
| 345 |
encoder.set_psnr(102.2); // Target PSNR = 102.2 |
| 346 |
|
| 347 |
// Step 5: perform the compression task: |
| 348 |
encoder.compress(); |
| 349 |
|
| 350 |
// Step 6: retrieve the compressed bitstream: |
| 351 |
auto bitstream = std::vector<uint8_t>(); |
| 352 |
encoder.append_encoded_bitstream(bitstream); |
| 353 |
\end{minted} |
| 354 |
|
| 355 |
\begin{minted}{cpp} |
| 356 |
// |
| 357 |
// Example of using a sperr::SPECK2D_FLT() to decompress a bitstream. |
| 358 |
// This is a 5-step process. |
| 359 |
// |
| 360 |
#include "SPECK2D_FLT.h" |
| 361 |
|
| 362 |
// Step 1: create a decoder: |
| 363 |
auto decoder = sperr::SPECK2D_FLT(); |
| 364 |
|
| 365 |
// Step 2: specify the 2D slice dimension (the third dimension is left with 1): |
| 366 |
// This information is often saved once somewhere for many same-sized slices. |
| 367 |
decoder.set_dims({128, 128, 1}); |
| 368 |
|
| 369 |
// Step 3: pass in the compressed bitstream as a raw pointer (uint8_t*): |
| 370 |
decoder.use_bitstream(ptr, 16'384); // 16,384 is the length of the bitstream. |
| 371 |
|
| 372 |
// Step 4: perform the decompression task: |
| 373 |
decoder.decompress(multi_res); // a boolean, if to enable multi-resolution decoding |
| 374 |
|
| 375 |
// Step 5: retrieve the decompressed volume: |
| 376 |
std::vector<double> vol = decoder.view_decoded_data(); |
| 377 |
auto hierarchy = decoder.view_hierarchy(); // if multi-resolution was enabled |
| 378 |
// Step 5 alternative: one can take ownership of the data buffer to avoid a memory copy. |
| 379 |
std::vector<double> vol = decoder.release_decoded_data(); |
| 380 |
auto hierarchy = decoder.release_hierarchy(); // if multi-resolution was enabled |
| 381 |
\end{minted} |
| 382 |
|
| 383 |
\subsubsection{3D Compression and Decompression} |
| 384 |
C++ class \texttt{sperr::SPERR3D\_OMP\_C} is responsible for 3D compression, and |
| 385 |
\texttt{sperr::SPERR3D\_OMP\_D} is responsible for 3D decompression. |
| 386 |
The sample code walks through necessary steps to perform a compression and decompression |
| 387 |
task, and a more concrete example can be found |
| 388 |
\href{https://github.com/NCAR/SPERR/blob/main/utilities/sperr3d.cpp}{here}. |
| 389 |
|
| 390 |
\begin{minted}{cpp} |
| 391 |
// |
| 392 |
// Example of using a sperr::SPERR3D_OMP_C() to compress a 3D volume. |
| 393 |
// This is a 6-step process. |
| 394 |
// |
| 395 |
#include "SPERR3D_OMP_C.h" |
| 396 |
|
| 397 |
// Step 1: create an encoder: |
| 398 |
auto encoder = sperr::SPERR3D_OMP_C(); |
| 399 |
|
| 400 |
// Step 2: specify the volume and chunk dimensions, respectively: |
| 401 |
encoder.set_dims_and chunks({384, 384, 256}, {192, 192, 128}); |
| 402 |
|
| 403 |
// Step 3: specify the number of OpenMP threads to use: |
| 404 |
encoder.set_num_threads(4); |
| 405 |
|
| 406 |
// Step 4: specify the compression quality measured in one of three metrics; |
| 407 |
// only the last invoked quality metric is honored. |
| 408 |
encoder.set_tolerance(1e-9); // PWE tolerance = 1e-9 |
| 409 |
encoder.set_bitrate(2.2); // Target bitrate = 2.2 |
| 410 |
encoder.set_psnr(102.2); // Target PSNR = 102.2 |
| 411 |
|
| 412 |
// Step 5: perform the compression task: |
| 413 |
// The input data is passed in in the form of a raw pointer (float* or double*), |
| 414 |
// and the total number of values will be passed in here too. |
| 415 |
encoder.compress(ptr, 384 * 384 * 256); |
| 416 |
|
| 417 |
// Step 6: retrieve the compressed bitstream: |
| 418 |
std::vector<uint8_t> stream = encoder.get_encoded_bitstream(); |
| 419 |
\end{minted} |
| 420 |
|
| 421 |
\begin{minted}{cpp} |
| 422 |
// |
| 423 |
// Example of using a sperr::SPERR3D_OMP_D() to decompress a bitstream. |
| 424 |
// This is a 5-step process. |
| 425 |
// |
| 426 |
#include "SPERR3D_OMP_D.h" |
| 427 |
|
| 428 |
// Step 1: create a decoder: |
| 429 |
auto decoder = sperr::SPERR3D_OMP_D(); |
| 430 |
|
| 431 |
// Step 2: specify the number of OpenMP threads to use: |
| 432 |
decoder.set_num_threads(4); |
| 433 |
|
| 434 |
// Step 3: pass in the compressed bitstream as a raw pointer (uint8_t*): |
| 435 |
decoder.use_bitstream(ptr, 16'384); // 16,384 is the length of the bitstream. |
| 436 |
|
| 437 |
// Step 4: perform the decompression task: |
| 438 |
// Note that the pointer to the bitstream is passed in again! |
| 439 |
decoder.decompress(ptr, multi_res); // a boolean, if to enable multi-resolution decoding |
| 440 |
|
| 441 |
// Step 5: retrieve the decompressed volume: |
| 442 |
auto [dimx, dimy, dimz] = decoder.get_dims(); // dimension of the volume |
| 443 |
std::vector<double> vol = decoder.view_decoded_data(); |
| 444 |
auto hierarchy = decoder.view_hierarchy(); // if multi-resolution was enabled |
| 445 |
// Step 5 alternative: one can take ownership of the data buffer to avoid memory copies. |
| 446 |
std::vector<double> vol = decoder.release_decoded_data(); |
| 447 |
auto hierarchy = decoder.release_hierarchy(); // if multi-resolution was enabled |
| 448 |
\end{minted} |
| 449 |
|
| 450 |
\callout{To achieve higher performance with repeated compression and |
| 451 |
decompression tasks, the encoder and decoder objects are better to be |
| 452 |
re-used rather than repeatedly destroyed and created.} |
| 453 |
|
| 454 |
\subsection{C Interface} |
| 455 |
\label{sec:c} |
| 456 |
SPERR provides a C wrapper with a set of C functions. |
| 457 |
All of the C interface is in the header file |
| 458 |
\href{https://github.com/NCAR/SPERR/blob/main/include/SPERR_C_API.h}{\texttt{SPERR\_C\_API.h}}, |
| 459 |
which itself documents the C functions and parameters, etc. |
| 460 |
The following example code walks through key steps to use the C API to perform |
| 461 |
compression and decompression, while more concrete examples are available |
| 462 |
for \href{https://github.com/NCAR/SPERR/blob/main/examples/C_API/2d.c}{2D} |
| 463 |
and \href{https://github.com/NCAR/SPERR/blob/main/examples/C_API/3d.c}{3D} cases. |
| 464 |
|
| 465 |
\subsubsection{Example: 2D} |
| 466 |
|
| 467 |
\begin{minted}{C} |
| 468 |
/* |
| 469 |
* Example of using the SPERR C API to perform 2D compression and decompression tasks. |
| 470 |
*/ |
| 471 |
#include "SPERR_C_API.h" |
| 472 |
|
| 473 |
/* Step 1: create variables to keep the output: */ |
| 474 |
void* stream = NULL; /* caller is responsible for free'ing it after use. */ |
| 475 |
size_t stream_len = 0; |
| 476 |
|
| 477 |
/* Step 2: call the 2D compression function: |
| 478 |
* Assume that we have a buffer of 128 * 128 floats (in float* type) to be compressed, |
| 479 |
* using PWE tolerance = 1e-3. |
| 480 |
*/ |
| 481 |
int ret = sperr_comp_2d(ptr, /* memory buffer containing the input */ |
| 482 |
1, /* the input is of type float; 0 means double. */ |
| 483 |
128, /* dimx */ |
| 484 |
128, /* dimy */ |
| 485 |
3, /* compression mode; 3 means fixed PWE */ |
| 486 |
1e-3, /* actual PWE tolerance */ |
| 487 |
0, /* not using a header for the output bitstream */ |
| 488 |
&stream, /* will hold the compressed bitstream */ |
| 489 |
&stream_len); /* length of the compressed bitstream */ |
| 490 |
assert(ret == 0); |
| 491 |
|
| 492 |
/* |
| 493 |
* Now that the 2D compression is completed, one can decompress the bitstream to |
| 494 |
* retrieve the raw values, as the rest of this example shows. |
| 495 |
*/ |
| 496 |
|
| 497 |
/* Step 3: create a pointer to hold the decompressed values: */ |
| 498 |
void* output = NULL; /* caller is responsible for free'ing it after use. */ |
| 499 |
|
| 500 |
/* Step 4: call the 2D decompression function: */ |
| 501 |
int ret2 = sperr_decomp_2d(stream, /* compressed bitstream */ |
| 502 |
stream_len, /* compressed bitstream length */ |
| 503 |
1, /* decompress to floats. 0 means to doubles. */ |
| 504 |
128, /* dimx */ |
| 505 |
128, /* dimy */ |
| 506 |
&output); /* decompressed data is stored here */ |
| 507 |
assert(ret2 == 0); |
| 508 |
free(output); /* cleanup */ |
| 509 |
free(stream); /* cleanup */ |
| 510 |
\end{minted} |
| 511 |
|
| 512 |
\subsubsection{Example: 3D} |
| 513 |
|
| 514 |
\begin{minted}{C} |
| 515 |
/* |
| 516 |
* Example of using the SPERR C API to perform 3D compression and decompression tasks. |
| 517 |
*/ |
| 518 |
#include "SPERR_C_API.h" |
| 519 |
|
| 520 |
/* Step 1: create variables to keep the output: */ |
| 521 |
void* stream = NULL; /* caller is responsible for free'ing it after use. */ |
| 522 |
size_t stream_len = 0; |
| 523 |
|
| 524 |
/* Step 2: call the 3D compression function: |
| 525 |
* Assume that we have a buffer of 256^3 floats (in float* type) to be compressed, |
| 526 |
* using PWE tolerance = 1e-3 and chunk dimension of 128^3. |
| 527 |
*/ |
| 528 |
int ret = sperr_comp_3d(ptr, /* memory buffer containing the input */ |
| 529 |
1, /* the input is of type float; 0 means double. */ |
| 530 |
256, /* dimx */ |
| 531 |
256, /* dimy */ |
| 532 |
256, /* dimz */ |
| 533 |
128, /* chunk_x */ |
| 534 |
128, /* chunk_y */ |
| 535 |
128, /* chunk_z */ |
| 536 |
3, /* compression mode; 3 means fixed PWE */ |
| 537 |
1e-3, /* actual PWE tolerance */ |
| 538 |
4, /* use 4 OpenMP threads */ |
| 539 |
&stream, /* will hold the compressed bitstream */ |
| 540 |
&stream_len); /* length of the compressed bitstream */ |
| 541 |
assert(ret == 0); |
| 542 |
|
| 543 |
/* |
| 544 |
* Now that the 3D compression is completed, one can decompress the bitstream to |
| 545 |
* retrieve the raw values, as the rest of this example shows. |
| 546 |
*/ |
| 547 |
|
| 548 |
/* Step 3: create a pointer to hold the decompressed values, |
| 549 |
* and also variables to hold the volume dimensions. |
| 550 |
*/ |
| 551 |
void* output = NULL; /* caller is responsible for free'ing it after use. */ |
| 552 |
size_t dimx = 0, dimy = 0, dimz = 0; |
| 553 |
|
| 554 |
/* Step 4: call the 3D decompression function: */ |
| 555 |
int ret2 = sperr_decomp_3d(stream, /* compressed bitstream */ |
| 556 |
stream_len, /* compressed bitstream length */ |
| 557 |
1, /* decompress to floats. 0 means to doubles. */ |
| 558 |
4, /* use 4 OpenMP threads */ |
| 559 |
&dimx, /* dimx of the decompressed volume */ |
| 560 |
&dimy, /* dimy of the decompressed volume */ |
| 561 |
&dimz, /* dimz of the decompressed volume */ |
| 562 |
&output); /* decompressed data is stored here */ |
| 563 |
assert(ret2 == 0); |
| 564 |
free(output); /* cleanup */ |
| 565 |
free(stream); /* cleanup */ |
| 566 |
\end{minted} |
| 567 |
|
| 568 |
\end{document} |