Debian Codegraph @

third_party/smol-v/README.md

raw
1 # SMOL-V: like Vulkan/Khronos SPIR-V, but smaller. [![Build Status](https://api.travis-ci.org/aras-p/smol-v.svg?branch=master)](https://travis-ci.org/aras-p/smol-v)
2
3 ## Overview
4
5 SMOL-V encodes Vulkan/Khronos [SPIR-V](https://www.khronos.org/registry/spir-v/)
6 format programs into a form that is *smoller*, and is more
7 compressible. Normally no changes to the programs are done; they decode
8 into exactly same program as what was encoded. Optionally, debug information
9 can be removed too.
10
11 SPIR-V is a very verbose format, several times larger than same programs expressed in other
12 shader formats *(e.g. DX11 bytecode, GLSL, DX9 bytecode etc.)*. The SSA-form with ever increasing
13 IDs is not very appreciated by regular data compressors either. SMOL-V does several things
14 to improve this:
15
16 - Many words, especially ones that most often have small values, are encoded using
17 ["varint" scheme](https://developers.google.com/protocol-buffers/docs/encoding) (1-5 bytes per
18 word, with just one byte for values in 0..127 range).
19 - Some IDs used in the program are delta-encoded, relative to previously seen IDs (e.g. Result
20 IDs). Often instructions reference things that were computed just before, so this results in
21 small deltas. These values are also encoded using "varint" scheme.
22 - Reordering instruction opcodes so that the most common ones are the smallest values, for smaller
23 varint encoding.
24 - Encoding several instructions in a more compact form, e.g. the "typical <=4 component swizzle"
25 shape of a VectorShuffle instruction, or sequences of MemberDecorate instructions.
26
27 A somewhat similar utility is [spirv-remap from glslang](https://github.com/KhronosGroup/glslang/blob/master/README-spirv-remap.txt).
28
29 See [this blog post](http://aras-p.info/blog/2016/09/01/SPIR-V-Compression/) for more information about
30 how I did SMOL-V.
31
32
33 ## Usage
34
35 Add [`source/smolv.h`](source/smolv.h) and [`source/smolv.cpp`](source/smolv.cpp) to your C++ project build.
36 It might require C++11 or somesuch; I only tested with Visual Studio 2017 and Mac Xcode 9.4. There's
37 a [Travis-CI](https://travis-ci.org/aras-p/smol-v) set up for this project, that builds on Windows (VS2017),
38 Mac (Xcode 10.1) and Linux (gcc 4.8).
39
40 `smolv::Encode` and `smolv::Decode` is the basic functionality. See [smolv.h](source/smolv.h).
41
42 Other functions are for development/statistics purposes, to figure out frequencies and
43 distributions of the instructions.
44
45 There's a test + compression benchmarking suite in `testing/testmain.cpp`, using that needs adding
46 other files under testing/external to the build too (3rd party code: glslang remapper, Zstd, LZ4, miniz).
47
48 ## Changelog
49
50 See [**Changelog**](Changelog.md).
51
52
53 ## Limitations / TODO
54
55 - SPIR-V where the words got stored in big-endian layout is not supported yet.
56 - The whole thing might not work on Big-Endian CPUs. It might, but I'm not 100% sure.
57 - Not much prevention is done against malformed/corrupted inputs, TODO.
58 - Out of memory cases are not handled. The code will either throw exception
59 or crash, depending on your compilation flags.
60
61
62 ## License
63
64 Code itself: **Public Domain** or **MIT**, pick whichever works better for you.
65
66 There is 3rd party code under the testing framework (`testing/external`); it is not required for
67 using SMOL-V. Most of that code ([glslang](https://github.com/KhronosGroup/glslang),
68 [LZ4](https://github.com/Cyan4973/lz4), [Zstd](https://github.com/facebook/zstd), [sokol_time.h](https://github.com/floooh/sokol/blob/master/sokol_time.h))
69 is BSD or zlib-licensed, and taken from github repositories of the respective projects. [miniz](https://github.com/richgel999/miniz)
70 is public domain.
71
72 There are SPIR-V binary shader dumps under `tests/spirv-dumps` for compression testing;
73 these are not required for using SMOL-V. Not sure how to appropriately
74 "license" them (but hey they are kinda useless by themselves out of context),
75 so I'll go with this: "Binary shader dumps under 'tests' folder are only to be
76 used for SMOL-V testing". Details on them:
77
78 * `tests/spirv-dumps/dota2` - some shaders from [DOTA2](http://blog.dota2.com/), Copyright Valve Corporation, all rights reserved.
79 * `tests/spirv-dumps/shadertoy` - several most popular shaders from [Shadertoy](https://www.shadertoy.com/), converted to Vulkan
80 SPIR-V via glslang. Copyrights by their individual authors (filename matches last component of shadertoy URL).
81 * `tests/spirv-dumps/talos` - some shaders from [The Talos Principle](http://www.croteam.com/talosprinciple/),
82 Copyright (c) 2002-2016 Croteam All rights reserved.
83 * `tests/spirv-dumps/unity` - various [Unity](https://unity3d.com/) shaders, produced
84 through a HLSL -> DX11 bytecode -> HLSLcc -> glslang toolchain.
85
86
87
88 ## Results
89
90 As of 2018 October 29, results on 342 shaders (under `tests/spirv-dumps`) are:
91
92 ```
93 Compressed with <none>:
94 Raw 4869.9KB 100.0%
95 Remapper 4541.4KB 93.3%
96 SmolV 1629.5KB 33.5%
97
98 Compressed with zlib:
99 Raw 1213.3KB 24.9%
100 Remapper 1079.7KB 22.2%
101 SmolV 602.1KB 12.4%
102
103 Compressed with LZ4 HC:
104 Raw 1343.4KB 27.6%
105 Remapper 1148.1KB 23.6%
106 SmolV 606.3KB 12.5%
107
108 Compressed with Zstandard:
109 Raw 899.3KB 18.5%
110 Remapper 742.1KB 15.2%
111 SmolV 445.6KB 9.1%
112
113 Compressed with Zstandard 20:
114 Raw 589.7KB 12.1%
115 Remapper 508.9KB 10.5%
116 SmolV 348.0KB 7.1%
117 ```
118
119 * "Raw" is just raw SPIR-V, with no extra processing.
120 * "Remapper" is spirv-remap from glslang, with debug info stripping.
121 * SmolV is what you're looking at, with debug info stripping too.
122 * zlib, LZ4HC and Zstd are general compression algorithms at default settings (Zstd20 is Zstd compression with almost max setting of 20).
123 * Compression is done on the whole blob of all the test programs (not individually for each program).