| 1 |
# SMOL-V: like Vulkan/Khronos SPIR-V, but smaller. [](https://travis-ci.org/aras-p/smol-v) |
| 2 |
|
| 3 |
## Overview |
| 4 |
|
| 5 |
SMOL-V encodes Vulkan/Khronos [SPIR-V](https://www.khronos.org/registry/spir-v/) |
| 6 |
format programs into a form that is *smoller*, and is more |
| 7 |
compressible. Normally no changes to the programs are done; they decode |
| 8 |
into exactly same program as what was encoded. Optionally, debug information |
| 9 |
can be removed too. |
| 10 |
|
| 11 |
SPIR-V is a very verbose format, several times larger than same programs expressed in other |
| 12 |
shader formats *(e.g. DX11 bytecode, GLSL, DX9 bytecode etc.)*. The SSA-form with ever increasing |
| 13 |
IDs is not very appreciated by regular data compressors either. SMOL-V does several things |
| 14 |
to improve this: |
| 15 |
|
| 16 |
- Many words, especially ones that most often have small values, are encoded using |
| 17 |
["varint" scheme](https://developers.google.com/protocol-buffers/docs/encoding) (1-5 bytes per |
| 18 |
word, with just one byte for values in 0..127 range). |
| 19 |
- Some IDs used in the program are delta-encoded, relative to previously seen IDs (e.g. Result |
| 20 |
IDs). Often instructions reference things that were computed just before, so this results in |
| 21 |
small deltas. These values are also encoded using "varint" scheme. |
| 22 |
- Reordering instruction opcodes so that the most common ones are the smallest values, for smaller |
| 23 |
varint encoding. |
| 24 |
- Encoding several instructions in a more compact form, e.g. the "typical <=4 component swizzle" |
| 25 |
shape of a VectorShuffle instruction, or sequences of MemberDecorate instructions. |
| 26 |
|
| 27 |
A somewhat similar utility is [spirv-remap from glslang](https://github.com/KhronosGroup/glslang/blob/master/README-spirv-remap.txt). |
| 28 |
|
| 29 |
See [this blog post](http://aras-p.info/blog/2016/09/01/SPIR-V-Compression/) for more information about |
| 30 |
how I did SMOL-V. |
| 31 |
|
| 32 |
|
| 33 |
## Usage |
| 34 |
|
| 35 |
Add [`source/smolv.h`](source/smolv.h) and [`source/smolv.cpp`](source/smolv.cpp) to your C++ project build. |
| 36 |
It might require C++11 or somesuch; I only tested with Visual Studio 2017 and Mac Xcode 9.4. There's |
| 37 |
a [Travis-CI](https://travis-ci.org/aras-p/smol-v) set up for this project, that builds on Windows (VS2017), |
| 38 |
Mac (Xcode 10.1) and Linux (gcc 4.8). |
| 39 |
|
| 40 |
`smolv::Encode` and `smolv::Decode` is the basic functionality. See [smolv.h](source/smolv.h). |
| 41 |
|
| 42 |
Other functions are for development/statistics purposes, to figure out frequencies and |
| 43 |
distributions of the instructions. |
| 44 |
|
| 45 |
There's a test + compression benchmarking suite in `testing/testmain.cpp`, using that needs adding |
| 46 |
other files under testing/external to the build too (3rd party code: glslang remapper, Zstd, LZ4, miniz). |
| 47 |
|
| 48 |
## Changelog |
| 49 |
|
| 50 |
See [**Changelog**](Changelog.md). |
| 51 |
|
| 52 |
|
| 53 |
## Limitations / TODO |
| 54 |
|
| 55 |
- SPIR-V where the words got stored in big-endian layout is not supported yet. |
| 56 |
- The whole thing might not work on Big-Endian CPUs. It might, but I'm not 100% sure. |
| 57 |
- Not much prevention is done against malformed/corrupted inputs, TODO. |
| 58 |
- Out of memory cases are not handled. The code will either throw exception |
| 59 |
or crash, depending on your compilation flags. |
| 60 |
|
| 61 |
|
| 62 |
## License |
| 63 |
|
| 64 |
Code itself: **Public Domain** or **MIT**, pick whichever works better for you. |
| 65 |
|
| 66 |
There is 3rd party code under the testing framework (`testing/external`); it is not required for |
| 67 |
using SMOL-V. Most of that code ([glslang](https://github.com/KhronosGroup/glslang), |
| 68 |
[LZ4](https://github.com/Cyan4973/lz4), [Zstd](https://github.com/facebook/zstd), [sokol_time.h](https://github.com/floooh/sokol/blob/master/sokol_time.h)) |
| 69 |
is BSD or zlib-licensed, and taken from github repositories of the respective projects. [miniz](https://github.com/richgel999/miniz) |
| 70 |
is public domain. |
| 71 |
|
| 72 |
There are SPIR-V binary shader dumps under `tests/spirv-dumps` for compression testing; |
| 73 |
these are not required for using SMOL-V. Not sure how to appropriately |
| 74 |
"license" them (but hey they are kinda useless by themselves out of context), |
| 75 |
so I'll go with this: "Binary shader dumps under 'tests' folder are only to be |
| 76 |
used for SMOL-V testing". Details on them: |
| 77 |
|
| 78 |
* `tests/spirv-dumps/dota2` - some shaders from [DOTA2](http://blog.dota2.com/), Copyright Valve Corporation, all rights reserved. |
| 79 |
* `tests/spirv-dumps/shadertoy` - several most popular shaders from [Shadertoy](https://www.shadertoy.com/), converted to Vulkan |
| 80 |
SPIR-V via glslang. Copyrights by their individual authors (filename matches last component of shadertoy URL). |
| 81 |
* `tests/spirv-dumps/talos` - some shaders from [The Talos Principle](http://www.croteam.com/talosprinciple/), |
| 82 |
Copyright (c) 2002-2016 Croteam All rights reserved. |
| 83 |
* `tests/spirv-dumps/unity` - various [Unity](https://unity3d.com/) shaders, produced |
| 84 |
through a HLSL -> DX11 bytecode -> HLSLcc -> glslang toolchain. |
| 85 |
|
| 86 |
|
| 87 |
|
| 88 |
## Results |
| 89 |
|
| 90 |
As of 2018 October 29, results on 342 shaders (under `tests/spirv-dumps`) are: |
| 91 |
|
| 92 |
``` |
| 93 |
Compressed with <none>: |
| 94 |
Raw 4869.9KB 100.0% |
| 95 |
Remapper 4541.4KB 93.3% |
| 96 |
SmolV 1629.5KB 33.5% |
| 97 |
|
| 98 |
Compressed with zlib: |
| 99 |
Raw 1213.3KB 24.9% |
| 100 |
Remapper 1079.7KB 22.2% |
| 101 |
SmolV 602.1KB 12.4% |
| 102 |
|
| 103 |
Compressed with LZ4 HC: |
| 104 |
Raw 1343.4KB 27.6% |
| 105 |
Remapper 1148.1KB 23.6% |
| 106 |
SmolV 606.3KB 12.5% |
| 107 |
|
| 108 |
Compressed with Zstandard: |
| 109 |
Raw 899.3KB 18.5% |
| 110 |
Remapper 742.1KB 15.2% |
| 111 |
SmolV 445.6KB 9.1% |
| 112 |
|
| 113 |
Compressed with Zstandard 20: |
| 114 |
Raw 589.7KB 12.1% |
| 115 |
Remapper 508.9KB 10.5% |
| 116 |
SmolV 348.0KB 7.1% |
| 117 |
``` |
| 118 |
|
| 119 |
* "Raw" is just raw SPIR-V, with no extra processing. |
| 120 |
* "Remapper" is spirv-remap from glslang, with debug info stripping. |
| 121 |
* SmolV is what you're looking at, with debug info stripping too. |
| 122 |
* zlib, LZ4HC and Zstd are general compression algorithms at default settings (Zstd20 is Zstd compression with almost max setting of 20). |
| 123 |
* Compression is done on the whole blob of all the test programs (not individually for each program). |