Debian Codegraph @

utf-8-validate/deps/is_utf8/README.md

raw
1 # is_utf8
2
3 Most strings online are in unicode using the UTF-8 encoding. Validating strings
4 quickly before accepting them is important.
5
6 ## How to use is_utf8
7
8 This is a simple one-source file library to validate UTF-8 strings at high
9 speeds using SIMD instructions. It works on all platforms (ARM, x64).
10
11 Build and link `is_utf8.cpp` with your project. Code usage:
12
13 ```C++
14 #include "is_utf8.h"
15
16 char * mystring = ...
17 bool is_it_valid = is_utf8(mystring, thestringlength);
18 ```
19
20 It should be able to validate strings using less than 1 cycle per input byte.
21
22 ## Requirements
23
24 - C++11 compatible compiler. We support LLVM clang, GCC, Visual Studio. (Our
25 optional benchmark tool requires C++17.)
26 - For high speed, you should have a recent 64-bit system (e.g., ARM or x64).
27 - If you rely on CMake, you should use a recent CMake (at least 3.15).
28 - AVX-512 support require a processor with AVX512-VBMI2 (Ice Lake or better) and
29 a recent compiler (GCC 8 or better, Visual Studio 2019 or better, LLVM clang 6
30 or better). You need a correspondingly recent assembler such as gas (2.30+) or
31 nasm (2.14+): recent compilers usually come with recent assemblers. If you mix
32 a recent compiler with an incompatible/old assembler (e.g., when using a
33 recent compiler with an old Linux distribution), you may get errors at build
34 time because the compiler produces instructions that the assembler does not
35 recognize: you should update your assembler to match your compiler (e.g.,
36 upgrade binutils to version 2.30 or better under Linux) or use an older
37 compiler matching the capabilities of your assembler.
38
39 ## Build with CMake
40
41 ```
42 cmake -B build
43 cmake --build build
44 cd build
45 ctest .
46 ```
47
48 Visual Studio users must specify whether they want to build the Release or Debug
49 version.
50
51 To run benchmarks, build and execute the `bench` command.
52
53 ```
54 cmake -B build
55 cmake --build build
56 ./build/benchmarks/bench
57 ```
58
59 Instructions are similar for Visual Studio users.
60
61 ## Real-word usage
62
63 This C++ library is part of the JavaScript package
64 [utf-8-validate](https://github.com/websockets/utf-8-validate). The
65 utf-8-validate package is routinely downloaded more than
66 [a million times per week](https://www.npmjs.com/package/utf-8-validate).
67
68 If you are using Node JS (19.4.0 or better), you already have access to this
69 function as
70 [`buffer.isUtf8(input)`](https://nodejs.org/api/buffer.html#bufferisutf8input).
71
72 ## Reference
73
74 - John Keiser, Daniel Lemire,
75 [Validating UTF-8 In Less Than One Instruction Per Byte](https://arxiv.org/abs/2010.03090),
76 Software: Practice & Experience 51 (5), 2021
77
78 ## Want more?
79
80 If you want a wide range of fast Unicode function for production use, you can
81 rely on the simdutf library. It is as simple as the following:
82
83 ```C++
84 #include "simdutf.cpp"
85 #include "simdutf.h"
86
87 int main(int argc, char *argv[]) {
88 const char *source = "1234";
89 // 4 == strlen(source)
90 bool validutf8 = simdutf::validate_utf8(source, 4);
91 if (validutf8) {
92 std::cout << "valid UTF-8" << std::endl;
93 } else {
94 std::cerr << "invalid UTF-8" << std::endl;
95 return EXIT_FAILURE;
96 }
97 }
98 ```
99
100 See https://github.com/simdutf/
101
102 ## License
103
104 This library is distributed under the terms of any of the following licenses, at
105 your option:
106
107 - Apache License (Version 2.0) [LICENSE-APACHE](LICENSE-APACHE),
108 - Boost Software License [LICENSE-BOOST](LICENSE-BOOST), or
109 - MIT License [LICENSE-MIT](LICENSE-MIT).