Debian Codegraph @

LspCpp/third_party/utfcpp/README.md

raw
1 # UTF8-CPP: UTF-8 with C++ in a Portable Way
2
3
4 ## Introduction
5
6 C++ developers miss an easy and portable way of handling Unicode encoded strings. The original C++ Standard (known as C++98 or C++03) is Unicode agnostic. C++11 provides some support for Unicode on core language and library level: u8, u, and U character and string literals, char16_t and char32_t character types, u16string and u32string library classes, and codecvt support for conversions between Unicode encoding forms. In the meantime, developers use third party libraries like ICU, OS specific capabilities, or simply roll out their own solutions.
7
8 In order to easily handle UTF-8 encoded Unicode strings, I came up with a small, C++98 compatible generic library. For anybody used to work with STL algorithms and iterators, it should be easy and natural to use. The code is freely available for any purpose - check out the [license](./LICENSE). The library has been used a lot in the past ten years both in commercial and open-source projects and is considered feature-complete now. If you run into bugs or performance issues, please let me know and I'll do my best to address them.
9
10 The purpose of this article is not to offer an introduction to Unicode in general, and UTF-8 in particular. If you are not familiar with Unicode, be sure to check out [Unicode Home Page](http://www.unicode.org/) or some other source of information for Unicode. Also, it is not my aim to advocate the use of UTF-8 encoded strings in C++ programs; if you want to handle UTF-8 encoded strings from C++, I am sure you have good reasons for it.
11
12 ## Examples of use
13
14 ### Introductory Sample
15
16 To illustrate the use of the library, let's start with a small but complete program that opens a file containing UTF-8 encoded text, reads it line by line, checks each line for invalid UTF-8 byte sequences, and converts it to UTF-16 encoding and back to UTF-8:
17
18 ```cpp
19 #include <fstream>
20 #include <iostream>
21 #include <string>
22 #include <vector>
23 #include "utf8.h"
24 using namespace std;
25 int main(int argc, char** argv)
26 {
27 if (argc != 2) {
28 cout << "\nUsage: docsample filename\n";
29 return 0;
30 }
31 const char* test_file_path = argv[1];
32 // Open the test file (must be UTF-8 encoded)
33 ifstream fs8(test_file_path);
34 if (!fs8.is_open()) {
35 cout << "Could not open " << test_file_path << endl;
36 return 0;
37 }
38
39 unsigned line_count = 1;
40 string line;
41 // Play with all the lines in the file
42 while (getline(fs8, line)) {
43 // check for invalid utf-8 (for a simple yes/no check, there is also utf8::is_valid function)
44 #if __cplusplus >= 201103L // C++ 11 or later
45 auto end_it = utf8::find_invalid(line.begin(), line.end());
46 #else
47 string::iterator end_it = utf8::find_invalid(line.begin(), line.end());
48 #endif // C++ 11
49 if (end_it != line.end()) {
50 cout << "Invalid UTF-8 encoding detected at line " << line_count << "\n";
51 cout << "This part is fine: " << string(line.begin(), end_it) << "\n";
52 }
53 // Get the line length (at least for the valid part)
54 int length = utf8::distance(line.begin(), end_it);
55 cout << "Length of line " << line_count << " is " << length << "\n";
56
57 // Convert it to utf-16
58 #if __cplusplus >= 201103L // C++ 11 or later
59 u16string utf16line = utf8::utf8to16(line);
60 #else
61 vector<unsigned short> utf16line;
62 utf8::utf8to16(line.begin(), end_it, back_inserter(utf16line));
63 #endif // C++ 11
64 // And back to utf-8;
65 #if __cplusplus >= 201103L // C++ 11 or later
66 string utf8line = utf8::utf16to8(utf16line);
67 #else
68 string utf8line;
69 utf8::utf16to8(utf16line.begin(), utf16line.end(), back_inserter(utf8line));
70 #endif // C++ 11
71 // Confirm that the conversion went OK:
72 if (utf8line != string(line.begin(), end_it))
73 cout << "Error in UTF-16 conversion at line: " << line_count << "\n";
74
75 line_count++;
76 }
77
78 return 0;
79 }
80 ```
81
82 In the previous code sample, for each line we performed a detection of invalid UTF-8 sequences with `find_invalid`; the number of characters (more precisely - the number of Unicode code points, including the end of line and even BOM if there is one) in each line was determined with a use of `utf8::distance`; finally, we have converted each line to UTF-16 encoding with `utf8to16` and back to UTF-8 with `utf16to8`.
83
84 Note a different pattern of usage for old compilers. For instance, this is how we convert
85 a UTF-8 encoded string to a UTF-16 encoded one with a pre - C++11 compiler:
86 ```cpp
87 vector<unsigned short> utf16line;
88 utf8::utf8to16(line.begin(), end_it, back_inserter(utf16line));
89 ```
90
91 With a more modern compiler, the same operation would look like:
92 ```cpp
93 u16string utf16line = utf8::utf8to16(line);
94 ```
95 If `__cplusplus` macro points to a C++ 11 or later, the library exposes API that takes into
96 account C++ standard Unicode strings and move semantics. With an older compiler, it is still
97 possible to use the same functionality, just in a little less convenient way
98
99 In case you do not trust the `__cplusplus` macro or, for instance, do not want to include
100 the C++ 11 helper functions even with a modern compiler, define `UTF_CPP_CPLUSPLUS` macro
101 before including `utf8.h` and assign it a value for the standard you want to use - the values are the same as for the `__cplusplus` macro. This can be also useful with compilers that are conservative in setting the `__cplusplus` macro even if they have a good support for a recent standard edition - Microsoft's Visual C++ is one example.
102
103 ### Checking if a file contains valid UTF-8 text
104
105 Here is a function that checks whether the content of a file is valid UTF-8 encoded text without reading the content into the memory:
106
107 ```cpp
108 bool valid_utf8_file(const char* file_name)
109 {
110 ifstream ifs(file_name);
111 if (!ifs)
112 return false; // even better, throw here
113
114 istreambuf_iterator<char> it(ifs.rdbuf());
115 istreambuf_iterator<char> eos;
116
117 return utf8::is_valid(it, eos);
118 }
119 ```
120
121 Because the function `utf8::is_valid()` works with input iterators, we were able to pass an `istreambuf_iterator` to `it` and read the content of the file directly without loading it to the memory first.
122
123 Note that other functions that take input iterator arguments can be used in a similar way. For instance, to read the content of a UTF-8 encoded text file and convert the text to UTF-16, just do something like:
124
125 ```cpp
126 utf8::utf8to16(it, eos, back_inserter(u16string));
127 ```
128
129 ### Ensure that a string contains valid UTF-8 text
130
131 If we have some text that "probably" contains UTF-8 encoded text and we want to replace any invalid UTF-8 sequence with a replacement character, something like the following function may be used:
132
133 ```cpp
134 void fix_utf8_string(std::string& str)
135 {
136 std::string temp;
137 utf8::replace_invalid(str.begin(), str.end(), back_inserter(temp));
138 str = temp;
139 }
140 ```
141
142 The function will replace any invalid UTF-8 sequence with a Unicode replacement character. There is an overloaded function that enables the caller to supply their own replacement character.
143
144
145 ## Points of interest
146
147 #### Design goals and decisions
148
149 The library was designed to be:
150
151 1. Generic: for better or worse, there are many C++ string classes out there, and the library should work with as many of them as possible.
152 2. Portable: the library should be portable both across different platforms and compilers. The only non-portable code is a small section that declares unsigned integers of different sizes: three typedefs. They can be changed by the users of the library if they don't match their platform. The default setting should work for Windows (both 32 and 64 bit), and most 32 bit and 64 bit Unix derivatives. Support for post C++03 language features is included for modern compilers at API level only, so the library should work even with pretty old compilers.
153 3. Lightweight: follow the "pay only for what you use" guideline.
154 4. Unintrusive: avoid forcing any particular design or even programming style on the user. This is a library, not a framework.
155
156 #### Alternatives
157
158 In case you want to look into other means of working with UTF-8 strings from C++, here is the list of solutions I am aware of:
159
160 1. [ICU Library](http://icu.sourceforge.net/). It is very powerful, complete, feature-rich, mature, and widely used. Also big, intrusive, non-generic, and doesn't play well with the Standard Library. I definitely recommend looking at ICU even if you don't plan to use it.
161 2. C++11 language and library features. Still far from complete, and not easy to use.
162 3. [Glib::ustring](http://www.gtkmm.org/gtkmm2/docs/tutorial/html/ch03s04.html). A class specifically made to work with UTF-8 strings, and also feel like `std::string`. If you prefer to have yet another string class in your code, it may be worth a look. Be aware of the licensing issues, though.
163 4. Platform dependent solutions: Windows and POSIX have functions to convert strings from one encoding to another. That is only a subset of what my library offers, but if that is all you need it may be good enough.
164
165
166 ## Reference
167
168 ### Functions From utf8 Namespace
169
170 #### utf8::append
171
172 Available in version 3.0 and later. Requires a C++ 11 compliant compiler.
173
174 Encodes a 32 bit code point as a UTF-8 sequence of octets and appends the sequence to a UTF-8 string.
175
176 ```cpp
177 void append(char32_t cp, std::string& s);
178 ```
179
180 `cp`: a code point to append to the string.
181 `s`: a utf-8 encoded string to append the code point to.
182
183 Example of use:
184
185 ```cpp
186 std::string u;
187 append(0x0448, u);
188 assert (u[0] == char(0xd1) && u[1] == char(0x88) && u.length() == 2);
189 ```
190
191 In case of an invalid code point, a `utf8::invalid_code_point` exception is thrown.
192
193
194 #### utf8::append
195
196 Available in version 1.0 and later.
197
198 Encodes a 32 bit code point as a UTF-8 sequence of octets and appends the sequence to a UTF-8 string.
199
200 ```cpp
201 template <typename octet_iterator>
202 octet_iterator append(uint32_t cp, octet_iterator result);
203 ```
204
205 `octet_iterator`: an output iterator.
206 `cp`: a 32 bit integer representing a code point to append to the sequence.
207 `result`: an output iterator to the place in the sequence where to append the code point.
208 Return value: an iterator pointing to the place after the newly appended sequence.
209
210 Example of use:
211
212 ```cpp
213 unsigned char u[5] = {0,0,0,0,0};
214 unsigned char* end = append(0x0448, u);
215 assert (u[0] == 0xd1 && u[1] == 0x88 && u[2] == 0 && u[3] == 0 && u[4] == 0);
216 ```
217
218 Note that `append` does not allocate any memory - it is the burden of the caller to make sure there is enough memory allocated for the operation. To make things more interesting, `append` can add anywhere between 1 and 4 octets to the sequence. In practice, you would most often want to use `std::back_inserter` to ensure that the necessary memory is allocated.
219
220 In case of an invalid code point, a `utf8::invalid_code_point` exception is thrown.
221
222 #### utf8::next
223
224 Available in version 1.0 and later.
225
226 Given the iterator to the beginning of the UTF-8 sequence, it returns the code point and moves the iterator to the next position.
227
228 ```cpp
229 template <typename octet_iterator>
230 uint32_t next(octet_iterator& it, octet_iterator end);
231 ```
232
233 `octet_iterator`: an input iterator.
234 `it`: a reference to an iterator pointing to the beginning of an UTF-8 encoded code point. After the function returns, it is incremented to point to the beginning of the next code point.
235 `end`: end of the UTF-8 sequence to be processed. If `it` gets equal to `end` during the extraction of a code point, an `utf8::not_enough_room` exception is thrown.
236 Return value: the 32 bit representation of the processed UTF-8 code point.
237
238 Example of use:
239
240 ```cpp
241 char* twochars = "\xe6\x97\xa5\xd1\x88";
242 char* w = twochars;
243 int cp = next(w, twochars + 6);
244 assert (cp == 0x65e5);
245 assert (w == twochars + 3);
246 ```
247
248 This function is typically used to iterate through a UTF-8 encoded string.
249
250 In case of an invalid UTF-8 sequence, a `utf8::invalid_utf8` exception is thrown.
251
252 #### utf8::peek_next
253
254 Available in version 2.1 and later.
255
256 Given the iterator to the beginning of the UTF-8 sequence, it returns the code point for the following sequence without changing the value of the iterator.
257
258 ```cpp
259 template <typename octet_iterator>
260 uint32_t peek_next(octet_iterator it, octet_iterator end);
261 ```
262
263
264 `octet_iterator`: an input iterator.
265 `it`: an iterator pointing to the beginning of an UTF-8 encoded code point.
266 `end`: end of the UTF-8 sequence to be processed. If `it` gets equal to `end` during the extraction of a code point, an `utf8::not_enough_room` exception is thrown.
267 Return value: the 32 bit representation of the processed UTF-8 code point.
268
269 Example of use:
270
271 ```cpp
272 char* twochars = "\xe6\x97\xa5\xd1\x88";
273 char* w = twochars;
274 int cp = peek_next(w, twochars + 6);
275 assert (cp == 0x65e5);
276 assert (w == twochars);
277 ```
278
279 In case of an invalid UTF-8 sequence, a `utf8::invalid_utf8` exception is thrown.
280
281 #### utf8::prior
282
283 Available in version 1.02 and later.
284
285 Given a reference to an iterator pointing to an octet in a UTF-8 sequence, it decreases the iterator until it hits the beginning of the previous UTF-8 encoded code point and returns the 32 bits representation of the code point.
286
287 ```cpp
288 template <typename octet_iterator>
289 uint32_t prior(octet_iterator& it, octet_iterator start);
290 ```
291
292 `octet_iterator`: a bidirectional iterator.
293 `it`: a reference pointing to an octet within a UTF-8 encoded string. After the function returns, it is decremented to point to the beginning of the previous code point.
294 `start`: an iterator to the beginning of the sequence where the search for the beginning of a code point is performed. It is a safety measure to prevent passing the beginning of the string in the search for a UTF-8 lead octet.
295 Return value: the 32 bit representation of the previous code point.
296
297 Example of use:
298
299 ```cpp
300 char* twochars = "\xe6\x97\xa5\xd1\x88";
301 unsigned char* w = twochars + 3;
302 int cp = prior (w, twochars);
303 assert (cp == 0x65e5);
304 assert (w == twochars);
305 ```
306
307 This function has two purposes: one is two iterate backwards through a UTF-8 encoded string. Note that it is usually a better idea to iterate forward instead, since `utf8::next` is faster. The second purpose is to find a beginning of a UTF-8 sequence if we have a random position within a string. Note that in that case `utf8::prior` may not detect an invalid UTF-8 sequence in some scenarios: for instance if there are superfluous trail octets, it will just skip them.
308
309 `it` will typically point to the beginning of a code point, and `start` will point to the beginning of the string to ensure we don't go backwards too far. `it` is decreased until it points to a lead UTF-8 octet, and then the UTF-8 sequence beginning with that octet is decoded to a 32 bit representation and returned.
310
311 In case `start` is reached before a UTF-8 lead octet is hit, or if an invalid UTF-8 sequence is started by the lead octet, an `invalid_utf8` exception is thrown.
312
313 In case `start` equals `it`, a `not_enough_room` exception is thrown.
314
315 #### utf8::advance
316 Available in version 1.0 and later.
317
318 Advances an iterator by the specified number of code points within an UTF-8 sequence.
319
320 ```cpp
321 template <typename octet_iterator, typename distance_type>
322 void advance (octet_iterator& it, distance_type n, octet_iterator end);
323 ```
324
325 `octet_iterator`: an input iterator.
326 `distance_type`: an integral type convertible to `octet_iterator`'s difference type.
327 `it`: a reference to an iterator pointing to the beginning of an UTF-8 encoded code point. After the function returns, it is incremented to point to the nth following code point.
328 `n`: number of code points `it` should be advanced. A negative value means decrement.
329 `end`: limit of the UTF-8 sequence to be processed. If `n` is positive and `it` gets equal to `end` during the extraction of a code point, an `utf8::not_enough_room` exception is thrown. If `n` is negative and `it` reaches `end` while `it` points t a trail byte of a UTF-8 sequence, a `utf8::invalid_code_point` exception is thrown.
330
331 Example of use:
332
333 ```cpp
334 char* twochars = "\xe6\x97\xa5\xd1\x88";
335 unsigned char* w = twochars;
336 advance (w, 2, twochars + 6);
337 assert (w == twochars + 5);
338 advance (w, -2, twochars);
339 assert (w == twochars);
340 ```
341
342 In case of an invalid code point, a `utf8::invalid_code_point` exception is thrown.
343
344 #### utf8::distance
345
346 Available in version 1.0 and later.
347
348 Given the iterators to two UTF-8 encoded code points in a sequence, returns the number of code points between them.
349
350 ```cpp
351 template <typename octet_iterator>
352 typename std::iterator_traits<octet_iterator>::difference_type distance (octet_iterator first, octet_iterator last);
353 ```
354
355 `octet_iterator`: an input iterator.
356 `first`: an iterator to a beginning of a UTF-8 encoded code point.
357 `last`: an iterator to a "post-end" of the last UTF-8 encoded code point in the sequence we are trying to determine the length. It can be the beginning of a new code point, or not.
358 Return value the distance between the iterators, in code points.
359
360 Example of use:
361
362 ```cpp
363 char* twochars = "\xe6\x97\xa5\xd1\x88";
364 size_t dist = utf8::distance(twochars, twochars + 5);
365 assert (dist == 2);
366 ```
367
368 This function is used to find the length (in code points) of a UTF-8 encoded string. The reason it is called _distance_, rather than, say, _length_ is mainly because developers are used that _length_ is an O(1) function. Computing the length of an UTF-8 string is a linear operation, and it looked better to model it after `std::distance` algorithm.
369
370 In case of an invalid UTF-8 sequence, a `utf8::invalid_utf8` exception is thrown. If `last` does not point to the past-of-end of a UTF-8 sequence, a `utf8::not_enough_room` exception is thrown.
371
372 #### utf8::utf16to8
373
374 Available in version 3.0 and later. Requires a C++ 11 compliant compiler.
375
376 Converts a UTF-16 encoded string to UTF-8.
377
378 ```cpp
379 std::string utf16to8(const std::u16string& s);
380 ```
381
382 `s`: a UTF-16 encoded string.
383 Return value: A UTF-8 encoded string.
384
385 Example of use:
386
387 ```cpp
388 u16string utf16string = {0x41, 0x0448, 0x65e5, 0xd834, 0xdd1e};
389 string u = utf16to8(utf16string);
390 assert (u.size() == 10);
391 ```
392
393 In case of invalid UTF-16 sequence, a `utf8::invalid_utf16` exception is thrown.
394
395 #### utf8::utf16to8
396
397 Available in version 3.2 and later. Requires a C++ 17 compliant compiler.
398
399 Converts a UTF-16 encoded string to UTF-8.
400
401 ```cpp
402 std::string utf16to8(std::u16string_view s);
403 ```
404
405 `s`: a UTF-16 encoded string.
406 Return value: A UTF-8 encoded string.
407
408 Example of use:
409
410 ```cpp
411 u16string utf16string = {0x41, 0x0448, 0x65e5, 0xd834, 0xdd1e};
412 u16string_view utf16stringview(u16string);
413 string u = utf16to8(utf16string);
414 assert (u.size() == 10);
415 ```
416
417 In case of invalid UTF-16 sequence, a `utf8::invalid_utf16` exception is thrown.
418
419
420 #### utf8::utf16to8
421
422 Available in version 1.0 and later.
423
424 Converts a UTF-16 encoded string to UTF-8.
425
426 ```cpp
427 template <typename u16bit_iterator, typename octet_iterator>
428 octet_iterator utf16to8 (u16bit_iterator start, u16bit_iterator end, octet_iterator result);
429 ```
430
431 `u16bit_iterator`: an input iterator.
432 `octet_iterator`: an output iterator.
433 `start`: an iterator pointing to the beginning of the UTF-16 encoded string to convert.
434 `end`: an iterator pointing to pass-the-end of the UTF-16 encoded string to convert.
435 `result`: an output iterator to the place in the UTF-8 string where to append the result of conversion.
436 Return value: An iterator pointing to the place after the appended UTF-8 string.
437
438 Example of use:
439
440 ```cpp
441 unsigned short utf16string[] = {0x41, 0x0448, 0x65e5, 0xd834, 0xdd1e};
442 vector<unsigned char> utf8result;
443 utf16to8(utf16string, utf16string + 5, back_inserter(utf8result));
444 assert (utf8result.size() == 10);
445 ```
446
447 In case of invalid UTF-16 sequence, a `utf8::invalid_utf16` exception is thrown.
448
449 #### utf8::utf8to16
450
451 Available in version 3.0 and later. Requires a C++ 11 compliant compiler.
452
453 Converts an UTF-8 encoded string to UTF-16.
454
455 ```cpp
456 std::u16string utf8to16(const std::string& s);
457 ```
458
459 `s`: an UTF-8 encoded string to convert.
460 Return value: A UTF-16 encoded string
461
462 Example of use:
463
464 ```cpp
465 string utf8_with_surrogates = "\xe6\x97\xa5\xd1\x88\xf0\x9d\x84\x9e";
466 u16string utf16result = utf8to16(utf8_with_surrogates);
467 assert (utf16result.length() == 4);
468 assert (utf16result[2] == 0xd834);
469 assert (utf16result[3] == 0xdd1e);
470 ```
471
472 In case of an invalid UTF-8 sequence, a `utf8::invalid_utf8` exception is thrown.
473
474 #### utf8::utf8to16
475
476 Available in version 3.2 and later. Requires a C++ 17 compliant compiler.
477
478 Converts an UTF-8 encoded string to UTF-16.
479
480 ```cpp
481 std::u16string utf8to16(std::string_view s);
482 ```
483
484 `s`: an UTF-8 encoded string to convert.
485 Return value: A UTF-16 encoded string
486
487 Example of use:
488
489 ```cpp
490 string_view utf8_with_surrogates = "\xe6\x97\xa5\xd1\x88\xf0\x9d\x84\x9e";
491 u16string utf16result = utf8to16(utf8_with_surrogates);
492 assert (utf16result.length() == 4);
493 assert (utf16result[2] == 0xd834);
494 assert (utf16result[3] == 0xdd1e);
495 ```
496
497 In case of an invalid UTF-8 sequence, a `utf8::invalid_utf8` exception is thrown.
498
499
500 #### utf8::utf8to16
501
502 Available in version 1.0 and later.
503
504 Converts an UTF-8 encoded string to UTF-16
505
506 ```cpp
507 template <typename u16bit_iterator, typename octet_iterator>
508 u16bit_iterator utf8to16 (octet_iterator start, octet_iterator end, u16bit_iterator result);
509 ```
510
511 `octet_iterator`: an input iterator.
512 `u16bit_iterator`: an output iterator.
513 `start`: an iterator pointing to the beginning of the UTF-8 encoded string to convert. < br /> `end`: an iterator pointing to pass-the-end of the UTF-8 encoded string to convert.
514 `result`: an output iterator to the place in the UTF-16 string where to append the result of conversion.
515 Return value: An iterator pointing to the place after the appended UTF-16 string.
516
517 Example of use:
518
519 ```cpp
520 char utf8_with_surrogates[] = "\xe6\x97\xa5\xd1\x88\xf0\x9d\x84\x9e";
521 vector <unsigned short> utf16result;
522 utf8to16(utf8_with_surrogates, utf8_with_surrogates + 9, back_inserter(utf16result));
523 assert (utf16result.size() == 4);
524 assert (utf16result[2] == 0xd834);
525 assert (utf16result[3] == 0xdd1e);
526 ```
527
528 In case of an invalid UTF-8 sequence, a `utf8::invalid_utf8` exception is thrown. If `end` does not point to the past-of-end of a UTF-8 sequence, a `utf8::not_enough_room` exception is thrown.
529
530 #### utf8::utf32to8
531
532 Available in version 3.0 and later. Requires a C++ 11 compliant compiler.
533
534 Converts a UTF-32 encoded string to UTF-8.
535
536 ```cpp
537 std::string utf32to8(const std::u32string& s);
538 ```
539
540 `s`: a UTF-32 encoded string.
541 Return value: a UTF-8 encoded string.
542
543 Example of use:
544
545 ```cpp
546 u32string utf32string = {0x448, 0x65E5, 0x10346};
547 string utf8result = utf32to8(utf32string);
548 assert (utf8result.size() == 9);
549 ```
550
551 In case of invalid UTF-32 string, a `utf8::invalid_code_point` exception is thrown.
552
553 #### utf8::utf32to8
554
555 Available in version 3.2 and later. Requires a C++ 17 compliant compiler.
556
557 Converts a UTF-32 encoded string to UTF-8.
558
559 ```cpp
560 std::string utf32to8(std::u32string_view s);
561 ```
562
563 `s`: a UTF-32 encoded string.
564 Return value: a UTF-8 encoded string.
565
566 Example of use:
567
568 ```cpp
569 u32string utf32string = {0x448, 0x65E5, 0x10346};
570 u32string_view utf32stringview(utf32string);
571 string utf8result = utf32to8(utf32stringview);
572 assert (utf8result.size() == 9);
573 ```
574
575 In case of invalid UTF-32 string, a `utf8::invalid_code_point` exception is thrown.
576
577
578 #### utf8::utf32to8
579
580 Available in version 1.0 and later.
581
582 Converts a UTF-32 encoded string to UTF-8.
583
584 ```cpp
585 template <typename octet_iterator, typename u32bit_iterator>
586 octet_iterator utf32to8 (u32bit_iterator start, u32bit_iterator end, octet_iterator result);
587 ```
588
589 `octet_iterator`: an output iterator.
590 `u32bit_iterator`: an input iterator.
591 `start`: an iterator pointing to the beginning of the UTF-32 encoded string to convert.
592 `end`: an iterator pointing to pass-the-end of the UTF-32 encoded string to convert.
593 `result`: an output iterator to the place in the UTF-8 string where to append the result of conversion.
594 Return value: An iterator pointing to the place after the appended UTF-8 string.
595
596 Example of use:
597
598 ```cpp
599 int utf32string[] = {0x448, 0x65E5, 0x10346, 0};
600 vector<unsigned char> utf8result;
601 utf32to8(utf32string, utf32string + 3, back_inserter(utf8result));
602 assert (utf8result.size() == 9);
603 ```
604
605 In case of invalid UTF-32 string, a `utf8::invalid_code_point` exception is thrown.
606
607 #### utf8::utf8to32
608
609 Available in version 3.0 and later. Requires a C++ 11 compliant compiler.
610
611 Converts a UTF-8 encoded string to UTF-32.
612
613 ```cpp
614 std::u32string utf8to32(const std::string& s);
615 ```
616
617 `s`: a UTF-8 encoded string.
618 Return value: a UTF-32 encoded string.
619
620 Example of use:
621
622 ```cpp
623 const char* twochars = "\xe6\x97\xa5\xd1\x88";
624 u32string utf32result = utf8to32(twochars);
625 assert (utf32result.size() == 2);
626 ```
627
628 In case of an invalid UTF-8 sequence, a `utf8::invalid_utf8` exception is thrown.
629
630 #### utf8::utf8to32
631
632 Available in version 3.2 and later. Requires a C++ 17 compliant compiler.
633
634 Converts a UTF-8 encoded string to UTF-32.
635
636 ```cpp
637 std::u32string utf8to32(std::string_view s);
638 ```
639
640 `s`: a UTF-8 encoded string.
641 Return value: a UTF-32 encoded string.
642
643 Example of use:
644
645 ```cpp
646 string_view twochars = "\xe6\x97\xa5\xd1\x88";
647 u32string utf32result = utf8to32(twochars);
648 assert (utf32result.size() == 2);
649 ```
650
651 In case of an invalid UTF-8 sequence, a `utf8::invalid_utf8` exception is thrown.
652
653
654 #### utf8::utf8to32
655
656 Available in version 1.0 and later.
657
658 Converts a UTF-8 encoded string to UTF-32.
659
660 ```cpp
661 template <typename octet_iterator, typename u32bit_iterator>
662 u32bit_iterator utf8to32 (octet_iterator start, octet_iterator end, u32bit_iterator result);
663 ```
664
665 `octet_iterator`: an input iterator.
666 `u32bit_iterator`: an output iterator.
667 `start`: an iterator pointing to the beginning of the UTF-8 encoded string to convert.
668 `end`: an iterator pointing to pass-the-end of the UTF-8 encoded string to convert.
669 `result`: an output iterator to the place in the UTF-32 string where to append the result of conversion.
670 Return value: An iterator pointing to the place after the appended UTF-32 string.
671
672 Example of use:
673
674 ```cpp
675 char* twochars = "\xe6\x97\xa5\xd1\x88";
676 vector<int> utf32result;
677 utf8to32(twochars, twochars + 5, back_inserter(utf32result));
678 assert (utf32result.size() == 2);
679 ```
680
681 In case of an invalid UTF-8 sequence, a `utf8::invalid_utf8` exception is thrown. If `end` does not point to the past-of-end of a UTF-8 sequence, a `utf8::not_enough_room` exception is thrown.
682
683 #### utf8::find_invalid
684
685 Available in version 3.0 and later. Requires a C++ 11 compliant compiler.
686
687 Detects an invalid sequence within a UTF-8 string.
688
689 ```cpp
690 std::size_t find_invalid(const std::string& s);
691 ```
692
693 `s`: a UTF-8 encoded string.
694 Return value: the index of the first invalid octet in the UTF-8 string. In case none were found, equals `std::string::npos`.
695
696 Example of use:
697
698 ```cpp
699 string utf_invalid = "\xe6\x97\xa5\xd1\x88\xfa";
700 auto invalid = find_invalid(utf_invalid);
701 assert (invalid == 5);
702 ```
703
704 This function is typically used to make sure a UTF-8 string is valid before processing it with other functions. It is especially important to call it if before doing any of the _unchecked_ operations on it.
705
706 #### utf8::find_invalid
707
708 Available in version 3.2 and later. Requires a C++ 17 compliant compiler.
709
710 Detects an invalid sequence within a UTF-8 string.
711
712 ```cpp
713 std::size_t find_invalid(std::string_view s);
714 ```
715
716 `s`: a UTF-8 encoded string.
717 Return value: the index of the first invalid octet in the UTF-8 string. In case none were found, equals `std::string_view::npos`.
718
719 Example of use:
720
721 ```cpp
722 string_view utf_invalid = "\xe6\x97\xa5\xd1\x88\xfa";
723 auto invalid = find_invalid(utf_invalid);
724 assert (invalid == 5);
725 ```
726
727 This function is typically used to make sure a UTF-8 string is valid before processing it with other functions. It is especially important to call it if before doing any of the _unchecked_ operations on it.
728
729
730 #### utf8::find_invalid
731
732 Available in version 1.0 and later.
733
734 Detects an invalid sequence within a UTF-8 string.
735
736 ```cpp
737 template <typename octet_iterator>
738 octet_iterator find_invalid(octet_iterator start, octet_iterator end);
739 ```
740
741 `octet_iterator`: an input iterator.
742 `start`: an iterator pointing to the beginning of the UTF-8 string to test for validity.
743 `end`: an iterator pointing to pass-the-end of the UTF-8 string to test for validity.
744 Return value: an iterator pointing to the first invalid octet in the UTF-8 string. In case none were found, equals `end`.
745
746 Example of use:
747
748 ```cpp
749 char utf_invalid[] = "\xe6\x97\xa5\xd1\x88\xfa";
750 char* invalid = find_invalid(utf_invalid, utf_invalid + 6);
751 assert (invalid == utf_invalid + 5);
752 ```
753
754 This function is typically used to make sure a UTF-8 string is valid before processing it with other functions. It is especially important to call it if before doing any of the _unchecked_ operations on it.
755
756 #### utf8::is_valid
757
758 Available in version 3.0 and later. Requires a C++ 11 compliant compiler.
759
760 Checks whether a string object contains valid UTF-8 encoded text.
761
762 ```cpp
763 bool is_valid(const std::string& s);
764 ```
765
766 `s`: a UTF-8 encoded string.
767 Return value: `true` if the string contains valid UTF-8 encoded text; `false` if not.
768
769 Example of use:
770
771 ```cpp
772 char utf_invalid[] = "\xe6\x97\xa5\xd1\x88\xfa";
773 bool bvalid = is_valid(utf_invalid);
774 assert (bvalid == false);
775 ```
776
777 You may want to use `is_valid` to make sure that a string contains valid UTF-8 text without the need to know where it fails if it is not valid.
778
779 #### utf8::is_valid
780
781 Available in version 3.2 and later. Requires a C++ 17 compliant compiler.
782
783 Checks whether a string object contains valid UTF-8 encoded text.
784
785 ```cpp
786 bool is_valid(std::string_view s);
787 ```
788
789 `s`: a UTF-8 encoded string.
790 Return value: `true` if the string contains valid UTF-8 encoded text; `false` if not.
791
792 Example of use:
793
794 ```cpp
795 string_view utf_invalid = "\xe6\x97\xa5\xd1\x88\xfa";
796 bool bvalid = is_valid(utf_invalid);
797 assert (bvalid == false);
798 ```
799
800 You may want to use `is_valid` to make sure that a string contains valid UTF-8 text without the need to know where it fails if it is not valid.
801
802
803 #### utf8::is_valid
804
805 Available in version 1.0 and later.
806
807 Checks whether a sequence of octets is a valid UTF-8 string.
808
809 ```cpp
810 template <typename octet_iterator>
811 bool is_valid(octet_iterator start, octet_iterator end);
812 ```
813
814 `octet_iterator`: an input iterator.
815 `start`: an iterator pointing to the beginning of the UTF-8 string to test for validity.
816 `end`: an iterator pointing to pass-the-end of the UTF-8 string to test for validity.
817 Return value: `true` if the sequence is a valid UTF-8 string; `false` if not.
818
819 Example of use:
820
821 ```cpp
822 char utf_invalid[] = "\xe6\x97\xa5\xd1\x88\xfa";
823 bool bvalid = is_valid(utf_invalid, utf_invalid + 6);
824 assert (bvalid == false);
825 ```
826
827 `is_valid` is a shorthand for `find_invalid(start, end) == end;`. You may want to use it to make sure that a byte sequence is a valid UTF-8 string without the need to know where it fails if it is not valid.
828
829 #### utf8::replace_invalid
830
831 Available in version 3.0 and later. Requires a C++ 11 compliant compiler.
832
833 Replaces all invalid UTF-8 sequences within a string with a replacement marker.
834
835 ```cpp
836 std::string replace_invalid(const std::string& s, char32_t replacement);
837 std::string replace_invalid(const std::string& s);
838 ```
839
840 `s`: a UTF-8 encoded string.
841 `replacement`: A Unicode code point for the replacement marker. The version without this parameter assumes the value `0xfffd`
842 Return value: A UTF-8 encoded string with replaced invalid sequences.
843
844 Example of use:
845
846 ```cpp
847 string invalid_sequence = "a\x80\xe0\xa0\xc0\xaf\xed\xa0\x80z";
848 string replace_invalid_result = replace_invalid(invalid_sequence, '?');
849 bvalid = is_valid(replace_invalid_result);
850 assert (bvalid);
851 const string fixed_invalid_sequence = "a????z";
852 assert (fixed_invalid_sequence == replace_invalid_result);
853 ```
854
855 #### utf8::replace_invalid
856
857 Available in version 3.2 and later. Requires a C++ 17 compliant compiler.
858
859 Replaces all invalid UTF-8 sequences within a string with a replacement marker.
860
861 ```cpp
862 std::string replace_invalid(std::string_view s, char32_t replacement);
863 std::string replace_invalid(std::string_view s);
864 ```
865
866 `s`: a UTF-8 encoded string.
867 `replacement`: A Unicode code point for the replacement marker. The version without this parameter assumes the value `0xfffd`
868 Return value: A UTF-8 encoded string with replaced invalid sequences.
869
870 Example of use:
871
872 ```cpp
873 string_view invalid_sequence = "a\x80\xe0\xa0\xc0\xaf\xed\xa0\x80z";
874 string replace_invalid_result = replace_invalid(invalid_sequence, '?');
875 bool bvalid = is_valid(replace_invalid_result);
876 assert (bvalid);
877 const string fixed_invalid_sequence = "a????z";
878 assert(fixed_invalid_sequence, replace_invalid_result);
879 ```
880
881
882 #### utf8::replace_invalid
883
884 Available in version 2.0 and later.
885
886 Replaces all invalid UTF-8 sequences within a string with a replacement marker.
887
888 ```cpp
889 template <typename octet_iterator, typename output_iterator>
890 output_iterator replace_invalid(octet_iterator start, octet_iterator end, output_iterator out, uint32_t replacement);
891 template <typename octet_iterator, typename output_iterator>
892 output_iterator replace_invalid(octet_iterator start, octet_iterator end, output_iterator out);
893 ```
894
895 `octet_iterator`: an input iterator.
896 `output_iterator`: an output iterator.
897 `start`: an iterator pointing to the beginning of the UTF-8 string to look for invalid UTF-8 sequences.
898 `end`: an iterator pointing to pass-the-end of the UTF-8 string to look for invalid UTF-8 sequences.
899 `out`: An output iterator to the range where the result of replacement is stored.
900 `replacement`: A Unicode code point for the replacement marker. The version without this parameter assumes the value `0xfffd`
901 Return value: An iterator pointing to the place after the UTF-8 string with replaced invalid sequences.
902
903 Example of use:
904
905 ```cpp
906 char invalid_sequence[] = "a\x80\xe0\xa0\xc0\xaf\xed\xa0\x80z";
907 vector<char> replace_invalid_result;
908 replace_invalid (invalid_sequence, invalid_sequence + sizeof(invalid_sequence), back_inserter(replace_invalid_result), '?');
909 bvalid = is_valid(replace_invalid_result.begin(), replace_invalid_result.end());
910 assert (bvalid);
911 char* fixed_invalid_sequence = "a????z";
912 assert (std::equal(replace_invalid_result.begin(), replace_invalid_result.end(), fixed_invalid_sequence));
913 ```
914
915 `replace_invalid` does not perform in-place replacement of invalid sequences. Rather, it produces a copy of the original string with the invalid sequences replaced with a replacement marker. Therefore, `out` must not be in the `[start, end]` range.
916
917 #### utf8::starts_with_bom
918
919 Available in version 3.0 and later. Requires a C++ 11 compliant compiler.
920
921 Checks whether a string starts with a UTF-8 byte order mark (BOM)
922
923 ```cpp
924 bool starts_with_bom(const std::string& s);
925 ```
926
927 `s`: a UTF-8 encoded string.
928 Return value: `true` if the string starts with a UTF-8 byte order mark; `false` if not.
929
930 Example of use:
931
932 ```cpp
933 string byte_order_mark = {char(0xef), char(0xbb), char(0xbf)};
934 bool bbom = starts_with_bom(byte_order_mark);
935 assert (bbom == true);
936 string threechars = "\xf0\x90\x8d\x86\xe6\x97\xa5\xd1\x88";
937 bool no_bbom = starts_with_bom(threechars);
938 assert (no_bbom == false);
939 ```
940
941 The typical use of this function is to check the first three bytes of a file. If they form the UTF-8 BOM, we want to skip them before processing the actual UTF-8 encoded text.
942
943
944 #### utf8::starts_with_bom
945
946 Available in version 3.2 and later. Requires a C++ 17 compliant compiler.
947
948 Checks whether a string starts with a UTF-8 byte order mark (BOM)
949
950 ```cpp
951 bool starts_with_bom(std::string_view s);
952 ```
953
954 `s`: a UTF-8 encoded string.
955 Return value: `true` if the string starts with a UTF-8 byte order mark; `false` if not.
956
957 Example of use:
958
959 ```cpp
960 string byte_order_mark = {char(0xef), char(0xbb), char(0xbf)};
961 string_view byte_order_mark_view(byte_order_mark);
962 bool bbom = starts_with_bom(byte_order_mark_view);
963 assert (bbom);
964 string_view threechars = "\xf0\x90\x8d\x86\xe6\x97\xa5\xd1\x88";
965 bool no_bbom = starts_with_bom(threechars);
966 assert (!no_bbom);
967 ```
968
969 The typical use of this function is to check the first three bytes of a file. If they form the UTF-8 BOM, we want to skip them before processing the actual UTF-8 encoded text.
970
971
972 #### utf8::starts_with_bom
973
974 Available in version 2.3 and later.
975
976 Checks whether an octet sequence starts with a UTF-8 byte order mark (BOM)
977
978 ```cpp
979 template <typename octet_iterator>
980 bool starts_with_bom (octet_iterator it, octet_iterator end);
981 ```
982
983 `octet_iterator`: an input iterator.
984 `it`: beginning of the octet sequence to check
985 `end`: pass-end of the sequence to check
986 Return value: `true` if the sequence starts with a UTF-8 byte order mark; `false` if not.
987
988 Example of use:
989
990 ```cpp
991 unsigned char byte_order_mark[] = {0xef, 0xbb, 0xbf};
992 bool bbom = starts_with_bom(byte_order_mark, byte_order_mark + sizeof(byte_order_mark));
993 assert (bbom == true);
994 ```
995
996 The typical use of this function is to check the first three bytes of a file. If they form the UTF-8 BOM, we want to skip them before processing the actual UTF-8 encoded text.
997
998 ### Types From utf8 Namespace
999
1000 #### utf8::exception
1001
1002 Available in version 2.3 and later.
1003
1004 Base class for the exceptions thrown by UTF CPP library functions.
1005
1006 ```cpp
1007 class exception : public std::exception {};
1008 ```
1009
1010 Example of use:
1011
1012 ```cpp
1013 try {
1014 code_that_uses_utf_cpp_library();
1015 }
1016 catch(const utf8::exception& utfcpp_ex) {
1017 cerr << utfcpp_ex.what();
1018 }
1019 ```
1020
1021 #### utf8::invalid_code_point
1022
1023 Available in version 1.0 and later.
1024
1025 Thrown by UTF8 CPP functions such as `advance` and `next` if an UTF-8 sequence represents and invalid code point.
1026
1027 ```cpp
1028 class invalid_code_point : public exception {
1029 public:
1030 uint32_t code_point() const;
1031 };
1032 ```
1033
1034 Member function `code_point()` can be used to determine the invalid code point that caused the exception to be thrown.
1035
1036 #### utf8::invalid_utf8
1037
1038 Available in version 1.0 and later.
1039
1040 Thrown by UTF8 CPP functions such as `next` and `prior` if an invalid UTF-8 sequence is detected during decoding.
1041
1042 ```cpp
1043 class invalid_utf8 : public exception {
1044 public:
1045 uint8_t utf8_octet() const;
1046 };
1047 ```
1048
1049 Member function `utf8_octet()` can be used to determine the beginning of the byte sequence that caused the exception to be thrown.
1050
1051 #### utf8::invalid_utf16
1052
1053 Available in version 1.0 and later.
1054
1055 Thrown by UTF8 CPP function `utf16to8` if an invalid UTF-16 sequence is detected during decoding.
1056
1057 ```cpp
1058 class invalid_utf16 : public exception {
1059 public:
1060 uint16_t utf16_word() const;
1061 };
1062 ```
1063
1064 Member function `utf16_word()` can be used to determine the UTF-16 code unit that caused the exception to be thrown.
1065
1066 #### utf8::not_enough_room
1067
1068 Available in version 1.0 and later.
1069
1070 Thrown by UTF8 CPP functions such as `next` if the end of the decoded UTF-8 sequence was reached before the code point was decoded.
1071
1072 ```cpp
1073 class not_enough_room : public exception {};
1074 ```
1075
1076 #### utf8::iterator
1077
1078 Available in version 2.0 and later.
1079
1080 Adapts the underlying octet iterator to iterate over the sequence of code points, rather than raw octets.
1081
1082 ```cpp
1083 template <typename octet_iterator>
1084 class iterator;
1085 ```
1086
1087 ##### Member functions
1088
1089 `iterator();` the deafult constructor; the underlying octet_iterator is constructed with its default constructor.
1090
1091 `explicit iterator (const octet_iterator& octet_it, const octet_iterator& range_start, const octet_iterator& range_end);` a constructor that initializes the underlying octet_iterator with octet_it and sets the range in which the iterator is considered valid.
1092
1093 `octet_iterator base () const;` returns the underlying octet_iterator.
1094
1095 `uint32_t operator * () const;` decodes the utf-8 sequence the underlying octet_iterator is pointing to and returns the code point.
1096
1097 `bool operator == (const iterator& rhs) const;` returns `true` if the two underlying iterators are equal.
1098
1099 `bool operator != (const iterator& rhs) const;` returns `true` if the two underlying iterators are not equal.
1100
1101 `iterator& operator ++ ();` the prefix increment - moves the iterator to the next UTF-8 encoded code point.
1102
1103 `iterator operator ++ (int);` the postfix increment - moves the iterator to the next UTF-8 encoded code point and returns the current one.
1104
1105 `iterator& operator -- ();` the prefix decrement - moves the iterator to the previous UTF-8 encoded code point.
1106
1107 `iterator operator -- (int);` the postfix decrement - moves the iterator to the previous UTF-8 encoded code point and returns the current one.
1108
1109 Example of use:
1110
1111 ```cpp
1112 char* threechars = "\xf0\x90\x8d\x86\xe6\x97\xa5\xd1\x88";
1113 utf8::iterator<char*> it(threechars, threechars, threechars + 9);
1114 utf8::iterator<char*> it2 = it;
1115 assert (it2 == it);
1116 assert (*it == 0x10346);
1117 assert (*(++it) == 0x65e5);
1118 assert ((*it++) == 0x65e5);
1119 assert (*it == 0x0448);
1120 assert (it != it2);
1121 utf8::iterator<char*> endit (threechars + 9, threechars, threechars + 9);
1122 assert (++it == endit);
1123 assert (*(--it) == 0x0448);
1124 assert ((*it--) == 0x0448);
1125 assert (*it == 0x65e5);
1126 assert (--it == utf8::iterator<char*>(threechars, threechars, threechars + 9));
1127 assert (*it == 0x10346);
1128 ```
1129
1130 The purpose of `utf8::iterator` adapter is to enable easy iteration as well as the use of STL algorithms with UTF-8 encoded strings. Increment and decrement operators are implemented in terms of `utf8::next()` and `utf8::prior()` functions.
1131
1132 Note that `utf8::iterator` adapter is a checked iterator. It operates on the range specified in the constructor; any attempt to go out of that range will result in an exception. Even the comparison operators require both iterator object to be constructed against the same range - otherwise an exception is thrown. Typically, the range will be determined by sequence container functions `begin` and `end`, i.e.:
1133
1134 ```cpp
1135 std::string s = "example";
1136 utf8::iterator i (s.begin(), s.begin(), s.end());
1137 ```
1138
1139 ### Functions From utf8::unchecked Namespace
1140
1141 #### utf8::unchecked::append
1142
1143 Available in version 1.0 and later.
1144
1145 Encodes a 32 bit code point as a UTF-8 sequence of octets and appends the sequence to a UTF-8 string.
1146
1147 ```cpp
1148 template <typename octet_iterator>
1149 octet_iterator append(uint32_t cp, octet_iterator result);
1150 ```
1151
1152 `cp`: A 32 bit integer representing a code point to append to the sequence.
1153 `result`: An output iterator to the place in the sequence where to append the code point.
1154 Return value: An iterator pointing to the place after the newly appended sequence.
1155
1156 Example of use:
1157
1158 ```cpp
1159 unsigned char u[5] = {0,0,0,0,0};
1160 unsigned char* end = unchecked::append(0x0448, u);
1161 assert (u[0] == 0xd1 && u[1] == 0x88 && u[2] == 0 && u[3] == 0 && u[4] == 0);
1162 ```
1163
1164 This is a faster but less safe version of `utf8::append`. It does not check for validity of the supplied code point, and may produce an invalid UTF-8 sequence.
1165
1166 #### utf8::unchecked::next
1167
1168 Available in version 1.0 and later.
1169
1170 Given the iterator to the beginning of a UTF-8 sequence, it returns the code point and moves the iterator to the next position.
1171
1172 ```cpp
1173 template <typename octet_iterator>
1174 uint32_t next(octet_iterator& it);
1175 ```
1176
1177 `it`: a reference to an iterator pointing to the beginning of an UTF-8 encoded code point. After the function returns, it is incremented to point to the beginning of the next code point.
1178 Return value: the 32 bit representation of the processed UTF-8 code point.
1179
1180 Example of use:
1181
1182 ```cpp
1183 char* twochars = "\xe6\x97\xa5\xd1\x88";
1184 char* w = twochars;
1185 int cp = unchecked::next(w);
1186 assert (cp == 0x65e5);
1187 assert (w == twochars + 3);
1188 ```
1189
1190 This is a faster but less safe version of `utf8::next`. It does not check for validity of the supplied UTF-8 sequence.
1191
1192 #### utf8::unchecked::peek_next
1193
1194 Available in version 2.1 and later.
1195
1196 Given the iterator to the beginning of a UTF-8 sequence, it returns the code point.
1197
1198 ```cpp
1199 template <typename octet_iterator>
1200 uint32_t peek_next(octet_iterator it);
1201 ```
1202
1203 `it`: an iterator pointing to the beginning of an UTF-8 encoded code point.
1204 Return value: the 32 bit representation of the processed UTF-8 code point.
1205
1206 Example of use:
1207
1208 ```cpp
1209 char* twochars = "\xe6\x97\xa5\xd1\x88";
1210 char* w = twochars;
1211 int cp = unchecked::peek_next(w);
1212 assert (cp == 0x65e5);
1213 assert (w == twochars);
1214 ```
1215
1216 This is a faster but less safe version of `utf8::peek_next`. It does not check for validity of the supplied UTF-8 sequence.
1217
1218 #### utf8::unchecked::prior
1219
1220 Available in version 1.02 and later.
1221
1222 Given a reference to an iterator pointing to an octet in a UTF-8 sequence, it decreases the iterator until it hits the beginning of the previous UTF-8 encoded code point and returns the 32 bits representation of the code point.
1223
1224 ```cpp
1225 template <typename octet_iterator>
1226 uint32_t prior(octet_iterator& it);
1227 ```
1228
1229 `it`: a reference pointing to an octet within a UTF-8 encoded string. After the function returns, it is decremented to point to the beginning of the previous code point.
1230 Return value: the 32 bit representation of the previous code point.
1231
1232 Example of use:
1233
1234 ```cpp
1235 char* twochars = "\xe6\x97\xa5\xd1\x88";
1236 char* w = twochars + 3;
1237 int cp = unchecked::prior (w);
1238 assert (cp == 0x65e5);
1239 assert (w == twochars);
1240 ```
1241
1242 This is a faster but less safe version of `utf8::prior`. It does not check for validity of the supplied UTF-8 sequence and offers no boundary checking.
1243
1244 #### utf8::unchecked::advance
1245
1246 Available in version 1.0 and later.
1247
1248 Advances an iterator by the specified number of code points within an UTF-8 sequence.
1249
1250 ```cpp
1251 template <typename octet_iterator, typename distance_type>
1252 void advance (octet_iterator& it, distance_type n);
1253 ```
1254
1255 `it`: a reference to an iterator pointing to the beginning of an UTF-8 encoded code point. After the function returns, it is incremented to point to the nth following code point.
1256 `n`: number of code points `it` should be advanced. A negative value means decrement.
1257
1258 Example of use:
1259
1260 ```cpp
1261 char* twochars = "\xe6\x97\xa5\xd1\x88";
1262 char* w = twochars;
1263 unchecked::advance (w, 2);
1264 assert (w == twochars + 5);
1265 ```
1266
1267 This is a faster but less safe version of `utf8::advance`. It does not check for validity of the supplied UTF-8 sequence and offers no boundary checking.
1268
1269 #### utf8::unchecked::distance
1270
1271 Available in version 1.0 and later.
1272
1273 Given the iterators to two UTF-8 encoded code points in a sequence, returns the number of code points between them.
1274
1275 ```cpp
1276 template <typename octet_iterator>
1277 typename std::iterator_traits<octet_iterator>::difference_type distance (octet_iterator first, octet_iterator last);
1278 ```
1279
1280 `first`: an iterator to a beginning of a UTF-8 encoded code point.
1281 `last`: an iterator to a "post-end" of the last UTF-8 encoded code point in the sequence we are trying to determine the length. It can be the beginning of a new code point, or not.
1282 Return value: the distance between the iterators, in code points.
1283
1284 Example of use:
1285
1286 ```cpp
1287 char* twochars = "\xe6\x97\xa5\xd1\x88";
1288 size_t dist = utf8::unchecked::distance(twochars, twochars + 5);
1289 assert (dist == 2);
1290 ```
1291
1292 This is a faster but less safe version of `utf8::distance`. It does not check for validity of the supplied UTF-8 sequence.
1293
1294 #### utf8::unchecked::utf16to8
1295
1296 Available in version 1.0 and later.
1297
1298 Converts a UTF-16 encoded string to UTF-8.
1299
1300 ```cpp
1301 template <typename u16bit_iterator, typename octet_iterator>
1302 octet_iterator utf16to8 (u16bit_iterator start, u16bit_iterator end, octet_iterator result);
1303 ```
1304
1305 `start`: an iterator pointing to the beginning of the UTF-16 encoded string to convert.
1306 `end`: an iterator pointing to pass-the-end of the UTF-16 encoded string to convert.
1307 `result`: an output iterator to the place in the UTF-8 string where to append the result of conversion.
1308 Return value: An iterator pointing to the place after the appended UTF-8 string.
1309
1310 Example of use:
1311
1312 ```cpp
1313 unsigned short utf16string[] = {0x41, 0x0448, 0x65e5, 0xd834, 0xdd1e};
1314 vector<unsigned char> utf8result;
1315 unchecked::utf16to8(utf16string, utf16string + 5, back_inserter(utf8result));
1316 assert (utf8result.size() == 10);
1317 ```
1318
1319 This is a faster but less safe version of `utf8::utf16to8`. It does not check for validity of the supplied UTF-16 sequence.
1320
1321 #### utf8::unchecked::utf8to16
1322
1323 Available in version 1.0 and later.
1324
1325 Converts an UTF-8 encoded string to UTF-16
1326
1327 ```cpp
1328 template <typename u16bit_iterator, typename octet_iterator>
1329 u16bit_iterator utf8to16 (octet_iterator start, octet_iterator end, u16bit_iterator result);
1330 ```
1331
1332 `start`: an iterator pointing to the beginning of the UTF-8 encoded string to convert. < br /> `end`: an iterator pointing to pass-the-end of the UTF-8 encoded string to convert.
1333 `result`: an output iterator to the place in the UTF-16 string where to append the result of conversion.
1334 Return value: An iterator pointing to the place after the appended UTF-16 string.
1335
1336 Example of use:
1337
1338 ```cpp
1339 char utf8_with_surrogates[] = "\xe6\x97\xa5\xd1\x88\xf0\x9d\x84\x9e";
1340 vector <unsigned short> utf16result;
1341 unchecked::utf8to16(utf8_with_surrogates, utf8_with_surrogates + 9, back_inserter(utf16result));
1342 assert (utf16result.size() == 4);
1343 assert (utf16result[2] == 0xd834);
1344 assert (utf16result[3] == 0xdd1e);
1345 ```
1346
1347 This is a faster but less safe version of `utf8::utf8to16`. It does not check for validity of the supplied UTF-8 sequence.
1348
1349 #### utf8::unchecked::utf32to8
1350
1351 Available in version 1.0 and later.
1352
1353 Converts a UTF-32 encoded string to UTF-8.
1354
1355 ```cpp
1356 template <typename octet_iterator, typename u32bit_iterator>
1357 octet_iterator utf32to8 (u32bit_iterator start, u32bit_iterator end, octet_iterator result);
1358 ```
1359
1360 `start`: an iterator pointing to the beginning of the UTF-32 encoded string to convert.
1361 `end`: an iterator pointing to pass-the-end of the UTF-32 encoded string to convert.
1362 `result`: an output iterator to the place in the UTF-8 string where to append the result of conversion.
1363 Return value: An iterator pointing to the place after the appended UTF-8 string.
1364
1365 Example of use:
1366
1367 ```cpp
1368 int utf32string[] = {0x448, 0x65e5, 0x10346, 0};
1369 vector<unsigned char> utf8result;
1370 utf32to8(utf32string, utf32string + 3, back_inserter(utf8result));
1371 assert (utf8result.size() == 9);
1372 ```
1373
1374 This is a faster but less safe version of `utf8::utf32to8`. It does not check for validity of the supplied UTF-32 sequence.
1375
1376 #### utf8::unchecked::utf8to32
1377
1378 Available in version 1.0 and later.
1379
1380 Converts a UTF-8 encoded string to UTF-32.
1381
1382 ```cpp
1383 template <typename octet_iterator, typename u32bit_iterator>
1384 u32bit_iterator utf8to32 (octet_iterator start, octet_iterator end, u32bit_iterator result);
1385 ```
1386
1387 `start`: an iterator pointing to the beginning of the UTF-8 encoded string to convert.
1388 `end`: an iterator pointing to pass-the-end of the UTF-8 encoded string to convert.
1389 `result`: an output iterator to the place in the UTF-32 string where to append the result of conversion.
1390 Return value: An iterator pointing to the place after the appended UTF-32 string.
1391
1392 Example of use:
1393
1394 ```cpp
1395 char* twochars = "\xe6\x97\xa5\xd1\x88";
1396 vector<int> utf32result;
1397 unchecked::utf8to32(twochars, twochars + 5, back_inserter(utf32result));
1398 assert (utf32result.size() == 2);
1399 ```
1400
1401 This is a faster but less safe version of `utf8::utf8to32`. It does not check for validity of the supplied UTF-8 sequence.
1402
1403 #### utf8::unchecked::replace_invalid
1404
1405 Available in version 3.1 and later.
1406
1407 Replaces all invalid UTF-8 sequences within a string with a replacement marker.
1408
1409 ```cpp
1410 template <typename octet_iterator, typename output_iterator>
1411 output_iterator replace_invalid(octet_iterator start, octet_iterator end, output_iterator out, uint32_t replacement);
1412 template <typename octet_iterator, typename output_iterator>
1413 output_iterator replace_invalid(octet_iterator start, octet_iterator end, output_iterator out);
1414 ```
1415
1416 `octet_iterator`: an input iterator.
1417 `output_iterator`: an output iterator.
1418 `start`: an iterator pointing to the beginning of the UTF-8 string to look for invalid UTF-8 sequences.
1419 `end`: an iterator pointing to pass-the-end of the UTF-8 string to look for invalid UTF-8 sequences.
1420 `out`: An output iterator to the range where the result of replacement is stored.
1421 `replacement`: A Unicode code point for the replacement marker. The version without this parameter assumes the value `0xfffd`
1422 Return value: An iterator pointing to the place after the UTF-8 string with replaced invalid sequences.
1423
1424 Example of use:
1425
1426 ```cpp
1427 char invalid_sequence[] = "a\x80\xe0\xa0\xc0\xaf\xed\xa0\x80z";
1428 vector<char> replace_invalid_result;
1429 unchecked::replace_invalid (invalid_sequence, invalid_sequence + sizeof(invalid_sequence), back_inserter(replace_invalid_result), '?');
1430 bvalid = utf8::is_valid(replace_invalid_result.begin(), replace_invalid_result.end());
1431 assert (bvalid);
1432 char* fixed_invalid_sequence = "a????z";
1433 assert (std::equal(replace_invalid_result.begin(), replace_invalid_result.end(), fixed_invalid_sequence));
1434 ```
1435
1436 `replace_invalid` does not perform in-place replacement of invalid sequences. Rather, it produces a copy of the original string with the invalid sequences replaced with a replacement marker. Therefore, `out` must not be in the `[start, end]` range.
1437
1438 Unlike `utf8::replace_invalid`, this function does not verify validity of the replacement marker.
1439
1440 ### Types From utf8::unchecked Namespace
1441
1442 #### utf8::iterator
1443
1444 Available in version 2.0 and later.
1445
1446 Adapts the underlying octet iterator to iterate over the sequence of code points, rather than raw octets.
1447
1448 ```cpp
1449 template <typename octet_iterator>
1450 class iterator;
1451 ```
1452
1453 ##### Member functions
1454
1455 `iterator();` the deafult constructor; the underlying octet_iterator is constructed with its default constructor.
1456
1457 `explicit iterator (const octet_iterator& octet_it);` a constructor that initializes the underlying octet_iterator with `octet_it`.
1458
1459 `octet_iterator base () const;` returns the underlying octet_iterator.
1460
1461 `uint32_t operator * () const;` decodes the utf-8 sequence the underlying octet_iterator is pointing to and returns the code point.
1462
1463 `bool operator == (const iterator& rhs) const;` returns `true` if the two underlying iterators are equal.
1464
1465 `bool operator != (const iterator& rhs) const;` returns `true` if the two underlying iterators are not equal.
1466
1467 `iterator& operator ++ ();` the prefix increment - moves the iterator to the next UTF-8 encoded code point.
1468
1469 `iterator operator ++ (int);` the postfix increment - moves the iterator to the next UTF-8 encoded code point and returns the current one.
1470
1471 `iterator& operator -- ();` the prefix decrement - moves the iterator to the previous UTF-8 encoded code point.
1472
1473 `iterator operator -- (int);` the postfix decrement - moves the iterator to the previous UTF-8 encoded code point and returns the current one.
1474
1475 Example of use:
1476
1477 ```cpp
1478 char* threechars = "\xf0\x90\x8d\x86\xe6\x97\xa5\xd1\x88";
1479 utf8::unchecked::iterator<char*> un_it(threechars);
1480 utf8::unchecked::iterator<char*> un_it2 = un_it;
1481 assert (un_it2 == un_it);
1482 assert (*un_it == 0x10346);
1483 assert (*(++un_it) == 0x65e5);
1484 assert ((*un_it++) == 0x65e5);
1485 assert (*un_it == 0x0448);
1486 assert (un_it != un_it2);
1487 utf8::::unchecked::iterator<char*> un_endit (threechars + 9);
1488 assert (++un_it == un_endit);
1489 assert (*(--un_it) == 0x0448);
1490 assert ((*un_it--) == 0x0448);
1491 assert (*un_it == 0x65e5);
1492 assert (--un_it == utf8::unchecked::iterator<char*>(threechars));
1493 assert (*un_it == 0x10346);
1494 ```
1495
1496 This is an unchecked version of `utf8::iterator`. It is faster in many cases, but offers no validity or range checks.
1497
1498 ## Links
1499
1500 1. [The Unicode Consortium](http://www.unicode.org/).
1501 2. [ICU Library](http://icu.sourceforge.net/).
1502 3. [UTF-8 at Wikipedia](http://en.wikipedia.org/wiki/UTF-8)
1503 4. [UTF-8 and Unicode FAQ for Unix/Linux](http://www.cl.cam.ac.uk/~mgk25/unicode.html)