Debian Codegraph @

.cargo/vendor/encoding_rs/README.md

raw
1 # encoding_rs
2
3 [![Build Status](https://travis-ci.org/hsivonen/encoding_rs.svg?branch=master)](https://travis-ci.org/hsivonen/encoding_rs)
4 [![crates.io](https://img.shields.io/crates/v/encoding_rs.svg)](https://crates.io/crates/encoding_rs)
5 [![docs.rs](https://docs.rs/encoding_rs/badge.svg)](https://docs.rs/encoding_rs/)
6
7 encoding_rs an implementation of the (non-JavaScript parts of) the
8 [Encoding Standard](https://encoding.spec.whatwg.org/) written in Rust.
9
10 The Encoding Standard defines the Web-compatible set of character encodings,
11 which means this crate can be used to decode Web content. encoding_rs is
12 used in Gecko starting with Firefox 56. Due to the notable overlap between
13 the legacy encodings on the Web and the legacy encodings used on Windows,
14 this crate may be of use for non-Web-related situations as well; see below
15 for links to adjacent crates.
16
17 Additionally, the `mem` module provides various operations for dealing with
18 in-RAM text (as opposed to data that's coming from or going to an IO boundary).
19 The `mem` module is a module instead of a separate crate due to internal
20 implementation detail efficiencies.
21
22 ## Functionality
23
24 Due to the Gecko use case, encoding_rs supports decoding to and encoding from
25 UTF-16 in addition to supporting the usual Rust use case of decoding to and
26 encoding from UTF-8. Additionally, the API has been designed to be FFI-friendly
27 to accommodate the C++ side of Gecko.
28
29 Specifically, encoding_rs does the following:
30
31 * Decodes a stream of bytes in an Encoding Standard-defined character encoding
32 into valid aligned native-endian in-RAM UTF-16 (units of `u16` / `char16_t`).
33 * Encodes a stream of potentially-invalid aligned native-endian in-RAM UTF-16
34 (units of `u16` / `char16_t`) into a sequence of bytes in an Encoding
35 Standard-defined character encoding as if the lone surrogates had been
36 replaced with the REPLACEMENT CHARACTER before performing the encode.
37 (Gecko's UTF-16 is potentially invalid.)
38 * Decodes a stream of bytes in an Encoding Standard-defined character
39 encoding into valid UTF-8.
40 * Encodes a stream of valid UTF-8 into a sequence of bytes in an Encoding
41 Standard-defined character encoding. (Rust's UTF-8 is guaranteed-valid.)
42 * Does the above in streaming (input and output split across multiple
43 buffers) and non-streaming (whole input in a single buffer and whole
44 output in a single buffer) variants.
45 * Avoids copying (borrows) when possible in the non-streaming cases when
46 decoding to or encoding from UTF-8.
47 * Resolves textual labels that identify character encodings in
48 protocol text into type-safe objects representing the those encodings
49 conceptually.
50 * Maps the type-safe encoding objects onto strings suitable for
51 returning from `document.characterSet`.
52 * Validates UTF-8 (in common instruction set scenarios a bit faster for Web
53 workloads than the standard library; hopefully will get upstreamed some
54 day) and ASCII.
55
56 Additionally, `encoding_rs::mem` does the following:
57
58 * Checks if a byte buffer contains only ASCII.
59 * Checks if a potentially-invalid UTF-16 buffer contains only Basic Latin (ASCII).
60 * Checks if a valid UTF-8, potentially-invalid UTF-8 or potentially-invalid UTF-16
61 buffer contains only Latin1 code points (below U+0100).
62 * Checks if a valid UTF-8, potentially-invalid UTF-8 or potentially-invalid UTF-16
63 buffer or a code point or a UTF-16 code unit can trigger right-to-left behavior
64 (suitable for checking if the Unicode Bidirectional Algorithm can be optimized
65 out).
66 * Combined versions of the above two checks.
67 * Converts valid UTF-8, potentially-invalid UTF-8 and Latin1 to UTF-16.
68 * Converts potentially-invalid UTF-16 and Latin1 to UTF-8.
69 * Converts UTF-8 and UTF-16 to Latin1 (if in range).
70 * Finds the first invalid code unit in a buffer of potentially-invalid UTF-16.
71 * Makes a mutable buffer of potential-invalid UTF-16 contain valid UTF-16.
72 * Copies ASCII from one buffer to another up to the first non-ASCII byte.
73 * Converts ASCII to UTF-16 up to the first non-ASCII byte.
74 * Converts UTF-16 to ASCII up to the first non-Basic Latin code unit.
75
76 ## Integration with `std::io`
77
78 Notably, the above feature list doesn't include the capability to wrap
79 a `std::io::Read`, decode it into UTF-8 and presenting the result via
80 `std::io::Read`. The [`encoding_rs_io`](https://crates.io/crates/encoding_rs_io)
81 crate provides that capability.
82
83 ## `no_std` Environment
84
85 The crate works in a `no_std` environment. By default, the `alloc` feature,
86 which assumes that an allocator is present is enabled. For a no-allocator
87 environment, the default features (i.e. `alloc`) can be turned off. This
88 makes the part of the API that returns `Vec`/`String`/`Cow` unavailable.
89
90 ## Decoding Email
91
92 For decoding character encodings that occur in email, use the
93 [`charset`](https://crates.io/crates/charset) crate instead of using this
94 one directly. (It wraps this crate and adds UTF-7 decoding.)
95
96 ## Windows Code Page Identifier Mappings
97
98 For mappings to and from Windows code page identifiers, use the
99 [`codepage`](https://crates.io/crates/codepage) crate.
100
101 ## DOS Encodings
102
103 This crate does not support single-byte DOS encodings that aren't required by
104 the Web Platform, but the [`oem_cp`](https://crates.io/crates/oem_cp) crate does.
105
106 ## Preparing Text for the Encoders
107
108 Normalizing text into Unicode Normalization Form C prior to encoding text into
109 a legacy encoding minimizes unmappable characters. Text can be normalized to
110 Unicode Normalization Form C using the
111 [`unic-normal`](https://crates.io/crates/unic-normal) crate.
112
113 The exception is windows-1258, which after normalizing to Unicode Normalization
114 Form C requires tone marks to be decomposed in order to minimize unmappable
115 characters. Vietnamese tone marks can be decomposed using the
116 [`detone`](https://crates.io/crates/detone) crate.
117
118 ## Licensing
119
120 TL;DR: `(Apache-2.0 OR MIT) AND BSD-3-Clause` for the code and data combination.
121
122 Please see the file named
123 [COPYRIGHT](https://github.com/hsivonen/encoding_rs/blob/master/COPYRIGHT).
124
125 The non-test code that isn't generated from the WHATWG data in this crate is
126 under Apache-2.0 OR MIT. Test code is under CC0.
127
128 This crate contains code/data generated from WHATWG-supplied data. The WHATWG
129 upstream changed its license for portions of specs incorporated into source code
130 from CC0 to BSD-3-Clause between the initial release of this crate and the present
131 version of this crate. The in-source licensing legends have been updated for the
132 parts of the generated code that have changed since the upstream license change.
133
134 ## Documentation
135
136 Generated [API documentation](https://docs.rs/encoding_rs/) is available
137 online.
138
139 There is a [long-form write-up](https://hsivonen.fi/encoding_rs/) about the
140 design and internals of the crate.
141
142 ## C and C++ bindings
143
144 An FFI layer for encoding_rs is available as a
145 [separate crate](https://github.com/hsivonen/encoding_c). The crate comes
146 with a [demo C++ wrapper](https://github.com/hsivonen/encoding_c/blob/master/include/encoding_rs_cpp.h)
147 using the C++ standard library and [GSL](https://github.com/Microsoft/GSL/) types.
148
149 The bindings for the `mem` module are in the
150 [encoding_c_mem crate](https://github.com/hsivonen/encoding_c_mem).
151
152 For the Gecko context, there's a
153 [C++ wrapper using the MFBT/XPCOM types](https://searchfox.org/mozilla-central/source/intl/Encoding.h#100).
154
155 There's a [write-up](https://hsivonen.fi/modern-cpp-in-rust/) about the C++
156 wrappers.
157
158 ## Sample programs
159
160 * [Rust](https://github.com/hsivonen/recode_rs)
161 * [C](https://github.com/hsivonen/recode_c)
162 * [C++](https://github.com/hsivonen/recode_cpp)
163
164 ## Optional features
165
166 There are currently these optional cargo features:
167
168 ### `simd-accel`
169
170 Enables SIMD acceleration using the nightly-dependent `portable_simd` standard
171 library feature.
172
173 This is an opt-in feature, because enabling this feature _opts out_ of Rust's
174 guarantees of future compilers compiling old code (aka. "stability story").
175
176 Currently, this has not been tested to be an improvement except for these
177 targets and enabling the `simd-accel` feature is expected to break the build
178 on other targets:
179
180 * x86_64
181 * i686
182 * aarch64
183 * thumbv7neon
184
185 If you use nightly Rust, you use targets whose first component is one of the
186 above, and you are prepared _to have to revise your configuration when updating
187 Rust_, you should enable this feature. Otherwise, please _do not_ enable this
188 feature.
189
190 Used by Firefox.
191
192 ### `serde`
193
194 Enables support for serializing and deserializing `&'static Encoding`-typed
195 struct fields using [Serde][1].
196
197 [1]: https://serde.rs/
198
199 Not used by Firefox.
200
201 ### `fast-legacy-encode`
202
203 A catch-all option for enabling the fastest legacy encode options. _Does not
204 affect decode speed or UTF-8 encode speed._
205
206 At present, this option is equivalent to enabling the following options:
207 * `fast-hangul-encode`
208 * `fast-hanja-encode`
209 * `fast-kanji-encode`
210 * `fast-gb-hanzi-encode`
211 * `fast-big5-hanzi-encode`
212
213 Adds 176 KB to the binary size.
214
215 Not used by Firefox.
216
217 ### `fast-hangul-encode`
218
219 Changes encoding precomposed Hangul syllables into EUC-KR from binary
220 search over the decode-optimized tables to lookup by index making Korean
221 plain-text encode about 4 times as fast as without this option.
222
223 Adds 20 KB to the binary size.
224
225 Does _not_ affect decode speed.
226
227 Not used by Firefox.
228
229 ### `fast-hanja-encode`
230
231 Changes encoding of Hanja into EUC-KR from linear search over the
232 decode-optimized table to lookup by index. Since Hanja is practically absent
233 in modern Korean text, this option doesn't affect perfomance in the common
234 case and mainly makes sense if you want to make your application resilient
235 agaist denial of service by someone intentionally feeding it a lot of Hanja
236 to encode into EUC-KR.
237
238 Adds 40 KB to the binary size.
239
240 Does _not_ affect decode speed.
241
242 Not used by Firefox.
243
244 ### `fast-kanji-encode`
245
246 Changes encoding of Kanji into Shift_JIS, EUC-JP and ISO-2022-JP from linear
247 search over the decode-optimized tables to lookup by index making Japanese
248 plain-text encode to legacy encodings 30 to 50 times as fast as without this
249 option (about 2 times as fast as with `less-slow-kanji-encode`).
250
251 Takes precedence over `less-slow-kanji-encode`.
252
253 Adds 36 KB to the binary size (24 KB compared to `less-slow-kanji-encode`).
254
255 Does _not_ affect decode speed.
256
257 Not used by Firefox.
258
259 ### `less-slow-kanji-encode`
260
261 Makes JIS X 0208 Level 1 Kanji (the most common Kanji in Shift_JIS, EUC-JP and
262 ISO-2022-JP) encode less slow (binary search instead of linear search) making
263 Japanese plain-text encode to legacy encodings 14 to 23 times as fast as
264 without this option.
265
266 Adds 12 KB to the binary size.
267
268 Does _not_ affect decode speed.
269
270 Not used by Firefox.
271
272 ### `fast-gb-hanzi-encode`
273
274 Changes encoding of Hanzi in the CJK Unified Ideographs block into GBK and
275 gb18030 from linear search over a part the decode-optimized tables followed
276 by a binary search over another part of the decode-optimized tables to lookup
277 by index making Simplified Chinese plain-text encode to the legacy encodings
278 100 to 110 times as fast as without this option (about 2.5 times as fast as
279 with `less-slow-gb-hanzi-encode`).
280
281 Takes precedence over `less-slow-gb-hanzi-encode`.
282
283 Adds 36 KB to the binary size (24 KB compared to `less-slow-gb-hanzi-encode`).
284
285 Does _not_ affect decode speed.
286
287 Not used by Firefox.
288
289 ### `less-slow-gb-hanzi-encode`
290
291 Makes GB2312 Level 1 Hanzi (the most common Hanzi in gb18030 and GBK) encode
292 less slow (binary search instead of linear search) making Simplified Chinese
293 plain-text encode to the legacy encodings about 40 times as fast as without
294 this option.
295
296 Adds 12 KB to the binary size.
297
298 Does _not_ affect decode speed.
299
300 Not used by Firefox.
301
302 ### `fast-big5-hanzi-encode`
303
304 Changes encoding of Hanzi in the CJK Unified Ideographs block into Big5 from
305 linear search over a part the decode-optimized tables to lookup by index
306 making Traditional Chinese plain-text encode to Big5 105 to 125 times as fast
307 as without this option (about 3 times as fast as with
308 `less-slow-big5-hanzi-encode`).
309
310 Takes precedence over `less-slow-big5-hanzi-encode`.
311
312 Adds 40 KB to the binary size (20 KB compared to `less-slow-big5-hanzi-encode`).
313
314 Does _not_ affect decode speed.
315
316 Not used by Firefox.
317
318 ### `less-slow-big5-hanzi-encode`
319
320 Makes Big5 Level 1 Hanzi (the most common Hanzi in Big5) encode less slow
321 (binary search instead of linear search) making Traditional Chinese
322 plain-text encode to Big5 about 36 times as fast as without this option.
323
324 Adds 20 KB to the binary size.
325
326 Does _not_ affect decode speed.
327
328 Not used by Firefox.
329
330 ## Performance goals
331
332 For decoding to UTF-16, the goal is to perform at least as well as Gecko's old
333 uconv. For decoding to UTF-8, the goal is to perform at least as well as
334 rust-encoding. These goals have been achieved.
335
336 Encoding to UTF-8 should be fast. (UTF-8 to UTF-8 encode should be equivalent
337 to `memcpy` and UTF-16 to UTF-8 should be fast.)
338
339 Speed is a non-goal when encoding to legacy encodings. By default, encoding to
340 legacy encodings should not be optimized for speed at the expense of code size
341 as long as form submission and URL parsing in Gecko don't become noticeably
342 too slow in real-world use.
343
344 In the interest of binary size, by default, encoding_rs does not have
345 encode-specific data tables beyond 32 bits of encode-specific data for each
346 single-byte encoding. Therefore, encoders search the decode-optimized data
347 tables. This is a linear search in most cases. As a result, by default, encode
348 to legacy encodings varies from slow to extremely slow relative to other
349 libraries. Still, with realistic work loads, this seemed fast enough not to be
350 user-visibly slow on Raspberry Pi 3 (which stood in for a phone for testing)
351 in the Web-exposed encoder use cases.
352
353 See the cargo features above for optionally making CJK legacy encode fast.
354
355 A framework for measuring performance is [available separately][2].
356
357 [2]: https://github.com/hsivonen/encoding_bench/
358
359 ## Rust Version Compatibility
360
361 It is a goal to support the latest stable Rust, the latest nightly Rust and
362 the version of Rust that's used for Firefox Nightly.
363
364 At this time, there is no firm commitment to support a version older than
365 what's required by Firefox, and there is no commitment to treat MSRV changes
366 as semver-breaking, because this crate depends on `cfg-if`, which doesn't
367 appear to treat MSRV changes as semver-breaking, so it would be useless for
368 this crate to treat MSRV changes as semver-breaking.
369
370 As of 2024-04-04, MSRV appears to be Rust 1.36.0 for using the crate and
371 1.42.0 for doc tests to pass without errors about the global allocator.
372 With the `simd-accel` feature, the MSRV is even higher.
373
374 ## Compatibility with rust-encoding
375
376 A compatibility layer that implements the rust-encoding API on top of
377 encoding_rs is
378 [provided as a separate crate](https://github.com/hsivonen/encoding_rs_compat)
379 (cannot be uploaded to crates.io). The compatibility layer was originally
380 written with the assuption that Firefox would need it, but it is not currently
381 used in Firefox.
382
383 ## Regenerating Generated Code
384
385 To regenerate the generated code:
386
387 * Have Python 2 installed.
388 * Clone [`https://github.com/hsivonen/encoding_c`](https://github.com/hsivonen/encoding_c)
389 next to the `encoding_rs` directory.
390 * Clone [`https://github.com/hsivonen/codepage`](https://github.com/hsivonen/codepage)
391 next to the `encoding_rs` directory.
392 * Clone [`https://github.com/whatwg/encoding`](https://github.com/whatwg/encoding)
393 next to the `encoding_rs` directory.
394 * Checkout revision `be3337450e7df1c49dca7872153c4c4670dd8256` of the `encoding` repo.
395 (Note: `f381389` was the revision of `encoding` used from before the `encoding` repo
396 license change. So far, only output changed since then has been updated to
397 the new license legend.)
398 * With the `encoding_rs` directory as the working directory, run
399 `python generate-encoding-data.py`.
400
401 ## Roadmap
402
403 - [x] Design the low-level API.
404 - [x] Provide Rust-only convenience features.
405 - [x] Provide an stl/gsl-flavored C++ API.
406 - [x] Implement all decoders and encoders.
407 - [x] Add unit tests for all decoders and encoders.
408 - [x] Finish BOM sniffing variants in Rust-only convenience features.
409 - [x] Document the API.
410 - [x] Publish the crate on crates.io.
411 - [x] Create a solution for measuring performance.
412 - [x] Accelerate ASCII conversions using SSE2 on x86.
413 - [x] Accelerate ASCII conversions using ALU register-sized operations on
414 non-x86 architectures (process an `usize` instead of `u8` at a time).
415 - [x] Split FFI into a separate crate so that the FFI doesn't interfere with
416 LTO in pure-Rust usage.
417 - [x] Compress CJK indices by making use of sequential code points as well
418 as Unicode-ordered parts of indices.
419 - [x] Make lookups by label or name use binary search that searches from the
420 end of the label/name to the start.
421 - [x] Make labels with non-ASCII bytes fail fast.
422 - [ ] ~Parallelize UTF-8 validation using [Rayon](https://github.com/nikomatsakis/rayon).~
423 (This turned out to be a pessimization in the ASCII case due to memory bandwidth reasons.)
424 - [x] Provide an XPCOM/MFBT-flavored C++ API.
425 - [x] Investigate accelerating single-byte encode with a single fast-tracked
426 range per encoding.
427 - [x] Replace uconv with encoding_rs in Gecko.
428 - [x] Implement the rust-encoding API in terms of encoding_rs.
429 - [x] Add SIMD acceleration for Aarch64.
430 - [x] Investigate the use of NEON on 32-bit ARM.
431 - [ ] ~Investigate Björn Höhrmann's lookup table acceleration for UTF-8 as
432 adapted to Rust in rust-encoding.~
433 - [x] Add actually fast CJK encode options.
434 - [ ] ~Investigate [Bob Steagall's lookup table acceleration for UTF-8](https://github.com/BobSteagall/CppNow2018/blob/master/FastConversionFromUTF-8/Fast%20Conversion%20From%20UTF-8%20with%20C%2B%2B%2C%20DFAs%2C%20and%20SSE%20Intrinsics%20-%20Bob%20Steagall%20-%20C%2B%2BNow%202018.pdf).~
435 - [x] Provide a build mode that works without `alloc` (with lesser API surface).
436 - [x] Migrate to `std::simd` ~once it is stable and declare 1.0.~
437 - [ ] Migrate `unsafe` slice access by larger types than `u8`/`u16` to `align_to`.
438
439 ## Release Notes
440
441 ### 0.8.34
442
443 * Use the `portable_simd` nightly feature of the standard library instead of the `packed_simd` crate. Only affects the `simd-accel` optional nightly feature.
444 * Internal documentation improvements and minor code improvements around `unsafe`.
445 * Added `rust-version` to `Cargo.toml`.
446
447 ### 0.8.33
448
449 * Use `packed_simd` instead of `packed_simd_2` again now that updates are back under the `packed_simd` name. Only affects the `simd-accel` optional nightly feature.
450
451 ### 0.8.32
452
453 * Removed `build.rs`. (This removal should resolve false positives reported by some antivirus products. This may break some build configurations that have opted out of Rust's guarantees against future build breakage.)
454 * Internal change to what API is used for reinterpreting the lane configuration of SIMD vectors.
455 * Documentation improvements.
456
457 ### 0.8.31
458
459 * Use SPDX with parentheses now that crates.io supports parentheses.
460
461 ### 0.8.30
462
463 * Update the licensing information to take into account the WHATWG data license change.
464
465 ### 0.8.29
466
467 * Make the parts that use an allocator optional.
468
469 ### 0.8.28
470
471 * Fix error in Serde support introduced as part of `no_std` support.
472
473 ### 0.8.27
474
475 * Make the crate works in a `no_std` environment (with `alloc`).
476
477 ### 0.8.26
478
479 * Fix oversights in edition 2018 migration that broke the `simd-accel` feature.
480
481 ### 0.8.25
482
483 * Do pointer alignment checks in a way where intermediate steps aren't defined to be Undefined Behavior.
484 * Update the `packed_simd` dependency to `packed_simd_2`.
485 * Update the `cfg-if` dependency to 1.0.
486 * Address warnings that have been introduced by newer Rust versions along the way.
487 * Update to edition 2018, since even prior to 1.0 `cfg-if` updated to edition 2018 without a semver break.
488
489 ### 0.8.24
490
491 * Avoid computing an intermediate (not dereferenced) pointer value in a manner designated as Undefined Behavior when computing pointer alignment.
492
493 ### 0.8.23
494
495 * Remove year from copyright notices. (No features or bug fixes.)
496
497 ### 0.8.22
498
499 * Formatting fix and new unit test. (No features or bug fixes.)
500
501 ### 0.8.21
502
503 * Fixed a panic with invalid UTF-16[BE|LE] input at the end of the stream.
504
505 ### 0.8.20
506
507 * Make `Decoder::latin1_byte_compatible_up_to` return `None` in more
508 cases to make the method actually useful. While this could be argued
509 to be a breaking change due to the bug fix changing semantics, it does
510 not break callers that had to handle the `None` case in a reasonable
511 way anyway.
512
513 ### 0.8.19
514
515 * Removed a bunch of bound checks in `convert_str_to_utf16`.
516 * Added `mem::convert_utf8_to_utf16_without_replacement`.
517
518 ### 0.8.18
519
520 * Added `mem::utf8_latin1_up_to` and `mem::str_latin1_up_to`.
521 * Added `Decoder::latin1_byte_compatible_up_to`.
522
523 ### 0.8.17
524
525 * Update `bincode` (dev dependency) version requirement to 1.0.
526
527 ### 0.8.16
528
529 * Switch from the `simd` crate to `packed_simd`.
530
531 ### 0.8.15
532
533 * Adjust documentation for `simd-accel` (README-only release).
534
535 ### 0.8.14
536
537 * Made UTF-16 to UTF-8 encode conversion fill the output buffer as
538 closely as possible.
539
540 ### 0.8.13
541
542 * Made the UTF-8 to UTF-16 decoder compare the number of code units written
543 with the length of the right slice (the output slice) to fix a panic
544 introduced in 0.8.11.
545
546 ### 0.8.12
547
548 * Removed the `clippy::` prefix from clippy lint names.
549
550 ### 0.8.11
551
552 * Changed minimum Rust requirement to 1.29.0 (for the ability to refer
553 to the interior of a `static` when defining another `static`).
554 * Explicitly aligned the lookup tables for single-byte encodings and
555 UTF-8 to cache lines in the hope of freeing up one cache line for
556 other data. (Perhaps the tables were already aligned and this is
557 placebo.)
558 * Added 32 bits of encode-oriented data for each single-byte encoding.
559 The change was performance-neutral for non-Latin1-ish Latin legacy
560 encodings, improved Latin1-ish and Arabic legacy encode speed
561 somewhat (new speed is 2.4x the old speed for German, 2.3x for
562 Arabic, 1.7x for Portuguese and 1.4x for French) and improved
563 non-Latin1, non-Arabic legacy single-byte encode a lot (7.2x for
564 Thai, 6x for Greek, 5x for Russian, 4x for Hebrew).
565 * Added compile-time options for fast CJK legacy encode options (at
566 the cost of binary size (up to 176 KB) and run-time memory usage).
567 These options still retain the overall code structure instead of
568 rewriting the CJK encoders totally, so the speed isn't as good as
569 what could be achieved by using even more memory / making the
570 binary even langer.
571 * Made UTF-8 decode and validation faster.
572 * Added method `is_single_byte()` on `Encoding`.
573 * Added `mem::decode_latin1()` and `mem::encode_latin1_lossy()`.
574
575 ### 0.8.10
576
577 * Disabled a unit test that tests a panic condition when the assertion
578 being tested is disabled.
579
580 ### 0.8.9
581
582 * Made `--features simd-accel` work with stable-channel compiler to
583 simplify the Firefox build system.
584
585 ### 0.8.8
586
587 * Made the `is_foo_bidi()` not treat U+FEFF (ZERO WIDTH NO-BREAK SPACE
588 aka. BYTE ORDER MARK) as right-to-left.
589 * Made the `is_foo_bidi()` functions report `true` if the input contains
590 Hebrew presentations forms (which are right-to-left but not in a
591 right-to-left-roadmapped block).
592
593 ### 0.8.7
594
595 * Fixed a panic in the UTF-16LE/UTF-16BE decoder when decoding to UTF-8.
596
597 ### 0.8.6
598
599 * Temporarily removed the debug assertion added in version 0.8.5 from
600 `convert_utf16_to_latin1_lossy`.
601
602 ### 0.8.5
603
604 * If debug assertions are enabled but fuzzing isn't enabled, lossy conversions
605 to Latin1 in the `mem` module assert that the input is in the range
606 U+0000...U+00FF (inclusive).
607 * In the `mem` module provide conversions from Latin1 and UTF-16 to UTF-8
608 that can deal with insufficient output space. The idea is to use them
609 first with an allocation rounded up to jemalloc bucket size and do the
610 worst-case allocation only if the jemalloc rounding up was insufficient
611 as the first guess.
612
613 ### 0.8.4
614
615 * Fix SSE2-specific, `simd-accel`-specific memory corruption introduced in
616 version 0.8.1 in conversions between UTF-16 and Latin1 in the `mem` module.
617
618 ### 0.8.3
619
620 * Removed an `#[inline(never)]` annotation that was not meant for release.
621
622 ### 0.8.2
623
624 * Made non-ASCII UTF-16 to UTF-8 encode faster by manually omitting bound
625 checks and manually adding branch prediction annotations.
626
627 ### 0.8.1
628
629 * Tweaked loop unrolling and memory alignment for SSE2 conversions between
630 UTF-16 and Latin1 in the `mem` module to increase the performance when
631 converting long buffers.
632
633 ### 0.8.0
634
635 * Changed the minimum supported version of Rust to 1.21.0 (semver breaking
636 change).
637 * Flipped around the defaults vs. optional features for controlling the size
638 vs. speed trade-off for Kanji and Hanzi legacy encode (semver breaking
639 change).
640 * Added NEON support on ARMv7.
641 * SIMD-accelerated x-user-defined to UTF-16 decode.
642 * Made UTF-16LE and UTF-16BE decode a lot faster (including SIMD
643 acceleration).
644
645 ### 0.7.2
646
647 * Add the `mem` module.
648 * Refactor SIMD code which can affect performance outside the `mem`
649 module.
650
651 ### 0.7.1
652
653 * When encoding from invalid UTF-16, correctly handle U+DC00 followed by
654 another low surrogate.
655
656 ### 0.7.0
657
658 * [Make `replacement` a label of the replacement
659 encoding.](https://github.com/whatwg/encoding/issues/70) (Spec change.)
660 * Remove `Encoding::for_name()`. (`Encoding::for_label(foo).unwrap()` is
661 now close enough after the above label change.)
662 * Remove the `parallel-utf8` cargo feature.
663 * Add optional Serde support for `&'static Encoding`.
664 * Performance tweaks for ASCII handling.
665 * Performance tweaks for UTF-8 validation.
666 * SIMD support on aarch64.
667
668 ### 0.6.11
669
670 * Make `Encoder::has_pending_state()` public.
671 * Update the `simd` crate dependency to 0.2.0.
672
673 ### 0.6.10
674
675 * Reserve enough space for NCRs when encoding to ISO-2022-JP.
676 * Correct max length calculations for multibyte decoders.
677 * Correct max length calculations before BOM sniffing has been
678 performed.
679 * Correctly calculate max length when encoding from UTF-16 to GBK.
680
681 ### 0.6.9
682
683 * [Don't prepend anything when gb18030 range decode
684 fails](https://github.com/whatwg/encoding/issues/110). (Spec change.)
685
686 ### 0.6.8
687
688 * Correcly handle the case where the first buffer contains potentially
689 partial BOM and the next buffer is the last buffer.
690 * Decode byte `7F` correctly in ISO-2022-JP.
691 * Make UTF-16 to UTF-8 encode write closer to the end of the buffer.
692 * Implement `Hash` for `Encoding`.
693
694 ### 0.6.7
695
696 * [Map half-width katakana to full-width katana in ISO-2022-JP
697 encoder](https://github.com/whatwg/encoding/issues/105). (Spec change.)
698 * Give `InputEmpty` correct precedence over `OutputFull` when encoding
699 with replacement and the output buffer passed in is too short or the
700 remaining space in the output buffer is too small after a replacement.
701
702 ### 0.6.6
703
704 * Correct max length calculation when a partial BOM prefix is part of
705 the decoder's state.
706
707 ### 0.6.5
708
709 * Correct max length calculation in various encoders.
710 * Correct max length calculation in the UTF-16 decoder.
711 * Derive `PartialEq` and `Eq` for the `CoderResult`, `DecoderResult`
712 and `EncoderResult` types.
713
714 ### 0.6.4
715
716 * Avoid panic when encoding with replacement and the destination buffer is
717 too short to hold one numeric character reference.
718
719 ### 0.6.3
720
721 * Add support for 32-bit big-endian hosts. (For real this time.)
722
723 ### 0.6.2
724
725 * Fix a panic from subslicing with bad indices in
726 `Encoder::encode_from_utf16`. (Due to an oversight, it lacked the fix that
727 `Encoder::encode_from_utf8` already had.)
728 * Micro-optimize error status accumulation in non-streaming case.
729
730 ### 0.6.1
731
732 * Avoid panic near integer overflow in a case that's unlikely to actually
733 happen.
734 * Address Clippy lints.
735
736 ### 0.6.0
737
738 * Make the methods for computing worst-case buffer size requirements check
739 for integer overflow.
740 * Upgrade rayon to 0.7.0.
741
742 ### 0.5.1
743
744 * Reorder methods for better documentation readability.
745 * Add support for big-endian hosts. (Only 64-bit case actually tested.)
746 * Optimize the ALU (non-SIMD) case for 32-bit ARM instead of x86_64.
747
748 ### 0.5.0
749
750 * Avoid allocating an excessively long buffers in non-streaming decode.
751 * Fix the behavior of ISO-2022-JP and replacement decoders near the end of the
752 output buffer.
753 * Annotate the result structs with `#[must_use]`.
754
755 ### 0.4.0
756
757 * Split FFI into a separate crate.
758 * Performance tweaks.
759 * CJK binary size and encoding performance changes.
760 * Parallelize UTF-8 validation in the case of long buffers (with optional
761 feature `parallel-utf8`).
762 * Borrow even with ISO-2022-JP when possible.
763
764 ### 0.3.2
765
766 * Fix moving pointers to alignment in ALU-based ASCII acceleration.
767 * Fix errors in documentation and improve documentation.
768
769 ### 0.3.1
770
771 * Fix UTF-8 to UTF-16 decode for byte sequences beginning with 0xEE.
772 * Make UTF-8 to UTF-8 decode SSE2-accelerated when feature `simd-accel` is used.
773 * When decoding and encoding ASCII-only input from or to an ASCII-compatible
774 encoding using the non-streaming API, return a borrow of the input.
775 * Make encode from UTF-16 to UTF-8 faster.
776
777 ### 0.3
778
779 * Change the references to the instances of `Encoding` from `const` to `static`
780 to make the referents unique across crates that use the refernces.
781 * Introduce non-reference-typed `FOO_INIT` instances of `Encoding` to allow
782 foreign crates to initialize `static` arrays with references to `Encoding`
783 instances even under Rust's constraints that prohibit the initialization of
784 `&'static Encoding`-typed array items with `&'static Encoding`-typed
785 `statics`.
786 * Document that the above two points will be reverted if Rust changes `const`
787 to work so that cross-crate usage keeps the referents unique.
788 * Return `Cow`s from Rust-only non-streaming methods for encode and decode.
789 * `Encoding::for_bom()` returns the length of the BOM.
790 * ASCII-accelerated conversions for encodings other than UTF-16LE, UTF-16BE,
791 ISO-2022-JP and x-user-defined.
792 * Add SSE2 acceleration behind the `simd-accel` feature flag. (Requires
793 nightly Rust.)
794 * Fix panic with long bogus labels.
795 * Map [0xCA to U+05BA in windows-1255](https://github.com/whatwg/encoding/issues/73).
796 (Spec change.)
797 * Correct the [end of the Shift_JIS EUDC range](https://github.com/whatwg/encoding/issues/53).
798 (Spec change.)
799
800 ### 0.2.4
801
802 * Polish FFI documentation.
803
804 ### 0.2.3
805
806 * Fix UTF-16 to UTF-8 encode.
807
808 ### 0.2.2
809
810 * Add `Encoder.encode_from_utf8_to_vec_without_replacement()`.
811
812 ### 0.2.1
813
814 * Add `Encoding.is_ascii_compatible()`.
815
816 * Add `Encoding::for_bom()`.
817
818 * Make `==` for `Encoding` use name comparison instead of pointer comparison,
819 because uses of the encoding constants in different crates result in
820 different addresses and the constant cannot be turned into statics without
821 breaking other things.
822
823 ### 0.2.0
824
825 The initial release.