Debian Codegraph @

.cargo/vendor/widestring/src/lib.rs

raw
1 //! A wide string library for converting to and from wide string variants.
2 //!
3 //! This library provides multiple types of wide strings, each corresponding to a string types in
4 //! the Rust standard library. [`Utf16String`] and [`Utf32String`] are analogous to the standard
5 //! [`String`] type, providing a similar interface, and are always encoded as valid UTF-16 and
6 //! UTF-32, respectively. They are the only type in this library that can losslessly and infallibly
7 //! convert to and from [`String`], and are the easiest type to work with. They are not designed for
8 //! working with FFI, but do support efficient conversions from the FFI types.
9 //!
10 //! [`U16String`] and [`U32String`], on the other hand, are similar to (but not the same as),
11 //! [`OsString`], and are designed around working with FFI. Unlike the UTF variants, these strings
12 //! do not have a defined encoding, and can work with any wide character strings, regardless of
13 //! the encoding. They can be converted to and from [`OsString`] (but may require an encoding
14 //! conversion depending on the platform), although that string type is an OS-specified
15 //! encoding, so take special care.
16 //!
17 //! [`U16String`] and [`U32String`] also allow access and mutation that relies on the user
18 //! to enforce any constraints on the data. Some methods do assume a UTF encoding, but do so in a
19 //! way that handles malformed encoding data. For FFI, use [`U16String`] or [`U32String`] when you
20 //! simply need to pass-through string data, or when you're not dealing with a nul-terminated data.
21 //!
22 //! Finally, [`U16CString`] and [`U32CString`] are wide version of the standard [`CString`] type.
23 //! Like [`U16String`] and [`U32String`], they do not have defined encoding, but are designed to
24 //! work with FFI, particularly C-style nul-terminated wide string data. These C-style strings are
25 //! always terminated in a nul value, and are guaranteed to contain no interior nul values (unless
26 //! unchecked methods are used). Again, these types may contain ill-formed encoding data, and
27 //! methods handle it appropriately. Use [`U16CString`] or [`U32CString`] anytime you must properly
28 //! handle nul values for when dealing with wide string C FFI.
29 //!
30 //! Like the standard Rust string types, each wide string type has its corresponding wide string
31 //! slice type, as shown in the following table:
32 //!
33 //! | String Type | Slice Type |
34 //! |-----------------|--------------|
35 //! | [`Utf16String`] | [`Utf16Str`] |
36 //! | [`Utf32String`] | [`Utf32Str`] |
37 //! | [`U16String`] | [`U16Str`] |
38 //! | [`U32String`] | [`U32Str`] |
39 //! | [`U16CString`] | [`U16CStr`] |
40 //! | [`U32CString`] | [`U32CStr`] |
41 //!
42 //! All the string types in this library can be converted between string types of the same bit
43 //! width, as well as appropriate standard Rust types, but be lossy and/or require knowledge of the
44 //! underlying encoding. The UTF strings additionally can be converted between the two sizes of
45 //! string, re-encoding the strings.
46 //!
47 //! # Wide string literals
48 //!
49 //! Macros are provided for each wide string slice type that convert standard Rust [`str`] literals
50 //! into UTF-16 or UTF-32 encoded versions of the slice type at *compile time*.
51 //!
52 //! ```
53 //! use widestring::u16str;
54 //! let hello = u16str!("Hello, world!"); // `hello` will be a &U16Str value
55 //! ```
56 //!
57 //! These can be used anywhere a `const` function can be used, and provide a convenient method of
58 //! specifying wide string literals instead of coding values by hand. The resulting string slices
59 //! are always valid UTF encoding, and the [`u16cstr!`] and [`u32cstr!`] macros are automatically
60 //! nul-terminated.
61 //!
62 //! # Cargo features
63 //!
64 //! This crate supports `no_std` when default cargo features are disabled. The `std` and `alloc`
65 //! cargo features (enabled by default) enable the owned string types: [`U16String`], [`U32String`],
66 //! [`U16CString`], [`U32CString`], [`Utf16String`], and [`Utf32String`] types and their modules.
67 //! Other types such as the string slices do not require allocation and can be used in a `no_std`
68 //! environment, even without the [`alloc`](https://doc.rust-lang.org/stable/alloc/index.html)
69 //! crate.
70 //!
71 //! # Remarks on UTF-16 and UTF-32
72 //!
73 //! UTF-16 encoding is a variable-length encoding. The 16-bit code units can specificy Unicode code
74 //! points either as single units or in _surrogate pairs_. Because every value might be part of a
75 //! surrogate pair, many regular string operations on UTF-16 data, including indexing, writing, or
76 //! even iterating, require considering either one or two values at a time. This library provides
77 //! safe methods for these operations when the data is known to be UTF-16, such as with
78 //! [`Utf16String`]. In those cases, keep in mind that the number of elements (`len()`) of the
79 //! wide string is _not_ equivalent to the number of Unicode code points in the string, but is
80 //! instead the number of code unit values.
81 //!
82 //! For [`U16String`] and [`U16CString`], which do not define an encoding, these same operations
83 //! (indexing, mutating, iterating) do _not_ take into account UTF-16 encoding and may result in
84 //! sequences that are ill-formed UTF-16. Some methods are provided that do make an exception to
85 //! this and treat the strings as malformed UTF-16, which are specified in their documentation as to
86 //! how they handle the invalid data.
87 //!
88 //! UTF-32 simply encodes Unicode code points as-is in 32-bit Unicode Scalar Values, but Unicode
89 //! character code points are reserved only for 21-bits, and UTF-16 surrogates are invalid in
90 //! UTF-32. Since UTF-32 is a fixed-width encoding, it is much easier to deal with, but equivalent
91 //! methods to the 16-bit strings are provided for compatibility.
92 //!
93 //! All the 32-bit wide strings provide efficient methods to convert to and from sequences of
94 //! [`char`] data, as the representation of UTF-32 strings is functionally equivalent to sequences
95 //! of [`char`]s. Keep in mind that only [`Utf32String`] guaruntees this equivalence, however, since
96 //! the other strings may contain invalid values.
97 //!
98 //! # FFI with C/C++ `wchar_t`
99 //!
100 //! C/C++'s `wchar_t` (and C++'s corresponding `widestring`) varies in size depending on compiler
101 //! and platform. Typically, `wchar_t` is 16-bits on Windows and 32-bits on most Unix-based
102 //! platforms. For convenience when using `wchar_t`-based FFI's, type aliases for the corresponding
103 //! string types are provided: [`WideString`] aliases [`U16String`] on Windows or [`U32String`]
104 //! elsewhere, [`WideCString`] aliases [`U16CString`] or [`U32CString`], and [`WideUtfString`]
105 //! aliases [`Utf16String`] or [`Utf32String`]. [`WideStr`], [`WideCStr`], and [`WideUtfStr`] are
106 //! provided for the string slice types. The [`WideChar`] alias is also provided, aliasing [`u16`]
107 //! or [`u32`] depending on platform.
108 //!
109 //! When not interacting with a FFI that uses `wchar_t`, it is recommended to use the string types
110 //! directly rather than via the wide alias.
111 //!
112 //! # Nul values
113 //!
114 //! This crate uses the term legacy ASCII term "nul" to refer to Unicode code point `U+0000 NULL`
115 //! and its associated code unit representation as zero-value bytes. This is to disambiguate this
116 //! zero value from null pointer values. C-style strings end in a nul value, while regular Rust
117 //! strings allow interior nul values and are not terminated with nul.
118 //!
119 //! # Examples
120 //!
121 //! The following example uses [`U16String`] to get Windows error messages, since `FormatMessageW`
122 //! returns a string length for us and we don't need to pass error messages into other FFI
123 //! functions so we don't need to worry about nul values.
124 //!
125 //! ```rust
126 //! # #[cfg(any(not(windows), not(feature = "alloc")))]
127 //! # fn main() {}
128 //! # extern crate winapi;
129 //! # extern crate widestring;
130 //! # #[cfg(all(windows, feature = "alloc"))]
131 //! # fn main() {
132 //! use winapi::um::winbase::{FormatMessageW, LocalFree, FORMAT_MESSAGE_FROM_SYSTEM,
133 //! FORMAT_MESSAGE_ALLOCATE_BUFFER, FORMAT_MESSAGE_IGNORE_INSERTS};
134 //! use winapi::shared::ntdef::LPWSTR;
135 //! use winapi::shared::minwindef::HLOCAL;
136 //! use std::ptr;
137 //! use widestring::U16String;
138 //! # use winapi::shared::minwindef::DWORD;
139 //! # let error_code: DWORD = 0;
140 //!
141 //! let s: U16String;
142 //! unsafe {
143 //! // First, get a string buffer from some windows api such as FormatMessageW...
144 //! let mut buffer: LPWSTR = ptr::null_mut();
145 //! let strlen = FormatMessageW(FORMAT_MESSAGE_FROM_SYSTEM |
146 //! FORMAT_MESSAGE_ALLOCATE_BUFFER |
147 //! FORMAT_MESSAGE_IGNORE_INSERTS,
148 //! ptr::null(),
149 //! error_code, // error code from GetLastError()
150 //! 0,
151 //! (&mut buffer as *mut LPWSTR) as LPWSTR,
152 //! 0,
153 //! ptr::null_mut());
154 //!
155 //! // Get the buffer as a wide string
156 //! s = U16String::from_ptr(buffer, strlen as usize);
157 //! // Since U16String creates an owned copy, it's safe to free original buffer now
158 //! // If you didn't want an owned copy, you could use &U16Str.
159 //! LocalFree(buffer as HLOCAL);
160 //! }
161 //! // Convert to a regular Rust String and use it to your heart's desire!
162 //! let message = s.to_string_lossy();
163 //! # assert_eq!(message, "The operation completed successfully.\r\n");
164 //! # }
165 //! ```
166 //!
167 //! The following example is the functionally the same, only using [`U16CString`] instead.
168 //!
169 //! ```rust
170 //! # #[cfg(any(not(windows), not(feature = "alloc")))]
171 //! # fn main() {}
172 //! # extern crate winapi;
173 //! # extern crate widestring;
174 //! # #[cfg(all(windows, feature = "alloc"))]
175 //! # fn main() {
176 //! use winapi::um::winbase::{FormatMessageW, LocalFree, FORMAT_MESSAGE_FROM_SYSTEM,
177 //! FORMAT_MESSAGE_ALLOCATE_BUFFER, FORMAT_MESSAGE_IGNORE_INSERTS};
178 //! use winapi::shared::ntdef::LPWSTR;
179 //! use winapi::shared::minwindef::HLOCAL;
180 //! use std::ptr;
181 //! use widestring::U16CString;
182 //! # use winapi::shared::minwindef::DWORD;
183 //! # let error_code: DWORD = 0;
184 //!
185 //! let s: U16CString;
186 //! unsafe {
187 //! // First, get a string buffer from some windows api such as FormatMessageW...
188 //! let mut buffer: LPWSTR = ptr::null_mut();
189 //! FormatMessageW(FORMAT_MESSAGE_FROM_SYSTEM |
190 //! FORMAT_MESSAGE_ALLOCATE_BUFFER |
191 //! FORMAT_MESSAGE_IGNORE_INSERTS,
192 //! ptr::null(),
193 //! error_code, // error code from GetLastError()
194 //! 0,
195 //! (&mut buffer as *mut LPWSTR) as LPWSTR,
196 //! 0,
197 //! ptr::null_mut());
198 //!
199 //! // Get the buffer as a wide string
200 //! s = U16CString::from_ptr_str(buffer);
201 //! // Since U16CString creates an owned copy, it's safe to free original buffer now
202 //! // If you didn't want an owned copy, you could use &U16CStr.
203 //! LocalFree(buffer as HLOCAL);
204 //! }
205 //! // Convert to a regular Rust String and use it to your heart's desire!
206 //! let message = s.to_string_lossy();
207 //! # assert_eq!(message, "The operation completed successfully.\r\n");
208 //! # }
209 //! ```
210 //!
211 //! [`OsString`]: std::ffi::OsString
212 //! [`OsStr`]: std::ffi::OsStr
213 //! [`CString`]: std::ffi::CString
214 //! [`CStr`]: std::ffi::CStr
215
216 #![warn(
217 missing_docs,
218 missing_debug_implementations,
219 trivial_casts,
220 trivial_numeric_casts,
221 future_incompatible
222 )]
223 #![allow(renamed_and_removed_lints, stable_features)] // Until min version gets bumped
224 #![cfg_attr(not(feature = "std"), no_std)]
225 #![doc(html_root_url = "https://docs.rs/widestring/1.1.0")]
226 #![doc(test(attr(deny(warnings), allow(unused))))]
227 #![cfg_attr(docsrs, feature(doc_cfg))]
228 #![cfg_attr(
229 feature = "debugger_visualizer",
230 feature(debugger_visualizer),
231 debugger_visualizer(natvis_file = "../debug_metadata/widestring.natvis")
232 )]
233
234 #[cfg(feature = "alloc")]
235 extern crate alloc;
236
237 use crate::error::{DecodeUtf16Error, DecodeUtf32Error};
238 #[cfg(feature = "alloc")]
239 #[allow(unused_imports)]
240 use alloc::vec::Vec;
241 use core::fmt::Write;
242
243 pub mod error;
244 pub mod iter;
245 mod macros;
246 #[cfg(feature = "std")]
247 #[cfg_attr(docsrs, doc(cfg(feature = "alloc")))]
248 mod platform;
249 pub mod ucstr;
250 #[cfg(feature = "alloc")]
251 #[cfg_attr(docsrs, doc(cfg(feature = "alloc")))]
252 pub mod ucstring;
253 pub mod ustr;
254 #[cfg(feature = "alloc")]
255 #[cfg_attr(docsrs, doc(cfg(feature = "alloc")))]
256 pub mod ustring;
257 pub mod utfstr;
258 #[cfg(feature = "alloc")]
259 #[cfg_attr(docsrs, doc(cfg(feature = "alloc")))]
260 pub mod utfstring;
261
262 #[doc(hidden)]
263 pub use macros::internals;
264 pub use ucstr::{U16CStr, U32CStr, WideCStr};
265 #[cfg(feature = "alloc")]
266 #[cfg_attr(docsrs, doc(cfg(feature = "alloc")))]
267 pub use ucstring::{U16CString, U32CString, WideCString};
268 pub use ustr::{U16Str, U32Str, WideStr};
269 #[cfg(feature = "alloc")]
270 #[cfg_attr(docsrs, doc(cfg(feature = "alloc")))]
271 pub use ustring::{U16String, U32String, WideString};
272 pub use utfstr::{Utf16Str, Utf32Str, WideUtfStr};
273 #[cfg(feature = "alloc")]
274 #[cfg_attr(docsrs, doc(cfg(feature = "alloc")))]
275 pub use utfstring::{Utf16String, Utf32String, WideUtfString};
276
277 #[cfg(not(windows))]
278 /// Alias for [`u16`] or [`u32`] depending on platform. Intended to match typical C `wchar_t` size
279 /// on platform.
280 pub type WideChar = u32;
281
282 #[cfg(windows)]
283 /// Alias for [`u16`] or [`u32`] depending on platform. Intended to match typical C `wchar_t` size
284 /// on platform.
285 pub type WideChar = u16;
286
287 /// Creates an iterator over the UTF-16 encoded code points in `iter`, returning unpaired surrogates
288 /// as `Err`s.
289 ///
290 /// # Examples
291 ///
292 /// Basic usage:
293 ///
294 /// ```
295 /// use std::char::decode_utf16;
296 ///
297 /// // 𝄞mus<invalid>ic<invalid>
298 /// let v = [
299 /// 0xD834, 0xDD1E, 0x006d, 0x0075, 0x0073, 0xDD1E, 0x0069, 0x0063, 0xD834,
300 /// ];
301 ///
302 /// assert_eq!(
303 /// decode_utf16(v.iter().cloned())
304 /// .map(|r| r.map_err(|e| e.unpaired_surrogate()))
305 /// .collect::<Vec<_>>(),
306 /// vec![
307 /// Ok('𝄞'),
308 /// Ok('m'), Ok('u'), Ok('s'),
309 /// Err(0xDD1E),
310 /// Ok('i'), Ok('c'),
311 /// Err(0xD834)
312 /// ]
313 /// );
314 /// ```
315 ///
316 /// A lossy decoder can be obtained by replacing Err results with the replacement character:
317 ///
318 /// ```
319 /// use std::char::{decode_utf16, REPLACEMENT_CHARACTER};
320 ///
321 /// // 𝄞mus<invalid>ic<invalid>
322 /// let v = [
323 /// 0xD834, 0xDD1E, 0x006d, 0x0075, 0x0073, 0xDD1E, 0x0069, 0x0063, 0xD834,
324 /// ];
325 ///
326 /// assert_eq!(
327 /// decode_utf16(v.iter().cloned())
328 /// .map(|r| r.unwrap_or(REPLACEMENT_CHARACTER))
329 /// .collect::<String>(),
330 /// "𝄞mus�ic�"
331 /// );
332 /// ```
333 #[must_use]
334 pub fn decode_utf16<I: IntoIterator<Item = u16>>(iter: I) -> iter::DecodeUtf16<I::IntoIter> {
335 iter::DecodeUtf16::new(iter.into_iter())
336 }
337
338 /// Creates a lossy decoder iterator over the possibly ill-formed UTF-16 encoded code points in
339 /// `iter`.
340 ///
341 /// This is equivalent to [`char::decode_utf16`][core::char::decode_utf16] except that any unpaired
342 /// UTF-16 surrogate values are replaced by
343 /// [`U+FFFD REPLACEMENT_CHARACTER`][core::char::REPLACEMENT_CHARACTER] (�) instead of returning
344 /// errors.
345 ///
346 /// # Examples
347 ///
348 /// ```
349 /// use widestring::decode_utf16_lossy;
350 ///
351 /// // 𝄞mus<invalid>ic<invalid>
352 /// let v = [
353 /// 0xD834, 0xDD1E, 0x006d, 0x0075, 0x0073, 0xDD1E, 0x0069, 0x0063, 0xD834,
354 /// ];
355 ///
356 /// assert_eq!(
357 /// decode_utf16_lossy(v.iter().copied()).collect::<String>(),
358 /// "𝄞mus�ic�"
359 /// );
360 /// ```
361 #[inline]
362 #[must_use]
363 pub fn decode_utf16_lossy<I: IntoIterator<Item = u16>>(
364 iter: I,
365 ) -> iter::DecodeUtf16Lossy<I::IntoIter> {
366 iter::DecodeUtf16Lossy {
367 iter: decode_utf16(iter),
368 }
369 }
370
371 /// Creates a decoder iterator over UTF-32 encoded code points in `iter`, returning invalid values
372 /// as `Err`s.
373 ///
374 /// # Examples
375 ///
376 /// ```
377 /// use widestring::decode_utf32;
378 ///
379 /// // 𝄞mus<invalid>ic<invalid>
380 /// let v = [
381 /// 0x1D11E, 0x6d, 0x75, 0x73, 0xDD1E, 0x69, 0x63, 0x23FD5A,
382 /// ];
383 ///
384 /// assert_eq!(
385 /// decode_utf32(v.iter().copied())
386 /// .map(|r| r.map_err(|e| e.invalid_code_point()))
387 /// .collect::<Vec<_>>(),
388 /// vec![
389 /// Ok('𝄞'),
390 /// Ok('m'), Ok('u'), Ok('s'),
391 /// Err(0xDD1E),
392 /// Ok('i'), Ok('c'),
393 /// Err(0x23FD5A)
394 /// ]
395 /// );
396 /// ```
397 #[inline]
398 #[must_use]
399 pub fn decode_utf32<I: IntoIterator<Item = u32>>(iter: I) -> iter::DecodeUtf32<I::IntoIter> {
400 iter::DecodeUtf32 {
401 iter: iter.into_iter(),
402 }
403 }
404
405 /// Creates a lossy decoder iterator over the possibly ill-formed UTF-32 encoded code points in
406 /// `iter`.
407 ///
408 /// This is equivalent to [`decode_utf32`] except that any invalid UTF-32 values are replaced by
409 /// [`U+FFFD REPLACEMENT_CHARACTER`][core::char::REPLACEMENT_CHARACTER] (�) instead of returning
410 /// errors.
411 ///
412 /// # Examples
413 ///
414 /// ```
415 /// use widestring::decode_utf32_lossy;
416 ///
417 /// // 𝄞mus<invalid>ic<invalid>
418 /// let v = [
419 /// 0x1D11E, 0x6d, 0x75, 0x73, 0xDD1E, 0x69, 0x63, 0x23FD5A,
420 /// ];
421 ///
422 /// assert_eq!(
423 /// decode_utf32_lossy(v.iter().copied()).collect::<String>(),
424 /// "𝄞mus�ic�"
425 /// );
426 /// ```
427 #[inline]
428 #[must_use]
429 pub fn decode_utf32_lossy<I: IntoIterator<Item = u32>>(
430 iter: I,
431 ) -> iter::DecodeUtf32Lossy<I::IntoIter> {
432 iter::DecodeUtf32Lossy {
433 iter: decode_utf32(iter),
434 }
435 }
436
437 /// Creates an iterator that encodes an iterator over [`char`]s into UTF-8 bytes.
438 ///
439 /// # Examples
440 ///
441 /// ```
442 /// use widestring::encode_utf8;
443 ///
444 /// let music = "𝄞music";
445 ///
446 /// let encoded: Vec<u8> = encode_utf8(music.chars()).collect();
447 ///
448 /// assert_eq!(encoded, music.as_bytes());
449 /// ```
450 #[must_use]
451 pub fn encode_utf8<I: IntoIterator<Item = char>>(iter: I) -> iter::EncodeUtf8<I::IntoIter> {
452 iter::EncodeUtf8::new(iter.into_iter())
453 }
454
455 /// Creates an iterator that encodes an iterator over [`char`]s into UTF-16 [`u16`] code units.
456 ///
457 /// # Examples
458 ///
459 /// ```
460 /// use widestring::encode_utf16;
461 ///
462 /// let encoded: Vec<u16> = encode_utf16("𝄞music".chars()).collect();
463 ///
464 /// let v = [
465 /// 0xD834, 0xDD1E, 0x006d, 0x0075, 0x0073, 0x0069, 0x0063,
466 /// ];
467 ///
468 /// assert_eq!(encoded, v);
469 /// ```
470 #[must_use]
471 pub fn encode_utf16<I: IntoIterator<Item = char>>(iter: I) -> iter::EncodeUtf16<I::IntoIter> {
472 iter::EncodeUtf16::new(iter.into_iter())
473 }
474
475 /// Creates an iterator that encodes an iterator over [`char`]s into UTF-32 [`u32`] values.
476 ///
477 /// This iterator is a simple type cast from [`char`] to [`u32`], as any sequence of [`char`]s is
478 /// valid UTF-32.
479 ///
480 /// # Examples
481 ///
482 /// ```
483 /// use widestring::encode_utf32;
484 ///
485 /// let encoded: Vec<u32> = encode_utf32("𝄞music".chars()).collect();
486 ///
487 /// let v = [
488 /// 0x1D11E, 0x006d, 0x0075, 0x0073, 0x0069, 0x0063,
489 /// ];
490 ///
491 /// assert_eq!(encoded, v);
492 /// ```
493 #[must_use]
494 pub fn encode_utf32<I: IntoIterator<Item = char>>(iter: I) -> iter::EncodeUtf32<I::IntoIter> {
495 iter::EncodeUtf32::new(iter.into_iter())
496 }
497
498 /// Debug implementation for any U16 string slice.
499 ///
500 /// Properly encoded input data will output valid strings with escape sequences, however invalid
501 /// encoding will purposefully output any unpaired surrogate as \<XXXX> which is not a valid escape
502 /// sequence. This is intentional, as debug output is not meant to be parsed but read by humans.
503 fn debug_fmt_u16(s: &[u16], fmt: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
504 debug_fmt_utf16_iter(decode_utf16(s.iter().copied()), fmt)
505 }
506
507 /// Debug implementation for any U16 string iterator.
508 ///
509 /// Properly encoded input data will output valid strings with escape sequences, however invalid
510 /// encoding will purposefully output any unpaired surrogate as \<XXXX> which is not a valid escape
511 /// sequence. This is intentional, as debug output is not meant to be parsed but read by humans.
512 fn debug_fmt_utf16_iter(
513 iter: impl Iterator<Item = Result<char, DecodeUtf16Error>>,
514 fmt: &mut core::fmt::Formatter<'_>,
515 ) -> core::fmt::Result {
516 fmt.write_char('"')?;
517 for res in iter {
518 match res {
519 Ok(ch) => {
520 for c in ch.escape_debug() {
521 fmt.write_char(c)?;
522 }
523 }
524 Err(e) => {
525 write!(fmt, "\\<{:X}>", e.unpaired_surrogate())?;
526 }
527 }
528 }
529 fmt.write_char('"')
530 }
531
532 /// Debug implementation for any U16 string slice.
533 ///
534 /// Properly encoded input data will output valid strings with escape sequences, however invalid
535 /// encoding will purposefully output any invalid code point as \<XXXX> which is not a valid escape
536 /// sequence. This is intentional, as debug output is not meant to be parsed but read by humans.
537 fn debug_fmt_u32(s: &[u32], fmt: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
538 debug_fmt_utf32_iter(decode_utf32(s.iter().copied()), fmt)
539 }
540
541 /// Debug implementation for any U16 string iterator.
542 ///
543 /// Properly encoded input data will output valid strings with escape sequences, however invalid
544 /// encoding will purposefully output any invalid code point as \<XXXX> which is not a valid escape
545 /// sequence. This is intentional, as debug output is not meant to be parsed but read by humans.
546 fn debug_fmt_utf32_iter(
547 iter: impl Iterator<Item = Result<char, DecodeUtf32Error>>,
548 fmt: &mut core::fmt::Formatter<'_>,
549 ) -> core::fmt::Result {
550 fmt.write_char('"')?;
551 for res in iter {
552 match res {
553 Ok(ch) => {
554 for c in ch.escape_debug() {
555 fmt.write_char(c)?;
556 }
557 }
558 Err(e) => {
559 write!(fmt, "\\<{:X}>", e.invalid_code_point())?;
560 }
561 }
562 }
563 fmt.write_char('"')
564 }
565
566 /// Debug implementation for any `char` iterator.
567 fn debug_fmt_char_iter(
568 iter: impl Iterator<Item = char>,
569 fmt: &mut core::fmt::Formatter<'_>,
570 ) -> core::fmt::Result {
571 fmt.write_char('"')?;
572 iter.flat_map(|c| c.escape_debug())
573 .try_for_each(|c| fmt.write_char(c))?;
574 fmt.write_char('"')
575 }
576
577 /// Returns whether the code unit a UTF-16 surrogate value.
578 #[inline(always)]
579 #[allow(dead_code)]
580 const fn is_utf16_surrogate(u: u16) -> bool {
581 u >= 0xD800 && u <= 0xDFFF
582 }
583
584 /// Returns whether the code unit a UTF-16 high surrogate value.
585 #[inline(always)]
586 #[allow(dead_code)]
587 const fn is_utf16_high_surrogate(u: u16) -> bool {
588 u >= 0xD800 && u <= 0xDBFF
589 }
590
591 /// Returns whether the code unit a UTF-16 low surrogate value.
592 #[inline(always)]
593 const fn is_utf16_low_surrogate(u: u16) -> bool {
594 u >= 0xDC00 && u <= 0xDFFF
595 }
596
597 /// Convert a UTF-16 surrogate pair to a `char`. Does not validate if the surrogates are valid.
598 #[inline(always)]
599 unsafe fn decode_utf16_surrogate_pair(high: u16, low: u16) -> char {
600 let c: u32 = (((high - 0xD800) as u32) << 10 | ((low) - 0xDC00) as u32) + 0x1_0000;
601 // SAFETY: we checked that it's a legal unicode value
602 core::char::from_u32_unchecked(c)
603 }
604
605 /// Validates whether a slice of 16-bit values is valid UTF-16, returning an error if it is not.
606 #[inline(always)]
607 fn validate_utf16(s: &[u16]) -> Result<(), crate::error::Utf16Error> {
608 for (index, result) in crate::decode_utf16(s.iter().copied()).enumerate() {
609 if let Err(e) = result {
610 return Err(crate::error::Utf16Error::empty(index, e));
611 }
612 }
613 Ok(())
614 }
615
616 /// Validates whether a vector of 16-bit values is valid UTF-16, returning an error if it is not.
617 #[inline(always)]
618 #[cfg(feature = "alloc")]
619 fn validate_utf16_vec(v: Vec<u16>) -> Result<Vec<u16>, crate::error::Utf16Error> {
620 for (index, result) in crate::decode_utf16(v.iter().copied()).enumerate() {
621 if let Err(e) = result {
622 return Err(crate::error::Utf16Error::new(v, index, e));
623 }
624 }
625 Ok(v)
626 }
627
628 /// Validates whether a slice of 32-bit values is valid UTF-32, returning an error if it is not.
629 #[inline(always)]
630 fn validate_utf32(s: &[u32]) -> Result<(), crate::error::Utf32Error> {
631 for (index, result) in crate::decode_utf32(s.iter().copied()).enumerate() {
632 if let Err(e) = result {
633 return Err(crate::error::Utf32Error::empty(index, e));
634 }
635 }
636 Ok(())
637 }
638
639 /// Validates whether a vector of 32-bit values is valid UTF-32, returning an error if it is not.
640 #[inline(always)]
641 #[cfg(feature = "alloc")]
642 fn validate_utf32_vec(v: Vec<u32>) -> Result<Vec<u32>, crate::error::Utf32Error> {
643 for (index, result) in crate::decode_utf32(v.iter().copied()).enumerate() {
644 if let Err(e) = result {
645 return Err(crate::error::Utf32Error::new(v, index, e));
646 }
647 }
648 Ok(v)
649 }
650
651 /// Copy of unstable core::slice::range to soundly handle ranges
652 /// TODO: Replace with core::slice::range when it is stabilized
653 #[track_caller]
654 #[allow(dead_code, clippy::redundant_closure)]
655 fn range<R>(range: R, bounds: core::ops::RangeTo<usize>) -> core::ops::Range<usize>
656 where
657 R: core::ops::RangeBounds<usize>,
658 {
659 #[inline(never)]
660 #[cold]
661 #[track_caller]
662 fn slice_end_index_len_fail(index: usize, len: usize) -> ! {
663 panic!(
664 "range end index {} out of range for slice of length {}",
665 index, len
666 );
667 }
668
669 #[inline(never)]
670 #[cold]
671 #[track_caller]
672 fn slice_index_order_fail(index: usize, end: usize) -> ! {
673 panic!("slice index starts at {} but ends at {}", index, end);
674 }
675
676 #[inline(never)]
677 #[cold]
678 #[track_caller]
679 fn slice_start_index_overflow_fail() -> ! {
680 panic!("attempted to index slice from after maximum usize");
681 }
682
683 #[inline(never)]
684 #[cold]
685 #[track_caller]
686 fn slice_end_index_overflow_fail() -> ! {
687 panic!("attempted to index slice up to maximum usize");
688 }
689
690 use core::ops::Bound::*;
691
692 let len = bounds.end;
693
694 let start = range.start_bound();
695 let start = match start {
696 Included(&start) => start,
697 Excluded(start) => start
698 .checked_add(1)
699 .unwrap_or_else(|| slice_start_index_overflow_fail()),
700 Unbounded => 0,
701 };
702
703 let end = range.end_bound();
704 let end = match end {
705 Included(end) => end
706 .checked_add(1)
707 .unwrap_or_else(|| slice_end_index_overflow_fail()),
708 Excluded(&end) => end,
709 Unbounded => len,
710 };
711
712 if start > end {
713 slice_index_order_fail(start, end);
714 }
715 if end > len {
716 slice_end_index_len_fail(end, len);
717 }
718
719 core::ops::Range { start, end }
720 }
721
722 /// Similar to core::slice::range, but returns [`None`] instead of panicking.
723 fn range_check<R>(range: R, bounds: core::ops::RangeTo<usize>) -> Option<core::ops::Range<usize>>
724 where
725 R: core::ops::RangeBounds<usize>,
726 {
727 use core::ops::Bound::*;
728
729 let len = bounds.end;
730
731 let start = range.start_bound();
732 let start = match start {
733 Included(&start) => start,
734 Excluded(start) => start.checked_add(1)?,
735 Unbounded => 0,
736 };
737
738 let end = range.end_bound();
739 let end = match end {
740 Included(end) => end.checked_add(1)?,
741 Excluded(&end) => end,
742 Unbounded => len,
743 };
744
745 if start > end || end > len {
746 return None;
747 }
748 Some(core::ops::Range { start, end })
749 }