Debian Codegraph @

Doc/c-api/unicode.rst

raw
1 .. highlight:: c
2
3 .. _unicodeobjects:
4
5 Unicode Objects and Codecs
6 --------------------------
7
8 .. sectionauthor:: Marc-André Lemburg <[email protected]>
9 .. sectionauthor:: Georg Brandl <[email protected]>
10
11 Unicode Objects
12 ^^^^^^^^^^^^^^^
13
14 Since the implementation of :pep:`393` in Python 3.3, Unicode objects internally
15 use a variety of representations, in order to allow handling the complete range
16 of Unicode characters while staying memory efficient. There are special cases
17 for strings where all code points are below 128, 256, or 65536; otherwise, code
18 points must be below 1114112 (which is the full Unicode range).
19
20 UTF-8 representation is created on demand and cached in the Unicode object.
21
22 .. note::
23 The :c:type:`Py_UNICODE` representation has been removed since Python 3.12
24 with deprecated APIs.
25 See :pep:`623` for more information.
26
27
28 Unicode Type
29 """"""""""""
30
31 These are the basic Unicode object types used for the Unicode implementation in
32 Python:
33
34 .. c:var:: PyTypeObject PyUnicode_Type
35
36 This instance of :c:type:`PyTypeObject` represents the Python Unicode type.
37 It is exposed to Python code as :py:class:`str`.
38
39
40 .. c:var:: PyTypeObject PyUnicodeIter_Type
41
42 This instance of :c:type:`PyTypeObject` represents the Python Unicode
43 iterator type. It is used to iterate over Unicode string objects.
44
45
46 .. c:type:: Py_UCS4
47 Py_UCS2
48 Py_UCS1
49
50 These types are typedefs for unsigned integer types wide enough to contain
51 characters of 32 bits, 16 bits and 8 bits, respectively. When dealing with
52 single Unicode characters, use :c:type:`Py_UCS4`.
53
54 .. versionadded:: 3.3
55
56
57 .. c:type:: PyASCIIObject
58 PyCompactUnicodeObject
59 PyUnicodeObject
60
61 These subtypes of :c:type:`PyObject` represent a Python Unicode object. In
62 almost all cases, they shouldn't be used directly, since all API functions
63 that deal with Unicode objects take and return :c:type:`PyObject` pointers.
64
65 .. versionadded:: 3.3
66
67
68 The structure of a particular object can be determined using the following
69 macros.
70 The macros cannot fail; their behavior is undefined if their argument
71 is not a Python Unicode object.
72
73 .. c:namespace:: NULL
74
75 .. c:macro:: PyUnicode_IS_COMPACT(o)
76
77 True if *o* uses the :c:struct:`PyCompactUnicodeObject` structure.
78
79 .. versionadded:: 3.3
80
81
82 .. c:macro:: PyUnicode_IS_COMPACT_ASCII(o)
83
84 True if *o* uses the :c:struct:`PyASCIIObject` structure.
85
86 .. versionadded:: 3.3
87
88
89 The following APIs are C macros and static inlined functions for fast checks and
90 access to internal read-only data of Unicode objects:
91
92 .. c:function:: int PyUnicode_Check(PyObject *obj)
93
94 Return true if the object *obj* is a Unicode object or an instance of a Unicode
95 subtype. This function always succeeds.
96
97
98 .. c:function:: int PyUnicode_CheckExact(PyObject *obj)
99
100 Return true if the object *obj* is a Unicode object, but not an instance of a
101 subtype. This function always succeeds.
102
103
104 .. c:function:: Py_ssize_t PyUnicode_GET_LENGTH(PyObject *unicode)
105
106 Return the length of the Unicode string, in code points. *unicode* has to be a
107 Unicode object in the "canonical" representation (not checked).
108
109 .. versionadded:: 3.3
110
111
112 .. c:function:: Py_UCS1* PyUnicode_1BYTE_DATA(PyObject *unicode)
113 Py_UCS2* PyUnicode_2BYTE_DATA(PyObject *unicode)
114 Py_UCS4* PyUnicode_4BYTE_DATA(PyObject *unicode)
115
116 Return a pointer to the canonical representation cast to UCS1, UCS2 or UCS4
117 integer types for direct character access. No checks are performed if the
118 canonical representation has the correct character size; use
119 :c:func:`PyUnicode_KIND` to select the right function.
120
121 .. versionadded:: 3.3
122
123
124 .. c:macro:: PyUnicode_1BYTE_KIND
125 PyUnicode_2BYTE_KIND
126 PyUnicode_4BYTE_KIND
127
128 Return values of the :c:func:`PyUnicode_KIND` macro.
129
130 .. versionadded:: 3.3
131
132 .. versionchanged:: 3.12
133 ``PyUnicode_WCHAR_KIND`` has been removed.
134
135
136 .. c:function:: int PyUnicode_KIND(PyObject *unicode)
137
138 Return one of the PyUnicode kind constants (see above) that indicate how many
139 bytes per character this Unicode object uses to store its data. *unicode* has to
140 be a Unicode object in the "canonical" representation (not checked).
141
142 .. versionadded:: 3.3
143
144
145 .. c:function:: void* PyUnicode_DATA(PyObject *unicode)
146
147 Return a void pointer to the raw Unicode buffer. *unicode* has to be a Unicode
148 object in the "canonical" representation (not checked).
149
150 .. versionadded:: 3.3
151
152
153 .. c:function:: void PyUnicode_WRITE(int kind, void *data, \
154 Py_ssize_t index, Py_UCS4 value)
155
156 Write the code point *value* to the given zero-based *index* in a string.
157
158 The *kind* value and *data* pointer must have been obtained from a
159 string using :c:func:`PyUnicode_KIND` and :c:func:`PyUnicode_DATA`
160 respectively. You must hold a reference to that string while calling
161 :c:func:`!PyUnicode_WRITE`. All requirements of
162 :c:func:`PyUnicode_WriteChar` also apply.
163
164 The function performs no checks for any of its requirements,
165 and is intended for usage in loops.
166
167 .. versionadded:: 3.3
168
169
170 .. c:function:: Py_UCS4 PyUnicode_READ(int kind, void *data, \
171 Py_ssize_t index)
172
173 Read a code point from a canonical representation *data* (as obtained with
174 :c:func:`PyUnicode_DATA`). No checks or ready calls are performed.
175
176 .. versionadded:: 3.3
177
178
179 .. c:function:: Py_UCS4 PyUnicode_READ_CHAR(PyObject *unicode, Py_ssize_t index)
180
181 Read a character from a Unicode object *unicode*, which must be in the "canonical"
182 representation. This is less efficient than :c:func:`PyUnicode_READ` if you
183 do multiple consecutive reads.
184
185 .. versionadded:: 3.3
186
187
188 .. c:function:: Py_UCS4 PyUnicode_MAX_CHAR_VALUE(PyObject *unicode)
189
190 Return the maximum code point that is suitable for creating another string
191 based on *unicode*, which must be in the "canonical" representation. This is
192 always an approximation but more efficient than iterating over the string.
193
194 .. versionadded:: 3.3
195
196
197 .. c:function:: int PyUnicode_IsIdentifier(PyObject *unicode)
198
199 Return ``1`` if the string is a valid identifier according to the language
200 definition, section :ref:`identifiers`. Return ``0`` otherwise.
201
202 .. versionchanged:: 3.9
203 The function does not call :c:func:`Py_FatalError` anymore if the string
204 is not ready.
205
206
207 .. c:function:: unsigned int PyUnicode_IS_ASCII(PyObject *unicode)
208
209 Return true if the string only contains ASCII characters.
210 Equivalent to :py:meth:`str.isascii`.
211
212 .. versionadded:: 3.2
213
214
215 Unicode Character Properties
216 """"""""""""""""""""""""""""
217
218 Unicode provides many different character properties. The most often needed ones
219 are available through these macros which are mapped to C functions depending on
220 the Python configuration.
221
222
223 .. c:function:: int Py_UNICODE_ISSPACE(Py_UCS4 ch)
224
225 Return ``1`` or ``0`` depending on whether *ch* is a whitespace character.
226
227
228 .. c:function:: int Py_UNICODE_ISLOWER(Py_UCS4 ch)
229
230 Return ``1`` or ``0`` depending on whether *ch* is a lowercase character.
231
232
233 .. c:function:: int Py_UNICODE_ISUPPER(Py_UCS4 ch)
234
235 Return ``1`` or ``0`` depending on whether *ch* is an uppercase character.
236
237
238 .. c:function:: int Py_UNICODE_ISTITLE(Py_UCS4 ch)
239
240 Return ``1`` or ``0`` depending on whether *ch* is a titlecase character.
241
242
243 .. c:function:: int Py_UNICODE_ISLINEBREAK(Py_UCS4 ch)
244
245 Return ``1`` or ``0`` depending on whether *ch* is a linebreak character.
246
247
248 .. c:function:: int Py_UNICODE_ISDECIMAL(Py_UCS4 ch)
249
250 Return ``1`` or ``0`` depending on whether *ch* is a decimal character.
251
252
253 .. c:function:: int Py_UNICODE_ISDIGIT(Py_UCS4 ch)
254
255 Return ``1`` or ``0`` depending on whether *ch* is a digit character.
256
257
258 .. c:function:: int Py_UNICODE_ISNUMERIC(Py_UCS4 ch)
259
260 Return ``1`` or ``0`` depending on whether *ch* is a numeric character.
261
262
263 .. c:function:: int Py_UNICODE_ISALPHA(Py_UCS4 ch)
264
265 Return ``1`` or ``0`` depending on whether *ch* is an alphabetic character.
266
267
268 .. c:function:: int Py_UNICODE_ISALNUM(Py_UCS4 ch)
269
270 Return ``1`` or ``0`` depending on whether *ch* is an alphanumeric character.
271
272
273 .. c:function:: int Py_UNICODE_ISPRINTABLE(Py_UCS4 ch)
274
275 Return ``1`` or ``0`` depending on whether *ch* is a printable character,
276 in the sense of :meth:`str.isprintable`.
277
278
279 These APIs can be used for fast direct character conversions:
280
281
282 .. c:function:: Py_UCS4 Py_UNICODE_TOLOWER(Py_UCS4 ch)
283
284 Return the character *ch* converted to lower case.
285
286
287 .. c:function:: Py_UCS4 Py_UNICODE_TOUPPER(Py_UCS4 ch)
288
289 Return the character *ch* converted to upper case.
290
291
292 .. c:function:: Py_UCS4 Py_UNICODE_TOTITLE(Py_UCS4 ch)
293
294 Return the character *ch* converted to title case.
295
296
297 .. c:function:: int Py_UNICODE_TODECIMAL(Py_UCS4 ch)
298
299 Return the character *ch* converted to a decimal positive integer. Return
300 ``-1`` if this is not possible. This function does not raise exceptions.
301
302
303 .. c:function:: int Py_UNICODE_TODIGIT(Py_UCS4 ch)
304
305 Return the character *ch* converted to a single digit integer. Return ``-1`` if
306 this is not possible. This function does not raise exceptions.
307
308
309 .. c:function:: double Py_UNICODE_TONUMERIC(Py_UCS4 ch)
310
311 Return the character *ch* converted to a double. Return ``-1.0`` if this is not
312 possible. This function does not raise exceptions.
313
314
315 These APIs can be used to work with surrogates:
316
317 .. c:function:: int Py_UNICODE_IS_SURROGATE(Py_UCS4 ch)
318
319 Check if *ch* is a surrogate (``0xD800 <= ch <= 0xDFFF``).
320
321 .. c:function:: int Py_UNICODE_IS_HIGH_SURROGATE(Py_UCS4 ch)
322
323 Check if *ch* is a high surrogate (``0xD800 <= ch <= 0xDBFF``).
324
325 .. c:function:: int Py_UNICODE_IS_LOW_SURROGATE(Py_UCS4 ch)
326
327 Check if *ch* is a low surrogate (``0xDC00 <= ch <= 0xDFFF``).
328
329 .. c:function:: Py_UCS4 Py_UNICODE_HIGH_SURROGATE(Py_UCS4 ch)
330
331 Return the high UTF-16 surrogate (``0xD800`` to ``0xDBFF``) for a Unicode
332 code point in the range ``[0x10000; 0x10FFFF]``.
333
334 .. c:function:: Py_UCS4 Py_UNICODE_LOW_SURROGATE(Py_UCS4 ch)
335
336 Return the low UTF-16 surrogate (``0xDC00`` to ``0xDFFF``) for a Unicode
337 code point in the range ``[0x10000; 0x10FFFF]``.
338
339 .. c:function:: Py_UCS4 Py_UNICODE_JOIN_SURROGATES(Py_UCS4 high, Py_UCS4 low)
340
341 Join two surrogate code points and return a single :c:type:`Py_UCS4` value.
342 *high* and *low* are respectively the leading and trailing surrogates in a
343 surrogate pair. *high* must be in the range ``[0xD800; 0xDBFF]`` and *low* must
344 be in the range ``[0xDC00; 0xDFFF]``.
345
346
347 Creating and accessing Unicode strings
348 """"""""""""""""""""""""""""""""""""""
349
350 To create Unicode objects and access their basic sequence properties, use these
351 APIs:
352
353 .. c:function:: PyObject* PyUnicode_New(Py_ssize_t size, Py_UCS4 maxchar)
354
355 Create a new Unicode object. *maxchar* should be the true maximum code point
356 to be placed in the string. As an approximation, it can be rounded up to the
357 nearest value in the sequence 127, 255, 65535, 1114111.
358
359 On error, set an exception and return ``NULL``.
360
361 After creation, the string can be filled by :c:func:`PyUnicode_WriteChar`,
362 :c:func:`PyUnicode_CopyCharacters`, :c:func:`PyUnicode_Fill`,
363 :c:func:`PyUnicode_WRITE` or similar.
364 Since strings are supposed to be immutable, take care to not “use” the
365 result while it is being modified. In particular, before it's filled
366 with its final contents, a string:
367
368 - must not be hashed,
369 - must not be :c:func:`converted to UTF-8 <PyUnicode_AsUTF8AndSize>`,
370 or another non-"canonical" representation,
371 - must not have its reference count changed,
372 - must not be shared with code that might do one of the above.
373
374 This list is not exhaustive. Avoiding these uses is your responsibility;
375 Python does not always check these requirements.
376
377 To avoid accidentally exposing a partially-written string object, prefer
378 using the :c:type:`PyUnicodeWriter` API, or one of the ``PyUnicode_From*``
379 functions below.
380
381
382 .. versionadded:: 3.3
383
384
385 .. c:function:: PyObject* PyUnicode_FromKindAndData(int kind, const void *buffer, \
386 Py_ssize_t size)
387
388 Create a new Unicode object with the given *kind* (possible values are
389 :c:macro:`PyUnicode_1BYTE_KIND` etc., as returned by
390 :c:func:`PyUnicode_KIND`). The *buffer* must point to an array of *size*
391 units of 1, 2 or 4 bytes per character, as given by the kind.
392
393 If necessary, the input *buffer* is copied and transformed into the
394 canonical representation. For example, if the *buffer* is a UCS4 string
395 (:c:macro:`PyUnicode_4BYTE_KIND`) and it consists only of codepoints in
396 the UCS1 range, it will be transformed into UCS1
397 (:c:macro:`PyUnicode_1BYTE_KIND`).
398
399 .. versionadded:: 3.3
400
401
402 .. c:function:: PyObject* PyUnicode_FromStringAndSize(const char *str, Py_ssize_t size)
403
404 Create a Unicode object from the char buffer *str*. The bytes will be
405 interpreted as being UTF-8 encoded. The buffer is copied into the new
406 object.
407 The return value might be a shared object, i.e. modification of the data is
408 not allowed.
409
410 This function raises :exc:`SystemError` when:
411
412 * *size* < 0,
413 * *str* is ``NULL`` and *size* > 0
414
415 .. versionchanged:: 3.12
416 *str* == ``NULL`` with *size* > 0 is not allowed anymore.
417
418
419 .. c:function:: PyObject *PyUnicode_FromString(const char *str)
420
421 Create a Unicode object from a UTF-8 encoded null-terminated char buffer
422 *str*.
423
424
425 .. c:function:: PyObject* PyUnicode_FromFormat(const char *format, ...)
426
427 Take a C :c:func:`printf`\ -style *format* string and a variable number of
428 arguments, calculate the size of the resulting Python Unicode string and return
429 a string with the values formatted into it. The variable arguments must be C
430 types and must correspond exactly to the format characters in the *format*
431 ASCII-encoded string.
432
433 A conversion specifier contains two or more characters and has the following
434 components, which must occur in this order:
435
436 #. The ``'%'`` character, which marks the start of the specifier.
437
438 #. Conversion flags (optional), which affect the result of some conversion
439 types.
440
441 #. Minimum field width (optional).
442 If specified as an ``'*'`` (asterisk), the actual width is given in the
443 next argument, which must be of type :c:expr:`int`, and the object to
444 convert comes after the minimum field width and optional precision.
445
446 #. Precision (optional), given as a ``'.'`` (dot) followed by the precision.
447 If specified as ``'*'`` (an asterisk), the actual precision is given in
448 the next argument, which must be of type :c:expr:`int`, and the value to
449 convert comes after the precision.
450
451 #. Length modifier (optional).
452
453 #. Conversion type.
454
455 The conversion flag characters are:
456
457 .. tabularcolumns:: |l|L|
458
459 +-------+-------------------------------------------------------------+
460 | Flag | Meaning |
461 +=======+=============================================================+
462 | ``0`` | The conversion will be zero padded for numeric values. |
463 +-------+-------------------------------------------------------------+
464 | ``-`` | The converted value is left adjusted (overrides the ``0`` |
465 | | flag if both are given). |
466 +-------+-------------------------------------------------------------+
467
468 The length modifiers for following integer conversions (``d``, ``i``,
469 ``o``, ``u``, ``x``, or ``X``) specify the type of the argument
470 (:c:expr:`int` by default):
471
472 .. tabularcolumns:: |l|L|
473
474 +----------+-----------------------------------------------------+
475 | Modifier | Types |
476 +==========+=====================================================+
477 | ``l`` | :c:expr:`long` or :c:expr:`unsigned long` |
478 +----------+-----------------------------------------------------+
479 | ``ll`` | :c:expr:`long long` or :c:expr:`unsigned long long` |
480 +----------+-----------------------------------------------------+
481 | ``j`` | :c:type:`intmax_t` or :c:type:`uintmax_t` |
482 +----------+-----------------------------------------------------+
483 | ``z`` | :c:type:`size_t` or :c:type:`ssize_t` |
484 +----------+-----------------------------------------------------+
485 | ``t`` | :c:type:`ptrdiff_t` |
486 +----------+-----------------------------------------------------+
487
488 The length modifier ``l`` for following conversions ``s`` or ``V`` specify
489 that the type of the argument is :c:expr:`const wchar_t*`.
490
491 The conversion specifiers are:
492
493 .. list-table::
494 :widths: auto
495 :header-rows: 1
496
497 * - Conversion Specifier
498 - Type
499 - Comment
500
501 * - ``%``
502 - *n/a*
503 - The literal ``%`` character.
504
505 * - ``d``, ``i``
506 - Specified by the length modifier
507 - The decimal representation of a signed C integer.
508
509 * - ``u``
510 - Specified by the length modifier
511 - The decimal representation of an unsigned C integer.
512
513 * - ``o``
514 - Specified by the length modifier
515 - The octal representation of an unsigned C integer.
516
517 * - ``x``
518 - Specified by the length modifier
519 - The hexadecimal representation of an unsigned C integer (lowercase).
520
521 * - ``X``
522 - Specified by the length modifier
523 - The hexadecimal representation of an unsigned C integer (uppercase).
524
525 * - ``c``
526 - :c:expr:`int`
527 - A single character.
528
529 * - ``s``
530 - :c:expr:`const char*` or :c:expr:`const wchar_t*`
531 - A null-terminated C character array.
532
533 * - ``p``
534 - :c:expr:`const void*`
535 - The hex representation of a C pointer.
536 Mostly equivalent to ``printf("%p")`` except that it is guaranteed to
537 start with the literal ``0x`` regardless of what the platform's
538 ``printf`` yields.
539
540 * - ``A``
541 - :c:expr:`PyObject*`
542 - The result of calling :func:`ascii`.
543
544 * - ``U``
545 - :c:expr:`PyObject*`
546 - A Unicode object.
547
548 * - ``V``
549 - :c:expr:`PyObject*`, :c:expr:`const char*` or :c:expr:`const wchar_t*`
550 - A Unicode object (which may be ``NULL``) and a null-terminated
551 C character array as a second parameter (which will be used,
552 if the first parameter is ``NULL``).
553
554 * - ``S``
555 - :c:expr:`PyObject*`
556 - The result of calling :c:func:`PyObject_Str`.
557
558 * - ``R``
559 - :c:expr:`PyObject*`
560 - The result of calling :c:func:`PyObject_Repr`.
561
562 * - ``T``
563 - :c:expr:`PyObject*`
564 - Get the fully qualified name of an object type;
565 call :c:func:`PyType_GetFullyQualifiedName`.
566
567 * - ``#T``
568 - :c:expr:`PyObject*`
569 - Similar to ``T`` format, but use a colon (``:``) as separator between
570 the module name and the qualified name.
571
572 * - ``N``
573 - :c:expr:`PyTypeObject*`
574 - Get the fully qualified name of a type;
575 call :c:func:`PyType_GetFullyQualifiedName`.
576
577 * - ``#N``
578 - :c:expr:`PyTypeObject*`
579 - Similar to ``N`` format, but use a colon (``:``) as separator between
580 the module name and the qualified name.
581
582 .. note::
583 The width formatter unit is number of characters rather than bytes.
584 The precision formatter unit is number of bytes or :c:type:`wchar_t`
585 items (if the length modifier ``l`` is used) for ``"%s"`` and
586 ``"%V"`` (if the ``PyObject*`` argument is ``NULL``), and a number of
587 characters for ``"%A"``, ``"%U"``, ``"%S"``, ``"%R"`` and ``"%V"``
588 (if the ``PyObject*`` argument is not ``NULL``).
589
590 .. note::
591 Unlike to C :c:func:`printf` the ``0`` flag has effect even when
592 a precision is given for integer conversions (``d``, ``i``, ``u``, ``o``,
593 ``x``, or ``X``).
594
595 .. versionchanged:: 3.2
596 Support for ``"%lld"`` and ``"%llu"`` added.
597
598 .. versionchanged:: 3.3
599 Support for ``"%li"``, ``"%lli"`` and ``"%zi"`` added.
600
601 .. versionchanged:: 3.4
602 Support width and precision formatter for ``"%s"``, ``"%A"``, ``"%U"``,
603 ``"%V"``, ``"%S"``, ``"%R"`` added.
604
605 .. versionchanged:: 3.12
606 Support for conversion specifiers ``o`` and ``X``.
607 Support for length modifiers ``j`` and ``t``.
608 Length modifiers are now applied to all integer conversions.
609 Length modifier ``l`` is now applied to conversion specifiers ``s`` and ``V``.
610 Support for variable width and precision ``*``.
611 Support for flag ``-``.
612
613 An unrecognized format character now sets a :exc:`SystemError`.
614 In previous versions it caused all the rest of the format string to be
615 copied as-is to the result string, and any extra arguments discarded.
616
617 .. versionchanged:: 3.13
618 Support for ``%T``, ``%#T``, ``%N`` and ``%#N`` formats added.
619
620
621 .. c:function:: PyObject* PyUnicode_FromFormatV(const char *format, va_list vargs)
622
623 Identical to :c:func:`PyUnicode_FromFormat` except that it takes exactly two
624 arguments.
625
626
627 .. c:function:: PyObject* PyUnicode_FromObject(PyObject *obj)
628
629 Copy an instance of a Unicode subtype to a new true Unicode object if
630 necessary. If *obj* is already a true Unicode object (not a subtype),
631 return a new :term:`strong reference` to the object.
632
633 Objects other than Unicode or its subtypes will cause a :exc:`TypeError`.
634
635
636 .. c:function:: PyObject* PyUnicode_FromOrdinal(int ordinal)
637
638 Create a Unicode Object from the given Unicode code point *ordinal*.
639
640 The ordinal must be in ``range(0x110000)``. A :exc:`ValueError` is
641 raised in the case it is not.
642
643
644 .. c:function:: PyObject* PyUnicode_FromEncodedObject(PyObject *obj, \
645 const char *encoding, const char *errors)
646
647 Decode an encoded object *obj* to a Unicode object.
648
649 :class:`bytes`, :class:`bytearray` and other
650 :term:`bytes-like objects <bytes-like object>`
651 are decoded according to the given *encoding* and using the error handling
652 defined by *errors*. Both can be ``NULL`` to have the interface use the default
653 values (see :ref:`builtincodecs` for details).
654
655 All other objects, including Unicode objects, cause a :exc:`TypeError` to be
656 set.
657
658 The API returns ``NULL`` if there was an error. The caller is responsible for
659 decref'ing the returned objects.
660
661
662 .. c:function:: void PyUnicode_Append(PyObject **p_left, PyObject *right)
663
664 Append the string *right* to the end of *p_left*.
665 *p_left* must point to a :term:`strong reference` to a Unicode object;
666 :c:func:`!PyUnicode_Append` releases ("steals") this reference.
667
668 On error, set *\*p_left* to ``NULL`` and set an exception.
669
670 On success, set *\*p_left* to a new strong reference to the result.
671
672
673 .. c:function:: void PyUnicode_AppendAndDel(PyObject **p_left, PyObject *right)
674
675 The function is similar to :c:func:`PyUnicode_Append`, with the only
676 difference being that it decrements the reference count of *right* by one.
677
678
679 .. c:function:: PyObject* PyUnicode_BuildEncodingMap(PyObject* string)
680
681 Return a mapping suitable for decoding a custom single-byte encoding.
682 Given a Unicode string *string* of up to 256 characters representing an encoding
683 table, returns either a compact internal mapping object or a dictionary
684 mapping character ordinals to byte values. Raises a :exc:`TypeError` and
685 return ``NULL`` on invalid input.
686
687 .. versionadded:: 3.2
688
689
690 .. c:function:: const char* PyUnicode_GetDefaultEncoding(void)
691
692 Return the name of the default string encoding, ``"utf-8"``.
693 See :func:`sys.getdefaultencoding`.
694
695 The returned string does not need to be freed, and is valid
696 until interpreter shutdown.
697
698
699 .. c:function:: Py_ssize_t PyUnicode_GetLength(PyObject *unicode)
700
701 Return the length of the Unicode object, in code points.
702
703 On error, set an exception and return ``-1``.
704
705 .. versionadded:: 3.3
706
707
708 .. c:function:: Py_ssize_t PyUnicode_CopyCharacters(PyObject *to, \
709 Py_ssize_t to_start, \
710 PyObject *from, \
711 Py_ssize_t from_start, \
712 Py_ssize_t how_many)
713
714 Copy characters from one Unicode object into another. This function performs
715 character conversion when necessary and falls back to :c:func:`!memcpy` if
716 possible. Returns ``-1`` and sets an exception on error, otherwise returns
717 the number of copied characters.
718
719 The string must not have been “used” yet.
720 See :c:func:`PyUnicode_New` for details.
721
722 .. versionadded:: 3.3
723
724
725 .. c:function:: int PyUnicode_Resize(PyObject **unicode, Py_ssize_t length);
726
727 Resize a Unicode object *\*unicode* to the new *length* in code points.
728
729 Try to resize the string in place (which is usually faster than allocating
730 a new string and copying characters), or create a new string.
731
732 *\*unicode* is modified to point to the new (resized) object and ``0`` is
733 returned on success. Otherwise, ``-1`` is returned and an exception is set,
734 and *\*unicode* is left untouched.
735
736 The function doesn't check string content, the result may not be a
737 string in canonical representation.
738
739
740 .. c:function:: Py_ssize_t PyUnicode_Fill(PyObject *unicode, Py_ssize_t start, \
741 Py_ssize_t length, Py_UCS4 fill_char)
742
743 Fill a string with a character: write *fill_char* into
744 ``unicode[start:start+length]``.
745
746 Fail if *fill_char* is bigger than the string maximum character, or if the
747 string has more than 1 reference.
748
749 The string must not have been “used” yet.
750 See :c:func:`PyUnicode_New` for details.
751
752 Return the number of written characters, or return ``-1`` and raise an
753 exception on error.
754
755 .. versionadded:: 3.3
756
757
758 .. c:function:: int PyUnicode_WriteChar(PyObject *unicode, Py_ssize_t index, \
759 Py_UCS4 character)
760
761 Write a *character* to the string *unicode* at the zero-based *index*.
762 Return ``0`` on success, ``-1`` on error with an exception set.
763
764 This function checks that *unicode* is a Unicode object, that the index is
765 not out of bounds, and that the object's reference count is one.
766 See :c:func:`PyUnicode_WRITE` for a version that skips these checks,
767 making them your responsibility.
768
769 The string must not have been “used” yet.
770 See :c:func:`PyUnicode_New` for details.
771
772 .. versionadded:: 3.3
773
774
775 .. c:function:: Py_UCS4 PyUnicode_ReadChar(PyObject *unicode, Py_ssize_t index)
776
777 Read a character from a string. This function checks that *unicode* is a
778 Unicode object and the index is not out of bounds, in contrast to
779 :c:func:`PyUnicode_READ_CHAR`, which performs no error checking.
780
781 Return character on success, ``-1`` on error with an exception set.
782
783 .. versionadded:: 3.3
784
785
786 .. c:function:: PyObject* PyUnicode_Substring(PyObject *unicode, Py_ssize_t start, \
787 Py_ssize_t end)
788
789 Return a substring of *unicode*, from character index *start* (included) to
790 character index *end* (excluded). Negative indices are not supported.
791 On error, set an exception and return ``NULL``.
792
793 .. versionadded:: 3.3
794
795
796 .. c:function:: Py_UCS4* PyUnicode_AsUCS4(PyObject *unicode, Py_UCS4 *buffer, \
797 Py_ssize_t buflen, int copy_null)
798
799 Copy the string *unicode* into a UCS4 buffer, including a null character, if
800 *copy_null* is set. Returns ``NULL`` and sets an exception on error (in
801 particular, a :exc:`SystemError` if *buflen* is smaller than the length of
802 *unicode*). *buffer* is returned on success.
803
804 .. versionadded:: 3.3
805
806
807 .. c:function:: Py_UCS4* PyUnicode_AsUCS4Copy(PyObject *unicode)
808
809 Copy the string *unicode* into a new UCS4 buffer that is allocated using
810 :c:func:`PyMem_Malloc`. If this fails, ``NULL`` is returned with a
811 :exc:`MemoryError` set. The returned buffer always has an extra
812 null code point appended.
813
814 .. versionadded:: 3.3
815
816
817 Locale Encoding
818 """""""""""""""
819
820 The current locale encoding can be used to decode text from the operating
821 system.
822
823 .. c:function:: PyObject* PyUnicode_DecodeLocaleAndSize(const char *str, \
824 Py_ssize_t length, \
825 const char *errors)
826
827 Decode a string from UTF-8 on Android and VxWorks, or from the current
828 locale encoding on other platforms. The supported
829 error handlers are ``"strict"`` and ``"surrogateescape"``
830 (:pep:`383`). The decoder uses ``"strict"`` error handler if
831 *errors* is ``NULL``. *str* must end with a null character but
832 cannot contain embedded null characters.
833
834 Use :c:func:`PyUnicode_DecodeFSDefaultAndSize` to decode a string from
835 the :term:`filesystem encoding and error handler`.
836
837 This function ignores the :ref:`Python UTF-8 Mode <utf8-mode>`.
838
839 .. seealso::
840
841 The :c:func:`Py_DecodeLocale` function.
842
843 .. versionadded:: 3.3
844
845 .. versionchanged:: 3.7
846 The function now also uses the current locale encoding for the
847 ``surrogateescape`` error handler, except on Android. Previously, :c:func:`Py_DecodeLocale`
848 was used for the ``surrogateescape``, and the current locale encoding was
849 used for ``strict``.
850
851
852 .. c:function:: PyObject* PyUnicode_DecodeLocale(const char *str, const char *errors)
853
854 Similar to :c:func:`PyUnicode_DecodeLocaleAndSize`, but compute the string
855 length using :c:func:`!strlen`.
856
857 .. versionadded:: 3.3
858
859
860 .. c:function:: PyObject* PyUnicode_EncodeLocale(PyObject *unicode, const char *errors)
861
862 Encode a Unicode object to UTF-8 on Android and VxWorks, or to the current
863 locale encoding on other platforms. The
864 supported error handlers are ``"strict"`` and ``"surrogateescape"``
865 (:pep:`383`). The encoder uses ``"strict"`` error handler if
866 *errors* is ``NULL``. Return a :class:`bytes` object. *unicode* cannot
867 contain embedded null characters.
868
869 Use :c:func:`PyUnicode_EncodeFSDefault` to encode a string to the
870 :term:`filesystem encoding and error handler`.
871
872 This function ignores the :ref:`Python UTF-8 Mode <utf8-mode>`.
873
874 .. seealso::
875
876 The :c:func:`Py_EncodeLocale` function.
877
878 .. versionadded:: 3.3
879
880 .. versionchanged:: 3.7
881 The function now also uses the current locale encoding for the
882 ``surrogateescape`` error handler, except on Android. Previously,
883 :c:func:`Py_EncodeLocale`
884 was used for the ``surrogateescape``, and the current locale encoding was
885 used for ``strict``.
886
887
888 File System Encoding
889 """"""""""""""""""""
890
891 Functions encoding to and decoding from the :term:`filesystem encoding and
892 error handler` (:pep:`383` and :pep:`529`).
893
894 To encode file names to :class:`bytes` during argument parsing, the ``"O&"``
895 converter should be used, passing :c:func:`!PyUnicode_FSConverter` as the
896 conversion function:
897
898 .. c:function:: int PyUnicode_FSConverter(PyObject* obj, void* result)
899
900 :ref:`PyArg_Parse\* converter <arg-parsing>`: encode :class:`str` objects -- obtained directly or
901 through the :class:`os.PathLike` interface -- to :class:`bytes` using
902 :c:func:`PyUnicode_EncodeFSDefault`; :class:`bytes` objects are output as-is.
903 *result* must be an address of a C variable of type :c:expr:`PyObject*`
904 (or :c:expr:`PyBytesObject*`).
905 On success, set the variable to a new :term:`strong reference` to
906 a :ref:`bytes object <bytesobjects>` which must be released
907 when it is no longer used and return a non-zero value
908 (:c:macro:`Py_CLEANUP_SUPPORTED`).
909 Embedded null bytes are not allowed in the result.
910 On failure, return ``0`` with an exception set.
911
912 If *obj* is ``NULL``, the function releases a strong reference
913 stored in the variable referred by *result* and returns ``1``.
914
915 .. versionadded:: 3.1
916
917 .. versionchanged:: 3.6
918 Accepts a :term:`path-like object`.
919
920 To decode file names to :class:`str` during argument parsing, the ``"O&"``
921 converter should be used, passing :c:func:`!PyUnicode_FSDecoder` as the
922 conversion function:
923
924 .. c:function:: int PyUnicode_FSDecoder(PyObject* obj, void* result)
925
926 :ref:`PyArg_Parse\* converter <arg-parsing>`: decode :class:`bytes` objects -- obtained either
927 directly or indirectly through the :class:`os.PathLike` interface -- to
928 :class:`str` using :c:func:`PyUnicode_DecodeFSDefaultAndSize`; :class:`str`
929 objects are output as-is.
930 *result* must be an address of a C variable of type :c:expr:`PyObject*`
931 (or :c:expr:`PyUnicodeObject*`).
932 On success, set the variable to a new :term:`strong reference` to
933 a :ref:`Unicode object <unicodeobjects>` which must be released
934 when it is no longer used and return a non-zero value
935 (:c:macro:`Py_CLEANUP_SUPPORTED`).
936 Embedded null characters are not allowed in the result.
937 On failure, return ``0`` with an exception set.
938
939 If *obj* is ``NULL``, release the strong reference
940 to the object referred to by *result* and return ``1``.
941
942 .. versionadded:: 3.2
943
944 .. versionchanged:: 3.6
945 Accepts a :term:`path-like object`.
946
947
948 .. c:function:: PyObject* PyUnicode_DecodeFSDefaultAndSize(const char *str, Py_ssize_t size)
949
950 Decode a string from the :term:`filesystem encoding and error handler`.
951
952 If you need to decode a string from the current locale encoding, use
953 :c:func:`PyUnicode_DecodeLocaleAndSize`.
954
955 .. seealso::
956
957 The :c:func:`Py_DecodeLocale` function.
958
959 .. versionchanged:: 3.6
960 The :term:`filesystem error handler <filesystem encoding and error
961 handler>` is now used.
962
963
964 .. c:function:: PyObject* PyUnicode_DecodeFSDefault(const char *str)
965
966 Decode a null-terminated string from the :term:`filesystem encoding and
967 error handler`.
968
969 If the string length is known, use
970 :c:func:`PyUnicode_DecodeFSDefaultAndSize`.
971
972 .. versionchanged:: 3.6
973 The :term:`filesystem error handler <filesystem encoding and error
974 handler>` is now used.
975
976
977 .. c:function:: PyObject* PyUnicode_EncodeFSDefault(PyObject *unicode)
978
979 Encode a Unicode object to the :term:`filesystem encoding and error
980 handler`, and return :class:`bytes`. Note that the resulting :class:`bytes`
981 object can contain null bytes.
982
983 If you need to encode a string to the current locale encoding, use
984 :c:func:`PyUnicode_EncodeLocale`.
985
986 .. seealso::
987
988 The :c:func:`Py_EncodeLocale` function.
989
990 .. versionadded:: 3.2
991
992 .. versionchanged:: 3.6
993 The :term:`filesystem error handler <filesystem encoding and error
994 handler>` is now used.
995
996 wchar_t Support
997 """""""""""""""
998
999 :c:type:`wchar_t` support for platforms which support it:
1000
1001 .. c:function:: PyObject* PyUnicode_FromWideChar(const wchar_t *wstr, Py_ssize_t size)
1002
1003 Create a Unicode object from the :c:type:`wchar_t` buffer *wstr* of the given *size*.
1004 Passing ``-1`` as the *size* indicates that the function must itself compute the length,
1005 using :c:func:`!wcslen`.
1006 Return ``NULL`` on failure.
1007
1008
1009 .. c:function:: Py_ssize_t PyUnicode_AsWideChar(PyObject *unicode, wchar_t *wstr, Py_ssize_t size)
1010
1011 Copy the Unicode object contents into the :c:type:`wchar_t` buffer *wstr*. At most
1012 *size* :c:type:`wchar_t` characters are copied (excluding a possibly trailing
1013 null termination character). Return the number of :c:type:`wchar_t` characters
1014 copied or ``-1`` in case of an error.
1015
1016 When *wstr* is ``NULL``, instead return the *size* that would be required
1017 to store all of *unicode* including a terminating null.
1018
1019 Note that the resulting :c:expr:`wchar_t*`
1020 string may or may not be null-terminated. It is the responsibility of the caller
1021 to make sure that the :c:expr:`wchar_t*` string is null-terminated in case this is
1022 required by the application. Also, note that the :c:expr:`wchar_t*` string
1023 might contain null characters, which would cause the string to be truncated
1024 when used with most C functions.
1025
1026
1027 .. c:function:: wchar_t* PyUnicode_AsWideCharString(PyObject *unicode, Py_ssize_t *size)
1028
1029 Convert the Unicode object to a wide character string. The output string
1030 always ends with a null character. If *size* is not ``NULL``, write the number
1031 of wide characters (excluding the trailing null termination character) into
1032 *\*size*. Note that the resulting :c:type:`wchar_t` string might contain
1033 null characters, which would cause the string to be truncated when used with
1034 most C functions. If *size* is ``NULL`` and the :c:expr:`wchar_t*` string
1035 contains null characters a :exc:`ValueError` is raised.
1036
1037 Returns a buffer allocated by :c:macro:`PyMem_New` (use
1038 :c:func:`PyMem_Free` to free it) on success. On error, returns ``NULL``
1039 and *\*size* is undefined. Raises a :exc:`MemoryError` if memory allocation
1040 is failed.
1041
1042 .. versionadded:: 3.2
1043
1044 .. versionchanged:: 3.7
1045 Raises a :exc:`ValueError` if *size* is ``NULL`` and the :c:expr:`wchar_t*`
1046 string contains null characters.
1047
1048
1049 .. _builtincodecs:
1050
1051 Built-in Codecs
1052 ^^^^^^^^^^^^^^^
1053
1054 Python provides a set of built-in codecs which are written in C for speed. All of
1055 these codecs are directly usable via the following functions.
1056
1057 Many of the following APIs take two arguments encoding and errors, and they
1058 have the same semantics as the ones of the built-in :func:`str` string object
1059 constructor.
1060
1061 Setting encoding to ``NULL`` causes the default encoding to be used
1062 which is UTF-8. The file system calls should use
1063 :c:func:`PyUnicode_FSConverter` for encoding file names. This uses the
1064 :term:`filesystem encoding and error handler` internally.
1065
1066 Error handling is set by errors which may also be set to ``NULL`` meaning to use
1067 the default handling defined for the codec. Default error handling for all
1068 built-in codecs is "strict" (:exc:`ValueError` is raised).
1069
1070 The codecs all use a similar interface. Only deviations from the following
1071 generic ones are documented for simplicity.
1072
1073
1074 Generic Codecs
1075 """"""""""""""
1076
1077 The following macro is provided:
1078
1079
1080 .. c:macro:: Py_UNICODE_REPLACEMENT_CHARACTER
1081
1082 The Unicode code point ``U+FFFD`` (replacement character).
1083
1084 This Unicode character is used as the replacement character during
1085 decoding if the *errors* argument is set to "replace".
1086
1087
1088 These are the generic codec APIs:
1089
1090
1091 .. c:function:: PyObject* PyUnicode_Decode(const char *str, Py_ssize_t size, \
1092 const char *encoding, const char *errors)
1093
1094 Create a Unicode object by decoding *size* bytes of the encoded string *str*.
1095 *encoding* and *errors* have the same meaning as the parameters of the same name
1096 in the :func:`str` built-in function. The codec to be used is looked up
1097 using the Python codec registry. Return ``NULL`` if an exception was raised by
1098 the codec.
1099
1100
1101 .. c:function:: PyObject* PyUnicode_AsEncodedString(PyObject *unicode, \
1102 const char *encoding, const char *errors)
1103
1104 Encode a Unicode object and return the result as Python bytes object.
1105 *encoding* and *errors* have the same meaning as the parameters of the same
1106 name in the Unicode :meth:`~str.encode` method. The codec to be used is looked up
1107 using the Python codec registry. Return ``NULL`` if an exception was raised by
1108 the codec.
1109
1110
1111 UTF-8 Codecs
1112 """"""""""""
1113
1114 These are the UTF-8 codec APIs:
1115
1116
1117 .. c:function:: PyObject* PyUnicode_DecodeUTF8(const char *str, Py_ssize_t size, const char *errors)
1118
1119 Create a Unicode object by decoding *size* bytes of the UTF-8 encoded string
1120 *str*. Return ``NULL`` if an exception was raised by the codec.
1121
1122
1123 .. c:function:: PyObject* PyUnicode_DecodeUTF8Stateful(const char *str, Py_ssize_t size, \
1124 const char *errors, Py_ssize_t *consumed)
1125
1126 If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeUTF8`. If
1127 *consumed* is not ``NULL``, trailing incomplete UTF-8 byte sequences will not be
1128 treated as an error. Those bytes will not be decoded and the number of bytes
1129 that have been decoded will be stored in *consumed*.
1130
1131
1132 .. c:function:: PyObject* PyUnicode_AsUTF8String(PyObject *unicode)
1133
1134 Encode a Unicode object using UTF-8 and return the result as Python bytes
1135 object. Error handling is "strict". Return ``NULL`` if an exception was
1136 raised by the codec.
1137
1138 The function fails if the string contains surrogate code points
1139 (``U+D800`` - ``U+DFFF``).
1140
1141
1142 .. c:function:: const char* PyUnicode_AsUTF8AndSize(PyObject *unicode, Py_ssize_t *size)
1143
1144 Return a pointer to the UTF-8 encoding of the Unicode object, and
1145 store the size of the encoded representation (in bytes) in *size*. The
1146 *size* argument can be ``NULL``; in this case no size will be stored. The
1147 returned buffer always has an extra null byte appended (not included in
1148 *size*), regardless of whether there are any other null code points.
1149
1150 On error, set an exception, set *size* to ``-1`` (if it's not NULL) and
1151 return ``NULL``.
1152
1153 The function fails if the string contains surrogate code points
1154 (``U+D800`` - ``U+DFFF``).
1155
1156 This caches the UTF-8 representation of the string in the Unicode object, and
1157 subsequent calls will return a pointer to the same buffer. The caller is not
1158 responsible for deallocating the buffer. The buffer is deallocated and
1159 pointers to it become invalid when the Unicode object is garbage collected.
1160
1161 .. versionadded:: 3.3
1162
1163 .. versionchanged:: 3.7
1164 The return type is now ``const char *`` rather than ``char *``.
1165
1166 .. versionchanged:: 3.10
1167 This function is a part of the :ref:`limited API <limited-c-api>`.
1168
1169
1170 .. c:function:: const char* PyUnicode_AsUTF8(PyObject *unicode)
1171
1172 As :c:func:`PyUnicode_AsUTF8AndSize`, but does not store the size.
1173
1174 .. warning::
1175
1176 This function does not have any special behavior for
1177 `null characters <https://en.wikipedia.org/wiki/Null_character>`_ embedded within
1178 *unicode*. As a result, strings containing null characters will remain in the returned
1179 string, which some C functions might interpret as the end of the string, leading to
1180 truncation. If truncation is an issue, it is recommended to use :c:func:`PyUnicode_AsUTF8AndSize`
1181 instead.
1182
1183 .. versionadded:: 3.3
1184
1185 .. versionchanged:: 3.7
1186 The return type is now ``const char *`` rather than ``char *``.
1187
1188
1189 UTF-32 Codecs
1190 """""""""""""
1191
1192 These are the UTF-32 codec APIs:
1193
1194
1195 .. c:function:: PyObject* PyUnicode_DecodeUTF32(const char *str, Py_ssize_t size, \
1196 const char *errors, int *byteorder)
1197
1198 Decode *size* bytes from a UTF-32 encoded buffer string and return the
1199 corresponding Unicode object. *errors* (if non-``NULL``) defines the error
1200 handling. It defaults to "strict".
1201
1202 If *byteorder* is non-``NULL``, the decoder starts decoding using the given byte
1203 order::
1204
1205 *byteorder == -1: little endian
1206 *byteorder == 0: native order
1207 *byteorder == 1: big endian
1208
1209 If ``*byteorder`` is zero, and the first four bytes of the input data are a
1210 byte order mark (BOM), the decoder switches to this byte order and the BOM is
1211 not copied into the resulting Unicode string. If ``*byteorder`` is ``-1`` or
1212 ``1``, any byte order mark is copied to the output.
1213
1214 After completion, *\*byteorder* is set to the current byte order at the end
1215 of input data.
1216
1217 If *byteorder* is ``NULL``, the codec starts in native order mode.
1218
1219 Return ``NULL`` if an exception was raised by the codec.
1220
1221
1222 .. c:function:: PyObject* PyUnicode_DecodeUTF32Stateful(const char *str, Py_ssize_t size, \
1223 const char *errors, int *byteorder, Py_ssize_t *consumed)
1224
1225 If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeUTF32`. If
1226 *consumed* is not ``NULL``, :c:func:`PyUnicode_DecodeUTF32Stateful` will not treat
1227 trailing incomplete UTF-32 byte sequences (such as a number of bytes not divisible
1228 by four) as an error. Those bytes will not be decoded and the number of bytes
1229 that have been decoded will be stored in *consumed*.
1230
1231
1232 .. c:function:: PyObject* PyUnicode_AsUTF32String(PyObject *unicode)
1233
1234 Return a Python byte string using the UTF-32 encoding in native byte
1235 order. The string always starts with a BOM mark. Error handling is "strict".
1236 Return ``NULL`` if an exception was raised by the codec.
1237
1238
1239 UTF-16 Codecs
1240 """""""""""""
1241
1242 These are the UTF-16 codec APIs:
1243
1244
1245 .. c:function:: PyObject* PyUnicode_DecodeUTF16(const char *str, Py_ssize_t size, \
1246 const char *errors, int *byteorder)
1247
1248 Decode *size* bytes from a UTF-16 encoded buffer string and return the
1249 corresponding Unicode object. *errors* (if non-``NULL``) defines the error
1250 handling. It defaults to "strict".
1251
1252 If *byteorder* is non-``NULL``, the decoder starts decoding using the given byte
1253 order::
1254
1255 *byteorder == -1: little endian
1256 *byteorder == 0: native order
1257 *byteorder == 1: big endian
1258
1259 If ``*byteorder`` is zero, and the first two bytes of the input data are a
1260 byte order mark (BOM), the decoder switches to this byte order and the BOM is
1261 not copied into the resulting Unicode string. If ``*byteorder`` is ``-1`` or
1262 ``1``, any byte order mark is copied to the output (where it will result in
1263 either a ``\ufeff`` or a ``\ufffe`` character).
1264
1265 After completion, ``*byteorder`` is set to the current byte order at the end
1266 of input data.
1267
1268 If *byteorder* is ``NULL``, the codec starts in native order mode.
1269
1270 Return ``NULL`` if an exception was raised by the codec.
1271
1272
1273 .. c:function:: PyObject* PyUnicode_DecodeUTF16Stateful(const char *str, Py_ssize_t size, \
1274 const char *errors, int *byteorder, Py_ssize_t *consumed)
1275
1276 If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeUTF16`. If
1277 *consumed* is not ``NULL``, :c:func:`PyUnicode_DecodeUTF16Stateful` will not treat
1278 trailing incomplete UTF-16 byte sequences (such as an odd number of bytes or a
1279 split surrogate pair) as an error. Those bytes will not be decoded and the
1280 number of bytes that have been decoded will be stored in *consumed*.
1281
1282
1283 .. c:function:: PyObject* PyUnicode_AsUTF16String(PyObject *unicode)
1284
1285 Return a Python byte string using the UTF-16 encoding in native byte
1286 order. The string always starts with a BOM mark. Error handling is "strict".
1287 Return ``NULL`` if an exception was raised by the codec.
1288
1289
1290 UTF-7 Codecs
1291 """"""""""""
1292
1293 These are the UTF-7 codec APIs:
1294
1295
1296 .. c:function:: PyObject* PyUnicode_DecodeUTF7(const char *str, Py_ssize_t size, const char *errors)
1297
1298 Create a Unicode object by decoding *size* bytes of the UTF-7 encoded string
1299 *str*. Return ``NULL`` if an exception was raised by the codec.
1300
1301
1302 .. c:function:: PyObject* PyUnicode_DecodeUTF7Stateful(const char *str, Py_ssize_t size, \
1303 const char *errors, Py_ssize_t *consumed)
1304
1305 If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeUTF7`. If
1306 *consumed* is not ``NULL``, trailing incomplete UTF-7 base-64 sections will not
1307 be treated as an error. Those bytes will not be decoded and the number of
1308 bytes that have been decoded will be stored in *consumed*.
1309
1310
1311 Unicode-Escape Codecs
1312 """""""""""""""""""""
1313
1314 These are the "Unicode Escape" codec APIs:
1315
1316
1317 .. c:function:: PyObject* PyUnicode_DecodeUnicodeEscape(const char *str, \
1318 Py_ssize_t size, const char *errors)
1319
1320 Create a Unicode object by decoding *size* bytes of the Unicode-Escape encoded
1321 string *str*. Return ``NULL`` if an exception was raised by the codec.
1322
1323
1324 .. c:function:: PyObject* PyUnicode_AsUnicodeEscapeString(PyObject *unicode)
1325
1326 Encode a Unicode object using Unicode-Escape and return the result as a
1327 bytes object. Error handling is "strict". Return ``NULL`` if an exception was
1328 raised by the codec.
1329
1330
1331 Raw-Unicode-Escape Codecs
1332 """""""""""""""""""""""""
1333
1334 These are the "Raw Unicode Escape" codec APIs:
1335
1336
1337 .. c:function:: PyObject* PyUnicode_DecodeRawUnicodeEscape(const char *str, \
1338 Py_ssize_t size, const char *errors)
1339
1340 Create a Unicode object by decoding *size* bytes of the Raw-Unicode-Escape
1341 encoded string *str*. Return ``NULL`` if an exception was raised by the codec.
1342
1343
1344 .. c:function:: PyObject* PyUnicode_AsRawUnicodeEscapeString(PyObject *unicode)
1345
1346 Encode a Unicode object using Raw-Unicode-Escape and return the result as
1347 a bytes object. Error handling is "strict". Return ``NULL`` if an exception
1348 was raised by the codec.
1349
1350
1351 Latin-1 Codecs
1352 """"""""""""""
1353
1354 These are the Latin-1 codec APIs: Latin-1 corresponds to the first 256 Unicode
1355 ordinals and only these are accepted by the codecs during encoding.
1356
1357
1358 .. c:function:: PyObject* PyUnicode_DecodeLatin1(const char *str, Py_ssize_t size, const char *errors)
1359
1360 Create a Unicode object by decoding *size* bytes of the Latin-1 encoded string
1361 *str*. Return ``NULL`` if an exception was raised by the codec.
1362
1363
1364 .. c:function:: PyObject* PyUnicode_AsLatin1String(PyObject *unicode)
1365
1366 Encode a Unicode object using Latin-1 and return the result as Python bytes
1367 object. Error handling is "strict". Return ``NULL`` if an exception was
1368 raised by the codec.
1369
1370
1371 ASCII Codecs
1372 """"""""""""
1373
1374 These are the ASCII codec APIs. Only 7-bit ASCII data is accepted. All other
1375 codes generate errors.
1376
1377
1378 .. c:function:: PyObject* PyUnicode_DecodeASCII(const char *str, Py_ssize_t size, const char *errors)
1379
1380 Create a Unicode object by decoding *size* bytes of the ASCII encoded string
1381 *str*. Return ``NULL`` if an exception was raised by the codec.
1382
1383
1384 .. c:function:: PyObject* PyUnicode_AsASCIIString(PyObject *unicode)
1385
1386 Encode a Unicode object using ASCII and return the result as Python bytes
1387 object. Error handling is "strict". Return ``NULL`` if an exception was
1388 raised by the codec.
1389
1390
1391 Character Map Codecs
1392 """"""""""""""""""""
1393
1394 This codec is special in that it can be used to implement many different codecs
1395 (and this is in fact what was done to obtain most of the standard codecs
1396 included in the :mod:`!encodings` package). The codec uses mappings to encode and
1397 decode characters. The mapping objects provided must support the
1398 :meth:`~object.__getitem__` mapping interface; dictionaries and sequences work well.
1399
1400 These are the mapping codec APIs:
1401
1402 .. c:function:: PyObject* PyUnicode_DecodeCharmap(const char *str, Py_ssize_t length, \
1403 PyObject *mapping, const char *errors)
1404
1405 Create a Unicode object by decoding *size* bytes of the encoded string *str*
1406 using the given *mapping* object. Return ``NULL`` if an exception was raised
1407 by the codec.
1408
1409 If *mapping* is ``NULL``, Latin-1 decoding will be applied. Else
1410 *mapping* must map bytes ordinals (integers in the range from 0 to 255)
1411 to Unicode strings, integers (which are then interpreted as Unicode
1412 ordinals) or ``None``. Unmapped data bytes -- ones which cause a
1413 :exc:`LookupError`, as well as ones which get mapped to ``None``,
1414 ``0xFFFE`` or ``'\ufffe'``, are treated as undefined mappings and cause
1415 an error.
1416
1417
1418 .. c:function:: PyObject* PyUnicode_AsCharmapString(PyObject *unicode, PyObject *mapping)
1419
1420 Encode a Unicode object using the given *mapping* object and return the
1421 result as a bytes object. Error handling is "strict". Return ``NULL`` if an
1422 exception was raised by the codec.
1423
1424 The *mapping* object must map Unicode ordinal integers to bytes objects,
1425 integers in the range from 0 to 255 or ``None``. Unmapped character
1426 ordinals (ones which cause a :exc:`LookupError`) as well as mapped to
1427 ``None`` are treated as "undefined mapping" and cause an error.
1428
1429
1430 The following codec API is special in that maps Unicode to Unicode.
1431
1432 .. c:function:: PyObject* PyUnicode_Translate(PyObject *unicode, PyObject *table, const char *errors)
1433
1434 Translate a string by applying a character mapping table to it and return the
1435 resulting Unicode object. Return ``NULL`` if an exception was raised by the
1436 codec.
1437
1438 The mapping table must map Unicode ordinal integers to Unicode ordinal integers
1439 or ``None`` (causing deletion of the character).
1440
1441 Mapping tables need only provide the :meth:`~object.__getitem__` interface; dictionaries
1442 and sequences work well. Unmapped character ordinals (ones which cause a
1443 :exc:`LookupError`) are left untouched and are copied as-is.
1444
1445 *errors* has the usual meaning for codecs. It may be ``NULL`` which indicates to
1446 use the default error handling.
1447
1448
1449 MBCS codecs for Windows
1450 """""""""""""""""""""""
1451
1452 These are the MBCS codec APIs. They are currently only available on Windows and
1453 use the Win32 MBCS converters to implement the conversions. Note that MBCS (or
1454 DBCS) is a class of encodings, not just one. The target encoding is defined by
1455 the user settings on the machine running the codec.
1456
1457 .. c:function:: PyObject* PyUnicode_DecodeMBCS(const char *str, Py_ssize_t size, const char *errors)
1458
1459 Create a Unicode object by decoding *size* bytes of the MBCS encoded string *str*.
1460 Return ``NULL`` if an exception was raised by the codec.
1461
1462
1463 .. c:function:: PyObject* PyUnicode_DecodeMBCSStateful(const char *str, Py_ssize_t size, \
1464 const char *errors, Py_ssize_t *consumed)
1465
1466 If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeMBCS`. If
1467 *consumed* is not ``NULL``, :c:func:`PyUnicode_DecodeMBCSStateful` will not decode
1468 trailing lead byte and the number of bytes that have been decoded will be stored
1469 in *consumed*.
1470
1471
1472 .. c:function:: PyObject* PyUnicode_DecodeCodePageStateful(int code_page, const char *str, \
1473 Py_ssize_t size, const char *errors, Py_ssize_t *consumed)
1474
1475 Similar to :c:func:`PyUnicode_DecodeMBCSStateful`, except uses the code page
1476 specified by *code_page*.
1477
1478
1479 .. c:function:: PyObject* PyUnicode_AsMBCSString(PyObject *unicode)
1480
1481 Encode a Unicode object using MBCS and return the result as Python bytes
1482 object. Error handling is "strict". Return ``NULL`` if an exception was
1483 raised by the codec.
1484
1485
1486 .. c:function:: PyObject* PyUnicode_EncodeCodePage(int code_page, PyObject *unicode, const char *errors)
1487
1488 Encode the Unicode object using the specified code page and return a Python
1489 bytes object. Return ``NULL`` if an exception was raised by the codec. Use
1490 :c:macro:`!CP_ACP` code page to get the MBCS encoder.
1491
1492 .. versionadded:: 3.3
1493
1494
1495 .. _unicodemethodsandslots:
1496
1497 Methods and Slot Functions
1498 ^^^^^^^^^^^^^^^^^^^^^^^^^^
1499
1500 The following APIs are capable of handling Unicode objects and strings on input
1501 (we refer to them as strings in the descriptions) and return Unicode objects or
1502 integers as appropriate.
1503
1504 They all return ``NULL`` or ``-1`` if an exception occurs.
1505
1506
1507 .. c:function:: PyObject* PyUnicode_Concat(PyObject *left, PyObject *right)
1508
1509 Concat two strings giving a new Unicode string.
1510
1511
1512 .. c:function:: PyObject* PyUnicode_Split(PyObject *unicode, PyObject *sep, Py_ssize_t maxsplit)
1513
1514 Split a string giving a list of Unicode strings. If *sep* is ``NULL``, splitting
1515 will be done at all whitespace substrings. Otherwise, splits occur at the given
1516 separator. At most *maxsplit* splits will be done. If negative, no limit is
1517 set. Separators are not included in the resulting list.
1518
1519 On error, return ``NULL`` with an exception set.
1520
1521 Equivalent to :py:meth:`str.split`.
1522
1523
1524 .. c:function:: PyObject* PyUnicode_RSplit(PyObject *unicode, PyObject *sep, Py_ssize_t maxsplit)
1525
1526 Similar to :c:func:`PyUnicode_Split`, but splitting will be done beginning
1527 at the end of the string.
1528
1529 On error, return ``NULL`` with an exception set.
1530
1531 Equivalent to :py:meth:`str.rsplit`.
1532
1533
1534 .. c:function:: PyObject* PyUnicode_Splitlines(PyObject *unicode, int keepends)
1535
1536 Split a Unicode string at line breaks, returning a list of Unicode strings.
1537 CRLF is considered to be one line break. If *keepends* is ``0``, the Line break
1538 characters are not included in the resulting strings.
1539
1540
1541 .. c:function:: PyObject* PyUnicode_Partition(PyObject *unicode, PyObject *sep)
1542
1543 Split a Unicode string at the first occurrence of *sep*, and return
1544 a 3-tuple containing the part before the separator, the separator itself,
1545 and the part after the separator. If the separator is not found,
1546 return a 3-tuple containing the string itself, followed by two empty strings.
1547
1548 *sep* must not be empty.
1549
1550 On error, return ``NULL`` with an exception set.
1551
1552 Equivalent to :py:meth:`str.partition`.
1553
1554
1555 .. c:function:: PyObject* PyUnicode_RPartition(PyObject *unicode, PyObject *sep)
1556
1557 Similar to :c:func:`PyUnicode_Partition`, but split a Unicode string at the
1558 last occurrence of *sep*. If the separator is not found, return a 3-tuple
1559 containing two empty strings, followed by the string itself.
1560
1561 *sep* must not be empty.
1562
1563 On error, return ``NULL`` with an exception set.
1564
1565 Equivalent to :py:meth:`str.rpartition`.
1566
1567
1568 .. c:function:: PyObject* PyUnicode_Join(PyObject *separator, PyObject *seq)
1569
1570 Join a sequence of strings using the given *separator* and return the resulting
1571 Unicode string.
1572
1573
1574 .. c:function:: Py_ssize_t PyUnicode_Tailmatch(PyObject *unicode, PyObject *substr, \
1575 Py_ssize_t start, Py_ssize_t end, int direction)
1576
1577 Return ``1`` if *substr* matches ``unicode[start:end]`` at the given tail end
1578 (*direction* == ``-1`` means to do a prefix match, *direction* == ``1`` a suffix match),
1579 ``0`` otherwise. Return ``-1`` if an error occurred.
1580
1581
1582 .. c:function:: Py_ssize_t PyUnicode_Find(PyObject *unicode, PyObject *substr, \
1583 Py_ssize_t start, Py_ssize_t end, int direction)
1584
1585 Return the first position of *substr* in ``unicode[start:end]`` using the given
1586 *direction* (*direction* == ``1`` means to do a forward search, *direction* == ``-1`` a
1587 backward search). The return value is the index of the first match; a value of
1588 ``-1`` indicates that no match was found, and ``-2`` indicates that an error
1589 occurred and an exception has been set.
1590
1591
1592 .. c:function:: Py_ssize_t PyUnicode_FindChar(PyObject *unicode, Py_UCS4 ch, \
1593 Py_ssize_t start, Py_ssize_t end, int direction)
1594
1595 Return the first position of the character *ch* in ``unicode[start:end]`` using
1596 the given *direction* (*direction* == ``1`` means to do a forward search,
1597 *direction* == ``-1`` a backward search). The return value is the index of the
1598 first match; a value of ``-1`` indicates that no match was found, and ``-2``
1599 indicates that an error occurred and an exception has been set.
1600
1601 .. versionadded:: 3.3
1602
1603 .. versionchanged:: 3.7
1604 *start* and *end* are now adjusted to behave like ``unicode[start:end]``.
1605
1606
1607 .. c:function:: Py_ssize_t PyUnicode_Count(PyObject *unicode, PyObject *substr, \
1608 Py_ssize_t start, Py_ssize_t end)
1609
1610 Return the number of non-overlapping occurrences of *substr* in
1611 ``unicode[start:end]``. Return ``-1`` if an error occurred.
1612
1613
1614 .. c:function:: PyObject* PyUnicode_Replace(PyObject *unicode, PyObject *substr, \
1615 PyObject *replstr, Py_ssize_t maxcount)
1616
1617 Replace at most *maxcount* occurrences of *substr* in *unicode* with *replstr* and
1618 return the resulting Unicode object. *maxcount* == ``-1`` means replace all
1619 occurrences.
1620
1621
1622 .. c:function:: int PyUnicode_Compare(PyObject *left, PyObject *right)
1623
1624 Compare two strings and return ``-1``, ``0``, ``1`` for less than, equal, and greater than,
1625 respectively.
1626
1627 This function returns ``-1`` upon failure, so one should call
1628 :c:func:`PyErr_Occurred` to check for errors.
1629
1630 .. seealso::
1631
1632 The :c:func:`PyUnicode_Equal` function.
1633
1634
1635 .. c:function:: int PyUnicode_Equal(PyObject *a, PyObject *b)
1636
1637 Test if two strings are equal:
1638
1639 * Return ``1`` if *a* is equal to *b*.
1640 * Return ``0`` if *a* is not equal to *b*.
1641 * Set a :exc:`TypeError` exception and return ``-1`` if *a* or *b* is not a
1642 :class:`str` object.
1643
1644 The function always succeeds if *a* and *b* are :class:`str` objects.
1645
1646 The function works for :class:`str` subclasses, but does not honor custom
1647 ``__eq__()`` method.
1648
1649 .. seealso::
1650
1651 The :c:func:`PyUnicode_Compare` function.
1652
1653 .. versionadded:: 3.14
1654
1655
1656 .. c:function:: int PyUnicode_EqualToUTF8AndSize(PyObject *unicode, const char *string, Py_ssize_t size)
1657
1658 Compare a Unicode object with a char buffer which is interpreted as
1659 being UTF-8 or ASCII encoded and return true (``1``) if they are equal,
1660 or false (``0``) otherwise.
1661 If the Unicode object contains surrogate code points
1662 (``U+D800`` - ``U+DFFF``) or the C string is not valid UTF-8,
1663 false (``0``) is returned.
1664
1665 This function does not raise exceptions.
1666
1667 .. versionadded:: 3.13
1668
1669
1670 .. c:function:: int PyUnicode_EqualToUTF8(PyObject *unicode, const char *string)
1671
1672 Similar to :c:func:`PyUnicode_EqualToUTF8AndSize`, but compute *string*
1673 length using :c:func:`!strlen`.
1674 If the Unicode object contains null characters, false (``0``) is returned.
1675
1676 .. versionadded:: 3.13
1677
1678
1679 .. c:function:: int PyUnicode_CompareWithASCIIString(PyObject *unicode, const char *string)
1680
1681 Compare a Unicode object, *unicode*, with *string* and return ``-1``, ``0``, ``1`` for less
1682 than, equal, and greater than, respectively. It is best to pass only
1683 ASCII-encoded strings, but the function interprets the input string as
1684 ISO-8859-1 if it contains non-ASCII characters.
1685
1686 This function does not raise exceptions.
1687
1688
1689 .. c:function:: PyObject* PyUnicode_RichCompare(PyObject *left, PyObject *right, int op)
1690
1691 Rich compare two Unicode strings and return one of the following:
1692
1693 * ``NULL`` in case an exception was raised
1694 * :c:data:`Py_True` or :c:data:`Py_False` for successful comparisons
1695 * :c:data:`Py_NotImplemented` in case the type combination is unknown
1696
1697 Possible values for *op* are :c:macro:`Py_GT`, :c:macro:`Py_GE`, :c:macro:`Py_EQ`,
1698 :c:macro:`Py_NE`, :c:macro:`Py_LT`, and :c:macro:`Py_LE`.
1699
1700
1701 .. c:function:: PyObject* PyUnicode_Format(PyObject *format, PyObject *args)
1702
1703 Return a new string object from *format* and *args*; this is analogous to
1704 ``format % args``.
1705
1706
1707 .. c:function:: int PyUnicode_Contains(PyObject *unicode, PyObject *substr)
1708
1709 Check whether *substr* is contained in *unicode* and return true or false
1710 accordingly.
1711
1712 *substr* has to coerce to a one element Unicode string. ``-1`` is returned
1713 if there was an error.
1714
1715
1716 .. c:function:: void PyUnicode_InternInPlace(PyObject **p_unicode)
1717
1718 Intern the argument :c:expr:`*p_unicode` in place. The argument must be the address of a
1719 pointer variable pointing to a Python Unicode string object. If there is an
1720 existing interned string that is the same as :c:expr:`*p_unicode`, it sets :c:expr:`*p_unicode` to
1721 it (releasing the reference to the old string object and creating a new
1722 :term:`strong reference` to the interned string object), otherwise it leaves
1723 :c:expr:`*p_unicode` alone and interns it.
1724
1725 (Clarification: even though there is a lot of talk about references, think
1726 of this function as reference-neutral. You must own the object you pass in;
1727 after the call you no longer own the passed-in reference, but you newly own
1728 the result.)
1729
1730 This function never raises an exception.
1731 On error, it leaves its argument unchanged without interning it.
1732
1733 Instances of subclasses of :py:class:`str` may not be interned, that is,
1734 :c:expr:`PyUnicode_CheckExact(*p_unicode)` must be true. If it is not,
1735 then -- as with any other error -- the argument is left unchanged.
1736
1737 Note that interned strings are not “immortal”.
1738 You must keep a reference to the result to benefit from interning.
1739
1740
1741 .. c:function:: PyObject* PyUnicode_InternFromString(const char *str)
1742
1743 A combination of :c:func:`PyUnicode_FromString` and
1744 :c:func:`PyUnicode_InternInPlace`, meant for statically allocated strings.
1745
1746 Return a new ("owned") reference to either a new Unicode string object
1747 that has been interned, or an earlier interned string object with the
1748 same value.
1749
1750 Python may keep a reference to the result, or make it :term:`immortal`,
1751 preventing it from being garbage-collected promptly.
1752 For interning an unbounded number of different strings, such as ones coming
1753 from user input, prefer calling :c:func:`PyUnicode_FromString` and
1754 :c:func:`PyUnicode_InternInPlace` directly.
1755
1756
1757 .. c:function:: unsigned int PyUnicode_CHECK_INTERNED(PyObject *str)
1758
1759 Return a non-zero value if *str* is interned, zero if not.
1760 The *str* argument must be a string; this is not checked.
1761 This function always succeeds.
1762
1763 .. impl-detail::
1764
1765 A non-zero return value may carry additional information
1766 about *how* the string is interned.
1767 The meaning of such non-zero values, as well as each specific string's
1768 intern-related details, may change between CPython versions.
1769
1770
1771 PyUnicodeWriter
1772 ^^^^^^^^^^^^^^^
1773
1774 The :c:type:`PyUnicodeWriter` API can be used to create a Python :class:`str`
1775 object.
1776
1777 .. versionadded:: 3.14
1778
1779 .. c:type:: PyUnicodeWriter
1780
1781 A Unicode writer instance.
1782
1783 The instance must be destroyed by :c:func:`PyUnicodeWriter_Finish` on
1784 success, or :c:func:`PyUnicodeWriter_Discard` on error.
1785
1786 .. c:function:: PyUnicodeWriter* PyUnicodeWriter_Create(Py_ssize_t length)
1787
1788 Create a Unicode writer instance.
1789
1790 *length* must be greater than or equal to ``0``.
1791
1792 If *length* is greater than ``0``, preallocate an internal buffer of
1793 *length* characters.
1794
1795 Set an exception and return ``NULL`` on error.
1796
1797 .. c:function:: PyObject* PyUnicodeWriter_Finish(PyUnicodeWriter *writer)
1798
1799 Return the final Python :class:`str` object and destroy the writer instance.
1800
1801 Set an exception and return ``NULL`` on error.
1802
1803 The writer instance is invalid after this call.
1804
1805 .. c:function:: void PyUnicodeWriter_Discard(PyUnicodeWriter *writer)
1806
1807 Discard the internal Unicode buffer and destroy the writer instance.
1808
1809 If *writer* is ``NULL``, no operation is performed.
1810
1811 The writer instance is invalid after this call.
1812
1813 .. c:function:: int PyUnicodeWriter_WriteChar(PyUnicodeWriter *writer, Py_UCS4 ch)
1814
1815 Write the single Unicode character *ch* into *writer*.
1816
1817 On success, return ``0``.
1818 On error, set an exception, leave the writer unchanged, and return ``-1``.
1819
1820 .. c:function:: int PyUnicodeWriter_WriteUTF8(PyUnicodeWriter *writer, const char *str, Py_ssize_t size)
1821
1822 Decode the string *str* from UTF-8 in strict mode and write the output into *writer*.
1823
1824 *size* is the string length in bytes. If *size* is equal to ``-1``, call
1825 ``strlen(str)`` to get the string length.
1826
1827 On success, return ``0``.
1828 On error, set an exception, leave the writer unchanged, and return ``-1``.
1829
1830 See also :c:func:`PyUnicodeWriter_DecodeUTF8Stateful`.
1831
1832 .. c:function:: int PyUnicodeWriter_WriteASCII(PyUnicodeWriter *writer, const char *str, Py_ssize_t size)
1833
1834 Write the ASCII string *str* into *writer*.
1835
1836 *size* is the string length in bytes. If *size* is equal to ``-1``, call
1837 ``strlen(str)`` to get the string length.
1838
1839 *str* must only contain ASCII characters. The behavior is undefined if
1840 *str* contains non-ASCII characters.
1841
1842 On success, return ``0``.
1843 On error, set an exception, leave the writer unchanged, and return ``-1``.
1844
1845 .. c:function:: int PyUnicodeWriter_WriteWideChar(PyUnicodeWriter *writer, const wchar_t *str, Py_ssize_t size)
1846
1847 Write the wide string *str* into *writer*.
1848
1849 *size* is a number of wide characters. If *size* is equal to ``-1``, call
1850 ``wcslen(str)`` to get the string length.
1851
1852 On success, return ``0``.
1853 On error, set an exception, leave the writer unchanged, and return ``-1``.
1854
1855 .. c:function:: int PyUnicodeWriter_WriteUCS4(PyUnicodeWriter *writer, Py_UCS4 *str, Py_ssize_t size)
1856
1857 Writer the UCS4 string *str* into *writer*.
1858
1859 *size* is a number of UCS4 characters.
1860
1861 On success, return ``0``.
1862 On error, set an exception, leave the writer unchanged, and return ``-1``.
1863
1864 .. c:function:: int PyUnicodeWriter_WriteStr(PyUnicodeWriter *writer, PyObject *obj)
1865
1866 Call :c:func:`PyObject_Str` on *obj* and write the output into *writer*.
1867
1868 On success, return ``0``.
1869 On error, set an exception, leave the writer unchanged, and return ``-1``.
1870
1871 To write a :class:`str` subclass which overrides the :meth:`~object.__str__`
1872 method, :c:func:`PyUnicode_FromObject` can be used to get the original
1873 string.
1874
1875 .. c:function:: int PyUnicodeWriter_WriteRepr(PyUnicodeWriter *writer, PyObject *obj)
1876
1877 Call :c:func:`PyObject_Repr` on *obj* and write the output into *writer*.
1878
1879 If *obj* is ``NULL``, write the string ``"<NULL>"`` into *writer*.
1880
1881 On success, return ``0``.
1882 On error, set an exception, leave the writer unchanged, and return ``-1``.
1883
1884 .. versionchanged:: 3.14.4
1885
1886 Added support for ``NULL``.
1887
1888 .. c:function:: int PyUnicodeWriter_WriteSubstring(PyUnicodeWriter *writer, PyObject *str, Py_ssize_t start, Py_ssize_t end)
1889
1890 Write the substring ``str[start:end]`` into *writer*.
1891
1892 *str* must be Python :class:`str` object. *start* must be greater than or
1893 equal to 0, and less than or equal to *end*. *end* must be less than or
1894 equal to *str* length.
1895
1896 On success, return ``0``.
1897 On error, set an exception, leave the writer unchanged, and return ``-1``.
1898
1899 .. c:function:: int PyUnicodeWriter_Format(PyUnicodeWriter *writer, const char *format, ...)
1900
1901 Similar to :c:func:`PyUnicode_FromFormat`, but write the output directly into *writer*.
1902
1903 On success, return ``0``.
1904 On error, set an exception, leave the writer unchanged, and return ``-1``.
1905
1906 .. c:function:: int PyUnicodeWriter_DecodeUTF8Stateful(PyUnicodeWriter *writer, const char *string, Py_ssize_t length, const char *errors, Py_ssize_t *consumed)
1907
1908 Decode the string *str* from UTF-8 with *errors* error handler and write the
1909 output into *writer*.
1910
1911 *size* is the string length in bytes. If *size* is equal to ``-1``, call
1912 ``strlen(str)`` to get the string length.
1913
1914 *errors* is an :ref:`error handler <error-handlers>` name, such as
1915 ``"replace"``. If *errors* is ``NULL``, use the strict error handler.
1916
1917 If *consumed* is not ``NULL``, set *\*consumed* to the number of decoded
1918 bytes on success.
1919 If *consumed* is ``NULL``, treat trailing incomplete UTF-8 byte sequences
1920 as an error.
1921
1922 On success, return ``0``.
1923 On error, set an exception, leave the writer unchanged, and return ``-1``.
1924
1925 See also :c:func:`PyUnicodeWriter_WriteUTF8`.
1926
1927 Deprecated API
1928 ^^^^^^^^^^^^^^
1929
1930 The following API is deprecated.
1931
1932 .. c:type:: Py_UNICODE
1933
1934 This is a typedef of :c:type:`wchar_t`, which is a 16-bit type or 32-bit type
1935 depending on the platform.
1936 Please use :c:type:`wchar_t` directly instead.
1937
1938 .. versionchanged:: 3.3
1939 In previous versions, this was a 16-bit type or a 32-bit type depending on
1940 whether you selected a "narrow" or "wide" Unicode version of Python at
1941 build time.
1942
1943 .. deprecated-removed:: 3.13 3.15
1944
1945
1946 .. c:function:: int PyUnicode_READY(PyObject *unicode)
1947
1948 Do nothing and return ``0``.
1949 This API is kept only for backward compatibility, but there are no plans
1950 to remove it.
1951
1952 .. versionadded:: 3.3
1953
1954 .. deprecated:: 3.10
1955 This API does nothing since Python 3.12.
1956 Previously, this needed to be called for each string created using
1957 the old API (:c:func:`!PyUnicode_FromUnicode` or similar).
1958
1959
1960 .. c:function:: unsigned int PyUnicode_IS_READY(PyObject *unicode)
1961
1962 Do nothing and return ``1``.
1963 This API is kept only for backward compatibility, but there are no plans
1964 to remove it.
1965
1966 .. versionadded:: 3.3
1967
1968 .. deprecated:: 3.14
1969 This API does nothing since Python 3.12.
1970 Previously, this could be called to check if
1971 :c:func:`PyUnicode_READY` is necessary.