Debian Codegraph @

Doc/library/struct.rst

raw
1 :mod:`!struct` --- Interpret bytes as packed binary data
2 ========================================================
3
4 .. testsetup:: *
5
6 from struct import *
7
8 .. module:: struct
9 :synopsis: Interpret bytes as packed binary data.
10
11 **Source code:** :source:`Lib/struct.py`
12
13 .. index::
14 pair: C; structures
15 triple: packing; binary; data
16
17 --------------
18
19 This module converts between Python values and C structs represented
20 as Python :class:`bytes` objects. Compact :ref:`format strings <struct-format-strings>`
21 describe the intended conversions to/from Python values.
22 The module's functions and objects can be used for two largely
23 distinct applications, data exchange with external sources (files or
24 network connections), or data transfer between the Python application
25 and the C layer.
26
27 .. note::
28
29 When no prefix character is given, native mode is the default. It
30 packs or unpacks data based on the platform and compiler on which
31 the Python interpreter was built.
32 The result of packing a given C struct includes pad bytes which
33 maintain proper alignment for the C types involved; similarly,
34 alignment is taken into account when unpacking. In contrast, when
35 communicating data between external sources, the programmer is
36 responsible for defining byte ordering and padding between elements.
37 See :ref:`struct-alignment` for details.
38
39 Several :mod:`struct` functions (and methods of :class:`Struct`) take a *buffer*
40 argument. This refers to objects that implement the :ref:`bufferobjects` and
41 provide either a readable or read-writable buffer. The most common types used
42 for that purpose are :class:`bytes` and :class:`bytearray`, but many other types
43 that can be viewed as an array of bytes implement the buffer protocol, so that
44 they can be read/filled without additional copying from a :class:`bytes` object.
45
46
47 Functions and Exceptions
48 ------------------------
49
50 The module defines the following exception and functions:
51
52
53 .. exception:: error
54
55 Exception raised on various occasions; argument is a string describing what
56 is wrong.
57
58
59 .. function:: pack(format, v1, v2, ...)
60
61 Return a bytes object containing the values *v1*, *v2*, ... packed according
62 to the format string *format*. The arguments must match the values required by
63 the format exactly.
64
65
66 .. function:: pack_into(format, buffer, offset, v1, v2, ...)
67
68 Pack the values *v1*, *v2*, ... according to the format string *format* and
69 write the packed bytes into the writable buffer *buffer* starting at
70 position *offset*. Note that *offset* is a required argument.
71 A negative *offset* counts from the end of *buffer*.
72
73
74 .. function:: unpack(format, buffer)
75
76 Unpack from the buffer *buffer* (presumably packed by ``pack(format, ...)``)
77 according to the format string *format*. The result is a tuple even if it
78 contains exactly one item. The buffer's size in bytes must match the
79 size required by the format, as reflected by :func:`calcsize`.
80
81
82 .. function:: unpack_from(format, /, buffer, offset=0)
83
84 Unpack from *buffer* starting at position *offset*, according to the format
85 string *format*. The result is a tuple even if it contains exactly one
86 item. The buffer's size in bytes, starting at position *offset*, must be at
87 least the size required by the format, as reflected by :func:`calcsize`.
88 A negative *offset* counts from the end of *buffer*.
89
90
91 .. function:: iter_unpack(format, buffer)
92
93 Iteratively unpack from the buffer *buffer* according to the format
94 string *format*. This function returns an iterator which will read
95 equally sized chunks from the buffer until all its contents have been
96 consumed. The buffer's size in bytes must be a multiple of the size
97 required by the format, as reflected by :func:`calcsize`.
98
99 Each iteration yields a tuple as specified by the format string.
100
101 .. versionadded:: 3.4
102
103
104 .. function:: calcsize(format)
105
106 Return the size of the struct (and hence of the bytes object produced by
107 ``pack(format, ...)``) corresponding to the format string *format*.
108
109
110 .. _struct-format-strings:
111
112 Format Strings
113 --------------
114
115 Format strings describe the data layout when
116 packing and unpacking data. They are built up from :ref:`format characters<format-characters>`,
117 which specify the type of data being packed/unpacked. In addition,
118 special characters control the :ref:`byte order, size and alignment<struct-alignment>`.
119 Each format string consists of an optional prefix character which
120 describes the overall properties of the data and one or more format
121 characters which describe the actual data values and padding.
122
123
124 .. _struct-alignment:
125
126 Byte Order, Size, and Alignment
127 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
128
129 By default, C types are represented in the machine's native format and byte
130 order, and properly aligned by skipping pad bytes if necessary (according to the
131 rules used by the C compiler).
132 This behavior is chosen so
133 that the bytes of a packed struct correspond exactly to the memory layout
134 of the corresponding C struct.
135 Whether to use native byte ordering
136 and padding or standard formats depends on the application.
137
138 .. index::
139 single: @ (at); in struct format strings
140 single: = (equals); in struct format strings
141 single: < (less); in struct format strings
142 single: > (greater); in struct format strings
143 single: ! (exclamation); in struct format strings
144
145 Alternatively, the first character of the format string can be used to indicate
146 the byte order, size and alignment of the packed data, according to the
147 following table:
148
149 +-----------+------------------------+----------+-----------+
150 | Character | Byte order | Size | Alignment |
151 +===========+========================+==========+===========+
152 | ``@`` | native | native | native |
153 +-----------+------------------------+----------+-----------+
154 | ``=`` | native | standard | none |
155 +-----------+------------------------+----------+-----------+
156 | ``<`` | little-endian | standard | none |
157 +-----------+------------------------+----------+-----------+
158 | ``>`` | big-endian | standard | none |
159 +-----------+------------------------+----------+-----------+
160 | ``!`` | network (= big-endian) | standard | none |
161 +-----------+------------------------+----------+-----------+
162
163 If the first character is not one of these, ``'@'`` is assumed.
164
165 .. note::
166
167 The number 1023 (``0x3ff`` in hexadecimal) has the following byte representations:
168
169 * ``03 ff`` in big-endian (``>``)
170 * ``ff 03`` in little-endian (``<``)
171
172 Python example:
173
174 >>> import struct
175 >>> struct.pack('>h', 1023)
176 b'\x03\xff'
177 >>> struct.pack('<h', 1023)
178 b'\xff\x03'
179
180 Native byte order is big-endian or little-endian, depending on the
181 host system. For example, Intel x86, AMD64 (x86-64), and Apple M1 are
182 little-endian; IBM z and many legacy architectures are big-endian.
183 Use :data:`sys.byteorder` to check the endianness of your system.
184
185 Native size and alignment are determined using the C compiler's
186 ``sizeof`` expression. This is always combined with native byte order.
187
188 Standard size depends only on the format character; see the table in
189 the :ref:`format-characters` section.
190
191 Note the difference between ``'@'`` and ``'='``: both use native byte order, but
192 the size and alignment of the latter is standardized.
193
194 The form ``'!'`` represents the network byte order which is always big-endian
195 as defined in `IETF RFC 1700 <IETF RFC 1700_>`_.
196
197 There is no way to indicate non-native byte order (force byte-swapping); use the
198 appropriate choice of ``'<'`` or ``'>'``.
199
200 Notes:
201
202 (1) Padding is only automatically added between successive structure members.
203 No padding is added at the beginning or the end of the encoded struct.
204
205 (2) No padding is added when using non-native size and alignment, e.g.
206 with '<', '>', '=', and '!'.
207
208 (3) To align the end of a structure to the alignment requirement of a
209 particular type, end the format with the code for that type with a repeat
210 count of zero. See :ref:`struct-examples`.
211
212
213 .. _format-characters:
214
215 Format Characters
216 ^^^^^^^^^^^^^^^^^
217
218 Format characters have the following meaning; the conversion between C and
219 Python values should be obvious given their types. The 'Standard size' column
220 refers to the size of the packed value in bytes when using standard size; that
221 is, when the format string starts with one of ``'<'``, ``'>'``, ``'!'`` or
222 ``'='``. When using native size, the size of the packed value is
223 platform-dependent.
224
225 +--------+--------------------------+--------------------+----------------+------------+
226 | Format | C Type | Python type | Standard size | Notes |
227 +========+==========================+====================+================+============+
228 | ``x`` | pad byte | no value | | \(7) |
229 +--------+--------------------------+--------------------+----------------+------------+
230 | ``c`` | :c:expr:`char` | bytes of length 1 | 1 | |
231 +--------+--------------------------+--------------------+----------------+------------+
232 | ``b`` | :c:expr:`signed char` | int | 1 | \(2) |
233 +--------+--------------------------+--------------------+----------------+------------+
234 | ``B`` | :c:expr:`unsigned char` | int | 1 | \(2) |
235 +--------+--------------------------+--------------------+----------------+------------+
236 | ``?`` | :c:expr:`_Bool` | bool | 1 | \(1) |
237 +--------+--------------------------+--------------------+----------------+------------+
238 | ``h`` | :c:expr:`short` | int | 2 | \(2) |
239 +--------+--------------------------+--------------------+----------------+------------+
240 | ``H`` | :c:expr:`unsigned short` | int | 2 | \(2) |
241 +--------+--------------------------+--------------------+----------------+------------+
242 | ``i`` | :c:expr:`int` | int | 4 | \(2) |
243 +--------+--------------------------+--------------------+----------------+------------+
244 | ``I`` | :c:expr:`unsigned int` | int | 4 | \(2) |
245 +--------+--------------------------+--------------------+----------------+------------+
246 | ``l`` | :c:expr:`long` | int | 4 | \(2) |
247 +--------+--------------------------+--------------------+----------------+------------+
248 | ``L`` | :c:expr:`unsigned long` | int | 4 | \(2) |
249 +--------+--------------------------+--------------------+----------------+------------+
250 | ``q`` | :c:expr:`long long` | int | 8 | \(2) |
251 +--------+--------------------------+--------------------+----------------+------------+
252 | ``Q`` | :c:expr:`unsigned long | int | 8 | \(2) |
253 | | long` | | | |
254 +--------+--------------------------+--------------------+----------------+------------+
255 | ``n`` | :c:type:`ssize_t` | int | | \(2), \(3) |
256 +--------+--------------------------+--------------------+----------------+------------+
257 | ``N`` | :c:type:`size_t` | int | | \(2), \(3) |
258 +--------+--------------------------+--------------------+----------------+------------+
259 | ``e`` | :c:expr:`_Float16` | float | 2 | \(4), \(6) |
260 +--------+--------------------------+--------------------+----------------+------------+
261 | ``f`` | :c:expr:`float` | float | 4 | \(4) |
262 +--------+--------------------------+--------------------+----------------+------------+
263 | ``d`` | :c:expr:`double` | float | 8 | \(4) |
264 +--------+--------------------------+--------------------+----------------+------------+
265 | ``s`` | :c:expr:`char[]` | bytes | | \(9) |
266 +--------+--------------------------+--------------------+----------------+------------+
267 | ``p`` | :c:expr:`char[]` | bytes | | \(8) |
268 +--------+--------------------------+--------------------+----------------+------------+
269 | ``P`` | :c:expr:`void \*` | int | | \(2), \(5) |
270 +--------+--------------------------+--------------------+----------------+------------+
271
272 .. versionchanged:: 3.3
273 Added support for the ``'n'`` and ``'N'`` formats.
274
275 .. versionchanged:: 3.6
276 Added support for the ``'e'`` format.
277
278
279 Notes:
280
281 (1)
282 .. index:: single: ? (question mark); in struct format strings
283
284 The ``'?'`` conversion code corresponds to the :c:expr:`_Bool` type
285 defined by C standards since C99. In standard mode, it is
286 represented by one byte.
287
288 (2)
289 When attempting to pack a non-integer using any of the integer conversion
290 codes, if the non-integer has a :meth:`~object.__index__` method then that method is
291 called to convert the argument to an integer before packing.
292
293 .. versionchanged:: 3.2
294 Added use of the :meth:`~object.__index__` method for non-integers.
295
296 (3)
297 The ``'n'`` and ``'N'`` conversion codes are only available for the native
298 size (selected as the default or with the ``'@'`` byte order character).
299 For the standard size, you can use whichever of the other integer formats
300 fits your application.
301
302 (4)
303 For the ``'f'``, ``'d'`` and ``'e'`` conversion codes, the packed
304 representation uses the IEEE 754 binary32, binary64 or binary16 format (for
305 ``'f'``, ``'d'`` or ``'e'`` respectively), regardless of the floating-point
306 format used by the platform.
307
308 (5)
309 The ``'P'`` format character is only available for the native byte ordering
310 (selected as the default or with the ``'@'`` byte order character). The byte
311 order character ``'='`` chooses to use little- or big-endian ordering based
312 on the host system. The struct module does not interpret this as native
313 ordering, so the ``'P'`` format is not available.
314
315 (6)
316 The IEEE 754 binary16 "half precision" type was introduced in the 2008
317 revision of the `IEEE 754 standard <ieee 754 standard_>`_. It has a sign
318 bit, a 5-bit exponent and 11-bit precision (with 10 bits explicitly stored),
319 and can represent numbers between approximately ``6.1e-05`` and ``6.5e+04``
320 at full precision. This type is not widely supported by C compilers:
321 it's available as :c:expr:`_Float16` type, if the compiler supports the Annex H
322 of the C23 standard. On a
323 typical machine, an unsigned short can be used for storage, but not for math
324 operations. See the Wikipedia page on the `half-precision floating-point
325 format <half precision format_>`_ for more information.
326
327 (7)
328 When packing, ``'x'`` inserts one NUL byte.
329
330 (8)
331 The ``'p'`` format character encodes a "Pascal string", meaning a short
332 variable-length string stored in a *fixed number of bytes*, given by the count.
333 The first byte stored is the length of the string, or 255, whichever is
334 smaller. The bytes of the string follow. If the byte string passed in to
335 :func:`pack` is too long (longer than the count minus 1), only the leading
336 ``count-1`` bytes of the string are stored. If the byte string is shorter than
337 ``count-1``, it is padded with null bytes so that exactly count bytes in all
338 are used. Note that for :func:`unpack`, the ``'p'`` format character consumes
339 ``count`` bytes, but that the :class:`!bytes` object returned can never contain more than 255
340 bytes.
341 When packing, arguments of types :class:`bytes` and :class:`bytearray`
342 are accepted.
343
344 (9)
345 For the ``'s'`` format character, the count is interpreted as the length of the
346 byte string, not a repeat count like for the other format characters; for example,
347 ``'10s'`` means a single 10-byte string mapping to or from a single
348 Python byte string, while ``'10c'`` means 10
349 separate one byte character elements (e.g., ``cccccccccc``) mapping
350 to or from ten different Python byte objects. (See :ref:`struct-examples`
351 for a concrete demonstration of the difference.)
352 If a count is not given, it defaults to 1. For packing, the byte string is
353 truncated or padded with null bytes as appropriate to make it fit. For
354 unpacking, the resulting :class:`!bytes` object always has exactly the specified number
355 of bytes. As a special case, ``'0s'`` means a single, empty byte string (while
356 ``'0c'`` means 0 characters).
357 When packing, arguments of types :class:`bytes` and :class:`bytearray`
358 are accepted.
359
360 A format character may be preceded by an integral repeat count. For example,
361 the format string ``'4h'`` means exactly the same as ``'hhhh'``.
362
363 Whitespace characters between formats are ignored; a count and its format must
364 not contain whitespace though.
365
366 When packing a value ``x`` using one of the integer formats (``'b'``,
367 ``'B'``, ``'h'``, ``'H'``, ``'i'``, ``'I'``, ``'l'``, ``'L'``,
368 ``'q'``, ``'Q'``), if ``x`` is outside the valid range for that format
369 then :exc:`struct.error` is raised.
370
371 .. versionchanged:: 3.1
372 Previously, some of the integer formats wrapped out-of-range values and
373 raised :exc:`DeprecationWarning` instead of :exc:`struct.error`.
374
375 .. index:: single: ? (question mark); in struct format strings
376
377 For the ``'?'`` format character, the return value is either :const:`True` or
378 :const:`False`. When packing, the truth value of the argument object is used.
379 Either 0 or 1 in the native or standard bool representation will be packed, and
380 any non-zero value will be ``True`` when unpacking.
381
382
383
384 .. _struct-examples:
385
386 Examples
387 ^^^^^^^^
388
389 .. note::
390 Native byte order examples (designated by the ``'@'`` format prefix or
391 lack of any prefix character) may not match what the reader's
392 machine produces as
393 that depends on the platform and compiler.
394
395 Pack and unpack integers of three different sizes, using big endian
396 ordering::
397
398 >>> from struct import *
399 >>> pack(">bhl", 1, 2, 3)
400 b'\x01\x00\x02\x00\x00\x00\x03'
401 >>> unpack('>bhl', b'\x01\x00\x02\x00\x00\x00\x03')
402 (1, 2, 3)
403 >>> calcsize('>bhl')
404 7
405
406 Attempt to pack an integer which is too large for the defined field::
407
408 >>> pack(">h", 99999)
409 Traceback (most recent call last):
410 File "<stdin>", line 1, in <module>
411 struct.error: 'h' format requires -32768 <= number <= 32767
412
413 Demonstrate the difference between ``'s'`` and ``'c'`` format
414 characters::
415
416 >>> pack("@ccc", b'1', b'2', b'3')
417 b'123'
418 >>> pack("@3s", b'123')
419 b'123'
420
421 Unpacked fields can be named by assigning them to variables or by wrapping
422 the result in a named tuple::
423
424 >>> record = b'raymond \x32\x12\x08\x01\x08'
425 >>> name, serialnum, school, gradelevel = unpack('<10sHHb', record)
426
427 >>> from collections import namedtuple
428 >>> Student = namedtuple('Student', 'name serialnum school gradelevel')
429 >>> Student._make(unpack('<10sHHb', record))
430 Student(name=b'raymond ', serialnum=4658, school=264, gradelevel=8)
431
432 The ordering of format characters may have an impact on size in native
433 mode since padding is implicit. In standard mode, the user is
434 responsible for inserting any desired padding.
435 Note in
436 the first ``pack`` call below that three NUL bytes were added after the
437 packed ``'#'`` to align the following integer on a four-byte boundary.
438 In this example, the output was produced on a little endian machine::
439
440 >>> pack('@ci', b'#', 0x12131415)
441 b'#\x00\x00\x00\x15\x14\x13\x12'
442 >>> pack('@ic', 0x12131415, b'#')
443 b'\x15\x14\x13\x12#'
444 >>> calcsize('@ci')
445 8
446 >>> calcsize('@ic')
447 5
448
449 The following format ``'llh0l'`` results in two pad bytes being added
450 at the end, assuming the platform's longs are aligned on 4-byte boundaries::
451
452 >>> pack('@llh0l', 1, 2, 3)
453 b'\x00\x00\x00\x01\x00\x00\x00\x02\x00\x03\x00\x00'
454
455
456 .. seealso::
457
458 Module :mod:`array`
459 Packed binary storage of homogeneous data.
460
461 Module :mod:`json`
462 JSON encoder and decoder.
463
464 Module :mod:`pickle`
465 Python object serialization.
466
467
468 .. _applications:
469
470 Applications
471 ------------
472
473 Two main applications for the :mod:`struct` module exist, data
474 interchange between Python and C code within an application or another
475 application compiled using the same compiler (:ref:`native formats<struct-native-formats>`), and
476 data interchange between applications using agreed upon data layout
477 (:ref:`standard formats<struct-standard-formats>`). Generally speaking, the format strings
478 constructed for these two domains are distinct.
479
480
481 .. _struct-native-formats:
482
483 Native Formats
484 ^^^^^^^^^^^^^^
485
486 When constructing format strings which mimic native layouts, the
487 compiler and machine architecture determine byte ordering and padding.
488 In such cases, the ``@`` format character should be used to specify
489 native byte ordering and data sizes. Internal pad bytes are normally inserted
490 automatically. It is possible that a zero-repeat format code will be
491 needed at the end of a format string to round up to the correct
492 byte boundary for proper alignment of consecutive chunks of data.
493
494 Consider these two simple examples (on a 64-bit, little-endian
495 machine)::
496
497 >>> calcsize('@lhl')
498 24
499 >>> calcsize('@llh')
500 18
501
502 Data is not padded to an 8-byte boundary at the end of the second
503 format string without the use of extra padding. A zero-repeat format
504 code solves that problem::
505
506 >>> calcsize('@llh0l')
507 24
508
509 The ``'x'`` format code can be used to specify the repeat, but for
510 native formats it is better to use a zero-repeat format like ``'0l'``.
511
512 By default, native byte ordering and alignment is used, but it is
513 better to be explicit and use the ``'@'`` prefix character.
514
515
516 .. _struct-standard-formats:
517
518 Standard Formats
519 ^^^^^^^^^^^^^^^^
520
521 When exchanging data beyond your process such as networking or storage,
522 be precise. Specify the exact byte order, size, and alignment. Do
523 not assume they match the native order of a particular machine.
524 For example, network byte order is big-endian, while many popular CPUs
525 are little-endian. By defining this explicitly, the user need not
526 care about the specifics of the platform their code is running on.
527 The first character should typically be ``<`` or ``>``
528 (or ``!``). Padding is the responsibility of the programmer. The
529 zero-repeat format character won't work. Instead, the user must
530 explicitly add ``'x'`` pad bytes where needed. Revisiting the
531 examples from the previous section, we have::
532
533 >>> calcsize('<qh6xq')
534 24
535 >>> pack('<qh6xq', 1, 2, 3) == pack('@lhl', 1, 2, 3)
536 True
537 >>> calcsize('@llh')
538 18
539 >>> pack('@llh', 1, 2, 3) == pack('<qqh', 1, 2, 3)
540 True
541 >>> calcsize('<qqh6x')
542 24
543 >>> calcsize('@llh0l')
544 24
545 >>> pack('@llh0l', 1, 2, 3) == pack('<qqh6x', 1, 2, 3)
546 True
547
548 The above results (executed on a 64-bit machine) aren't guaranteed to
549 match when executed on different machines. For example, the examples
550 below were executed on a 32-bit machine::
551
552 >>> calcsize('<qqh6x')
553 24
554 >>> calcsize('@llh0l')
555 12
556 >>> pack('@llh0l', 1, 2, 3) == pack('<qqh6x', 1, 2, 3)
557 False
558
559
560 .. _struct-objects:
561
562 Classes
563 -------
564
565 The :mod:`struct` module also defines the following type:
566
567
568 .. class:: Struct(format)
569
570 Return a new Struct object which writes and reads binary data according to
571 the format string *format*. Creating a ``Struct`` object once and calling its
572 methods is more efficient than calling module-level functions with the
573 same format since the format string is only compiled once.
574
575 .. note::
576
577 The compiled versions of the most recent format strings passed to
578 the module-level functions are cached, so programs that use only a few
579 format strings needn't worry about reusing a single :class:`Struct`
580 instance.
581
582 Compiled Struct objects support the following methods and attributes:
583
584 .. method:: pack(v1, v2, ...)
585
586 Identical to the :func:`pack` function, using the compiled format.
587 (``len(result)`` will equal :attr:`size`.)
588
589
590 .. method:: pack_into(buffer, offset, v1, v2, ...)
591
592 Identical to the :func:`pack_into` function, using the compiled format.
593
594
595 .. method:: unpack(buffer)
596
597 Identical to the :func:`unpack` function, using the compiled format.
598 The buffer's size in bytes must equal :attr:`size`.
599
600
601 .. method:: unpack_from(buffer, offset=0)
602
603 Identical to the :func:`unpack_from` function, using the compiled format.
604 The buffer's size in bytes, starting at position *offset*, must be at least
605 :attr:`size`.
606
607
608 .. method:: iter_unpack(buffer)
609
610 Identical to the :func:`iter_unpack` function, using the compiled format.
611 The buffer's size in bytes must be a multiple of :attr:`size`.
612
613 .. versionadded:: 3.4
614
615 .. attribute:: format
616
617 The format string used to construct this Struct object.
618
619 .. versionchanged:: 3.7
620 The format string type is now :class:`str` instead of :class:`bytes`.
621
622 .. attribute:: size
623
624 The calculated size of the struct (and hence of the bytes object produced
625 by the :meth:`pack` method) corresponding to :attr:`format`.
626
627 .. versionchanged:: 3.13 The *repr()* of structs has changed. It
628 is now:
629
630 >>> Struct('i')
631 Struct('i')
632
633 .. _half precision format: https://en.wikipedia.org/wiki/Half-precision_floating-point_format
634
635 .. _ieee 754 standard: https://en.wikipedia.org/wiki/IEEE_754-2008_revision
636
637 .. _IETF RFC 1700: https://datatracker.ietf.org/doc/html/rfc1700