Debian Codegraph @

Doc/library/pickle.rst

raw
1 :mod:`!pickle` --- Python object serialization
2 ==============================================
3
4 .. module:: pickle
5 :synopsis: Convert Python objects to streams of bytes and back.
6
7 .. sectionauthor:: Jim Kerr <[email protected]>.
8 .. sectionauthor:: Barry Warsaw <[email protected]>
9
10 **Source code:** :source:`Lib/pickle.py`
11
12 .. index::
13 single: persistence
14 pair: persistent; objects
15 pair: serializing; objects
16 pair: marshalling; objects
17 pair: flattening; objects
18 pair: pickling; objects
19
20 --------------
21
22 The :mod:`!pickle` module implements binary protocols for serializing and
23 de-serializing a Python object structure. *"Pickling"* is the process
24 whereby a Python object hierarchy is converted into a byte stream, and
25 *"unpickling"* is the inverse operation, whereby a byte stream
26 (from a :term:`binary file` or :term:`bytes-like object`) is converted
27 back into an object hierarchy. Pickling (and unpickling) is alternatively
28 known as "serialization", "marshalling," [#]_ or "flattening"; however, to
29 avoid confusion, the terms used here are "pickling" and "unpickling".
30
31 .. warning::
32
33 The ``pickle`` module **is not secure**. Only unpickle data you trust.
34
35 It is possible to construct malicious pickle data which will **execute
36 arbitrary code during unpickling**. Never unpickle data that could have come
37 from an untrusted source, or that could have been tampered with.
38
39 Consider signing data with :mod:`hmac` if you need to ensure that it has not
40 been tampered with.
41
42 Safer serialization formats such as :mod:`json` may be more appropriate if
43 you are processing untrusted data. See :ref:`comparison-with-json`.
44
45
46 Relationship to other Python modules
47 ------------------------------------
48
49 Comparison with ``marshal``
50 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
51
52 Python has a more primitive serialization module called :mod:`marshal`, but in
53 general :mod:`!pickle` should always be the preferred way to serialize Python
54 objects. :mod:`marshal` exists primarily to support Python's :file:`.pyc`
55 files.
56
57 The :mod:`!pickle` module differs from :mod:`marshal` in several significant ways:
58
59 * :mod:`marshal` cannot be used to serialize user-defined classes and their
60 instances. :mod:`!pickle` can save and restore class instances transparently,
61 however the class definition must be importable and live in the same module as
62 when the object was pickled.
63
64 * The :mod:`marshal` serialization format is not guaranteed to be portable
65 across Python versions. Because its primary job in life is to support
66 :file:`.pyc` files, the Python implementers reserve the right to change the
67 serialization format in non-backwards compatible ways should the need arise.
68 The :mod:`!pickle` serialization format is guaranteed to be backwards compatible
69 across Python releases provided a compatible pickle protocol is chosen and
70 pickling and unpickling code deals with Python 2 to Python 3 type differences
71 if your data is crossing that unique breaking change language boundary.
72
73
74 .. _comparison-with-json:
75
76 Comparison with ``json``
77 ^^^^^^^^^^^^^^^^^^^^^^^^
78
79 There are fundamental differences between the pickle protocols and
80 `JSON (JavaScript Object Notation) <https://json.org>`_:
81
82 * JSON is a text serialization format (it outputs unicode text, although
83 most of the time it is then encoded to ``utf-8``), while pickle is
84 a binary serialization format;
85
86 * JSON is human-readable, while pickle is not;
87
88 * JSON is interoperable and widely used outside of the Python ecosystem,
89 while pickle is Python-specific;
90
91 * JSON, by default, can only represent a subset of the Python built-in
92 types, and no custom classes; pickle can represent an extremely large
93 number of Python types (many of them automatically, by clever usage
94 of Python's introspection facilities; complex cases can be tackled by
95 implementing :ref:`specific object APIs <pickle-inst>`);
96
97 * Unlike pickle, deserializing untrusted JSON does not in itself create an
98 arbitrary code execution vulnerability.
99
100 .. seealso::
101 The :mod:`json` module: a standard library module allowing JSON
102 serialization and deserialization.
103
104
105 .. _pickle-protocols:
106
107 Data stream format
108 ------------------
109
110 .. index::
111 single: External Data Representation
112
113 The data format used by :mod:`!pickle` is Python-specific. This has the
114 advantage that there are no restrictions imposed by external standards such as
115 JSON (which can't represent pointer sharing); however it means that
116 non-Python programs may not be able to reconstruct pickled Python objects.
117
118 By default, the :mod:`!pickle` data format uses a relatively compact binary
119 representation. If you need optimal size characteristics, you can efficiently
120 :doc:`compress <archiving>` pickled data.
121
122 The module :mod:`pickletools` contains tools for analyzing data streams
123 generated by :mod:`!pickle`. :mod:`pickletools` source code has extensive
124 comments about opcodes used by pickle protocols.
125
126 There are currently 6 different protocols which can be used for pickling.
127 The higher the protocol used, the more recent the version of Python needed
128 to read the pickle produced.
129
130 * Protocol version 0 is the original "human-readable" protocol and is
131 backwards compatible with earlier versions of Python.
132
133 * Protocol version 1 is an old binary format which is also compatible with
134 earlier versions of Python.
135
136 * Protocol version 2 was introduced in Python 2.3. It provides much more
137 efficient pickling of :term:`new-style classes <new-style class>`. Refer to :pep:`307` for
138 information about improvements brought by protocol 2.
139
140 * Protocol version 3 was added in Python 3.0. It has explicit support for
141 :class:`bytes` objects and cannot be unpickled by Python 2.x. This was
142 the default protocol in Python 3.0--3.7.
143
144 * Protocol version 4 was added in Python 3.4. It adds support for very large
145 objects, pickling more kinds of objects, and some data format
146 optimizations. This was the default protocol in Python 3.8--3.13.
147 Refer to :pep:`3154` for information about improvements brought by
148 protocol 4.
149
150 * Protocol version 5 was added in Python 3.8. It adds support for out-of-band
151 data and speedup for in-band data. It is the default protocol starting with
152 Python 3.14. Refer to :pep:`574` for information about improvements brought
153 by protocol 5.
154
155 .. note::
156 Serialization is a more primitive notion than persistence; although
157 :mod:`!pickle` reads and writes file objects, it does not handle the issue of
158 naming persistent objects, nor the (even more complicated) issue of concurrent
159 access to persistent objects. The :mod:`!pickle` module can transform a complex
160 object into a byte stream and it can transform the byte stream into an object
161 with the same internal structure. Perhaps the most obvious thing to do with
162 these byte streams is to write them onto a file, but it is also conceivable to
163 send them across a network or store them in a database. The :mod:`shelve`
164 module provides a simple interface to pickle and unpickle objects on
165 DBM-style database files.
166
167
168 Module Interface
169 ----------------
170
171 To serialize an object hierarchy, you simply call the :func:`dumps` function.
172 Similarly, to de-serialize a data stream, you call the :func:`loads` function.
173 However, if you want more control over serialization and de-serialization,
174 you can create a :class:`Pickler` or an :class:`Unpickler` object, respectively.
175
176 The :mod:`!pickle` module provides the following constants:
177
178
179 .. data:: HIGHEST_PROTOCOL
180
181 An integer, the highest :ref:`protocol version <pickle-protocols>`
182 available. This value can be passed as a *protocol* value to functions
183 :func:`dump` and :func:`dumps` as well as the :class:`Pickler`
184 constructor.
185
186 .. data:: DEFAULT_PROTOCOL
187
188 An integer, the default :ref:`protocol version <pickle-protocols>` used
189 for pickling. May be less than :data:`HIGHEST_PROTOCOL`. Currently the
190 default protocol is 5, introduced in Python 3.8 and incompatible
191 with previous versions. This version introduces support for out-of-band
192 buffers, where :pep:`3118`-compatible data can be transmitted separately
193 from the main pickle stream.
194
195 .. versionchanged:: 3.0
196
197 The default protocol is 3.
198
199 .. versionchanged:: 3.8
200
201 The default protocol is 4.
202
203 .. versionchanged:: 3.14
204
205 The default protocol is 5.
206
207 The :mod:`!pickle` module provides the following functions to make the pickling
208 process more convenient:
209
210 .. function:: dump(obj, file, protocol=None, *, fix_imports=True, buffer_callback=None)
211
212 Write the pickled representation of the object *obj* to the open
213 :term:`file object` *file*. This is equivalent to
214 ``Pickler(file, protocol).dump(obj)``.
215
216 Arguments *file*, *protocol*, *fix_imports* and *buffer_callback* have
217 the same meaning as in the :class:`Pickler` constructor.
218
219 .. versionchanged:: 3.8
220 The *buffer_callback* argument was added.
221
222 .. function:: dumps(obj, protocol=None, *, fix_imports=True, buffer_callback=None)
223
224 Return the pickled representation of the object *obj* as a :class:`bytes` object,
225 instead of writing it to a file.
226
227 Arguments *protocol*, *fix_imports* and *buffer_callback* have the same
228 meaning as in the :class:`Pickler` constructor.
229
230 .. versionchanged:: 3.8
231 The *buffer_callback* argument was added.
232
233 .. function:: load(file, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None)
234
235 Read the pickled representation of an object from the open :term:`file object`
236 *file* and return the reconstituted object hierarchy specified therein.
237 This is equivalent to ``Unpickler(file).load()``.
238
239 The protocol version of the pickle is detected automatically, so no
240 protocol argument is needed. Bytes past the pickled representation
241 of the object are ignored.
242
243 Arguments *file*, *fix_imports*, *encoding*, *errors*, *strict* and *buffers*
244 have the same meaning as in the :class:`Unpickler` constructor.
245
246 .. versionchanged:: 3.8
247 The *buffers* argument was added.
248
249 .. function:: loads(data, /, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None)
250
251 Return the reconstituted object hierarchy of the pickled representation
252 *data* of an object. *data* must be a :term:`bytes-like object`.
253
254 The protocol version of the pickle is detected automatically, so no
255 protocol argument is needed. Bytes past the pickled representation
256 of the object are ignored.
257
258 Arguments *fix_imports*, *encoding*, *errors*, *strict* and *buffers*
259 have the same meaning as in the :class:`Unpickler` constructor.
260
261 .. versionchanged:: 3.8
262 The *buffers* argument was added.
263
264
265 The :mod:`!pickle` module defines three exceptions:
266
267 .. exception:: PickleError
268
269 Common base class for the other pickling exceptions. It inherits from
270 :exc:`Exception`.
271
272 .. exception:: PicklingError
273
274 Error raised when an unpicklable object is encountered by :class:`Pickler`.
275 It inherits from :exc:`PickleError`.
276
277 Refer to :ref:`pickle-picklable` to learn what kinds of objects can be
278 pickled.
279
280 .. exception:: UnpicklingError
281
282 Error raised when there is a problem unpickling an object, such as a data
283 corruption or a security violation. It inherits from :exc:`PickleError`.
284
285 Note that other exceptions may also be raised during unpickling, including
286 (but not necessarily limited to) AttributeError, EOFError, ImportError, and
287 IndexError.
288
289
290 The :mod:`!pickle` module exports three classes, :class:`Pickler`,
291 :class:`Unpickler` and :class:`PickleBuffer`:
292
293 .. class:: Pickler(file, protocol=None, *, fix_imports=True, buffer_callback=None)
294
295 This takes a binary file for writing a pickle data stream.
296
297 The optional *protocol* argument, an integer, tells the pickler to use
298 the given protocol; supported protocols are 0 to :data:`HIGHEST_PROTOCOL`.
299 If not specified, the default is :data:`DEFAULT_PROTOCOL`. If a negative
300 number is specified, :data:`HIGHEST_PROTOCOL` is selected.
301
302 The *file* argument must have a write() method that accepts a single bytes
303 argument. It can thus be an on-disk file opened for binary writing, an
304 :class:`io.BytesIO` instance, or any other custom object that meets this
305 interface.
306
307 If *fix_imports* is true and *protocol* is less than 3, pickle will try to
308 map the new Python 3 names to the old module names used in Python 2, so
309 that the pickle data stream is readable with Python 2.
310
311 If *buffer_callback* is ``None`` (the default), buffer views are
312 serialized into *file* as part of the pickle stream.
313
314 If *buffer_callback* is not ``None``, then it can be called any number
315 of times with a buffer view. If the callback returns a false value
316 (such as ``None``), the given buffer is :ref:`out-of-band <pickle-oob>`;
317 otherwise the buffer is serialized in-band, i.e. inside the pickle stream.
318
319 It is an error if *buffer_callback* is not ``None`` and *protocol* is
320 ``None`` or smaller than 5.
321
322 .. versionchanged:: 3.8
323 The *buffer_callback* argument was added.
324
325 .. method:: dump(obj)
326
327 Write the pickled representation of *obj* to the open file object given in
328 the constructor.
329
330 .. method:: persistent_id(obj)
331
332 Do nothing by default. This exists so a subclass can override it.
333
334 If :meth:`persistent_id` returns ``None``, *obj* is pickled as usual. Any
335 other value causes :class:`Pickler` to emit the returned value as a
336 persistent ID for *obj*. The meaning of this persistent ID should be
337 defined by :meth:`Unpickler.persistent_load`. Note that the value
338 returned by :meth:`persistent_id` cannot itself have a persistent ID.
339
340 See :ref:`pickle-persistent` for details and examples of uses.
341
342 .. versionchanged:: 3.13
343 Add the default implementation of this method in the C implementation
344 of :class:`!Pickler`.
345
346 .. attribute:: dispatch_table
347
348 A pickler object's dispatch table is a registry of *reduction
349 functions* of the kind which can be declared using
350 :func:`copyreg.pickle`. It is a mapping whose keys are classes
351 and whose values are reduction functions. A reduction function
352 takes a single argument of the associated class and should
353 conform to the same interface as a :meth:`~object.__reduce__`
354 method.
355
356 By default, a pickler object will not have a
357 :attr:`dispatch_table` attribute, and it will instead use the
358 global dispatch table managed by the :mod:`copyreg` module.
359 However, to customize the pickling for a specific pickler object
360 one can set the :attr:`dispatch_table` attribute to a dict-like
361 object. Alternatively, if a subclass of :class:`Pickler` has a
362 :attr:`dispatch_table` attribute then this will be used as the
363 default dispatch table for instances of that class.
364
365 See :ref:`pickle-dispatch` for usage examples.
366
367 .. versionadded:: 3.3
368
369 .. method:: reducer_override(obj)
370
371 Special reducer that can be defined in :class:`Pickler` subclasses. This
372 method has priority over any reducer in the :attr:`dispatch_table`. It
373 should conform to the same interface as a :meth:`~object.__reduce__` method, and
374 can optionally return :data:`NotImplemented` to fallback on
375 :attr:`dispatch_table`-registered reducers to pickle ``obj``.
376
377 For a detailed example, see :ref:`reducer_override`.
378
379 .. versionadded:: 3.8
380
381 .. attribute:: fast
382
383 Deprecated. Enable fast mode if set to a true value. The fast mode
384 disables the usage of memo, therefore speeding the pickling process by not
385 generating superfluous PUT opcodes. It should not be used with
386 self-referential objects, doing otherwise will cause :class:`Pickler` to
387 recurse infinitely.
388
389 Use :func:`pickletools.optimize` if you need more compact pickles.
390
391 .. method:: clear_memo()
392
393 Clears the pickler's "memo".
394
395 The memo is the data structure that remembers which objects the
396 pickler has already seen, so that shared or recursive objects
397 are pickled by reference and not by value. This method is
398 useful when re-using picklers.
399
400
401 .. class:: Unpickler(file, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None)
402
403 This takes a binary file for reading a pickle data stream.
404
405 The protocol version of the pickle is detected automatically, so no
406 protocol argument is needed.
407
408 The argument *file* must have three methods, a read() method that takes an
409 integer argument, a readinto() method that takes a buffer argument
410 and a readline() method that requires no arguments, as in the
411 :class:`io.BufferedIOBase` interface. Thus *file* can be an on-disk file
412 opened for binary reading, an :class:`io.BytesIO` object, or any other
413 custom object that meets this interface.
414
415 The optional arguments *fix_imports*, *encoding* and *errors* are used
416 to control compatibility support for pickle stream generated by Python 2.
417 If *fix_imports* is true, pickle will try to map the old Python 2 names
418 to the new names used in Python 3. The *encoding* and *errors* tell
419 pickle how to decode 8-bit string instances pickled by Python 2;
420 these default to 'ASCII' and 'strict', respectively. The *encoding* can
421 be 'bytes' to read these 8-bit string instances as bytes objects.
422 Using ``encoding='latin1'`` is required for unpickling NumPy arrays and
423 instances of :class:`~datetime.datetime`, :class:`~datetime.date` and
424 :class:`~datetime.time` pickled by Python 2.
425
426 If *buffers* is ``None`` (the default), then all data necessary for
427 deserialization must be contained in the pickle stream. This means
428 that the *buffer_callback* argument was ``None`` when a :class:`Pickler`
429 was instantiated (or when :func:`dump` or :func:`dumps` was called).
430
431 If *buffers* is not ``None``, it should be an iterable of buffer-enabled
432 objects that is consumed each time the pickle stream references
433 an :ref:`out-of-band <pickle-oob>` buffer view. Such buffers have been
434 given in order to the *buffer_callback* of a Pickler object.
435
436 .. versionchanged:: 3.8
437 The *buffers* argument was added.
438
439 .. method:: load()
440
441 Read the pickled representation of an object from the open file object
442 given in the constructor, and return the reconstituted object hierarchy
443 specified therein. Bytes past the pickled representation of the object
444 are ignored.
445
446 .. method:: persistent_load(pid)
447
448 Raise an :exc:`UnpicklingError` by default.
449
450 If defined, :meth:`persistent_load` should return the object specified by
451 the persistent ID *pid*. If an invalid persistent ID is encountered, an
452 :exc:`UnpicklingError` should be raised.
453
454 See :ref:`pickle-persistent` for details and examples of uses.
455
456 .. versionchanged:: 3.13
457 Add the default implementation of this method in the C implementation
458 of :class:`!Unpickler`.
459
460 .. method:: find_class(module, name)
461
462 Import *module* if necessary and return the object called *name* from it,
463 where the *module* and *name* arguments are :class:`str` objects. Note,
464 unlike its name suggests, :meth:`find_class` is also used for finding
465 functions.
466
467 Subclasses may override this to gain control over what type of objects and
468 how they can be loaded, potentially reducing security risks. Refer to
469 :ref:`pickle-restrict` for details.
470
471 .. audit-event:: pickle.find_class module,name pickle.Unpickler.find_class
472
473 .. class:: PickleBuffer(buffer)
474
475 A wrapper for a buffer representing picklable data. *buffer* must be a
476 :ref:`buffer-providing <bufferobjects>` object, such as a
477 :term:`bytes-like object` or a N-dimensional array.
478
479 :class:`PickleBuffer` is itself a buffer provider, therefore it is
480 possible to pass it to other APIs expecting a buffer-providing object,
481 such as :class:`memoryview`.
482
483 :class:`PickleBuffer` objects can only be serialized using pickle
484 protocol 5 or higher. They are eligible for
485 :ref:`out-of-band serialization <pickle-oob>`.
486
487 .. versionadded:: 3.8
488
489 .. method:: raw()
490
491 Return a :class:`memoryview` of the memory area underlying this buffer.
492 The returned object is a one-dimensional, C-contiguous memoryview
493 with format ``B`` (unsigned bytes). :exc:`BufferError` is raised if
494 the buffer is neither C- nor Fortran-contiguous.
495
496 .. method:: release()
497
498 Release the underlying buffer exposed by the PickleBuffer object.
499
500
501 .. _pickle-picklable:
502
503 What can be pickled and unpickled?
504 ----------------------------------
505
506 The following types can be pickled:
507
508 * built-in constants (``None``, ``True``, ``False``, ``Ellipsis``, and
509 :data:`NotImplemented`);
510
511 * integers, floating-point numbers, complex numbers;
512
513 * strings, bytes, bytearrays;
514
515 * tuples, lists, sets, and dictionaries containing only picklable objects;
516
517 * functions (built-in and user-defined) accessible from the top level of a
518 module (using :keyword:`def`, not :keyword:`lambda`);
519
520 * classes accessible from the top level of a module;
521
522 * instances of such classes for which the result of calling :meth:`~object.__getstate__`
523 is picklable (see section :ref:`pickle-inst` for details).
524
525 Attempts to pickle unpicklable objects will raise the :exc:`PicklingError`
526 exception; when this happens, an unspecified number of bytes may have already
527 been written to the underlying file. Trying to pickle a highly recursive data
528 structure may exceed the maximum recursion depth, a :exc:`RecursionError` will be
529 raised in this case. You can carefully raise this limit with
530 :func:`sys.setrecursionlimit`.
531
532 Note that functions (built-in and user-defined) are pickled by fully
533 :term:`qualified name`, not by value. [#]_ This means that only the function name is
534 pickled, along with the name of the containing module and classes. Neither
535 the function's code, nor any of its function attributes are pickled. Thus the
536 defining module must be importable in the unpickling environment, and the module
537 must contain the named object, otherwise an exception will be raised. [#]_
538
539 Similarly, classes are pickled by fully qualified name, so the same restrictions in
540 the unpickling environment apply. Note that none of the class's code or data is
541 pickled, so in the following example the class attribute ``attr`` is not
542 restored in the unpickling environment::
543
544 class Foo:
545 attr = 'A class attribute'
546
547 picklestring = pickle.dumps(Foo)
548
549 These restrictions are why picklable functions and classes must be defined at
550 the top level of a module.
551
552 Similarly, when class instances are pickled, their class's code and data are not
553 pickled along with them. Only the instance data are pickled. This is done on
554 purpose, so you can fix bugs in a class or add methods to the class and still
555 load objects that were created with an earlier version of the class. If you
556 plan to have long-lived objects that will see many versions of a class, it may
557 be worthwhile to put a version number in the objects so that suitable
558 conversions can be made by the class's :meth:`~object.__setstate__` method.
559
560
561 .. _pickle-inst:
562
563 Pickling Class Instances
564 ------------------------
565
566 .. currentmodule:: None
567
568 In this section, we describe the general mechanisms available to you to define,
569 customize, and control how class instances are pickled and unpickled.
570
571 In most cases, no additional code is needed to make instances picklable. By
572 default, pickle will retrieve the class and the attributes of an instance via
573 introspection. When a class instance is unpickled, its :meth:`~object.__init__` method
574 is usually *not* invoked. The default behaviour first creates an uninitialized
575 instance and then restores the saved attributes. The following code shows an
576 implementation of this behaviour::
577
578 def save(obj):
579 return (obj.__class__, obj.__dict__)
580
581 def restore(cls, attributes):
582 obj = cls.__new__(cls)
583 obj.__dict__.update(attributes)
584 return obj
585
586 Classes can alter the default behaviour by providing one or several special
587 methods:
588
589 .. method:: object.__getnewargs_ex__()
590
591 In protocols 2 and newer, classes that implement the
592 :meth:`__getnewargs_ex__` method can dictate the values passed to the
593 :meth:`__new__` method upon unpickling. The method must return a pair
594 ``(args, kwargs)`` where *args* is a tuple of positional arguments
595 and *kwargs* a dictionary of named arguments for constructing the
596 object. Those will be passed to the :meth:`__new__` method upon
597 unpickling.
598
599 You should implement this method if the :meth:`__new__` method of your
600 class requires keyword-only arguments. Otherwise, it is recommended for
601 compatibility to implement :meth:`__getnewargs__`.
602
603 .. versionchanged:: 3.6
604 :meth:`__getnewargs_ex__` is now used in protocols 2 and 3.
605
606
607 .. method:: object.__getnewargs__()
608
609 This method serves a similar purpose as :meth:`__getnewargs_ex__`, but
610 supports only positional arguments. It must return a tuple of arguments
611 ``args`` which will be passed to the :meth:`__new__` method upon unpickling.
612
613 :meth:`__getnewargs__` will not be called if :meth:`__getnewargs_ex__` is
614 defined.
615
616 .. versionchanged:: 3.6
617 Before Python 3.6, :meth:`__getnewargs__` was called instead of
618 :meth:`__getnewargs_ex__` in protocols 2 and 3.
619
620
621 .. method:: object.__getstate__()
622
623 Classes can further influence how their instances are pickled by overriding
624 the method :meth:`__getstate__`. It is called and the returned object
625 is pickled as the contents for the instance, instead of a default state.
626 There are several cases:
627
628 * For a class that has no instance :attr:`~object.__dict__` and no
629 :attr:`~object.__slots__`, the default state is ``None``.
630
631 * For a class that has an instance :attr:`~object.__dict__` and no
632 :attr:`~object.__slots__`, the default state is ``self.__dict__``.
633
634 * For a class that has an instance :attr:`~object.__dict__` and
635 :attr:`~object.__slots__`, the default state is a tuple consisting of two
636 dictionaries: ``self.__dict__``, and a dictionary mapping slot
637 names to slot values. Only slots that have a value are
638 included in the latter.
639
640 * For a class that has :attr:`~object.__slots__` and no instance
641 :attr:`~object.__dict__`, the default state is a tuple whose first item
642 is ``None`` and whose second item is a dictionary mapping slot names
643 to slot values described in the previous bullet.
644
645 .. versionchanged:: 3.11
646 Added the default implementation of the ``__getstate__()`` method in the
647 :class:`object` class.
648
649
650 .. method:: object.__setstate__(state)
651
652 Upon unpickling, if the class defines :meth:`__setstate__`, it is called with
653 the unpickled state. In that case, there is no requirement for the state
654 object to be a dictionary. Otherwise, the pickled state must be a dictionary
655 and its items are assigned to the new instance's dictionary.
656
657 .. note::
658
659 If :meth:`__reduce__` returns a state with value ``None`` at pickling,
660 the :meth:`__setstate__` method will not be called upon unpickling.
661
662
663 Refer to the section :ref:`pickle-state` for more information about how to use
664 the methods :meth:`~object.__getstate__` and :meth:`~object.__setstate__`.
665
666 .. note::
667
668 At unpickling time, some methods like :meth:`~object.__getattr__`,
669 :meth:`~object.__getattribute__`, or :meth:`~object.__setattr__` may be called upon the
670 instance. In case those methods rely on some internal invariant being
671 true, the type should implement :meth:`~object.__new__` to establish such an
672 invariant, as :meth:`~object.__init__` is not called when unpickling an
673 instance.
674
675 .. index:: pair: copy; protocol
676
677 As we shall see, pickle does not use directly the methods described above. In
678 fact, these methods are part of the copy protocol which implements the
679 :meth:`~object.__reduce__` special method. The copy protocol provides a unified
680 interface for retrieving the data necessary for pickling and copying
681 objects. [#]_
682
683 Although powerful, implementing :meth:`~object.__reduce__` directly in your classes is
684 error prone. For this reason, class designers should use the high-level
685 interface (i.e., :meth:`~object.__getnewargs_ex__`, :meth:`~object.__getstate__` and
686 :meth:`~object.__setstate__`) whenever possible. We will show, however, cases where
687 using :meth:`!__reduce__` is the only option or leads to more efficient pickling
688 or both.
689
690 .. method:: object.__reduce__()
691
692 The interface is currently defined as follows. The :meth:`__reduce__` method
693 takes no argument and shall return either a string or preferably a tuple (the
694 returned object is often referred to as the "reduce value").
695
696 If a string is returned, the string should be interpreted as the name of a
697 global variable. It should be the object's local name relative to its
698 module; the pickle module searches the module namespace to determine the
699 object's module: for a given ``obj`` to be pickled, the ``__module__``
700 attribute is looked up on ``obj`` directly, which falls back to a lookup
701 on the type of ``obj`` if no ``__module__`` instance attribute is set.
702 This behaviour is typically useful for singletons.
703
704 When a tuple is returned, it must be between two and six items long.
705 Optional items can either be omitted, or ``None`` can be provided as their
706 value. The semantics of each item are in order:
707
708 .. XXX Mention __newobj__ special-case?
709
710 * A callable object that will be called to create the initial version of the
711 object.
712
713 * A tuple of arguments for the callable object. An empty tuple must be given
714 if the callable does not accept any argument.
715
716 * Optionally, the object's state, which will be passed to the object's
717 :meth:`__setstate__` method as previously described. If the object has no
718 such method then, the value must be a dictionary and it will be added to
719 the object's :attr:`~object.__dict__` attribute.
720
721 * Optionally, an iterator (and not a sequence) yielding successive items.
722 These items will be appended to the object either using
723 ``obj.append(item)`` or, in batch, using ``obj.extend(list_of_items)``.
724 This is primarily used for list subclasses, but may be used by other
725 classes as long as they have :meth:`~sequence.append`
726 and :meth:`~sequence.extend` methods with
727 the appropriate signature. (Whether :meth:`!append` or :meth:`!extend` is
728 used depends on which pickle protocol version is used as well as the number
729 of items to append, so both must be supported.)
730
731 * Optionally, an iterator (not a sequence) yielding successive key-value
732 pairs. These items will be stored to the object using ``obj[key] =
733 value``. This is primarily used for dictionary subclasses, but may be used
734 by other classes as long as they implement :meth:`__setitem__`.
735
736 * Optionally, a callable with a ``(obj, state)`` signature. This
737 callable allows the user to programmatically control the state-updating
738 behavior of a specific object, instead of using ``obj``'s static
739 :meth:`__setstate__` method. If not ``None``, this callable will have
740 priority over ``obj``'s :meth:`__setstate__`.
741
742 .. versionadded:: 3.8
743 The optional sixth tuple item, ``(obj, state)``, was added.
744
745
746 .. method:: object.__reduce_ex__(protocol)
747
748 Alternatively, a :meth:`__reduce_ex__` method may be defined. The only
749 difference is this method should take a single integer argument, the protocol
750 version. When defined, pickle will prefer it over the :meth:`__reduce__`
751 method. In addition, :meth:`__reduce__` automatically becomes a synonym for
752 the extended version. The main use for this method is to provide
753 backwards-compatible reduce values for older Python releases.
754
755 .. currentmodule:: pickle
756
757 .. _pickle-persistent:
758
759 Persistence of External Objects
760 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
761
762 .. index::
763 single: persistent_id (pickle protocol)
764 single: persistent_load (pickle protocol)
765
766 For the benefit of object persistence, the :mod:`!pickle` module supports the
767 notion of a reference to an object outside the pickled data stream. Such
768 objects are referenced by a persistent ID, which should be either a string of
769 alphanumeric characters (for protocol 0) [#]_ or just an arbitrary object (for
770 any newer protocol).
771
772 The resolution of such persistent IDs is not defined by the :mod:`!pickle`
773 module; it will delegate this resolution to the user-defined methods on the
774 pickler and unpickler, :meth:`~Pickler.persistent_id` and
775 :meth:`~Unpickler.persistent_load` respectively.
776
777 To pickle objects that have an external persistent ID, the pickler must have a
778 custom :meth:`~Pickler.persistent_id` method that takes an object as an
779 argument and returns either ``None`` or the persistent ID for that object.
780 When ``None`` is returned, the pickler simply pickles the object as normal.
781 When a persistent ID string is returned, the pickler will pickle that object,
782 along with a marker so that the unpickler will recognize it as a persistent ID.
783
784 To unpickle external objects, the unpickler must have a custom
785 :meth:`~Unpickler.persistent_load` method that takes a persistent ID object and
786 returns the referenced object.
787
788 Here is a comprehensive example presenting how persistent ID can be used to
789 pickle external objects by reference.
790
791 .. literalinclude:: ../includes/dbpickle.py
792
793 .. _pickle-dispatch:
794
795 Dispatch Tables
796 ^^^^^^^^^^^^^^^
797
798 If one wants to customize pickling of some classes without disturbing
799 any other code which depends on pickling, then one can create a
800 pickler with a private dispatch table.
801
802 The global dispatch table managed by the :mod:`copyreg` module is
803 available as :data:`!copyreg.dispatch_table`. Therefore, one may
804 choose to use a modified copy of :data:`!copyreg.dispatch_table` as a
805 private dispatch table.
806
807 For example ::
808
809 f = io.BytesIO()
810 p = pickle.Pickler(f)
811 p.dispatch_table = copyreg.dispatch_table.copy()
812 p.dispatch_table[SomeClass] = reduce_SomeClass
813
814 creates an instance of :class:`pickle.Pickler` with a private dispatch
815 table which handles the ``SomeClass`` class specially. Alternatively,
816 the code ::
817
818 class MyPickler(pickle.Pickler):
819 dispatch_table = copyreg.dispatch_table.copy()
820 dispatch_table[SomeClass] = reduce_SomeClass
821 f = io.BytesIO()
822 p = MyPickler(f)
823
824 does the same but all instances of ``MyPickler`` will by default
825 share the private dispatch table. On the other hand, the code ::
826
827 copyreg.pickle(SomeClass, reduce_SomeClass)
828 f = io.BytesIO()
829 p = pickle.Pickler(f)
830
831 modifies the global dispatch table shared by all users of the :mod:`copyreg` module.
832
833 .. _pickle-state:
834
835 Handling Stateful Objects
836 ^^^^^^^^^^^^^^^^^^^^^^^^^
837
838 .. index::
839 single: __getstate__() (copy protocol)
840 single: __setstate__() (copy protocol)
841
842 Here's an example that shows how to modify pickling behavior for a class.
843 The :class:`!TextReader` class below opens a text file, and returns the line number and
844 line contents each time its :meth:`!readline` method is called. If a
845 :class:`!TextReader` instance is pickled, all attributes *except* the file object
846 member are saved. When the instance is unpickled, the file is reopened, and
847 reading resumes from the last location. The :meth:`!__setstate__` and
848 :meth:`!__getstate__` methods are used to implement this behavior. ::
849
850 class TextReader:
851 """Print and number lines in a text file."""
852
853 def __init__(self, filename):
854 self.filename = filename
855 self.file = open(filename)
856 self.lineno = 0
857
858 def readline(self):
859 self.lineno += 1
860 line = self.file.readline()
861 if not line:
862 return None
863 if line.endswith('\n'):
864 line = line[:-1]
865 return "%i: %s" % (self.lineno, line)
866
867 def __getstate__(self):
868 # Copy the object's state from self.__dict__ which contains
869 # all our instance attributes. Always use the dict.copy()
870 # method to avoid modifying the original state.
871 state = self.__dict__.copy()
872 # Remove the unpicklable entries.
873 del state['file']
874 return state
875
876 def __setstate__(self, state):
877 # Restore instance attributes (i.e., filename and lineno).
878 self.__dict__.update(state)
879 # Restore the previously opened file's state. To do so, we need to
880 # reopen it and read from it until the line count is restored.
881 file = open(self.filename)
882 for _ in range(self.lineno):
883 file.readline()
884 # Finally, save the file.
885 self.file = file
886
887
888 A sample usage might be something like this::
889
890 >>> reader = TextReader("hello.txt")
891 >>> reader.readline()
892 '1: Hello world!'
893 >>> reader.readline()
894 '2: I am line number two.'
895 >>> new_reader = pickle.loads(pickle.dumps(reader))
896 >>> new_reader.readline()
897 '3: Goodbye!'
898
899 .. _reducer_override:
900
901 Custom Reduction for Types, Functions, and Other Objects
902 --------------------------------------------------------
903
904 .. versionadded:: 3.8
905
906 Sometimes, :attr:`~Pickler.dispatch_table` may not be flexible enough.
907 In particular we may want to customize pickling based on another criterion
908 than the object's type, or we may want to customize the pickling of
909 functions and classes.
910
911 For those cases, it is possible to subclass from the :class:`Pickler` class and
912 implement a :meth:`~Pickler.reducer_override` method. This method can return an
913 arbitrary reduction tuple (see :meth:`~object.__reduce__`). It can alternatively return
914 :data:`NotImplemented` to fallback to the traditional behavior.
915
916 If both the :attr:`~Pickler.dispatch_table` and
917 :meth:`~Pickler.reducer_override` are defined, then
918 :meth:`~Pickler.reducer_override` method takes priority.
919
920 .. Note::
921 For performance reasons, :meth:`~Pickler.reducer_override` may not be
922 called for the following objects: ``None``, ``True``, ``False``, and
923 exact instances of :class:`int`, :class:`float`, :class:`bytes`,
924 :class:`str`, :class:`dict`, :class:`set`, :class:`frozenset`, :class:`list`
925 and :class:`tuple`.
926
927 Here is a simple example where we allow pickling and reconstructing
928 a given class::
929
930 import io
931 import pickle
932
933 class MyClass:
934 my_attribute = 1
935
936 class MyPickler(pickle.Pickler):
937 def reducer_override(self, obj):
938 """Custom reducer for MyClass."""
939 if getattr(obj, "__name__", None) == "MyClass":
940 return type, (obj.__name__, obj.__bases__,
941 {'my_attribute': obj.my_attribute})
942 else:
943 # For any other object, fallback to usual reduction
944 return NotImplemented
945
946 f = io.BytesIO()
947 p = MyPickler(f)
948 p.dump(MyClass)
949
950 del MyClass
951
952 unpickled_class = pickle.loads(f.getvalue())
953
954 assert isinstance(unpickled_class, type)
955 assert unpickled_class.__name__ == "MyClass"
956 assert unpickled_class.my_attribute == 1
957
958
959 .. _pickle-oob:
960
961 Out-of-band Buffers
962 -------------------
963
964 .. versionadded:: 3.8
965
966 In some contexts, the :mod:`!pickle` module is used to transfer massive amounts
967 of data. Therefore, it can be important to minimize the number of memory
968 copies, to preserve performance and resource consumption. However, normal
969 operation of the :mod:`!pickle` module, as it transforms a graph-like structure
970 of objects into a sequential stream of bytes, intrinsically involves copying
971 data to and from the pickle stream.
972
973 This constraint can be eschewed if both the *provider* (the implementation
974 of the object types to be transferred) and the *consumer* (the implementation
975 of the communications system) support the out-of-band transfer facilities
976 provided by pickle protocol 5 and higher.
977
978 Provider API
979 ^^^^^^^^^^^^
980
981 The large data objects to be pickled must implement a :meth:`~object.__reduce_ex__`
982 method specialized for protocol 5 and higher, which returns a
983 :class:`PickleBuffer` instance (instead of e.g. a :class:`bytes` object)
984 for any large data.
985
986 A :class:`PickleBuffer` object *signals* that the underlying buffer is
987 eligible for out-of-band data transfer. Those objects remain compatible
988 with normal usage of the :mod:`!pickle` module. However, consumers can also
989 opt-in to tell :mod:`!pickle` that they will handle those buffers by
990 themselves.
991
992 Consumer API
993 ^^^^^^^^^^^^
994
995 A communications system can enable custom handling of the :class:`PickleBuffer`
996 objects generated when serializing an object graph.
997
998 On the sending side, it needs to pass a *buffer_callback* argument to
999 :class:`Pickler` (or to the :func:`dump` or :func:`dumps` function), which
1000 will be called with each :class:`PickleBuffer` generated while pickling
1001 the object graph. Buffers accumulated by the *buffer_callback* will not
1002 see their data copied into the pickle stream, only a cheap marker will be
1003 inserted.
1004
1005 On the receiving side, it needs to pass a *buffers* argument to
1006 :class:`Unpickler` (or to the :func:`load` or :func:`loads` function),
1007 which is an iterable of the buffers which were passed to *buffer_callback*.
1008 That iterable should produce buffers in the same order as they were passed
1009 to *buffer_callback*. Those buffers will provide the data expected by the
1010 reconstructors of the objects whose pickling produced the original
1011 :class:`PickleBuffer` objects.
1012
1013 Between the sending side and the receiving side, the communications system
1014 is free to implement its own transfer mechanism for out-of-band buffers.
1015 Potential optimizations include the use of shared memory or datatype-dependent
1016 compression.
1017
1018 Example
1019 ^^^^^^^
1020
1021 Here is a trivial example where we implement a :class:`bytearray` subclass
1022 able to participate in out-of-band buffer pickling::
1023
1024 class ZeroCopyByteArray(bytearray):
1025
1026 def __reduce_ex__(self, protocol):
1027 if protocol >= 5:
1028 return type(self)._reconstruct, (PickleBuffer(self),), None
1029 else:
1030 # PickleBuffer is forbidden with pickle protocols <= 4.
1031 return type(self)._reconstruct, (bytearray(self),)
1032
1033 @classmethod
1034 def _reconstruct(cls, obj):
1035 with memoryview(obj) as m:
1036 # Get a handle over the original buffer object
1037 obj = m.obj
1038 if type(obj) is cls:
1039 # Original buffer object is a ZeroCopyByteArray, return it
1040 # as-is.
1041 return obj
1042 else:
1043 return cls(obj)
1044
1045 The reconstructor (the ``_reconstruct`` class method) returns the buffer's
1046 providing object if it has the right type. This is an easy way to simulate
1047 zero-copy behaviour on this toy example.
1048
1049 On the consumer side, we can pickle those objects the usual way, which
1050 when unserialized will give us a copy of the original object::
1051
1052 b = ZeroCopyByteArray(b"abc")
1053 data = pickle.dumps(b, protocol=5)
1054 new_b = pickle.loads(data)
1055 print(b == new_b) # True
1056 print(b is new_b) # False: a copy was made
1057
1058 But if we pass a *buffer_callback* and then give back the accumulated
1059 buffers when unserializing, we are able to get back the original object::
1060
1061 b = ZeroCopyByteArray(b"abc")
1062 buffers = []
1063 data = pickle.dumps(b, protocol=5, buffer_callback=buffers.append)
1064 new_b = pickle.loads(data, buffers=buffers)
1065 print(b == new_b) # True
1066 print(b is new_b) # True: no copy was made
1067
1068 This example is limited by the fact that :class:`bytearray` allocates its
1069 own memory: you cannot create a :class:`bytearray` instance that is backed
1070 by another object's memory. However, third-party datatypes such as NumPy
1071 arrays do not have this limitation, and allow use of zero-copy pickling
1072 (or making as few copies as possible) when transferring between distinct
1073 processes or systems.
1074
1075 .. seealso:: :pep:`574` -- Pickle protocol 5 with out-of-band data
1076
1077
1078 .. _pickle-restrict:
1079
1080 Restricting Globals
1081 -------------------
1082
1083 .. index::
1084 single: find_class() (pickle protocol)
1085
1086 By default, unpickling will import any class or function that it finds in the
1087 pickle data. For many applications, this behaviour is unacceptable as it
1088 permits the unpickler to import and invoke arbitrary code. Just consider what
1089 this hand-crafted pickle data stream does when loaded::
1090
1091 >>> import pickle
1092 >>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
1093 hello world
1094 0
1095
1096 In this example, the unpickler imports the :func:`os.system` function and then
1097 apply the string argument "echo hello world". Although this example is
1098 inoffensive, it is not difficult to imagine one that could damage your system.
1099
1100 For this reason, you may want to control what gets unpickled by customizing
1101 :meth:`Unpickler.find_class`. Unlike its name suggests,
1102 :meth:`Unpickler.find_class` is called whenever a global (i.e., a class or
1103 a function) is requested. Thus it is possible to either completely forbid
1104 globals or restrict them to a safe subset.
1105
1106 Here is an example of an unpickler allowing only few safe classes from the
1107 :mod:`builtins` module to be loaded::
1108
1109 import builtins
1110 import io
1111 import pickle
1112
1113 safe_builtins = {
1114 'range',
1115 'complex',
1116 'set',
1117 'frozenset',
1118 'slice',
1119 }
1120
1121 class RestrictedUnpickler(pickle.Unpickler):
1122
1123 def find_class(self, module, name):
1124 # Only allow safe classes from builtins.
1125 if module == "builtins" and name in safe_builtins:
1126 return getattr(builtins, name)
1127 # Forbid everything else.
1128 raise pickle.UnpicklingError("global '%s.%s' is forbidden" %
1129 (module, name))
1130
1131 def restricted_loads(s):
1132 """Helper function analogous to pickle.loads()."""
1133 return RestrictedUnpickler(io.BytesIO(s)).load()
1134
1135 A sample usage of our unpickler working as intended::
1136
1137 >>> restricted_loads(pickle.dumps([1, 2, range(15)]))
1138 [1, 2, range(0, 15)]
1139 >>> restricted_loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
1140 Traceback (most recent call last):
1141 ...
1142 pickle.UnpicklingError: global 'os.system' is forbidden
1143 >>> restricted_loads(b'cbuiltins\neval\n'
1144 ... b'(S\'getattr(__import__("os"), "system")'
1145 ... b'("echo hello world")\'\ntR.')
1146 Traceback (most recent call last):
1147 ...
1148 pickle.UnpicklingError: global 'builtins.eval' is forbidden
1149
1150
1151 .. XXX Add note about how extension codes could evade our protection
1152 mechanism (e.g. cached classes do not invokes find_class()).
1153
1154 As our examples shows, you have to be careful with what you allow to be
1155 unpickled. Therefore if security is a concern, you may want to consider
1156 alternatives such as the marshalling API in :mod:`xmlrpc.client` or
1157 third-party solutions.
1158
1159
1160 Performance
1161 -----------
1162
1163 Recent versions of the pickle protocol (from protocol 2 and upwards) feature
1164 efficient binary encodings for several common features and built-in types.
1165 Also, the :mod:`!pickle` module has a transparent optimizer written in C.
1166
1167
1168 .. _pickle-example:
1169
1170 Examples
1171 --------
1172
1173 For the simplest code, use the :func:`dump` and :func:`load` functions. ::
1174
1175 import pickle
1176
1177 # An arbitrary collection of objects supported by pickle.
1178 data = {
1179 'a': [1, 2.0, 3+4j],
1180 'b': ("character string", b"byte string"),
1181 'c': {None, True, False}
1182 }
1183
1184 with open('data.pickle', 'wb') as f:
1185 # Pickle the 'data' dictionary using the highest protocol available.
1186 pickle.dump(data, f, pickle.HIGHEST_PROTOCOL)
1187
1188
1189 The following example reads the resulting pickled data. ::
1190
1191 import pickle
1192
1193 with open('data.pickle', 'rb') as f:
1194 # The protocol version used is detected automatically, so we do not
1195 # have to specify it.
1196 data = pickle.load(f)
1197
1198
1199 .. XXX: Add examples showing how to optimize pickles for size (like using
1200 .. pickletools.optimize() or the gzip module).
1201
1202
1203 .. _pickle-cli:
1204
1205 Command-line interface
1206 ----------------------
1207
1208 The :mod:`!pickle` module can be invoked as a script from the command line,
1209 it will display contents of the pickle files. However, when the pickle file
1210 that you want to examine comes from an untrusted source, ``-m pickletools``
1211 is a safer option because it does not execute pickle bytecode, see
1212 :ref:`pickletools CLI usage <pickletools-cli>`.
1213
1214 .. code-block:: bash
1215
1216 python -m pickle pickle_file [pickle_file ...]
1217
1218 The following option is accepted:
1219
1220 .. program:: pickle
1221
1222 .. option:: pickle_file
1223
1224 A pickle file to read, or ``-`` to indicate reading from standard input.
1225
1226
1227 .. seealso::
1228
1229 Module :mod:`copyreg`
1230 Pickle interface constructor registration for extension types.
1231
1232 Module :mod:`pickletools`
1233 Tools for working with and analyzing pickled data.
1234
1235 Module :mod:`shelve`
1236 Indexed databases of objects; uses :mod:`!pickle`.
1237
1238 Module :mod:`copy`
1239 Shallow and deep object copying.
1240
1241 Module :mod:`marshal`
1242 High-performance serialization of built-in types.
1243
1244
1245 .. rubric:: Footnotes
1246
1247 .. [#] Don't confuse this with the :mod:`marshal` module
1248
1249 .. [#] This is why :keyword:`lambda` functions cannot be pickled: all
1250 :keyword:`!lambda` functions share the same name: ``<lambda>``.
1251
1252 .. [#] The exception raised will likely be an :exc:`ImportError` or an
1253 :exc:`AttributeError` but it could be something else.
1254
1255 .. [#] The :mod:`copy` module uses this protocol for shallow and deep copying
1256 operations.
1257
1258 .. [#] The limitation on alphanumeric characters is due to the fact
1259 that persistent IDs in protocol 0 are delimited by the newline
1260 character. Therefore if any kind of newline characters occurs in
1261 persistent IDs, the resulting pickled data will become unreadable.