Debian Codegraph @

Doc/library/pickle.rst

raw
1 :mod:`!pickle` --- Python object serialization
2 ==============================================
3
4 .. module:: pickle
5 :synopsis: Convert Python objects to streams of bytes and back.
6
7 .. sectionauthor:: Jim Kerr <[email protected]>.
8 .. sectionauthor:: Barry Warsaw <[email protected]>
9
10 **Source code:** :source:`Lib/pickle.py`
11
12 .. index::
13 single: persistence
14 pair: persistent; objects
15 pair: serializing; objects
16 pair: marshalling; objects
17 pair: flattening; objects
18 pair: pickling; objects
19
20 --------------
21
22 The :mod:`pickle` module implements binary protocols for serializing and
23 de-serializing a Python object structure. *"Pickling"* is the process
24 whereby a Python object hierarchy is converted into a byte stream, and
25 *"unpickling"* is the inverse operation, whereby a byte stream
26 (from a :term:`binary file` or :term:`bytes-like object`) is converted
27 back into an object hierarchy. Pickling (and unpickling) is alternatively
28 known as "serialization", "marshalling," [#]_ or "flattening"; however, to
29 avoid confusion, the terms used here are "pickling" and "unpickling".
30
31 .. warning::
32
33 The ``pickle`` module **is not secure**. Only unpickle data you trust.
34
35 It is possible to construct malicious pickle data which will **execute
36 arbitrary code during unpickling**. Never unpickle data that could have come
37 from an untrusted source, or that could have been tampered with.
38
39 Consider signing data with :mod:`hmac` if you need to ensure that it has not
40 been tampered with.
41
42 Safer serialization formats such as :mod:`json` may be more appropriate if
43 you are processing untrusted data. See :ref:`comparison-with-json`.
44
45
46 Relationship to other Python modules
47 ------------------------------------
48
49 Comparison with ``marshal``
50 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
51
52 Python has a more primitive serialization module called :mod:`marshal`, but in
53 general :mod:`pickle` should always be the preferred way to serialize Python
54 objects. :mod:`marshal` exists primarily to support Python's :file:`.pyc`
55 files.
56
57 The :mod:`pickle` module differs from :mod:`marshal` in several significant ways:
58
59 * :mod:`marshal` cannot be used to serialize user-defined classes and their
60 instances. :mod:`pickle` can save and restore class instances transparently,
61 however the class definition must be importable and live in the same module as
62 when the object was pickled.
63
64 * The :mod:`marshal` serialization format is not guaranteed to be portable
65 across Python versions. Because its primary job in life is to support
66 :file:`.pyc` files, the Python implementers reserve the right to change the
67 serialization format in non-backwards compatible ways should the need arise.
68 The :mod:`pickle` serialization format is guaranteed to be backwards compatible
69 across Python releases provided a compatible pickle protocol is chosen and
70 pickling and unpickling code deals with Python 2 to Python 3 type differences
71 if your data is crossing that unique breaking change language boundary.
72
73
74 .. _comparison-with-json:
75
76 Comparison with ``json``
77 ^^^^^^^^^^^^^^^^^^^^^^^^
78
79 There are fundamental differences between the pickle protocols and
80 `JSON (JavaScript Object Notation) <https://json.org>`_:
81
82 * JSON is a text serialization format (it outputs unicode text, although
83 most of the time it is then encoded to ``utf-8``), while pickle is
84 a binary serialization format;
85
86 * JSON is human-readable, while pickle is not;
87
88 * JSON is interoperable and widely used outside of the Python ecosystem,
89 while pickle is Python-specific;
90
91 * JSON, by default, can only represent a subset of the Python built-in
92 types, and no custom classes; pickle can represent an extremely large
93 number of Python types (many of them automatically, by clever usage
94 of Python's introspection facilities; complex cases can be tackled by
95 implementing :ref:`specific object APIs <pickle-inst>`);
96
97 * Unlike pickle, deserializing untrusted JSON does not in itself create an
98 arbitrary code execution vulnerability.
99
100 .. seealso::
101 The :mod:`json` module: a standard library module allowing JSON
102 serialization and deserialization.
103
104
105 .. _pickle-protocols:
106
107 Data stream format
108 ------------------
109
110 .. index::
111 single: External Data Representation
112
113 The data format used by :mod:`pickle` is Python-specific. This has the
114 advantage that there are no restrictions imposed by external standards such as
115 JSON (which can't represent pointer sharing); however it means that
116 non-Python programs may not be able to reconstruct pickled Python objects.
117
118 By default, the :mod:`pickle` data format uses a relatively compact binary
119 representation. If you need optimal size characteristics, you can efficiently
120 :doc:`compress <archiving>` pickled data.
121
122 The module :mod:`pickletools` contains tools for analyzing data streams
123 generated by :mod:`pickle`. :mod:`pickletools` source code has extensive
124 comments about opcodes used by pickle protocols.
125
126 There are currently 6 different protocols which can be used for pickling.
127 The higher the protocol used, the more recent the version of Python needed
128 to read the pickle produced.
129
130 * Protocol version 0 is the original "human-readable" protocol and is
131 backwards compatible with earlier versions of Python.
132
133 * Protocol version 1 is an old binary format which is also compatible with
134 earlier versions of Python.
135
136 * Protocol version 2 was introduced in Python 2.3. It provides much more
137 efficient pickling of :term:`new-style classes <new-style class>`. Refer to :pep:`307` for
138 information about improvements brought by protocol 2.
139
140 * Protocol version 3 was added in Python 3.0. It has explicit support for
141 :class:`bytes` objects and cannot be unpickled by Python 2.x. This was
142 the default protocol in Python 3.0--3.7.
143
144 * Protocol version 4 was added in Python 3.4. It adds support for very large
145 objects, pickling more kinds of objects, and some data format
146 optimizations. It is the default protocol starting with Python 3.8.
147 Refer to :pep:`3154` for information about improvements brought by
148 protocol 4.
149
150 * Protocol version 5 was added in Python 3.8. It adds support for out-of-band
151 data and speedup for in-band data. Refer to :pep:`574` for information about
152 improvements brought by protocol 5.
153
154 .. note::
155 Serialization is a more primitive notion than persistence; although
156 :mod:`pickle` reads and writes file objects, it does not handle the issue of
157 naming persistent objects, nor the (even more complicated) issue of concurrent
158 access to persistent objects. The :mod:`pickle` module can transform a complex
159 object into a byte stream and it can transform the byte stream into an object
160 with the same internal structure. Perhaps the most obvious thing to do with
161 these byte streams is to write them onto a file, but it is also conceivable to
162 send them across a network or store them in a database. The :mod:`shelve`
163 module provides a simple interface to pickle and unpickle objects on
164 DBM-style database files.
165
166
167 Module Interface
168 ----------------
169
170 To serialize an object hierarchy, you simply call the :func:`dumps` function.
171 Similarly, to de-serialize a data stream, you call the :func:`loads` function.
172 However, if you want more control over serialization and de-serialization,
173 you can create a :class:`Pickler` or an :class:`Unpickler` object, respectively.
174
175 The :mod:`pickle` module provides the following constants:
176
177
178 .. data:: HIGHEST_PROTOCOL
179
180 An integer, the highest :ref:`protocol version <pickle-protocols>`
181 available. This value can be passed as a *protocol* value to functions
182 :func:`dump` and :func:`dumps` as well as the :class:`Pickler`
183 constructor.
184
185 .. data:: DEFAULT_PROTOCOL
186
187 An integer, the default :ref:`protocol version <pickle-protocols>` used
188 for pickling. May be less than :data:`HIGHEST_PROTOCOL`. Currently the
189 default protocol is 4, first introduced in Python 3.4 and incompatible
190 with previous versions.
191
192 .. versionchanged:: 3.0
193
194 The default protocol is 3.
195
196 .. versionchanged:: 3.8
197
198 The default protocol is 4.
199
200 The :mod:`pickle` module provides the following functions to make the pickling
201 process more convenient:
202
203 .. function:: dump(obj, file, protocol=None, *, fix_imports=True, buffer_callback=None)
204
205 Write the pickled representation of the object *obj* to the open
206 :term:`file object` *file*. This is equivalent to
207 ``Pickler(file, protocol).dump(obj)``.
208
209 Arguments *file*, *protocol*, *fix_imports* and *buffer_callback* have
210 the same meaning as in the :class:`Pickler` constructor.
211
212 .. versionchanged:: 3.8
213 The *buffer_callback* argument was added.
214
215 .. function:: dumps(obj, protocol=None, *, fix_imports=True, buffer_callback=None)
216
217 Return the pickled representation of the object *obj* as a :class:`bytes` object,
218 instead of writing it to a file.
219
220 Arguments *protocol*, *fix_imports* and *buffer_callback* have the same
221 meaning as in the :class:`Pickler` constructor.
222
223 .. versionchanged:: 3.8
224 The *buffer_callback* argument was added.
225
226 .. function:: load(file, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None)
227
228 Read the pickled representation of an object from the open :term:`file object`
229 *file* and return the reconstituted object hierarchy specified therein.
230 This is equivalent to ``Unpickler(file).load()``.
231
232 The protocol version of the pickle is detected automatically, so no
233 protocol argument is needed. Bytes past the pickled representation
234 of the object are ignored.
235
236 Arguments *file*, *fix_imports*, *encoding*, *errors*, *strict* and *buffers*
237 have the same meaning as in the :class:`Unpickler` constructor.
238
239 .. versionchanged:: 3.8
240 The *buffers* argument was added.
241
242 .. function:: loads(data, /, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None)
243
244 Return the reconstituted object hierarchy of the pickled representation
245 *data* of an object. *data* must be a :term:`bytes-like object`.
246
247 The protocol version of the pickle is detected automatically, so no
248 protocol argument is needed. Bytes past the pickled representation
249 of the object are ignored.
250
251 Arguments *fix_imports*, *encoding*, *errors*, *strict* and *buffers*
252 have the same meaning as in the :class:`Unpickler` constructor.
253
254 .. versionchanged:: 3.8
255 The *buffers* argument was added.
256
257
258 The :mod:`pickle` module defines three exceptions:
259
260 .. exception:: PickleError
261
262 Common base class for the other pickling exceptions. It inherits from
263 :exc:`Exception`.
264
265 .. exception:: PicklingError
266
267 Error raised when an unpicklable object is encountered by :class:`Pickler`.
268 It inherits from :exc:`PickleError`.
269
270 Refer to :ref:`pickle-picklable` to learn what kinds of objects can be
271 pickled.
272
273 .. exception:: UnpicklingError
274
275 Error raised when there is a problem unpickling an object, such as a data
276 corruption or a security violation. It inherits from :exc:`PickleError`.
277
278 Note that other exceptions may also be raised during unpickling, including
279 (but not necessarily limited to) AttributeError, EOFError, ImportError, and
280 IndexError.
281
282
283 The :mod:`pickle` module exports three classes, :class:`Pickler`,
284 :class:`Unpickler` and :class:`PickleBuffer`:
285
286 .. class:: Pickler(file, protocol=None, *, fix_imports=True, buffer_callback=None)
287
288 This takes a binary file for writing a pickle data stream.
289
290 The optional *protocol* argument, an integer, tells the pickler to use
291 the given protocol; supported protocols are 0 to :data:`HIGHEST_PROTOCOL`.
292 If not specified, the default is :data:`DEFAULT_PROTOCOL`. If a negative
293 number is specified, :data:`HIGHEST_PROTOCOL` is selected.
294
295 The *file* argument must have a write() method that accepts a single bytes
296 argument. It can thus be an on-disk file opened for binary writing, an
297 :class:`io.BytesIO` instance, or any other custom object that meets this
298 interface.
299
300 If *fix_imports* is true and *protocol* is less than 3, pickle will try to
301 map the new Python 3 names to the old module names used in Python 2, so
302 that the pickle data stream is readable with Python 2.
303
304 If *buffer_callback* is ``None`` (the default), buffer views are
305 serialized into *file* as part of the pickle stream.
306
307 If *buffer_callback* is not ``None``, then it can be called any number
308 of times with a buffer view. If the callback returns a false value
309 (such as ``None``), the given buffer is :ref:`out-of-band <pickle-oob>`;
310 otherwise the buffer is serialized in-band, i.e. inside the pickle stream.
311
312 It is an error if *buffer_callback* is not ``None`` and *protocol* is
313 ``None`` or smaller than 5.
314
315 .. versionchanged:: 3.8
316 The *buffer_callback* argument was added.
317
318 .. method:: dump(obj)
319
320 Write the pickled representation of *obj* to the open file object given in
321 the constructor.
322
323 .. method:: persistent_id(obj)
324
325 Do nothing by default. This exists so a subclass can override it.
326
327 If :meth:`persistent_id` returns ``None``, *obj* is pickled as usual. Any
328 other value causes :class:`Pickler` to emit the returned value as a
329 persistent ID for *obj*. The meaning of this persistent ID should be
330 defined by :meth:`Unpickler.persistent_load`. Note that the value
331 returned by :meth:`persistent_id` cannot itself have a persistent ID.
332
333 See :ref:`pickle-persistent` for details and examples of uses.
334
335 .. versionchanged:: 3.13
336 Add the default implementation of this method in the C implementation
337 of :class:`!Pickler`.
338
339 .. attribute:: dispatch_table
340
341 A pickler object's dispatch table is a registry of *reduction
342 functions* of the kind which can be declared using
343 :func:`copyreg.pickle`. It is a mapping whose keys are classes
344 and whose values are reduction functions. A reduction function
345 takes a single argument of the associated class and should
346 conform to the same interface as a :meth:`~object.__reduce__`
347 method.
348
349 By default, a pickler object will not have a
350 :attr:`dispatch_table` attribute, and it will instead use the
351 global dispatch table managed by the :mod:`copyreg` module.
352 However, to customize the pickling for a specific pickler object
353 one can set the :attr:`dispatch_table` attribute to a dict-like
354 object. Alternatively, if a subclass of :class:`Pickler` has a
355 :attr:`dispatch_table` attribute then this will be used as the
356 default dispatch table for instances of that class.
357
358 See :ref:`pickle-dispatch` for usage examples.
359
360 .. versionadded:: 3.3
361
362 .. method:: reducer_override(obj)
363
364 Special reducer that can be defined in :class:`Pickler` subclasses. This
365 method has priority over any reducer in the :attr:`dispatch_table`. It
366 should conform to the same interface as a :meth:`~object.__reduce__` method, and
367 can optionally return :data:`NotImplemented` to fallback on
368 :attr:`dispatch_table`-registered reducers to pickle ``obj``.
369
370 For a detailed example, see :ref:`reducer_override`.
371
372 .. versionadded:: 3.8
373
374 .. attribute:: fast
375
376 Deprecated. Enable fast mode if set to a true value. The fast mode
377 disables the usage of memo, therefore speeding the pickling process by not
378 generating superfluous PUT opcodes. It should not be used with
379 self-referential objects, doing otherwise will cause :class:`Pickler` to
380 recurse infinitely.
381
382 Use :func:`pickletools.optimize` if you need more compact pickles.
383
384
385 .. class:: Unpickler(file, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None)
386
387 This takes a binary file for reading a pickle data stream.
388
389 The protocol version of the pickle is detected automatically, so no
390 protocol argument is needed.
391
392 The argument *file* must have three methods, a read() method that takes an
393 integer argument, a readinto() method that takes a buffer argument
394 and a readline() method that requires no arguments, as in the
395 :class:`io.BufferedIOBase` interface. Thus *file* can be an on-disk file
396 opened for binary reading, an :class:`io.BytesIO` object, or any other
397 custom object that meets this interface.
398
399 The optional arguments *fix_imports*, *encoding* and *errors* are used
400 to control compatibility support for pickle stream generated by Python 2.
401 If *fix_imports* is true, pickle will try to map the old Python 2 names
402 to the new names used in Python 3. The *encoding* and *errors* tell
403 pickle how to decode 8-bit string instances pickled by Python 2;
404 these default to 'ASCII' and 'strict', respectively. The *encoding* can
405 be 'bytes' to read these 8-bit string instances as bytes objects.
406 Using ``encoding='latin1'`` is required for unpickling NumPy arrays and
407 instances of :class:`~datetime.datetime`, :class:`~datetime.date` and
408 :class:`~datetime.time` pickled by Python 2.
409
410 If *buffers* is ``None`` (the default), then all data necessary for
411 deserialization must be contained in the pickle stream. This means
412 that the *buffer_callback* argument was ``None`` when a :class:`Pickler`
413 was instantiated (or when :func:`dump` or :func:`dumps` was called).
414
415 If *buffers* is not ``None``, it should be an iterable of buffer-enabled
416 objects that is consumed each time the pickle stream references
417 an :ref:`out-of-band <pickle-oob>` buffer view. Such buffers have been
418 given in order to the *buffer_callback* of a Pickler object.
419
420 .. versionchanged:: 3.8
421 The *buffers* argument was added.
422
423 .. method:: load()
424
425 Read the pickled representation of an object from the open file object
426 given in the constructor, and return the reconstituted object hierarchy
427 specified therein. Bytes past the pickled representation of the object
428 are ignored.
429
430 .. method:: persistent_load(pid)
431
432 Raise an :exc:`UnpicklingError` by default.
433
434 If defined, :meth:`persistent_load` should return the object specified by
435 the persistent ID *pid*. If an invalid persistent ID is encountered, an
436 :exc:`UnpicklingError` should be raised.
437
438 See :ref:`pickle-persistent` for details and examples of uses.
439
440 .. versionchanged:: 3.13
441 Add the default implementation of this method in the C implementation
442 of :class:`!Unpickler`.
443
444 .. method:: find_class(module, name)
445
446 Import *module* if necessary and return the object called *name* from it,
447 where the *module* and *name* arguments are :class:`str` objects. Note,
448 unlike its name suggests, :meth:`find_class` is also used for finding
449 functions.
450
451 Subclasses may override this to gain control over what type of objects and
452 how they can be loaded, potentially reducing security risks. Refer to
453 :ref:`pickle-restrict` for details.
454
455 .. audit-event:: pickle.find_class module,name pickle.Unpickler.find_class
456
457 .. class:: PickleBuffer(buffer)
458
459 A wrapper for a buffer representing picklable data. *buffer* must be a
460 :ref:`buffer-providing <bufferobjects>` object, such as a
461 :term:`bytes-like object` or a N-dimensional array.
462
463 :class:`PickleBuffer` is itself a buffer provider, therefore it is
464 possible to pass it to other APIs expecting a buffer-providing object,
465 such as :class:`memoryview`.
466
467 :class:`PickleBuffer` objects can only be serialized using pickle
468 protocol 5 or higher. They are eligible for
469 :ref:`out-of-band serialization <pickle-oob>`.
470
471 .. versionadded:: 3.8
472
473 .. method:: raw()
474
475 Return a :class:`memoryview` of the memory area underlying this buffer.
476 The returned object is a one-dimensional, C-contiguous memoryview
477 with format ``B`` (unsigned bytes). :exc:`BufferError` is raised if
478 the buffer is neither C- nor Fortran-contiguous.
479
480 .. method:: release()
481
482 Release the underlying buffer exposed by the PickleBuffer object.
483
484
485 .. _pickle-picklable:
486
487 What can be pickled and unpickled?
488 ----------------------------------
489
490 The following types can be pickled:
491
492 * built-in constants (``None``, ``True``, ``False``, ``Ellipsis``, and
493 :data:`NotImplemented`);
494
495 * integers, floating-point numbers, complex numbers;
496
497 * strings, bytes, bytearrays;
498
499 * tuples, lists, sets, and dictionaries containing only picklable objects;
500
501 * functions (built-in and user-defined) accessible from the top level of a
502 module (using :keyword:`def`, not :keyword:`lambda`);
503
504 * classes accessible from the top level of a module;
505
506 * instances of such classes for which the result of calling :meth:`~object.__getstate__`
507 is picklable (see section :ref:`pickle-inst` for details).
508
509 Attempts to pickle unpicklable objects will raise the :exc:`PicklingError`
510 exception; when this happens, an unspecified number of bytes may have already
511 been written to the underlying file. Trying to pickle a highly recursive data
512 structure may exceed the maximum recursion depth, a :exc:`RecursionError` will be
513 raised in this case. You can carefully raise this limit with
514 :func:`sys.setrecursionlimit`.
515
516 Note that functions (built-in and user-defined) are pickled by fully
517 :term:`qualified name`, not by value. [#]_ This means that only the function name is
518 pickled, along with the name of the containing module and classes. Neither
519 the function's code, nor any of its function attributes are pickled. Thus the
520 defining module must be importable in the unpickling environment, and the module
521 must contain the named object, otherwise an exception will be raised. [#]_
522
523 Similarly, classes are pickled by fully qualified name, so the same restrictions in
524 the unpickling environment apply. Note that none of the class's code or data is
525 pickled, so in the following example the class attribute ``attr`` is not
526 restored in the unpickling environment::
527
528 class Foo:
529 attr = 'A class attribute'
530
531 picklestring = pickle.dumps(Foo)
532
533 These restrictions are why picklable functions and classes must be defined at
534 the top level of a module.
535
536 Similarly, when class instances are pickled, their class's code and data are not
537 pickled along with them. Only the instance data are pickled. This is done on
538 purpose, so you can fix bugs in a class or add methods to the class and still
539 load objects that were created with an earlier version of the class. If you
540 plan to have long-lived objects that will see many versions of a class, it may
541 be worthwhile to put a version number in the objects so that suitable
542 conversions can be made by the class's :meth:`~object.__setstate__` method.
543
544
545 .. _pickle-inst:
546
547 Pickling Class Instances
548 ------------------------
549
550 .. currentmodule:: None
551
552 In this section, we describe the general mechanisms available to you to define,
553 customize, and control how class instances are pickled and unpickled.
554
555 In most cases, no additional code is needed to make instances picklable. By
556 default, pickle will retrieve the class and the attributes of an instance via
557 introspection. When a class instance is unpickled, its :meth:`~object.__init__` method
558 is usually *not* invoked. The default behaviour first creates an uninitialized
559 instance and then restores the saved attributes. The following code shows an
560 implementation of this behaviour::
561
562 def save(obj):
563 return (obj.__class__, obj.__dict__)
564
565 def restore(cls, attributes):
566 obj = cls.__new__(cls)
567 obj.__dict__.update(attributes)
568 return obj
569
570 Classes can alter the default behaviour by providing one or several special
571 methods:
572
573 .. method:: object.__getnewargs_ex__()
574
575 In protocols 2 and newer, classes that implement the
576 :meth:`__getnewargs_ex__` method can dictate the values passed to the
577 :meth:`__new__` method upon unpickling. The method must return a pair
578 ``(args, kwargs)`` where *args* is a tuple of positional arguments
579 and *kwargs* a dictionary of named arguments for constructing the
580 object. Those will be passed to the :meth:`__new__` method upon
581 unpickling.
582
583 You should implement this method if the :meth:`__new__` method of your
584 class requires keyword-only arguments. Otherwise, it is recommended for
585 compatibility to implement :meth:`__getnewargs__`.
586
587 .. versionchanged:: 3.6
588 :meth:`__getnewargs_ex__` is now used in protocols 2 and 3.
589
590
591 .. method:: object.__getnewargs__()
592
593 This method serves a similar purpose as :meth:`__getnewargs_ex__`, but
594 supports only positional arguments. It must return a tuple of arguments
595 ``args`` which will be passed to the :meth:`__new__` method upon unpickling.
596
597 :meth:`__getnewargs__` will not be called if :meth:`__getnewargs_ex__` is
598 defined.
599
600 .. versionchanged:: 3.6
601 Before Python 3.6, :meth:`__getnewargs__` was called instead of
602 :meth:`__getnewargs_ex__` in protocols 2 and 3.
603
604
605 .. method:: object.__getstate__()
606
607 Classes can further influence how their instances are pickled by overriding
608 the method :meth:`__getstate__`. It is called and the returned object
609 is pickled as the contents for the instance, instead of a default state.
610 There are several cases:
611
612 * For a class that has no instance :attr:`~object.__dict__` and no
613 :attr:`~object.__slots__`, the default state is ``None``.
614
615 * For a class that has an instance :attr:`~object.__dict__` and no
616 :attr:`~object.__slots__`, the default state is ``self.__dict__``.
617
618 * For a class that has an instance :attr:`~object.__dict__` and
619 :attr:`~object.__slots__`, the default state is a tuple consisting of two
620 dictionaries: ``self.__dict__``, and a dictionary mapping slot
621 names to slot values. Only slots that have a value are
622 included in the latter.
623
624 * For a class that has :attr:`~object.__slots__` and no instance
625 :attr:`~object.__dict__`, the default state is a tuple whose first item
626 is ``None`` and whose second item is a dictionary mapping slot names
627 to slot values described in the previous bullet.
628
629 .. versionchanged:: 3.11
630 Added the default implementation of the ``__getstate__()`` method in the
631 :class:`object` class.
632
633
634 .. method:: object.__setstate__(state)
635
636 Upon unpickling, if the class defines :meth:`__setstate__`, it is called with
637 the unpickled state. In that case, there is no requirement for the state
638 object to be a dictionary. Otherwise, the pickled state must be a dictionary
639 and its items are assigned to the new instance's dictionary.
640
641 .. note::
642
643 If :meth:`__reduce__` returns a state with value ``None`` at pickling,
644 the :meth:`__setstate__` method will not be called upon unpickling.
645
646
647 Refer to the section :ref:`pickle-state` for more information about how to use
648 the methods :meth:`~object.__getstate__` and :meth:`~object.__setstate__`.
649
650 .. note::
651
652 At unpickling time, some methods like :meth:`~object.__getattr__`,
653 :meth:`~object.__getattribute__`, or :meth:`~object.__setattr__` may be called upon the
654 instance. In case those methods rely on some internal invariant being
655 true, the type should implement :meth:`~object.__new__` to establish such an
656 invariant, as :meth:`~object.__init__` is not called when unpickling an
657 instance.
658
659 .. index:: pair: copy; protocol
660
661 As we shall see, pickle does not use directly the methods described above. In
662 fact, these methods are part of the copy protocol which implements the
663 :meth:`~object.__reduce__` special method. The copy protocol provides a unified
664 interface for retrieving the data necessary for pickling and copying
665 objects. [#]_
666
667 Although powerful, implementing :meth:`~object.__reduce__` directly in your classes is
668 error prone. For this reason, class designers should use the high-level
669 interface (i.e., :meth:`~object.__getnewargs_ex__`, :meth:`~object.__getstate__` and
670 :meth:`~object.__setstate__`) whenever possible. We will show, however, cases where
671 using :meth:`!__reduce__` is the only option or leads to more efficient pickling
672 or both.
673
674 .. method:: object.__reduce__()
675
676 The interface is currently defined as follows. The :meth:`__reduce__` method
677 takes no argument and shall return either a string or preferably a tuple (the
678 returned object is often referred to as the "reduce value").
679
680 If a string is returned, the string should be interpreted as the name of a
681 global variable. It should be the object's local name relative to its
682 module; the pickle module searches the module namespace to determine the
683 object's module: for a given ``obj`` to be pickled, the ``__module__``
684 attribute is looked up on ``obj`` directly, which falls back to a lookup
685 on the type of ``obj`` if no ``__module__`` instance attribute is set.
686 This behaviour is typically useful for singletons.
687
688 When a tuple is returned, it must be between two and six items long.
689 Optional items can either be omitted, or ``None`` can be provided as their
690 value. The semantics of each item are in order:
691
692 .. XXX Mention __newobj__ special-case?
693
694 * A callable object that will be called to create the initial version of the
695 object.
696
697 * A tuple of arguments for the callable object. An empty tuple must be given
698 if the callable does not accept any argument.
699
700 * Optionally, the object's state, which will be passed to the object's
701 :meth:`__setstate__` method as previously described. If the object has no
702 such method then, the value must be a dictionary and it will be added to
703 the object's :attr:`~object.__dict__` attribute.
704
705 * Optionally, an iterator (and not a sequence) yielding successive items.
706 These items will be appended to the object either using
707 ``obj.append(item)`` or, in batch, using ``obj.extend(list_of_items)``.
708 This is primarily used for list subclasses, but may be used by other
709 classes as long as they have :meth:`~sequence.append`
710 and :meth:`~sequence.extend` methods with
711 the appropriate signature. (Whether :meth:`!append` or :meth:`!extend` is
712 used depends on which pickle protocol version is used as well as the number
713 of items to append, so both must be supported.)
714
715 * Optionally, an iterator (not a sequence) yielding successive key-value
716 pairs. These items will be stored to the object using ``obj[key] =
717 value``. This is primarily used for dictionary subclasses, but may be used
718 by other classes as long as they implement :meth:`__setitem__`.
719
720 * Optionally, a callable with a ``(obj, state)`` signature. This
721 callable allows the user to programmatically control the state-updating
722 behavior of a specific object, instead of using ``obj``'s static
723 :meth:`__setstate__` method. If not ``None``, this callable will have
724 priority over ``obj``'s :meth:`__setstate__`.
725
726 .. versionadded:: 3.8
727 The optional sixth tuple item, ``(obj, state)``, was added.
728
729
730 .. method:: object.__reduce_ex__(protocol)
731
732 Alternatively, a :meth:`__reduce_ex__` method may be defined. The only
733 difference is this method should take a single integer argument, the protocol
734 version. When defined, pickle will prefer it over the :meth:`__reduce__`
735 method. In addition, :meth:`__reduce__` automatically becomes a synonym for
736 the extended version. The main use for this method is to provide
737 backwards-compatible reduce values for older Python releases.
738
739 .. currentmodule:: pickle
740
741 .. _pickle-persistent:
742
743 Persistence of External Objects
744 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
745
746 .. index::
747 single: persistent_id (pickle protocol)
748 single: persistent_load (pickle protocol)
749
750 For the benefit of object persistence, the :mod:`pickle` module supports the
751 notion of a reference to an object outside the pickled data stream. Such
752 objects are referenced by a persistent ID, which should be either a string of
753 alphanumeric characters (for protocol 0) [#]_ or just an arbitrary object (for
754 any newer protocol).
755
756 The resolution of such persistent IDs is not defined by the :mod:`pickle`
757 module; it will delegate this resolution to the user-defined methods on the
758 pickler and unpickler, :meth:`~Pickler.persistent_id` and
759 :meth:`~Unpickler.persistent_load` respectively.
760
761 To pickle objects that have an external persistent ID, the pickler must have a
762 custom :meth:`~Pickler.persistent_id` method that takes an object as an
763 argument and returns either ``None`` or the persistent ID for that object.
764 When ``None`` is returned, the pickler simply pickles the object as normal.
765 When a persistent ID string is returned, the pickler will pickle that object,
766 along with a marker so that the unpickler will recognize it as a persistent ID.
767
768 To unpickle external objects, the unpickler must have a custom
769 :meth:`~Unpickler.persistent_load` method that takes a persistent ID object and
770 returns the referenced object.
771
772 Here is a comprehensive example presenting how persistent ID can be used to
773 pickle external objects by reference.
774
775 .. literalinclude:: ../includes/dbpickle.py
776
777 .. _pickle-dispatch:
778
779 Dispatch Tables
780 ^^^^^^^^^^^^^^^
781
782 If one wants to customize pickling of some classes without disturbing
783 any other code which depends on pickling, then one can create a
784 pickler with a private dispatch table.
785
786 The global dispatch table managed by the :mod:`copyreg` module is
787 available as :data:`!copyreg.dispatch_table`. Therefore, one may
788 choose to use a modified copy of :data:`!copyreg.dispatch_table` as a
789 private dispatch table.
790
791 For example ::
792
793 f = io.BytesIO()
794 p = pickle.Pickler(f)
795 p.dispatch_table = copyreg.dispatch_table.copy()
796 p.dispatch_table[SomeClass] = reduce_SomeClass
797
798 creates an instance of :class:`pickle.Pickler` with a private dispatch
799 table which handles the ``SomeClass`` class specially. Alternatively,
800 the code ::
801
802 class MyPickler(pickle.Pickler):
803 dispatch_table = copyreg.dispatch_table.copy()
804 dispatch_table[SomeClass] = reduce_SomeClass
805 f = io.BytesIO()
806 p = MyPickler(f)
807
808 does the same but all instances of ``MyPickler`` will by default
809 share the private dispatch table. On the other hand, the code ::
810
811 copyreg.pickle(SomeClass, reduce_SomeClass)
812 f = io.BytesIO()
813 p = pickle.Pickler(f)
814
815 modifies the global dispatch table shared by all users of the :mod:`copyreg` module.
816
817 .. _pickle-state:
818
819 Handling Stateful Objects
820 ^^^^^^^^^^^^^^^^^^^^^^^^^
821
822 .. index::
823 single: __getstate__() (copy protocol)
824 single: __setstate__() (copy protocol)
825
826 Here's an example that shows how to modify pickling behavior for a class.
827 The :class:`!TextReader` class below opens a text file, and returns the line number and
828 line contents each time its :meth:`!readline` method is called. If a
829 :class:`!TextReader` instance is pickled, all attributes *except* the file object
830 member are saved. When the instance is unpickled, the file is reopened, and
831 reading resumes from the last location. The :meth:`!__setstate__` and
832 :meth:`!__getstate__` methods are used to implement this behavior. ::
833
834 class TextReader:
835 """Print and number lines in a text file."""
836
837 def __init__(self, filename):
838 self.filename = filename
839 self.file = open(filename)
840 self.lineno = 0
841
842 def readline(self):
843 self.lineno += 1
844 line = self.file.readline()
845 if not line:
846 return None
847 if line.endswith('\n'):
848 line = line[:-1]
849 return "%i: %s" % (self.lineno, line)
850
851 def __getstate__(self):
852 # Copy the object's state from self.__dict__ which contains
853 # all our instance attributes. Always use the dict.copy()
854 # method to avoid modifying the original state.
855 state = self.__dict__.copy()
856 # Remove the unpicklable entries.
857 del state['file']
858 return state
859
860 def __setstate__(self, state):
861 # Restore instance attributes (i.e., filename and lineno).
862 self.__dict__.update(state)
863 # Restore the previously opened file's state. To do so, we need to
864 # reopen it and read from it until the line count is restored.
865 file = open(self.filename)
866 for _ in range(self.lineno):
867 file.readline()
868 # Finally, save the file.
869 self.file = file
870
871
872 A sample usage might be something like this::
873
874 >>> reader = TextReader("hello.txt")
875 >>> reader.readline()
876 '1: Hello world!'
877 >>> reader.readline()
878 '2: I am line number two.'
879 >>> new_reader = pickle.loads(pickle.dumps(reader))
880 >>> new_reader.readline()
881 '3: Goodbye!'
882
883 .. _reducer_override:
884
885 Custom Reduction for Types, Functions, and Other Objects
886 --------------------------------------------------------
887
888 .. versionadded:: 3.8
889
890 Sometimes, :attr:`~Pickler.dispatch_table` may not be flexible enough.
891 In particular we may want to customize pickling based on another criterion
892 than the object's type, or we may want to customize the pickling of
893 functions and classes.
894
895 For those cases, it is possible to subclass from the :class:`Pickler` class and
896 implement a :meth:`~Pickler.reducer_override` method. This method can return an
897 arbitrary reduction tuple (see :meth:`~object.__reduce__`). It can alternatively return
898 :data:`NotImplemented` to fallback to the traditional behavior.
899
900 If both the :attr:`~Pickler.dispatch_table` and
901 :meth:`~Pickler.reducer_override` are defined, then
902 :meth:`~Pickler.reducer_override` method takes priority.
903
904 .. Note::
905 For performance reasons, :meth:`~Pickler.reducer_override` may not be
906 called for the following objects: ``None``, ``True``, ``False``, and
907 exact instances of :class:`int`, :class:`float`, :class:`bytes`,
908 :class:`str`, :class:`dict`, :class:`set`, :class:`frozenset`, :class:`list`
909 and :class:`tuple`.
910
911 Here is a simple example where we allow pickling and reconstructing
912 a given class::
913
914 import io
915 import pickle
916
917 class MyClass:
918 my_attribute = 1
919
920 class MyPickler(pickle.Pickler):
921 def reducer_override(self, obj):
922 """Custom reducer for MyClass."""
923 if getattr(obj, "__name__", None) == "MyClass":
924 return type, (obj.__name__, obj.__bases__,
925 {'my_attribute': obj.my_attribute})
926 else:
927 # For any other object, fallback to usual reduction
928 return NotImplemented
929
930 f = io.BytesIO()
931 p = MyPickler(f)
932 p.dump(MyClass)
933
934 del MyClass
935
936 unpickled_class = pickle.loads(f.getvalue())
937
938 assert isinstance(unpickled_class, type)
939 assert unpickled_class.__name__ == "MyClass"
940 assert unpickled_class.my_attribute == 1
941
942
943 .. _pickle-oob:
944
945 Out-of-band Buffers
946 -------------------
947
948 .. versionadded:: 3.8
949
950 In some contexts, the :mod:`pickle` module is used to transfer massive amounts
951 of data. Therefore, it can be important to minimize the number of memory
952 copies, to preserve performance and resource consumption. However, normal
953 operation of the :mod:`pickle` module, as it transforms a graph-like structure
954 of objects into a sequential stream of bytes, intrinsically involves copying
955 data to and from the pickle stream.
956
957 This constraint can be eschewed if both the *provider* (the implementation
958 of the object types to be transferred) and the *consumer* (the implementation
959 of the communications system) support the out-of-band transfer facilities
960 provided by pickle protocol 5 and higher.
961
962 Provider API
963 ^^^^^^^^^^^^
964
965 The large data objects to be pickled must implement a :meth:`~object.__reduce_ex__`
966 method specialized for protocol 5 and higher, which returns a
967 :class:`PickleBuffer` instance (instead of e.g. a :class:`bytes` object)
968 for any large data.
969
970 A :class:`PickleBuffer` object *signals* that the underlying buffer is
971 eligible for out-of-band data transfer. Those objects remain compatible
972 with normal usage of the :mod:`pickle` module. However, consumers can also
973 opt-in to tell :mod:`pickle` that they will handle those buffers by
974 themselves.
975
976 Consumer API
977 ^^^^^^^^^^^^
978
979 A communications system can enable custom handling of the :class:`PickleBuffer`
980 objects generated when serializing an object graph.
981
982 On the sending side, it needs to pass a *buffer_callback* argument to
983 :class:`Pickler` (or to the :func:`dump` or :func:`dumps` function), which
984 will be called with each :class:`PickleBuffer` generated while pickling
985 the object graph. Buffers accumulated by the *buffer_callback* will not
986 see their data copied into the pickle stream, only a cheap marker will be
987 inserted.
988
989 On the receiving side, it needs to pass a *buffers* argument to
990 :class:`Unpickler` (or to the :func:`load` or :func:`loads` function),
991 which is an iterable of the buffers which were passed to *buffer_callback*.
992 That iterable should produce buffers in the same order as they were passed
993 to *buffer_callback*. Those buffers will provide the data expected by the
994 reconstructors of the objects whose pickling produced the original
995 :class:`PickleBuffer` objects.
996
997 Between the sending side and the receiving side, the communications system
998 is free to implement its own transfer mechanism for out-of-band buffers.
999 Potential optimizations include the use of shared memory or datatype-dependent
1000 compression.
1001
1002 Example
1003 ^^^^^^^
1004
1005 Here is a trivial example where we implement a :class:`bytearray` subclass
1006 able to participate in out-of-band buffer pickling::
1007
1008 class ZeroCopyByteArray(bytearray):
1009
1010 def __reduce_ex__(self, protocol):
1011 if protocol >= 5:
1012 return type(self)._reconstruct, (PickleBuffer(self),), None
1013 else:
1014 # PickleBuffer is forbidden with pickle protocols <= 4.
1015 return type(self)._reconstruct, (bytearray(self),)
1016
1017 @classmethod
1018 def _reconstruct(cls, obj):
1019 with memoryview(obj) as m:
1020 # Get a handle over the original buffer object
1021 obj = m.obj
1022 if type(obj) is cls:
1023 # Original buffer object is a ZeroCopyByteArray, return it
1024 # as-is.
1025 return obj
1026 else:
1027 return cls(obj)
1028
1029 The reconstructor (the ``_reconstruct`` class method) returns the buffer's
1030 providing object if it has the right type. This is an easy way to simulate
1031 zero-copy behaviour on this toy example.
1032
1033 On the consumer side, we can pickle those objects the usual way, which
1034 when unserialized will give us a copy of the original object::
1035
1036 b = ZeroCopyByteArray(b"abc")
1037 data = pickle.dumps(b, protocol=5)
1038 new_b = pickle.loads(data)
1039 print(b == new_b) # True
1040 print(b is new_b) # False: a copy was made
1041
1042 But if we pass a *buffer_callback* and then give back the accumulated
1043 buffers when unserializing, we are able to get back the original object::
1044
1045 b = ZeroCopyByteArray(b"abc")
1046 buffers = []
1047 data = pickle.dumps(b, protocol=5, buffer_callback=buffers.append)
1048 new_b = pickle.loads(data, buffers=buffers)
1049 print(b == new_b) # True
1050 print(b is new_b) # True: no copy was made
1051
1052 This example is limited by the fact that :class:`bytearray` allocates its
1053 own memory: you cannot create a :class:`bytearray` instance that is backed
1054 by another object's memory. However, third-party datatypes such as NumPy
1055 arrays do not have this limitation, and allow use of zero-copy pickling
1056 (or making as few copies as possible) when transferring between distinct
1057 processes or systems.
1058
1059 .. seealso:: :pep:`574` -- Pickle protocol 5 with out-of-band data
1060
1061
1062 .. _pickle-restrict:
1063
1064 Restricting Globals
1065 -------------------
1066
1067 .. index::
1068 single: find_class() (pickle protocol)
1069
1070 By default, unpickling will import any class or function that it finds in the
1071 pickle data. For many applications, this behaviour is unacceptable as it
1072 permits the unpickler to import and invoke arbitrary code. Just consider what
1073 this hand-crafted pickle data stream does when loaded::
1074
1075 >>> import pickle
1076 >>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
1077 hello world
1078 0
1079
1080 In this example, the unpickler imports the :func:`os.system` function and then
1081 apply the string argument "echo hello world". Although this example is
1082 inoffensive, it is not difficult to imagine one that could damage your system.
1083
1084 For this reason, you may want to control what gets unpickled by customizing
1085 :meth:`Unpickler.find_class`. Unlike its name suggests,
1086 :meth:`Unpickler.find_class` is called whenever a global (i.e., a class or
1087 a function) is requested. Thus it is possible to either completely forbid
1088 globals or restrict them to a safe subset.
1089
1090 Here is an example of an unpickler allowing only few safe classes from the
1091 :mod:`builtins` module to be loaded::
1092
1093 import builtins
1094 import io
1095 import pickle
1096
1097 safe_builtins = {
1098 'range',
1099 'complex',
1100 'set',
1101 'frozenset',
1102 'slice',
1103 }
1104
1105 class RestrictedUnpickler(pickle.Unpickler):
1106
1107 def find_class(self, module, name):
1108 # Only allow safe classes from builtins.
1109 if module == "builtins" and name in safe_builtins:
1110 return getattr(builtins, name)
1111 # Forbid everything else.
1112 raise pickle.UnpicklingError("global '%s.%s' is forbidden" %
1113 (module, name))
1114
1115 def restricted_loads(s):
1116 """Helper function analogous to pickle.loads()."""
1117 return RestrictedUnpickler(io.BytesIO(s)).load()
1118
1119 A sample usage of our unpickler working as intended::
1120
1121 >>> restricted_loads(pickle.dumps([1, 2, range(15)]))
1122 [1, 2, range(0, 15)]
1123 >>> restricted_loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
1124 Traceback (most recent call last):
1125 ...
1126 pickle.UnpicklingError: global 'os.system' is forbidden
1127 >>> restricted_loads(b'cbuiltins\neval\n'
1128 ... b'(S\'getattr(__import__("os"), "system")'
1129 ... b'("echo hello world")\'\ntR.')
1130 Traceback (most recent call last):
1131 ...
1132 pickle.UnpicklingError: global 'builtins.eval' is forbidden
1133
1134
1135 .. XXX Add note about how extension codes could evade our protection
1136 mechanism (e.g. cached classes do not invokes find_class()).
1137
1138 As our examples shows, you have to be careful with what you allow to be
1139 unpickled. Therefore if security is a concern, you may want to consider
1140 alternatives such as the marshalling API in :mod:`xmlrpc.client` or
1141 third-party solutions.
1142
1143
1144 Performance
1145 -----------
1146
1147 Recent versions of the pickle protocol (from protocol 2 and upwards) feature
1148 efficient binary encodings for several common features and built-in types.
1149 Also, the :mod:`pickle` module has a transparent optimizer written in C.
1150
1151
1152 .. _pickle-example:
1153
1154 Examples
1155 --------
1156
1157 For the simplest code, use the :func:`dump` and :func:`load` functions. ::
1158
1159 import pickle
1160
1161 # An arbitrary collection of objects supported by pickle.
1162 data = {
1163 'a': [1, 2.0, 3+4j],
1164 'b': ("character string", b"byte string"),
1165 'c': {None, True, False}
1166 }
1167
1168 with open('data.pickle', 'wb') as f:
1169 # Pickle the 'data' dictionary using the highest protocol available.
1170 pickle.dump(data, f, pickle.HIGHEST_PROTOCOL)
1171
1172
1173 The following example reads the resulting pickled data. ::
1174
1175 import pickle
1176
1177 with open('data.pickle', 'rb') as f:
1178 # The protocol version used is detected automatically, so we do not
1179 # have to specify it.
1180 data = pickle.load(f)
1181
1182
1183 .. XXX: Add examples showing how to optimize pickles for size (like using
1184 .. pickletools.optimize() or the gzip module).
1185
1186
1187 .. seealso::
1188
1189 Module :mod:`copyreg`
1190 Pickle interface constructor registration for extension types.
1191
1192 Module :mod:`pickletools`
1193 Tools for working with and analyzing pickled data.
1194
1195 Module :mod:`shelve`
1196 Indexed databases of objects; uses :mod:`pickle`.
1197
1198 Module :mod:`copy`
1199 Shallow and deep object copying.
1200
1201 Module :mod:`marshal`
1202 High-performance serialization of built-in types.
1203
1204
1205 .. rubric:: Footnotes
1206
1207 .. [#] Don't confuse this with the :mod:`marshal` module
1208
1209 .. [#] This is why :keyword:`lambda` functions cannot be pickled: all
1210 :keyword:`!lambda` functions share the same name: ``<lambda>``.
1211
1212 .. [#] The exception raised will likely be an :exc:`ImportError` or an
1213 :exc:`AttributeError` but it could be something else.
1214
1215 .. [#] The :mod:`copy` module uses this protocol for shallow and deep copying
1216 operations.
1217
1218 .. [#] The limitation on alphanumeric characters is due to the fact
1219 that persistent IDs in protocol 0 are delimited by the newline
1220 character. Therefore if any kind of newline characters occurs in
1221 persistent IDs, the resulting pickled data will become unreadable.