| 1 |
:mod:`!pickle` --- Python object serialization |
| 2 |
============================================== |
| 3 |
|
| 4 |
.. module:: pickle |
| 5 |
:synopsis: Convert Python objects to streams of bytes and back. |
| 6 |
|
| 7 |
.. sectionauthor:: Jim Kerr <[email protected]>. |
| 8 |
.. sectionauthor:: Barry Warsaw <[email protected]> |
| 9 |
|
| 10 |
**Source code:** :source:`Lib/pickle.py` |
| 11 |
|
| 12 |
.. index:: |
| 13 |
single: persistence |
| 14 |
pair: persistent; objects |
| 15 |
pair: serializing; objects |
| 16 |
pair: marshalling; objects |
| 17 |
pair: flattening; objects |
| 18 |
pair: pickling; objects |
| 19 |
|
| 20 |
-------------- |
| 21 |
|
| 22 |
The :mod:`!pickle` module implements binary protocols for serializing and |
| 23 |
de-serializing a Python object structure. *"Pickling"* is the process |
| 24 |
whereby a Python object hierarchy is converted into a byte stream, and |
| 25 |
*"unpickling"* is the inverse operation, whereby a byte stream |
| 26 |
(from a :term:`binary file` or :term:`bytes-like object`) is converted |
| 27 |
back into an object hierarchy. Pickling (and unpickling) is alternatively |
| 28 |
known as "serialization", "marshalling," [#]_ or "flattening"; however, to |
| 29 |
avoid confusion, the terms used here are "pickling" and "unpickling". |
| 30 |
|
| 31 |
.. warning:: |
| 32 |
|
| 33 |
The ``pickle`` module **is not secure**. Only unpickle data you trust. |
| 34 |
|
| 35 |
It is possible to construct malicious pickle data which will **execute |
| 36 |
arbitrary code during unpickling**. Never unpickle data that could have come |
| 37 |
from an untrusted source, or that could have been tampered with. |
| 38 |
|
| 39 |
Consider signing data with :mod:`hmac` if you need to ensure that it has not |
| 40 |
been tampered with. |
| 41 |
|
| 42 |
Safer serialization formats such as :mod:`json` may be more appropriate if |
| 43 |
you are processing untrusted data. See :ref:`comparison-with-json`. |
| 44 |
|
| 45 |
|
| 46 |
Relationship to other Python modules |
| 47 |
------------------------------------ |
| 48 |
|
| 49 |
Comparison with ``marshal`` |
| 50 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 51 |
|
| 52 |
Python has a more primitive serialization module called :mod:`marshal`, but in |
| 53 |
general :mod:`!pickle` should always be the preferred way to serialize Python |
| 54 |
objects. :mod:`marshal` exists primarily to support Python's :file:`.pyc` |
| 55 |
files. |
| 56 |
|
| 57 |
The :mod:`!pickle` module differs from :mod:`marshal` in several significant ways: |
| 58 |
|
| 59 |
* :mod:`marshal` cannot be used to serialize user-defined classes and their |
| 60 |
instances. :mod:`!pickle` can save and restore class instances transparently, |
| 61 |
however the class definition must be importable and live in the same module as |
| 62 |
when the object was pickled. |
| 63 |
|
| 64 |
* The :mod:`marshal` serialization format is not guaranteed to be portable |
| 65 |
across Python versions. Because its primary job in life is to support |
| 66 |
:file:`.pyc` files, the Python implementers reserve the right to change the |
| 67 |
serialization format in non-backwards compatible ways should the need arise. |
| 68 |
The :mod:`!pickle` serialization format is guaranteed to be backwards compatible |
| 69 |
across Python releases provided a compatible pickle protocol is chosen and |
| 70 |
pickling and unpickling code deals with Python 2 to Python 3 type differences |
| 71 |
if your data is crossing that unique breaking change language boundary. |
| 72 |
|
| 73 |
|
| 74 |
.. _comparison-with-json: |
| 75 |
|
| 76 |
Comparison with ``json`` |
| 77 |
^^^^^^^^^^^^^^^^^^^^^^^^ |
| 78 |
|
| 79 |
There are fundamental differences between the pickle protocols and |
| 80 |
`JSON (JavaScript Object Notation) <https://json.org>`_: |
| 81 |
|
| 82 |
* JSON is a text serialization format (it outputs unicode text, although |
| 83 |
most of the time it is then encoded to ``utf-8``), while pickle is |
| 84 |
a binary serialization format; |
| 85 |
|
| 86 |
* JSON is human-readable, while pickle is not; |
| 87 |
|
| 88 |
* JSON is interoperable and widely used outside of the Python ecosystem, |
| 89 |
while pickle is Python-specific; |
| 90 |
|
| 91 |
* JSON, by default, can only represent a subset of the Python built-in |
| 92 |
types, and no custom classes; pickle can represent an extremely large |
| 93 |
number of Python types (many of them automatically, by clever usage |
| 94 |
of Python's introspection facilities; complex cases can be tackled by |
| 95 |
implementing :ref:`specific object APIs <pickle-inst>`); |
| 96 |
|
| 97 |
* Unlike pickle, deserializing untrusted JSON does not in itself create an |
| 98 |
arbitrary code execution vulnerability. |
| 99 |
|
| 100 |
.. seealso:: |
| 101 |
The :mod:`json` module: a standard library module allowing JSON |
| 102 |
serialization and deserialization. |
| 103 |
|
| 104 |
|
| 105 |
.. _pickle-protocols: |
| 106 |
|
| 107 |
Data stream format |
| 108 |
------------------ |
| 109 |
|
| 110 |
.. index:: |
| 111 |
single: External Data Representation |
| 112 |
|
| 113 |
The data format used by :mod:`!pickle` is Python-specific. This has the |
| 114 |
advantage that there are no restrictions imposed by external standards such as |
| 115 |
JSON (which can't represent pointer sharing); however it means that |
| 116 |
non-Python programs may not be able to reconstruct pickled Python objects. |
| 117 |
|
| 118 |
By default, the :mod:`!pickle` data format uses a relatively compact binary |
| 119 |
representation. If you need optimal size characteristics, you can efficiently |
| 120 |
:doc:`compress <archiving>` pickled data. |
| 121 |
|
| 122 |
The module :mod:`pickletools` contains tools for analyzing data streams |
| 123 |
generated by :mod:`!pickle`. :mod:`pickletools` source code has extensive |
| 124 |
comments about opcodes used by pickle protocols. |
| 125 |
|
| 126 |
There are currently 6 different protocols which can be used for pickling. |
| 127 |
The higher the protocol used, the more recent the version of Python needed |
| 128 |
to read the pickle produced. |
| 129 |
|
| 130 |
* Protocol version 0 is the original "human-readable" protocol and is |
| 131 |
backwards compatible with earlier versions of Python. |
| 132 |
|
| 133 |
* Protocol version 1 is an old binary format which is also compatible with |
| 134 |
earlier versions of Python. |
| 135 |
|
| 136 |
* Protocol version 2 was introduced in Python 2.3. It provides much more |
| 137 |
efficient pickling of :term:`new-style classes <new-style class>`. Refer to :pep:`307` for |
| 138 |
information about improvements brought by protocol 2. |
| 139 |
|
| 140 |
* Protocol version 3 was added in Python 3.0. It has explicit support for |
| 141 |
:class:`bytes` objects and cannot be unpickled by Python 2.x. This was |
| 142 |
the default protocol in Python 3.0--3.7. |
| 143 |
|
| 144 |
* Protocol version 4 was added in Python 3.4. It adds support for very large |
| 145 |
objects, pickling more kinds of objects, and some data format |
| 146 |
optimizations. This was the default protocol in Python 3.8--3.13. |
| 147 |
Refer to :pep:`3154` for information about improvements brought by |
| 148 |
protocol 4. |
| 149 |
|
| 150 |
* Protocol version 5 was added in Python 3.8. It adds support for out-of-band |
| 151 |
data and speedup for in-band data. It is the default protocol starting with |
| 152 |
Python 3.14. Refer to :pep:`574` for information about improvements brought |
| 153 |
by protocol 5. |
| 154 |
|
| 155 |
.. note:: |
| 156 |
Serialization is a more primitive notion than persistence; although |
| 157 |
:mod:`!pickle` reads and writes file objects, it does not handle the issue of |
| 158 |
naming persistent objects, nor the (even more complicated) issue of concurrent |
| 159 |
access to persistent objects. The :mod:`!pickle` module can transform a complex |
| 160 |
object into a byte stream and it can transform the byte stream into an object |
| 161 |
with the same internal structure. Perhaps the most obvious thing to do with |
| 162 |
these byte streams is to write them onto a file, but it is also conceivable to |
| 163 |
send them across a network or store them in a database. The :mod:`shelve` |
| 164 |
module provides a simple interface to pickle and unpickle objects on |
| 165 |
DBM-style database files. |
| 166 |
|
| 167 |
|
| 168 |
Module Interface |
| 169 |
---------------- |
| 170 |
|
| 171 |
To serialize an object hierarchy, you simply call the :func:`dumps` function. |
| 172 |
Similarly, to de-serialize a data stream, you call the :func:`loads` function. |
| 173 |
However, if you want more control over serialization and de-serialization, |
| 174 |
you can create a :class:`Pickler` or an :class:`Unpickler` object, respectively. |
| 175 |
|
| 176 |
The :mod:`!pickle` module provides the following constants: |
| 177 |
|
| 178 |
|
| 179 |
.. data:: HIGHEST_PROTOCOL |
| 180 |
|
| 181 |
An integer, the highest :ref:`protocol version <pickle-protocols>` |
| 182 |
available. This value can be passed as a *protocol* value to functions |
| 183 |
:func:`dump` and :func:`dumps` as well as the :class:`Pickler` |
| 184 |
constructor. |
| 185 |
|
| 186 |
.. data:: DEFAULT_PROTOCOL |
| 187 |
|
| 188 |
An integer, the default :ref:`protocol version <pickle-protocols>` used |
| 189 |
for pickling. May be less than :data:`HIGHEST_PROTOCOL`. Currently the |
| 190 |
default protocol is 5, introduced in Python 3.8 and incompatible |
| 191 |
with previous versions. This version introduces support for out-of-band |
| 192 |
buffers, where :pep:`3118`-compatible data can be transmitted separately |
| 193 |
from the main pickle stream. |
| 194 |
|
| 195 |
.. versionchanged:: 3.0 |
| 196 |
|
| 197 |
The default protocol is 3. |
| 198 |
|
| 199 |
.. versionchanged:: 3.8 |
| 200 |
|
| 201 |
The default protocol is 4. |
| 202 |
|
| 203 |
.. versionchanged:: 3.14 |
| 204 |
|
| 205 |
The default protocol is 5. |
| 206 |
|
| 207 |
The :mod:`!pickle` module provides the following functions to make the pickling |
| 208 |
process more convenient: |
| 209 |
|
| 210 |
.. function:: dump(obj, file, protocol=None, *, fix_imports=True, buffer_callback=None) |
| 211 |
|
| 212 |
Write the pickled representation of the object *obj* to the open |
| 213 |
:term:`file object` *file*. This is equivalent to |
| 214 |
``Pickler(file, protocol).dump(obj)``. |
| 215 |
|
| 216 |
Arguments *file*, *protocol*, *fix_imports* and *buffer_callback* have |
| 217 |
the same meaning as in the :class:`Pickler` constructor. |
| 218 |
|
| 219 |
.. versionchanged:: 3.8 |
| 220 |
The *buffer_callback* argument was added. |
| 221 |
|
| 222 |
.. function:: dumps(obj, protocol=None, *, fix_imports=True, buffer_callback=None) |
| 223 |
|
| 224 |
Return the pickled representation of the object *obj* as a :class:`bytes` object, |
| 225 |
instead of writing it to a file. |
| 226 |
|
| 227 |
Arguments *protocol*, *fix_imports* and *buffer_callback* have the same |
| 228 |
meaning as in the :class:`Pickler` constructor. |
| 229 |
|
| 230 |
.. versionchanged:: 3.8 |
| 231 |
The *buffer_callback* argument was added. |
| 232 |
|
| 233 |
.. function:: load(file, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None) |
| 234 |
|
| 235 |
Read the pickled representation of an object from the open :term:`file object` |
| 236 |
*file* and return the reconstituted object hierarchy specified therein. |
| 237 |
This is equivalent to ``Unpickler(file).load()``. |
| 238 |
|
| 239 |
The protocol version of the pickle is detected automatically, so no |
| 240 |
protocol argument is needed. Bytes past the pickled representation |
| 241 |
of the object are ignored. |
| 242 |
|
| 243 |
Arguments *file*, *fix_imports*, *encoding*, *errors*, *strict* and *buffers* |
| 244 |
have the same meaning as in the :class:`Unpickler` constructor. |
| 245 |
|
| 246 |
.. versionchanged:: 3.8 |
| 247 |
The *buffers* argument was added. |
| 248 |
|
| 249 |
.. function:: loads(data, /, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None) |
| 250 |
|
| 251 |
Return the reconstituted object hierarchy of the pickled representation |
| 252 |
*data* of an object. *data* must be a :term:`bytes-like object`. |
| 253 |
|
| 254 |
The protocol version of the pickle is detected automatically, so no |
| 255 |
protocol argument is needed. Bytes past the pickled representation |
| 256 |
of the object are ignored. |
| 257 |
|
| 258 |
Arguments *fix_imports*, *encoding*, *errors*, *strict* and *buffers* |
| 259 |
have the same meaning as in the :class:`Unpickler` constructor. |
| 260 |
|
| 261 |
.. versionchanged:: 3.8 |
| 262 |
The *buffers* argument was added. |
| 263 |
|
| 264 |
|
| 265 |
The :mod:`!pickle` module defines three exceptions: |
| 266 |
|
| 267 |
.. exception:: PickleError |
| 268 |
|
| 269 |
Common base class for the other pickling exceptions. It inherits from |
| 270 |
:exc:`Exception`. |
| 271 |
|
| 272 |
.. exception:: PicklingError |
| 273 |
|
| 274 |
Error raised when an unpicklable object is encountered by :class:`Pickler`. |
| 275 |
It inherits from :exc:`PickleError`. |
| 276 |
|
| 277 |
Refer to :ref:`pickle-picklable` to learn what kinds of objects can be |
| 278 |
pickled. |
| 279 |
|
| 280 |
.. exception:: UnpicklingError |
| 281 |
|
| 282 |
Error raised when there is a problem unpickling an object, such as a data |
| 283 |
corruption or a security violation. It inherits from :exc:`PickleError`. |
| 284 |
|
| 285 |
Note that other exceptions may also be raised during unpickling, including |
| 286 |
(but not necessarily limited to) AttributeError, EOFError, ImportError, and |
| 287 |
IndexError. |
| 288 |
|
| 289 |
|
| 290 |
The :mod:`!pickle` module exports three classes, :class:`Pickler`, |
| 291 |
:class:`Unpickler` and :class:`PickleBuffer`: |
| 292 |
|
| 293 |
.. class:: Pickler(file, protocol=None, *, fix_imports=True, buffer_callback=None) |
| 294 |
|
| 295 |
This takes a binary file for writing a pickle data stream. |
| 296 |
|
| 297 |
The optional *protocol* argument, an integer, tells the pickler to use |
| 298 |
the given protocol; supported protocols are 0 to :data:`HIGHEST_PROTOCOL`. |
| 299 |
If not specified, the default is :data:`DEFAULT_PROTOCOL`. If a negative |
| 300 |
number is specified, :data:`HIGHEST_PROTOCOL` is selected. |
| 301 |
|
| 302 |
The *file* argument must have a write() method that accepts a single bytes |
| 303 |
argument. It can thus be an on-disk file opened for binary writing, an |
| 304 |
:class:`io.BytesIO` instance, or any other custom object that meets this |
| 305 |
interface. |
| 306 |
|
| 307 |
If *fix_imports* is true and *protocol* is less than 3, pickle will try to |
| 308 |
map the new Python 3 names to the old module names used in Python 2, so |
| 309 |
that the pickle data stream is readable with Python 2. |
| 310 |
|
| 311 |
If *buffer_callback* is ``None`` (the default), buffer views are |
| 312 |
serialized into *file* as part of the pickle stream. |
| 313 |
|
| 314 |
If *buffer_callback* is not ``None``, then it can be called any number |
| 315 |
of times with a buffer view. If the callback returns a false value |
| 316 |
(such as ``None``), the given buffer is :ref:`out-of-band <pickle-oob>`; |
| 317 |
otherwise the buffer is serialized in-band, i.e. inside the pickle stream. |
| 318 |
|
| 319 |
It is an error if *buffer_callback* is not ``None`` and *protocol* is |
| 320 |
``None`` or smaller than 5. |
| 321 |
|
| 322 |
.. versionchanged:: 3.8 |
| 323 |
The *buffer_callback* argument was added. |
| 324 |
|
| 325 |
.. method:: dump(obj) |
| 326 |
|
| 327 |
Write the pickled representation of *obj* to the open file object given in |
| 328 |
the constructor. |
| 329 |
|
| 330 |
.. method:: persistent_id(obj) |
| 331 |
|
| 332 |
Do nothing by default. This exists so a subclass can override it. |
| 333 |
|
| 334 |
If :meth:`persistent_id` returns ``None``, *obj* is pickled as usual. Any |
| 335 |
other value causes :class:`Pickler` to emit the returned value as a |
| 336 |
persistent ID for *obj*. The meaning of this persistent ID should be |
| 337 |
defined by :meth:`Unpickler.persistent_load`. Note that the value |
| 338 |
returned by :meth:`persistent_id` cannot itself have a persistent ID. |
| 339 |
|
| 340 |
See :ref:`pickle-persistent` for details and examples of uses. |
| 341 |
|
| 342 |
.. versionchanged:: 3.13 |
| 343 |
Add the default implementation of this method in the C implementation |
| 344 |
of :class:`!Pickler`. |
| 345 |
|
| 346 |
.. attribute:: dispatch_table |
| 347 |
|
| 348 |
A pickler object's dispatch table is a registry of *reduction |
| 349 |
functions* of the kind which can be declared using |
| 350 |
:func:`copyreg.pickle`. It is a mapping whose keys are classes |
| 351 |
and whose values are reduction functions. A reduction function |
| 352 |
takes a single argument of the associated class and should |
| 353 |
conform to the same interface as a :meth:`~object.__reduce__` |
| 354 |
method. |
| 355 |
|
| 356 |
By default, a pickler object will not have a |
| 357 |
:attr:`dispatch_table` attribute, and it will instead use the |
| 358 |
global dispatch table managed by the :mod:`copyreg` module. |
| 359 |
However, to customize the pickling for a specific pickler object |
| 360 |
one can set the :attr:`dispatch_table` attribute to a dict-like |
| 361 |
object. Alternatively, if a subclass of :class:`Pickler` has a |
| 362 |
:attr:`dispatch_table` attribute then this will be used as the |
| 363 |
default dispatch table for instances of that class. |
| 364 |
|
| 365 |
See :ref:`pickle-dispatch` for usage examples. |
| 366 |
|
| 367 |
.. versionadded:: 3.3 |
| 368 |
|
| 369 |
.. method:: reducer_override(obj) |
| 370 |
|
| 371 |
Special reducer that can be defined in :class:`Pickler` subclasses. This |
| 372 |
method has priority over any reducer in the :attr:`dispatch_table`. It |
| 373 |
should conform to the same interface as a :meth:`~object.__reduce__` method, and |
| 374 |
can optionally return :data:`NotImplemented` to fallback on |
| 375 |
:attr:`dispatch_table`-registered reducers to pickle ``obj``. |
| 376 |
|
| 377 |
For a detailed example, see :ref:`reducer_override`. |
| 378 |
|
| 379 |
.. versionadded:: 3.8 |
| 380 |
|
| 381 |
.. attribute:: fast |
| 382 |
|
| 383 |
Deprecated. Enable fast mode if set to a true value. The fast mode |
| 384 |
disables the usage of memo, therefore speeding the pickling process by not |
| 385 |
generating superfluous PUT opcodes. It should not be used with |
| 386 |
self-referential objects, doing otherwise will cause :class:`Pickler` to |
| 387 |
recurse infinitely. |
| 388 |
|
| 389 |
Use :func:`pickletools.optimize` if you need more compact pickles. |
| 390 |
|
| 391 |
.. method:: clear_memo() |
| 392 |
|
| 393 |
Clears the pickler's "memo". |
| 394 |
|
| 395 |
The memo is the data structure that remembers which objects the |
| 396 |
pickler has already seen, so that shared or recursive objects |
| 397 |
are pickled by reference and not by value. This method is |
| 398 |
useful when re-using picklers. |
| 399 |
|
| 400 |
|
| 401 |
.. class:: Unpickler(file, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None) |
| 402 |
|
| 403 |
This takes a binary file for reading a pickle data stream. |
| 404 |
|
| 405 |
The protocol version of the pickle is detected automatically, so no |
| 406 |
protocol argument is needed. |
| 407 |
|
| 408 |
The argument *file* must have three methods, a read() method that takes an |
| 409 |
integer argument, a readinto() method that takes a buffer argument |
| 410 |
and a readline() method that requires no arguments, as in the |
| 411 |
:class:`io.BufferedIOBase` interface. Thus *file* can be an on-disk file |
| 412 |
opened for binary reading, an :class:`io.BytesIO` object, or any other |
| 413 |
custom object that meets this interface. |
| 414 |
|
| 415 |
The optional arguments *fix_imports*, *encoding* and *errors* are used |
| 416 |
to control compatibility support for pickle stream generated by Python 2. |
| 417 |
If *fix_imports* is true, pickle will try to map the old Python 2 names |
| 418 |
to the new names used in Python 3. The *encoding* and *errors* tell |
| 419 |
pickle how to decode 8-bit string instances pickled by Python 2; |
| 420 |
these default to 'ASCII' and 'strict', respectively. The *encoding* can |
| 421 |
be 'bytes' to read these 8-bit string instances as bytes objects. |
| 422 |
Using ``encoding='latin1'`` is required for unpickling NumPy arrays and |
| 423 |
instances of :class:`~datetime.datetime`, :class:`~datetime.date` and |
| 424 |
:class:`~datetime.time` pickled by Python 2. |
| 425 |
|
| 426 |
If *buffers* is ``None`` (the default), then all data necessary for |
| 427 |
deserialization must be contained in the pickle stream. This means |
| 428 |
that the *buffer_callback* argument was ``None`` when a :class:`Pickler` |
| 429 |
was instantiated (or when :func:`dump` or :func:`dumps` was called). |
| 430 |
|
| 431 |
If *buffers* is not ``None``, it should be an iterable of buffer-enabled |
| 432 |
objects that is consumed each time the pickle stream references |
| 433 |
an :ref:`out-of-band <pickle-oob>` buffer view. Such buffers have been |
| 434 |
given in order to the *buffer_callback* of a Pickler object. |
| 435 |
|
| 436 |
.. versionchanged:: 3.8 |
| 437 |
The *buffers* argument was added. |
| 438 |
|
| 439 |
.. method:: load() |
| 440 |
|
| 441 |
Read the pickled representation of an object from the open file object |
| 442 |
given in the constructor, and return the reconstituted object hierarchy |
| 443 |
specified therein. Bytes past the pickled representation of the object |
| 444 |
are ignored. |
| 445 |
|
| 446 |
.. method:: persistent_load(pid) |
| 447 |
|
| 448 |
Raise an :exc:`UnpicklingError` by default. |
| 449 |
|
| 450 |
If defined, :meth:`persistent_load` should return the object specified by |
| 451 |
the persistent ID *pid*. If an invalid persistent ID is encountered, an |
| 452 |
:exc:`UnpicklingError` should be raised. |
| 453 |
|
| 454 |
See :ref:`pickle-persistent` for details and examples of uses. |
| 455 |
|
| 456 |
.. versionchanged:: 3.13 |
| 457 |
Add the default implementation of this method in the C implementation |
| 458 |
of :class:`!Unpickler`. |
| 459 |
|
| 460 |
.. method:: find_class(module, name) |
| 461 |
|
| 462 |
Import *module* if necessary and return the object called *name* from it, |
| 463 |
where the *module* and *name* arguments are :class:`str` objects. Note, |
| 464 |
unlike its name suggests, :meth:`find_class` is also used for finding |
| 465 |
functions. |
| 466 |
|
| 467 |
Subclasses may override this to gain control over what type of objects and |
| 468 |
how they can be loaded, potentially reducing security risks. Refer to |
| 469 |
:ref:`pickle-restrict` for details. |
| 470 |
|
| 471 |
.. audit-event:: pickle.find_class module,name pickle.Unpickler.find_class |
| 472 |
|
| 473 |
.. class:: PickleBuffer(buffer) |
| 474 |
|
| 475 |
A wrapper for a buffer representing picklable data. *buffer* must be a |
| 476 |
:ref:`buffer-providing <bufferobjects>` object, such as a |
| 477 |
:term:`bytes-like object` or a N-dimensional array. |
| 478 |
|
| 479 |
:class:`PickleBuffer` is itself a buffer provider, therefore it is |
| 480 |
possible to pass it to other APIs expecting a buffer-providing object, |
| 481 |
such as :class:`memoryview`. |
| 482 |
|
| 483 |
:class:`PickleBuffer` objects can only be serialized using pickle |
| 484 |
protocol 5 or higher. They are eligible for |
| 485 |
:ref:`out-of-band serialization <pickle-oob>`. |
| 486 |
|
| 487 |
.. versionadded:: 3.8 |
| 488 |
|
| 489 |
.. method:: raw() |
| 490 |
|
| 491 |
Return a :class:`memoryview` of the memory area underlying this buffer. |
| 492 |
The returned object is a one-dimensional, C-contiguous memoryview |
| 493 |
with format ``B`` (unsigned bytes). :exc:`BufferError` is raised if |
| 494 |
the buffer is neither C- nor Fortran-contiguous. |
| 495 |
|
| 496 |
.. method:: release() |
| 497 |
|
| 498 |
Release the underlying buffer exposed by the PickleBuffer object. |
| 499 |
|
| 500 |
|
| 501 |
.. _pickle-picklable: |
| 502 |
|
| 503 |
What can be pickled and unpickled? |
| 504 |
---------------------------------- |
| 505 |
|
| 506 |
The following types can be pickled: |
| 507 |
|
| 508 |
* built-in constants (``None``, ``True``, ``False``, ``Ellipsis``, and |
| 509 |
:data:`NotImplemented`); |
| 510 |
|
| 511 |
* integers, floating-point numbers, complex numbers; |
| 512 |
|
| 513 |
* strings, bytes, bytearrays; |
| 514 |
|
| 515 |
* tuples, lists, sets, and dictionaries containing only picklable objects; |
| 516 |
|
| 517 |
* functions (built-in and user-defined) accessible from the top level of a |
| 518 |
module (using :keyword:`def`, not :keyword:`lambda`); |
| 519 |
|
| 520 |
* classes accessible from the top level of a module; |
| 521 |
|
| 522 |
* instances of such classes for which the result of calling :meth:`~object.__getstate__` |
| 523 |
is picklable (see section :ref:`pickle-inst` for details). |
| 524 |
|
| 525 |
Attempts to pickle unpicklable objects will raise the :exc:`PicklingError` |
| 526 |
exception; when this happens, an unspecified number of bytes may have already |
| 527 |
been written to the underlying file. Trying to pickle a highly recursive data |
| 528 |
structure may exceed the maximum recursion depth, a :exc:`RecursionError` will be |
| 529 |
raised in this case. You can carefully raise this limit with |
| 530 |
:func:`sys.setrecursionlimit`. |
| 531 |
|
| 532 |
Note that functions (built-in and user-defined) are pickled by fully |
| 533 |
:term:`qualified name`, not by value. [#]_ This means that only the function name is |
| 534 |
pickled, along with the name of the containing module and classes. Neither |
| 535 |
the function's code, nor any of its function attributes are pickled. Thus the |
| 536 |
defining module must be importable in the unpickling environment, and the module |
| 537 |
must contain the named object, otherwise an exception will be raised. [#]_ |
| 538 |
|
| 539 |
Similarly, classes are pickled by fully qualified name, so the same restrictions in |
| 540 |
the unpickling environment apply. Note that none of the class's code or data is |
| 541 |
pickled, so in the following example the class attribute ``attr`` is not |
| 542 |
restored in the unpickling environment:: |
| 543 |
|
| 544 |
class Foo: |
| 545 |
attr = 'A class attribute' |
| 546 |
|
| 547 |
picklestring = pickle.dumps(Foo) |
| 548 |
|
| 549 |
These restrictions are why picklable functions and classes must be defined at |
| 550 |
the top level of a module. |
| 551 |
|
| 552 |
Similarly, when class instances are pickled, their class's code and data are not |
| 553 |
pickled along with them. Only the instance data are pickled. This is done on |
| 554 |
purpose, so you can fix bugs in a class or add methods to the class and still |
| 555 |
load objects that were created with an earlier version of the class. If you |
| 556 |
plan to have long-lived objects that will see many versions of a class, it may |
| 557 |
be worthwhile to put a version number in the objects so that suitable |
| 558 |
conversions can be made by the class's :meth:`~object.__setstate__` method. |
| 559 |
|
| 560 |
|
| 561 |
.. _pickle-inst: |
| 562 |
|
| 563 |
Pickling Class Instances |
| 564 |
------------------------ |
| 565 |
|
| 566 |
.. currentmodule:: None |
| 567 |
|
| 568 |
In this section, we describe the general mechanisms available to you to define, |
| 569 |
customize, and control how class instances are pickled and unpickled. |
| 570 |
|
| 571 |
In most cases, no additional code is needed to make instances picklable. By |
| 572 |
default, pickle will retrieve the class and the attributes of an instance via |
| 573 |
introspection. When a class instance is unpickled, its :meth:`~object.__init__` method |
| 574 |
is usually *not* invoked. The default behaviour first creates an uninitialized |
| 575 |
instance and then restores the saved attributes. The following code shows an |
| 576 |
implementation of this behaviour:: |
| 577 |
|
| 578 |
def save(obj): |
| 579 |
return (obj.__class__, obj.__dict__) |
| 580 |
|
| 581 |
def restore(cls, attributes): |
| 582 |
obj = cls.__new__(cls) |
| 583 |
obj.__dict__.update(attributes) |
| 584 |
return obj |
| 585 |
|
| 586 |
Classes can alter the default behaviour by providing one or several special |
| 587 |
methods: |
| 588 |
|
| 589 |
.. method:: object.__getnewargs_ex__() |
| 590 |
|
| 591 |
In protocols 2 and newer, classes that implement the |
| 592 |
:meth:`__getnewargs_ex__` method can dictate the values passed to the |
| 593 |
:meth:`__new__` method upon unpickling. The method must return a pair |
| 594 |
``(args, kwargs)`` where *args* is a tuple of positional arguments |
| 595 |
and *kwargs* a dictionary of named arguments for constructing the |
| 596 |
object. Those will be passed to the :meth:`__new__` method upon |
| 597 |
unpickling. |
| 598 |
|
| 599 |
You should implement this method if the :meth:`__new__` method of your |
| 600 |
class requires keyword-only arguments. Otherwise, it is recommended for |
| 601 |
compatibility to implement :meth:`__getnewargs__`. |
| 602 |
|
| 603 |
.. versionchanged:: 3.6 |
| 604 |
:meth:`__getnewargs_ex__` is now used in protocols 2 and 3. |
| 605 |
|
| 606 |
|
| 607 |
.. method:: object.__getnewargs__() |
| 608 |
|
| 609 |
This method serves a similar purpose as :meth:`__getnewargs_ex__`, but |
| 610 |
supports only positional arguments. It must return a tuple of arguments |
| 611 |
``args`` which will be passed to the :meth:`__new__` method upon unpickling. |
| 612 |
|
| 613 |
:meth:`__getnewargs__` will not be called if :meth:`__getnewargs_ex__` is |
| 614 |
defined. |
| 615 |
|
| 616 |
.. versionchanged:: 3.6 |
| 617 |
Before Python 3.6, :meth:`__getnewargs__` was called instead of |
| 618 |
:meth:`__getnewargs_ex__` in protocols 2 and 3. |
| 619 |
|
| 620 |
|
| 621 |
.. method:: object.__getstate__() |
| 622 |
|
| 623 |
Classes can further influence how their instances are pickled by overriding |
| 624 |
the method :meth:`__getstate__`. It is called and the returned object |
| 625 |
is pickled as the contents for the instance, instead of a default state. |
| 626 |
There are several cases: |
| 627 |
|
| 628 |
* For a class that has no instance :attr:`~object.__dict__` and no |
| 629 |
:attr:`~object.__slots__`, the default state is ``None``. |
| 630 |
|
| 631 |
* For a class that has an instance :attr:`~object.__dict__` and no |
| 632 |
:attr:`~object.__slots__`, the default state is ``self.__dict__``. |
| 633 |
|
| 634 |
* For a class that has an instance :attr:`~object.__dict__` and |
| 635 |
:attr:`~object.__slots__`, the default state is a tuple consisting of two |
| 636 |
dictionaries: ``self.__dict__``, and a dictionary mapping slot |
| 637 |
names to slot values. Only slots that have a value are |
| 638 |
included in the latter. |
| 639 |
|
| 640 |
* For a class that has :attr:`~object.__slots__` and no instance |
| 641 |
:attr:`~object.__dict__`, the default state is a tuple whose first item |
| 642 |
is ``None`` and whose second item is a dictionary mapping slot names |
| 643 |
to slot values described in the previous bullet. |
| 644 |
|
| 645 |
.. versionchanged:: 3.11 |
| 646 |
Added the default implementation of the ``__getstate__()`` method in the |
| 647 |
:class:`object` class. |
| 648 |
|
| 649 |
|
| 650 |
.. method:: object.__setstate__(state) |
| 651 |
|
| 652 |
Upon unpickling, if the class defines :meth:`__setstate__`, it is called with |
| 653 |
the unpickled state. In that case, there is no requirement for the state |
| 654 |
object to be a dictionary. Otherwise, the pickled state must be a dictionary |
| 655 |
and its items are assigned to the new instance's dictionary. |
| 656 |
|
| 657 |
.. note:: |
| 658 |
|
| 659 |
If :meth:`__reduce__` returns a state with value ``None`` at pickling, |
| 660 |
the :meth:`__setstate__` method will not be called upon unpickling. |
| 661 |
|
| 662 |
|
| 663 |
Refer to the section :ref:`pickle-state` for more information about how to use |
| 664 |
the methods :meth:`~object.__getstate__` and :meth:`~object.__setstate__`. |
| 665 |
|
| 666 |
.. note:: |
| 667 |
|
| 668 |
At unpickling time, some methods like :meth:`~object.__getattr__`, |
| 669 |
:meth:`~object.__getattribute__`, or :meth:`~object.__setattr__` may be called upon the |
| 670 |
instance. In case those methods rely on some internal invariant being |
| 671 |
true, the type should implement :meth:`~object.__new__` to establish such an |
| 672 |
invariant, as :meth:`~object.__init__` is not called when unpickling an |
| 673 |
instance. |
| 674 |
|
| 675 |
.. index:: pair: copy; protocol |
| 676 |
|
| 677 |
As we shall see, pickle does not use directly the methods described above. In |
| 678 |
fact, these methods are part of the copy protocol which implements the |
| 679 |
:meth:`~object.__reduce__` special method. The copy protocol provides a unified |
| 680 |
interface for retrieving the data necessary for pickling and copying |
| 681 |
objects. [#]_ |
| 682 |
|
| 683 |
Although powerful, implementing :meth:`~object.__reduce__` directly in your classes is |
| 684 |
error prone. For this reason, class designers should use the high-level |
| 685 |
interface (i.e., :meth:`~object.__getnewargs_ex__`, :meth:`~object.__getstate__` and |
| 686 |
:meth:`~object.__setstate__`) whenever possible. We will show, however, cases where |
| 687 |
using :meth:`!__reduce__` is the only option or leads to more efficient pickling |
| 688 |
or both. |
| 689 |
|
| 690 |
.. method:: object.__reduce__() |
| 691 |
|
| 692 |
The interface is currently defined as follows. The :meth:`__reduce__` method |
| 693 |
takes no argument and shall return either a string or preferably a tuple (the |
| 694 |
returned object is often referred to as the "reduce value"). |
| 695 |
|
| 696 |
If a string is returned, the string should be interpreted as the name of a |
| 697 |
global variable. It should be the object's local name relative to its |
| 698 |
module; the pickle module searches the module namespace to determine the |
| 699 |
object's module: for a given ``obj`` to be pickled, the ``__module__`` |
| 700 |
attribute is looked up on ``obj`` directly, which falls back to a lookup |
| 701 |
on the type of ``obj`` if no ``__module__`` instance attribute is set. |
| 702 |
This behaviour is typically useful for singletons. |
| 703 |
|
| 704 |
When a tuple is returned, it must be between two and six items long. |
| 705 |
Optional items can either be omitted, or ``None`` can be provided as their |
| 706 |
value. The semantics of each item are in order: |
| 707 |
|
| 708 |
.. XXX Mention __newobj__ special-case? |
| 709 |
|
| 710 |
* A callable object that will be called to create the initial version of the |
| 711 |
object. |
| 712 |
|
| 713 |
* A tuple of arguments for the callable object. An empty tuple must be given |
| 714 |
if the callable does not accept any argument. |
| 715 |
|
| 716 |
* Optionally, the object's state, which will be passed to the object's |
| 717 |
:meth:`__setstate__` method as previously described. If the object has no |
| 718 |
such method then, the value must be a dictionary and it will be added to |
| 719 |
the object's :attr:`~object.__dict__` attribute. |
| 720 |
|
| 721 |
* Optionally, an iterator (and not a sequence) yielding successive items. |
| 722 |
These items will be appended to the object either using |
| 723 |
``obj.append(item)`` or, in batch, using ``obj.extend(list_of_items)``. |
| 724 |
This is primarily used for list subclasses, but may be used by other |
| 725 |
classes as long as they have :meth:`~sequence.append` |
| 726 |
and :meth:`~sequence.extend` methods with |
| 727 |
the appropriate signature. (Whether :meth:`!append` or :meth:`!extend` is |
| 728 |
used depends on which pickle protocol version is used as well as the number |
| 729 |
of items to append, so both must be supported.) |
| 730 |
|
| 731 |
* Optionally, an iterator (not a sequence) yielding successive key-value |
| 732 |
pairs. These items will be stored to the object using ``obj[key] = |
| 733 |
value``. This is primarily used for dictionary subclasses, but may be used |
| 734 |
by other classes as long as they implement :meth:`__setitem__`. |
| 735 |
|
| 736 |
* Optionally, a callable with a ``(obj, state)`` signature. This |
| 737 |
callable allows the user to programmatically control the state-updating |
| 738 |
behavior of a specific object, instead of using ``obj``'s static |
| 739 |
:meth:`__setstate__` method. If not ``None``, this callable will have |
| 740 |
priority over ``obj``'s :meth:`__setstate__`. |
| 741 |
|
| 742 |
.. versionadded:: 3.8 |
| 743 |
The optional sixth tuple item, ``(obj, state)``, was added. |
| 744 |
|
| 745 |
|
| 746 |
.. method:: object.__reduce_ex__(protocol) |
| 747 |
|
| 748 |
Alternatively, a :meth:`__reduce_ex__` method may be defined. The only |
| 749 |
difference is this method should take a single integer argument, the protocol |
| 750 |
version. When defined, pickle will prefer it over the :meth:`__reduce__` |
| 751 |
method. In addition, :meth:`__reduce__` automatically becomes a synonym for |
| 752 |
the extended version. The main use for this method is to provide |
| 753 |
backwards-compatible reduce values for older Python releases. |
| 754 |
|
| 755 |
.. currentmodule:: pickle |
| 756 |
|
| 757 |
.. _pickle-persistent: |
| 758 |
|
| 759 |
Persistence of External Objects |
| 760 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 761 |
|
| 762 |
.. index:: |
| 763 |
single: persistent_id (pickle protocol) |
| 764 |
single: persistent_load (pickle protocol) |
| 765 |
|
| 766 |
For the benefit of object persistence, the :mod:`!pickle` module supports the |
| 767 |
notion of a reference to an object outside the pickled data stream. Such |
| 768 |
objects are referenced by a persistent ID, which should be either a string of |
| 769 |
alphanumeric characters (for protocol 0) [#]_ or just an arbitrary object (for |
| 770 |
any newer protocol). |
| 771 |
|
| 772 |
The resolution of such persistent IDs is not defined by the :mod:`!pickle` |
| 773 |
module; it will delegate this resolution to the user-defined methods on the |
| 774 |
pickler and unpickler, :meth:`~Pickler.persistent_id` and |
| 775 |
:meth:`~Unpickler.persistent_load` respectively. |
| 776 |
|
| 777 |
To pickle objects that have an external persistent ID, the pickler must have a |
| 778 |
custom :meth:`~Pickler.persistent_id` method that takes an object as an |
| 779 |
argument and returns either ``None`` or the persistent ID for that object. |
| 780 |
When ``None`` is returned, the pickler simply pickles the object as normal. |
| 781 |
When a persistent ID string is returned, the pickler will pickle that object, |
| 782 |
along with a marker so that the unpickler will recognize it as a persistent ID. |
| 783 |
|
| 784 |
To unpickle external objects, the unpickler must have a custom |
| 785 |
:meth:`~Unpickler.persistent_load` method that takes a persistent ID object and |
| 786 |
returns the referenced object. |
| 787 |
|
| 788 |
Here is a comprehensive example presenting how persistent ID can be used to |
| 789 |
pickle external objects by reference. |
| 790 |
|
| 791 |
.. literalinclude:: ../includes/dbpickle.py |
| 792 |
|
| 793 |
.. _pickle-dispatch: |
| 794 |
|
| 795 |
Dispatch Tables |
| 796 |
^^^^^^^^^^^^^^^ |
| 797 |
|
| 798 |
If one wants to customize pickling of some classes without disturbing |
| 799 |
any other code which depends on pickling, then one can create a |
| 800 |
pickler with a private dispatch table. |
| 801 |
|
| 802 |
The global dispatch table managed by the :mod:`copyreg` module is |
| 803 |
available as :data:`!copyreg.dispatch_table`. Therefore, one may |
| 804 |
choose to use a modified copy of :data:`!copyreg.dispatch_table` as a |
| 805 |
private dispatch table. |
| 806 |
|
| 807 |
For example :: |
| 808 |
|
| 809 |
f = io.BytesIO() |
| 810 |
p = pickle.Pickler(f) |
| 811 |
p.dispatch_table = copyreg.dispatch_table.copy() |
| 812 |
p.dispatch_table[SomeClass] = reduce_SomeClass |
| 813 |
|
| 814 |
creates an instance of :class:`pickle.Pickler` with a private dispatch |
| 815 |
table which handles the ``SomeClass`` class specially. Alternatively, |
| 816 |
the code :: |
| 817 |
|
| 818 |
class MyPickler(pickle.Pickler): |
| 819 |
dispatch_table = copyreg.dispatch_table.copy() |
| 820 |
dispatch_table[SomeClass] = reduce_SomeClass |
| 821 |
f = io.BytesIO() |
| 822 |
p = MyPickler(f) |
| 823 |
|
| 824 |
does the same but all instances of ``MyPickler`` will by default |
| 825 |
share the private dispatch table. On the other hand, the code :: |
| 826 |
|
| 827 |
copyreg.pickle(SomeClass, reduce_SomeClass) |
| 828 |
f = io.BytesIO() |
| 829 |
p = pickle.Pickler(f) |
| 830 |
|
| 831 |
modifies the global dispatch table shared by all users of the :mod:`copyreg` module. |
| 832 |
|
| 833 |
.. _pickle-state: |
| 834 |
|
| 835 |
Handling Stateful Objects |
| 836 |
^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 837 |
|
| 838 |
.. index:: |
| 839 |
single: __getstate__() (copy protocol) |
| 840 |
single: __setstate__() (copy protocol) |
| 841 |
|
| 842 |
Here's an example that shows how to modify pickling behavior for a class. |
| 843 |
The :class:`!TextReader` class below opens a text file, and returns the line number and |
| 844 |
line contents each time its :meth:`!readline` method is called. If a |
| 845 |
:class:`!TextReader` instance is pickled, all attributes *except* the file object |
| 846 |
member are saved. When the instance is unpickled, the file is reopened, and |
| 847 |
reading resumes from the last location. The :meth:`!__setstate__` and |
| 848 |
:meth:`!__getstate__` methods are used to implement this behavior. :: |
| 849 |
|
| 850 |
class TextReader: |
| 851 |
"""Print and number lines in a text file.""" |
| 852 |
|
| 853 |
def __init__(self, filename): |
| 854 |
self.filename = filename |
| 855 |
self.file = open(filename) |
| 856 |
self.lineno = 0 |
| 857 |
|
| 858 |
def readline(self): |
| 859 |
self.lineno += 1 |
| 860 |
line = self.file.readline() |
| 861 |
if not line: |
| 862 |
return None |
| 863 |
if line.endswith('\n'): |
| 864 |
line = line[:-1] |
| 865 |
return "%i: %s" % (self.lineno, line) |
| 866 |
|
| 867 |
def __getstate__(self): |
| 868 |
# Copy the object's state from self.__dict__ which contains |
| 869 |
# all our instance attributes. Always use the dict.copy() |
| 870 |
# method to avoid modifying the original state. |
| 871 |
state = self.__dict__.copy() |
| 872 |
# Remove the unpicklable entries. |
| 873 |
del state['file'] |
| 874 |
return state |
| 875 |
|
| 876 |
def __setstate__(self, state): |
| 877 |
# Restore instance attributes (i.e., filename and lineno). |
| 878 |
self.__dict__.update(state) |
| 879 |
# Restore the previously opened file's state. To do so, we need to |
| 880 |
# reopen it and read from it until the line count is restored. |
| 881 |
file = open(self.filename) |
| 882 |
for _ in range(self.lineno): |
| 883 |
file.readline() |
| 884 |
# Finally, save the file. |
| 885 |
self.file = file |
| 886 |
|
| 887 |
|
| 888 |
A sample usage might be something like this:: |
| 889 |
|
| 890 |
>>> reader = TextReader("hello.txt") |
| 891 |
>>> reader.readline() |
| 892 |
'1: Hello world!' |
| 893 |
>>> reader.readline() |
| 894 |
'2: I am line number two.' |
| 895 |
>>> new_reader = pickle.loads(pickle.dumps(reader)) |
| 896 |
>>> new_reader.readline() |
| 897 |
'3: Goodbye!' |
| 898 |
|
| 899 |
.. _reducer_override: |
| 900 |
|
| 901 |
Custom Reduction for Types, Functions, and Other Objects |
| 902 |
-------------------------------------------------------- |
| 903 |
|
| 904 |
.. versionadded:: 3.8 |
| 905 |
|
| 906 |
Sometimes, :attr:`~Pickler.dispatch_table` may not be flexible enough. |
| 907 |
In particular we may want to customize pickling based on another criterion |
| 908 |
than the object's type, or we may want to customize the pickling of |
| 909 |
functions and classes. |
| 910 |
|
| 911 |
For those cases, it is possible to subclass from the :class:`Pickler` class and |
| 912 |
implement a :meth:`~Pickler.reducer_override` method. This method can return an |
| 913 |
arbitrary reduction tuple (see :meth:`~object.__reduce__`). It can alternatively return |
| 914 |
:data:`NotImplemented` to fallback to the traditional behavior. |
| 915 |
|
| 916 |
If both the :attr:`~Pickler.dispatch_table` and |
| 917 |
:meth:`~Pickler.reducer_override` are defined, then |
| 918 |
:meth:`~Pickler.reducer_override` method takes priority. |
| 919 |
|
| 920 |
.. Note:: |
| 921 |
For performance reasons, :meth:`~Pickler.reducer_override` may not be |
| 922 |
called for the following objects: ``None``, ``True``, ``False``, and |
| 923 |
exact instances of :class:`int`, :class:`float`, :class:`bytes`, |
| 924 |
:class:`str`, :class:`dict`, :class:`set`, :class:`frozenset`, :class:`list` |
| 925 |
and :class:`tuple`. |
| 926 |
|
| 927 |
Here is a simple example where we allow pickling and reconstructing |
| 928 |
a given class:: |
| 929 |
|
| 930 |
import io |
| 931 |
import pickle |
| 932 |
|
| 933 |
class MyClass: |
| 934 |
my_attribute = 1 |
| 935 |
|
| 936 |
class MyPickler(pickle.Pickler): |
| 937 |
def reducer_override(self, obj): |
| 938 |
"""Custom reducer for MyClass.""" |
| 939 |
if getattr(obj, "__name__", None) == "MyClass": |
| 940 |
return type, (obj.__name__, obj.__bases__, |
| 941 |
{'my_attribute': obj.my_attribute}) |
| 942 |
else: |
| 943 |
# For any other object, fallback to usual reduction |
| 944 |
return NotImplemented |
| 945 |
|
| 946 |
f = io.BytesIO() |
| 947 |
p = MyPickler(f) |
| 948 |
p.dump(MyClass) |
| 949 |
|
| 950 |
del MyClass |
| 951 |
|
| 952 |
unpickled_class = pickle.loads(f.getvalue()) |
| 953 |
|
| 954 |
assert isinstance(unpickled_class, type) |
| 955 |
assert unpickled_class.__name__ == "MyClass" |
| 956 |
assert unpickled_class.my_attribute == 1 |
| 957 |
|
| 958 |
|
| 959 |
.. _pickle-oob: |
| 960 |
|
| 961 |
Out-of-band Buffers |
| 962 |
------------------- |
| 963 |
|
| 964 |
.. versionadded:: 3.8 |
| 965 |
|
| 966 |
In some contexts, the :mod:`!pickle` module is used to transfer massive amounts |
| 967 |
of data. Therefore, it can be important to minimize the number of memory |
| 968 |
copies, to preserve performance and resource consumption. However, normal |
| 969 |
operation of the :mod:`!pickle` module, as it transforms a graph-like structure |
| 970 |
of objects into a sequential stream of bytes, intrinsically involves copying |
| 971 |
data to and from the pickle stream. |
| 972 |
|
| 973 |
This constraint can be eschewed if both the *provider* (the implementation |
| 974 |
of the object types to be transferred) and the *consumer* (the implementation |
| 975 |
of the communications system) support the out-of-band transfer facilities |
| 976 |
provided by pickle protocol 5 and higher. |
| 977 |
|
| 978 |
Provider API |
| 979 |
^^^^^^^^^^^^ |
| 980 |
|
| 981 |
The large data objects to be pickled must implement a :meth:`~object.__reduce_ex__` |
| 982 |
method specialized for protocol 5 and higher, which returns a |
| 983 |
:class:`PickleBuffer` instance (instead of e.g. a :class:`bytes` object) |
| 984 |
for any large data. |
| 985 |
|
| 986 |
A :class:`PickleBuffer` object *signals* that the underlying buffer is |
| 987 |
eligible for out-of-band data transfer. Those objects remain compatible |
| 988 |
with normal usage of the :mod:`!pickle` module. However, consumers can also |
| 989 |
opt-in to tell :mod:`!pickle` that they will handle those buffers by |
| 990 |
themselves. |
| 991 |
|
| 992 |
Consumer API |
| 993 |
^^^^^^^^^^^^ |
| 994 |
|
| 995 |
A communications system can enable custom handling of the :class:`PickleBuffer` |
| 996 |
objects generated when serializing an object graph. |
| 997 |
|
| 998 |
On the sending side, it needs to pass a *buffer_callback* argument to |
| 999 |
:class:`Pickler` (or to the :func:`dump` or :func:`dumps` function), which |
| 1000 |
will be called with each :class:`PickleBuffer` generated while pickling |
| 1001 |
the object graph. Buffers accumulated by the *buffer_callback* will not |
| 1002 |
see their data copied into the pickle stream, only a cheap marker will be |
| 1003 |
inserted. |
| 1004 |
|
| 1005 |
On the receiving side, it needs to pass a *buffers* argument to |
| 1006 |
:class:`Unpickler` (or to the :func:`load` or :func:`loads` function), |
| 1007 |
which is an iterable of the buffers which were passed to *buffer_callback*. |
| 1008 |
That iterable should produce buffers in the same order as they were passed |
| 1009 |
to *buffer_callback*. Those buffers will provide the data expected by the |
| 1010 |
reconstructors of the objects whose pickling produced the original |
| 1011 |
:class:`PickleBuffer` objects. |
| 1012 |
|
| 1013 |
Between the sending side and the receiving side, the communications system |
| 1014 |
is free to implement its own transfer mechanism for out-of-band buffers. |
| 1015 |
Potential optimizations include the use of shared memory or datatype-dependent |
| 1016 |
compression. |
| 1017 |
|
| 1018 |
Example |
| 1019 |
^^^^^^^ |
| 1020 |
|
| 1021 |
Here is a trivial example where we implement a :class:`bytearray` subclass |
| 1022 |
able to participate in out-of-band buffer pickling:: |
| 1023 |
|
| 1024 |
class ZeroCopyByteArray(bytearray): |
| 1025 |
|
| 1026 |
def __reduce_ex__(self, protocol): |
| 1027 |
if protocol >= 5: |
| 1028 |
return type(self)._reconstruct, (PickleBuffer(self),), None |
| 1029 |
else: |
| 1030 |
# PickleBuffer is forbidden with pickle protocols <= 4. |
| 1031 |
return type(self)._reconstruct, (bytearray(self),) |
| 1032 |
|
| 1033 |
@classmethod |
| 1034 |
def _reconstruct(cls, obj): |
| 1035 |
with memoryview(obj) as m: |
| 1036 |
# Get a handle over the original buffer object |
| 1037 |
obj = m.obj |
| 1038 |
if type(obj) is cls: |
| 1039 |
# Original buffer object is a ZeroCopyByteArray, return it |
| 1040 |
# as-is. |
| 1041 |
return obj |
| 1042 |
else: |
| 1043 |
return cls(obj) |
| 1044 |
|
| 1045 |
The reconstructor (the ``_reconstruct`` class method) returns the buffer's |
| 1046 |
providing object if it has the right type. This is an easy way to simulate |
| 1047 |
zero-copy behaviour on this toy example. |
| 1048 |
|
| 1049 |
On the consumer side, we can pickle those objects the usual way, which |
| 1050 |
when unserialized will give us a copy of the original object:: |
| 1051 |
|
| 1052 |
b = ZeroCopyByteArray(b"abc") |
| 1053 |
data = pickle.dumps(b, protocol=5) |
| 1054 |
new_b = pickle.loads(data) |
| 1055 |
print(b == new_b) # True |
| 1056 |
print(b is new_b) # False: a copy was made |
| 1057 |
|
| 1058 |
But if we pass a *buffer_callback* and then give back the accumulated |
| 1059 |
buffers when unserializing, we are able to get back the original object:: |
| 1060 |
|
| 1061 |
b = ZeroCopyByteArray(b"abc") |
| 1062 |
buffers = [] |
| 1063 |
data = pickle.dumps(b, protocol=5, buffer_callback=buffers.append) |
| 1064 |
new_b = pickle.loads(data, buffers=buffers) |
| 1065 |
print(b == new_b) # True |
| 1066 |
print(b is new_b) # True: no copy was made |
| 1067 |
|
| 1068 |
This example is limited by the fact that :class:`bytearray` allocates its |
| 1069 |
own memory: you cannot create a :class:`bytearray` instance that is backed |
| 1070 |
by another object's memory. However, third-party datatypes such as NumPy |
| 1071 |
arrays do not have this limitation, and allow use of zero-copy pickling |
| 1072 |
(or making as few copies as possible) when transferring between distinct |
| 1073 |
processes or systems. |
| 1074 |
|
| 1075 |
.. seealso:: :pep:`574` -- Pickle protocol 5 with out-of-band data |
| 1076 |
|
| 1077 |
|
| 1078 |
.. _pickle-restrict: |
| 1079 |
|
| 1080 |
Restricting Globals |
| 1081 |
------------------- |
| 1082 |
|
| 1083 |
.. index:: |
| 1084 |
single: find_class() (pickle protocol) |
| 1085 |
|
| 1086 |
By default, unpickling will import any class or function that it finds in the |
| 1087 |
pickle data. For many applications, this behaviour is unacceptable as it |
| 1088 |
permits the unpickler to import and invoke arbitrary code. Just consider what |
| 1089 |
this hand-crafted pickle data stream does when loaded:: |
| 1090 |
|
| 1091 |
>>> import pickle |
| 1092 |
>>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.") |
| 1093 |
hello world |
| 1094 |
0 |
| 1095 |
|
| 1096 |
In this example, the unpickler imports the :func:`os.system` function and then |
| 1097 |
apply the string argument "echo hello world". Although this example is |
| 1098 |
inoffensive, it is not difficult to imagine one that could damage your system. |
| 1099 |
|
| 1100 |
For this reason, you may want to control what gets unpickled by customizing |
| 1101 |
:meth:`Unpickler.find_class`. Unlike its name suggests, |
| 1102 |
:meth:`Unpickler.find_class` is called whenever a global (i.e., a class or |
| 1103 |
a function) is requested. Thus it is possible to either completely forbid |
| 1104 |
globals or restrict them to a safe subset. |
| 1105 |
|
| 1106 |
Here is an example of an unpickler allowing only few safe classes from the |
| 1107 |
:mod:`builtins` module to be loaded:: |
| 1108 |
|
| 1109 |
import builtins |
| 1110 |
import io |
| 1111 |
import pickle |
| 1112 |
|
| 1113 |
safe_builtins = { |
| 1114 |
'range', |
| 1115 |
'complex', |
| 1116 |
'set', |
| 1117 |
'frozenset', |
| 1118 |
'slice', |
| 1119 |
} |
| 1120 |
|
| 1121 |
class RestrictedUnpickler(pickle.Unpickler): |
| 1122 |
|
| 1123 |
def find_class(self, module, name): |
| 1124 |
# Only allow safe classes from builtins. |
| 1125 |
if module == "builtins" and name in safe_builtins: |
| 1126 |
return getattr(builtins, name) |
| 1127 |
# Forbid everything else. |
| 1128 |
raise pickle.UnpicklingError("global '%s.%s' is forbidden" % |
| 1129 |
(module, name)) |
| 1130 |
|
| 1131 |
def restricted_loads(s): |
| 1132 |
"""Helper function analogous to pickle.loads().""" |
| 1133 |
return RestrictedUnpickler(io.BytesIO(s)).load() |
| 1134 |
|
| 1135 |
A sample usage of our unpickler working as intended:: |
| 1136 |
|
| 1137 |
>>> restricted_loads(pickle.dumps([1, 2, range(15)])) |
| 1138 |
[1, 2, range(0, 15)] |
| 1139 |
>>> restricted_loads(b"cos\nsystem\n(S'echo hello world'\ntR.") |
| 1140 |
Traceback (most recent call last): |
| 1141 |
... |
| 1142 |
pickle.UnpicklingError: global 'os.system' is forbidden |
| 1143 |
>>> restricted_loads(b'cbuiltins\neval\n' |
| 1144 |
... b'(S\'getattr(__import__("os"), "system")' |
| 1145 |
... b'("echo hello world")\'\ntR.') |
| 1146 |
Traceback (most recent call last): |
| 1147 |
... |
| 1148 |
pickle.UnpicklingError: global 'builtins.eval' is forbidden |
| 1149 |
|
| 1150 |
|
| 1151 |
.. XXX Add note about how extension codes could evade our protection |
| 1152 |
mechanism (e.g. cached classes do not invokes find_class()). |
| 1153 |
|
| 1154 |
As our examples shows, you have to be careful with what you allow to be |
| 1155 |
unpickled. Therefore if security is a concern, you may want to consider |
| 1156 |
alternatives such as the marshalling API in :mod:`xmlrpc.client` or |
| 1157 |
third-party solutions. |
| 1158 |
|
| 1159 |
|
| 1160 |
Performance |
| 1161 |
----------- |
| 1162 |
|
| 1163 |
Recent versions of the pickle protocol (from protocol 2 and upwards) feature |
| 1164 |
efficient binary encodings for several common features and built-in types. |
| 1165 |
Also, the :mod:`!pickle` module has a transparent optimizer written in C. |
| 1166 |
|
| 1167 |
|
| 1168 |
.. _pickle-example: |
| 1169 |
|
| 1170 |
Examples |
| 1171 |
-------- |
| 1172 |
|
| 1173 |
For the simplest code, use the :func:`dump` and :func:`load` functions. :: |
| 1174 |
|
| 1175 |
import pickle |
| 1176 |
|
| 1177 |
# An arbitrary collection of objects supported by pickle. |
| 1178 |
data = { |
| 1179 |
'a': [1, 2.0, 3+4j], |
| 1180 |
'b': ("character string", b"byte string"), |
| 1181 |
'c': {None, True, False} |
| 1182 |
} |
| 1183 |
|
| 1184 |
with open('data.pickle', 'wb') as f: |
| 1185 |
# Pickle the 'data' dictionary using the highest protocol available. |
| 1186 |
pickle.dump(data, f, pickle.HIGHEST_PROTOCOL) |
| 1187 |
|
| 1188 |
|
| 1189 |
The following example reads the resulting pickled data. :: |
| 1190 |
|
| 1191 |
import pickle |
| 1192 |
|
| 1193 |
with open('data.pickle', 'rb') as f: |
| 1194 |
# The protocol version used is detected automatically, so we do not |
| 1195 |
# have to specify it. |
| 1196 |
data = pickle.load(f) |
| 1197 |
|
| 1198 |
|
| 1199 |
.. XXX: Add examples showing how to optimize pickles for size (like using |
| 1200 |
.. pickletools.optimize() or the gzip module). |
| 1201 |
|
| 1202 |
|
| 1203 |
.. _pickle-cli: |
| 1204 |
|
| 1205 |
Command-line interface |
| 1206 |
---------------------- |
| 1207 |
|
| 1208 |
The :mod:`!pickle` module can be invoked as a script from the command line, |
| 1209 |
it will display contents of the pickle files. However, when the pickle file |
| 1210 |
that you want to examine comes from an untrusted source, ``-m pickletools`` |
| 1211 |
is a safer option because it does not execute pickle bytecode, see |
| 1212 |
:ref:`pickletools CLI usage <pickletools-cli>`. |
| 1213 |
|
| 1214 |
.. code-block:: bash |
| 1215 |
|
| 1216 |
python -m pickle pickle_file [pickle_file ...] |
| 1217 |
|
| 1218 |
The following option is accepted: |
| 1219 |
|
| 1220 |
.. program:: pickle |
| 1221 |
|
| 1222 |
.. option:: pickle_file |
| 1223 |
|
| 1224 |
A pickle file to read, or ``-`` to indicate reading from standard input. |
| 1225 |
|
| 1226 |
|
| 1227 |
.. seealso:: |
| 1228 |
|
| 1229 |
Module :mod:`copyreg` |
| 1230 |
Pickle interface constructor registration for extension types. |
| 1231 |
|
| 1232 |
Module :mod:`pickletools` |
| 1233 |
Tools for working with and analyzing pickled data. |
| 1234 |
|
| 1235 |
Module :mod:`shelve` |
| 1236 |
Indexed databases of objects; uses :mod:`!pickle`. |
| 1237 |
|
| 1238 |
Module :mod:`copy` |
| 1239 |
Shallow and deep object copying. |
| 1240 |
|
| 1241 |
Module :mod:`marshal` |
| 1242 |
High-performance serialization of built-in types. |
| 1243 |
|
| 1244 |
|
| 1245 |
.. rubric:: Footnotes |
| 1246 |
|
| 1247 |
.. [#] Don't confuse this with the :mod:`marshal` module |
| 1248 |
|
| 1249 |
.. [#] This is why :keyword:`lambda` functions cannot be pickled: all |
| 1250 |
:keyword:`!lambda` functions share the same name: ``<lambda>``. |
| 1251 |
|
| 1252 |
.. [#] The exception raised will likely be an :exc:`ImportError` or an |
| 1253 |
:exc:`AttributeError` but it could be something else. |
| 1254 |
|
| 1255 |
.. [#] The :mod:`copy` module uses this protocol for shallow and deep copying |
| 1256 |
operations. |
| 1257 |
|
| 1258 |
.. [#] The limitation on alphanumeric characters is due to the fact |
| 1259 |
that persistent IDs in protocol 0 are delimited by the newline |
| 1260 |
character. Therefore if any kind of newline characters occurs in |
| 1261 |
persistent IDs, the resulting pickled data will become unreadable. |