| 1 |
:mod:`!shelve` --- Python object persistence |
| 2 |
============================================ |
| 3 |
|
| 4 |
.. module:: shelve |
| 5 |
:synopsis: Python object persistence. |
| 6 |
|
| 7 |
**Source code:** :source:`Lib/shelve.py` |
| 8 |
|
| 9 |
.. index:: pair: module; pickle |
| 10 |
|
| 11 |
-------------- |
| 12 |
|
| 13 |
A "shelf" is a persistent, dictionary-like object. The difference with "dbm" |
| 14 |
databases is that the values (not the keys!) in a shelf can be essentially |
| 15 |
arbitrary Python objects --- anything that the :mod:`pickle` module can handle. |
| 16 |
This includes most class instances, recursive data types, and objects containing |
| 17 |
lots of shared sub-objects. The keys are ordinary strings. |
| 18 |
|
| 19 |
|
| 20 |
.. function:: open(filename, flag='c', protocol=None, writeback=False) |
| 21 |
|
| 22 |
Open a persistent dictionary. The filename specified is the base filename for |
| 23 |
the underlying database. As a side-effect, an extension may be added to the |
| 24 |
filename and more than one file may be created. By default, the underlying |
| 25 |
database file is opened for reading and writing. The optional *flag* parameter |
| 26 |
has the same interpretation as the *flag* parameter of :func:`dbm.open`. |
| 27 |
|
| 28 |
By default, pickles created with :const:`pickle.DEFAULT_PROTOCOL` are used |
| 29 |
to serialize values. The version of the pickle protocol can be specified |
| 30 |
with the *protocol* parameter. |
| 31 |
|
| 32 |
Because of Python semantics, a shelf cannot know when a mutable |
| 33 |
persistent-dictionary entry is modified. By default modified objects are |
| 34 |
written *only* when assigned to the shelf (see :ref:`shelve-example`). If the |
| 35 |
optional *writeback* parameter is set to ``True``, all entries accessed are also |
| 36 |
cached in memory, and written back on :meth:`~Shelf.sync` and |
| 37 |
:meth:`~Shelf.close`; this can make it handier to mutate mutable entries in |
| 38 |
the persistent dictionary, but, if many entries are accessed, it can consume |
| 39 |
vast amounts of memory for the cache, and it can make the close operation |
| 40 |
very slow since all accessed entries are written back (there is no way to |
| 41 |
determine which accessed entries are mutable, nor which ones were actually |
| 42 |
mutated). |
| 43 |
|
| 44 |
.. versionchanged:: 3.10 |
| 45 |
:const:`pickle.DEFAULT_PROTOCOL` is now used as the default pickle |
| 46 |
protocol. |
| 47 |
|
| 48 |
.. versionchanged:: 3.11 |
| 49 |
Accepts :term:`path-like object` for filename. |
| 50 |
|
| 51 |
.. note:: |
| 52 |
|
| 53 |
Do not rely on the shelf being closed automatically; always call |
| 54 |
:meth:`~Shelf.close` explicitly when you don't need it any more, or |
| 55 |
use :func:`shelve.open` as a context manager:: |
| 56 |
|
| 57 |
with shelve.open('spam') as db: |
| 58 |
db['eggs'] = 'eggs' |
| 59 |
|
| 60 |
.. _shelve-security: |
| 61 |
|
| 62 |
.. warning:: |
| 63 |
|
| 64 |
Because the :mod:`!shelve` module is backed by :mod:`pickle`, it is insecure |
| 65 |
to load a shelf from an untrusted source. Like with pickle, loading a shelf |
| 66 |
can execute arbitrary code. |
| 67 |
|
| 68 |
Shelf objects support most of the methods and operations supported by dictionaries |
| 69 |
(except copying, constructors and operators ``|`` and ``|=``). This eases the |
| 70 |
transition from dictionary based scripts to those requiring persistent storage. |
| 71 |
|
| 72 |
Two additional methods are supported: |
| 73 |
|
| 74 |
.. method:: Shelf.sync() |
| 75 |
|
| 76 |
Write back all entries in the cache if the shelf was opened with *writeback* |
| 77 |
set to :const:`True`. Also empty the cache and synchronize the persistent |
| 78 |
dictionary on disk, if feasible. This is called automatically when the shelf |
| 79 |
is closed with :meth:`close`. |
| 80 |
|
| 81 |
.. method:: Shelf.close() |
| 82 |
|
| 83 |
Synchronize and close the persistent *dict* object. Operations on a closed |
| 84 |
shelf will fail with a :exc:`ValueError`. |
| 85 |
|
| 86 |
|
| 87 |
.. seealso:: |
| 88 |
|
| 89 |
`Persistent dictionary recipe <https://code.activestate.com/recipes/576642-persistent-dict-with-multiple-standard-file-format/>`_ |
| 90 |
with widely supported storage formats and having the speed of native |
| 91 |
dictionaries. |
| 92 |
|
| 93 |
|
| 94 |
Restrictions |
| 95 |
------------ |
| 96 |
|
| 97 |
.. index:: |
| 98 |
pair: module; dbm.ndbm |
| 99 |
pair: module; dbm.gnu |
| 100 |
|
| 101 |
* The choice of which database package will be used (such as :mod:`dbm.ndbm` or |
| 102 |
:mod:`dbm.gnu`) depends on which interface is available. Therefore it is not |
| 103 |
safe to open the database directly using :mod:`dbm`. The database is also |
| 104 |
(unfortunately) subject to the limitations of :mod:`dbm`, if it is used --- |
| 105 |
this means that (the pickled representation of) the objects stored in the |
| 106 |
database should be fairly small, and in rare cases key collisions may cause |
| 107 |
the database to refuse updates. |
| 108 |
|
| 109 |
* The :mod:`!shelve` module does not support *concurrent* read/write access to |
| 110 |
shelved objects. (Multiple simultaneous read accesses are safe.) When a |
| 111 |
program has a shelf open for writing, no other program should have it open for |
| 112 |
reading or writing. Unix file locking can be used to solve this, but this |
| 113 |
differs across Unix versions and requires knowledge about the database |
| 114 |
implementation used. |
| 115 |
|
| 116 |
* On macOS :mod:`dbm.ndbm` can silently corrupt the database file on updates, |
| 117 |
which can cause hard crashes when trying to read from the database. |
| 118 |
|
| 119 |
|
| 120 |
.. class:: Shelf(dict, protocol=None, writeback=False, keyencoding='utf-8') |
| 121 |
|
| 122 |
A subclass of :class:`collections.abc.MutableMapping` which stores pickled |
| 123 |
values in the *dict* object. |
| 124 |
|
| 125 |
By default, pickles created with :const:`pickle.DEFAULT_PROTOCOL` are used |
| 126 |
to serialize values. The version of the pickle protocol can be specified |
| 127 |
with the *protocol* parameter. See the :mod:`pickle` documentation for a |
| 128 |
discussion of the pickle protocols. |
| 129 |
|
| 130 |
If the *writeback* parameter is ``True``, the object will hold a cache of all |
| 131 |
entries accessed and write them back to the *dict* at sync and close times. |
| 132 |
This allows natural operations on mutable entries, but can consume much more |
| 133 |
memory and make sync and close take a long time. |
| 134 |
|
| 135 |
The *keyencoding* parameter is the encoding used to encode keys before they |
| 136 |
are used with the underlying dict. |
| 137 |
|
| 138 |
A :class:`Shelf` object can also be used as a context manager, in which |
| 139 |
case it will be automatically closed when the :keyword:`with` block ends. |
| 140 |
|
| 141 |
.. versionchanged:: 3.2 |
| 142 |
Added the *keyencoding* parameter; previously, keys were always encoded in |
| 143 |
UTF-8. |
| 144 |
|
| 145 |
.. versionchanged:: 3.4 |
| 146 |
Added context manager support. |
| 147 |
|
| 148 |
.. versionchanged:: 3.10 |
| 149 |
:const:`pickle.DEFAULT_PROTOCOL` is now used as the default pickle |
| 150 |
protocol. |
| 151 |
|
| 152 |
|
| 153 |
.. class:: BsdDbShelf(dict, protocol=None, writeback=False, keyencoding='utf-8') |
| 154 |
|
| 155 |
A subclass of :class:`Shelf` which exposes :meth:`!first`, :meth:`!next`, |
| 156 |
:meth:`!previous`, :meth:`!last` and :meth:`!set_location` methods. |
| 157 |
These are available |
| 158 |
in the third-party :mod:`!bsddb` module from `pybsddb |
| 159 |
<https://www.jcea.es/programacion/pybsddb.htm>`_ but not in other database |
| 160 |
modules. The *dict* object passed to the constructor must support those |
| 161 |
methods. This is generally accomplished by calling one of |
| 162 |
:func:`!bsddb.hashopen`, :func:`!bsddb.btopen` or :func:`!bsddb.rnopen`. The |
| 163 |
optional *protocol*, *writeback*, and *keyencoding* parameters have the same |
| 164 |
interpretation as for the :class:`Shelf` class. |
| 165 |
|
| 166 |
|
| 167 |
.. class:: DbfilenameShelf(filename, flag='c', protocol=None, writeback=False) |
| 168 |
|
| 169 |
A subclass of :class:`Shelf` which accepts a *filename* instead of a dict-like |
| 170 |
object. The underlying file will be opened using :func:`dbm.open`. By |
| 171 |
default, the file will be created and opened for both read and write. The |
| 172 |
optional *flag* parameter has the same interpretation as for the :func:`.open` |
| 173 |
function. The optional *protocol* and *writeback* parameters have the same |
| 174 |
interpretation as for the :class:`Shelf` class. |
| 175 |
|
| 176 |
|
| 177 |
.. _shelve-example: |
| 178 |
|
| 179 |
Example |
| 180 |
------- |
| 181 |
|
| 182 |
To summarize the interface (``key`` is a string, ``data`` is an arbitrary |
| 183 |
object):: |
| 184 |
|
| 185 |
import shelve |
| 186 |
|
| 187 |
d = shelve.open(filename) # open -- file may get suffix added by low-level |
| 188 |
# library |
| 189 |
|
| 190 |
d[key] = data # store data at key (overwrites old data if |
| 191 |
# using an existing key) |
| 192 |
data = d[key] # retrieve a COPY of data at key (raise KeyError |
| 193 |
# if no such key) |
| 194 |
del d[key] # delete data stored at key (raises KeyError |
| 195 |
# if no such key) |
| 196 |
|
| 197 |
flag = key in d # true if the key exists |
| 198 |
klist = list(d.keys()) # a list of all existing keys (slow!) |
| 199 |
|
| 200 |
# as d was opened WITHOUT writeback=True, beware: |
| 201 |
d['xx'] = [0, 1, 2] # this works as expected, but... |
| 202 |
d['xx'].append(3) # *this doesn't!* -- d['xx'] is STILL [0, 1, 2]! |
| 203 |
|
| 204 |
# having opened d without writeback=True, you need to code carefully: |
| 205 |
temp = d['xx'] # extracts the copy |
| 206 |
temp.append(5) # mutates the copy |
| 207 |
d['xx'] = temp # stores the copy right back, to persist it |
| 208 |
|
| 209 |
# or, d=shelve.open(filename,writeback=True) would let you just code |
| 210 |
# d['xx'].append(5) and have it work as expected, BUT it would also |
| 211 |
# consume more memory and make the d.close() operation slower. |
| 212 |
|
| 213 |
d.close() # close it |
| 214 |
|
| 215 |
|
| 216 |
.. seealso:: |
| 217 |
|
| 218 |
Module :mod:`dbm` |
| 219 |
Generic interface to ``dbm``-style databases. |
| 220 |
|
| 221 |
Module :mod:`pickle` |
| 222 |
Object serialization used by :mod:`!shelve`. |
| 223 |
|