Debian Codegraph @

Doc/library/weakref.rst

raw
1 .. _mod-weakref:
2
3 :mod:`!weakref` --- Weak references
4 ===================================
5
6 .. module:: weakref
7 :synopsis: Support for weak references and weak dictionaries.
8
9 .. moduleauthor:: Fred L. Drake, Jr. <[email protected]>
10 .. moduleauthor:: Neil Schemenauer <[email protected]>
11 .. moduleauthor:: Martin von Löwis <[email protected]>
12 .. sectionauthor:: Fred L. Drake, Jr. <[email protected]>
13
14 **Source code:** :source:`Lib/weakref.py`
15
16 --------------
17
18 The :mod:`!weakref` module allows the Python programmer to create :dfn:`weak
19 references` to objects.
20
21 .. When making changes to the examples in this file, be sure to update
22 Lib/test/test_weakref.py::libreftest too!
23
24 In the following, the term :dfn:`referent` means the object which is referred to
25 by a weak reference.
26
27 A weak reference to an object is not enough to keep the object alive: when the
28 only remaining references to a referent are weak references,
29 :term:`garbage collection` is free to destroy the referent and reuse its memory
30 for something else. However, until the object is actually destroyed the weak
31 reference may return the object even if there are no strong references to it.
32
33 A primary use for weak references is to implement caches or
34 mappings holding large objects, where it's desired that a large object not be
35 kept alive solely because it appears in a cache or mapping.
36
37 For example, if you have a number of large binary image objects, you may wish to
38 associate a name with each. If you used a Python dictionary to map names to
39 images, or images to names, the image objects would remain alive just because
40 they appeared as values or keys in the dictionaries. The
41 :class:`WeakKeyDictionary` and :class:`WeakValueDictionary` classes supplied by
42 the :mod:`!weakref` module are an alternative, using weak references to construct
43 mappings that don't keep objects alive solely because they appear in the mapping
44 objects. If, for example, an image object is a value in a
45 :class:`WeakValueDictionary`, then when the last remaining references to that
46 image object are the weak references held by weak mappings, garbage collection
47 can reclaim the object, and its corresponding entries in weak mappings are
48 simply deleted.
49
50 :class:`WeakKeyDictionary` and :class:`WeakValueDictionary` use weak references
51 in their implementation, setting up callback functions on the weak references
52 that notify the weak dictionaries when a key or value has been reclaimed by
53 garbage collection. :class:`WeakSet` implements the :class:`set` interface,
54 but keeps weak references to its elements, just like a
55 :class:`WeakKeyDictionary` does.
56
57 :class:`finalize` provides a straight forward way to register a
58 cleanup function to be called when an object is garbage collected.
59 This is simpler to use than setting up a callback function on a raw
60 weak reference, since the module automatically ensures that the finalizer
61 remains alive until the object is collected.
62
63 Most programs should find that using one of these weak container types
64 or :class:`finalize` is all they need -- it's not usually necessary to
65 create your own weak references directly. The low-level machinery is
66 exposed by the :mod:`!weakref` module for the benefit of advanced uses.
67
68 Not all objects can be weakly referenced. Objects which support weak references
69 include class instances, functions written in Python (but not in C), instance methods,
70 sets, frozensets, some :term:`file objects <file object>`, :term:`generators <generator>`,
71 type objects, sockets, arrays, deques, regular expression pattern objects, and code
72 objects.
73
74 .. versionchanged:: 3.2
75 Added support for thread.lock, threading.Lock, and code objects.
76
77 Several built-in types such as :class:`list` and :class:`dict` do not directly
78 support weak references but can add support through subclassing::
79
80 class Dict(dict):
81 pass
82
83 obj = Dict(red=1, green=2, blue=3) # this object is weak referenceable
84
85 .. impl-detail::
86
87 Other built-in types such as :class:`tuple` and :class:`int` do not support weak
88 references even when subclassed.
89
90 Extension types can easily be made to support weak references; see
91 :ref:`weakref-support`.
92
93 When ``__slots__`` are defined for a given type, weak reference support is
94 disabled unless a ``'__weakref__'`` string is also present in the sequence of
95 strings in the ``__slots__`` declaration.
96 See :ref:`__slots__ documentation <slots>` for details.
97
98 .. class:: ref(object[, callback])
99
100 Return a weak reference to *object*. The original object can be retrieved by
101 calling the reference object if the referent is still alive; if the referent is
102 no longer alive, calling the reference object will cause :const:`None` to be
103 returned. If *callback* is provided and not :const:`None`, and the returned
104 weakref object is still alive, the callback will be called when the object is
105 about to be finalized; the weak reference object will be passed as the only
106 parameter to the callback; the referent will no longer be available.
107
108 It is allowable for many weak references to be constructed for the same object.
109 Callbacks registered for each weak reference will be called from the most
110 recently registered callback to the oldest registered callback.
111
112 Exceptions raised by the callback will be noted on the standard error output,
113 but cannot be propagated; they are handled in exactly the same way as exceptions
114 raised from an object's :meth:`~object.__del__` method.
115
116 Weak references are :term:`hashable` if the *object* is hashable. They will
117 maintain their hash value even after the *object* was deleted. If
118 :func:`hash` is called the first time only after the *object* was deleted,
119 the call will raise :exc:`TypeError`.
120
121 Weak references support tests for equality, but not ordering. If the referents
122 are still alive, two references have the same equality relationship as their
123 referents (regardless of the *callback*). If either referent has been deleted,
124 the references are equal only if the reference objects are the same object.
125
126 This is a subclassable type rather than a factory function.
127
128 .. attribute:: __callback__
129
130 This read-only attribute returns the callback currently associated to the
131 weakref. If there is no callback or if the referent of the weakref is
132 no longer alive then this attribute will have value ``None``.
133
134 .. versionchanged:: 3.4
135 Added the :attr:`__callback__` attribute.
136
137
138 .. function:: proxy(object[, callback])
139
140 Return a proxy to *object* which uses a weak reference. This supports use of
141 the proxy in most contexts instead of requiring the explicit dereferencing used
142 with weak reference objects. The returned object will have a type of either
143 ``ProxyType`` or ``CallableProxyType``, depending on whether *object* is
144 callable. Proxy objects are not :term:`hashable` regardless of the referent; this
145 avoids a number of problems related to their fundamentally mutable nature, and
146 prevents their use as dictionary keys. *callback* is the same as the parameter
147 of the same name to the :func:`ref` function.
148
149 Accessing an attribute of the proxy object after the referent is
150 garbage collected raises :exc:`ReferenceError`.
151
152 .. versionchanged:: 3.8
153 Extended the operator support on proxy objects to include the matrix
154 multiplication operators ``@`` and ``@=``.
155
156
157 .. function:: getweakrefcount(object)
158
159 Return the number of weak references and proxies which refer to *object*.
160
161
162 .. function:: getweakrefs(object)
163
164 Return a list of all weak reference and proxy objects which refer to *object*.
165
166
167 .. class:: WeakKeyDictionary([dict])
168
169 Mapping class that references keys weakly. Entries in the dictionary will be
170 discarded when there is no longer a strong reference to the key. This can be
171 used to associate additional data with an object owned by other parts of an
172 application without adding attributes to those objects. This can be especially
173 useful with objects that override attribute accesses.
174
175 Note that when a key with equal value to an existing key (but not equal identity)
176 is inserted into the dictionary, it replaces the value but does not replace the
177 existing key. Due to this, when the reference to the original key is deleted, it
178 also deletes the entry in the dictionary::
179
180 >>> class T(str): pass
181 ...
182 >>> k1, k2 = T(), T()
183 >>> d = weakref.WeakKeyDictionary()
184 >>> d[k1] = 1 # d = {k1: 1}
185 >>> d[k2] = 2 # d = {k1: 2}
186 >>> del k1 # d = {}
187
188 A workaround would be to remove the key prior to reassignment::
189
190 >>> class T(str): pass
191 ...
192 >>> k1, k2 = T(), T()
193 >>> d = weakref.WeakKeyDictionary()
194 >>> d[k1] = 1 # d = {k1: 1}
195 >>> del d[k1]
196 >>> d[k2] = 2 # d = {k2: 2}
197 >>> del k1 # d = {k2: 2}
198
199 .. versionchanged:: 3.9
200 Added support for ``|`` and ``|=`` operators, as specified in :pep:`584`.
201
202 :class:`WeakKeyDictionary` objects have an additional method that
203 exposes the internal references directly. The references are not guaranteed to
204 be "live" at the time they are used, so the result of calling the references
205 needs to be checked before being used. This can be used to avoid creating
206 references that will cause the garbage collector to keep the keys around longer
207 than needed.
208
209
210 .. method:: WeakKeyDictionary.keyrefs()
211
212 Return an iterable of the weak references to the keys.
213
214
215 .. class:: WeakValueDictionary([dict])
216
217 Mapping class that references values weakly. Entries in the dictionary will be
218 discarded when no strong reference to the value exists any more.
219
220 .. versionchanged:: 3.9
221 Added support for ``|`` and ``|=`` operators, as specified in :pep:`584`.
222
223 :class:`WeakValueDictionary` objects have an additional method that has the
224 same issues as the :meth:`WeakKeyDictionary.keyrefs` method.
225
226
227 .. method:: WeakValueDictionary.valuerefs()
228
229 Return an iterable of the weak references to the values.
230
231
232 .. class:: WeakSet([elements])
233
234 Set class that keeps weak references to its elements. An element will be
235 discarded when no strong reference to it exists any more.
236
237
238 .. class:: WeakMethod(method[, callback])
239
240 A custom :class:`ref` subclass which simulates a weak reference to a bound
241 method (i.e., a method defined on a class and looked up on an instance).
242 Since a bound method is ephemeral, a standard weak reference cannot keep
243 hold of it. :class:`WeakMethod` has special code to recreate the bound
244 method until either the object or the original function dies::
245
246 >>> class C:
247 ... def method(self):
248 ... print("method called!")
249 ...
250 >>> c = C()
251 >>> r = weakref.ref(c.method)
252 >>> r()
253 >>> r = weakref.WeakMethod(c.method)
254 >>> r()
255 <bound method C.method of <__main__.C object at 0x7fc859830220>>
256 >>> r()()
257 method called!
258 >>> del c
259 >>> gc.collect()
260 0
261 >>> r()
262 >>>
263
264 *callback* is the same as the parameter of the same name to the :func:`ref` function.
265
266 .. versionadded:: 3.4
267
268 .. class:: finalize(obj, func, /, *args, **kwargs)
269
270 Return a callable finalizer object which will be called when *obj*
271 is garbage collected. Unlike an ordinary weak reference, a finalizer
272 will always survive until the reference object is collected, greatly
273 simplifying lifecycle management.
274
275 A finalizer is considered *alive* until it is called (either explicitly
276 or at garbage collection), and after that it is *dead*. Calling a live
277 finalizer returns the result of evaluating ``func(*arg, **kwargs)``,
278 whereas calling a dead finalizer returns :const:`None`.
279
280 Exceptions raised by finalizer callbacks during garbage collection
281 will be shown on the standard error output, but cannot be
282 propagated. They are handled in the same way as exceptions raised
283 from an object's :meth:`~object.__del__` method or a weak reference's
284 callback.
285
286 When the program exits, each remaining live finalizer is called
287 unless its :attr:`atexit` attribute has been set to false. They
288 are called in reverse order of creation.
289
290 A finalizer will never invoke its callback during the later part of
291 the :term:`interpreter shutdown` when module globals are liable to have
292 been replaced by :const:`None`.
293
294 .. method:: __call__()
295
296 If *self* is alive then mark it as dead and return the result of
297 calling ``func(*args, **kwargs)``. If *self* is dead then return
298 :const:`None`.
299
300 .. method:: detach()
301
302 If *self* is alive then mark it as dead and return the tuple
303 ``(obj, func, args, kwargs)``. If *self* is dead then return
304 :const:`None`.
305
306 .. method:: peek()
307
308 If *self* is alive then return the tuple ``(obj, func, args,
309 kwargs)``. If *self* is dead then return :const:`None`.
310
311 .. attribute:: alive
312
313 Property which is true if the finalizer is alive, false otherwise.
314
315 .. attribute:: atexit
316
317 A writable boolean property which by default is true. When the
318 program exits, it calls all remaining live finalizers for which
319 :attr:`.atexit` is true. They are called in reverse order of
320 creation.
321
322 .. note::
323
324 It is important to ensure that *func*, *args* and *kwargs* do
325 not own any references to *obj*, either directly or indirectly,
326 since otherwise *obj* will never be garbage collected. In
327 particular, *func* should not be a bound method of *obj*.
328
329 .. versionadded:: 3.4
330
331
332 .. data:: ReferenceType
333
334 The type object for weak references objects.
335
336
337 .. data:: ProxyType
338
339 The type object for proxies of objects which are not callable.
340
341
342 .. data:: CallableProxyType
343
344 The type object for proxies of callable objects.
345
346
347 .. data:: ProxyTypes
348
349 Sequence containing all the type objects for proxies. This can make it simpler
350 to test if an object is a proxy without being dependent on naming both proxy
351 types.
352
353
354 .. seealso::
355
356 :pep:`205` - Weak References
357 The proposal and rationale for this feature, including links to earlier
358 implementations and information about similar features in other languages.
359
360
361 .. _weakref-objects:
362
363 Weak Reference Objects
364 ----------------------
365
366 Weak reference objects have no methods and no attributes besides
367 :attr:`ref.__callback__`. A weak reference object allows the referent to be
368 obtained, if it still exists, by calling it:
369
370 >>> import weakref
371 >>> class Object:
372 ... pass
373 ...
374 >>> o = Object()
375 >>> r = weakref.ref(o)
376 >>> o2 = r()
377 >>> o is o2
378 True
379
380 If the referent no longer exists, calling the reference object returns
381 :const:`None`:
382
383 >>> del o, o2
384 >>> print(r())
385 None
386
387 Testing that a weak reference object is still live should be done using the
388 expression ``ref() is not None``. Normally, application code that needs to use
389 a reference object should follow this pattern::
390
391 # r is a weak reference object
392 o = r()
393 if o is None:
394 # referent has been garbage collected
395 print("Object has been deallocated; can't frobnicate.")
396 else:
397 print("Object is still live!")
398 o.do_something_useful()
399
400 Using a separate test for "liveness" creates race conditions in threaded
401 applications; another thread can cause a weak reference to become invalidated
402 before the weak reference is called; the idiom shown above is safe in threaded
403 applications as well as single-threaded applications.
404
405 Specialized versions of :class:`ref` objects can be created through subclassing.
406 This is used in the implementation of the :class:`WeakValueDictionary` to reduce
407 the memory overhead for each entry in the mapping. This may be most useful to
408 associate additional information with a reference, but could also be used to
409 insert additional processing on calls to retrieve the referent.
410
411 This example shows how a subclass of :class:`ref` can be used to store
412 additional information about an object and affect the value that's returned when
413 the referent is accessed::
414
415 import weakref
416
417 class ExtendedRef(weakref.ref):
418 def __init__(self, ob, callback=None, /, **annotations):
419 super().__init__(ob, callback)
420 self.__counter = 0
421 for k, v in annotations.items():
422 setattr(self, k, v)
423
424 def __call__(self):
425 """Return a pair containing the referent and the number of
426 times the reference has been called.
427 """
428 ob = super().__call__()
429 if ob is not None:
430 self.__counter += 1
431 ob = (ob, self.__counter)
432 return ob
433
434
435 .. _weakref-example:
436
437 Example
438 -------
439
440 This simple example shows how an application can use object IDs to retrieve
441 objects that it has seen before. The IDs of the objects can then be used in
442 other data structures without forcing the objects to remain alive, but the
443 objects can still be retrieved by ID if they do.
444
445 .. Example contributed by Tim Peters.
446
447 ::
448
449 import weakref
450
451 _id2obj_dict = weakref.WeakValueDictionary()
452
453 def remember(obj):
454 oid = id(obj)
455 _id2obj_dict[oid] = obj
456 return oid
457
458 def id2obj(oid):
459 return _id2obj_dict[oid]
460
461
462 .. _finalize-examples:
463
464 Finalizer Objects
465 -----------------
466
467 The main benefit of using :class:`finalize` is that it makes it simple
468 to register a callback without needing to preserve the returned finalizer
469 object. For instance
470
471 >>> import weakref
472 >>> class Object:
473 ... pass
474 ...
475 >>> kenny = Object()
476 >>> weakref.finalize(kenny, print, "You killed Kenny!") #doctest:+ELLIPSIS
477 <finalize object at ...; for 'Object' at ...>
478 >>> del kenny
479 You killed Kenny!
480
481 The finalizer can be called directly as well. However the finalizer
482 will invoke the callback at most once.
483
484 >>> def callback(x, y, z):
485 ... print("CALLBACK")
486 ... return x + y + z
487 ...
488 >>> obj = Object()
489 >>> f = weakref.finalize(obj, callback, 1, 2, z=3)
490 >>> assert f.alive
491 >>> assert f() == 6
492 CALLBACK
493 >>> assert not f.alive
494 >>> f() # callback not called because finalizer dead
495 >>> del obj # callback not called because finalizer dead
496
497 You can unregister a finalizer using its :meth:`~finalize.detach`
498 method. This kills the finalizer and returns the arguments passed to
499 the constructor when it was created.
500
501 >>> obj = Object()
502 >>> f = weakref.finalize(obj, callback, 1, 2, z=3)
503 >>> f.detach() #doctest:+ELLIPSIS
504 (<...Object object ...>, <function callback ...>, (1, 2), {'z': 3})
505 >>> newobj, func, args, kwargs = _
506 >>> assert not f.alive
507 >>> assert newobj is obj
508 >>> assert func(*args, **kwargs) == 6
509 CALLBACK
510
511 Unless you set the :attr:`~finalize.atexit` attribute to
512 :const:`False`, a finalizer will be called when the program exits if it
513 is still alive. For instance
514
515 .. doctest::
516 :options: +SKIP
517
518 >>> obj = Object()
519 >>> weakref.finalize(obj, print, "obj dead or exiting")
520 <finalize object at ...; for 'Object' at ...>
521 >>> exit()
522 obj dead or exiting
523
524
525 Comparing finalizers with :meth:`~object.__del__` methods
526 ---------------------------------------------------------
527
528 Suppose we want to create a class whose instances represent temporary
529 directories. The directories should be deleted with their contents
530 when the first of the following events occurs:
531
532 * the object is garbage collected,
533 * the object's :meth:`!remove` method is called, or
534 * the program exits.
535
536 We might try to implement the class using a :meth:`~object.__del__` method as
537 follows::
538
539 class TempDir:
540 def __init__(self):
541 self.name = tempfile.mkdtemp()
542
543 def remove(self):
544 if self.name is not None:
545 shutil.rmtree(self.name)
546 self.name = None
547
548 @property
549 def removed(self):
550 return self.name is None
551
552 def __del__(self):
553 self.remove()
554
555 Starting with Python 3.4, :meth:`~object.__del__` methods no longer prevent
556 reference cycles from being garbage collected, and module globals are
557 no longer forced to :const:`None` during :term:`interpreter shutdown`.
558 So this code should work without any issues on CPython.
559
560 However, handling of :meth:`~object.__del__` methods is notoriously implementation
561 specific, since it depends on internal details of the interpreter's garbage
562 collector implementation.
563
564 A more robust alternative can be to define a finalizer which only references
565 the specific functions and objects that it needs, rather than having access
566 to the full state of the object::
567
568 class TempDir:
569 def __init__(self):
570 self.name = tempfile.mkdtemp()
571 self._finalizer = weakref.finalize(self, shutil.rmtree, self.name)
572
573 def remove(self):
574 self._finalizer()
575
576 @property
577 def removed(self):
578 return not self._finalizer.alive
579
580 Defined like this, our finalizer only receives a reference to the details
581 it needs to clean up the directory appropriately. If the object never gets
582 garbage collected the finalizer will still be called at exit.
583
584 The other advantage of weakref based finalizers is that they can be used to
585 register finalizers for classes where the definition is controlled by a
586 third party, such as running code when a module is unloaded::
587
588 import weakref, sys
589 def unloading_module():
590 # implicit reference to the module globals from the function body
591 weakref.finalize(sys.modules[__name__], unloading_module)
592
593
594 .. note::
595
596 If you create a finalizer object in a daemonic thread just as the program
597 exits then there is the possibility that the finalizer
598 does not get called at exit. However, in a daemonic thread
599 :func:`atexit.register`, ``try: ... finally: ...`` and ``with: ...``
600 do not guarantee that cleanup occurs either.