Debian Codegraph @

Doc/library/weakref.rst

raw
1 .. _mod-weakref:
2
3 :mod:`!weakref` --- Weak references
4 ===================================
5
6 .. module:: weakref
7 :synopsis: Support for weak references and weak dictionaries.
8
9 .. moduleauthor:: Fred L. Drake, Jr. <[email protected]>
10 .. moduleauthor:: Neil Schemenauer <[email protected]>
11 .. moduleauthor:: Martin von Löwis <[email protected]>
12 .. sectionauthor:: Fred L. Drake, Jr. <[email protected]>
13
14 **Source code:** :source:`Lib/weakref.py`
15
16 --------------
17
18 The :mod:`!weakref` module allows the Python programmer to create :dfn:`weak
19 references` to objects.
20
21 .. When making changes to the examples in this file, be sure to update
22 Lib/test/test_weakref.py::libreftest too!
23
24 In the following, the term :dfn:`referent` means the object which is referred to
25 by a weak reference.
26
27 A weak reference to an object is not enough to keep the object alive: when the
28 only remaining references to a referent are weak references,
29 :term:`garbage collection` is free to destroy the referent and reuse its memory
30 for something else. However, until the object is actually destroyed the weak
31 reference may return the object even if there are no strong references to it.
32
33 A primary use for weak references is to implement caches or
34 mappings holding large objects, where it's desired that a large object not be
35 kept alive solely because it appears in a cache or mapping.
36
37 For example, if you have a number of large binary image objects, you may wish to
38 associate a name with each. If you used a Python dictionary to map names to
39 images, or images to names, the image objects would remain alive just because
40 they appeared as values or keys in the dictionaries. The
41 :class:`WeakKeyDictionary` and :class:`WeakValueDictionary` classes supplied by
42 the :mod:`!weakref` module are an alternative, using weak references to construct
43 mappings that don't keep objects alive solely because they appear in the mapping
44 objects. If, for example, an image object is a value in a
45 :class:`WeakValueDictionary`, then when the last remaining references to that
46 image object are the weak references held by weak mappings, garbage collection
47 can reclaim the object, and its corresponding entries in weak mappings are
48 simply deleted.
49
50 :class:`WeakKeyDictionary` and :class:`WeakValueDictionary` use weak references
51 in their implementation, setting up callback functions on the weak references
52 that notify the weak dictionaries when a key or value has been reclaimed by
53 garbage collection. :class:`WeakSet` implements the :class:`set` interface,
54 but keeps weak references to its elements, just like a
55 :class:`WeakKeyDictionary` does.
56
57 :class:`finalize` provides a straight forward way to register a
58 cleanup function to be called when an object is garbage collected.
59 This is simpler to use than setting up a callback function on a raw
60 weak reference, since the module automatically ensures that the finalizer
61 remains alive until the object is collected.
62
63 Most programs should find that using one of these weak container types
64 or :class:`finalize` is all they need -- it's not usually necessary to
65 create your own weak references directly. The low-level machinery is
66 exposed by the :mod:`!weakref` module for the benefit of advanced uses.
67
68 Not all objects can be weakly referenced. Objects which support weak references
69 include class instances, functions written in Python (but not in C), instance methods,
70 sets, frozensets, some :term:`file objects <file object>`, :term:`generators <generator>`,
71 type objects, sockets, arrays, deques, regular expression pattern objects, and code
72 objects.
73
74 .. versionchanged:: 3.2
75 Added support for thread.lock, threading.Lock, and code objects.
76
77 Several built-in types such as :class:`list` and :class:`dict` do not directly
78 support weak references but can add support through subclassing::
79
80 class Dict(dict):
81 pass
82
83 obj = Dict(red=1, green=2, blue=3) # this object is weak referenceable
84
85 .. impl-detail::
86
87 Other built-in types such as :class:`tuple` and :class:`int` do not support weak
88 references even when subclassed.
89
90 Extension types can easily be made to support weak references; see
91 :ref:`weakref-support`.
92
93 When ``__slots__`` are defined for a given type, weak reference support is
94 disabled unless a ``'__weakref__'`` string is also present in the sequence of
95 strings in the ``__slots__`` declaration.
96 See :ref:`__slots__ documentation <slots>` for details.
97
98 .. class:: ref(object[, callback])
99
100 Return a weak reference to *object*. The original object can be retrieved by
101 calling the reference object if the referent is still alive; if the referent is
102 no longer alive, calling the reference object will cause :const:`None` to be
103 returned. If *callback* is provided and not :const:`None`, and the returned
104 weakref object is still alive, the callback will be called when the object is
105 about to be finalized; the weak reference object will be passed as the only
106 parameter to the callback; the referent will no longer be available.
107
108 It is allowable for many weak references to be constructed for the same object.
109 Callbacks registered for each weak reference will be called from the most
110 recently registered callback to the oldest registered callback.
111
112 Exceptions raised by the callback will be noted on the standard error output,
113 but cannot be propagated; they are handled in exactly the same way as exceptions
114 raised from an object's :meth:`~object.__del__` method.
115
116 Weak references are :term:`hashable` if the *object* is hashable. They will
117 maintain their hash value even after the *object* was deleted. If
118 :func:`hash` is called the first time only after the *object* was deleted,
119 the call will raise :exc:`TypeError`.
120
121 Weak references support tests for equality, but not ordering. If the referents
122 are still alive, two references have the same equality relationship as their
123 referents (regardless of the *callback*). If either referent has been deleted,
124 the references are equal only if the reference objects are the same object.
125
126 This is a subclassable type rather than a factory function.
127
128 Weak references are :ref:`generic <generics>` over the type of the object they
129 reference.
130
131 .. attribute:: __callback__
132
133 This read-only attribute returns the callback currently associated to the
134 weakref. If there is no callback or if the referent of the weakref is
135 no longer alive then this attribute will have value ``None``.
136
137 .. versionchanged:: 3.4
138 Added the :attr:`__callback__` attribute.
139
140
141 .. function:: proxy(object[, callback])
142
143 Return a proxy to *object* which uses a weak reference. This supports use of
144 the proxy in most contexts instead of requiring the explicit dereferencing used
145 with weak reference objects. The returned object will have a type of either
146 ``ProxyType`` or ``CallableProxyType``, depending on whether *object* is
147 callable. Proxy objects are not :term:`hashable` regardless of the referent; this
148 avoids a number of problems related to their fundamentally mutable nature, and
149 prevents their use as dictionary keys. *callback* is the same as the parameter
150 of the same name to the :func:`ref` function.
151
152 Accessing an attribute of the proxy object after the referent is
153 garbage collected raises :exc:`ReferenceError`.
154
155 .. versionchanged:: 3.8
156 Extended the operator support on proxy objects to include the matrix
157 multiplication operators ``@`` and ``@=``.
158
159
160 .. function:: getweakrefcount(object)
161
162 Return the number of weak references and proxies which refer to *object*.
163
164
165 .. function:: getweakrefs(object)
166
167 Return a list of all weak reference and proxy objects which refer to *object*.
168
169
170 .. class:: WeakKeyDictionary([dict])
171
172 Mapping class that references keys weakly. Entries in the dictionary will be
173 discarded when there is no longer a strong reference to the key. This can be
174 used to associate additional data with an object owned by other parts of an
175 application without adding attributes to those objects. This can be especially
176 useful with objects that override attribute accesses.
177
178 Note that when a key with equal value to an existing key (but not equal identity)
179 is inserted into the dictionary, it replaces the value but does not replace the
180 existing key. Due to this, when the reference to the original key is deleted, it
181 also deletes the entry in the dictionary::
182
183 >>> class T(str): pass
184 ...
185 >>> k1, k2 = T(), T()
186 >>> d = weakref.WeakKeyDictionary()
187 >>> d[k1] = 1 # d = {k1: 1}
188 >>> d[k2] = 2 # d = {k1: 2}
189 >>> del k1 # d = {}
190
191 A workaround would be to remove the key prior to reassignment::
192
193 >>> class T(str): pass
194 ...
195 >>> k1, k2 = T(), T()
196 >>> d = weakref.WeakKeyDictionary()
197 >>> d[k1] = 1 # d = {k1: 1}
198 >>> del d[k1]
199 >>> d[k2] = 2 # d = {k2: 2}
200 >>> del k1 # d = {k2: 2}
201
202 .. versionchanged:: 3.9
203 Added support for ``|`` and ``|=`` operators, as specified in :pep:`584`.
204
205 :class:`WeakKeyDictionary` objects have an additional method that
206 exposes the internal references directly. The references are not guaranteed to
207 be "live" at the time they are used, so the result of calling the references
208 needs to be checked before being used. This can be used to avoid creating
209 references that will cause the garbage collector to keep the keys around longer
210 than needed.
211
212
213 .. method:: WeakKeyDictionary.keyrefs()
214
215 Return an iterable of the weak references to the keys.
216
217
218 .. class:: WeakValueDictionary([dict])
219
220 Mapping class that references values weakly. Entries in the dictionary will be
221 discarded when no strong reference to the value exists any more.
222
223 .. versionchanged:: 3.9
224 Added support for ``|`` and ``|=`` operators, as specified in :pep:`584`.
225
226 :class:`WeakValueDictionary` objects have an additional method that has the
227 same issues as the :meth:`WeakKeyDictionary.keyrefs` method.
228
229
230 .. method:: WeakValueDictionary.valuerefs()
231
232 Return an iterable of the weak references to the values.
233
234
235 .. class:: WeakSet([elements])
236
237 Set class that keeps weak references to its elements. An element will be
238 discarded when no strong reference to it exists any more.
239
240
241 .. class:: WeakMethod(method[, callback])
242
243 A custom :class:`ref` subclass which simulates a weak reference to a bound
244 method (i.e., a method defined on a class and looked up on an instance).
245 Since a bound method is ephemeral, a standard weak reference cannot keep
246 hold of it. :class:`WeakMethod` has special code to recreate the bound
247 method until either the object or the original function dies::
248
249 >>> class C:
250 ... def method(self):
251 ... print("method called!")
252 ...
253 >>> c = C()
254 >>> r = weakref.ref(c.method)
255 >>> r()
256 >>> r = weakref.WeakMethod(c.method)
257 >>> r()
258 <bound method C.method of <__main__.C object at 0x7fc859830220>>
259 >>> r()()
260 method called!
261 >>> del c
262 >>> gc.collect()
263 0
264 >>> r()
265 >>>
266
267 *callback* is the same as the parameter of the same name to the :func:`ref` function.
268
269 .. versionadded:: 3.4
270
271 .. class:: finalize(obj, func, /, *args, **kwargs)
272
273 Return a callable finalizer object which will be called when *obj*
274 is garbage collected. Unlike an ordinary weak reference, a finalizer
275 will always survive until the reference object is collected, greatly
276 simplifying lifecycle management.
277
278 A finalizer is considered *alive* until it is called (either explicitly
279 or at garbage collection), and after that it is *dead*. Calling a live
280 finalizer returns the result of evaluating ``func(*arg, **kwargs)``,
281 whereas calling a dead finalizer returns :const:`None`.
282
283 Exceptions raised by finalizer callbacks during garbage collection
284 will be shown on the standard error output, but cannot be
285 propagated. They are handled in the same way as exceptions raised
286 from an object's :meth:`~object.__del__` method or a weak reference's
287 callback.
288
289 When the program exits, each remaining live finalizer is called
290 unless its :attr:`atexit` attribute has been set to false. They
291 are called in reverse order of creation.
292
293 A finalizer will never invoke its callback during the later part of
294 the :term:`interpreter shutdown` when module globals are liable to have
295 been replaced by :const:`None`.
296
297 .. method:: __call__()
298
299 If *self* is alive then mark it as dead and return the result of
300 calling ``func(*args, **kwargs)``. If *self* is dead then return
301 :const:`None`.
302
303 .. method:: detach()
304
305 If *self* is alive then mark it as dead and return the tuple
306 ``(obj, func, args, kwargs)``. If *self* is dead then return
307 :const:`None`.
308
309 .. method:: peek()
310
311 If *self* is alive then return the tuple ``(obj, func, args,
312 kwargs)``. If *self* is dead then return :const:`None`.
313
314 .. attribute:: alive
315
316 Property which is true if the finalizer is alive, false otherwise.
317
318 .. attribute:: atexit
319
320 A writable boolean property which by default is true. When the
321 program exits, it calls all remaining live finalizers for which
322 :attr:`.atexit` is true. They are called in reverse order of
323 creation.
324
325 .. note::
326
327 It is important to ensure that *func*, *args* and *kwargs* do
328 not own any references to *obj*, either directly or indirectly,
329 since otherwise *obj* will never be garbage collected. In
330 particular, *func* should not be a bound method of *obj*.
331
332 .. versionadded:: 3.4
333
334
335 .. class:: ReferenceType
336
337 The type object for weak references objects.
338
339
340 .. class:: ProxyType
341
342 The type object for proxies of objects which are not callable.
343
344
345 .. class:: CallableProxyType
346
347 The type object for proxies of callable objects.
348
349
350 .. data:: ProxyTypes
351
352 Sequence containing all the type objects for proxies. This can make it simpler
353 to test if an object is a proxy without being dependent on naming both proxy
354 types.
355
356
357 .. seealso::
358
359 :pep:`205` - Weak References
360 The proposal and rationale for this feature, including links to earlier
361 implementations and information about similar features in other languages.
362
363
364 .. _weakref-objects:
365
366 Weak Reference Objects
367 ----------------------
368
369 Weak reference objects have no methods and no attributes besides
370 :attr:`ref.__callback__`. A weak reference object allows the referent to be
371 obtained, if it still exists, by calling it:
372
373 >>> import weakref
374 >>> class Object:
375 ... pass
376 ...
377 >>> o = Object()
378 >>> r = weakref.ref(o)
379 >>> o2 = r()
380 >>> o is o2
381 True
382
383 If the referent no longer exists, calling the reference object returns
384 :const:`None`:
385
386 >>> del o, o2
387 >>> print(r())
388 None
389
390 Testing that a weak reference object is still live should be done using the
391 expression ``ref() is not None``. Normally, application code that needs to use
392 a reference object should follow this pattern::
393
394 # r is a weak reference object
395 o = r()
396 if o is None:
397 # referent has been garbage collected
398 print("Object has been deallocated; can't frobnicate.")
399 else:
400 print("Object is still live!")
401 o.do_something_useful()
402
403 Using a separate test for "liveness" creates race conditions in threaded
404 applications; another thread can cause a weak reference to become invalidated
405 before the weak reference is called; the idiom shown above is safe in threaded
406 applications as well as single-threaded applications.
407
408 Specialized versions of :class:`ref` objects can be created through subclassing.
409 This is used in the implementation of the :class:`WeakValueDictionary` to reduce
410 the memory overhead for each entry in the mapping. This may be most useful to
411 associate additional information with a reference, but could also be used to
412 insert additional processing on calls to retrieve the referent.
413
414 This example shows how a subclass of :class:`ref` can be used to store
415 additional information about an object and affect the value that's returned when
416 the referent is accessed::
417
418 import weakref
419
420 class ExtendedRef(weakref.ref):
421 def __init__(self, ob, callback=None, /, **annotations):
422 super().__init__(ob, callback)
423 self.__counter = 0
424 for k, v in annotations.items():
425 setattr(self, k, v)
426
427 def __call__(self):
428 """Return a pair containing the referent and the number of
429 times the reference has been called.
430 """
431 ob = super().__call__()
432 if ob is not None:
433 self.__counter += 1
434 ob = (ob, self.__counter)
435 return ob
436
437
438 .. _weakref-example:
439
440 Example
441 -------
442
443 This simple example shows how an application can use object IDs to retrieve
444 objects that it has seen before. The IDs of the objects can then be used in
445 other data structures without forcing the objects to remain alive, but the
446 objects can still be retrieved by ID if they do.
447
448 .. Example contributed by Tim Peters.
449
450 ::
451
452 import weakref
453
454 _id2obj_dict = weakref.WeakValueDictionary()
455
456 def remember(obj):
457 oid = id(obj)
458 _id2obj_dict[oid] = obj
459 return oid
460
461 def id2obj(oid):
462 return _id2obj_dict[oid]
463
464
465 .. _finalize-examples:
466
467 Finalizer Objects
468 -----------------
469
470 The main benefit of using :class:`finalize` is that it makes it simple
471 to register a callback without needing to preserve the returned finalizer
472 object. For instance
473
474 >>> import weakref
475 >>> class Object:
476 ... pass
477 ...
478 >>> kenny = Object()
479 >>> weakref.finalize(kenny, print, "You killed Kenny!") #doctest:+ELLIPSIS
480 <finalize object at ...; for 'Object' at ...>
481 >>> del kenny
482 You killed Kenny!
483
484 The finalizer can be called directly as well. However the finalizer
485 will invoke the callback at most once.
486
487 >>> def callback(x, y, z):
488 ... print("CALLBACK")
489 ... return x + y + z
490 ...
491 >>> obj = Object()
492 >>> f = weakref.finalize(obj, callback, 1, 2, z=3)
493 >>> assert f.alive
494 >>> assert f() == 6
495 CALLBACK
496 >>> assert not f.alive
497 >>> f() # callback not called because finalizer dead
498 >>> del obj # callback not called because finalizer dead
499
500 You can unregister a finalizer using its :meth:`~finalize.detach`
501 method. This kills the finalizer and returns the arguments passed to
502 the constructor when it was created.
503
504 >>> obj = Object()
505 >>> f = weakref.finalize(obj, callback, 1, 2, z=3)
506 >>> f.detach() #doctest:+ELLIPSIS
507 (<...Object object ...>, <function callback ...>, (1, 2), {'z': 3})
508 >>> newobj, func, args, kwargs = _
509 >>> assert not f.alive
510 >>> assert newobj is obj
511 >>> assert func(*args, **kwargs) == 6
512 CALLBACK
513
514 Unless you set the :attr:`~finalize.atexit` attribute to
515 :const:`False`, a finalizer will be called when the program exits if it
516 is still alive. For instance
517
518 .. doctest::
519 :options: +SKIP
520
521 >>> obj = Object()
522 >>> weakref.finalize(obj, print, "obj dead or exiting")
523 <finalize object at ...; for 'Object' at ...>
524 >>> exit()
525 obj dead or exiting
526
527
528 Comparing finalizers with :meth:`~object.__del__` methods
529 ---------------------------------------------------------
530
531 Suppose we want to create a class whose instances represent temporary
532 directories. The directories should be deleted with their contents
533 when the first of the following events occurs:
534
535 * the object is garbage collected,
536 * the object's :meth:`!remove` method is called, or
537 * the program exits.
538
539 We might try to implement the class using a :meth:`~object.__del__` method as
540 follows::
541
542 class TempDir:
543 def __init__(self):
544 self.name = tempfile.mkdtemp()
545
546 def remove(self):
547 if self.name is not None:
548 shutil.rmtree(self.name)
549 self.name = None
550
551 @property
552 def removed(self):
553 return self.name is None
554
555 def __del__(self):
556 self.remove()
557
558 Starting with Python 3.4, :meth:`~object.__del__` methods no longer prevent
559 reference cycles from being garbage collected, and module globals are
560 no longer forced to :const:`None` during :term:`interpreter shutdown`.
561 So this code should work without any issues on CPython.
562
563 However, handling of :meth:`~object.__del__` methods is notoriously implementation
564 specific, since it depends on internal details of the interpreter's garbage
565 collector implementation.
566
567 A more robust alternative can be to define a finalizer which only references
568 the specific functions and objects that it needs, rather than having access
569 to the full state of the object::
570
571 class TempDir:
572 def __init__(self):
573 self.name = tempfile.mkdtemp()
574 self._finalizer = weakref.finalize(self, shutil.rmtree, self.name)
575
576 def remove(self):
577 self._finalizer()
578
579 @property
580 def removed(self):
581 return not self._finalizer.alive
582
583 Defined like this, our finalizer only receives a reference to the details
584 it needs to clean up the directory appropriately. If the object never gets
585 garbage collected the finalizer will still be called at exit.
586
587 The other advantage of weakref based finalizers is that they can be used to
588 register finalizers for classes where the definition is controlled by a
589 third party, such as running code when a module is unloaded::
590
591 import weakref, sys
592 def unloading_module():
593 # implicit reference to the module globals from the function body
594 weakref.finalize(sys.modules[__name__], unloading_module)
595
596
597 .. note::
598
599 If you create a finalizer object in a daemonic thread just as the program
600 exits then there is the possibility that the finalizer
601 does not get called at exit. However, in a daemonic thread
602 :func:`atexit.register`, ``try: ... finally: ...`` and ``with: ...``
603 do not guarantee that cleanup occurs either.