Debian Codegraph @

Doc/howto/descriptor.rst

raw
1 .. _descriptorhowto:
2
3 ================
4 Descriptor Guide
5 ================
6
7 :Author: Raymond Hettinger
8 :Contact: <python at rcn dot com>
9
10 .. Contents::
11
12
13 :term:`Descriptors <descriptor>` let objects customize attribute lookup,
14 storage, and deletion.
15
16 This guide has four major sections:
17
18 1) The "primer" gives a basic overview, moving gently from simple examples,
19 adding one feature at a time. Start here if you're new to descriptors.
20
21 2) The second section shows a complete, practical descriptor example. If you
22 already know the basics, start there.
23
24 3) The third section provides a more technical tutorial that goes into the
25 detailed mechanics of how descriptors work. Most people don't need this
26 level of detail.
27
28 4) The last section has pure Python equivalents for built-in descriptors that
29 are written in C. Read this if you're curious about how functions turn
30 into bound methods or about the implementation of common tools like
31 :func:`classmethod`, :func:`staticmethod`, :func:`property`, and
32 :term:`__slots__`.
33
34
35 Primer
36 ^^^^^^
37
38 In this primer, we start with the most basic possible example and then we'll
39 add new capabilities one by one.
40
41
42 Simple example: A descriptor that returns a constant
43 ----------------------------------------------------
44
45 The :class:`!Ten` class is a descriptor whose :meth:`~object.__get__` method always
46 returns the constant ``10``:
47
48 .. testcode::
49
50 class Ten:
51 def __get__(self, obj, objtype=None):
52 return 10
53
54 To use the descriptor, it must be stored as a class variable in another class:
55
56 .. testcode::
57
58 class A:
59 x = 5 # Regular class attribute
60 y = Ten() # Descriptor instance
61
62 An interactive session shows the difference between normal attribute lookup
63 and descriptor lookup:
64
65 .. doctest::
66
67 >>> a = A() # Make an instance of class A
68 >>> a.x # Normal attribute lookup
69 5
70 >>> a.y # Descriptor lookup
71 10
72
73 In the ``a.x`` attribute lookup, the dot operator finds ``'x': 5``
74 in the class dictionary. In the ``a.y`` lookup, the dot operator
75 finds a descriptor instance, recognized by its ``__get__`` method.
76 Calling that method returns ``10``.
77
78 Note that the value ``10`` is not stored in either the class dictionary or the
79 instance dictionary. Instead, the value ``10`` is computed on demand.
80
81 This example shows how a simple descriptor works, but it isn't very useful.
82 For retrieving constants, normal attribute lookup would be better.
83
84 In the next section, we'll create something more useful, a dynamic lookup.
85
86
87 Dynamic lookups
88 ---------------
89
90 Interesting descriptors typically run computations instead of returning
91 constants:
92
93 .. testcode::
94
95 import os
96
97 class DirectorySize:
98
99 def __get__(self, obj, objtype=None):
100 return len(os.listdir(obj.dirname))
101
102 class Directory:
103
104 size = DirectorySize() # Descriptor instance
105
106 def __init__(self, dirname):
107 self.dirname = dirname # Regular instance attribute
108
109 An interactive session shows that the lookup is dynamic — it computes
110 different, updated answers each time::
111
112 >>> s = Directory('songs')
113 >>> g = Directory('games')
114 >>> s.size # The songs directory has twenty files
115 20
116 >>> g.size # The games directory has three files
117 3
118 >>> os.remove('games/chess') # Delete a game
119 >>> g.size # File count is automatically updated
120 2
121
122 Besides showing how descriptors can run computations, this example also
123 reveals the purpose of the parameters to :meth:`~object.__get__`. The *self*
124 parameter is *size*, an instance of *DirectorySize*. The *obj* parameter is
125 either *g* or *s*, an instance of *Directory*. It is the *obj* parameter that
126 lets the :meth:`~object.__get__` method learn the target directory. The *objtype*
127 parameter is the class *Directory*.
128
129
130 Managed attributes
131 ------------------
132
133 A popular use for descriptors is managing access to instance data. The
134 descriptor is assigned to a public attribute in the class dictionary while the
135 actual data is stored as a private attribute in the instance dictionary. The
136 descriptor's :meth:`~object.__get__` and :meth:`~object.__set__` methods are triggered when
137 the public attribute is accessed.
138
139 In the following example, *age* is the public attribute and *_age* is the
140 private attribute. When the public attribute is accessed, the descriptor logs
141 the lookup or update:
142
143 .. testcode::
144
145 import logging
146
147 logging.basicConfig(level=logging.INFO)
148
149 class LoggedAgeAccess:
150
151 def __get__(self, obj, objtype=None):
152 value = obj._age
153 logging.info('Accessing %r giving %r', 'age', value)
154 return value
155
156 def __set__(self, obj, value):
157 logging.info('Updating %r to %r', 'age', value)
158 obj._age = value
159
160 class Person:
161
162 age = LoggedAgeAccess() # Descriptor instance
163
164 def __init__(self, name, age):
165 self.name = name # Regular instance attribute
166 self.age = age # Calls __set__()
167
168 def birthday(self):
169 self.age += 1 # Calls both __get__() and __set__()
170
171
172 An interactive session shows that all access to the managed attribute *age* is
173 logged, but that the regular attribute *name* is not logged:
174
175 .. testcode::
176 :hide:
177
178 import logging, sys
179 logging.basicConfig(level=logging.INFO, stream=sys.stdout, force=True)
180
181 .. doctest::
182
183 >>> mary = Person('Mary M', 30) # The initial age update is logged
184 INFO:root:Updating 'age' to 30
185 >>> dave = Person('David D', 40)
186 INFO:root:Updating 'age' to 40
187
188 >>> vars(mary) # The actual data is in a private attribute
189 {'name': 'Mary M', '_age': 30}
190 >>> vars(dave)
191 {'name': 'David D', '_age': 40}
192
193 >>> mary.age # Access the data and log the lookup
194 INFO:root:Accessing 'age' giving 30
195 30
196 >>> mary.birthday() # Updates are logged as well
197 INFO:root:Accessing 'age' giving 30
198 INFO:root:Updating 'age' to 31
199
200 >>> dave.name # Regular attribute lookup isn't logged
201 'David D'
202 >>> dave.age # Only the managed attribute is logged
203 INFO:root:Accessing 'age' giving 40
204 40
205
206 One major issue with this example is that the private name *_age* is hardwired in
207 the *LoggedAgeAccess* class. That means that each instance can only have one
208 logged attribute and that its name is unchangeable. In the next example,
209 we'll fix that problem.
210
211
212 Customized names
213 ----------------
214
215 When a class uses descriptors, it can inform each descriptor about which
216 variable name was used.
217
218 In this example, the :class:`!Person` class has two descriptor instances,
219 *name* and *age*. When the :class:`!Person` class is defined, it makes a
220 callback to :meth:`~object.__set_name__` in *LoggedAccess* so that the field names can
221 be recorded, giving each descriptor its own *public_name* and *private_name*:
222
223 .. testcode::
224
225 import logging
226
227 logging.basicConfig(level=logging.INFO)
228
229 class LoggedAccess:
230
231 def __set_name__(self, owner, name):
232 self.public_name = name
233 self.private_name = '_' + name
234
235 def __get__(self, obj, objtype=None):
236 value = getattr(obj, self.private_name)
237 logging.info('Accessing %r giving %r', self.public_name, value)
238 return value
239
240 def __set__(self, obj, value):
241 logging.info('Updating %r to %r', self.public_name, value)
242 setattr(obj, self.private_name, value)
243
244 class Person:
245
246 name = LoggedAccess() # First descriptor instance
247 age = LoggedAccess() # Second descriptor instance
248
249 def __init__(self, name, age):
250 self.name = name # Calls the first descriptor
251 self.age = age # Calls the second descriptor
252
253 def birthday(self):
254 self.age += 1
255
256 An interactive session shows that the :class:`!Person` class has called
257 :meth:`~object.__set_name__` so that the field names would be recorded. Here
258 we call :func:`vars` to look up the descriptor without triggering it:
259
260 .. doctest::
261
262 >>> vars(vars(Person)['name'])
263 {'public_name': 'name', 'private_name': '_name'}
264 >>> vars(vars(Person)['age'])
265 {'public_name': 'age', 'private_name': '_age'}
266
267 The new class now logs access to both *name* and *age*:
268
269 .. testcode::
270 :hide:
271
272 import logging, sys
273 logging.basicConfig(level=logging.INFO, stream=sys.stdout, force=True)
274
275 .. doctest::
276
277 >>> pete = Person('Peter P', 10)
278 INFO:root:Updating 'name' to 'Peter P'
279 INFO:root:Updating 'age' to 10
280 >>> kate = Person('Catherine C', 20)
281 INFO:root:Updating 'name' to 'Catherine C'
282 INFO:root:Updating 'age' to 20
283
284 The two *Person* instances contain only the private names:
285
286 .. doctest::
287
288 >>> vars(pete)
289 {'_name': 'Peter P', '_age': 10}
290 >>> vars(kate)
291 {'_name': 'Catherine C', '_age': 20}
292
293
294 Closing thoughts
295 ----------------
296
297 A :term:`descriptor` is what we call any object that defines :meth:`~object.__get__`,
298 :meth:`~object.__set__`, or :meth:`~object.__delete__`.
299
300 Optionally, descriptors can have a :meth:`~object.__set_name__` method. This is only
301 used in cases where a descriptor needs to know either the class where it was
302 created or the name of class variable it was assigned to. (This method, if
303 present, is called even if the class is not a descriptor.)
304
305 Descriptors get invoked by the dot operator during attribute lookup. If a
306 descriptor is accessed indirectly with ``vars(some_class)[descriptor_name]``,
307 the descriptor instance is returned without invoking it.
308
309 Descriptors only work when used as class variables. When put in instances,
310 they have no effect.
311
312 The main motivation for descriptors is to provide a hook allowing objects
313 stored in class variables to control what happens during attribute lookup.
314
315 Traditionally, the calling class controls what happens during lookup.
316 Descriptors invert that relationship and allow the data being looked-up to
317 have a say in the matter.
318
319 Descriptors are used throughout the language. It is how functions turn into
320 bound methods. Common tools like :func:`classmethod`, :func:`staticmethod`,
321 :func:`property`, and :func:`functools.cached_property` are all implemented as
322 descriptors.
323
324
325 Complete Practical Example
326 ^^^^^^^^^^^^^^^^^^^^^^^^^^
327
328 In this example, we create a practical and powerful tool for locating
329 notoriously hard to find data corruption bugs.
330
331
332 Validator class
333 ---------------
334
335 A validator is a descriptor for managed attribute access. Prior to storing
336 any data, it verifies that the new value meets various type and range
337 restrictions. If those restrictions aren't met, it raises an exception to
338 prevent data corruption at its source.
339
340 This :class:`!Validator` class is both an :term:`abstract base class` and a
341 managed attribute descriptor:
342
343 .. testcode::
344
345 from abc import ABC, abstractmethod
346
347 class Validator(ABC):
348
349 def __set_name__(self, owner, name):
350 self.private_name = '_' + name
351
352 def __get__(self, obj, objtype=None):
353 return getattr(obj, self.private_name)
354
355 def __set__(self, obj, value):
356 self.validate(value)
357 setattr(obj, self.private_name, value)
358
359 @abstractmethod
360 def validate(self, value):
361 pass
362
363 Custom validators need to inherit from :class:`!Validator` and must supply a
364 :meth:`!validate` method to test various restrictions as needed.
365
366
367 Custom validators
368 -----------------
369
370 Here are three practical data validation utilities:
371
372 1) :class:`!OneOf` verifies that a value is one of a restricted set of options.
373
374 2) :class:`!Number` verifies that a value is either an :class:`int` or
375 :class:`float`. Optionally, it verifies that a value is between a given
376 minimum or maximum.
377
378 3) :class:`!String` verifies that a value is a :class:`str`. Optionally, it
379 validates a given minimum or maximum length. It can validate a
380 user-defined `predicate
381 <https://en.wikipedia.org/wiki/Predicate_(mathematical_logic)>`_ as well.
382
383 .. testcode::
384
385 class OneOf(Validator):
386
387 def __init__(self, *options):
388 self.options = set(options)
389
390 def validate(self, value):
391 if value not in self.options:
392 raise ValueError(
393 f'Expected {value!r} to be one of {self.options!r}'
394 )
395
396 class Number(Validator):
397
398 def __init__(self, minvalue=None, maxvalue=None):
399 self.minvalue = minvalue
400 self.maxvalue = maxvalue
401
402 def validate(self, value):
403 if not isinstance(value, (int, float)):
404 raise TypeError(f'Expected {value!r} to be an int or float')
405 if self.minvalue is not None and value < self.minvalue:
406 raise ValueError(
407 f'Expected {value!r} to be at least {self.minvalue!r}'
408 )
409 if self.maxvalue is not None and value > self.maxvalue:
410 raise ValueError(
411 f'Expected {value!r} to be no more than {self.maxvalue!r}'
412 )
413
414 class String(Validator):
415
416 def __init__(self, minsize=None, maxsize=None, predicate=None):
417 self.minsize = minsize
418 self.maxsize = maxsize
419 self.predicate = predicate
420
421 def validate(self, value):
422 if not isinstance(value, str):
423 raise TypeError(f'Expected {value!r} to be a str')
424 if self.minsize is not None and len(value) < self.minsize:
425 raise ValueError(
426 f'Expected {value!r} to be no smaller than {self.minsize!r}'
427 )
428 if self.maxsize is not None and len(value) > self.maxsize:
429 raise ValueError(
430 f'Expected {value!r} to be no bigger than {self.maxsize!r}'
431 )
432 if self.predicate is not None and not self.predicate(value):
433 raise ValueError(
434 f'Expected {self.predicate} to be true for {value!r}'
435 )
436
437
438 Practical application
439 ---------------------
440
441 Here's how the data validators can be used in a real class:
442
443 .. testcode::
444
445 class Component:
446
447 name = String(minsize=3, maxsize=10, predicate=str.isupper)
448 kind = OneOf('wood', 'metal', 'plastic')
449 quantity = Number(minvalue=0)
450
451 def __init__(self, name, kind, quantity):
452 self.name = name
453 self.kind = kind
454 self.quantity = quantity
455
456 The descriptors prevent invalid instances from being created:
457
458 .. doctest::
459
460 >>> Component('Widget', 'metal', 5) # Blocked: 'Widget' is not all uppercase
461 Traceback (most recent call last):
462 ...
463 ValueError: Expected <method 'isupper' of 'str' objects> to be true for 'Widget'
464
465 >>> Component('WIDGET', 'metle', 5) # Blocked: 'metle' is misspelled
466 Traceback (most recent call last):
467 ...
468 ValueError: Expected 'metle' to be one of {'metal', 'plastic', 'wood'}
469
470 >>> Component('WIDGET', 'metal', -5) # Blocked: -5 is negative
471 Traceback (most recent call last):
472 ...
473 ValueError: Expected -5 to be at least 0
474
475 >>> Component('WIDGET', 'metal', 'V') # Blocked: 'V' isn't a number
476 Traceback (most recent call last):
477 ...
478 TypeError: Expected 'V' to be an int or float
479
480 >>> c = Component('WIDGET', 'metal', 5) # Allowed: The inputs are valid
481
482
483 Technical Tutorial
484 ^^^^^^^^^^^^^^^^^^
485
486 What follows is a more technical tutorial for the mechanics and details of how
487 descriptors work.
488
489
490 Abstract
491 --------
492
493 Defines descriptors, summarizes the protocol, and shows how descriptors are
494 called. Provides an example showing how object relational mappings work.
495
496 Learning about descriptors not only provides access to a larger toolset, it
497 creates a deeper understanding of how Python works.
498
499
500 Definition and introduction
501 ---------------------------
502
503 In general, a descriptor is an attribute value that has one of the methods in
504 the descriptor protocol. Those methods are :meth:`~object.__get__`, :meth:`~object.__set__`,
505 and :meth:`~object.__delete__`. If any of those methods are defined for an
506 attribute, it is said to be a :term:`descriptor`.
507
508 The default behavior for attribute access is to get, set, or delete the
509 attribute from an object's dictionary. For instance, ``a.x`` has a lookup chain
510 starting with ``a.__dict__['x']``, then ``type(a).__dict__['x']``, and
511 continuing through the method resolution order of ``type(a)``. If the
512 looked-up value is an object defining one of the descriptor methods, then Python
513 may override the default behavior and invoke the descriptor method instead.
514 Where this occurs in the precedence chain depends on which descriptor methods
515 were defined.
516
517 Descriptors are a powerful, general purpose protocol. They are the mechanism
518 behind properties, methods, static methods, class methods, and
519 :func:`super`. They are used throughout Python itself. Descriptors
520 simplify the underlying C code and offer a flexible set of new tools for
521 everyday Python programs.
522
523
524 Descriptor protocol
525 -------------------
526
527 ``descr.__get__(self, obj, type=None)``
528
529 ``descr.__set__(self, obj, value)``
530
531 ``descr.__delete__(self, obj)``
532
533 That is all there is to it. Define any of these methods and an object is
534 considered a descriptor and can override default behavior upon being looked up
535 as an attribute.
536
537 If an object defines :meth:`~object.__set__` or :meth:`~object.__delete__`, it is considered
538 a data descriptor. Descriptors that only define :meth:`~object.__get__` are called
539 non-data descriptors (they are often used for methods but other uses are
540 possible).
541
542 Data and non-data descriptors differ in how overrides are calculated with
543 respect to entries in an instance's dictionary. If an instance's dictionary
544 has an entry with the same name as a data descriptor, the data descriptor
545 takes precedence. If an instance's dictionary has an entry with the same
546 name as a non-data descriptor, the dictionary entry takes precedence.
547
548 To make a read-only data descriptor, define both :meth:`~object.__get__` and
549 :meth:`~object.__set__` with the :meth:`~object.__set__` raising an :exc:`AttributeError` when
550 called. Defining the :meth:`~object.__set__` method with an exception raising
551 placeholder is enough to make it a data descriptor.
552
553
554 Overview of descriptor invocation
555 ---------------------------------
556
557 A descriptor can be called directly with ``desc.__get__(obj)`` or
558 ``desc.__get__(None, cls)``.
559
560 But it is more common for a descriptor to be invoked automatically from
561 attribute access.
562
563 The expression ``obj.x`` looks up the attribute ``x`` in the chain of
564 namespaces for ``obj``. If the search finds a descriptor outside of the
565 instance :attr:`~object.__dict__`, its :meth:`~object.__get__` method is
566 invoked according to the precedence rules listed below.
567
568 The details of invocation depend on whether ``obj`` is an object, class, or
569 instance of super.
570
571
572 Invocation from an instance
573 ---------------------------
574
575 Instance lookup scans through a chain of namespaces giving data descriptors
576 the highest priority, followed by instance variables, then non-data
577 descriptors, then class variables, and lastly :meth:`~object.__getattr__` if it is
578 provided.
579
580 If a descriptor is found for ``a.x``, then it is invoked with:
581 ``desc.__get__(a, type(a))``.
582
583 The logic for a dotted lookup is in :meth:`object.__getattribute__`. Here is
584 a pure Python equivalent:
585
586 .. testcode::
587
588 def find_name_in_mro(cls, name, default):
589 "Emulate _PyType_Lookup() in Objects/typeobject.c"
590 for base in cls.__mro__:
591 if name in vars(base):
592 return vars(base)[name]
593 return default
594
595 def object_getattribute(obj, name):
596 "Emulate PyObject_GenericGetAttr() in Objects/object.c"
597 null = object()
598 objtype = type(obj)
599 cls_var = find_name_in_mro(objtype, name, null)
600 descr_get = getattr(type(cls_var), '__get__', null)
601 if descr_get is not null:
602 if (hasattr(type(cls_var), '__set__')
603 or hasattr(type(cls_var), '__delete__')):
604 return descr_get(cls_var, obj, objtype) # data descriptor
605 if hasattr(obj, '__dict__') and name in vars(obj):
606 return vars(obj)[name] # instance variable
607 if descr_get is not null:
608 return descr_get(cls_var, obj, objtype) # non-data descriptor
609 if cls_var is not null:
610 return cls_var # class variable
611 raise AttributeError(name)
612
613
614 .. testcode::
615 :hide:
616
617 # Test the fidelity of object_getattribute() by comparing it with the
618 # normal object.__getattribute__(). The former will be accessed by
619 # square brackets and the latter by the dot operator.
620
621 class Object:
622
623 def __getitem__(obj, name):
624 try:
625 return object_getattribute(obj, name)
626 except AttributeError:
627 if not hasattr(type(obj), '__getattr__'):
628 raise
629 return type(obj).__getattr__(obj, name) # __getattr__
630
631 class DualOperator(Object):
632
633 x = 10
634
635 def __init__(self, z):
636 self.z = z
637
638 @property
639 def p2(self):
640 return 2 * self.x
641
642 @property
643 def p3(self):
644 return 3 * self.x
645
646 def m5(self, y):
647 return 5 * y
648
649 def m7(self, y):
650 return 7 * y
651
652 def __getattr__(self, name):
653 return ('getattr_hook', self, name)
654
655 class DualOperatorWithSlots:
656
657 __getitem__ = Object.__getitem__
658
659 __slots__ = ['z']
660
661 x = 15
662
663 def __init__(self, z):
664 self.z = z
665
666 @property
667 def p2(self):
668 return 2 * self.x
669
670 def m5(self, y):
671 return 5 * y
672
673 def __getattr__(self, name):
674 return ('getattr_hook', self, name)
675
676 class D1:
677 def __get__(self, obj, objtype=None):
678 return type(self), obj, objtype
679
680 class U1:
681 x = D1()
682
683 class U2(U1):
684 pass
685
686 .. doctest::
687 :hide:
688
689 >>> a = DualOperator(11)
690 >>> vars(a).update(p3 = '_p3', m7 = '_m7')
691 >>> a.x == a['x'] == 10
692 True
693 >>> a.z == a['z'] == 11
694 True
695 >>> a.p2 == a['p2'] == 20
696 True
697 >>> a.p3 == a['p3'] == 30
698 True
699 >>> a.m5(100) == a.m5(100) == 500
700 True
701 >>> a.m7 == a['m7'] == '_m7'
702 True
703 >>> a.g == a['g'] == ('getattr_hook', a, 'g')
704 True
705
706 >>> b = DualOperatorWithSlots(22)
707 >>> b.x == b['x'] == 15
708 True
709 >>> b.z == b['z'] == 22
710 True
711 >>> b.p2 == b['p2'] == 30
712 True
713 >>> b.m5(200) == b['m5'](200) == 1000
714 True
715 >>> b.g == b['g'] == ('getattr_hook', b, 'g')
716 True
717
718 >>> u2 = U2()
719 >>> object_getattribute(u2, 'x') == u2.x == (D1, u2, U2)
720 True
721
722 Note, there is no :meth:`~object.__getattr__` hook in the :meth:`~object.__getattribute__`
723 code. That is why calling :meth:`~object.__getattribute__` directly or with
724 ``super().__getattribute__`` will bypass :meth:`~object.__getattr__` entirely.
725
726 Instead, it is the dot operator and the :func:`getattr` function that are
727 responsible for invoking :meth:`~object.__getattr__` whenever :meth:`~object.__getattribute__`
728 raises an :exc:`AttributeError`. Their logic is encapsulated in a helper
729 function:
730
731 .. testcode::
732
733 def getattr_hook(obj, name):
734 "Emulate slot_tp_getattr_hook() in Objects/typeobject.c"
735 try:
736 return obj.__getattribute__(name)
737 except AttributeError:
738 if not hasattr(type(obj), '__getattr__'):
739 raise
740 return type(obj).__getattr__(obj, name) # __getattr__
741
742 .. doctest::
743 :hide:
744
745
746 >>> class ClassWithGetAttr:
747 ... x = 123
748 ... def __getattr__(self, attr):
749 ... return attr.upper()
750 ...
751 >>> cw = ClassWithGetAttr()
752 >>> cw.y = 456
753 >>> getattr_hook(cw, 'x')
754 123
755 >>> getattr_hook(cw, 'y')
756 456
757 >>> getattr_hook(cw, 'z')
758 'Z'
759
760 >>> class ClassWithoutGetAttr:
761 ... x = 123
762 ...
763 >>> cwo = ClassWithoutGetAttr()
764 >>> cwo.y = 456
765 >>> getattr_hook(cwo, 'x')
766 123
767 >>> getattr_hook(cwo, 'y')
768 456
769 >>> getattr_hook(cwo, 'z')
770 Traceback (most recent call last):
771 ...
772 AttributeError: 'ClassWithoutGetAttr' object has no attribute 'z'
773
774
775 Invocation from a class
776 -----------------------
777
778 The logic for a dotted lookup such as ``A.x`` is in
779 :meth:`!type.__getattribute__`. The steps are similar to those for
780 :meth:`!object.__getattribute__` but the instance dictionary lookup is replaced
781 by a search through the class's :term:`method resolution order`.
782
783 If a descriptor is found, it is invoked with ``desc.__get__(None, A)``.
784
785 The full C implementation can be found in :c:func:`!type_getattro` and
786 :c:func:`!_PyType_Lookup` in :source:`Objects/typeobject.c`.
787
788
789 Invocation from super
790 ---------------------
791
792 The logic for super's dotted lookup is in the :meth:`~object.__getattribute__` method for
793 object returned by :func:`super`.
794
795 A dotted lookup such as ``super(A, obj).m`` searches ``obj.__class__.__mro__``
796 for the base class ``B`` immediately following ``A`` and then returns
797 ``B.__dict__['m'].__get__(obj, A)``. If not a descriptor, ``m`` is returned
798 unchanged.
799
800 The full C implementation can be found in :c:func:`!super_getattro` in
801 :source:`Objects/typeobject.c`. A pure Python equivalent can be found in
802 `Guido's Tutorial
803 <https://www.python.org/download/releases/2.2.3/descrintro/#cooperation>`_.
804
805
806 Summary of invocation logic
807 ---------------------------
808
809 The mechanism for descriptors is embedded in the :meth:`~object.__getattribute__`
810 methods for :class:`object`, :class:`type`, and :func:`super`.
811
812 The important points to remember are:
813
814 * Descriptors are invoked by the :meth:`~object.__getattribute__` method.
815
816 * Classes inherit this machinery from :class:`object`, :class:`type`, or
817 :func:`super`.
818
819 * Overriding :meth:`~object.__getattribute__` prevents automatic descriptor calls
820 because all the descriptor logic is in that method.
821
822 * :meth:`!object.__getattribute__` and :meth:`!type.__getattribute__` make
823 different calls to :meth:`~object.__get__`. The first includes the instance and may
824 include the class. The second puts in ``None`` for the instance and always
825 includes the class.
826
827 * Data descriptors always override instance dictionaries.
828
829 * Non-data descriptors may be overridden by instance dictionaries.
830
831
832 Automatic name notification
833 ---------------------------
834
835 Sometimes it is desirable for a descriptor to know what class variable name it
836 was assigned to. When a new class is created, the :class:`type` metaclass
837 scans the dictionary of the new class. If any of the entries are descriptors
838 and if they define :meth:`~object.__set_name__`, that method is called with two
839 arguments. The *owner* is the class where the descriptor is used, and the
840 *name* is the class variable the descriptor was assigned to.
841
842 The implementation details are in :c:func:`!type_new` and
843 :c:func:`!set_names` in :source:`Objects/typeobject.c`.
844
845 Since the update logic is in :meth:`!type.__new__`, notifications only take
846 place at the time of class creation. If descriptors are added to the class
847 afterwards, :meth:`~object.__set_name__` will need to be called manually.
848
849
850 ORM example
851 -----------
852
853 The following code is a simplified skeleton showing how data descriptors could
854 be used to implement an `object relational mapping
855 <https://en.wikipedia.org/wiki/Object%E2%80%93relational_mapping>`_.
856
857 The essential idea is that the data is stored in an external database. The
858 Python instances only hold keys to the database's tables. Descriptors take
859 care of lookups or updates:
860
861 .. testcode::
862
863 class Field:
864
865 def __set_name__(self, owner, name):
866 self.fetch = f'SELECT {name} FROM {owner.table} WHERE {owner.key}=?;'
867 self.store = f'UPDATE {owner.table} SET {name}=? WHERE {owner.key}=?;'
868
869 def __get__(self, obj, objtype=None):
870 return conn.execute(self.fetch, [obj.key]).fetchone()[0]
871
872 def __set__(self, obj, value):
873 conn.execute(self.store, [value, obj.key])
874 conn.commit()
875
876 We can use the :class:`!Field` class to define `models
877 <https://en.wikipedia.org/wiki/Database_model>`_ that describe the schema for
878 each table in a database:
879
880 .. testcode::
881
882 class Movie:
883 table = 'Movies' # Table name
884 key = 'title' # Primary key
885 director = Field()
886 year = Field()
887
888 def __init__(self, key):
889 self.key = key
890
891 class Song:
892 table = 'Music'
893 key = 'title'
894 artist = Field()
895 year = Field()
896 genre = Field()
897
898 def __init__(self, key):
899 self.key = key
900
901 To use the models, first connect to the database::
902
903 >>> import sqlite3
904 >>> conn = sqlite3.connect('entertainment.db')
905
906 An interactive session shows how data is retrieved from the database and how
907 it can be updated:
908
909 .. testsetup::
910
911 song_data = [
912 ('Country Roads', 'John Denver', 1972),
913 ('Me and Bobby McGee', 'Janice Joplin', 1971),
914 ('Coal Miners Daughter', 'Loretta Lynn', 1970),
915 ]
916
917 movie_data = [
918 ('Star Wars', 'George Lucas', 1977),
919 ('Jaws', 'Steven Spielberg', 1975),
920 ('Aliens', 'James Cameron', 1986),
921 ]
922
923 import sqlite3
924
925 conn = sqlite3.connect(':memory:')
926 conn.execute('CREATE TABLE Music (title text, artist text, year integer);')
927 conn.execute('CREATE INDEX MusicNdx ON Music (title);')
928 conn.executemany('INSERT INTO Music VALUES (?, ?, ?);', song_data)
929 conn.execute('CREATE TABLE Movies (title text, director text, year integer);')
930 conn.execute('CREATE INDEX MovieNdx ON Music (title);')
931 conn.executemany('INSERT INTO Movies VALUES (?, ?, ?);', movie_data)
932 conn.commit()
933
934 .. doctest::
935
936 >>> Movie('Star Wars').director
937 'George Lucas'
938 >>> jaws = Movie('Jaws')
939 >>> f'Released in {jaws.year} by {jaws.director}'
940 'Released in 1975 by Steven Spielberg'
941
942 >>> Song('Country Roads').artist
943 'John Denver'
944
945 >>> Movie('Star Wars').director = 'J.J. Abrams'
946 >>> Movie('Star Wars').director
947 'J.J. Abrams'
948
949 .. testcleanup::
950
951 conn.close()
952
953
954 Pure Python Equivalents
955 ^^^^^^^^^^^^^^^^^^^^^^^
956
957 The descriptor protocol is simple and offers exciting possibilities. Several
958 use cases are so common that they have been prepackaged into built-in tools.
959 Properties, bound methods, static methods, class methods, and \_\_slots\_\_ are
960 all based on the descriptor protocol.
961
962
963 Properties
964 ----------
965
966 Calling :func:`property` is a succinct way of building a data descriptor that
967 triggers a function call upon access to an attribute. Its signature is::
968
969 property(fget=None, fset=None, fdel=None, doc=None) -> property
970
971 The documentation shows a typical use to define a managed attribute ``x``:
972
973 .. testcode::
974
975 class C:
976 def getx(self): return self.__x
977 def setx(self, value): self.__x = value
978 def delx(self): del self.__x
979 x = property(getx, setx, delx, "I'm the 'x' property.")
980
981 .. doctest::
982 :hide:
983
984 >>> C.x.__doc__
985 "I'm the 'x' property."
986 >>> c.x = 2.71828
987 >>> c.x
988 2.71828
989 >>> del c.x
990 >>> c.x
991 Traceback (most recent call last):
992 ...
993 AttributeError: 'C' object has no attribute '_C__x'
994
995 To see how :func:`property` is implemented in terms of the descriptor protocol,
996 here is a pure Python equivalent that implements most of the core functionality:
997
998 .. testcode::
999
1000 class Property:
1001 "Emulate PyProperty_Type() in Objects/descrobject.c"
1002
1003 def __init__(self, fget=None, fset=None, fdel=None, doc=None):
1004 self.fget = fget
1005 self.fset = fset
1006 self.fdel = fdel
1007 if doc is None and fget is not None:
1008 doc = fget.__doc__
1009 self.__doc__ = doc
1010
1011 def __set_name__(self, owner, name):
1012 self.__name__ = name
1013
1014 def __get__(self, obj, objtype=None):
1015 if obj is None:
1016 return self
1017 if self.fget is None:
1018 raise AttributeError
1019 return self.fget(obj)
1020
1021 def __set__(self, obj, value):
1022 if self.fset is None:
1023 raise AttributeError
1024 self.fset(obj, value)
1025
1026 def __delete__(self, obj):
1027 if self.fdel is None:
1028 raise AttributeError
1029 self.fdel(obj)
1030
1031 def getter(self, fget):
1032 return type(self)(fget, self.fset, self.fdel, self.__doc__)
1033
1034 def setter(self, fset):
1035 return type(self)(self.fget, fset, self.fdel, self.__doc__)
1036
1037 def deleter(self, fdel):
1038 return type(self)(self.fget, self.fset, fdel, self.__doc__)
1039
1040 .. testcode::
1041 :hide:
1042
1043 # Verify the Property() emulation
1044
1045 class CC:
1046 def getx(self):
1047 return self.__x
1048 def setx(self, value):
1049 self.__x = value
1050 def delx(self):
1051 del self.__x
1052 x = Property(getx, setx, delx, "I'm the 'x' property.")
1053 no_getter = Property(None, setx, delx, "I'm the 'x' property.")
1054 no_setter = Property(getx, None, delx, "I'm the 'x' property.")
1055 no_deleter = Property(getx, setx, None, "I'm the 'x' property.")
1056 no_doc = Property(getx, setx, delx, None)
1057
1058
1059 # Now do it again but use the decorator style
1060
1061 class CCC:
1062 @Property
1063 def x(self):
1064 return self.__x
1065 @x.setter
1066 def x(self, value):
1067 self.__x = value
1068 @x.deleter
1069 def x(self):
1070 del self.__x
1071
1072
1073 .. doctest::
1074 :hide:
1075
1076 >>> cc = CC()
1077 >>> hasattr(cc, 'x')
1078 False
1079 >>> cc.x = 33
1080 >>> cc.x
1081 33
1082 >>> del cc.x
1083 >>> hasattr(cc, 'x')
1084 False
1085
1086 >>> ccc = CCC()
1087 >>> hasattr(ccc, 'x')
1088 False
1089 >>> ccc.x = 333
1090 >>> ccc.x == 333
1091 True
1092 >>> del ccc.x
1093 >>> hasattr(ccc, 'x')
1094 False
1095
1096 >>> cc = CC()
1097 >>> cc.x = 33
1098 >>> try:
1099 ... cc.no_getter
1100 ... except AttributeError as e:
1101 ... type(e).__name__
1102 ...
1103 'AttributeError'
1104
1105 >>> try:
1106 ... cc.no_setter = 33
1107 ... except AttributeError as e:
1108 ... type(e).__name__
1109 ...
1110 'AttributeError'
1111
1112 >>> try:
1113 ... del cc.no_deleter
1114 ... except AttributeError as e:
1115 ... type(e).__name__
1116 ...
1117 'AttributeError'
1118
1119 >>> CC.no_doc.__doc__ is None
1120 True
1121
1122 The :func:`property` builtin helps whenever a user interface has granted
1123 attribute access and then subsequent changes require the intervention of a
1124 method.
1125
1126 For instance, a spreadsheet class may grant access to a cell value through
1127 ``Cell('b10').value``. Subsequent improvements to the program require the cell
1128 to be recalculated on every access; however, the programmer does not want to
1129 affect existing client code accessing the attribute directly. The solution is
1130 to wrap access to the value attribute in a property data descriptor:
1131
1132 .. testcode::
1133
1134 class Cell:
1135 ...
1136
1137 @property
1138 def value(self):
1139 "Recalculate the cell before returning value"
1140 self.recalc()
1141 return self._value
1142
1143 Either the built-in :func:`property` or our :func:`!Property` equivalent would
1144 work in this example.
1145
1146
1147 Functions and methods
1148 ---------------------
1149
1150 Python's object oriented features are built upon a function based environment.
1151 Using non-data descriptors, the two are merged seamlessly.
1152
1153 Functions stored in class dictionaries get turned into methods when invoked.
1154 Methods only differ from regular functions in that the object instance is
1155 prepended to the other arguments. By convention, the instance is called
1156 *self* but could be called *this* or any other variable name.
1157
1158 Methods can be created manually with :class:`types.MethodType` which is
1159 roughly equivalent to:
1160
1161 .. testcode::
1162
1163 class MethodType:
1164 "Emulate PyMethod_Type in Objects/classobject.c"
1165
1166 def __init__(self, func, obj):
1167 self.__func__ = func
1168 self.__self__ = obj
1169
1170 def __call__(self, *args, **kwargs):
1171 func = self.__func__
1172 obj = self.__self__
1173 return func(obj, *args, **kwargs)
1174
1175 def __getattribute__(self, name):
1176 "Emulate method_getset() in Objects/classobject.c"
1177 if name == '__doc__':
1178 return self.__func__.__doc__
1179 return object.__getattribute__(self, name)
1180
1181 def __getattr__(self, name):
1182 "Emulate method_getattro() in Objects/classobject.c"
1183 return getattr(self.__func__, name)
1184
1185 def __get__(self, obj, objtype=None):
1186 "Emulate method_descr_get() in Objects/classobject.c"
1187 return self
1188
1189 To support automatic creation of methods, functions include the
1190 :meth:`~object.__get__` method for binding methods during attribute access. This
1191 means that functions are non-data descriptors that return bound methods
1192 during dotted lookup from an instance. Here's how it works:
1193
1194 .. testcode::
1195
1196 class Function:
1197 ...
1198
1199 def __get__(self, obj, objtype=None):
1200 "Simulate func_descr_get() in Objects/funcobject.c"
1201 if obj is None:
1202 return self
1203 return MethodType(self, obj)
1204
1205 Running the following class in the interpreter shows how the function
1206 descriptor works in practice:
1207
1208 .. testcode::
1209
1210 class D:
1211 def f(self):
1212 return self
1213
1214 class D2:
1215 pass
1216
1217 .. doctest::
1218 :hide:
1219
1220 >>> d = D()
1221 >>> d2 = D2()
1222 >>> d2.f = d.f.__get__(d2, D2)
1223 >>> d2.f() is d
1224 True
1225
1226 The function has a :term:`qualified name` attribute to support introspection:
1227
1228 .. doctest::
1229
1230 >>> D.f.__qualname__
1231 'D.f'
1232
1233 Accessing the function through the class dictionary does not invoke
1234 :meth:`~object.__get__`. Instead, it just returns the underlying function object::
1235
1236 >>> D.__dict__['f']
1237 <function D.f at 0x00C45070>
1238
1239 Dotted access from a class calls :meth:`~object.__get__` which just returns the
1240 underlying function unchanged::
1241
1242 >>> D.f
1243 <function D.f at 0x00C45070>
1244
1245 The interesting behavior occurs during dotted access from an instance. The
1246 dotted lookup calls :meth:`~object.__get__` which returns a bound method object::
1247
1248 >>> d = D()
1249 >>> d.f
1250 <bound method D.f of <__main__.D object at 0x00B18C90>>
1251
1252 Internally, the bound method stores the underlying function and the bound
1253 instance::
1254
1255 >>> d.f.__func__
1256 <function D.f at 0x00C45070>
1257
1258 >>> d.f.__self__
1259 <__main__.D object at 0x00B18C90>
1260
1261 If you have ever wondered where *self* comes from in regular methods or where
1262 *cls* comes from in class methods, this is it!
1263
1264
1265 Kinds of methods
1266 ----------------
1267
1268 Non-data descriptors provide a simple mechanism for variations on the usual
1269 patterns of binding functions into methods.
1270
1271 To recap, functions have a :meth:`~object.__get__` method so that they can be converted
1272 to a method when accessed as attributes. The non-data descriptor transforms an
1273 ``obj.f(*args)`` call into ``f(obj, *args)``. Calling ``cls.f(*args)``
1274 becomes ``f(*args)``.
1275
1276 This chart summarizes the binding and its two most useful variants:
1277
1278 +-----------------+----------------------+------------------+
1279 | Transformation | Called from an | Called from a |
1280 | | object | class |
1281 +=================+======================+==================+
1282 | function | f(obj, \*args) | f(\*args) |
1283 +-----------------+----------------------+------------------+
1284 | staticmethod | f(\*args) | f(\*args) |
1285 +-----------------+----------------------+------------------+
1286 | classmethod | f(type(obj), \*args) | f(cls, \*args) |
1287 +-----------------+----------------------+------------------+
1288
1289
1290 Static methods
1291 --------------
1292
1293 Static methods return the underlying function without changes. Calling either
1294 ``c.f`` or ``C.f`` is the equivalent of a direct lookup into
1295 ``object.__getattribute__(c, "f")`` or ``object.__getattribute__(C, "f")``. As a
1296 result, the function becomes identically accessible from either an object or a
1297 class.
1298
1299 Good candidates for static methods are methods that do not reference the
1300 ``self`` variable.
1301
1302 For instance, a statistics package may include a container class for
1303 experimental data. The class provides normal methods for computing the average,
1304 mean, median, and other descriptive statistics that depend on the data. However,
1305 there may be useful functions which are conceptually related but do not depend
1306 on the data. For instance, ``erf(x)`` is handy conversion routine that comes up
1307 in statistical work but does not directly depend on a particular dataset.
1308 It can be called either from an object or the class: ``s.erf(1.5) --> 0.9332``
1309 or ``Sample.erf(1.5) --> 0.9332``.
1310
1311 Since static methods return the underlying function with no changes, the
1312 example calls are unexciting:
1313
1314 .. testcode::
1315
1316 class E:
1317 @staticmethod
1318 def f(x):
1319 return x * 10
1320
1321 .. doctest::
1322
1323 >>> E.f(3)
1324 30
1325 >>> E().f(3)
1326 30
1327
1328 Using the non-data descriptor protocol, a pure Python version of
1329 :func:`staticmethod` would look like this:
1330
1331 .. testcode::
1332
1333 import functools
1334
1335 class StaticMethod:
1336 "Emulate PyStaticMethod_Type() in Objects/funcobject.c"
1337
1338 def __init__(self, f):
1339 self.f = f
1340 functools.update_wrapper(self, f)
1341
1342 def __get__(self, obj, objtype=None):
1343 return self.f
1344
1345 def __call__(self, *args, **kwds):
1346 return self.f(*args, **kwds)
1347
1348 @property
1349 def __annotations__(self):
1350 return self.f.__annotations__
1351
1352 The :func:`functools.update_wrapper` call adds a ``__wrapped__`` attribute
1353 that refers to the underlying function. Also it carries forward
1354 the attributes necessary to make the wrapper look like the wrapped
1355 function, including :attr:`~function.__name__`, :attr:`~function.__qualname__`,
1356 and :attr:`~function.__doc__`.
1357
1358 .. testcode::
1359 :hide:
1360
1361 class E_sim:
1362 @StaticMethod
1363 def f(x: int) -> str:
1364 "Simple function example"
1365 return "!" * x
1366
1367 wrapped_ord = StaticMethod(ord)
1368
1369 .. doctest::
1370 :hide:
1371
1372 >>> E_sim.f(3)
1373 '!!!'
1374 >>> E_sim().f(3)
1375 '!!!'
1376
1377 >>> sm = vars(E_sim)['f']
1378 >>> type(sm).__name__
1379 'StaticMethod'
1380 >>> f = E_sim.f
1381 >>> type(f).__name__
1382 'function'
1383 >>> sm.__name__
1384 'f'
1385 >>> f.__name__
1386 'f'
1387 >>> sm.__qualname__
1388 'E_sim.f'
1389 >>> f.__qualname__
1390 'E_sim.f'
1391 >>> sm.__doc__
1392 'Simple function example'
1393 >>> f.__doc__
1394 'Simple function example'
1395 >>> sm.__annotations__
1396 {'x': <class 'int'>, 'return': <class 'str'>}
1397 >>> f.__annotations__
1398 {'x': <class 'int'>, 'return': <class 'str'>}
1399 >>> sm.__module__ == f.__module__
1400 True
1401 >>> sm(3)
1402 '!!!'
1403 >>> f(3)
1404 '!!!'
1405
1406 >>> wrapped_ord('A')
1407 65
1408 >>> wrapped_ord.__module__ == ord.__module__
1409 True
1410 >>> wrapped_ord.__wrapped__ == ord
1411 True
1412 >>> wrapped_ord.__name__ == ord.__name__
1413 True
1414 >>> wrapped_ord.__qualname__ == ord.__qualname__
1415 True
1416 >>> wrapped_ord.__doc__ == ord.__doc__
1417 True
1418
1419
1420 Class methods
1421 -------------
1422
1423 Unlike static methods, class methods prepend the class reference to the
1424 argument list before calling the function. This format is the same
1425 for whether the caller is an object or a class:
1426
1427 .. testcode::
1428
1429 class F:
1430 @classmethod
1431 def f(cls, x):
1432 return cls.__name__, x
1433
1434 .. doctest::
1435
1436 >>> F.f(3)
1437 ('F', 3)
1438 >>> F().f(3)
1439 ('F', 3)
1440
1441 This behavior is useful whenever the method only needs to have a class
1442 reference and does not rely on data stored in a specific instance. One use for
1443 class methods is to create alternate class constructors. For example, the
1444 classmethod :func:`dict.fromkeys` creates a new dictionary from a list of
1445 keys. The pure Python equivalent is:
1446
1447 .. testcode::
1448
1449 class Dict(dict):
1450 @classmethod
1451 def fromkeys(cls, iterable, value=None):
1452 "Emulate dict_fromkeys() in Objects/dictobject.c"
1453 d = cls()
1454 for key in iterable:
1455 d[key] = value
1456 return d
1457
1458 Now a new dictionary of unique keys can be constructed like this:
1459
1460 .. doctest::
1461
1462 >>> d = Dict.fromkeys('abracadabra')
1463 >>> type(d) is Dict
1464 True
1465 >>> d
1466 {'a': None, 'b': None, 'r': None, 'c': None, 'd': None}
1467
1468 Using the non-data descriptor protocol, a pure Python version of
1469 :func:`classmethod` would look like this:
1470
1471 .. testcode::
1472
1473 import functools
1474
1475 class ClassMethod:
1476 "Emulate PyClassMethod_Type() in Objects/funcobject.c"
1477
1478 def __init__(self, f):
1479 self.f = f
1480 functools.update_wrapper(self, f)
1481
1482 def __get__(self, obj, cls=None):
1483 if cls is None:
1484 cls = type(obj)
1485 return MethodType(self.f, cls)
1486
1487 .. testcode::
1488 :hide:
1489
1490 # Verify the emulation works
1491 class T:
1492 @ClassMethod
1493 def cm(cls, x: int, y: str) -> tuple[str, int, str]:
1494 "Class method that returns a tuple"
1495 return (cls.__name__, x, y)
1496
1497
1498 .. doctest::
1499 :hide:
1500
1501 >>> T.cm(11, 22)
1502 ('T', 11, 22)
1503
1504 # Also call it from an instance
1505 >>> t = T()
1506 >>> t.cm(11, 22)
1507 ('T', 11, 22)
1508
1509 # Verify that T uses our emulation
1510 >>> type(vars(T)['cm']).__name__
1511 'ClassMethod'
1512
1513 # Verify that update_wrapper() correctly copied attributes
1514 >>> T.cm.__name__
1515 'cm'
1516 >>> T.cm.__qualname__
1517 'T.cm'
1518 >>> T.cm.__doc__
1519 'Class method that returns a tuple'
1520 >>> T.cm.__annotations__
1521 {'x': <class 'int'>, 'y': <class 'str'>, 'return': tuple[str, int, str]}
1522
1523 # Verify that __wrapped__ was added and works correctly
1524 >>> f = vars(T)['cm'].__wrapped__
1525 >>> type(f).__name__
1526 'function'
1527 >>> f.__name__
1528 'cm'
1529 >>> f(T, 11, 22)
1530 ('T', 11, 22)
1531
1532
1533 The :func:`functools.update_wrapper` call in ``ClassMethod`` adds a
1534 ``__wrapped__`` attribute that refers to the underlying function. Also
1535 it carries forward the attributes necessary to make the wrapper look
1536 like the wrapped function: :attr:`~function.__name__`,
1537 :attr:`~function.__qualname__`, :attr:`~function.__doc__`,
1538 and :attr:`~function.__annotations__`.
1539
1540
1541 Member objects and __slots__
1542 ----------------------------
1543
1544 When a class defines ``__slots__``, it replaces instance dictionaries with a
1545 fixed-length array of slot values. From a user point of view that has
1546 several effects:
1547
1548 1. Provides immediate detection of bugs due to misspelled attribute
1549 assignments. Only attribute names specified in ``__slots__`` are allowed:
1550
1551 .. testcode::
1552
1553 class Vehicle:
1554 __slots__ = ('id_number', 'make', 'model')
1555
1556 .. doctest::
1557
1558 >>> auto = Vehicle()
1559 >>> auto.id_nubmer = 'VYE483814LQEX'
1560 Traceback (most recent call last):
1561 ...
1562 AttributeError: 'Vehicle' object has no attribute 'id_nubmer'
1563
1564 2. Helps create immutable objects where descriptors manage access to private
1565 attributes stored in ``__slots__``:
1566
1567 .. testcode::
1568
1569 class Immutable:
1570
1571 __slots__ = ('_dept', '_name') # Replace the instance dictionary
1572
1573 def __init__(self, dept, name):
1574 self._dept = dept # Store to private attribute
1575 self._name = name # Store to private attribute
1576
1577 @property # Read-only descriptor
1578 def dept(self):
1579 return self._dept
1580
1581 @property
1582 def name(self): # Read-only descriptor
1583 return self._name
1584
1585 .. doctest::
1586
1587 >>> mark = Immutable('Botany', 'Mark Watney')
1588 >>> mark.dept
1589 'Botany'
1590 >>> mark.dept = 'Space Pirate'
1591 Traceback (most recent call last):
1592 ...
1593 AttributeError: property 'dept' of 'Immutable' object has no setter
1594 >>> mark.location = 'Mars'
1595 Traceback (most recent call last):
1596 ...
1597 AttributeError: 'Immutable' object has no attribute 'location'
1598
1599 3. Saves memory. On a 64-bit Linux build, an instance with two attributes
1600 takes 48 bytes with ``__slots__`` and 152 bytes without. This `flyweight
1601 design pattern <https://en.wikipedia.org/wiki/Flyweight_pattern>`_ likely only
1602 matters when a large number of instances are going to be created.
1603
1604 4. Improves speed. Reading instance variables is 35% faster with
1605 ``__slots__`` (as measured with Python 3.10 on an Apple M1 processor).
1606
1607 5. Blocks tools like :func:`functools.cached_property` which require an
1608 instance dictionary to function correctly:
1609
1610 .. testcode::
1611
1612 from functools import cached_property
1613
1614 class CP:
1615 __slots__ = () # Eliminates the instance dict
1616
1617 @cached_property # Requires an instance dict
1618 def pi(self):
1619 return 4 * sum((-1.0)**n / (2.0*n + 1.0)
1620 for n in reversed(range(100_000)))
1621
1622 .. doctest::
1623
1624 >>> CP().pi
1625 Traceback (most recent call last):
1626 ...
1627 TypeError: No '__dict__' attribute on 'CP' instance to cache 'pi' property.
1628
1629 It is not possible to create an exact drop-in pure Python version of
1630 ``__slots__`` because it requires direct access to C structures and control
1631 over object memory allocation. However, we can build a mostly faithful
1632 simulation where the actual C structure for slots is emulated by a private
1633 ``_slotvalues`` list. Reads and writes to that private structure are managed
1634 by member descriptors:
1635
1636 .. testcode::
1637
1638 null = object()
1639
1640 class Member:
1641
1642 def __init__(self, name, clsname, offset):
1643 'Emulate PyMemberDef in Include/descrobject.h'
1644 # Also see descr_new() in Objects/descrobject.c
1645 self.name = name
1646 self.clsname = clsname
1647 self.offset = offset
1648
1649 def __get__(self, obj, objtype=None):
1650 'Emulate member_get() in Objects/descrobject.c'
1651 # Also see PyMember_GetOne() in Python/structmember.c
1652 if obj is None:
1653 return self
1654 value = obj._slotvalues[self.offset]
1655 if value is null:
1656 raise AttributeError(self.name)
1657 return value
1658
1659 def __set__(self, obj, value):
1660 'Emulate member_set() in Objects/descrobject.c'
1661 obj._slotvalues[self.offset] = value
1662
1663 def __delete__(self, obj):
1664 'Emulate member_delete() in Objects/descrobject.c'
1665 value = obj._slotvalues[self.offset]
1666 if value is null:
1667 raise AttributeError(self.name)
1668 obj._slotvalues[self.offset] = null
1669
1670 def __repr__(self):
1671 'Emulate member_repr() in Objects/descrobject.c'
1672 return f'<Member {self.name!r} of {self.clsname!r}>'
1673
1674 The :meth:`!type.__new__` method takes care of adding member objects to class
1675 variables:
1676
1677 .. testcode::
1678
1679 class Type(type):
1680 'Simulate how the type metaclass adds member objects for slots'
1681
1682 def __new__(mcls, clsname, bases, mapping, **kwargs):
1683 'Emulate type_new() in Objects/typeobject.c'
1684 # type_new() calls PyTypeReady() which calls add_methods()
1685 slot_names = mapping.get('slot_names', [])
1686 for offset, name in enumerate(slot_names):
1687 mapping[name] = Member(name, clsname, offset)
1688 return type.__new__(mcls, clsname, bases, mapping, **kwargs)
1689
1690 The :meth:`object.__new__` method takes care of creating instances that have
1691 slots instead of an instance dictionary. Here is a rough simulation in pure
1692 Python:
1693
1694 .. testcode::
1695
1696 class Object:
1697 'Simulate how object.__new__() allocates memory for __slots__'
1698
1699 def __new__(cls, *args, **kwargs):
1700 'Emulate object_new() in Objects/typeobject.c'
1701 inst = super().__new__(cls)
1702 if hasattr(cls, 'slot_names'):
1703 empty_slots = [null] * len(cls.slot_names)
1704 object.__setattr__(inst, '_slotvalues', empty_slots)
1705 return inst
1706
1707 def __setattr__(self, name, value):
1708 'Emulate _PyObject_GenericSetAttrWithDict() Objects/object.c'
1709 cls = type(self)
1710 if hasattr(cls, 'slot_names') and name not in cls.slot_names:
1711 raise AttributeError(
1712 f'{cls.__name__!r} object has no attribute {name!r}'
1713 )
1714 super().__setattr__(name, value)
1715
1716 def __delattr__(self, name):
1717 'Emulate _PyObject_GenericSetAttrWithDict() Objects/object.c'
1718 cls = type(self)
1719 if hasattr(cls, 'slot_names') and name not in cls.slot_names:
1720 raise AttributeError(
1721 f'{cls.__name__!r} object has no attribute {name!r}'
1722 )
1723 super().__delattr__(name)
1724
1725 To use the simulation in a real class, just inherit from :class:`!Object` and
1726 set the :term:`metaclass` to :class:`Type`:
1727
1728 .. testcode::
1729
1730 class H(Object, metaclass=Type):
1731 'Instance variables stored in slots'
1732
1733 slot_names = ['x', 'y']
1734
1735 def __init__(self, x, y):
1736 self.x = x
1737 self.y = y
1738
1739 At this point, the metaclass has loaded member objects for *x* and *y*::
1740
1741 >>> from pprint import pp
1742 >>> pp(dict(vars(H)))
1743 {'__module__': '__main__',
1744 '__doc__': 'Instance variables stored in slots',
1745 'slot_names': ['x', 'y'],
1746 '__init__': <function H.__init__ at 0x7fb5d302f9d0>,
1747 'x': <Member 'x' of 'H'>,
1748 'y': <Member 'y' of 'H'>}
1749
1750 .. doctest::
1751 :hide:
1752
1753 # We test this separately because the preceding section is not
1754 # doctestable due to the hex memory address for the __init__ function
1755 >>> isinstance(vars(H)['x'], Member)
1756 True
1757 >>> isinstance(vars(H)['y'], Member)
1758 True
1759
1760 When instances are created, they have a ``slot_values`` list where the
1761 attributes are stored:
1762
1763 .. doctest::
1764
1765 >>> h = H(10, 20)
1766 >>> vars(h)
1767 {'_slotvalues': [10, 20]}
1768 >>> h.x = 55
1769 >>> vars(h)
1770 {'_slotvalues': [55, 20]}
1771
1772 Misspelled or unassigned attributes will raise an exception:
1773
1774 .. doctest::
1775
1776 >>> h.xz
1777 Traceback (most recent call last):
1778 ...
1779 AttributeError: 'H' object has no attribute 'xz'
1780
1781 .. doctest::
1782 :hide:
1783
1784 # Examples for deleted attributes are not shown because this section
1785 # is already a bit lengthy. We still test that code here.
1786 >>> del h.x
1787 >>> hasattr(h, 'x')
1788 False
1789
1790 # Also test the code for uninitialized slots
1791 >>> class HU(Object, metaclass=Type):
1792 ... slot_names = ['x', 'y']
1793 ...
1794 >>> hu = HU()
1795 >>> hasattr(hu, 'x')
1796 False
1797 >>> hasattr(hu, 'y')
1798 False