Debian Codegraph @

Doc/library/annotationlib.rst

raw
1 :mod:`!annotationlib` --- Functionality for introspecting annotations
2 =====================================================================
3
4 .. module:: annotationlib
5 :synopsis: Functionality for introspecting annotations
6
7 .. versionadded:: 3.14
8
9 **Source code:** :source:`Lib/annotationlib.py`
10
11 .. testsetup:: default
12
13 import annotationlib
14 from annotationlib import *
15
16 --------------
17
18 The :mod:`!annotationlib` module provides tools for introspecting
19 :term:`annotations <annotation>` on modules, classes, and functions.
20
21 Annotations are :ref:`lazily evaluated <lazy-evaluation>` and often contain
22 forward references to objects that are not yet defined when the annotation
23 is created. This module provides a set of low-level tools that can be used to retrieve annotations in a reliable way, even
24 in the presence of forward references and other edge cases.
25
26 This module supports retrieving annotations in three main formats
27 (see :class:`Format`), each of which works best for different use cases:
28
29 * :attr:`~Format.VALUE` evaluates the annotations and returns their value.
30 This is most straightforward to work with, but it may raise errors,
31 for example if the annotations contain references to undefined names.
32 * :attr:`~Format.FORWARDREF` returns :class:`ForwardRef` objects
33 for annotations that cannot be resolved, allowing you to inspect the
34 annotations without evaluating them. This is useful when you need to
35 work with annotations that may contain unresolved forward references.
36 * :attr:`~Format.STRING` returns the annotations as a string, similar
37 to how it would appear in the source file. This is useful for documentation
38 generators that want to display annotations in a readable way.
39
40 The :func:`get_annotations` function is the main entry point for
41 retrieving annotations. Given a function, class, or module, it returns
42 an annotations dictionary in the requested format. This module also provides
43 functionality for working directly with the :term:`annotate function`
44 that is used to evaluate annotations, such as :func:`get_annotate_from_class_namespace`
45 and :func:`call_annotate_function`, as well as the
46 :func:`call_evaluate_function` function for working with
47 :term:`evaluate functions <evaluate function>`.
48
49 .. caution::
50
51 Most functionality in this module can execute arbitrary code; see
52 :ref:`the security section <annotationlib-security>` for more information.
53
54 .. seealso::
55
56 :pep:`649` proposed the current model for how annotations work in Python.
57
58 :pep:`749` expanded on various aspects of :pep:`649` and introduced the
59 :mod:`!annotationlib` module.
60
61 :ref:`annotations-howto` provides best practices for working with
62 annotations.
63
64 :pypi:`typing-extensions` provides a backport of :func:`get_annotations`
65 that works on earlier versions of Python.
66
67 Annotation semantics
68 --------------------
69
70 The way annotations are evaluated has changed over the history of Python 3,
71 and currently still depends on a :ref:`future import <future>`.
72 There have been execution models for annotations:
73
74 * *Stock semantics* (default in Python 3.0 through 3.13; see :pep:`3107`
75 and :pep:`526`): Annotations are evaluated eagerly, as they are
76 encountered in the source code.
77 * *Stringified annotations* (used with ``from __future__ import annotations``
78 in Python 3.7 and newer; see :pep:`563`): Annotations are stored as
79 strings only.
80 * *Deferred evaluation* (default in Python 3.14 and newer; see :pep:`649` and
81 :pep:`749`): Annotations are evaluated lazily, only when they are accessed.
82
83 As an example, consider the following program::
84
85 def func(a: Cls) -> None:
86 print(a)
87
88 class Cls: pass
89
90 print(func.__annotations__)
91
92 This will behave as follows:
93
94 * Under stock semantics (Python 3.13 and earlier), it will throw a
95 :exc:`NameError` at the line where ``func`` is defined,
96 because ``Cls`` is an undefined name at that point.
97 * Under stringified annotations (if ``from __future__ import annotations``
98 is used), it will print ``{'a': 'Cls', 'return': 'None'}``.
99 * Under deferred evaluation (Python 3.14 and later), it will print
100 ``{'a': <class 'Cls'>, 'return': None}``.
101
102 Stock semantics were used when function annotations were first introduced
103 in Python 3.0 (by :pep:`3107`) because this was the simplest, most obvious
104 way to implement annotations. The same execution model was used when variable
105 annotations were introduced in Python 3.6 (by :pep:`526`). However,
106 stock semantics caused problems when using annotations as type hints,
107 such as a need to refer to names that are not yet defined when the
108 annotation is encountered. In addition, there were performance problems
109 with executing annotations at module import time. Therefore, in Python 3.7,
110 :pep:`563` introduced the ability to store annotations as strings using the
111 ``from __future__ import annotations`` syntax. The plan at the time was to
112 eventually make this behavior the default, but a problem appeared:
113 stringified annotations are more difficult to process for those who
114 introspect annotations at runtime. An alternative proposal, :pep:`649`,
115 introduced the third execution model, deferred evaluation, and was implemented
116 in Python 3.14. Stringified annotations are still used if
117 ``from __future__ import annotations`` is present, but this behavior will
118 eventually be removed.
119
120 Classes
121 -------
122
123 .. class:: Format
124
125 An :class:`~enum.IntEnum` describing the formats in which annotations
126 can be returned. Members of the enum, or their equivalent integer values,
127 can be passed to :func:`get_annotations` and other functions in this
128 module, as well as to :attr:`~object.__annotate__` functions.
129
130 .. attribute:: VALUE
131 :value: 1
132
133 Values are the result of evaluating the annotation expressions.
134
135 .. attribute:: VALUE_WITH_FAKE_GLOBALS
136 :value: 2
137
138 Special value used to signal that an annotate function is being
139 evaluated in a special environment with fake globals. When passed this
140 value, annotate functions should either return the same value as for
141 the :attr:`Format.VALUE` format, or raise :exc:`NotImplementedError`
142 to signal that they do not support execution in this environment.
143 This format is only used internally and should not be passed to
144 the functions in this module.
145
146 .. attribute:: FORWARDREF
147 :value: 3
148
149 Values are real annotation values (as per :attr:`Format.VALUE` format)
150 for defined values, and :class:`ForwardRef` proxies for undefined
151 values. Real objects may contain references to :class:`ForwardRef`
152 proxy objects.
153
154 .. attribute:: STRING
155 :value: 4
156
157 Values are the text string of the annotation as it appears in the
158 source code, up to modifications including, but not restricted to,
159 whitespace normalizations and constant values optimizations.
160
161 The exact values of these strings may change in future versions of Python.
162
163 .. versionadded:: 3.14
164
165 .. class:: ForwardRef
166
167 A proxy object for forward references in annotations.
168
169 Instances of this class are returned when the :attr:`~Format.FORWARDREF`
170 format is used and annotations contain a name that cannot be resolved.
171 This can happen when a forward reference is used in an annotation, such as
172 when a class is referenced before it is defined.
173
174 .. attribute:: __forward_arg__
175
176 A string containing the code that was evaluated to produce the
177 :class:`~ForwardRef`. The string may not be exactly equivalent
178 to the original source.
179
180 .. method:: evaluate(*, owner=None, globals=None, locals=None, type_params=None, format=Format.VALUE)
181
182 Evaluate the forward reference, returning its value.
183
184 If the *format* argument is :attr:`~Format.VALUE` (the default),
185 this method may throw an exception, such as :exc:`NameError`, if the forward
186 reference refers to a name that cannot be resolved. The arguments to this
187 method can be used to provide bindings for names that would otherwise
188 be undefined. If the *format* argument is :attr:`~Format.FORWARDREF`,
189 the method will never throw an exception, but may return a :class:`~ForwardRef`
190 instance. For example, if the forward reference object contains the code
191 ``list[undefined]``, where ``undefined`` is a name that is not defined,
192 evaluating it with the :attr:`~Format.FORWARDREF` format will return
193 ``list[ForwardRef('undefined')]``. If the *format* argument is
194 :attr:`~Format.STRING`, the method will return :attr:`~ForwardRef.__forward_arg__`.
195
196 The *owner* parameter provides the preferred mechanism for passing scope
197 information to this method. The owner of a :class:`~ForwardRef` is the
198 object that contains the annotation from which the :class:`~ForwardRef`
199 derives, such as a module object, type object, or function object.
200
201 The *globals*, *locals*, and *type_params* parameters provide a more precise
202 mechanism for influencing the names that are available when the :class:`~ForwardRef`
203 is evaluated. *globals* and *locals* are passed to :func:`eval`, representing
204 the global and local namespaces in which the name is evaluated.
205 The *type_params* parameter is relevant for objects created using the native
206 syntax for :ref:`generic classes <generic-classes>` and :ref:`functions <generic-functions>`.
207 It is a tuple of :ref:`type parameters <type-params>` that are in scope
208 while the forward reference is being evaluated. For example, if evaluating a
209 :class:`~ForwardRef` retrieved from an annotation found in the class namespace
210 of a generic class ``C``, *type_params* should be set to ``C.__type_params__``.
211
212 :class:`~ForwardRef` instances returned by :func:`get_annotations`
213 retain references to information about the scope they originated from,
214 so calling this method with no further arguments may be sufficient to
215 evaluate such objects. :class:`~ForwardRef` instances created by other
216 means may not have any information about their scope, so passing
217 arguments to this method may be necessary to evaluate them successfully.
218
219 If no *owner*, *globals*, *locals*, or *type_params* are provided and the
220 :class:`~ForwardRef` does not contain information about its origin,
221 empty globals and locals dictionaries are used.
222
223 .. versionadded:: 3.14
224
225
226 Functions
227 ---------
228
229 .. function:: annotations_to_string(annotations)
230
231 Convert an annotations dict containing runtime values to a
232 dict containing only strings. If the values are not already strings,
233 they are converted using :func:`type_repr`.
234 This is meant as a helper for user-provided
235 annotate functions that support the :attr:`~Format.STRING` format but
236 do not have access to the code creating the annotations.
237
238 For example, this is used to implement the :attr:`~Format.STRING` for
239 :class:`typing.TypedDict` classes created through the functional syntax:
240
241 .. doctest::
242
243 >>> from typing import TypedDict
244 >>> Movie = TypedDict("movie", {"name": str, "year": int})
245 >>> get_annotations(Movie, format=Format.STRING)
246 {'name': 'str', 'year': 'int'}
247
248 .. versionadded:: 3.14
249
250 .. function:: call_annotate_function(annotate, format, *, owner=None)
251
252 Call the :term:`annotate function` *annotate* with the given *format*,
253 a member of the :class:`Format` enum, and return the annotations
254 dictionary produced by the function.
255
256 This helper function is required because annotate functions generated by
257 the compiler for functions, classes, and modules only support
258 the :attr:`~Format.VALUE` format when called directly.
259 To support other formats, this function calls the annotate function
260 in a special environment that allows it to produce annotations in the
261 other formats. This is a useful building block when implementing
262 functionality that needs to partially evaluate annotations while a class
263 is being constructed.
264
265 *owner* is the object that owns the annotation function, usually
266 a function, class, or module. If provided, it is used in the
267 :attr:`~Format.FORWARDREF` format to produce a :class:`ForwardRef`
268 object that carries more information.
269
270 .. seealso::
271
272 :PEP:`PEP 649 <649#the-stringizer-and-the-fake-globals-environment>`
273 contains an explanation of the implementation technique used by this
274 function.
275
276 .. versionadded:: 3.14
277
278 .. function:: call_evaluate_function(evaluate, format, *, owner=None)
279
280 Call the :term:`evaluate function` *evaluate* with the given *format*,
281 a member of the :class:`Format` enum, and return the value produced by
282 the function. This is similar to :func:`call_annotate_function`,
283 but the latter always returns a dictionary mapping strings to annotations,
284 while this function returns a single value.
285
286 This is intended for use with the evaluate functions generated for lazily
287 evaluated elements related to type aliases and type parameters:
288
289 * :meth:`typing.TypeAliasType.evaluate_value`, the value of type aliases
290 * :meth:`typing.TypeVar.evaluate_bound`, the bound of type variables
291 * :meth:`typing.TypeVar.evaluate_constraints`, the constraints of
292 type variables
293 * :meth:`typing.TypeVar.evaluate_default`, the default value of
294 type variables
295 * :meth:`typing.ParamSpec.evaluate_default`, the default value of
296 parameter specifications
297 * :meth:`typing.TypeVarTuple.evaluate_default`, the default value of
298 type variable tuples
299
300 *owner* is the object that owns the evaluate function, such as the type
301 alias or type variable object.
302
303 *format* can be used to control the format in which the value is returned:
304
305 .. doctest::
306
307 >>> type Alias = undefined
308 >>> call_evaluate_function(Alias.evaluate_value, Format.VALUE)
309 Traceback (most recent call last):
310 ...
311 NameError: name 'undefined' is not defined
312 >>> call_evaluate_function(Alias.evaluate_value, Format.FORWARDREF)
313 ForwardRef('undefined')
314 >>> call_evaluate_function(Alias.evaluate_value, Format.STRING)
315 'undefined'
316
317 .. versionadded:: 3.14
318
319 .. function:: get_annotate_from_class_namespace(namespace)
320
321 Retrieve the :term:`annotate function` from a class namespace dictionary *namespace*.
322 Return :const:`!None` if the namespace does not contain an annotate function.
323 This is primarily useful before the class has been fully created (e.g., in a metaclass);
324 after the class exists, the annotate function can be retrieved with ``cls.__annotate__``.
325 See :ref:`below <annotationlib-metaclass>` for an example using this function in a metaclass.
326
327 .. versionadded:: 3.14
328
329 .. function:: get_annotations(obj, *, globals=None, locals=None, eval_str=False, format=Format.VALUE)
330
331 Compute the annotations dict for an object.
332
333 *obj* may be a callable, class, module, or other object with
334 :attr:`~object.__annotate__` or :attr:`~object.__annotations__` attributes.
335 Passing any other object raises :exc:`TypeError`.
336
337 The *format* parameter controls the format in which annotations are returned,
338 and must be a member of the :class:`Format` enum or its integer equivalent.
339 The different formats work as follows:
340
341 * VALUE: :attr:`!object.__annotations__` is tried first; if that does not exist,
342 the :attr:`!object.__annotate__` function is called if it exists.
343
344 * FORWARDREF: If :attr:`!object.__annotations__` exists and can be evaluated successfully,
345 it is used; otherwise, the :attr:`!object.__annotate__` function is called. If it
346 does not exist either, :attr:`!object.__annotations__` is tried again and any error
347 from accessing it is re-raised.
348
349 * When calling :attr:`!object.__annotate__` it is first called with :attr:`~Format.FORWARDREF`.
350 If this is not implemented, it will then check if :attr:`~Format.VALUE_WITH_FAKE_GLOBALS`
351 is supported and use that in the fake globals environment.
352 If neither of these formats are supported, it will fall back to using :attr:`~Format.VALUE`.
353 If :attr:`~Format.VALUE` fails, the error from this call will be raised.
354
355 * STRING: If :attr:`!object.__annotate__` exists, it is called first;
356 otherwise, :attr:`!object.__annotations__` is used and stringified
357 using :func:`annotations_to_string`.
358
359 * When calling :attr:`!object.__annotate__` it is first called with :attr:`~Format.STRING`.
360 If this is not implemented, it will then check if :attr:`~Format.VALUE_WITH_FAKE_GLOBALS`
361 is supported and use that in the fake globals environment.
362 If neither of these formats are supported, it will fall back to using :attr:`~Format.VALUE`
363 with the result converted using :func:`annotations_to_string`.
364 If :attr:`~Format.VALUE` fails, the error from this call will be raised.
365
366 Returns a dict. :func:`!get_annotations` returns a new dict every time
367 it's called; calling it twice on the same object will return two
368 different but equivalent dicts.
369
370 This function handles several details for you:
371
372 * If *eval_str* is true, values of type :class:`!str` will
373 be un-stringized using :func:`eval`. This is intended
374 for use with stringized annotations
375 (``from __future__ import annotations``). It is an error
376 to set *eval_str* to true with formats other than :attr:`Format.VALUE`.
377 * If *obj* doesn't have an annotations dict, returns an
378 empty dict. (Functions and methods always have an
379 annotations dict; classes, modules, and other types of
380 callables may not.)
381 * Ignores inherited annotations on classes, as well as annotations
382 on metaclasses. If a class
383 doesn't have its own annotations dict, returns an empty dict.
384 * All accesses to object members and dict values are done
385 using ``getattr()`` and ``dict.get()`` for safety.
386
387 *eval_str* controls whether or not values of type :class:`!str` are
388 replaced with the result of calling :func:`eval` on those values:
389
390 * If eval_str is true, :func:`eval` is called on values of type
391 :class:`!str`. (Note that :func:`!get_annotations` doesn't catch
392 exceptions; if :func:`eval` raises an exception, it will unwind
393 the stack past the :func:`!get_annotations` call.)
394 * If *eval_str* is false (the default), values of type :class:`!str` are
395 unchanged.
396
397 *globals* and *locals* are passed in to :func:`eval`; see the documentation
398 for :func:`eval` for more information. If *globals* or *locals*
399 is :const:`!None`, this function may replace that value with a
400 context-specific default, contingent on ``type(obj)``:
401
402 * If *obj* is a module, *globals* defaults to ``obj.__dict__``.
403 * If *obj* is a class, *globals* defaults to
404 ``sys.modules[obj.__module__].__dict__`` and *locals* defaults
405 to the *obj* class namespace.
406 * If *obj* is a callable, *globals* defaults to
407 :attr:`obj.__globals__ <function.__globals__>`,
408 although if *obj* is a wrapped function (using
409 :func:`functools.update_wrapper`) or a :class:`functools.partial` object,
410 it is unwrapped until a non-wrapped function is found.
411
412 Calling :func:`!get_annotations` is best practice for accessing the
413 annotations dict of any object. See :ref:`annotations-howto` for
414 more information on annotations best practices.
415
416 .. doctest::
417
418 >>> def f(a: int, b: str) -> float:
419 ... pass
420 >>> get_annotations(f)
421 {'a': <class 'int'>, 'b': <class 'str'>, 'return': <class 'float'>}
422
423 .. versionadded:: 3.14
424
425 .. function:: type_repr(value)
426
427 Convert an arbitrary Python value to a format suitable for use by the
428 :attr:`~Format.STRING` format. This calls :func:`repr` for most
429 objects, but has special handling for some objects, such as type objects.
430
431 This is meant as a helper for user-provided
432 annotate functions that support the :attr:`~Format.STRING` format but
433 do not have access to the code creating the annotations. It can also
434 be used to provide a user-friendly string representation for other
435 objects that contain values that are commonly encountered in annotations.
436
437 .. versionadded:: 3.14
438
439
440 Recipes
441 -------
442
443 .. _annotationlib-metaclass:
444
445 Using annotations in a metaclass
446 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
447
448 A :ref:`metaclass <metaclasses>` may want to inspect or even modify the annotations
449 in a class body during class creation. Doing so requires retrieving annotations
450 from the class namespace dictionary. For classes created with
451 ``from __future__ import annotations``, the annotations will be in the ``__annotations__``
452 key of the dictionary. For other classes with annotations,
453 :func:`get_annotate_from_class_namespace` can be used to get the
454 annotate function, and :func:`call_annotate_function` can be used to call it and
455 retrieve the annotations. Using the :attr:`~Format.FORWARDREF` format will usually
456 be best, because this allows the annotations to refer to names that cannot yet be
457 resolved when the class is created.
458
459 To modify the annotations, it is best to create a wrapper annotate function
460 that calls the original annotate function, makes any necessary adjustments, and
461 returns the result.
462
463 Below is an example of a metaclass that filters out all :class:`typing.ClassVar`
464 annotations from the class and puts them in a separate attribute:
465
466 .. code-block:: python
467
468 import annotationlib
469 import typing
470
471 class ClassVarSeparator(type):
472 def __new__(mcls, name, bases, ns):
473 if "__annotations__" in ns: # from __future__ import annotations
474 annotations = ns["__annotations__"]
475 classvar_keys = {
476 key for key, value in annotations.items()
477 # Use string comparison for simplicity; a more robust solution
478 # could use annotationlib.ForwardRef.evaluate
479 if value.startswith("ClassVar")
480 }
481 classvars = {key: annotations[key] for key in classvar_keys}
482 ns["__annotations__"] = {
483 key: value for key, value in annotations.items()
484 if key not in classvar_keys
485 }
486 wrapped_annotate = None
487 elif annotate := annotationlib.get_annotate_from_class_namespace(ns):
488 annotations = annotationlib.call_annotate_function(
489 annotate, format=annotationlib.Format.FORWARDREF
490 )
491 classvar_keys = {
492 key for key, value in annotations.items()
493 if typing.get_origin(value) is typing.ClassVar
494 }
495 classvars = {key: annotations[key] for key in classvar_keys}
496
497 def wrapped_annotate(format):
498 annos = annotationlib.call_annotate_function(annotate, format, owner=typ)
499 return {key: value for key, value in annos.items() if key not in classvar_keys}
500
501 else: # no annotations
502 classvars = {}
503 wrapped_annotate = None
504 typ = super().__new__(mcls, name, bases, ns)
505
506 if wrapped_annotate is not None:
507 # Wrap the original __annotate__ with a wrapper that removes ClassVars
508 typ.__annotate__ = wrapped_annotate
509 typ.classvars = classvars # Store the ClassVars in a separate attribute
510 return typ
511
512
513 Creating a custom callable annotate function
514 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
515
516 Custom :term:`annotate functions <annotate function>` may be literal functions like those
517 automatically generated for functions, classes, and modules. Or, they may wish to utilise
518 the encapsulation provided by classes, in which case any :term:`callable` can be used as
519 an :term:`annotate function`.
520
521 To provide the :attr:`~Format.VALUE`, :attr:`~Format.STRING`, or
522 :attr:`~Format.FORWARDREF` formats directly, an :term:`annotate function` must provide
523 the following attribute:
524
525 * A callable ``__call__`` with signature ``__call__(format, /) -> dict``, that does not
526 raise a :exc:`NotImplementedError` when called with a supported format.
527
528 To provide the :attr:`~Format.VALUE_WITH_FAKE_GLOBALS` format, which is used to
529 automatically generate :attr:`~Format.STRING` or :attr:`~Format.FORWARDREF` if they are
530 not supported directly, :term:`annotate functions <annotate function>` must provide the
531 following attributes:
532
533 * A callable ``__call__`` with signature ``__call__(format, /) -> dict``, that does not
534 raise a :exc:`NotImplementedError` when called with
535 :attr:`~Format.VALUE_WITH_FAKE_GLOBALS`.
536 * A :ref:`code object <code-objects>` ``__code__`` containing the compiled code for the
537 annotate function.
538 * Optional: A tuple of the function's positional defaults ``__kwdefaults__``, if the
539 function represented by ``__code__`` uses any positional defaults.
540 * Optional: A dict of the function's keyword defaults ``__defaults__``, if the function
541 represented by ``__code__`` uses any keyword defaults.
542 * Optional: All other :ref:`function attributes <inspect-types>`.
543
544 .. code-block:: python
545
546 class Annotate:
547 called_formats = []
548
549 def __call__(self, format=None, /, *, _self=None):
550 # When called with fake globals, `_self` will be the
551 # actual self value, and `self` will be the format.
552 if _self is not None:
553 self, format = _self, self
554
555 self.called_formats.append(format)
556 if format <= 2: # VALUE or VALUE_WITH_FAKE_GLOBALS
557 return {"x": MyType}
558 raise NotImplementedError
559
560 __code__ = __call__.__code__
561 __defaults__ = (None,)
562 __kwdefaults__ = property(lambda self: dict(_self=self))
563
564 __globals__ = {}
565 __builtins__ = {}
566 __closure__ = None
567
568 This can then be called with:
569
570 .. code-block:: pycon
571
572 >>> from annotationlib import call_annotate_function, Format
573 >>> call_annotate_function(Annotate(), format=Format.STRING)
574 {'x': 'MyType'}
575
576 Or used as the annotate function for an object:
577
578 .. code-block:: pycon
579
580 >>> from annotationlib import get_annotations, Format
581 >>> class C:
582 ... pass
583 >>> C.__annotate__ = Annotate()
584 >>> get_annotations(Annotate(), format=Format.STRING)
585 {'x': 'MyType'}
586
587
588 Limitations of the ``STRING`` format
589 ------------------------------------
590
591 The :attr:`~Format.STRING` format is meant to approximate the source code
592 of the annotation, but the implementation strategy used means that it is not
593 always possible to recover the exact source code.
594
595 First, the stringifier of course cannot recover any information that is not present in
596 the compiled code, including comments, whitespace, parenthesization, and operations that
597 get simplified by the compiler.
598
599 Second, the stringifier can intercept almost all operations that involve names looked
600 up in some scope, but it cannot intercept operations that operate fully on constants.
601 As a corollary, this also means it is not safe to request the ``STRING`` format on
602 untrusted code: Python is powerful enough that it is possible to achieve arbitrary
603 code execution even with no access to any globals or builtins. For example:
604
605 .. code-block:: pycon
606
607 >>> def f(x: (1).__class__.__base__.__subclasses__()[-1].__init__.__builtins__["print"]("Hello world")): pass
608 ...
609 >>> annotationlib.get_annotations(f, format=annotationlib.Format.STRING)
610 Hello world
611 {'x': 'None'}
612
613 .. note::
614 This particular example works as of the time of writing, but it relies on
615 implementation details and is not guaranteed to work in the future.
616
617 Among the different kinds of expressions that exist in Python,
618 as represented by the :mod:`ast` module, some expressions are supported,
619 meaning that the ``STRING`` format can generally recover the original source code;
620 others are unsupported, meaning that they may result in incorrect output or an error.
621
622 The following are supported (sometimes with caveats):
623
624 * :class:`ast.BinOp`
625 * :class:`ast.UnaryOp`
626
627 * :class:`ast.Invert` (``~``), :class:`ast.UAdd` (``+``), and :class:`ast.USub` (``-``) are supported
628 * :class:`ast.Not` (``not``) is not supported
629
630 * :class:`ast.Dict` (except when using ``**`` unpacking)
631 * :class:`ast.Set`
632 * :class:`ast.Compare`
633
634 * :class:`ast.Eq` and :class:`ast.NotEq` are supported
635 * :class:`ast.Lt`, :class:`ast.LtE`, :class:`ast.Gt`, and :class:`ast.GtE` are supported, but the operand may be flipped
636 * :class:`ast.Is`, :class:`ast.IsNot`, :class:`ast.In`, and :class:`ast.NotIn` are not supported
637
638 * :class:`ast.Call` (except when using ``**`` unpacking)
639 * :class:`ast.Constant` (though not the exact representation of the constant; for example, escape
640 sequences in strings are lost; hexadecimal numbers are converted to decimal)
641 * :class:`ast.Attribute` (assuming the value is not a constant)
642 * :class:`ast.Subscript` (assuming the value is not a constant)
643 * :class:`ast.Starred` (``*`` unpacking)
644 * :class:`ast.Name`
645 * :class:`ast.List`
646 * :class:`ast.Tuple`
647 * :class:`ast.Slice`
648
649 The following are unsupported, but throw an informative error when encountered by the
650 stringifier:
651
652 * :class:`ast.FormattedValue` (f-strings; error is not detected if conversion specifiers like ``!r``
653 are used)
654 * :class:`ast.JoinedStr` (f-strings)
655
656 The following are unsupported and result in incorrect output:
657
658 * :class:`ast.BoolOp` (``and`` and ``or``)
659 * :class:`ast.IfExp`
660 * :class:`ast.Lambda`
661 * :class:`ast.ListComp`
662 * :class:`ast.SetComp`
663 * :class:`ast.DictComp`
664 * :class:`ast.GeneratorExp`
665
666 The following are disallowed in annotation scopes and therefore not relevant:
667
668 * :class:`ast.NamedExpr` (``:=``)
669 * :class:`ast.Await`
670 * :class:`ast.Yield`
671 * :class:`ast.YieldFrom`
672
673
674 Limitations of the ``FORWARDREF`` format
675 ----------------------------------------
676
677 The :attr:`~Format.FORWARDREF` format aims to produce real values as much
678 as possible, with anything that cannot be resolved replaced with
679 :class:`ForwardRef` objects. It is affected by broadly the same Limitations
680 as the :attr:`~Format.STRING` format: annotations that perform operations on
681 literals or that use unsupported expression types may raise exceptions when
682 evaluated using the :attr:`~Format.FORWARDREF` format.
683
684 Below are a few examples of the behavior with unsupported expressions:
685
686 .. code-block:: pycon
687
688 >>> from annotationlib import get_annotations, Format
689 >>> def zerodiv(x: 1 / 0): ...
690 >>> get_annotations(zerodiv, format=Format.STRING)
691 Traceback (most recent call last):
692 ...
693 ZeroDivisionError: division by zero
694 >>> get_annotations(zerodiv, format=Format.FORWARDREF)
695 Traceback (most recent call last):
696 ...
697 ZeroDivisionError: division by zero
698 >>> def ifexp(x: 1 if y else 0): ...
699 >>> get_annotations(ifexp, format=Format.STRING)
700 {'x': '1'}
701
702 .. _annotationlib-security:
703
704 Security implications of introspecting annotations
705 --------------------------------------------------
706
707 Much of the functionality in this module involves executing code related to annotations,
708 which can then do arbitrary things. For example,
709 :func:`get_annotations` may call an arbitrary :term:`annotate function`, and
710 :meth:`ForwardRef.evaluate` may call :func:`eval` on an arbitrary string. Code contained
711 in an annotation might make arbitrary system calls, enter an infinite loop, or perform any
712 other operation. This is also true for any access of the :attr:`~object.__annotations__` attribute,
713 and for various functions in the :mod:`typing` module that work with annotations, such as
714 :func:`typing.get_type_hints`.
715
716 Any security issue arising from this also applies immediately after importing
717 code that may contain untrusted annotations: importing code can always cause arbitrary operations
718 to be performed. However, it is unsafe to accept strings or other input from an untrusted source and
719 pass them to any of the APIs for introspecting annotations, for example by editing an
720 ``__annotations__`` dictionary or directly creating a :class:`ForwardRef` object.