| 1 |
:mod:`!annotationlib` --- Functionality for introspecting annotations |
| 2 |
===================================================================== |
| 3 |
|
| 4 |
.. module:: annotationlib |
| 5 |
:synopsis: Functionality for introspecting annotations |
| 6 |
|
| 7 |
.. versionadded:: 3.14 |
| 8 |
|
| 9 |
**Source code:** :source:`Lib/annotationlib.py` |
| 10 |
|
| 11 |
.. testsetup:: default |
| 12 |
|
| 13 |
import annotationlib |
| 14 |
from annotationlib import * |
| 15 |
|
| 16 |
-------------- |
| 17 |
|
| 18 |
The :mod:`!annotationlib` module provides tools for introspecting |
| 19 |
:term:`annotations <annotation>` on modules, classes, and functions. |
| 20 |
|
| 21 |
Annotations are :ref:`lazily evaluated <lazy-evaluation>` and often contain |
| 22 |
forward references to objects that are not yet defined when the annotation |
| 23 |
is created. This module provides a set of low-level tools that can be used to retrieve annotations in a reliable way, even |
| 24 |
in the presence of forward references and other edge cases. |
| 25 |
|
| 26 |
This module supports retrieving annotations in three main formats |
| 27 |
(see :class:`Format`), each of which works best for different use cases: |
| 28 |
|
| 29 |
* :attr:`~Format.VALUE` evaluates the annotations and returns their value. |
| 30 |
This is most straightforward to work with, but it may raise errors, |
| 31 |
for example if the annotations contain references to undefined names. |
| 32 |
* :attr:`~Format.FORWARDREF` returns :class:`ForwardRef` objects |
| 33 |
for annotations that cannot be resolved, allowing you to inspect the |
| 34 |
annotations without evaluating them. This is useful when you need to |
| 35 |
work with annotations that may contain unresolved forward references. |
| 36 |
* :attr:`~Format.STRING` returns the annotations as a string, similar |
| 37 |
to how it would appear in the source file. This is useful for documentation |
| 38 |
generators that want to display annotations in a readable way. |
| 39 |
|
| 40 |
The :func:`get_annotations` function is the main entry point for |
| 41 |
retrieving annotations. Given a function, class, or module, it returns |
| 42 |
an annotations dictionary in the requested format. This module also provides |
| 43 |
functionality for working directly with the :term:`annotate function` |
| 44 |
that is used to evaluate annotations, such as :func:`get_annotate_from_class_namespace` |
| 45 |
and :func:`call_annotate_function`, as well as the |
| 46 |
:func:`call_evaluate_function` function for working with |
| 47 |
:term:`evaluate functions <evaluate function>`. |
| 48 |
|
| 49 |
.. caution:: |
| 50 |
|
| 51 |
Most functionality in this module can execute arbitrary code; see |
| 52 |
:ref:`the security section <annotationlib-security>` for more information. |
| 53 |
|
| 54 |
.. seealso:: |
| 55 |
|
| 56 |
:pep:`649` proposed the current model for how annotations work in Python. |
| 57 |
|
| 58 |
:pep:`749` expanded on various aspects of :pep:`649` and introduced the |
| 59 |
:mod:`!annotationlib` module. |
| 60 |
|
| 61 |
:ref:`annotations-howto` provides best practices for working with |
| 62 |
annotations. |
| 63 |
|
| 64 |
:pypi:`typing-extensions` provides a backport of :func:`get_annotations` |
| 65 |
that works on earlier versions of Python. |
| 66 |
|
| 67 |
Annotation semantics |
| 68 |
-------------------- |
| 69 |
|
| 70 |
The way annotations are evaluated has changed over the history of Python 3, |
| 71 |
and currently still depends on a :ref:`future import <future>`. |
| 72 |
There have been execution models for annotations: |
| 73 |
|
| 74 |
* *Stock semantics* (default in Python 3.0 through 3.13; see :pep:`3107` |
| 75 |
and :pep:`526`): Annotations are evaluated eagerly, as they are |
| 76 |
encountered in the source code. |
| 77 |
* *Stringified annotations* (used with ``from __future__ import annotations`` |
| 78 |
in Python 3.7 and newer; see :pep:`563`): Annotations are stored as |
| 79 |
strings only. |
| 80 |
* *Deferred evaluation* (default in Python 3.14 and newer; see :pep:`649` and |
| 81 |
:pep:`749`): Annotations are evaluated lazily, only when they are accessed. |
| 82 |
|
| 83 |
As an example, consider the following program:: |
| 84 |
|
| 85 |
def func(a: Cls) -> None: |
| 86 |
print(a) |
| 87 |
|
| 88 |
class Cls: pass |
| 89 |
|
| 90 |
print(func.__annotations__) |
| 91 |
|
| 92 |
This will behave as follows: |
| 93 |
|
| 94 |
* Under stock semantics (Python 3.13 and earlier), it will throw a |
| 95 |
:exc:`NameError` at the line where ``func`` is defined, |
| 96 |
because ``Cls`` is an undefined name at that point. |
| 97 |
* Under stringified annotations (if ``from __future__ import annotations`` |
| 98 |
is used), it will print ``{'a': 'Cls', 'return': 'None'}``. |
| 99 |
* Under deferred evaluation (Python 3.14 and later), it will print |
| 100 |
``{'a': <class 'Cls'>, 'return': None}``. |
| 101 |
|
| 102 |
Stock semantics were used when function annotations were first introduced |
| 103 |
in Python 3.0 (by :pep:`3107`) because this was the simplest, most obvious |
| 104 |
way to implement annotations. The same execution model was used when variable |
| 105 |
annotations were introduced in Python 3.6 (by :pep:`526`). However, |
| 106 |
stock semantics caused problems when using annotations as type hints, |
| 107 |
such as a need to refer to names that are not yet defined when the |
| 108 |
annotation is encountered. In addition, there were performance problems |
| 109 |
with executing annotations at module import time. Therefore, in Python 3.7, |
| 110 |
:pep:`563` introduced the ability to store annotations as strings using the |
| 111 |
``from __future__ import annotations`` syntax. The plan at the time was to |
| 112 |
eventually make this behavior the default, but a problem appeared: |
| 113 |
stringified annotations are more difficult to process for those who |
| 114 |
introspect annotations at runtime. An alternative proposal, :pep:`649`, |
| 115 |
introduced the third execution model, deferred evaluation, and was implemented |
| 116 |
in Python 3.14. Stringified annotations are still used if |
| 117 |
``from __future__ import annotations`` is present, but this behavior will |
| 118 |
eventually be removed. |
| 119 |
|
| 120 |
Classes |
| 121 |
------- |
| 122 |
|
| 123 |
.. class:: Format |
| 124 |
|
| 125 |
An :class:`~enum.IntEnum` describing the formats in which annotations |
| 126 |
can be returned. Members of the enum, or their equivalent integer values, |
| 127 |
can be passed to :func:`get_annotations` and other functions in this |
| 128 |
module, as well as to :attr:`~object.__annotate__` functions. |
| 129 |
|
| 130 |
.. attribute:: VALUE |
| 131 |
:value: 1 |
| 132 |
|
| 133 |
Values are the result of evaluating the annotation expressions. |
| 134 |
|
| 135 |
.. attribute:: VALUE_WITH_FAKE_GLOBALS |
| 136 |
:value: 2 |
| 137 |
|
| 138 |
Special value used to signal that an annotate function is being |
| 139 |
evaluated in a special environment with fake globals. When passed this |
| 140 |
value, annotate functions should either return the same value as for |
| 141 |
the :attr:`Format.VALUE` format, or raise :exc:`NotImplementedError` |
| 142 |
to signal that they do not support execution in this environment. |
| 143 |
This format is only used internally and should not be passed to |
| 144 |
the functions in this module. |
| 145 |
|
| 146 |
.. attribute:: FORWARDREF |
| 147 |
:value: 3 |
| 148 |
|
| 149 |
Values are real annotation values (as per :attr:`Format.VALUE` format) |
| 150 |
for defined values, and :class:`ForwardRef` proxies for undefined |
| 151 |
values. Real objects may contain references to :class:`ForwardRef` |
| 152 |
proxy objects. |
| 153 |
|
| 154 |
.. attribute:: STRING |
| 155 |
:value: 4 |
| 156 |
|
| 157 |
Values are the text string of the annotation as it appears in the |
| 158 |
source code, up to modifications including, but not restricted to, |
| 159 |
whitespace normalizations and constant values optimizations. |
| 160 |
|
| 161 |
The exact values of these strings may change in future versions of Python. |
| 162 |
|
| 163 |
.. versionadded:: 3.14 |
| 164 |
|
| 165 |
.. class:: ForwardRef |
| 166 |
|
| 167 |
A proxy object for forward references in annotations. |
| 168 |
|
| 169 |
Instances of this class are returned when the :attr:`~Format.FORWARDREF` |
| 170 |
format is used and annotations contain a name that cannot be resolved. |
| 171 |
This can happen when a forward reference is used in an annotation, such as |
| 172 |
when a class is referenced before it is defined. |
| 173 |
|
| 174 |
.. attribute:: __forward_arg__ |
| 175 |
|
| 176 |
A string containing the code that was evaluated to produce the |
| 177 |
:class:`~ForwardRef`. The string may not be exactly equivalent |
| 178 |
to the original source. |
| 179 |
|
| 180 |
.. method:: evaluate(*, owner=None, globals=None, locals=None, type_params=None, format=Format.VALUE) |
| 181 |
|
| 182 |
Evaluate the forward reference, returning its value. |
| 183 |
|
| 184 |
If the *format* argument is :attr:`~Format.VALUE` (the default), |
| 185 |
this method may throw an exception, such as :exc:`NameError`, if the forward |
| 186 |
reference refers to a name that cannot be resolved. The arguments to this |
| 187 |
method can be used to provide bindings for names that would otherwise |
| 188 |
be undefined. If the *format* argument is :attr:`~Format.FORWARDREF`, |
| 189 |
the method will never throw an exception, but may return a :class:`~ForwardRef` |
| 190 |
instance. For example, if the forward reference object contains the code |
| 191 |
``list[undefined]``, where ``undefined`` is a name that is not defined, |
| 192 |
evaluating it with the :attr:`~Format.FORWARDREF` format will return |
| 193 |
``list[ForwardRef('undefined')]``. If the *format* argument is |
| 194 |
:attr:`~Format.STRING`, the method will return :attr:`~ForwardRef.__forward_arg__`. |
| 195 |
|
| 196 |
The *owner* parameter provides the preferred mechanism for passing scope |
| 197 |
information to this method. The owner of a :class:`~ForwardRef` is the |
| 198 |
object that contains the annotation from which the :class:`~ForwardRef` |
| 199 |
derives, such as a module object, type object, or function object. |
| 200 |
|
| 201 |
The *globals*, *locals*, and *type_params* parameters provide a more precise |
| 202 |
mechanism for influencing the names that are available when the :class:`~ForwardRef` |
| 203 |
is evaluated. *globals* and *locals* are passed to :func:`eval`, representing |
| 204 |
the global and local namespaces in which the name is evaluated. |
| 205 |
The *type_params* parameter is relevant for objects created using the native |
| 206 |
syntax for :ref:`generic classes <generic-classes>` and :ref:`functions <generic-functions>`. |
| 207 |
It is a tuple of :ref:`type parameters <type-params>` that are in scope |
| 208 |
while the forward reference is being evaluated. For example, if evaluating a |
| 209 |
:class:`~ForwardRef` retrieved from an annotation found in the class namespace |
| 210 |
of a generic class ``C``, *type_params* should be set to ``C.__type_params__``. |
| 211 |
|
| 212 |
:class:`~ForwardRef` instances returned by :func:`get_annotations` |
| 213 |
retain references to information about the scope they originated from, |
| 214 |
so calling this method with no further arguments may be sufficient to |
| 215 |
evaluate such objects. :class:`~ForwardRef` instances created by other |
| 216 |
means may not have any information about their scope, so passing |
| 217 |
arguments to this method may be necessary to evaluate them successfully. |
| 218 |
|
| 219 |
If no *owner*, *globals*, *locals*, or *type_params* are provided and the |
| 220 |
:class:`~ForwardRef` does not contain information about its origin, |
| 221 |
empty globals and locals dictionaries are used. |
| 222 |
|
| 223 |
.. versionadded:: 3.14 |
| 224 |
|
| 225 |
|
| 226 |
Functions |
| 227 |
--------- |
| 228 |
|
| 229 |
.. function:: annotations_to_string(annotations) |
| 230 |
|
| 231 |
Convert an annotations dict containing runtime values to a |
| 232 |
dict containing only strings. If the values are not already strings, |
| 233 |
they are converted using :func:`type_repr`. |
| 234 |
This is meant as a helper for user-provided |
| 235 |
annotate functions that support the :attr:`~Format.STRING` format but |
| 236 |
do not have access to the code creating the annotations. |
| 237 |
|
| 238 |
For example, this is used to implement the :attr:`~Format.STRING` for |
| 239 |
:class:`typing.TypedDict` classes created through the functional syntax: |
| 240 |
|
| 241 |
.. doctest:: |
| 242 |
|
| 243 |
>>> from typing import TypedDict |
| 244 |
>>> Movie = TypedDict("movie", {"name": str, "year": int}) |
| 245 |
>>> get_annotations(Movie, format=Format.STRING) |
| 246 |
{'name': 'str', 'year': 'int'} |
| 247 |
|
| 248 |
.. versionadded:: 3.14 |
| 249 |
|
| 250 |
.. function:: call_annotate_function(annotate, format, *, owner=None) |
| 251 |
|
| 252 |
Call the :term:`annotate function` *annotate* with the given *format*, |
| 253 |
a member of the :class:`Format` enum, and return the annotations |
| 254 |
dictionary produced by the function. |
| 255 |
|
| 256 |
This helper function is required because annotate functions generated by |
| 257 |
the compiler for functions, classes, and modules only support |
| 258 |
the :attr:`~Format.VALUE` format when called directly. |
| 259 |
To support other formats, this function calls the annotate function |
| 260 |
in a special environment that allows it to produce annotations in the |
| 261 |
other formats. This is a useful building block when implementing |
| 262 |
functionality that needs to partially evaluate annotations while a class |
| 263 |
is being constructed. |
| 264 |
|
| 265 |
*owner* is the object that owns the annotation function, usually |
| 266 |
a function, class, or module. If provided, it is used in the |
| 267 |
:attr:`~Format.FORWARDREF` format to produce a :class:`ForwardRef` |
| 268 |
object that carries more information. |
| 269 |
|
| 270 |
.. seealso:: |
| 271 |
|
| 272 |
:PEP:`PEP 649 <649#the-stringizer-and-the-fake-globals-environment>` |
| 273 |
contains an explanation of the implementation technique used by this |
| 274 |
function. |
| 275 |
|
| 276 |
.. versionadded:: 3.14 |
| 277 |
|
| 278 |
.. function:: call_evaluate_function(evaluate, format, *, owner=None) |
| 279 |
|
| 280 |
Call the :term:`evaluate function` *evaluate* with the given *format*, |
| 281 |
a member of the :class:`Format` enum, and return the value produced by |
| 282 |
the function. This is similar to :func:`call_annotate_function`, |
| 283 |
but the latter always returns a dictionary mapping strings to annotations, |
| 284 |
while this function returns a single value. |
| 285 |
|
| 286 |
This is intended for use with the evaluate functions generated for lazily |
| 287 |
evaluated elements related to type aliases and type parameters: |
| 288 |
|
| 289 |
* :meth:`typing.TypeAliasType.evaluate_value`, the value of type aliases |
| 290 |
* :meth:`typing.TypeVar.evaluate_bound`, the bound of type variables |
| 291 |
* :meth:`typing.TypeVar.evaluate_constraints`, the constraints of |
| 292 |
type variables |
| 293 |
* :meth:`typing.TypeVar.evaluate_default`, the default value of |
| 294 |
type variables |
| 295 |
* :meth:`typing.ParamSpec.evaluate_default`, the default value of |
| 296 |
parameter specifications |
| 297 |
* :meth:`typing.TypeVarTuple.evaluate_default`, the default value of |
| 298 |
type variable tuples |
| 299 |
|
| 300 |
*owner* is the object that owns the evaluate function, such as the type |
| 301 |
alias or type variable object. |
| 302 |
|
| 303 |
*format* can be used to control the format in which the value is returned: |
| 304 |
|
| 305 |
.. doctest:: |
| 306 |
|
| 307 |
>>> type Alias = undefined |
| 308 |
>>> call_evaluate_function(Alias.evaluate_value, Format.VALUE) |
| 309 |
Traceback (most recent call last): |
| 310 |
... |
| 311 |
NameError: name 'undefined' is not defined |
| 312 |
>>> call_evaluate_function(Alias.evaluate_value, Format.FORWARDREF) |
| 313 |
ForwardRef('undefined') |
| 314 |
>>> call_evaluate_function(Alias.evaluate_value, Format.STRING) |
| 315 |
'undefined' |
| 316 |
|
| 317 |
.. versionadded:: 3.14 |
| 318 |
|
| 319 |
.. function:: get_annotate_from_class_namespace(namespace) |
| 320 |
|
| 321 |
Retrieve the :term:`annotate function` from a class namespace dictionary *namespace*. |
| 322 |
Return :const:`!None` if the namespace does not contain an annotate function. |
| 323 |
This is primarily useful before the class has been fully created (e.g., in a metaclass); |
| 324 |
after the class exists, the annotate function can be retrieved with ``cls.__annotate__``. |
| 325 |
See :ref:`below <annotationlib-metaclass>` for an example using this function in a metaclass. |
| 326 |
|
| 327 |
.. versionadded:: 3.14 |
| 328 |
|
| 329 |
.. function:: get_annotations(obj, *, globals=None, locals=None, eval_str=False, format=Format.VALUE) |
| 330 |
|
| 331 |
Compute the annotations dict for an object. |
| 332 |
|
| 333 |
*obj* may be a callable, class, module, or other object with |
| 334 |
:attr:`~object.__annotate__` or :attr:`~object.__annotations__` attributes. |
| 335 |
Passing any other object raises :exc:`TypeError`. |
| 336 |
|
| 337 |
The *format* parameter controls the format in which annotations are returned, |
| 338 |
and must be a member of the :class:`Format` enum or its integer equivalent. |
| 339 |
The different formats work as follows: |
| 340 |
|
| 341 |
* VALUE: :attr:`!object.__annotations__` is tried first; if that does not exist, |
| 342 |
the :attr:`!object.__annotate__` function is called if it exists. |
| 343 |
|
| 344 |
* FORWARDREF: If :attr:`!object.__annotations__` exists and can be evaluated successfully, |
| 345 |
it is used; otherwise, the :attr:`!object.__annotate__` function is called. If it |
| 346 |
does not exist either, :attr:`!object.__annotations__` is tried again and any error |
| 347 |
from accessing it is re-raised. |
| 348 |
|
| 349 |
* When calling :attr:`!object.__annotate__` it is first called with :attr:`~Format.FORWARDREF`. |
| 350 |
If this is not implemented, it will then check if :attr:`~Format.VALUE_WITH_FAKE_GLOBALS` |
| 351 |
is supported and use that in the fake globals environment. |
| 352 |
If neither of these formats are supported, it will fall back to using :attr:`~Format.VALUE`. |
| 353 |
If :attr:`~Format.VALUE` fails, the error from this call will be raised. |
| 354 |
|
| 355 |
* STRING: If :attr:`!object.__annotate__` exists, it is called first; |
| 356 |
otherwise, :attr:`!object.__annotations__` is used and stringified |
| 357 |
using :func:`annotations_to_string`. |
| 358 |
|
| 359 |
* When calling :attr:`!object.__annotate__` it is first called with :attr:`~Format.STRING`. |
| 360 |
If this is not implemented, it will then check if :attr:`~Format.VALUE_WITH_FAKE_GLOBALS` |
| 361 |
is supported and use that in the fake globals environment. |
| 362 |
If neither of these formats are supported, it will fall back to using :attr:`~Format.VALUE` |
| 363 |
with the result converted using :func:`annotations_to_string`. |
| 364 |
If :attr:`~Format.VALUE` fails, the error from this call will be raised. |
| 365 |
|
| 366 |
Returns a dict. :func:`!get_annotations` returns a new dict every time |
| 367 |
it's called; calling it twice on the same object will return two |
| 368 |
different but equivalent dicts. |
| 369 |
|
| 370 |
This function handles several details for you: |
| 371 |
|
| 372 |
* If *eval_str* is true, values of type :class:`!str` will |
| 373 |
be un-stringized using :func:`eval`. This is intended |
| 374 |
for use with stringized annotations |
| 375 |
(``from __future__ import annotations``). It is an error |
| 376 |
to set *eval_str* to true with formats other than :attr:`Format.VALUE`. |
| 377 |
* If *obj* doesn't have an annotations dict, returns an |
| 378 |
empty dict. (Functions and methods always have an |
| 379 |
annotations dict; classes, modules, and other types of |
| 380 |
callables may not.) |
| 381 |
* Ignores inherited annotations on classes, as well as annotations |
| 382 |
on metaclasses. If a class |
| 383 |
doesn't have its own annotations dict, returns an empty dict. |
| 384 |
* All accesses to object members and dict values are done |
| 385 |
using ``getattr()`` and ``dict.get()`` for safety. |
| 386 |
|
| 387 |
*eval_str* controls whether or not values of type :class:`!str` are |
| 388 |
replaced with the result of calling :func:`eval` on those values: |
| 389 |
|
| 390 |
* If eval_str is true, :func:`eval` is called on values of type |
| 391 |
:class:`!str`. (Note that :func:`!get_annotations` doesn't catch |
| 392 |
exceptions; if :func:`eval` raises an exception, it will unwind |
| 393 |
the stack past the :func:`!get_annotations` call.) |
| 394 |
* If *eval_str* is false (the default), values of type :class:`!str` are |
| 395 |
unchanged. |
| 396 |
|
| 397 |
*globals* and *locals* are passed in to :func:`eval`; see the documentation |
| 398 |
for :func:`eval` for more information. If *globals* or *locals* |
| 399 |
is :const:`!None`, this function may replace that value with a |
| 400 |
context-specific default, contingent on ``type(obj)``: |
| 401 |
|
| 402 |
* If *obj* is a module, *globals* defaults to ``obj.__dict__``. |
| 403 |
* If *obj* is a class, *globals* defaults to |
| 404 |
``sys.modules[obj.__module__].__dict__`` and *locals* defaults |
| 405 |
to the *obj* class namespace. |
| 406 |
* If *obj* is a callable, *globals* defaults to |
| 407 |
:attr:`obj.__globals__ <function.__globals__>`, |
| 408 |
although if *obj* is a wrapped function (using |
| 409 |
:func:`functools.update_wrapper`) or a :class:`functools.partial` object, |
| 410 |
it is unwrapped until a non-wrapped function is found. |
| 411 |
|
| 412 |
Calling :func:`!get_annotations` is best practice for accessing the |
| 413 |
annotations dict of any object. See :ref:`annotations-howto` for |
| 414 |
more information on annotations best practices. |
| 415 |
|
| 416 |
.. doctest:: |
| 417 |
|
| 418 |
>>> def f(a: int, b: str) -> float: |
| 419 |
... pass |
| 420 |
>>> get_annotations(f) |
| 421 |
{'a': <class 'int'>, 'b': <class 'str'>, 'return': <class 'float'>} |
| 422 |
|
| 423 |
.. versionadded:: 3.14 |
| 424 |
|
| 425 |
.. function:: type_repr(value) |
| 426 |
|
| 427 |
Convert an arbitrary Python value to a format suitable for use by the |
| 428 |
:attr:`~Format.STRING` format. This calls :func:`repr` for most |
| 429 |
objects, but has special handling for some objects, such as type objects. |
| 430 |
|
| 431 |
This is meant as a helper for user-provided |
| 432 |
annotate functions that support the :attr:`~Format.STRING` format but |
| 433 |
do not have access to the code creating the annotations. It can also |
| 434 |
be used to provide a user-friendly string representation for other |
| 435 |
objects that contain values that are commonly encountered in annotations. |
| 436 |
|
| 437 |
.. versionadded:: 3.14 |
| 438 |
|
| 439 |
|
| 440 |
Recipes |
| 441 |
------- |
| 442 |
|
| 443 |
.. _annotationlib-metaclass: |
| 444 |
|
| 445 |
Using annotations in a metaclass |
| 446 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 447 |
|
| 448 |
A :ref:`metaclass <metaclasses>` may want to inspect or even modify the annotations |
| 449 |
in a class body during class creation. Doing so requires retrieving annotations |
| 450 |
from the class namespace dictionary. For classes created with |
| 451 |
``from __future__ import annotations``, the annotations will be in the ``__annotations__`` |
| 452 |
key of the dictionary. For other classes with annotations, |
| 453 |
:func:`get_annotate_from_class_namespace` can be used to get the |
| 454 |
annotate function, and :func:`call_annotate_function` can be used to call it and |
| 455 |
retrieve the annotations. Using the :attr:`~Format.FORWARDREF` format will usually |
| 456 |
be best, because this allows the annotations to refer to names that cannot yet be |
| 457 |
resolved when the class is created. |
| 458 |
|
| 459 |
To modify the annotations, it is best to create a wrapper annotate function |
| 460 |
that calls the original annotate function, makes any necessary adjustments, and |
| 461 |
returns the result. |
| 462 |
|
| 463 |
Below is an example of a metaclass that filters out all :class:`typing.ClassVar` |
| 464 |
annotations from the class and puts them in a separate attribute: |
| 465 |
|
| 466 |
.. code-block:: python |
| 467 |
|
| 468 |
import annotationlib |
| 469 |
import typing |
| 470 |
|
| 471 |
class ClassVarSeparator(type): |
| 472 |
def __new__(mcls, name, bases, ns): |
| 473 |
if "__annotations__" in ns: # from __future__ import annotations |
| 474 |
annotations = ns["__annotations__"] |
| 475 |
classvar_keys = { |
| 476 |
key for key, value in annotations.items() |
| 477 |
# Use string comparison for simplicity; a more robust solution |
| 478 |
# could use annotationlib.ForwardRef.evaluate |
| 479 |
if value.startswith("ClassVar") |
| 480 |
} |
| 481 |
classvars = {key: annotations[key] for key in classvar_keys} |
| 482 |
ns["__annotations__"] = { |
| 483 |
key: value for key, value in annotations.items() |
| 484 |
if key not in classvar_keys |
| 485 |
} |
| 486 |
wrapped_annotate = None |
| 487 |
elif annotate := annotationlib.get_annotate_from_class_namespace(ns): |
| 488 |
annotations = annotationlib.call_annotate_function( |
| 489 |
annotate, format=annotationlib.Format.FORWARDREF |
| 490 |
) |
| 491 |
classvar_keys = { |
| 492 |
key for key, value in annotations.items() |
| 493 |
if typing.get_origin(value) is typing.ClassVar |
| 494 |
} |
| 495 |
classvars = {key: annotations[key] for key in classvar_keys} |
| 496 |
|
| 497 |
def wrapped_annotate(format): |
| 498 |
annos = annotationlib.call_annotate_function(annotate, format, owner=typ) |
| 499 |
return {key: value for key, value in annos.items() if key not in classvar_keys} |
| 500 |
|
| 501 |
else: # no annotations |
| 502 |
classvars = {} |
| 503 |
wrapped_annotate = None |
| 504 |
typ = super().__new__(mcls, name, bases, ns) |
| 505 |
|
| 506 |
if wrapped_annotate is not None: |
| 507 |
# Wrap the original __annotate__ with a wrapper that removes ClassVars |
| 508 |
typ.__annotate__ = wrapped_annotate |
| 509 |
typ.classvars = classvars # Store the ClassVars in a separate attribute |
| 510 |
return typ |
| 511 |
|
| 512 |
|
| 513 |
Creating a custom callable annotate function |
| 514 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 515 |
|
| 516 |
Custom :term:`annotate functions <annotate function>` may be literal functions like those |
| 517 |
automatically generated for functions, classes, and modules. Or, they may wish to utilise |
| 518 |
the encapsulation provided by classes, in which case any :term:`callable` can be used as |
| 519 |
an :term:`annotate function`. |
| 520 |
|
| 521 |
To provide the :attr:`~Format.VALUE`, :attr:`~Format.STRING`, or |
| 522 |
:attr:`~Format.FORWARDREF` formats directly, an :term:`annotate function` must provide |
| 523 |
the following attribute: |
| 524 |
|
| 525 |
* A callable ``__call__`` with signature ``__call__(format, /) -> dict``, that does not |
| 526 |
raise a :exc:`NotImplementedError` when called with a supported format. |
| 527 |
|
| 528 |
To provide the :attr:`~Format.VALUE_WITH_FAKE_GLOBALS` format, which is used to |
| 529 |
automatically generate :attr:`~Format.STRING` or :attr:`~Format.FORWARDREF` if they are |
| 530 |
not supported directly, :term:`annotate functions <annotate function>` must provide the |
| 531 |
following attributes: |
| 532 |
|
| 533 |
* A callable ``__call__`` with signature ``__call__(format, /) -> dict``, that does not |
| 534 |
raise a :exc:`NotImplementedError` when called with |
| 535 |
:attr:`~Format.VALUE_WITH_FAKE_GLOBALS`. |
| 536 |
* A :ref:`code object <code-objects>` ``__code__`` containing the compiled code for the |
| 537 |
annotate function. |
| 538 |
* Optional: A tuple of the function's positional defaults ``__kwdefaults__``, if the |
| 539 |
function represented by ``__code__`` uses any positional defaults. |
| 540 |
* Optional: A dict of the function's keyword defaults ``__defaults__``, if the function |
| 541 |
represented by ``__code__`` uses any keyword defaults. |
| 542 |
* Optional: All other :ref:`function attributes <inspect-types>`. |
| 543 |
|
| 544 |
.. code-block:: python |
| 545 |
|
| 546 |
class Annotate: |
| 547 |
called_formats = [] |
| 548 |
|
| 549 |
def __call__(self, format=None, /, *, _self=None): |
| 550 |
# When called with fake globals, `_self` will be the |
| 551 |
# actual self value, and `self` will be the format. |
| 552 |
if _self is not None: |
| 553 |
self, format = _self, self |
| 554 |
|
| 555 |
self.called_formats.append(format) |
| 556 |
if format <= 2: # VALUE or VALUE_WITH_FAKE_GLOBALS |
| 557 |
return {"x": MyType} |
| 558 |
raise NotImplementedError |
| 559 |
|
| 560 |
__code__ = __call__.__code__ |
| 561 |
__defaults__ = (None,) |
| 562 |
__kwdefaults__ = property(lambda self: dict(_self=self)) |
| 563 |
|
| 564 |
__globals__ = {} |
| 565 |
__builtins__ = {} |
| 566 |
__closure__ = None |
| 567 |
|
| 568 |
This can then be called with: |
| 569 |
|
| 570 |
.. code-block:: pycon |
| 571 |
|
| 572 |
>>> from annotationlib import call_annotate_function, Format |
| 573 |
>>> call_annotate_function(Annotate(), format=Format.STRING) |
| 574 |
{'x': 'MyType'} |
| 575 |
|
| 576 |
Or used as the annotate function for an object: |
| 577 |
|
| 578 |
.. code-block:: pycon |
| 579 |
|
| 580 |
>>> from annotationlib import get_annotations, Format |
| 581 |
>>> class C: |
| 582 |
... pass |
| 583 |
>>> C.__annotate__ = Annotate() |
| 584 |
>>> get_annotations(Annotate(), format=Format.STRING) |
| 585 |
{'x': 'MyType'} |
| 586 |
|
| 587 |
|
| 588 |
Limitations of the ``STRING`` format |
| 589 |
------------------------------------ |
| 590 |
|
| 591 |
The :attr:`~Format.STRING` format is meant to approximate the source code |
| 592 |
of the annotation, but the implementation strategy used means that it is not |
| 593 |
always possible to recover the exact source code. |
| 594 |
|
| 595 |
First, the stringifier of course cannot recover any information that is not present in |
| 596 |
the compiled code, including comments, whitespace, parenthesization, and operations that |
| 597 |
get simplified by the compiler. |
| 598 |
|
| 599 |
Second, the stringifier can intercept almost all operations that involve names looked |
| 600 |
up in some scope, but it cannot intercept operations that operate fully on constants. |
| 601 |
As a corollary, this also means it is not safe to request the ``STRING`` format on |
| 602 |
untrusted code: Python is powerful enough that it is possible to achieve arbitrary |
| 603 |
code execution even with no access to any globals or builtins. For example: |
| 604 |
|
| 605 |
.. code-block:: pycon |
| 606 |
|
| 607 |
>>> def f(x: (1).__class__.__base__.__subclasses__()[-1].__init__.__builtins__["print"]("Hello world")): pass |
| 608 |
... |
| 609 |
>>> annotationlib.get_annotations(f, format=annotationlib.Format.STRING) |
| 610 |
Hello world |
| 611 |
{'x': 'None'} |
| 612 |
|
| 613 |
.. note:: |
| 614 |
This particular example works as of the time of writing, but it relies on |
| 615 |
implementation details and is not guaranteed to work in the future. |
| 616 |
|
| 617 |
Among the different kinds of expressions that exist in Python, |
| 618 |
as represented by the :mod:`ast` module, some expressions are supported, |
| 619 |
meaning that the ``STRING`` format can generally recover the original source code; |
| 620 |
others are unsupported, meaning that they may result in incorrect output or an error. |
| 621 |
|
| 622 |
The following are supported (sometimes with caveats): |
| 623 |
|
| 624 |
* :class:`ast.BinOp` |
| 625 |
* :class:`ast.UnaryOp` |
| 626 |
|
| 627 |
* :class:`ast.Invert` (``~``), :class:`ast.UAdd` (``+``), and :class:`ast.USub` (``-``) are supported |
| 628 |
* :class:`ast.Not` (``not``) is not supported |
| 629 |
|
| 630 |
* :class:`ast.Dict` (except when using ``**`` unpacking) |
| 631 |
* :class:`ast.Set` |
| 632 |
* :class:`ast.Compare` |
| 633 |
|
| 634 |
* :class:`ast.Eq` and :class:`ast.NotEq` are supported |
| 635 |
* :class:`ast.Lt`, :class:`ast.LtE`, :class:`ast.Gt`, and :class:`ast.GtE` are supported, but the operand may be flipped |
| 636 |
* :class:`ast.Is`, :class:`ast.IsNot`, :class:`ast.In`, and :class:`ast.NotIn` are not supported |
| 637 |
|
| 638 |
* :class:`ast.Call` (except when using ``**`` unpacking) |
| 639 |
* :class:`ast.Constant` (though not the exact representation of the constant; for example, escape |
| 640 |
sequences in strings are lost; hexadecimal numbers are converted to decimal) |
| 641 |
* :class:`ast.Attribute` (assuming the value is not a constant) |
| 642 |
* :class:`ast.Subscript` (assuming the value is not a constant) |
| 643 |
* :class:`ast.Starred` (``*`` unpacking) |
| 644 |
* :class:`ast.Name` |
| 645 |
* :class:`ast.List` |
| 646 |
* :class:`ast.Tuple` |
| 647 |
* :class:`ast.Slice` |
| 648 |
|
| 649 |
The following are unsupported, but throw an informative error when encountered by the |
| 650 |
stringifier: |
| 651 |
|
| 652 |
* :class:`ast.FormattedValue` (f-strings; error is not detected if conversion specifiers like ``!r`` |
| 653 |
are used) |
| 654 |
* :class:`ast.JoinedStr` (f-strings) |
| 655 |
|
| 656 |
The following are unsupported and result in incorrect output: |
| 657 |
|
| 658 |
* :class:`ast.BoolOp` (``and`` and ``or``) |
| 659 |
* :class:`ast.IfExp` |
| 660 |
* :class:`ast.Lambda` |
| 661 |
* :class:`ast.ListComp` |
| 662 |
* :class:`ast.SetComp` |
| 663 |
* :class:`ast.DictComp` |
| 664 |
* :class:`ast.GeneratorExp` |
| 665 |
|
| 666 |
The following are disallowed in annotation scopes and therefore not relevant: |
| 667 |
|
| 668 |
* :class:`ast.NamedExpr` (``:=``) |
| 669 |
* :class:`ast.Await` |
| 670 |
* :class:`ast.Yield` |
| 671 |
* :class:`ast.YieldFrom` |
| 672 |
|
| 673 |
|
| 674 |
Limitations of the ``FORWARDREF`` format |
| 675 |
---------------------------------------- |
| 676 |
|
| 677 |
The :attr:`~Format.FORWARDREF` format aims to produce real values as much |
| 678 |
as possible, with anything that cannot be resolved replaced with |
| 679 |
:class:`ForwardRef` objects. It is affected by broadly the same Limitations |
| 680 |
as the :attr:`~Format.STRING` format: annotations that perform operations on |
| 681 |
literals or that use unsupported expression types may raise exceptions when |
| 682 |
evaluated using the :attr:`~Format.FORWARDREF` format. |
| 683 |
|
| 684 |
Below are a few examples of the behavior with unsupported expressions: |
| 685 |
|
| 686 |
.. code-block:: pycon |
| 687 |
|
| 688 |
>>> from annotationlib import get_annotations, Format |
| 689 |
>>> def zerodiv(x: 1 / 0): ... |
| 690 |
>>> get_annotations(zerodiv, format=Format.STRING) |
| 691 |
Traceback (most recent call last): |
| 692 |
... |
| 693 |
ZeroDivisionError: division by zero |
| 694 |
>>> get_annotations(zerodiv, format=Format.FORWARDREF) |
| 695 |
Traceback (most recent call last): |
| 696 |
... |
| 697 |
ZeroDivisionError: division by zero |
| 698 |
>>> def ifexp(x: 1 if y else 0): ... |
| 699 |
>>> get_annotations(ifexp, format=Format.STRING) |
| 700 |
{'x': '1'} |
| 701 |
|
| 702 |
.. _annotationlib-security: |
| 703 |
|
| 704 |
Security implications of introspecting annotations |
| 705 |
-------------------------------------------------- |
| 706 |
|
| 707 |
Much of the functionality in this module involves executing code related to annotations, |
| 708 |
which can then do arbitrary things. For example, |
| 709 |
:func:`get_annotations` may call an arbitrary :term:`annotate function`, and |
| 710 |
:meth:`ForwardRef.evaluate` may call :func:`eval` on an arbitrary string. Code contained |
| 711 |
in an annotation might make arbitrary system calls, enter an infinite loop, or perform any |
| 712 |
other operation. This is also true for any access of the :attr:`~object.__annotations__` attribute, |
| 713 |
and for various functions in the :mod:`typing` module that work with annotations, such as |
| 714 |
:func:`typing.get_type_hints`. |
| 715 |
|
| 716 |
Any security issue arising from this also applies immediately after importing |
| 717 |
code that may contain untrusted annotations: importing code can always cause arbitrary operations |
| 718 |
to be performed. However, it is unsafe to accept strings or other input from an untrusted source and |
| 719 |
pass them to any of the APIs for introspecting annotations, for example by editing an |
| 720 |
``__annotations__`` dictionary or directly creating a :class:`ForwardRef` object. |