| 1 |
:tocdepth: 2 |
| 2 |
|
| 3 |
=============== |
| 4 |
Programming FAQ |
| 5 |
=============== |
| 6 |
|
| 7 |
.. only:: html |
| 8 |
|
| 9 |
.. contents:: |
| 10 |
|
| 11 |
General questions |
| 12 |
================= |
| 13 |
|
| 14 |
Is there a source code-level debugger with breakpoints and single-stepping? |
| 15 |
--------------------------------------------------------------------------- |
| 16 |
|
| 17 |
Yes. |
| 18 |
|
| 19 |
Several debuggers for Python are described below, and the built-in function |
| 20 |
:func:`breakpoint` allows you to drop into any of them. |
| 21 |
|
| 22 |
The pdb module is a simple but adequate console-mode debugger for Python. It is |
| 23 |
part of the standard Python library, and is :mod:`documented in the Library |
| 24 |
Reference Manual <pdb>`. You can also write your own debugger by using the code |
| 25 |
for pdb as an example. |
| 26 |
|
| 27 |
The IDLE interactive development environment, which is part of the standard |
| 28 |
Python distribution (normally available as :mod:`idlelib`), |
| 29 |
includes a graphical debugger. |
| 30 |
|
| 31 |
PythonWin is a Python IDE that includes a GUI debugger based on pdb. The |
| 32 |
PythonWin debugger colors breakpoints and has quite a few cool features such as |
| 33 |
debugging non-PythonWin programs. PythonWin is available as part of |
| 34 |
`pywin32 <https://github.com/mhammond/pywin32>`_ project and |
| 35 |
as a part of the |
| 36 |
`ActivePython <https://www.activestate.com/products/python/>`_ distribution. |
| 37 |
|
| 38 |
`Eric <https://eric-ide.python-projects.org/>`_ is an IDE built on PyQt |
| 39 |
and the Scintilla editing component. |
| 40 |
|
| 41 |
`trepan3k <https://github.com/rocky/python3-trepan/>`_ is a gdb-like debugger. |
| 42 |
|
| 43 |
`Visual Studio Code <https://code.visualstudio.com/>`_ is an IDE with debugging |
| 44 |
tools that integrates with version-control software. |
| 45 |
|
| 46 |
There are a number of commercial Python IDEs that include graphical debuggers. |
| 47 |
They include: |
| 48 |
|
| 49 |
* `Wing IDE <https://wingware.com/>`_ |
| 50 |
* `PyCharm <https://www.jetbrains.com/pycharm/>`_ |
| 51 |
|
| 52 |
|
| 53 |
Are there tools to help find bugs or perform static analysis? |
| 54 |
------------------------------------------------------------- |
| 55 |
|
| 56 |
Yes. |
| 57 |
|
| 58 |
`Ruff <https://docs.astral.sh/ruff/>`__, |
| 59 |
`Pylint <https://pylint.readthedocs.io/>`__ and |
| 60 |
`Pyflakes <https://github.com/PyCQA/pyflakes>`__ do basic checking that will |
| 61 |
help you catch bugs sooner. |
| 62 |
|
| 63 |
Static type checkers such as `mypy <https://mypy-lang.org/>`__, |
| 64 |
`ty <https://docs.astral.sh/ty/>`__, |
| 65 |
`Pyrefly <https://pyrefly.org/>`__, and |
| 66 |
`pytype <https://github.com/google/pytype>`__ can check type hints in Python |
| 67 |
source code. |
| 68 |
|
| 69 |
|
| 70 |
.. _faq-create-standalone-binary: |
| 71 |
|
| 72 |
How can I create a stand-alone binary from a Python script? |
| 73 |
----------------------------------------------------------- |
| 74 |
|
| 75 |
You don't need the ability to compile Python to C code if all you want is a |
| 76 |
stand-alone program that users can download and run without having to install |
| 77 |
the Python distribution first. There are a number of tools that determine the |
| 78 |
set of modules required by a program and bind these modules together with a |
| 79 |
Python binary to produce a single executable. |
| 80 |
|
| 81 |
One is to use the freeze tool, which is included in the Python source tree as |
| 82 |
:source:`Tools/freeze`. |
| 83 |
It converts Python byte code to C arrays; with a C compiler you can |
| 84 |
embed all your modules into a new program, which is then linked with the |
| 85 |
standard Python modules. |
| 86 |
|
| 87 |
It works by scanning your source recursively for import statements (in both |
| 88 |
forms) and looking for the modules in the standard Python path as well as in the |
| 89 |
source directory (for built-in modules). It then turns the bytecode for modules |
| 90 |
written in Python into C code (array initializers that can be turned into code |
| 91 |
objects using the marshal module) and creates a custom-made config file that |
| 92 |
only contains those built-in modules which are actually used in the program. It |
| 93 |
then compiles the generated C code and links it with the rest of the Python |
| 94 |
interpreter to form a self-contained binary which acts exactly like your script. |
| 95 |
|
| 96 |
The following packages can help with the creation of console and GUI |
| 97 |
executables: |
| 98 |
|
| 99 |
* `Nuitka <https://nuitka.net/>`_ (Cross-platform) |
| 100 |
* `PyInstaller <https://pyinstaller.org/>`_ (Cross-platform) |
| 101 |
* `PyOxidizer <https://pyoxidizer.readthedocs.io/en/stable/>`_ (Cross-platform) |
| 102 |
* `cx_Freeze <https://marcelotduarte.github.io/cx_Freeze/>`_ (Cross-platform) |
| 103 |
* `py2app <https://github.com/ronaldoussoren/py2app>`_ (macOS only) |
| 104 |
* `py2exe <https://www.py2exe.org/>`_ (Windows only) |
| 105 |
|
| 106 |
|
| 107 |
Are there coding standards or a style guide for Python programs? |
| 108 |
---------------------------------------------------------------- |
| 109 |
|
| 110 |
Yes. The coding style required for standard library modules is documented as |
| 111 |
:pep:`8`. |
| 112 |
|
| 113 |
|
| 114 |
Core language |
| 115 |
============= |
| 116 |
|
| 117 |
.. _faq-unboundlocalerror: |
| 118 |
|
| 119 |
Why am I getting an UnboundLocalError when the variable has a value? |
| 120 |
-------------------------------------------------------------------- |
| 121 |
|
| 122 |
It can be a surprise to get the :exc:`UnboundLocalError` in previously working |
| 123 |
code when it is modified by adding an assignment statement somewhere in |
| 124 |
the body of a function. |
| 125 |
|
| 126 |
This code: |
| 127 |
|
| 128 |
>>> x = 10 |
| 129 |
>>> def bar(): |
| 130 |
... print(x) |
| 131 |
... |
| 132 |
>>> bar() |
| 133 |
10 |
| 134 |
|
| 135 |
works, but this code: |
| 136 |
|
| 137 |
>>> x = 10 |
| 138 |
>>> def foo(): |
| 139 |
... print(x) |
| 140 |
... x += 1 |
| 141 |
|
| 142 |
results in an :exc:`!UnboundLocalError`: |
| 143 |
|
| 144 |
>>> foo() |
| 145 |
Traceback (most recent call last): |
| 146 |
... |
| 147 |
UnboundLocalError: cannot access local variable 'x' where it is not associated with a value |
| 148 |
|
| 149 |
This is because when you make an assignment to a variable in a scope, that |
| 150 |
variable becomes local to that scope and shadows any similarly named variable |
| 151 |
in the outer scope. Since the last statement in foo assigns a new value to |
| 152 |
``x``, the compiler recognizes it as a local variable. Consequently when the |
| 153 |
earlier ``print(x)`` attempts to print the uninitialized local variable and |
| 154 |
an error results. |
| 155 |
|
| 156 |
In the example above you can access the outer scope variable by declaring it |
| 157 |
global: |
| 158 |
|
| 159 |
>>> x = 10 |
| 160 |
>>> def foobar(): |
| 161 |
... global x |
| 162 |
... print(x) |
| 163 |
... x += 1 |
| 164 |
... |
| 165 |
>>> foobar() |
| 166 |
10 |
| 167 |
|
| 168 |
This explicit declaration is required in order to remind you that (unlike the |
| 169 |
superficially analogous situation with class and instance variables) you are |
| 170 |
actually modifying the value of the variable in the outer scope: |
| 171 |
|
| 172 |
>>> print(x) |
| 173 |
11 |
| 174 |
|
| 175 |
You can do a similar thing in a nested scope using the :keyword:`nonlocal` |
| 176 |
keyword: |
| 177 |
|
| 178 |
>>> def foo(): |
| 179 |
... x = 10 |
| 180 |
... def bar(): |
| 181 |
... nonlocal x |
| 182 |
... print(x) |
| 183 |
... x += 1 |
| 184 |
... bar() |
| 185 |
... print(x) |
| 186 |
... |
| 187 |
>>> foo() |
| 188 |
10 |
| 189 |
11 |
| 190 |
|
| 191 |
|
| 192 |
What are the rules for local and global variables in Python? |
| 193 |
------------------------------------------------------------ |
| 194 |
|
| 195 |
In Python, variables that are only referenced inside a function are implicitly |
| 196 |
global. If a variable is assigned a value anywhere within the function's body, |
| 197 |
it's assumed to be a local unless explicitly declared as global. |
| 198 |
|
| 199 |
Though a bit surprising at first, a moment's consideration explains this. On |
| 200 |
one hand, requiring :keyword:`global` for assigned variables provides a bar |
| 201 |
against unintended side-effects. On the other hand, if ``global`` was required |
| 202 |
for all global references, you'd be using ``global`` all the time. You'd have |
| 203 |
to declare as global every reference to a built-in function or to a component of |
| 204 |
an imported module. This clutter would defeat the usefulness of the ``global`` |
| 205 |
declaration for identifying side-effects. |
| 206 |
|
| 207 |
|
| 208 |
Why do lambdas defined in a loop with different values all return the same result? |
| 209 |
---------------------------------------------------------------------------------- |
| 210 |
|
| 211 |
Assume you use a for loop to define a few different lambdas (or even plain |
| 212 |
functions), for example:: |
| 213 |
|
| 214 |
>>> squares = [] |
| 215 |
>>> for x in range(5): |
| 216 |
... squares.append(lambda: x**2) |
| 217 |
|
| 218 |
This gives you a list that contains 5 lambdas that calculate ``x**2``. You |
| 219 |
might expect that, when called, they would return, respectively, ``0``, ``1``, |
| 220 |
``4``, ``9``, and ``16``. However, when you actually try you will see that |
| 221 |
they all return ``16``:: |
| 222 |
|
| 223 |
>>> squares[2]() |
| 224 |
16 |
| 225 |
>>> squares[4]() |
| 226 |
16 |
| 227 |
|
| 228 |
This happens because ``x`` is not local to the lambdas, but is defined in |
| 229 |
the outer scope, and it is accessed when the lambda is called --- not when it |
| 230 |
is defined. At the end of the loop, the value of ``x`` is ``4``, so all the |
| 231 |
functions now return ``4**2``, that is ``16``. You can also verify this by |
| 232 |
changing the value of ``x`` and see how the results of the lambdas change:: |
| 233 |
|
| 234 |
>>> x = 8 |
| 235 |
>>> squares[2]() |
| 236 |
64 |
| 237 |
|
| 238 |
In order to avoid this, you need to save the values in variables local to the |
| 239 |
lambdas, so that they don't rely on the value of the global ``x``:: |
| 240 |
|
| 241 |
>>> squares = [] |
| 242 |
>>> for x in range(5): |
| 243 |
... squares.append(lambda n=x: n**2) |
| 244 |
|
| 245 |
Here, ``n=x`` creates a new variable ``n`` local to the lambda and computed |
| 246 |
when the lambda is defined so that it has the same value that ``x`` had at |
| 247 |
that point in the loop. This means that the value of ``n`` will be ``0`` |
| 248 |
in the first lambda, ``1`` in the second, ``2`` in the third, and so on. |
| 249 |
Therefore each lambda will now return the correct result:: |
| 250 |
|
| 251 |
>>> squares[2]() |
| 252 |
4 |
| 253 |
>>> squares[4]() |
| 254 |
16 |
| 255 |
|
| 256 |
Note that this behaviour is not peculiar to lambdas, but applies to regular |
| 257 |
functions too. |
| 258 |
|
| 259 |
|
| 260 |
How do I share global variables across modules? |
| 261 |
------------------------------------------------ |
| 262 |
|
| 263 |
The canonical way to share information across modules within a single program is |
| 264 |
to create a special module (often called config or cfg). Just import the config |
| 265 |
module in all modules of your application; the module then becomes available as |
| 266 |
a global name. Because there is only one instance of each module, any changes |
| 267 |
made to the module object get reflected everywhere. For example: |
| 268 |
|
| 269 |
config.py:: |
| 270 |
|
| 271 |
x = 0 # Default value of the 'x' configuration setting |
| 272 |
|
| 273 |
mod.py:: |
| 274 |
|
| 275 |
import config |
| 276 |
config.x = 1 |
| 277 |
|
| 278 |
main.py:: |
| 279 |
|
| 280 |
import config |
| 281 |
import mod |
| 282 |
print(config.x) |
| 283 |
|
| 284 |
Note that using a module is also the basis for implementing the singleton design |
| 285 |
pattern, for the same reason. |
| 286 |
|
| 287 |
|
| 288 |
What are the "best practices" for using import in a module? |
| 289 |
----------------------------------------------------------- |
| 290 |
|
| 291 |
In general, don't use ``from modulename import *``. Doing so clutters the |
| 292 |
importer's namespace, and makes it much harder for linters to detect undefined |
| 293 |
names. |
| 294 |
|
| 295 |
Import modules at the top of a file. Doing so makes it clear what other modules |
| 296 |
your code requires and avoids questions of whether the module name is in scope. |
| 297 |
Using one import per line makes it easy to add and delete module imports, but |
| 298 |
using multiple imports per line uses less screen space. |
| 299 |
|
| 300 |
It's good practice if you import modules in the following order: |
| 301 |
|
| 302 |
1. standard library modules -- such as :mod:`sys`, :mod:`os`, :mod:`argparse`, :mod:`re` |
| 303 |
2. third-party library modules (anything installed in Python's site-packages |
| 304 |
directory) -- such as :pypi:`dateutil`, :pypi:`requests`, :pypi:`tzdata` |
| 305 |
3. locally developed modules |
| 306 |
|
| 307 |
It is sometimes necessary to move imports to a function or class to avoid |
| 308 |
problems with circular imports. Gordon McMillan says: |
| 309 |
|
| 310 |
Circular imports are fine where both modules use the "import <module>" form |
| 311 |
of import. They fail when the 2nd module wants to grab a name out of the |
| 312 |
first ("from module import name") and the import is at the top level. That's |
| 313 |
because names in the 1st are not yet available, because the first module is |
| 314 |
busy importing the 2nd. |
| 315 |
|
| 316 |
In this case, if the second module is only used in one function, then the import |
| 317 |
can easily be moved into that function. By the time the import is called, the |
| 318 |
first module will have finished initializing, and the second module can do its |
| 319 |
import. |
| 320 |
|
| 321 |
It may also be necessary to move imports out of the top level of code if some of |
| 322 |
the modules are platform-specific. In that case, it may not even be possible to |
| 323 |
import all of the modules at the top of the file. In this case, importing the |
| 324 |
correct modules in the corresponding platform-specific code is a good option. |
| 325 |
|
| 326 |
Only move imports into a local scope, such as inside a function definition, if |
| 327 |
it's necessary to solve a problem such as avoiding a circular import or are |
| 328 |
trying to reduce the initialization time of a module. This technique is |
| 329 |
especially helpful if many of the imports are unnecessary depending on how the |
| 330 |
program executes. You may also want to move imports into a function if the |
| 331 |
modules are only ever used in that function. Note that loading a module the |
| 332 |
first time may be expensive because of the one time initialization of the |
| 333 |
module, but loading a module multiple times is virtually free, costing only a |
| 334 |
couple of dictionary lookups. Even if the module name has gone out of scope, |
| 335 |
the module is probably available in :data:`sys.modules`. |
| 336 |
|
| 337 |
|
| 338 |
Why are default values shared between objects? |
| 339 |
---------------------------------------------- |
| 340 |
|
| 341 |
This type of bug commonly bites neophyte programmers. Consider this function:: |
| 342 |
|
| 343 |
def foo(mydict={}): # Danger: shared reference to one dict for all calls |
| 344 |
... compute something ... |
| 345 |
mydict[key] = value |
| 346 |
return mydict |
| 347 |
|
| 348 |
The first time you call this function, ``mydict`` contains a single item. The |
| 349 |
second time, ``mydict`` contains two items because when ``foo()`` begins |
| 350 |
executing, ``mydict`` starts out with an item already in it. |
| 351 |
|
| 352 |
It is often expected that a function call creates new objects for default |
| 353 |
values. This is not what happens. Default values are created exactly once, when |
| 354 |
the function is defined. If that object is changed, like the dictionary in this |
| 355 |
example, subsequent calls to the function will refer to this changed object. |
| 356 |
|
| 357 |
By definition, immutable objects such as numbers, strings, tuples, and ``None``, |
| 358 |
are safe from change. Changes to mutable objects such as dictionaries, lists, |
| 359 |
and class instances can lead to confusion. |
| 360 |
|
| 361 |
Because of this feature, it is good programming practice to not use mutable |
| 362 |
objects as default values. Instead, use ``None`` as the default value and |
| 363 |
inside the function, check if the parameter is ``None`` and create a new |
| 364 |
list/dictionary/whatever if it is. For example, don't write:: |
| 365 |
|
| 366 |
def foo(mydict={}): |
| 367 |
... |
| 368 |
|
| 369 |
but:: |
| 370 |
|
| 371 |
def foo(mydict=None): |
| 372 |
if mydict is None: |
| 373 |
mydict = {} # create a new dict for local namespace |
| 374 |
|
| 375 |
This feature can be useful. When you have a function that's time-consuming to |
| 376 |
compute, a common technique is to cache the parameters and the resulting value |
| 377 |
of each call to the function, and return the cached value if the same value is |
| 378 |
requested again. This is called "memoizing", and can be implemented like this:: |
| 379 |
|
| 380 |
# Callers can only provide two parameters and optionally pass _cache by keyword |
| 381 |
def expensive(arg1, arg2, *, _cache={}): |
| 382 |
if (arg1, arg2) in _cache: |
| 383 |
return _cache[(arg1, arg2)] |
| 384 |
|
| 385 |
# Calculate the value |
| 386 |
result = ... expensive computation ... |
| 387 |
_cache[(arg1, arg2)] = result # Store result in the cache |
| 388 |
return result |
| 389 |
|
| 390 |
You could use a global variable containing a dictionary instead of the default |
| 391 |
value; it's a matter of taste. |
| 392 |
|
| 393 |
|
| 394 |
How can I pass optional or keyword parameters from one function to another? |
| 395 |
--------------------------------------------------------------------------- |
| 396 |
|
| 397 |
Collect the arguments using the ``*`` and ``**`` specifiers in the function's |
| 398 |
parameter list; this gives you the positional arguments as a tuple and the |
| 399 |
keyword arguments as a dictionary. You can then pass these arguments when |
| 400 |
calling another function by using ``*`` and ``**``:: |
| 401 |
|
| 402 |
def f(x, *args, **kwargs): |
| 403 |
... |
| 404 |
kwargs['width'] = '14.3c' |
| 405 |
... |
| 406 |
g(x, *args, **kwargs) |
| 407 |
|
| 408 |
|
| 409 |
.. index:: |
| 410 |
single: argument; difference from parameter |
| 411 |
single: parameter; difference from argument |
| 412 |
|
| 413 |
.. _faq-argument-vs-parameter: |
| 414 |
|
| 415 |
What is the difference between arguments and parameters? |
| 416 |
-------------------------------------------------------- |
| 417 |
|
| 418 |
:term:`Parameters <parameter>` are defined by the names that appear in a |
| 419 |
function definition, whereas :term:`arguments <argument>` are the values |
| 420 |
actually passed to a function when calling it. Parameters define what |
| 421 |
:term:`kind of arguments <parameter>` a function can accept. For |
| 422 |
example, given the function definition:: |
| 423 |
|
| 424 |
def func(foo, bar=None, **kwargs): |
| 425 |
pass |
| 426 |
|
| 427 |
*foo*, *bar* and *kwargs* are parameters of ``func``. However, when calling |
| 428 |
``func``, for example:: |
| 429 |
|
| 430 |
func(42, bar=314, extra=somevar) |
| 431 |
|
| 432 |
the values ``42``, ``314``, and ``somevar`` are arguments. |
| 433 |
|
| 434 |
|
| 435 |
Why did changing list 'y' also change list 'x'? |
| 436 |
------------------------------------------------ |
| 437 |
|
| 438 |
If you wrote code like:: |
| 439 |
|
| 440 |
>>> x = [] |
| 441 |
>>> y = x |
| 442 |
>>> y.append(10) |
| 443 |
>>> y |
| 444 |
[10] |
| 445 |
>>> x |
| 446 |
[10] |
| 447 |
|
| 448 |
you might be wondering why appending an element to ``y`` changed ``x`` too. |
| 449 |
|
| 450 |
There are two factors that produce this result: |
| 451 |
|
| 452 |
1) Variables are simply names that refer to objects. Doing ``y = x`` doesn't |
| 453 |
create a copy of the list -- it creates a new variable ``y`` that refers to |
| 454 |
the same object ``x`` refers to. This means that there is only one object |
| 455 |
(the list), and both ``x`` and ``y`` refer to it. |
| 456 |
2) Lists are :term:`mutable`, which means that you can change their content. |
| 457 |
|
| 458 |
After the call to :meth:`~sequence.append`, the content of the mutable object has |
| 459 |
changed from ``[]`` to ``[10]``. Since both the variables refer to the same |
| 460 |
object, using either name accesses the modified value ``[10]``. |
| 461 |
|
| 462 |
If we instead assign an immutable object to ``x``:: |
| 463 |
|
| 464 |
>>> x = 5 # ints are immutable |
| 465 |
>>> y = x |
| 466 |
>>> x = x + 1 # 5 can't be mutated, we are creating a new object here |
| 467 |
>>> x |
| 468 |
6 |
| 469 |
>>> y |
| 470 |
5 |
| 471 |
|
| 472 |
we can see that in this case ``x`` and ``y`` are not equal anymore. This is |
| 473 |
because integers are :term:`immutable`, and when we do ``x = x + 1`` we are not |
| 474 |
mutating the int ``5`` by incrementing its value; instead, we are creating a |
| 475 |
new object (the int ``6``) and assigning it to ``x`` (that is, changing which |
| 476 |
object ``x`` refers to). After this assignment we have two objects (the ints |
| 477 |
``6`` and ``5``) and two variables that refer to them (``x`` now refers to |
| 478 |
``6`` but ``y`` still refers to ``5``). |
| 479 |
|
| 480 |
Some operations (for example ``y.append(10)`` and ``y.sort()``) mutate the |
| 481 |
object, whereas superficially similar operations (for example ``y = y + [10]`` |
| 482 |
and :func:`sorted(y) <sorted>`) create a new object. In general in Python (and in all cases |
| 483 |
in the standard library) a method that mutates an object will return ``None`` |
| 484 |
to help avoid getting the two types of operations confused. So if you |
| 485 |
mistakenly write ``y.sort()`` thinking it will give you a sorted copy of ``y``, |
| 486 |
you'll instead end up with ``None``, which will likely cause your program to |
| 487 |
generate an easily diagnosed error. |
| 488 |
|
| 489 |
However, there is one class of operations where the same operation sometimes |
| 490 |
has different behaviors with different types: the augmented assignment |
| 491 |
operators. For example, ``+=`` mutates lists but not tuples or ints (``a_list |
| 492 |
+= [1, 2, 3]`` is equivalent to ``a_list.extend([1, 2, 3])`` and mutates |
| 493 |
``a_list``, whereas ``some_tuple += (1, 2, 3)`` and ``some_int += 1`` create |
| 494 |
new objects). |
| 495 |
|
| 496 |
In other words: |
| 497 |
|
| 498 |
* If we have a mutable object (such as :class:`list`, :class:`dict`, :class:`set`), |
| 499 |
we can use some specific operations to mutate it and all the variables |
| 500 |
that refer to it will see the change. |
| 501 |
* If we have an immutable object (such as :class:`str`, :class:`int`, :class:`tuple`), |
| 502 |
all the variables that refer to it will always see the same value, |
| 503 |
but operations that transform that value into a new value always return a new |
| 504 |
object. |
| 505 |
|
| 506 |
If you want to know if two variables refer to the same object or not, you can |
| 507 |
use the :keyword:`is` operator, or the built-in function :func:`id`. |
| 508 |
|
| 509 |
|
| 510 |
How do I write a function with output parameters (call by reference)? |
| 511 |
--------------------------------------------------------------------- |
| 512 |
|
| 513 |
Remember that arguments are passed by assignment in Python. Since assignment |
| 514 |
just creates references to objects, there's no alias between an argument name in |
| 515 |
the caller and callee, and consequently no call-by-reference. You can achieve the |
| 516 |
desired effect in a number of ways. |
| 517 |
|
| 518 |
1) By returning a tuple of the results:: |
| 519 |
|
| 520 |
>>> def func1(a, b): |
| 521 |
... a = 'new-value' # a and b are local names |
| 522 |
... b = b + 1 # assigned to new objects |
| 523 |
... return a, b # return new values |
| 524 |
... |
| 525 |
>>> x, y = 'old-value', 99 |
| 526 |
>>> func1(x, y) |
| 527 |
('new-value', 100) |
| 528 |
|
| 529 |
This is almost always the clearest solution. |
| 530 |
|
| 531 |
2) By using global variables. This isn't thread-safe, and is not recommended. |
| 532 |
|
| 533 |
3) By passing a mutable (changeable in-place) object:: |
| 534 |
|
| 535 |
>>> def func2(a): |
| 536 |
... a[0] = 'new-value' # 'a' references a mutable list |
| 537 |
... a[1] = a[1] + 1 # changes a shared object |
| 538 |
... |
| 539 |
>>> args = ['old-value', 99] |
| 540 |
>>> func2(args) |
| 541 |
>>> args |
| 542 |
['new-value', 100] |
| 543 |
|
| 544 |
4) By passing in a dictionary that gets mutated:: |
| 545 |
|
| 546 |
>>> def func3(args): |
| 547 |
... args['a'] = 'new-value' # args is a mutable dictionary |
| 548 |
... args['b'] = args['b'] + 1 # change it in-place |
| 549 |
... |
| 550 |
>>> args = {'a': 'old-value', 'b': 99} |
| 551 |
>>> func3(args) |
| 552 |
>>> args |
| 553 |
{'a': 'new-value', 'b': 100} |
| 554 |
|
| 555 |
5) Or bundle up values in a class instance:: |
| 556 |
|
| 557 |
>>> class Namespace: |
| 558 |
... def __init__(self, /, **args): |
| 559 |
... for key, value in args.items(): |
| 560 |
... setattr(self, key, value) |
| 561 |
... |
| 562 |
>>> def func4(args): |
| 563 |
... args.a = 'new-value' # args is a mutable Namespace |
| 564 |
... args.b = args.b + 1 # change object in-place |
| 565 |
... |
| 566 |
>>> args = Namespace(a='old-value', b=99) |
| 567 |
>>> func4(args) |
| 568 |
>>> vars(args) |
| 569 |
{'a': 'new-value', 'b': 100} |
| 570 |
|
| 571 |
|
| 572 |
There's almost never a good reason to get this complicated. |
| 573 |
|
| 574 |
Your best choice is to return a tuple containing the multiple results. |
| 575 |
|
| 576 |
|
| 577 |
How do you make a higher order function in Python? |
| 578 |
-------------------------------------------------- |
| 579 |
|
| 580 |
You have two choices: you can use nested scopes or you can use callable objects. |
| 581 |
For example, suppose you wanted to define ``linear(a,b)`` which returns a |
| 582 |
function ``f(x)`` that computes the value ``a*x+b``. Using nested scopes:: |
| 583 |
|
| 584 |
def linear(a, b): |
| 585 |
def result(x): |
| 586 |
return a * x + b |
| 587 |
return result |
| 588 |
|
| 589 |
Or using a callable object:: |
| 590 |
|
| 591 |
class linear: |
| 592 |
|
| 593 |
def __init__(self, a, b): |
| 594 |
self.a, self.b = a, b |
| 595 |
|
| 596 |
def __call__(self, x): |
| 597 |
return self.a * x + self.b |
| 598 |
|
| 599 |
In both cases, :: |
| 600 |
|
| 601 |
taxes = linear(0.3, 2) |
| 602 |
|
| 603 |
gives a callable object where ``taxes(10e6) == 0.3 * 10e6 + 2``. |
| 604 |
|
| 605 |
The callable object approach has the disadvantage that it is a bit slower and |
| 606 |
results in slightly longer code. However, note that a collection of callables |
| 607 |
can share their signature via inheritance:: |
| 608 |
|
| 609 |
class exponential(linear): |
| 610 |
# __init__ inherited |
| 611 |
def __call__(self, x): |
| 612 |
return self.a * (x ** self.b) |
| 613 |
|
| 614 |
Object can encapsulate state for several methods:: |
| 615 |
|
| 616 |
class counter: |
| 617 |
|
| 618 |
value = 0 |
| 619 |
|
| 620 |
def set(self, x): |
| 621 |
self.value = x |
| 622 |
|
| 623 |
def up(self): |
| 624 |
self.value = self.value + 1 |
| 625 |
|
| 626 |
def down(self): |
| 627 |
self.value = self.value - 1 |
| 628 |
|
| 629 |
count = counter() |
| 630 |
inc, dec, reset = count.up, count.down, count.set |
| 631 |
|
| 632 |
Here ``inc()``, ``dec()`` and ``reset()`` act like functions which share the |
| 633 |
same counting variable. |
| 634 |
|
| 635 |
|
| 636 |
How do I copy an object in Python? |
| 637 |
---------------------------------- |
| 638 |
|
| 639 |
In general, try :func:`copy.copy` or :func:`copy.deepcopy` for the general case. |
| 640 |
Not all objects can be copied, but most can. |
| 641 |
|
| 642 |
Some objects can be copied more easily. Dictionaries have a :meth:`~dict.copy` |
| 643 |
method:: |
| 644 |
|
| 645 |
newdict = olddict.copy() |
| 646 |
|
| 647 |
Sequences can be copied by slicing:: |
| 648 |
|
| 649 |
new_l = l[:] |
| 650 |
|
| 651 |
|
| 652 |
How can I find the methods or attributes of an object? |
| 653 |
------------------------------------------------------ |
| 654 |
|
| 655 |
For an instance ``x`` of a user-defined class, :func:`dir(x) <dir>` returns an alphabetized |
| 656 |
list of the names containing the instance attributes and methods and attributes |
| 657 |
defined by its class. |
| 658 |
|
| 659 |
|
| 660 |
How can my code discover the name of an object? |
| 661 |
----------------------------------------------- |
| 662 |
|
| 663 |
Generally speaking, it can't, because objects don't really have names. |
| 664 |
Essentially, assignment always binds a name to a value; the same is true of |
| 665 |
``def`` and ``class`` statements, but in that case the value is a |
| 666 |
callable. Consider the following code:: |
| 667 |
|
| 668 |
>>> class A: |
| 669 |
... pass |
| 670 |
... |
| 671 |
>>> B = A |
| 672 |
>>> a = B() |
| 673 |
>>> b = a |
| 674 |
>>> print(b) |
| 675 |
<__main__.A object at 0x16D07CC> |
| 676 |
>>> print(a) |
| 677 |
<__main__.A object at 0x16D07CC> |
| 678 |
|
| 679 |
Arguably the class has a name: even though it is bound to two names and invoked |
| 680 |
through the name ``B`` the created instance is still reported as an instance of |
| 681 |
class ``A``. However, it is impossible to say whether the instance's name is ``a`` or |
| 682 |
``b``, since both names are bound to the same value. |
| 683 |
|
| 684 |
Generally speaking it should not be necessary for your code to "know the names" |
| 685 |
of particular values. Unless you are deliberately writing introspective |
| 686 |
programs, this is usually an indication that a change of approach might be |
| 687 |
beneficial. |
| 688 |
|
| 689 |
In comp.lang.python, Fredrik Lundh once gave an excellent analogy in answer to |
| 690 |
this question: |
| 691 |
|
| 692 |
The same way as you get the name of that cat you found on your porch: the cat |
| 693 |
(object) itself cannot tell you its name, and it doesn't really care -- so |
| 694 |
the only way to find out what it's called is to ask all your neighbours |
| 695 |
(namespaces) if it's their cat (object)... |
| 696 |
|
| 697 |
....and don't be surprised if you'll find that it's known by many names, or |
| 698 |
no name at all! |
| 699 |
|
| 700 |
|
| 701 |
What's up with the comma operator's precedence? |
| 702 |
----------------------------------------------- |
| 703 |
|
| 704 |
Comma is not an operator in Python. Consider this session:: |
| 705 |
|
| 706 |
>>> "a" in "b", "a" |
| 707 |
(False, 'a') |
| 708 |
|
| 709 |
Since the comma is not an operator, but a separator between expressions the |
| 710 |
above is evaluated as if you had entered:: |
| 711 |
|
| 712 |
("a" in "b"), "a" |
| 713 |
|
| 714 |
not:: |
| 715 |
|
| 716 |
"a" in ("b", "a") |
| 717 |
|
| 718 |
The same is true of the various assignment operators (``=``, ``+=``, and so on). |
| 719 |
They are not truly operators but syntactic delimiters in assignment statements. |
| 720 |
|
| 721 |
|
| 722 |
Is there an equivalent of C's "?:" ternary operator? |
| 723 |
---------------------------------------------------- |
| 724 |
|
| 725 |
Yes, there is. The syntax is as follows:: |
| 726 |
|
| 727 |
[on_true] if [expression] else [on_false] |
| 728 |
|
| 729 |
x, y = 50, 25 |
| 730 |
small = x if x < y else y |
| 731 |
|
| 732 |
Before this syntax was introduced in Python 2.5, a common idiom was to use |
| 733 |
logical operators:: |
| 734 |
|
| 735 |
[expression] and [on_true] or [on_false] |
| 736 |
|
| 737 |
However, this idiom is unsafe, as it can give wrong results when *on_true* |
| 738 |
has a false boolean value. Therefore, it is always better to use |
| 739 |
the ``... if ... else ...`` form. |
| 740 |
|
| 741 |
|
| 742 |
Is it possible to write obfuscated one-liners in Python? |
| 743 |
-------------------------------------------------------- |
| 744 |
|
| 745 |
Yes. Usually this is done by nesting :keyword:`lambda` within |
| 746 |
:keyword:`!lambda`. See the following three examples, slightly adapted from Ulf Bartelt:: |
| 747 |
|
| 748 |
from functools import reduce |
| 749 |
|
| 750 |
# Primes < 1000 |
| 751 |
print(list(filter(None,map(lambda y:y*reduce(lambda x,y:x*y!=0, |
| 752 |
map(lambda x,y=y:y%x,range(2,int(pow(y,0.5)+1))),1),range(2,1000))))) |
| 753 |
|
| 754 |
# First 10 Fibonacci numbers |
| 755 |
print(list(map(lambda x,f=lambda x,f:(f(x-1,f)+f(x-2,f)) if x>1 else 1: |
| 756 |
f(x,f), range(10)))) |
| 757 |
|
| 758 |
# Mandelbrot set |
| 759 |
print((lambda Ru,Ro,Iu,Io,IM,Sx,Sy:reduce(lambda x,y:x+'\n'+y,map(lambda y, |
| 760 |
Iu=Iu,Io=Io,Ru=Ru,Ro=Ro,Sy=Sy,L=lambda yc,Iu=Iu,Io=Io,Ru=Ru,Ro=Ro,i=IM, |
| 761 |
Sx=Sx,Sy=Sy:reduce(lambda x,y:x+y,map(lambda x,xc=Ru,yc=yc,Ru=Ru,Ro=Ro, |
| 762 |
i=i,Sx=Sx,F=lambda xc,yc,x,y,k,f=lambda xc,yc,x,y,k,f:(k<=0)or (x*x+y*y |
| 763 |
>=4.0) or 1+f(xc,yc,x*x-y*y+xc,2.0*x*y+yc,k-1,f):f(xc,yc,x,y,k,f):chr( |
| 764 |
64+F(Ru+x*(Ro-Ru)/Sx,yc,0,0,i)),range(Sx))):L(Iu+y*(Io-Iu)/Sy),range(Sy |
| 765 |
))))(-2.1, 0.7, -1.2, 1.2, 30, 80, 24)) |
| 766 |
# \___ ___/ \___ ___/ | | |__ lines on screen |
| 767 |
# V V | |______ columns on screen |
| 768 |
# | | |__________ maximum of "iterations" |
| 769 |
# | |_________________ range on y axis |
| 770 |
# |____________________________ range on x axis |
| 771 |
|
| 772 |
Don't try this at home, kids! |
| 773 |
|
| 774 |
|
| 775 |
.. _faq-positional-only-arguments: |
| 776 |
|
| 777 |
What does the slash(/) in the parameter list of a function mean? |
| 778 |
---------------------------------------------------------------- |
| 779 |
|
| 780 |
A slash in the argument list of a function denotes that the parameters prior to |
| 781 |
it are positional-only. Positional-only parameters are the ones without an |
| 782 |
externally usable name. Upon calling a function that accepts positional-only |
| 783 |
parameters, arguments are mapped to parameters based solely on their position. |
| 784 |
For example, :func:`divmod` is a function that accepts positional-only |
| 785 |
parameters. Its documentation looks like this:: |
| 786 |
|
| 787 |
>>> help(divmod) |
| 788 |
Help on built-in function divmod in module builtins: |
| 789 |
|
| 790 |
divmod(x, y, /) |
| 791 |
Return the tuple (x//y, x%y). Invariant: div*y + mod == x. |
| 792 |
|
| 793 |
The slash at the end of the parameter list means that both parameters are |
| 794 |
positional-only. Thus, calling :func:`divmod` with keyword arguments would lead |
| 795 |
to an error:: |
| 796 |
|
| 797 |
>>> divmod(x=3, y=4) |
| 798 |
Traceback (most recent call last): |
| 799 |
File "<stdin>", line 1, in <module> |
| 800 |
TypeError: divmod() takes no keyword arguments |
| 801 |
|
| 802 |
|
| 803 |
Numbers and strings |
| 804 |
=================== |
| 805 |
|
| 806 |
How do I specify hexadecimal and octal integers? |
| 807 |
------------------------------------------------ |
| 808 |
|
| 809 |
To specify an octal digit, precede the octal value with a zero, and then a lower |
| 810 |
or uppercase "o". For example, to set the variable "a" to the octal value "10" |
| 811 |
(8 in decimal), type:: |
| 812 |
|
| 813 |
>>> a = 0o10 |
| 814 |
>>> a |
| 815 |
8 |
| 816 |
|
| 817 |
Hexadecimal is just as easy. Simply precede the hexadecimal number with a zero, |
| 818 |
and then a lower or uppercase "x". Hexadecimal digits can be specified in lower |
| 819 |
or uppercase. For example, in the Python interpreter:: |
| 820 |
|
| 821 |
>>> a = 0xa5 |
| 822 |
>>> a |
| 823 |
165 |
| 824 |
>>> b = 0XB2 |
| 825 |
>>> b |
| 826 |
178 |
| 827 |
|
| 828 |
|
| 829 |
Why does -22 // 10 return -3? |
| 830 |
----------------------------- |
| 831 |
|
| 832 |
It's primarily driven by the desire that ``i % j`` have the same sign as ``j``. |
| 833 |
If you want that, and also want:: |
| 834 |
|
| 835 |
i == (i // j) * j + (i % j) |
| 836 |
|
| 837 |
then integer division has to return the floor. C also requires that identity to |
| 838 |
hold, and then compilers that truncate ``i // j`` need to make ``i % j`` have |
| 839 |
the same sign as ``i``. |
| 840 |
|
| 841 |
There are few real use cases for ``i % j`` when ``j`` is negative. When ``j`` |
| 842 |
is positive, there are many, and in virtually all of them it's more useful for |
| 843 |
``i % j`` to be ``>= 0``. If the clock says 10 now, what did it say 200 hours |
| 844 |
ago? ``-190 % 12 == 2`` is useful; ``-190 % 12 == -10`` is a bug waiting to |
| 845 |
bite. |
| 846 |
|
| 847 |
|
| 848 |
How do I get int literal attribute instead of SyntaxError? |
| 849 |
---------------------------------------------------------- |
| 850 |
|
| 851 |
Trying to lookup an ``int`` literal attribute in the normal manner gives |
| 852 |
a :exc:`SyntaxError` because the period is seen as a decimal point:: |
| 853 |
|
| 854 |
>>> 1.__class__ |
| 855 |
File "<stdin>", line 1 |
| 856 |
1.__class__ |
| 857 |
^ |
| 858 |
SyntaxError: invalid decimal literal |
| 859 |
|
| 860 |
The solution is to separate the literal from the period |
| 861 |
with either a space or parentheses. |
| 862 |
|
| 863 |
>>> 1 .__class__ |
| 864 |
<class 'int'> |
| 865 |
>>> (1).__class__ |
| 866 |
<class 'int'> |
| 867 |
|
| 868 |
|
| 869 |
How do I convert a string to a number? |
| 870 |
-------------------------------------- |
| 871 |
|
| 872 |
For integers, use the built-in :func:`int` type constructor, for example, ``int('144') |
| 873 |
== 144``. Similarly, :func:`float` converts to a floating-point number, |
| 874 |
for example, ``float('144') == 144.0``. |
| 875 |
|
| 876 |
By default, these interpret the number as decimal, so that ``int('0144') == |
| 877 |
144`` holds true, and ``int('0x144')`` raises :exc:`ValueError`. ``int(string, |
| 878 |
base)`` takes the base to convert from as a second optional argument, so ``int( |
| 879 |
'0x144', 16) == 324``. If the base is specified as 0, the number is interpreted |
| 880 |
using Python's rules: a leading '0o' indicates octal, and '0x' indicates a hex |
| 881 |
number. |
| 882 |
|
| 883 |
Do not use the built-in function :func:`eval` if all you need is to convert |
| 884 |
strings to numbers. :func:`eval` will be significantly slower and it presents a |
| 885 |
security risk: someone could pass you a Python expression that might have |
| 886 |
unwanted side effects. For example, someone could pass |
| 887 |
``__import__('os').system("rm -rf $HOME")`` which would erase your home |
| 888 |
directory. |
| 889 |
|
| 890 |
:func:`eval` also has the effect of interpreting numbers as Python expressions, |
| 891 |
so that, for example, ``eval('09')`` gives a syntax error because Python does not allow |
| 892 |
leading '0' in a decimal number (except '0'). |
| 893 |
|
| 894 |
|
| 895 |
How do I convert a number to a string? |
| 896 |
-------------------------------------- |
| 897 |
|
| 898 |
For example, to convert the number ``144`` to the string ``'144'``, use the built-in type |
| 899 |
constructor :func:`str`. If you want a hexadecimal or octal representation, use |
| 900 |
the built-in functions :func:`hex` or :func:`oct`. For fancy formatting, see |
| 901 |
the :ref:`f-strings` and :ref:`formatstrings` sections. |
| 902 |
For example, ``"{:04d}".format(144)`` yields |
| 903 |
``'0144'`` and ``"{:.3f}".format(1.0/3.0)`` yields ``'0.333'``. |
| 904 |
|
| 905 |
|
| 906 |
How do I modify a string in place? |
| 907 |
---------------------------------- |
| 908 |
|
| 909 |
You can't, because strings are immutable. In most situations, you should |
| 910 |
simply construct a new string from the various parts you want to assemble |
| 911 |
it from. However, if you need an object with the ability to modify in-place |
| 912 |
Unicode data, try using an :class:`io.StringIO` object or the :mod:`array` |
| 913 |
module:: |
| 914 |
|
| 915 |
>>> import io |
| 916 |
>>> s = "Hello, world" |
| 917 |
>>> sio = io.StringIO(s) |
| 918 |
>>> sio.getvalue() |
| 919 |
'Hello, world' |
| 920 |
>>> sio.seek(7) |
| 921 |
7 |
| 922 |
>>> sio.write("there!") |
| 923 |
6 |
| 924 |
>>> sio.getvalue() |
| 925 |
'Hello, there!' |
| 926 |
|
| 927 |
>>> import array |
| 928 |
>>> a = array.array('w', s) |
| 929 |
>>> print(a) |
| 930 |
array('w', 'Hello, world') |
| 931 |
>>> a[0] = 'y' |
| 932 |
>>> print(a) |
| 933 |
array('w', 'yello, world') |
| 934 |
>>> a.tounicode() |
| 935 |
'yello, world' |
| 936 |
|
| 937 |
|
| 938 |
How do I use strings to call functions/methods? |
| 939 |
----------------------------------------------- |
| 940 |
|
| 941 |
There are various techniques. |
| 942 |
|
| 943 |
* The best is to use a dictionary that maps strings to functions. The primary |
| 944 |
advantage of this technique is that the strings do not need to match the names |
| 945 |
of the functions. This is also the primary technique used to emulate a case |
| 946 |
construct:: |
| 947 |
|
| 948 |
def a(): |
| 949 |
pass |
| 950 |
|
| 951 |
def b(): |
| 952 |
pass |
| 953 |
|
| 954 |
dispatch = {'go': a, 'stop': b} # Note lack of parens for funcs |
| 955 |
|
| 956 |
dispatch[get_input()]() # Note trailing parens to call function |
| 957 |
|
| 958 |
* Use the built-in function :func:`getattr`:: |
| 959 |
|
| 960 |
import foo |
| 961 |
getattr(foo, 'bar')() |
| 962 |
|
| 963 |
Note that :func:`getattr` works on any object, including classes, class |
| 964 |
instances, modules, and so on. |
| 965 |
|
| 966 |
This is used in several places in the standard library, like this:: |
| 967 |
|
| 968 |
class Foo: |
| 969 |
def do_foo(self): |
| 970 |
... |
| 971 |
|
| 972 |
def do_bar(self): |
| 973 |
... |
| 974 |
|
| 975 |
f = getattr(foo_instance, 'do_' + opname) |
| 976 |
f() |
| 977 |
|
| 978 |
|
| 979 |
* Use :func:`locals` to resolve the function name:: |
| 980 |
|
| 981 |
def myFunc(): |
| 982 |
print("hello") |
| 983 |
|
| 984 |
fname = "myFunc" |
| 985 |
|
| 986 |
f = locals()[fname] |
| 987 |
f() |
| 988 |
|
| 989 |
|
| 990 |
Is there an equivalent to Perl's ``chomp()`` for removing trailing newlines from strings? |
| 991 |
----------------------------------------------------------------------------------------- |
| 992 |
|
| 993 |
You can use ``S.rstrip("\r\n")`` to remove all occurrences of any line |
| 994 |
terminator from the end of the string ``S`` without removing other trailing |
| 995 |
whitespace. If the string ``S`` represents more than one line, with several |
| 996 |
empty lines at the end, the line terminators for all the blank lines will |
| 997 |
be removed:: |
| 998 |
|
| 999 |
>>> lines = ("line 1 \r\n" |
| 1000 |
... "\r\n" |
| 1001 |
... "\r\n") |
| 1002 |
>>> lines.rstrip("\n\r") |
| 1003 |
'line 1 ' |
| 1004 |
|
| 1005 |
Since this is typically only desired when reading text one line at a time, using |
| 1006 |
``S.rstrip()`` this way works well. |
| 1007 |
|
| 1008 |
|
| 1009 |
Is there a ``scanf()`` or ``sscanf()`` equivalent? |
| 1010 |
-------------------------------------------------- |
| 1011 |
|
| 1012 |
Not as such. |
| 1013 |
|
| 1014 |
For simple input parsing, the easiest approach is usually to split the line into |
| 1015 |
whitespace-delimited words using the :meth:`~str.split` method of string objects |
| 1016 |
and then convert decimal strings to numeric values using :func:`int` or |
| 1017 |
:func:`float`. :meth:`!split` supports an optional "sep" parameter which is useful |
| 1018 |
if the line uses something other than whitespace as a separator. |
| 1019 |
|
| 1020 |
For more complicated input parsing, regular expressions are more powerful |
| 1021 |
than C's ``sscanf`` and better suited for the task. |
| 1022 |
|
| 1023 |
|
| 1024 |
What does ``UnicodeDecodeError`` or ``UnicodeEncodeError`` error mean? |
| 1025 |
---------------------------------------------------------------------- |
| 1026 |
|
| 1027 |
See the :ref:`unicode-howto`. |
| 1028 |
|
| 1029 |
|
| 1030 |
.. _faq-programming-raw-string-backslash: |
| 1031 |
|
| 1032 |
Can I end a raw string with an odd number of backslashes? |
| 1033 |
--------------------------------------------------------- |
| 1034 |
|
| 1035 |
A raw string ending with an odd number of backslashes will escape the string's quote:: |
| 1036 |
|
| 1037 |
>>> r'C:\this\will\not\work\' |
| 1038 |
File "<stdin>", line 1 |
| 1039 |
r'C:\this\will\not\work\' |
| 1040 |
^ |
| 1041 |
SyntaxError: unterminated string literal (detected at line 1) |
| 1042 |
|
| 1043 |
There are several workarounds for this. One is to use regular strings and double |
| 1044 |
the backslashes:: |
| 1045 |
|
| 1046 |
>>> 'C:\\this\\will\\work\\' |
| 1047 |
'C:\\this\\will\\work\\' |
| 1048 |
|
| 1049 |
Another is to concatenate a regular string containing an escaped backslash to the |
| 1050 |
raw string:: |
| 1051 |
|
| 1052 |
>>> r'C:\this\will\work' '\\' |
| 1053 |
'C:\\this\\will\\work\\' |
| 1054 |
|
| 1055 |
It is also possible to use :func:`os.path.join` to append a backslash on Windows:: |
| 1056 |
|
| 1057 |
>>> os.path.join(r'C:\this\will\work', '') |
| 1058 |
'C:\\this\\will\\work\\' |
| 1059 |
|
| 1060 |
Note that while a backslash will "escape" a quote for the purposes of |
| 1061 |
determining where the raw string ends, no escaping occurs when interpreting the |
| 1062 |
value of the raw string. That is, the backslash remains present in the value of |
| 1063 |
the raw string:: |
| 1064 |
|
| 1065 |
>>> r'backslash\'preserved' |
| 1066 |
"backslash\\'preserved" |
| 1067 |
|
| 1068 |
Also see the specification in the :ref:`language reference <strings>`. |
| 1069 |
|
| 1070 |
|
| 1071 |
Performance |
| 1072 |
=========== |
| 1073 |
|
| 1074 |
My program is too slow. How do I speed it up? |
| 1075 |
--------------------------------------------- |
| 1076 |
|
| 1077 |
That's a tough one, in general. First, here is a list of things to |
| 1078 |
remember before diving further: |
| 1079 |
|
| 1080 |
* Performance characteristics vary across Python implementations. This FAQ |
| 1081 |
focuses on :term:`CPython`. |
| 1082 |
* Behaviour can vary across operating systems, especially when talking about |
| 1083 |
I/O or multi-threading. |
| 1084 |
* You should always find the hot spots in your program *before* attempting to |
| 1085 |
optimize any code (see the :mod:`profile` module). |
| 1086 |
* Writing benchmark scripts will allow you to iterate quickly when searching |
| 1087 |
for improvements (see the :mod:`timeit` module). |
| 1088 |
* It is highly recommended to have good code coverage (through unit testing |
| 1089 |
or any other technique) before potentially introducing regressions hidden |
| 1090 |
in sophisticated optimizations. |
| 1091 |
|
| 1092 |
That being said, there are many tricks to speed up Python code. Here are |
| 1093 |
some general principles which go a long way towards reaching acceptable |
| 1094 |
performance levels: |
| 1095 |
|
| 1096 |
* Making your algorithms faster (or changing to faster ones) can yield |
| 1097 |
much larger benefits than trying to sprinkle micro-optimization tricks |
| 1098 |
all over your code. |
| 1099 |
|
| 1100 |
* Use the right data structures. Study documentation for the :ref:`bltin-types` |
| 1101 |
and the :mod:`collections` module. |
| 1102 |
|
| 1103 |
* When the standard library provides a primitive for doing something, it is |
| 1104 |
likely (although not guaranteed) to be faster than any alternative you |
| 1105 |
may come up with. This is doubly true for primitives written in C, such |
| 1106 |
as builtins and some extension types. For example, be sure to use |
| 1107 |
either the :meth:`list.sort` built-in method or the related :func:`sorted` |
| 1108 |
function to do sorting (and see the :ref:`sortinghowto` for examples |
| 1109 |
of moderately advanced usage). |
| 1110 |
|
| 1111 |
* Abstractions tend to create indirections and force the interpreter to work |
| 1112 |
more. If the levels of indirection outweigh the amount of useful work |
| 1113 |
done, your program will be slower. You should avoid excessive abstraction, |
| 1114 |
especially under the form of tiny functions or methods (which are also often |
| 1115 |
detrimental to readability). |
| 1116 |
|
| 1117 |
If you have reached the limit of what pure Python can allow, there are tools |
| 1118 |
to take you further away. For example, `Cython <https://cython.org>`_ can |
| 1119 |
compile a slightly modified version of Python code into a C extension, and |
| 1120 |
can be used on many different platforms. Cython can take advantage of |
| 1121 |
compilation (and optional type annotations) to make your code significantly |
| 1122 |
faster than when interpreted. If you are confident in your C programming |
| 1123 |
skills, you can also :ref:`write a C extension module <extending-index>` |
| 1124 |
yourself. |
| 1125 |
|
| 1126 |
.. seealso:: |
| 1127 |
The wiki page devoted to `performance tips |
| 1128 |
<https://wiki.python.org/moin/PythonSpeed/PerformanceTips>`_. |
| 1129 |
|
| 1130 |
|
| 1131 |
.. _efficient_string_concatenation: |
| 1132 |
|
| 1133 |
What is the most efficient way to concatenate many strings together? |
| 1134 |
-------------------------------------------------------------------- |
| 1135 |
|
| 1136 |
:class:`str` and :class:`bytes` objects are immutable, therefore concatenating |
| 1137 |
many strings together is inefficient as each concatenation creates a new |
| 1138 |
object. In the general case, the total runtime cost is quadratic in the |
| 1139 |
total string length. |
| 1140 |
|
| 1141 |
To accumulate many :class:`str` objects, the recommended idiom is to place |
| 1142 |
them into a list and call :meth:`str.join` at the end:: |
| 1143 |
|
| 1144 |
chunks = [] |
| 1145 |
for s in my_strings: |
| 1146 |
chunks.append(s) |
| 1147 |
result = ''.join(chunks) |
| 1148 |
|
| 1149 |
(Another reasonably efficient idiom is to use :class:`io.StringIO`.) |
| 1150 |
|
| 1151 |
To accumulate many :class:`bytes` objects, the recommended idiom is to extend |
| 1152 |
a :class:`bytearray` object using in-place concatenation (the ``+=`` operator):: |
| 1153 |
|
| 1154 |
result = bytearray() |
| 1155 |
for b in my_bytes_objects: |
| 1156 |
result += b |
| 1157 |
|
| 1158 |
|
| 1159 |
Sequences (tuples/lists) |
| 1160 |
======================== |
| 1161 |
|
| 1162 |
How do I convert between tuples and lists? |
| 1163 |
------------------------------------------ |
| 1164 |
|
| 1165 |
The type constructor ``tuple(seq)`` converts any sequence (actually, any |
| 1166 |
iterable) into a tuple with the same items in the same order. |
| 1167 |
|
| 1168 |
For example, ``tuple([1, 2, 3])`` yields ``(1, 2, 3)`` and ``tuple('abc')`` |
| 1169 |
yields ``('a', 'b', 'c')``. If the argument is a tuple, it does not make a copy |
| 1170 |
but returns the same object, so it is cheap to call :func:`tuple` when you |
| 1171 |
aren't sure that an object is already a tuple. |
| 1172 |
|
| 1173 |
The type constructor ``list(seq)`` converts any sequence or iterable into a list |
| 1174 |
with the same items in the same order. For example, ``list((1, 2, 3))`` yields |
| 1175 |
``[1, 2, 3]`` and ``list('abc')`` yields ``['a', 'b', 'c']``. If the argument |
| 1176 |
is a list, it makes a copy just like ``seq[:]`` would. |
| 1177 |
|
| 1178 |
|
| 1179 |
What's a negative index? |
| 1180 |
------------------------ |
| 1181 |
|
| 1182 |
Python sequences are indexed with positive numbers and negative numbers. For |
| 1183 |
positive numbers 0 is the first index 1 is the second index and so forth. For |
| 1184 |
negative indices -1 is the last index and -2 is the penultimate (next to last) |
| 1185 |
index and so forth. Think of ``seq[-n]`` as the same as ``seq[len(seq)-n]``. |
| 1186 |
|
| 1187 |
Using negative indices can be very convenient. For example ``S[:-1]`` is all of |
| 1188 |
the string except for its last character, which is useful for removing the |
| 1189 |
trailing newline from a string. |
| 1190 |
|
| 1191 |
|
| 1192 |
How do I iterate over a sequence in reverse order? |
| 1193 |
-------------------------------------------------- |
| 1194 |
|
| 1195 |
Use the :func:`reversed` built-in function:: |
| 1196 |
|
| 1197 |
for x in reversed(sequence): |
| 1198 |
... # do something with x ... |
| 1199 |
|
| 1200 |
This won't touch your original sequence, but build a new copy with reversed |
| 1201 |
order to iterate over. |
| 1202 |
|
| 1203 |
|
| 1204 |
How do you remove duplicates from a list? |
| 1205 |
----------------------------------------- |
| 1206 |
|
| 1207 |
See the Python Cookbook for a long discussion of many ways to do this: |
| 1208 |
|
| 1209 |
https://code.activestate.com/recipes/52560/ |
| 1210 |
|
| 1211 |
If you don't mind reordering the list, sort it and then scan from the end of the |
| 1212 |
list, deleting duplicates as you go:: |
| 1213 |
|
| 1214 |
if mylist: |
| 1215 |
mylist.sort() |
| 1216 |
last = mylist[-1] |
| 1217 |
for i in range(len(mylist)-2, -1, -1): |
| 1218 |
if last == mylist[i]: |
| 1219 |
del mylist[i] |
| 1220 |
else: |
| 1221 |
last = mylist[i] |
| 1222 |
|
| 1223 |
If all elements of the list may be used as set keys (that is, they are all |
| 1224 |
:term:`hashable`) this is often faster:: |
| 1225 |
|
| 1226 |
mylist = list(set(mylist)) |
| 1227 |
|
| 1228 |
This converts the list into a set, thereby removing duplicates, and then back |
| 1229 |
into a list. |
| 1230 |
|
| 1231 |
|
| 1232 |
How do you remove multiple items from a list? |
| 1233 |
--------------------------------------------- |
| 1234 |
|
| 1235 |
As with removing duplicates, explicitly iterating in reverse with a |
| 1236 |
delete condition is one possibility. However, it is easier and faster |
| 1237 |
to use slice replacement with an implicit or explicit forward iteration. |
| 1238 |
Here are three variations:: |
| 1239 |
|
| 1240 |
mylist[:] = filter(keep_function, mylist) |
| 1241 |
mylist[:] = (x for x in mylist if keep_condition) |
| 1242 |
mylist[:] = [x for x in mylist if keep_condition] |
| 1243 |
|
| 1244 |
The list comprehension may be fastest. |
| 1245 |
|
| 1246 |
|
| 1247 |
How do you make an array in Python? |
| 1248 |
----------------------------------- |
| 1249 |
|
| 1250 |
Use a list:: |
| 1251 |
|
| 1252 |
["this", 1, "is", "an", "array"] |
| 1253 |
|
| 1254 |
Lists are equivalent to C or Pascal arrays in their time complexity; the primary |
| 1255 |
difference is that a Python list can contain objects of many different types. |
| 1256 |
|
| 1257 |
The ``array`` module also provides methods for creating arrays of fixed types |
| 1258 |
with compact representations, but they are slower to index than lists. Also |
| 1259 |
note that `NumPy <https://numpy.org/>`_ |
| 1260 |
and other third-party packages define array-like structures with |
| 1261 |
various characteristics as well. |
| 1262 |
|
| 1263 |
To get Lisp-style linked lists, you can emulate *cons cells* using tuples:: |
| 1264 |
|
| 1265 |
lisp_list = ("like", ("this", ("example", None) ) ) |
| 1266 |
|
| 1267 |
If mutability is desired, you could use lists instead of tuples. Here the |
| 1268 |
analogue of a Lisp *car* is ``lisp_list[0]`` and the analogue of *cdr* is |
| 1269 |
``lisp_list[1]``. Only do this if you're sure you really need to, because it's |
| 1270 |
usually a lot slower than using Python lists. |
| 1271 |
|
| 1272 |
|
| 1273 |
.. _faq-multidimensional-list: |
| 1274 |
|
| 1275 |
How do I create a multidimensional list? |
| 1276 |
---------------------------------------- |
| 1277 |
|
| 1278 |
You probably tried to make a multidimensional array like this:: |
| 1279 |
|
| 1280 |
>>> A = [[None] * 2] * 3 |
| 1281 |
|
| 1282 |
This looks correct if you print it: |
| 1283 |
|
| 1284 |
.. testsetup:: |
| 1285 |
|
| 1286 |
A = [[None] * 2] * 3 |
| 1287 |
|
| 1288 |
.. doctest:: |
| 1289 |
|
| 1290 |
>>> A |
| 1291 |
[[None, None], [None, None], [None, None]] |
| 1292 |
|
| 1293 |
But when you assign a value, it shows up in multiple places: |
| 1294 |
|
| 1295 |
.. testsetup:: |
| 1296 |
|
| 1297 |
A = [[None] * 2] * 3 |
| 1298 |
|
| 1299 |
.. doctest:: |
| 1300 |
|
| 1301 |
>>> A[0][0] = 5 |
| 1302 |
>>> A |
| 1303 |
[[5, None], [5, None], [5, None]] |
| 1304 |
|
| 1305 |
The reason is that replicating a list with ``*`` doesn't create copies, it only |
| 1306 |
creates references to the existing objects. The ``*3`` creates a list |
| 1307 |
containing 3 references to the same list of length two. Changes to one row will |
| 1308 |
show in all rows, which is almost certainly not what you want. |
| 1309 |
|
| 1310 |
The suggested approach is to create a list of the desired length first and then |
| 1311 |
fill in each element with a newly created list:: |
| 1312 |
|
| 1313 |
A = [None] * 3 |
| 1314 |
for i in range(3): |
| 1315 |
A[i] = [None] * 2 |
| 1316 |
|
| 1317 |
This generates a list containing 3 different lists of length two. You can also |
| 1318 |
use a list comprehension:: |
| 1319 |
|
| 1320 |
w, h = 2, 3 |
| 1321 |
A = [[None] * w for i in range(h)] |
| 1322 |
|
| 1323 |
Or, you can use an extension that provides a matrix datatype; `NumPy |
| 1324 |
<https://numpy.org/>`_ is the best known. |
| 1325 |
|
| 1326 |
|
| 1327 |
How do I apply a method or function to a sequence of objects? |
| 1328 |
------------------------------------------------------------- |
| 1329 |
|
| 1330 |
To call a method or function and accumulate the return values in a list, |
| 1331 |
a :term:`list comprehension` is an elegant solution:: |
| 1332 |
|
| 1333 |
result = [obj.method() for obj in mylist] |
| 1334 |
|
| 1335 |
result = [function(obj) for obj in mylist] |
| 1336 |
|
| 1337 |
To just run the method or function without saving the return values, |
| 1338 |
a plain :keyword:`for` loop will suffice:: |
| 1339 |
|
| 1340 |
for obj in mylist: |
| 1341 |
obj.method() |
| 1342 |
|
| 1343 |
for obj in mylist: |
| 1344 |
function(obj) |
| 1345 |
|
| 1346 |
|
| 1347 |
.. _faq-augmented-assignment-tuple-error: |
| 1348 |
|
| 1349 |
Why does a_tuple[i] += ['item'] raise an exception when the addition works? |
| 1350 |
--------------------------------------------------------------------------- |
| 1351 |
|
| 1352 |
This is because of a combination of the fact that augmented assignment |
| 1353 |
operators are *assignment* operators, and the difference between mutable and |
| 1354 |
immutable objects in Python. |
| 1355 |
|
| 1356 |
This discussion applies in general when augmented assignment operators are |
| 1357 |
applied to elements of a tuple that point to mutable objects, but we'll use |
| 1358 |
a ``list`` and ``+=`` as our exemplar. |
| 1359 |
|
| 1360 |
If you wrote:: |
| 1361 |
|
| 1362 |
>>> a_tuple = (1, 2) |
| 1363 |
>>> a_tuple[0] += 1 |
| 1364 |
Traceback (most recent call last): |
| 1365 |
... |
| 1366 |
TypeError: 'tuple' object does not support item assignment |
| 1367 |
|
| 1368 |
The reason for the exception should be immediately clear: ``1`` is added to the |
| 1369 |
object ``a_tuple[0]`` points to (``1``), producing the result object, ``2``, |
| 1370 |
but when we attempt to assign the result of the computation, ``2``, to element |
| 1371 |
``0`` of the tuple, we get an error because we can't change what an element of |
| 1372 |
a tuple points to. |
| 1373 |
|
| 1374 |
Under the covers, what this augmented assignment statement is doing is |
| 1375 |
approximately this:: |
| 1376 |
|
| 1377 |
>>> result = a_tuple[0] + 1 |
| 1378 |
>>> a_tuple[0] = result |
| 1379 |
Traceback (most recent call last): |
| 1380 |
... |
| 1381 |
TypeError: 'tuple' object does not support item assignment |
| 1382 |
|
| 1383 |
It is the assignment part of the operation that produces the error, since a |
| 1384 |
tuple is immutable. |
| 1385 |
|
| 1386 |
When you write something like:: |
| 1387 |
|
| 1388 |
>>> a_tuple = (['foo'], 'bar') |
| 1389 |
>>> a_tuple[0] += ['item'] |
| 1390 |
Traceback (most recent call last): |
| 1391 |
... |
| 1392 |
TypeError: 'tuple' object does not support item assignment |
| 1393 |
|
| 1394 |
The exception is a bit more surprising, and even more surprising is the fact |
| 1395 |
that even though there was an error, the append worked:: |
| 1396 |
|
| 1397 |
>>> a_tuple[0] |
| 1398 |
['foo', 'item'] |
| 1399 |
|
| 1400 |
To see why this happens, you need to know that (a) if an object implements an |
| 1401 |
:meth:`~object.__iadd__` magic method, it gets called when the ``+=`` augmented |
| 1402 |
assignment |
| 1403 |
is executed, and its return value is what gets used in the assignment statement; |
| 1404 |
and (b) for lists, :meth:`!__iadd__` is equivalent to calling |
| 1405 |
:meth:`~sequence.extend` on the list and returning the list. |
| 1406 |
That's why we say that for lists, ``+=`` is a "shorthand" for :meth:`list.extend`:: |
| 1407 |
|
| 1408 |
>>> a_list = [] |
| 1409 |
>>> a_list += [1] |
| 1410 |
>>> a_list |
| 1411 |
[1] |
| 1412 |
|
| 1413 |
This is equivalent to:: |
| 1414 |
|
| 1415 |
>>> result = a_list.__iadd__([1]) |
| 1416 |
>>> a_list = result |
| 1417 |
|
| 1418 |
The object pointed to by a_list has been mutated, and the pointer to the |
| 1419 |
mutated object is assigned back to ``a_list``. The end result of the |
| 1420 |
assignment is a no-op, since it is a pointer to the same object that ``a_list`` |
| 1421 |
was previously pointing to, but the assignment still happens. |
| 1422 |
|
| 1423 |
Thus, in our tuple example what is happening is equivalent to:: |
| 1424 |
|
| 1425 |
>>> result = a_tuple[0].__iadd__(['item']) |
| 1426 |
>>> a_tuple[0] = result |
| 1427 |
Traceback (most recent call last): |
| 1428 |
... |
| 1429 |
TypeError: 'tuple' object does not support item assignment |
| 1430 |
|
| 1431 |
The :meth:`!__iadd__` succeeds, and thus the list is extended, but even though |
| 1432 |
``result`` points to the same object that ``a_tuple[0]`` already points to, |
| 1433 |
that final assignment still results in an error, because tuples are immutable. |
| 1434 |
|
| 1435 |
|
| 1436 |
I want to do a complicated sort: can you do a Schwartzian Transform in Python? |
| 1437 |
------------------------------------------------------------------------------ |
| 1438 |
|
| 1439 |
The technique, attributed to Randal Schwartz of the Perl community, sorts the |
| 1440 |
elements of a list by a metric which maps each element to its "sort value". In |
| 1441 |
Python, use the ``key`` argument for the :meth:`list.sort` method:: |
| 1442 |
|
| 1443 |
Isorted = L[:] |
| 1444 |
Isorted.sort(key=lambda s: int(s[10:15])) |
| 1445 |
|
| 1446 |
|
| 1447 |
How can I sort one list by values from another list? |
| 1448 |
---------------------------------------------------- |
| 1449 |
|
| 1450 |
Merge them into an iterator of tuples, sort the resulting list, and then pick |
| 1451 |
out the element you want. |
| 1452 |
|
| 1453 |
>>> list1 = ["what", "I'm", "sorting", "by"] |
| 1454 |
>>> list2 = ["something", "else", "to", "sort"] |
| 1455 |
>>> pairs = zip(list1, list2) |
| 1456 |
>>> pairs = sorted(pairs) |
| 1457 |
>>> pairs |
| 1458 |
[("I'm", 'else'), ('by', 'sort'), ('sorting', 'to'), ('what', 'something')] |
| 1459 |
>>> result = [x[1] for x in pairs] |
| 1460 |
>>> result |
| 1461 |
['else', 'sort', 'to', 'something'] |
| 1462 |
|
| 1463 |
|
| 1464 |
Objects |
| 1465 |
======= |
| 1466 |
|
| 1467 |
What is a class? |
| 1468 |
---------------- |
| 1469 |
|
| 1470 |
A class is the particular object type created by executing a class statement. |
| 1471 |
Class objects are used as templates to create instance objects, which embody |
| 1472 |
both the data (attributes) and code (methods) specific to a datatype. |
| 1473 |
|
| 1474 |
A class can be based on one or more other classes, called its base class(es). It |
| 1475 |
then inherits the attributes and methods of its base classes. This allows an |
| 1476 |
object model to be successively refined by inheritance. You might have a |
| 1477 |
generic ``Mailbox`` class that provides basic accessor methods for a mailbox, |
| 1478 |
and subclasses such as ``MboxMailbox``, ``MaildirMailbox``, ``OutlookMailbox`` |
| 1479 |
that handle various specific mailbox formats. |
| 1480 |
|
| 1481 |
|
| 1482 |
What is a method? |
| 1483 |
----------------- |
| 1484 |
|
| 1485 |
A method is a function on some object ``x`` that you normally call as |
| 1486 |
``x.name(arguments...)``. Methods are defined as functions inside the class |
| 1487 |
definition:: |
| 1488 |
|
| 1489 |
class C: |
| 1490 |
def meth(self, arg): |
| 1491 |
return arg * 2 + self.attribute |
| 1492 |
|
| 1493 |
|
| 1494 |
What is self? |
| 1495 |
------------- |
| 1496 |
|
| 1497 |
Self is merely a conventional name for the first argument of a method. A method |
| 1498 |
defined as ``meth(self, a, b, c)`` should be called as ``x.meth(a, b, c)`` for |
| 1499 |
some instance ``x`` of the class in which the definition occurs; the called |
| 1500 |
method will think it is called as ``meth(x, a, b, c)``. |
| 1501 |
|
| 1502 |
See also :ref:`why-self`. |
| 1503 |
|
| 1504 |
|
| 1505 |
How do I check if an object is an instance of a given class or of a subclass of it? |
| 1506 |
----------------------------------------------------------------------------------- |
| 1507 |
|
| 1508 |
Use the built-in function :func:`isinstance(obj, cls) <isinstance>`. You can |
| 1509 |
check if an object |
| 1510 |
is an instance of any of a number of classes by providing a tuple instead of a |
| 1511 |
single class, for example, ``isinstance(obj, (class1, class2, ...))``, and can also |
| 1512 |
check whether an object is one of Python's built-in types, for example, |
| 1513 |
``isinstance(obj, str)`` or ``isinstance(obj, (int, float, complex))``. |
| 1514 |
|
| 1515 |
Note that :func:`isinstance` also checks for virtual inheritance from an |
| 1516 |
:term:`abstract base class`. So, the test will return ``True`` for a |
| 1517 |
registered class even if hasn't directly or indirectly inherited from it. To |
| 1518 |
test for "true inheritance", scan the :term:`method resolution order` (MRO) of |
| 1519 |
the class: |
| 1520 |
|
| 1521 |
.. testcode:: |
| 1522 |
|
| 1523 |
from collections.abc import Mapping |
| 1524 |
|
| 1525 |
class P: |
| 1526 |
pass |
| 1527 |
|
| 1528 |
class C(P): |
| 1529 |
pass |
| 1530 |
|
| 1531 |
Mapping.register(P) |
| 1532 |
|
| 1533 |
.. doctest:: |
| 1534 |
|
| 1535 |
>>> c = C() |
| 1536 |
>>> isinstance(c, C) # direct |
| 1537 |
True |
| 1538 |
>>> isinstance(c, P) # indirect |
| 1539 |
True |
| 1540 |
>>> isinstance(c, Mapping) # virtual |
| 1541 |
True |
| 1542 |
|
| 1543 |
# Actual inheritance chain |
| 1544 |
>>> type(c).__mro__ |
| 1545 |
(<class 'C'>, <class 'P'>, <class 'object'>) |
| 1546 |
|
| 1547 |
# Test for "true inheritance" |
| 1548 |
>>> Mapping in type(c).__mro__ |
| 1549 |
False |
| 1550 |
|
| 1551 |
Note that most programs do not use :func:`isinstance` on user-defined classes |
| 1552 |
very often. If you are developing the classes yourself, a more proper |
| 1553 |
object-oriented style is to define methods on the classes that encapsulate a |
| 1554 |
particular behaviour, instead of checking the object's class and doing a |
| 1555 |
different thing based on what class it is. For example, if you have a function |
| 1556 |
that does something:: |
| 1557 |
|
| 1558 |
def search(obj): |
| 1559 |
if isinstance(obj, Mailbox): |
| 1560 |
... # code to search a mailbox |
| 1561 |
elif isinstance(obj, Document): |
| 1562 |
... # code to search a document |
| 1563 |
elif ... |
| 1564 |
|
| 1565 |
A better approach is to define a ``search()`` method on all the classes and just |
| 1566 |
call it:: |
| 1567 |
|
| 1568 |
class Mailbox: |
| 1569 |
def search(self): |
| 1570 |
... # code to search a mailbox |
| 1571 |
|
| 1572 |
class Document: |
| 1573 |
def search(self): |
| 1574 |
... # code to search a document |
| 1575 |
|
| 1576 |
obj.search() |
| 1577 |
|
| 1578 |
|
| 1579 |
What is delegation? |
| 1580 |
------------------- |
| 1581 |
|
| 1582 |
Delegation is an object-oriented technique (also called a design pattern). |
| 1583 |
Let's say you have an object ``x`` and want to change the behaviour of just one |
| 1584 |
of its methods. You can create a new class that provides a new implementation |
| 1585 |
of the method you're interested in changing and delegates all other methods to |
| 1586 |
the corresponding method of ``x``. |
| 1587 |
|
| 1588 |
Python programmers can easily implement delegation. For example, the following |
| 1589 |
class implements a class that behaves like a file but converts all written data |
| 1590 |
to uppercase:: |
| 1591 |
|
| 1592 |
class UpperOut: |
| 1593 |
|
| 1594 |
def __init__(self, outfile): |
| 1595 |
self._outfile = outfile |
| 1596 |
|
| 1597 |
def write(self, s): |
| 1598 |
self._outfile.write(s.upper()) |
| 1599 |
|
| 1600 |
def __getattr__(self, name): |
| 1601 |
return getattr(self._outfile, name) |
| 1602 |
|
| 1603 |
Here the ``UpperOut`` class redefines the ``write()`` method to convert the |
| 1604 |
argument string to uppercase before calling the underlying |
| 1605 |
``self._outfile.write()`` method. All other methods are delegated to the |
| 1606 |
underlying ``self._outfile`` object. The delegation is accomplished via the |
| 1607 |
:meth:`~object.__getattr__` method; consult :ref:`the language reference <attribute-access>` |
| 1608 |
for more information about controlling attribute access. |
| 1609 |
|
| 1610 |
Note that for more general cases delegation can get trickier. When attributes |
| 1611 |
must be set as well as retrieved, the class must define a :meth:`~object.__setattr__` |
| 1612 |
method too, and it must do so carefully. The basic implementation of |
| 1613 |
:meth:`!__setattr__` is roughly equivalent to the following:: |
| 1614 |
|
| 1615 |
class X: |
| 1616 |
... |
| 1617 |
def __setattr__(self, name, value): |
| 1618 |
self.__dict__[name] = value |
| 1619 |
... |
| 1620 |
|
| 1621 |
Many :meth:`~object.__setattr__` implementations call :meth:`!object.__setattr__` to set |
| 1622 |
an attribute on self without causing infinite recursion:: |
| 1623 |
|
| 1624 |
class X: |
| 1625 |
def __setattr__(self, name, value): |
| 1626 |
# Custom logic here... |
| 1627 |
object.__setattr__(self, name, value) |
| 1628 |
|
| 1629 |
Alternatively, it is possible to set attributes by inserting |
| 1630 |
entries into :attr:`self.__dict__ <object.__dict__>` directly. |
| 1631 |
|
| 1632 |
|
| 1633 |
How do I call a method defined in a base class from a derived class that extends it? |
| 1634 |
------------------------------------------------------------------------------------ |
| 1635 |
|
| 1636 |
Use the built-in :func:`super` function:: |
| 1637 |
|
| 1638 |
class Derived(Base): |
| 1639 |
def meth(self): |
| 1640 |
super().meth() # calls Base.meth |
| 1641 |
|
| 1642 |
In the example, :func:`super` will automatically determine the instance from |
| 1643 |
which it was called (the ``self`` value), look up the :term:`method resolution |
| 1644 |
order` (MRO) with ``type(self).__mro__``, and return the next in line after |
| 1645 |
``Derived`` in the MRO: ``Base``. |
| 1646 |
|
| 1647 |
|
| 1648 |
How can I organize my code to make it easier to change the base class? |
| 1649 |
---------------------------------------------------------------------- |
| 1650 |
|
| 1651 |
You could assign the base class to an alias and derive from the alias. Then all |
| 1652 |
you have to change is the value assigned to the alias. Incidentally, this trick |
| 1653 |
is also handy if you want to decide dynamically (such as depending on availability |
| 1654 |
of resources) which base class to use. Example:: |
| 1655 |
|
| 1656 |
class Base: |
| 1657 |
... |
| 1658 |
|
| 1659 |
BaseAlias = Base |
| 1660 |
|
| 1661 |
class Derived(BaseAlias): |
| 1662 |
... |
| 1663 |
|
| 1664 |
|
| 1665 |
How do I create static class data and static class methods? |
| 1666 |
----------------------------------------------------------- |
| 1667 |
|
| 1668 |
Both static data and static methods (in the sense of C++ or Java) are supported |
| 1669 |
in Python. |
| 1670 |
|
| 1671 |
For static data, simply define a class attribute. To assign a new value to the |
| 1672 |
attribute, you have to explicitly use the class name in the assignment:: |
| 1673 |
|
| 1674 |
class C: |
| 1675 |
count = 0 # number of times C.__init__ called |
| 1676 |
|
| 1677 |
def __init__(self): |
| 1678 |
C.count = C.count + 1 |
| 1679 |
|
| 1680 |
def getcount(self): |
| 1681 |
return C.count # or return self.count |
| 1682 |
|
| 1683 |
``c.count`` also refers to ``C.count`` for any ``c`` such that ``isinstance(c, |
| 1684 |
C)`` holds, unless overridden by ``c`` itself or by some class on the base-class |
| 1685 |
search path from ``c.__class__`` back to ``C``. |
| 1686 |
|
| 1687 |
Caution: within a method of C, an assignment like ``self.count = 42`` creates a |
| 1688 |
new and unrelated instance named "count" in ``self``'s own dict. Rebinding of a |
| 1689 |
class-static data name must always specify the class whether inside a method or |
| 1690 |
not:: |
| 1691 |
|
| 1692 |
C.count = 314 |
| 1693 |
|
| 1694 |
Static methods are possible:: |
| 1695 |
|
| 1696 |
class C: |
| 1697 |
@staticmethod |
| 1698 |
def static(arg1, arg2, arg3): |
| 1699 |
# No 'self' parameter! |
| 1700 |
... |
| 1701 |
|
| 1702 |
However, a far more straightforward way to get the effect of a static method is |
| 1703 |
via a simple module-level function:: |
| 1704 |
|
| 1705 |
def getcount(): |
| 1706 |
return C.count |
| 1707 |
|
| 1708 |
If your code is structured so as to define one class (or tightly related class |
| 1709 |
hierarchy) per module, this supplies the desired encapsulation. |
| 1710 |
|
| 1711 |
|
| 1712 |
How can I overload constructors (or methods) in Python? |
| 1713 |
------------------------------------------------------- |
| 1714 |
|
| 1715 |
This answer actually applies to all methods, but the question usually comes up |
| 1716 |
first in the context of constructors. |
| 1717 |
|
| 1718 |
In C++ you'd write: |
| 1719 |
|
| 1720 |
.. code-block:: c++ |
| 1721 |
|
| 1722 |
class C { |
| 1723 |
C() { cout << "No arguments\n"; } |
| 1724 |
C(int i) { cout << "Argument is " << i << "\n"; } |
| 1725 |
} |
| 1726 |
|
| 1727 |
In Python you have to write a single constructor that catches all cases using |
| 1728 |
default arguments. For example:: |
| 1729 |
|
| 1730 |
class C: |
| 1731 |
def __init__(self, i=None): |
| 1732 |
if i is None: |
| 1733 |
print("No arguments") |
| 1734 |
else: |
| 1735 |
print("Argument is", i) |
| 1736 |
|
| 1737 |
This is not entirely equivalent, but close enough in practice. |
| 1738 |
|
| 1739 |
You could also try a variable-length argument list, for example:: |
| 1740 |
|
| 1741 |
def __init__(self, *args): |
| 1742 |
... |
| 1743 |
|
| 1744 |
The same approach works for all method definitions. |
| 1745 |
|
| 1746 |
|
| 1747 |
I try to use __spam and I get an error about _SomeClassName__spam. |
| 1748 |
------------------------------------------------------------------ |
| 1749 |
|
| 1750 |
Variable names with double leading underscores are "mangled" to provide a simple |
| 1751 |
but effective way to define class private variables. Any identifier of the form |
| 1752 |
``__spam`` (at least two leading underscores, at most one trailing underscore) |
| 1753 |
is textually replaced with ``_classname__spam``, where ``classname`` is the |
| 1754 |
current class name with any leading underscores stripped. |
| 1755 |
|
| 1756 |
The identifier can be used unchanged within the class, but to access it outside |
| 1757 |
the class, the mangled name must be used: |
| 1758 |
|
| 1759 |
.. code-block:: python |
| 1760 |
|
| 1761 |
class A: |
| 1762 |
def __one(self): |
| 1763 |
return 1 |
| 1764 |
def two(self): |
| 1765 |
return 2 * self.__one() |
| 1766 |
|
| 1767 |
class B(A): |
| 1768 |
def three(self): |
| 1769 |
return 3 * self._A__one() |
| 1770 |
|
| 1771 |
four = 4 * A()._A__one() |
| 1772 |
|
| 1773 |
In particular, this does not guarantee privacy since an outside user can still |
| 1774 |
deliberately access the private attribute; many Python programmers never bother |
| 1775 |
to use private variable names at all. |
| 1776 |
|
| 1777 |
.. seealso:: |
| 1778 |
|
| 1779 |
The :ref:`private name mangling specifications <private-name-mangling>` |
| 1780 |
for details and special cases. |
| 1781 |
|
| 1782 |
|
| 1783 |
My class defines __del__ but it is not called when I delete the object. |
| 1784 |
----------------------------------------------------------------------- |
| 1785 |
|
| 1786 |
There are several possible reasons for this. |
| 1787 |
|
| 1788 |
The :keyword:`del` statement does not necessarily call :meth:`~object.__del__` -- it simply |
| 1789 |
decrements the object's reference count, and if this reaches zero |
| 1790 |
:meth:`!__del__` is called. |
| 1791 |
|
| 1792 |
If your data structures contain circular links (for example, a tree where each child has |
| 1793 |
a parent reference and each parent has a list of children) the reference counts |
| 1794 |
will never go back to zero. Once in a while Python runs an algorithm to detect |
| 1795 |
such cycles, but the garbage collector might run some time after the last |
| 1796 |
reference to your data structure vanishes, so your :meth:`!__del__` method may be |
| 1797 |
called at an inconvenient and random time. This is inconvenient if you're trying |
| 1798 |
to reproduce a problem. Worse, the order in which object's :meth:`!__del__` |
| 1799 |
methods are executed is arbitrary. You can run :func:`gc.collect` to force a |
| 1800 |
collection, but there *are* pathological cases where objects will never be |
| 1801 |
collected. |
| 1802 |
|
| 1803 |
Despite the cycle collector, it's still a good idea to define an explicit |
| 1804 |
``close()`` method on objects to be called whenever you're done with them. The |
| 1805 |
``close()`` method can then remove attributes that refer to subobjects. Don't |
| 1806 |
call :meth:`!__del__` directly -- :meth:`!__del__` should call ``close()`` and |
| 1807 |
``close()`` should make sure that it can be called more than once for the same |
| 1808 |
object. |
| 1809 |
|
| 1810 |
Another way to avoid cyclical references is to use the :mod:`weakref` module, |
| 1811 |
which allows you to point to objects without incrementing their reference count. |
| 1812 |
Tree data structures, for instance, should use weak references for their parent |
| 1813 |
and sibling references (if they need them!). |
| 1814 |
|
| 1815 |
.. XXX relevant for Python 3? |
| 1816 |
|
| 1817 |
If the object has ever been a local variable in a function that caught an |
| 1818 |
expression in an except clause, chances are that a reference to the object |
| 1819 |
still exists in that function's stack frame as contained in the stack trace. |
| 1820 |
Normally, calling :func:`sys.exc_clear` will take care of this by clearing |
| 1821 |
the last recorded exception. |
| 1822 |
|
| 1823 |
Finally, if your :meth:`!__del__` method raises an exception, a warning message |
| 1824 |
is printed to :data:`sys.stderr`. |
| 1825 |
|
| 1826 |
|
| 1827 |
How do I get a list of all instances of a given class? |
| 1828 |
------------------------------------------------------ |
| 1829 |
|
| 1830 |
Python does not keep track of all instances of a class (or of a built-in type). |
| 1831 |
You can program the class's constructor to keep track of all instances by |
| 1832 |
keeping a list of weak references to each instance. |
| 1833 |
|
| 1834 |
|
| 1835 |
Why does the result of ``id()`` appear to be not unique? |
| 1836 |
-------------------------------------------------------- |
| 1837 |
|
| 1838 |
The :func:`id` builtin returns an integer that is guaranteed to be unique during |
| 1839 |
the lifetime of the object. Since in CPython, this is the object's memory |
| 1840 |
address, it happens frequently that after an object is deleted from memory, the |
| 1841 |
next freshly created object is allocated at the same position in memory. This |
| 1842 |
is illustrated by this example: |
| 1843 |
|
| 1844 |
>>> id(1000) # doctest: +SKIP |
| 1845 |
13901272 |
| 1846 |
>>> id(2000) # doctest: +SKIP |
| 1847 |
13901272 |
| 1848 |
|
| 1849 |
The two ids belong to different integer objects that are created before, and |
| 1850 |
deleted immediately after execution of the ``id()`` call. To be sure that |
| 1851 |
objects whose id you want to examine are still alive, create another reference |
| 1852 |
to the object: |
| 1853 |
|
| 1854 |
>>> a = 1000; b = 2000 |
| 1855 |
>>> id(a) # doctest: +SKIP |
| 1856 |
13901272 |
| 1857 |
>>> id(b) # doctest: +SKIP |
| 1858 |
13891296 |
| 1859 |
|
| 1860 |
|
| 1861 |
.. _faq-identity-with-is: |
| 1862 |
|
| 1863 |
When can I rely on identity tests with the *is* operator? |
| 1864 |
--------------------------------------------------------- |
| 1865 |
|
| 1866 |
The ``is`` operator tests for object identity. The test ``a is b`` is |
| 1867 |
equivalent to ``id(a) == id(b)``. |
| 1868 |
|
| 1869 |
The most important property of an identity test is that an object is always |
| 1870 |
identical to itself, ``a is a`` always returns ``True``. Identity tests are |
| 1871 |
usually faster than equality tests. And unlike equality tests, identity tests |
| 1872 |
are guaranteed to return a boolean ``True`` or ``False``. |
| 1873 |
|
| 1874 |
However, identity tests can *only* be substituted for equality tests when |
| 1875 |
object identity is assured. Generally, there are three circumstances where |
| 1876 |
identity is guaranteed: |
| 1877 |
|
| 1878 |
1) Assignments create new names but do not change object identity. After the |
| 1879 |
assignment ``new = old``, it is guaranteed that ``new is old``. |
| 1880 |
|
| 1881 |
2) Putting an object in a container that stores object references does not |
| 1882 |
change object identity. After the list assignment ``s[0] = x``, it is |
| 1883 |
guaranteed that ``s[0] is x``. |
| 1884 |
|
| 1885 |
3) If an object is a singleton, it means that only one instance of that object |
| 1886 |
can exist. After the assignments ``a = None`` and ``b = None``, it is |
| 1887 |
guaranteed that ``a is b`` because ``None`` is a singleton. |
| 1888 |
|
| 1889 |
In most other circumstances, identity tests are inadvisable and equality tests |
| 1890 |
are preferred. In particular, identity tests should not be used to check |
| 1891 |
constants such as :class:`int` and :class:`str` which aren't guaranteed to be |
| 1892 |
singletons:: |
| 1893 |
|
| 1894 |
>>> a = 10_000_000 |
| 1895 |
>>> b = 5_000_000 |
| 1896 |
>>> c = b + 5_000_000 |
| 1897 |
>>> a is c |
| 1898 |
False |
| 1899 |
|
| 1900 |
>>> a = 'Python' |
| 1901 |
>>> b = 'Py' |
| 1902 |
>>> c = b + 'thon' |
| 1903 |
>>> a is c |
| 1904 |
False |
| 1905 |
|
| 1906 |
Likewise, new instances of mutable containers are never identical:: |
| 1907 |
|
| 1908 |
>>> a = [] |
| 1909 |
>>> b = [] |
| 1910 |
>>> a is b |
| 1911 |
False |
| 1912 |
|
| 1913 |
In the standard library code, you will see several common patterns for |
| 1914 |
correctly using identity tests: |
| 1915 |
|
| 1916 |
1) As recommended by :pep:`8`, an identity test is the preferred way to check |
| 1917 |
for ``None``. This reads like plain English in code and avoids confusion |
| 1918 |
with other objects that may have boolean values that evaluate to false. |
| 1919 |
|
| 1920 |
2) Detecting optional arguments can be tricky when ``None`` is a valid input |
| 1921 |
value. In those situations, you can create a singleton sentinel object |
| 1922 |
guaranteed to be distinct from other objects. For example, here is how |
| 1923 |
to implement a method that behaves like :meth:`dict.pop`: |
| 1924 |
|
| 1925 |
.. code-block:: python |
| 1926 |
|
| 1927 |
_sentinel = object() |
| 1928 |
|
| 1929 |
def pop(self, key, default=_sentinel): |
| 1930 |
if key in self: |
| 1931 |
value = self[key] |
| 1932 |
del self[key] |
| 1933 |
return value |
| 1934 |
if default is _sentinel: |
| 1935 |
raise KeyError(key) |
| 1936 |
return default |
| 1937 |
|
| 1938 |
3) Container implementations sometimes need to augment equality tests with |
| 1939 |
identity tests. This prevents the code from being confused by objects |
| 1940 |
such as ``float('NaN')`` that are not equal to themselves. |
| 1941 |
|
| 1942 |
For example, here is the implementation of |
| 1943 |
:meth:`!collections.abc.Sequence.__contains__`:: |
| 1944 |
|
| 1945 |
def __contains__(self, value): |
| 1946 |
for v in self: |
| 1947 |
if v is value or v == value: |
| 1948 |
return True |
| 1949 |
return False |
| 1950 |
|
| 1951 |
|
| 1952 |
How can a subclass control what data is stored in an immutable instance? |
| 1953 |
------------------------------------------------------------------------ |
| 1954 |
|
| 1955 |
When subclassing an immutable type, override the :meth:`~object.__new__` method |
| 1956 |
instead of the :meth:`~object.__init__` method. The latter only runs *after* an |
| 1957 |
instance is created, which is too late to alter data in an immutable |
| 1958 |
instance. |
| 1959 |
|
| 1960 |
All of these immutable classes have a different signature than their |
| 1961 |
parent class: |
| 1962 |
|
| 1963 |
.. testcode:: |
| 1964 |
|
| 1965 |
import datetime as dt |
| 1966 |
|
| 1967 |
class FirstOfMonthDate(dt.date): |
| 1968 |
"Always choose the first day of the month" |
| 1969 |
def __new__(cls, year, month, day): |
| 1970 |
return super().__new__(cls, year, month, 1) |
| 1971 |
|
| 1972 |
class NamedInt(int): |
| 1973 |
"Allow text names for some numbers" |
| 1974 |
xlat = {'zero': 0, 'one': 1, 'ten': 10} |
| 1975 |
def __new__(cls, value): |
| 1976 |
value = cls.xlat.get(value, value) |
| 1977 |
return super().__new__(cls, value) |
| 1978 |
|
| 1979 |
class TitleStr(str): |
| 1980 |
"Convert str to name suitable for a URL path" |
| 1981 |
def __new__(cls, s): |
| 1982 |
s = s.lower().replace(' ', '-') |
| 1983 |
s = ''.join([c for c in s if c.isalnum() or c == '-']) |
| 1984 |
return super().__new__(cls, s) |
| 1985 |
|
| 1986 |
The classes can be used like this: |
| 1987 |
|
| 1988 |
.. doctest:: |
| 1989 |
|
| 1990 |
>>> FirstOfMonthDate(2012, 2, 14) |
| 1991 |
FirstOfMonthDate(2012, 2, 1) |
| 1992 |
>>> NamedInt('ten') |
| 1993 |
10 |
| 1994 |
>>> NamedInt(20) |
| 1995 |
20 |
| 1996 |
>>> TitleStr('Blog: Why Python Rocks') |
| 1997 |
'blog-why-python-rocks' |
| 1998 |
|
| 1999 |
|
| 2000 |
.. _faq-cache-method-calls: |
| 2001 |
|
| 2002 |
How do I cache method calls? |
| 2003 |
---------------------------- |
| 2004 |
|
| 2005 |
The two principal tools for caching methods are |
| 2006 |
:func:`functools.cached_property` and :func:`functools.lru_cache`. The |
| 2007 |
former stores results at the instance level and the latter at the class |
| 2008 |
level. |
| 2009 |
|
| 2010 |
The ``cached_property`` approach only works with methods that do not take |
| 2011 |
any arguments. It does not create a reference to the instance. The |
| 2012 |
cached method result will be kept only as long as the instance is alive. |
| 2013 |
|
| 2014 |
The advantage is that when an instance is no longer used, the cached |
| 2015 |
method result will be released right away. The disadvantage is that if |
| 2016 |
instances accumulate, so too will the accumulated method results. They |
| 2017 |
can grow without bound. |
| 2018 |
|
| 2019 |
The ``lru_cache`` approach works with methods that have :term:`hashable` |
| 2020 |
arguments. It creates a reference to the instance unless special |
| 2021 |
efforts are made to pass in weak references. |
| 2022 |
|
| 2023 |
The advantage of the least recently used algorithm is that the cache is |
| 2024 |
bounded by the specified *maxsize*. The disadvantage is that instances |
| 2025 |
are kept alive until they age out of the cache or until the cache is |
| 2026 |
cleared. |
| 2027 |
|
| 2028 |
This example shows the various techniques:: |
| 2029 |
|
| 2030 |
class Weather: |
| 2031 |
"Lookup weather information on a government website" |
| 2032 |
|
| 2033 |
def __init__(self, station_id): |
| 2034 |
self._station_id = station_id |
| 2035 |
# The _station_id is private and immutable |
| 2036 |
|
| 2037 |
def current_temperature(self): |
| 2038 |
"Latest hourly observation" |
| 2039 |
# Do not cache this because old results |
| 2040 |
# can be out of date. |
| 2041 |
|
| 2042 |
@cached_property |
| 2043 |
def location(self): |
| 2044 |
"Return the longitude/latitude coordinates of the station" |
| 2045 |
# Result only depends on the station_id |
| 2046 |
|
| 2047 |
@lru_cache(maxsize=20) |
| 2048 |
def historic_rainfall(self, date, units='mm'): |
| 2049 |
"Rainfall on a given date" |
| 2050 |
# Depends on the station_id, date, and units. |
| 2051 |
|
| 2052 |
The above example assumes that the *station_id* never changes. If the |
| 2053 |
relevant instance attributes are mutable, the ``cached_property`` approach |
| 2054 |
can't be made to work because it cannot detect changes to the |
| 2055 |
attributes. |
| 2056 |
|
| 2057 |
To make the ``lru_cache`` approach work when the *station_id* is mutable, |
| 2058 |
the class needs to define the :meth:`~object.__eq__` and :meth:`~object.__hash__` |
| 2059 |
methods so that the cache can detect relevant attribute updates:: |
| 2060 |
|
| 2061 |
class Weather: |
| 2062 |
"Example with a mutable station identifier" |
| 2063 |
|
| 2064 |
def __init__(self, station_id): |
| 2065 |
self.station_id = station_id |
| 2066 |
|
| 2067 |
def change_station(self, station_id): |
| 2068 |
self.station_id = station_id |
| 2069 |
|
| 2070 |
def __eq__(self, other): |
| 2071 |
return self.station_id == other.station_id |
| 2072 |
|
| 2073 |
def __hash__(self): |
| 2074 |
return hash(self.station_id) |
| 2075 |
|
| 2076 |
@lru_cache(maxsize=20) |
| 2077 |
def historic_rainfall(self, date, units='cm'): |
| 2078 |
'Rainfall on a given date' |
| 2079 |
# Depends on the station_id, date, and units. |
| 2080 |
|
| 2081 |
|
| 2082 |
Modules |
| 2083 |
======= |
| 2084 |
|
| 2085 |
How do I create a .pyc file? |
| 2086 |
---------------------------- |
| 2087 |
|
| 2088 |
When a module is imported for the first time (or when the source file has |
| 2089 |
changed since the current compiled file was created) a ``.pyc`` file containing |
| 2090 |
the compiled code should be created in a ``__pycache__`` subdirectory of the |
| 2091 |
directory containing the ``.py`` file. The ``.pyc`` file will have a |
| 2092 |
filename that starts with the same name as the ``.py`` file, and ends with |
| 2093 |
``.pyc``, with a middle component that depends on the particular ``python`` |
| 2094 |
binary that created it. (See :pep:`3147` for details.) |
| 2095 |
|
| 2096 |
One reason that a ``.pyc`` file may not be created is a permissions problem |
| 2097 |
with the directory containing the source file, meaning that the ``__pycache__`` |
| 2098 |
subdirectory cannot be created. This can happen, for example, if you develop as |
| 2099 |
one user but run as another, such as if you are testing with a web server. |
| 2100 |
|
| 2101 |
Unless the :envvar:`PYTHONDONTWRITEBYTECODE` environment variable is set, |
| 2102 |
creation of a .pyc file is automatic if you're importing a module and Python |
| 2103 |
has the ability (permissions, free space, and so on) to create a ``__pycache__`` |
| 2104 |
subdirectory and write the compiled module to that subdirectory. |
| 2105 |
|
| 2106 |
Running Python on a top-level script is not considered an import and no |
| 2107 |
``.pyc`` will be created. For example, if you have a top-level module |
| 2108 |
``foo.py`` that imports another module ``xyz.py``, when you run ``foo`` (by |
| 2109 |
typing ``python foo.py`` as a shell command), a ``.pyc`` will be created for |
| 2110 |
``xyz`` because ``xyz`` is imported, but no ``.pyc`` file will be created for |
| 2111 |
``foo`` since ``foo.py`` isn't being imported. |
| 2112 |
|
| 2113 |
If you need to create a ``.pyc`` file for ``foo`` -- that is, to create a |
| 2114 |
``.pyc`` file for a module that is not imported -- you can, using the |
| 2115 |
:mod:`py_compile` and :mod:`compileall` modules. |
| 2116 |
|
| 2117 |
The :mod:`py_compile` module can manually compile any module. One way is to use |
| 2118 |
the ``compile()`` function in that module interactively:: |
| 2119 |
|
| 2120 |
>>> import py_compile |
| 2121 |
>>> py_compile.compile('foo.py') # doctest: +SKIP |
| 2122 |
|
| 2123 |
This will write the ``.pyc`` to a ``__pycache__`` subdirectory in the same |
| 2124 |
location as ``foo.py`` (or you can override that with the optional parameter |
| 2125 |
*cfile*). |
| 2126 |
|
| 2127 |
You can also automatically compile all files in a directory or directories using |
| 2128 |
the :mod:`compileall` module. You can do it from the shell prompt by running |
| 2129 |
``compileall.py`` and providing the path of a directory containing Python files |
| 2130 |
to compile:: |
| 2131 |
|
| 2132 |
python -m compileall . |
| 2133 |
|
| 2134 |
|
| 2135 |
How do I find the current module name? |
| 2136 |
-------------------------------------- |
| 2137 |
|
| 2138 |
A module can find out its own module name by looking at the predefined global |
| 2139 |
variable ``__name__``. If this has the value ``'__main__'``, the program is |
| 2140 |
running as a script. Many modules that are usually used by importing them also |
| 2141 |
provide a command-line interface or a self-test, and only execute this code |
| 2142 |
after checking ``__name__``:: |
| 2143 |
|
| 2144 |
def main(): |
| 2145 |
print('Running test...') |
| 2146 |
... |
| 2147 |
|
| 2148 |
if __name__ == '__main__': |
| 2149 |
main() |
| 2150 |
|
| 2151 |
|
| 2152 |
How can I have modules that mutually import each other? |
| 2153 |
------------------------------------------------------- |
| 2154 |
|
| 2155 |
Suppose you have the following modules: |
| 2156 |
|
| 2157 |
:file:`foo.py`:: |
| 2158 |
|
| 2159 |
from bar import bar_var |
| 2160 |
foo_var = 1 |
| 2161 |
|
| 2162 |
:file:`bar.py`:: |
| 2163 |
|
| 2164 |
from foo import foo_var |
| 2165 |
bar_var = 2 |
| 2166 |
|
| 2167 |
The problem is that the interpreter will perform the following steps: |
| 2168 |
|
| 2169 |
* main imports ``foo`` |
| 2170 |
* Empty globals for ``foo`` are created |
| 2171 |
* ``foo`` is compiled and starts executing |
| 2172 |
* ``foo`` imports ``bar`` |
| 2173 |
* Empty globals for ``bar`` are created |
| 2174 |
* ``bar`` is compiled and starts executing |
| 2175 |
* ``bar`` imports ``foo`` (which is a no-op since there already is a module named ``foo``) |
| 2176 |
* The import mechanism tries to read ``foo_var`` from ``foo`` globals, to set ``bar.foo_var = foo.foo_var`` |
| 2177 |
|
| 2178 |
The last step fails, because Python isn't done with interpreting ``foo`` yet and |
| 2179 |
the global symbol dictionary for ``foo`` is still empty. |
| 2180 |
|
| 2181 |
The same thing happens when you use ``import foo``, and then try to access |
| 2182 |
``foo.foo_var`` in global code. |
| 2183 |
|
| 2184 |
There are (at least) three possible workarounds for this problem. |
| 2185 |
|
| 2186 |
Guido van Rossum recommends avoiding all uses of ``from <module> import ...``, |
| 2187 |
and placing all code inside functions. Initializations of global variables and |
| 2188 |
class variables should use constants or built-in functions only. This means |
| 2189 |
everything from an imported module is referenced as ``<module>.<name>``. |
| 2190 |
|
| 2191 |
Jim Roskind suggests performing steps in the following order in each module: |
| 2192 |
|
| 2193 |
* exports (globals, functions, and classes that don't need imported base |
| 2194 |
classes) |
| 2195 |
* ``import`` statements |
| 2196 |
* active code (including globals that are initialized from imported values). |
| 2197 |
|
| 2198 |
Van Rossum doesn't like this approach much because the imports appear in a |
| 2199 |
strange place, but it does work. |
| 2200 |
|
| 2201 |
Matthias Urlichs recommends restructuring your code so that the recursive import |
| 2202 |
is not necessary in the first place. |
| 2203 |
|
| 2204 |
These solutions are not mutually exclusive. |
| 2205 |
|
| 2206 |
|
| 2207 |
__import__('x.y.z') returns <module 'x'>; how do I get z? |
| 2208 |
--------------------------------------------------------- |
| 2209 |
|
| 2210 |
Consider using the convenience function :func:`~importlib.import_module` from |
| 2211 |
:mod:`importlib` instead:: |
| 2212 |
|
| 2213 |
z = importlib.import_module('x.y.z') |
| 2214 |
|
| 2215 |
|
| 2216 |
When I edit an imported module and reimport it, the changes don't show up. Why does this happen? |
| 2217 |
------------------------------------------------------------------------------------------------- |
| 2218 |
|
| 2219 |
For reasons of efficiency as well as consistency, Python only reads the module |
| 2220 |
file on the first time a module is imported. If it didn't, in a program |
| 2221 |
consisting of many modules where each one imports the same basic module, the |
| 2222 |
basic module would be parsed and re-parsed many times. To force re-reading of a |
| 2223 |
changed module, do this:: |
| 2224 |
|
| 2225 |
import importlib |
| 2226 |
import modname |
| 2227 |
importlib.reload(modname) |
| 2228 |
|
| 2229 |
Warning: this technique is not 100% fool-proof. In particular, modules |
| 2230 |
containing statements like:: |
| 2231 |
|
| 2232 |
from modname import some_objects |
| 2233 |
|
| 2234 |
will continue to work with the old version of the imported objects. If the |
| 2235 |
module contains class definitions, existing class instances will *not* be |
| 2236 |
updated to use the new class definition. This can result in the following |
| 2237 |
paradoxical behaviour:: |
| 2238 |
|
| 2239 |
>>> import importlib |
| 2240 |
>>> import cls |
| 2241 |
>>> c = cls.C() # Create an instance of C |
| 2242 |
>>> importlib.reload(cls) |
| 2243 |
<module 'cls' from 'cls.py'> |
| 2244 |
>>> isinstance(c, cls.C) # isinstance is false?!? |
| 2245 |
False |
| 2246 |
|
| 2247 |
The nature of the problem is made clear if you print out the "identity" of the |
| 2248 |
class objects:: |
| 2249 |
|
| 2250 |
>>> hex(id(c.__class__)) |
| 2251 |
'0x7352a0' |
| 2252 |
>>> hex(id(cls.C)) |
| 2253 |
'0x4198d0' |