Debian Codegraph @

Doc/faq/programming.rst

raw
1 :tocdepth: 2
2
3 ===============
4 Programming FAQ
5 ===============
6
7 .. only:: html
8
9 .. contents::
10
11 General questions
12 =================
13
14 Is there a source code-level debugger with breakpoints and single-stepping?
15 ---------------------------------------------------------------------------
16
17 Yes.
18
19 Several debuggers for Python are described below, and the built-in function
20 :func:`breakpoint` allows you to drop into any of them.
21
22 The pdb module is a simple but adequate console-mode debugger for Python. It is
23 part of the standard Python library, and is :mod:`documented in the Library
24 Reference Manual <pdb>`. You can also write your own debugger by using the code
25 for pdb as an example.
26
27 The IDLE interactive development environment, which is part of the standard
28 Python distribution (normally available as :mod:`idlelib`),
29 includes a graphical debugger.
30
31 PythonWin is a Python IDE that includes a GUI debugger based on pdb. The
32 PythonWin debugger colors breakpoints and has quite a few cool features such as
33 debugging non-PythonWin programs. PythonWin is available as part of
34 `pywin32 <https://github.com/mhammond/pywin32>`_ project and
35 as a part of the
36 `ActivePython <https://www.activestate.com/products/python/>`_ distribution.
37
38 `Eric <https://eric-ide.python-projects.org/>`_ is an IDE built on PyQt
39 and the Scintilla editing component.
40
41 `trepan3k <https://github.com/rocky/python3-trepan/>`_ is a gdb-like debugger.
42
43 `Visual Studio Code <https://code.visualstudio.com/>`_ is an IDE with debugging
44 tools that integrates with version-control software.
45
46 There are a number of commercial Python IDEs that include graphical debuggers.
47 They include:
48
49 * `Wing IDE <https://wingware.com/>`_
50 * `PyCharm <https://www.jetbrains.com/pycharm/>`_
51
52
53 Are there tools to help find bugs or perform static analysis?
54 -------------------------------------------------------------
55
56 Yes.
57
58 `Ruff <https://docs.astral.sh/ruff/>`__,
59 `Pylint <https://pylint.readthedocs.io/>`__ and
60 `Pyflakes <https://github.com/PyCQA/pyflakes>`__ do basic checking that will
61 help you catch bugs sooner.
62
63 Static type checkers such as `mypy <https://mypy-lang.org/>`__,
64 `ty <https://docs.astral.sh/ty/>`__,
65 `Pyrefly <https://pyrefly.org/>`__, and
66 `pytype <https://github.com/google/pytype>`__ can check type hints in Python
67 source code.
68
69
70 .. _faq-create-standalone-binary:
71
72 How can I create a stand-alone binary from a Python script?
73 -----------------------------------------------------------
74
75 You don't need the ability to compile Python to C code if all you want is a
76 stand-alone program that users can download and run without having to install
77 the Python distribution first. There are a number of tools that determine the
78 set of modules required by a program and bind these modules together with a
79 Python binary to produce a single executable.
80
81 One is to use the freeze tool, which is included in the Python source tree as
82 :source:`Tools/freeze`.
83 It converts Python byte code to C arrays; with a C compiler you can
84 embed all your modules into a new program, which is then linked with the
85 standard Python modules.
86
87 It works by scanning your source recursively for import statements (in both
88 forms) and looking for the modules in the standard Python path as well as in the
89 source directory (for built-in modules). It then turns the bytecode for modules
90 written in Python into C code (array initializers that can be turned into code
91 objects using the marshal module) and creates a custom-made config file that
92 only contains those built-in modules which are actually used in the program. It
93 then compiles the generated C code and links it with the rest of the Python
94 interpreter to form a self-contained binary which acts exactly like your script.
95
96 The following packages can help with the creation of console and GUI
97 executables:
98
99 * `Nuitka <https://nuitka.net/>`_ (Cross-platform)
100 * `PyInstaller <https://pyinstaller.org/>`_ (Cross-platform)
101 * `PyOxidizer <https://pyoxidizer.readthedocs.io/en/stable/>`_ (Cross-platform)
102 * `cx_Freeze <https://marcelotduarte.github.io/cx_Freeze/>`_ (Cross-platform)
103 * `py2app <https://github.com/ronaldoussoren/py2app>`_ (macOS only)
104 * `py2exe <https://www.py2exe.org/>`_ (Windows only)
105
106
107 Are there coding standards or a style guide for Python programs?
108 ----------------------------------------------------------------
109
110 Yes. The coding style required for standard library modules is documented as
111 :pep:`8`.
112
113
114 Core language
115 =============
116
117 .. _faq-unboundlocalerror:
118
119 Why am I getting an UnboundLocalError when the variable has a value?
120 --------------------------------------------------------------------
121
122 It can be a surprise to get the :exc:`UnboundLocalError` in previously working
123 code when it is modified by adding an assignment statement somewhere in
124 the body of a function.
125
126 This code:
127
128 >>> x = 10
129 >>> def bar():
130 ... print(x)
131 ...
132 >>> bar()
133 10
134
135 works, but this code:
136
137 >>> x = 10
138 >>> def foo():
139 ... print(x)
140 ... x += 1
141
142 results in an :exc:`!UnboundLocalError`:
143
144 >>> foo()
145 Traceback (most recent call last):
146 ...
147 UnboundLocalError: cannot access local variable 'x' where it is not associated with a value
148
149 This is because when you make an assignment to a variable in a scope, that
150 variable becomes local to that scope and shadows any similarly named variable
151 in the outer scope. Since the last statement in foo assigns a new value to
152 ``x``, the compiler recognizes it as a local variable. Consequently when the
153 earlier ``print(x)`` attempts to print the uninitialized local variable and
154 an error results.
155
156 In the example above you can access the outer scope variable by declaring it
157 global:
158
159 >>> x = 10
160 >>> def foobar():
161 ... global x
162 ... print(x)
163 ... x += 1
164 ...
165 >>> foobar()
166 10
167
168 This explicit declaration is required in order to remind you that (unlike the
169 superficially analogous situation with class and instance variables) you are
170 actually modifying the value of the variable in the outer scope:
171
172 >>> print(x)
173 11
174
175 You can do a similar thing in a nested scope using the :keyword:`nonlocal`
176 keyword:
177
178 >>> def foo():
179 ... x = 10
180 ... def bar():
181 ... nonlocal x
182 ... print(x)
183 ... x += 1
184 ... bar()
185 ... print(x)
186 ...
187 >>> foo()
188 10
189 11
190
191
192 What are the rules for local and global variables in Python?
193 ------------------------------------------------------------
194
195 In Python, variables that are only referenced inside a function are implicitly
196 global. If a variable is assigned a value anywhere within the function's body,
197 it's assumed to be a local unless explicitly declared as global.
198
199 Though a bit surprising at first, a moment's consideration explains this. On
200 one hand, requiring :keyword:`global` for assigned variables provides a bar
201 against unintended side-effects. On the other hand, if ``global`` was required
202 for all global references, you'd be using ``global`` all the time. You'd have
203 to declare as global every reference to a built-in function or to a component of
204 an imported module. This clutter would defeat the usefulness of the ``global``
205 declaration for identifying side-effects.
206
207
208 Why do lambdas defined in a loop with different values all return the same result?
209 ----------------------------------------------------------------------------------
210
211 Assume you use a for loop to define a few different lambdas (or even plain
212 functions), for example::
213
214 >>> squares = []
215 >>> for x in range(5):
216 ... squares.append(lambda: x**2)
217
218 This gives you a list that contains 5 lambdas that calculate ``x**2``. You
219 might expect that, when called, they would return, respectively, ``0``, ``1``,
220 ``4``, ``9``, and ``16``. However, when you actually try you will see that
221 they all return ``16``::
222
223 >>> squares[2]()
224 16
225 >>> squares[4]()
226 16
227
228 This happens because ``x`` is not local to the lambdas, but is defined in
229 the outer scope, and it is accessed when the lambda is called --- not when it
230 is defined. At the end of the loop, the value of ``x`` is ``4``, so all the
231 functions now return ``4**2``, that is ``16``. You can also verify this by
232 changing the value of ``x`` and see how the results of the lambdas change::
233
234 >>> x = 8
235 >>> squares[2]()
236 64
237
238 In order to avoid this, you need to save the values in variables local to the
239 lambdas, so that they don't rely on the value of the global ``x``::
240
241 >>> squares = []
242 >>> for x in range(5):
243 ... squares.append(lambda n=x: n**2)
244
245 Here, ``n=x`` creates a new variable ``n`` local to the lambda and computed
246 when the lambda is defined so that it has the same value that ``x`` had at
247 that point in the loop. This means that the value of ``n`` will be ``0``
248 in the first lambda, ``1`` in the second, ``2`` in the third, and so on.
249 Therefore each lambda will now return the correct result::
250
251 >>> squares[2]()
252 4
253 >>> squares[4]()
254 16
255
256 Note that this behaviour is not peculiar to lambdas, but applies to regular
257 functions too.
258
259
260 How do I share global variables across modules?
261 ------------------------------------------------
262
263 The canonical way to share information across modules within a single program is
264 to create a special module (often called config or cfg). Just import the config
265 module in all modules of your application; the module then becomes available as
266 a global name. Because there is only one instance of each module, any changes
267 made to the module object get reflected everywhere. For example:
268
269 config.py::
270
271 x = 0 # Default value of the 'x' configuration setting
272
273 mod.py::
274
275 import config
276 config.x = 1
277
278 main.py::
279
280 import config
281 import mod
282 print(config.x)
283
284 Note that using a module is also the basis for implementing the singleton design
285 pattern, for the same reason.
286
287
288 What are the "best practices" for using import in a module?
289 -----------------------------------------------------------
290
291 In general, don't use ``from modulename import *``. Doing so clutters the
292 importer's namespace, and makes it much harder for linters to detect undefined
293 names.
294
295 Import modules at the top of a file. Doing so makes it clear what other modules
296 your code requires and avoids questions of whether the module name is in scope.
297 Using one import per line makes it easy to add and delete module imports, but
298 using multiple imports per line uses less screen space.
299
300 It's good practice if you import modules in the following order:
301
302 1. standard library modules -- such as :mod:`sys`, :mod:`os`, :mod:`argparse`, :mod:`re`
303 2. third-party library modules (anything installed in Python's site-packages
304 directory) -- such as :pypi:`dateutil`, :pypi:`requests`, :pypi:`tzdata`
305 3. locally developed modules
306
307 It is sometimes necessary to move imports to a function or class to avoid
308 problems with circular imports. Gordon McMillan says:
309
310 Circular imports are fine where both modules use the "import <module>" form
311 of import. They fail when the 2nd module wants to grab a name out of the
312 first ("from module import name") and the import is at the top level. That's
313 because names in the 1st are not yet available, because the first module is
314 busy importing the 2nd.
315
316 In this case, if the second module is only used in one function, then the import
317 can easily be moved into that function. By the time the import is called, the
318 first module will have finished initializing, and the second module can do its
319 import.
320
321 It may also be necessary to move imports out of the top level of code if some of
322 the modules are platform-specific. In that case, it may not even be possible to
323 import all of the modules at the top of the file. In this case, importing the
324 correct modules in the corresponding platform-specific code is a good option.
325
326 Only move imports into a local scope, such as inside a function definition, if
327 it's necessary to solve a problem such as avoiding a circular import or are
328 trying to reduce the initialization time of a module. This technique is
329 especially helpful if many of the imports are unnecessary depending on how the
330 program executes. You may also want to move imports into a function if the
331 modules are only ever used in that function. Note that loading a module the
332 first time may be expensive because of the one time initialization of the
333 module, but loading a module multiple times is virtually free, costing only a
334 couple of dictionary lookups. Even if the module name has gone out of scope,
335 the module is probably available in :data:`sys.modules`.
336
337
338 Why are default values shared between objects?
339 ----------------------------------------------
340
341 This type of bug commonly bites neophyte programmers. Consider this function::
342
343 def foo(mydict={}): # Danger: shared reference to one dict for all calls
344 ... compute something ...
345 mydict[key] = value
346 return mydict
347
348 The first time you call this function, ``mydict`` contains a single item. The
349 second time, ``mydict`` contains two items because when ``foo()`` begins
350 executing, ``mydict`` starts out with an item already in it.
351
352 It is often expected that a function call creates new objects for default
353 values. This is not what happens. Default values are created exactly once, when
354 the function is defined. If that object is changed, like the dictionary in this
355 example, subsequent calls to the function will refer to this changed object.
356
357 By definition, immutable objects such as numbers, strings, tuples, and ``None``,
358 are safe from change. Changes to mutable objects such as dictionaries, lists,
359 and class instances can lead to confusion.
360
361 Because of this feature, it is good programming practice to not use mutable
362 objects as default values. Instead, use ``None`` as the default value and
363 inside the function, check if the parameter is ``None`` and create a new
364 list/dictionary/whatever if it is. For example, don't write::
365
366 def foo(mydict={}):
367 ...
368
369 but::
370
371 def foo(mydict=None):
372 if mydict is None:
373 mydict = {} # create a new dict for local namespace
374
375 This feature can be useful. When you have a function that's time-consuming to
376 compute, a common technique is to cache the parameters and the resulting value
377 of each call to the function, and return the cached value if the same value is
378 requested again. This is called "memoizing", and can be implemented like this::
379
380 # Callers can only provide two parameters and optionally pass _cache by keyword
381 def expensive(arg1, arg2, *, _cache={}):
382 if (arg1, arg2) in _cache:
383 return _cache[(arg1, arg2)]
384
385 # Calculate the value
386 result = ... expensive computation ...
387 _cache[(arg1, arg2)] = result # Store result in the cache
388 return result
389
390 You could use a global variable containing a dictionary instead of the default
391 value; it's a matter of taste.
392
393
394 How can I pass optional or keyword parameters from one function to another?
395 ---------------------------------------------------------------------------
396
397 Collect the arguments using the ``*`` and ``**`` specifiers in the function's
398 parameter list; this gives you the positional arguments as a tuple and the
399 keyword arguments as a dictionary. You can then pass these arguments when
400 calling another function by using ``*`` and ``**``::
401
402 def f(x, *args, **kwargs):
403 ...
404 kwargs['width'] = '14.3c'
405 ...
406 g(x, *args, **kwargs)
407
408
409 .. index::
410 single: argument; difference from parameter
411 single: parameter; difference from argument
412
413 .. _faq-argument-vs-parameter:
414
415 What is the difference between arguments and parameters?
416 --------------------------------------------------------
417
418 :term:`Parameters <parameter>` are defined by the names that appear in a
419 function definition, whereas :term:`arguments <argument>` are the values
420 actually passed to a function when calling it. Parameters define what
421 :term:`kind of arguments <parameter>` a function can accept. For
422 example, given the function definition::
423
424 def func(foo, bar=None, **kwargs):
425 pass
426
427 *foo*, *bar* and *kwargs* are parameters of ``func``. However, when calling
428 ``func``, for example::
429
430 func(42, bar=314, extra=somevar)
431
432 the values ``42``, ``314``, and ``somevar`` are arguments.
433
434
435 Why did changing list 'y' also change list 'x'?
436 ------------------------------------------------
437
438 If you wrote code like::
439
440 >>> x = []
441 >>> y = x
442 >>> y.append(10)
443 >>> y
444 [10]
445 >>> x
446 [10]
447
448 you might be wondering why appending an element to ``y`` changed ``x`` too.
449
450 There are two factors that produce this result:
451
452 1) Variables are simply names that refer to objects. Doing ``y = x`` doesn't
453 create a copy of the list -- it creates a new variable ``y`` that refers to
454 the same object ``x`` refers to. This means that there is only one object
455 (the list), and both ``x`` and ``y`` refer to it.
456 2) Lists are :term:`mutable`, which means that you can change their content.
457
458 After the call to :meth:`~sequence.append`, the content of the mutable object has
459 changed from ``[]`` to ``[10]``. Since both the variables refer to the same
460 object, using either name accesses the modified value ``[10]``.
461
462 If we instead assign an immutable object to ``x``::
463
464 >>> x = 5 # ints are immutable
465 >>> y = x
466 >>> x = x + 1 # 5 can't be mutated, we are creating a new object here
467 >>> x
468 6
469 >>> y
470 5
471
472 we can see that in this case ``x`` and ``y`` are not equal anymore. This is
473 because integers are :term:`immutable`, and when we do ``x = x + 1`` we are not
474 mutating the int ``5`` by incrementing its value; instead, we are creating a
475 new object (the int ``6``) and assigning it to ``x`` (that is, changing which
476 object ``x`` refers to). After this assignment we have two objects (the ints
477 ``6`` and ``5``) and two variables that refer to them (``x`` now refers to
478 ``6`` but ``y`` still refers to ``5``).
479
480 Some operations (for example ``y.append(10)`` and ``y.sort()``) mutate the
481 object, whereas superficially similar operations (for example ``y = y + [10]``
482 and :func:`sorted(y) <sorted>`) create a new object. In general in Python (and in all cases
483 in the standard library) a method that mutates an object will return ``None``
484 to help avoid getting the two types of operations confused. So if you
485 mistakenly write ``y.sort()`` thinking it will give you a sorted copy of ``y``,
486 you'll instead end up with ``None``, which will likely cause your program to
487 generate an easily diagnosed error.
488
489 However, there is one class of operations where the same operation sometimes
490 has different behaviors with different types: the augmented assignment
491 operators. For example, ``+=`` mutates lists but not tuples or ints (``a_list
492 += [1, 2, 3]`` is equivalent to ``a_list.extend([1, 2, 3])`` and mutates
493 ``a_list``, whereas ``some_tuple += (1, 2, 3)`` and ``some_int += 1`` create
494 new objects).
495
496 In other words:
497
498 * If we have a mutable object (such as :class:`list`, :class:`dict`, :class:`set`),
499 we can use some specific operations to mutate it and all the variables
500 that refer to it will see the change.
501 * If we have an immutable object (such as :class:`str`, :class:`int`, :class:`tuple`),
502 all the variables that refer to it will always see the same value,
503 but operations that transform that value into a new value always return a new
504 object.
505
506 If you want to know if two variables refer to the same object or not, you can
507 use the :keyword:`is` operator, or the built-in function :func:`id`.
508
509
510 How do I write a function with output parameters (call by reference)?
511 ---------------------------------------------------------------------
512
513 Remember that arguments are passed by assignment in Python. Since assignment
514 just creates references to objects, there's no alias between an argument name in
515 the caller and callee, and consequently no call-by-reference. You can achieve the
516 desired effect in a number of ways.
517
518 1) By returning a tuple of the results::
519
520 >>> def func1(a, b):
521 ... a = 'new-value' # a and b are local names
522 ... b = b + 1 # assigned to new objects
523 ... return a, b # return new values
524 ...
525 >>> x, y = 'old-value', 99
526 >>> func1(x, y)
527 ('new-value', 100)
528
529 This is almost always the clearest solution.
530
531 2) By using global variables. This isn't thread-safe, and is not recommended.
532
533 3) By passing a mutable (changeable in-place) object::
534
535 >>> def func2(a):
536 ... a[0] = 'new-value' # 'a' references a mutable list
537 ... a[1] = a[1] + 1 # changes a shared object
538 ...
539 >>> args = ['old-value', 99]
540 >>> func2(args)
541 >>> args
542 ['new-value', 100]
543
544 4) By passing in a dictionary that gets mutated::
545
546 >>> def func3(args):
547 ... args['a'] = 'new-value' # args is a mutable dictionary
548 ... args['b'] = args['b'] + 1 # change it in-place
549 ...
550 >>> args = {'a': 'old-value', 'b': 99}
551 >>> func3(args)
552 >>> args
553 {'a': 'new-value', 'b': 100}
554
555 5) Or bundle up values in a class instance::
556
557 >>> class Namespace:
558 ... def __init__(self, /, **args):
559 ... for key, value in args.items():
560 ... setattr(self, key, value)
561 ...
562 >>> def func4(args):
563 ... args.a = 'new-value' # args is a mutable Namespace
564 ... args.b = args.b + 1 # change object in-place
565 ...
566 >>> args = Namespace(a='old-value', b=99)
567 >>> func4(args)
568 >>> vars(args)
569 {'a': 'new-value', 'b': 100}
570
571
572 There's almost never a good reason to get this complicated.
573
574 Your best choice is to return a tuple containing the multiple results.
575
576
577 How do you make a higher order function in Python?
578 --------------------------------------------------
579
580 You have two choices: you can use nested scopes or you can use callable objects.
581 For example, suppose you wanted to define ``linear(a,b)`` which returns a
582 function ``f(x)`` that computes the value ``a*x+b``. Using nested scopes::
583
584 def linear(a, b):
585 def result(x):
586 return a * x + b
587 return result
588
589 Or using a callable object::
590
591 class linear:
592
593 def __init__(self, a, b):
594 self.a, self.b = a, b
595
596 def __call__(self, x):
597 return self.a * x + self.b
598
599 In both cases, ::
600
601 taxes = linear(0.3, 2)
602
603 gives a callable object where ``taxes(10e6) == 0.3 * 10e6 + 2``.
604
605 The callable object approach has the disadvantage that it is a bit slower and
606 results in slightly longer code. However, note that a collection of callables
607 can share their signature via inheritance::
608
609 class exponential(linear):
610 # __init__ inherited
611 def __call__(self, x):
612 return self.a * (x ** self.b)
613
614 Object can encapsulate state for several methods::
615
616 class counter:
617
618 value = 0
619
620 def set(self, x):
621 self.value = x
622
623 def up(self):
624 self.value = self.value + 1
625
626 def down(self):
627 self.value = self.value - 1
628
629 count = counter()
630 inc, dec, reset = count.up, count.down, count.set
631
632 Here ``inc()``, ``dec()`` and ``reset()`` act like functions which share the
633 same counting variable.
634
635
636 How do I copy an object in Python?
637 ----------------------------------
638
639 In general, try :func:`copy.copy` or :func:`copy.deepcopy` for the general case.
640 Not all objects can be copied, but most can.
641
642 Some objects can be copied more easily. Dictionaries have a :meth:`~dict.copy`
643 method::
644
645 newdict = olddict.copy()
646
647 Sequences can be copied by slicing::
648
649 new_l = l[:]
650
651
652 How can I find the methods or attributes of an object?
653 ------------------------------------------------------
654
655 For an instance ``x`` of a user-defined class, :func:`dir(x) <dir>` returns an alphabetized
656 list of the names containing the instance attributes and methods and attributes
657 defined by its class.
658
659
660 How can my code discover the name of an object?
661 -----------------------------------------------
662
663 Generally speaking, it can't, because objects don't really have names.
664 Essentially, assignment always binds a name to a value; the same is true of
665 ``def`` and ``class`` statements, but in that case the value is a
666 callable. Consider the following code::
667
668 >>> class A:
669 ... pass
670 ...
671 >>> B = A
672 >>> a = B()
673 >>> b = a
674 >>> print(b)
675 <__main__.A object at 0x16D07CC>
676 >>> print(a)
677 <__main__.A object at 0x16D07CC>
678
679 Arguably the class has a name: even though it is bound to two names and invoked
680 through the name ``B`` the created instance is still reported as an instance of
681 class ``A``. However, it is impossible to say whether the instance's name is ``a`` or
682 ``b``, since both names are bound to the same value.
683
684 Generally speaking it should not be necessary for your code to "know the names"
685 of particular values. Unless you are deliberately writing introspective
686 programs, this is usually an indication that a change of approach might be
687 beneficial.
688
689 In comp.lang.python, Fredrik Lundh once gave an excellent analogy in answer to
690 this question:
691
692 The same way as you get the name of that cat you found on your porch: the cat
693 (object) itself cannot tell you its name, and it doesn't really care -- so
694 the only way to find out what it's called is to ask all your neighbours
695 (namespaces) if it's their cat (object)...
696
697 ....and don't be surprised if you'll find that it's known by many names, or
698 no name at all!
699
700
701 What's up with the comma operator's precedence?
702 -----------------------------------------------
703
704 Comma is not an operator in Python. Consider this session::
705
706 >>> "a" in "b", "a"
707 (False, 'a')
708
709 Since the comma is not an operator, but a separator between expressions the
710 above is evaluated as if you had entered::
711
712 ("a" in "b"), "a"
713
714 not::
715
716 "a" in ("b", "a")
717
718 The same is true of the various assignment operators (``=``, ``+=``, and so on).
719 They are not truly operators but syntactic delimiters in assignment statements.
720
721
722 Is there an equivalent of C's "?:" ternary operator?
723 ----------------------------------------------------
724
725 Yes, there is. The syntax is as follows::
726
727 [on_true] if [expression] else [on_false]
728
729 x, y = 50, 25
730 small = x if x < y else y
731
732 Before this syntax was introduced in Python 2.5, a common idiom was to use
733 logical operators::
734
735 [expression] and [on_true] or [on_false]
736
737 However, this idiom is unsafe, as it can give wrong results when *on_true*
738 has a false boolean value. Therefore, it is always better to use
739 the ``... if ... else ...`` form.
740
741
742 Is it possible to write obfuscated one-liners in Python?
743 --------------------------------------------------------
744
745 Yes. Usually this is done by nesting :keyword:`lambda` within
746 :keyword:`!lambda`. See the following three examples, slightly adapted from Ulf Bartelt::
747
748 from functools import reduce
749
750 # Primes < 1000
751 print(list(filter(None,map(lambda y:y*reduce(lambda x,y:x*y!=0,
752 map(lambda x,y=y:y%x,range(2,int(pow(y,0.5)+1))),1),range(2,1000)))))
753
754 # First 10 Fibonacci numbers
755 print(list(map(lambda x,f=lambda x,f:(f(x-1,f)+f(x-2,f)) if x>1 else 1:
756 f(x,f), range(10))))
757
758 # Mandelbrot set
759 print((lambda Ru,Ro,Iu,Io,IM,Sx,Sy:reduce(lambda x,y:x+'\n'+y,map(lambda y,
760 Iu=Iu,Io=Io,Ru=Ru,Ro=Ro,Sy=Sy,L=lambda yc,Iu=Iu,Io=Io,Ru=Ru,Ro=Ro,i=IM,
761 Sx=Sx,Sy=Sy:reduce(lambda x,y:x+y,map(lambda x,xc=Ru,yc=yc,Ru=Ru,Ro=Ro,
762 i=i,Sx=Sx,F=lambda xc,yc,x,y,k,f=lambda xc,yc,x,y,k,f:(k<=0)or (x*x+y*y
763 >=4.0) or 1+f(xc,yc,x*x-y*y+xc,2.0*x*y+yc,k-1,f):f(xc,yc,x,y,k,f):chr(
764 64+F(Ru+x*(Ro-Ru)/Sx,yc,0,0,i)),range(Sx))):L(Iu+y*(Io-Iu)/Sy),range(Sy
765 ))))(-2.1, 0.7, -1.2, 1.2, 30, 80, 24))
766 # \___ ___/ \___ ___/ | | |__ lines on screen
767 # V V | |______ columns on screen
768 # | | |__________ maximum of "iterations"
769 # | |_________________ range on y axis
770 # |____________________________ range on x axis
771
772 Don't try this at home, kids!
773
774
775 .. _faq-positional-only-arguments:
776
777 What does the slash(/) in the parameter list of a function mean?
778 ----------------------------------------------------------------
779
780 A slash in the argument list of a function denotes that the parameters prior to
781 it are positional-only. Positional-only parameters are the ones without an
782 externally usable name. Upon calling a function that accepts positional-only
783 parameters, arguments are mapped to parameters based solely on their position.
784 For example, :func:`divmod` is a function that accepts positional-only
785 parameters. Its documentation looks like this::
786
787 >>> help(divmod)
788 Help on built-in function divmod in module builtins:
789
790 divmod(x, y, /)
791 Return the tuple (x//y, x%y). Invariant: div*y + mod == x.
792
793 The slash at the end of the parameter list means that both parameters are
794 positional-only. Thus, calling :func:`divmod` with keyword arguments would lead
795 to an error::
796
797 >>> divmod(x=3, y=4)
798 Traceback (most recent call last):
799 File "<stdin>", line 1, in <module>
800 TypeError: divmod() takes no keyword arguments
801
802
803 Numbers and strings
804 ===================
805
806 How do I specify hexadecimal and octal integers?
807 ------------------------------------------------
808
809 To specify an octal digit, precede the octal value with a zero, and then a lower
810 or uppercase "o". For example, to set the variable "a" to the octal value "10"
811 (8 in decimal), type::
812
813 >>> a = 0o10
814 >>> a
815 8
816
817 Hexadecimal is just as easy. Simply precede the hexadecimal number with a zero,
818 and then a lower or uppercase "x". Hexadecimal digits can be specified in lower
819 or uppercase. For example, in the Python interpreter::
820
821 >>> a = 0xa5
822 >>> a
823 165
824 >>> b = 0XB2
825 >>> b
826 178
827
828
829 Why does -22 // 10 return -3?
830 -----------------------------
831
832 It's primarily driven by the desire that ``i % j`` have the same sign as ``j``.
833 If you want that, and also want::
834
835 i == (i // j) * j + (i % j)
836
837 then integer division has to return the floor. C also requires that identity to
838 hold, and then compilers that truncate ``i // j`` need to make ``i % j`` have
839 the same sign as ``i``.
840
841 There are few real use cases for ``i % j`` when ``j`` is negative. When ``j``
842 is positive, there are many, and in virtually all of them it's more useful for
843 ``i % j`` to be ``>= 0``. If the clock says 10 now, what did it say 200 hours
844 ago? ``-190 % 12 == 2`` is useful; ``-190 % 12 == -10`` is a bug waiting to
845 bite.
846
847
848 How do I get int literal attribute instead of SyntaxError?
849 ----------------------------------------------------------
850
851 Trying to lookup an ``int`` literal attribute in the normal manner gives
852 a :exc:`SyntaxError` because the period is seen as a decimal point::
853
854 >>> 1.__class__
855 File "<stdin>", line 1
856 1.__class__
857 ^
858 SyntaxError: invalid decimal literal
859
860 The solution is to separate the literal from the period
861 with either a space or parentheses.
862
863 >>> 1 .__class__
864 <class 'int'>
865 >>> (1).__class__
866 <class 'int'>
867
868
869 How do I convert a string to a number?
870 --------------------------------------
871
872 For integers, use the built-in :func:`int` type constructor, for example, ``int('144')
873 == 144``. Similarly, :func:`float` converts to a floating-point number,
874 for example, ``float('144') == 144.0``.
875
876 By default, these interpret the number as decimal, so that ``int('0144') ==
877 144`` holds true, and ``int('0x144')`` raises :exc:`ValueError`. ``int(string,
878 base)`` takes the base to convert from as a second optional argument, so ``int(
879 '0x144', 16) == 324``. If the base is specified as 0, the number is interpreted
880 using Python's rules: a leading '0o' indicates octal, and '0x' indicates a hex
881 number.
882
883 Do not use the built-in function :func:`eval` if all you need is to convert
884 strings to numbers. :func:`eval` will be significantly slower and it presents a
885 security risk: someone could pass you a Python expression that might have
886 unwanted side effects. For example, someone could pass
887 ``__import__('os').system("rm -rf $HOME")`` which would erase your home
888 directory.
889
890 :func:`eval` also has the effect of interpreting numbers as Python expressions,
891 so that, for example, ``eval('09')`` gives a syntax error because Python does not allow
892 leading '0' in a decimal number (except '0').
893
894
895 How do I convert a number to a string?
896 --------------------------------------
897
898 For example, to convert the number ``144`` to the string ``'144'``, use the built-in type
899 constructor :func:`str`. If you want a hexadecimal or octal representation, use
900 the built-in functions :func:`hex` or :func:`oct`. For fancy formatting, see
901 the :ref:`f-strings` and :ref:`formatstrings` sections.
902 For example, ``"{:04d}".format(144)`` yields
903 ``'0144'`` and ``"{:.3f}".format(1.0/3.0)`` yields ``'0.333'``.
904
905
906 How do I modify a string in place?
907 ----------------------------------
908
909 You can't, because strings are immutable. In most situations, you should
910 simply construct a new string from the various parts you want to assemble
911 it from. However, if you need an object with the ability to modify in-place
912 Unicode data, try using an :class:`io.StringIO` object or the :mod:`array`
913 module::
914
915 >>> import io
916 >>> s = "Hello, world"
917 >>> sio = io.StringIO(s)
918 >>> sio.getvalue()
919 'Hello, world'
920 >>> sio.seek(7)
921 7
922 >>> sio.write("there!")
923 6
924 >>> sio.getvalue()
925 'Hello, there!'
926
927 >>> import array
928 >>> a = array.array('w', s)
929 >>> print(a)
930 array('w', 'Hello, world')
931 >>> a[0] = 'y'
932 >>> print(a)
933 array('w', 'yello, world')
934 >>> a.tounicode()
935 'yello, world'
936
937
938 How do I use strings to call functions/methods?
939 -----------------------------------------------
940
941 There are various techniques.
942
943 * The best is to use a dictionary that maps strings to functions. The primary
944 advantage of this technique is that the strings do not need to match the names
945 of the functions. This is also the primary technique used to emulate a case
946 construct::
947
948 def a():
949 pass
950
951 def b():
952 pass
953
954 dispatch = {'go': a, 'stop': b} # Note lack of parens for funcs
955
956 dispatch[get_input()]() # Note trailing parens to call function
957
958 * Use the built-in function :func:`getattr`::
959
960 import foo
961 getattr(foo, 'bar')()
962
963 Note that :func:`getattr` works on any object, including classes, class
964 instances, modules, and so on.
965
966 This is used in several places in the standard library, like this::
967
968 class Foo:
969 def do_foo(self):
970 ...
971
972 def do_bar(self):
973 ...
974
975 f = getattr(foo_instance, 'do_' + opname)
976 f()
977
978
979 * Use :func:`locals` to resolve the function name::
980
981 def myFunc():
982 print("hello")
983
984 fname = "myFunc"
985
986 f = locals()[fname]
987 f()
988
989
990 Is there an equivalent to Perl's ``chomp()`` for removing trailing newlines from strings?
991 -----------------------------------------------------------------------------------------
992
993 You can use ``S.rstrip("\r\n")`` to remove all occurrences of any line
994 terminator from the end of the string ``S`` without removing other trailing
995 whitespace. If the string ``S`` represents more than one line, with several
996 empty lines at the end, the line terminators for all the blank lines will
997 be removed::
998
999 >>> lines = ("line 1 \r\n"
1000 ... "\r\n"
1001 ... "\r\n")
1002 >>> lines.rstrip("\n\r")
1003 'line 1 '
1004
1005 Since this is typically only desired when reading text one line at a time, using
1006 ``S.rstrip()`` this way works well.
1007
1008
1009 Is there a ``scanf()`` or ``sscanf()`` equivalent?
1010 --------------------------------------------------
1011
1012 Not as such.
1013
1014 For simple input parsing, the easiest approach is usually to split the line into
1015 whitespace-delimited words using the :meth:`~str.split` method of string objects
1016 and then convert decimal strings to numeric values using :func:`int` or
1017 :func:`float`. :meth:`!split` supports an optional "sep" parameter which is useful
1018 if the line uses something other than whitespace as a separator.
1019
1020 For more complicated input parsing, regular expressions are more powerful
1021 than C's ``sscanf`` and better suited for the task.
1022
1023
1024 What does ``UnicodeDecodeError`` or ``UnicodeEncodeError`` error mean?
1025 ----------------------------------------------------------------------
1026
1027 See the :ref:`unicode-howto`.
1028
1029
1030 .. _faq-programming-raw-string-backslash:
1031
1032 Can I end a raw string with an odd number of backslashes?
1033 ---------------------------------------------------------
1034
1035 A raw string ending with an odd number of backslashes will escape the string's quote::
1036
1037 >>> r'C:\this\will\not\work\'
1038 File "<stdin>", line 1
1039 r'C:\this\will\not\work\'
1040 ^
1041 SyntaxError: unterminated string literal (detected at line 1)
1042
1043 There are several workarounds for this. One is to use regular strings and double
1044 the backslashes::
1045
1046 >>> 'C:\\this\\will\\work\\'
1047 'C:\\this\\will\\work\\'
1048
1049 Another is to concatenate a regular string containing an escaped backslash to the
1050 raw string::
1051
1052 >>> r'C:\this\will\work' '\\'
1053 'C:\\this\\will\\work\\'
1054
1055 It is also possible to use :func:`os.path.join` to append a backslash on Windows::
1056
1057 >>> os.path.join(r'C:\this\will\work', '')
1058 'C:\\this\\will\\work\\'
1059
1060 Note that while a backslash will "escape" a quote for the purposes of
1061 determining where the raw string ends, no escaping occurs when interpreting the
1062 value of the raw string. That is, the backslash remains present in the value of
1063 the raw string::
1064
1065 >>> r'backslash\'preserved'
1066 "backslash\\'preserved"
1067
1068 Also see the specification in the :ref:`language reference <strings>`.
1069
1070
1071 Performance
1072 ===========
1073
1074 My program is too slow. How do I speed it up?
1075 ---------------------------------------------
1076
1077 That's a tough one, in general. First, here is a list of things to
1078 remember before diving further:
1079
1080 * Performance characteristics vary across Python implementations. This FAQ
1081 focuses on :term:`CPython`.
1082 * Behaviour can vary across operating systems, especially when talking about
1083 I/O or multi-threading.
1084 * You should always find the hot spots in your program *before* attempting to
1085 optimize any code (see the :mod:`profile` module).
1086 * Writing benchmark scripts will allow you to iterate quickly when searching
1087 for improvements (see the :mod:`timeit` module).
1088 * It is highly recommended to have good code coverage (through unit testing
1089 or any other technique) before potentially introducing regressions hidden
1090 in sophisticated optimizations.
1091
1092 That being said, there are many tricks to speed up Python code. Here are
1093 some general principles which go a long way towards reaching acceptable
1094 performance levels:
1095
1096 * Making your algorithms faster (or changing to faster ones) can yield
1097 much larger benefits than trying to sprinkle micro-optimization tricks
1098 all over your code.
1099
1100 * Use the right data structures. Study documentation for the :ref:`bltin-types`
1101 and the :mod:`collections` module.
1102
1103 * When the standard library provides a primitive for doing something, it is
1104 likely (although not guaranteed) to be faster than any alternative you
1105 may come up with. This is doubly true for primitives written in C, such
1106 as builtins and some extension types. For example, be sure to use
1107 either the :meth:`list.sort` built-in method or the related :func:`sorted`
1108 function to do sorting (and see the :ref:`sortinghowto` for examples
1109 of moderately advanced usage).
1110
1111 * Abstractions tend to create indirections and force the interpreter to work
1112 more. If the levels of indirection outweigh the amount of useful work
1113 done, your program will be slower. You should avoid excessive abstraction,
1114 especially under the form of tiny functions or methods (which are also often
1115 detrimental to readability).
1116
1117 If you have reached the limit of what pure Python can allow, there are tools
1118 to take you further away. For example, `Cython <https://cython.org>`_ can
1119 compile a slightly modified version of Python code into a C extension, and
1120 can be used on many different platforms. Cython can take advantage of
1121 compilation (and optional type annotations) to make your code significantly
1122 faster than when interpreted. If you are confident in your C programming
1123 skills, you can also :ref:`write a C extension module <extending-index>`
1124 yourself.
1125
1126 .. seealso::
1127 The wiki page devoted to `performance tips
1128 <https://wiki.python.org/moin/PythonSpeed/PerformanceTips>`_.
1129
1130
1131 .. _efficient_string_concatenation:
1132
1133 What is the most efficient way to concatenate many strings together?
1134 --------------------------------------------------------------------
1135
1136 :class:`str` and :class:`bytes` objects are immutable, therefore concatenating
1137 many strings together is inefficient as each concatenation creates a new
1138 object. In the general case, the total runtime cost is quadratic in the
1139 total string length.
1140
1141 To accumulate many :class:`str` objects, the recommended idiom is to place
1142 them into a list and call :meth:`str.join` at the end::
1143
1144 chunks = []
1145 for s in my_strings:
1146 chunks.append(s)
1147 result = ''.join(chunks)
1148
1149 (Another reasonably efficient idiom is to use :class:`io.StringIO`.)
1150
1151 To accumulate many :class:`bytes` objects, the recommended idiom is to extend
1152 a :class:`bytearray` object using in-place concatenation (the ``+=`` operator)::
1153
1154 result = bytearray()
1155 for b in my_bytes_objects:
1156 result += b
1157
1158
1159 Sequences (tuples/lists)
1160 ========================
1161
1162 How do I convert between tuples and lists?
1163 ------------------------------------------
1164
1165 The type constructor ``tuple(seq)`` converts any sequence (actually, any
1166 iterable) into a tuple with the same items in the same order.
1167
1168 For example, ``tuple([1, 2, 3])`` yields ``(1, 2, 3)`` and ``tuple('abc')``
1169 yields ``('a', 'b', 'c')``. If the argument is a tuple, it does not make a copy
1170 but returns the same object, so it is cheap to call :func:`tuple` when you
1171 aren't sure that an object is already a tuple.
1172
1173 The type constructor ``list(seq)`` converts any sequence or iterable into a list
1174 with the same items in the same order. For example, ``list((1, 2, 3))`` yields
1175 ``[1, 2, 3]`` and ``list('abc')`` yields ``['a', 'b', 'c']``. If the argument
1176 is a list, it makes a copy just like ``seq[:]`` would.
1177
1178
1179 What's a negative index?
1180 ------------------------
1181
1182 Python sequences are indexed with positive numbers and negative numbers. For
1183 positive numbers 0 is the first index 1 is the second index and so forth. For
1184 negative indices -1 is the last index and -2 is the penultimate (next to last)
1185 index and so forth. Think of ``seq[-n]`` as the same as ``seq[len(seq)-n]``.
1186
1187 Using negative indices can be very convenient. For example ``S[:-1]`` is all of
1188 the string except for its last character, which is useful for removing the
1189 trailing newline from a string.
1190
1191
1192 How do I iterate over a sequence in reverse order?
1193 --------------------------------------------------
1194
1195 Use the :func:`reversed` built-in function::
1196
1197 for x in reversed(sequence):
1198 ... # do something with x ...
1199
1200 This won't touch your original sequence, but build a new copy with reversed
1201 order to iterate over.
1202
1203
1204 How do you remove duplicates from a list?
1205 -----------------------------------------
1206
1207 See the Python Cookbook for a long discussion of many ways to do this:
1208
1209 https://code.activestate.com/recipes/52560/
1210
1211 If you don't mind reordering the list, sort it and then scan from the end of the
1212 list, deleting duplicates as you go::
1213
1214 if mylist:
1215 mylist.sort()
1216 last = mylist[-1]
1217 for i in range(len(mylist)-2, -1, -1):
1218 if last == mylist[i]:
1219 del mylist[i]
1220 else:
1221 last = mylist[i]
1222
1223 If all elements of the list may be used as set keys (that is, they are all
1224 :term:`hashable`) this is often faster::
1225
1226 mylist = list(set(mylist))
1227
1228 This converts the list into a set, thereby removing duplicates, and then back
1229 into a list.
1230
1231
1232 How do you remove multiple items from a list?
1233 ---------------------------------------------
1234
1235 As with removing duplicates, explicitly iterating in reverse with a
1236 delete condition is one possibility. However, it is easier and faster
1237 to use slice replacement with an implicit or explicit forward iteration.
1238 Here are three variations::
1239
1240 mylist[:] = filter(keep_function, mylist)
1241 mylist[:] = (x for x in mylist if keep_condition)
1242 mylist[:] = [x for x in mylist if keep_condition]
1243
1244 The list comprehension may be fastest.
1245
1246
1247 How do you make an array in Python?
1248 -----------------------------------
1249
1250 Use a list::
1251
1252 ["this", 1, "is", "an", "array"]
1253
1254 Lists are equivalent to C or Pascal arrays in their time complexity; the primary
1255 difference is that a Python list can contain objects of many different types.
1256
1257 The ``array`` module also provides methods for creating arrays of fixed types
1258 with compact representations, but they are slower to index than lists. Also
1259 note that `NumPy <https://numpy.org/>`_
1260 and other third-party packages define array-like structures with
1261 various characteristics as well.
1262
1263 To get Lisp-style linked lists, you can emulate *cons cells* using tuples::
1264
1265 lisp_list = ("like", ("this", ("example", None) ) )
1266
1267 If mutability is desired, you could use lists instead of tuples. Here the
1268 analogue of a Lisp *car* is ``lisp_list[0]`` and the analogue of *cdr* is
1269 ``lisp_list[1]``. Only do this if you're sure you really need to, because it's
1270 usually a lot slower than using Python lists.
1271
1272
1273 .. _faq-multidimensional-list:
1274
1275 How do I create a multidimensional list?
1276 ----------------------------------------
1277
1278 You probably tried to make a multidimensional array like this::
1279
1280 >>> A = [[None] * 2] * 3
1281
1282 This looks correct if you print it:
1283
1284 .. testsetup::
1285
1286 A = [[None] * 2] * 3
1287
1288 .. doctest::
1289
1290 >>> A
1291 [[None, None], [None, None], [None, None]]
1292
1293 But when you assign a value, it shows up in multiple places:
1294
1295 .. testsetup::
1296
1297 A = [[None] * 2] * 3
1298
1299 .. doctest::
1300
1301 >>> A[0][0] = 5
1302 >>> A
1303 [[5, None], [5, None], [5, None]]
1304
1305 The reason is that replicating a list with ``*`` doesn't create copies, it only
1306 creates references to the existing objects. The ``*3`` creates a list
1307 containing 3 references to the same list of length two. Changes to one row will
1308 show in all rows, which is almost certainly not what you want.
1309
1310 The suggested approach is to create a list of the desired length first and then
1311 fill in each element with a newly created list::
1312
1313 A = [None] * 3
1314 for i in range(3):
1315 A[i] = [None] * 2
1316
1317 This generates a list containing 3 different lists of length two. You can also
1318 use a list comprehension::
1319
1320 w, h = 2, 3
1321 A = [[None] * w for i in range(h)]
1322
1323 Or, you can use an extension that provides a matrix datatype; `NumPy
1324 <https://numpy.org/>`_ is the best known.
1325
1326
1327 How do I apply a method or function to a sequence of objects?
1328 -------------------------------------------------------------
1329
1330 To call a method or function and accumulate the return values in a list,
1331 a :term:`list comprehension` is an elegant solution::
1332
1333 result = [obj.method() for obj in mylist]
1334
1335 result = [function(obj) for obj in mylist]
1336
1337 To just run the method or function without saving the return values,
1338 a plain :keyword:`for` loop will suffice::
1339
1340 for obj in mylist:
1341 obj.method()
1342
1343 for obj in mylist:
1344 function(obj)
1345
1346
1347 .. _faq-augmented-assignment-tuple-error:
1348
1349 Why does a_tuple[i] += ['item'] raise an exception when the addition works?
1350 ---------------------------------------------------------------------------
1351
1352 This is because of a combination of the fact that augmented assignment
1353 operators are *assignment* operators, and the difference between mutable and
1354 immutable objects in Python.
1355
1356 This discussion applies in general when augmented assignment operators are
1357 applied to elements of a tuple that point to mutable objects, but we'll use
1358 a ``list`` and ``+=`` as our exemplar.
1359
1360 If you wrote::
1361
1362 >>> a_tuple = (1, 2)
1363 >>> a_tuple[0] += 1
1364 Traceback (most recent call last):
1365 ...
1366 TypeError: 'tuple' object does not support item assignment
1367
1368 The reason for the exception should be immediately clear: ``1`` is added to the
1369 object ``a_tuple[0]`` points to (``1``), producing the result object, ``2``,
1370 but when we attempt to assign the result of the computation, ``2``, to element
1371 ``0`` of the tuple, we get an error because we can't change what an element of
1372 a tuple points to.
1373
1374 Under the covers, what this augmented assignment statement is doing is
1375 approximately this::
1376
1377 >>> result = a_tuple[0] + 1
1378 >>> a_tuple[0] = result
1379 Traceback (most recent call last):
1380 ...
1381 TypeError: 'tuple' object does not support item assignment
1382
1383 It is the assignment part of the operation that produces the error, since a
1384 tuple is immutable.
1385
1386 When you write something like::
1387
1388 >>> a_tuple = (['foo'], 'bar')
1389 >>> a_tuple[0] += ['item']
1390 Traceback (most recent call last):
1391 ...
1392 TypeError: 'tuple' object does not support item assignment
1393
1394 The exception is a bit more surprising, and even more surprising is the fact
1395 that even though there was an error, the append worked::
1396
1397 >>> a_tuple[0]
1398 ['foo', 'item']
1399
1400 To see why this happens, you need to know that (a) if an object implements an
1401 :meth:`~object.__iadd__` magic method, it gets called when the ``+=`` augmented
1402 assignment
1403 is executed, and its return value is what gets used in the assignment statement;
1404 and (b) for lists, :meth:`!__iadd__` is equivalent to calling
1405 :meth:`~sequence.extend` on the list and returning the list.
1406 That's why we say that for lists, ``+=`` is a "shorthand" for :meth:`list.extend`::
1407
1408 >>> a_list = []
1409 >>> a_list += [1]
1410 >>> a_list
1411 [1]
1412
1413 This is equivalent to::
1414
1415 >>> result = a_list.__iadd__([1])
1416 >>> a_list = result
1417
1418 The object pointed to by a_list has been mutated, and the pointer to the
1419 mutated object is assigned back to ``a_list``. The end result of the
1420 assignment is a no-op, since it is a pointer to the same object that ``a_list``
1421 was previously pointing to, but the assignment still happens.
1422
1423 Thus, in our tuple example what is happening is equivalent to::
1424
1425 >>> result = a_tuple[0].__iadd__(['item'])
1426 >>> a_tuple[0] = result
1427 Traceback (most recent call last):
1428 ...
1429 TypeError: 'tuple' object does not support item assignment
1430
1431 The :meth:`!__iadd__` succeeds, and thus the list is extended, but even though
1432 ``result`` points to the same object that ``a_tuple[0]`` already points to,
1433 that final assignment still results in an error, because tuples are immutable.
1434
1435
1436 I want to do a complicated sort: can you do a Schwartzian Transform in Python?
1437 ------------------------------------------------------------------------------
1438
1439 The technique, attributed to Randal Schwartz of the Perl community, sorts the
1440 elements of a list by a metric which maps each element to its "sort value". In
1441 Python, use the ``key`` argument for the :meth:`list.sort` method::
1442
1443 Isorted = L[:]
1444 Isorted.sort(key=lambda s: int(s[10:15]))
1445
1446
1447 How can I sort one list by values from another list?
1448 ----------------------------------------------------
1449
1450 Merge them into an iterator of tuples, sort the resulting list, and then pick
1451 out the element you want.
1452
1453 >>> list1 = ["what", "I'm", "sorting", "by"]
1454 >>> list2 = ["something", "else", "to", "sort"]
1455 >>> pairs = zip(list1, list2)
1456 >>> pairs = sorted(pairs)
1457 >>> pairs
1458 [("I'm", 'else'), ('by', 'sort'), ('sorting', 'to'), ('what', 'something')]
1459 >>> result = [x[1] for x in pairs]
1460 >>> result
1461 ['else', 'sort', 'to', 'something']
1462
1463
1464 Objects
1465 =======
1466
1467 What is a class?
1468 ----------------
1469
1470 A class is the particular object type created by executing a class statement.
1471 Class objects are used as templates to create instance objects, which embody
1472 both the data (attributes) and code (methods) specific to a datatype.
1473
1474 A class can be based on one or more other classes, called its base class(es). It
1475 then inherits the attributes and methods of its base classes. This allows an
1476 object model to be successively refined by inheritance. You might have a
1477 generic ``Mailbox`` class that provides basic accessor methods for a mailbox,
1478 and subclasses such as ``MboxMailbox``, ``MaildirMailbox``, ``OutlookMailbox``
1479 that handle various specific mailbox formats.
1480
1481
1482 What is a method?
1483 -----------------
1484
1485 A method is a function on some object ``x`` that you normally call as
1486 ``x.name(arguments...)``. Methods are defined as functions inside the class
1487 definition::
1488
1489 class C:
1490 def meth(self, arg):
1491 return arg * 2 + self.attribute
1492
1493
1494 What is self?
1495 -------------
1496
1497 Self is merely a conventional name for the first argument of a method. A method
1498 defined as ``meth(self, a, b, c)`` should be called as ``x.meth(a, b, c)`` for
1499 some instance ``x`` of the class in which the definition occurs; the called
1500 method will think it is called as ``meth(x, a, b, c)``.
1501
1502 See also :ref:`why-self`.
1503
1504
1505 How do I check if an object is an instance of a given class or of a subclass of it?
1506 -----------------------------------------------------------------------------------
1507
1508 Use the built-in function :func:`isinstance(obj, cls) <isinstance>`. You can
1509 check if an object
1510 is an instance of any of a number of classes by providing a tuple instead of a
1511 single class, for example, ``isinstance(obj, (class1, class2, ...))``, and can also
1512 check whether an object is one of Python's built-in types, for example,
1513 ``isinstance(obj, str)`` or ``isinstance(obj, (int, float, complex))``.
1514
1515 Note that :func:`isinstance` also checks for virtual inheritance from an
1516 :term:`abstract base class`. So, the test will return ``True`` for a
1517 registered class even if hasn't directly or indirectly inherited from it. To
1518 test for "true inheritance", scan the :term:`method resolution order` (MRO) of
1519 the class:
1520
1521 .. testcode::
1522
1523 from collections.abc import Mapping
1524
1525 class P:
1526 pass
1527
1528 class C(P):
1529 pass
1530
1531 Mapping.register(P)
1532
1533 .. doctest::
1534
1535 >>> c = C()
1536 >>> isinstance(c, C) # direct
1537 True
1538 >>> isinstance(c, P) # indirect
1539 True
1540 >>> isinstance(c, Mapping) # virtual
1541 True
1542
1543 # Actual inheritance chain
1544 >>> type(c).__mro__
1545 (<class 'C'>, <class 'P'>, <class 'object'>)
1546
1547 # Test for "true inheritance"
1548 >>> Mapping in type(c).__mro__
1549 False
1550
1551 Note that most programs do not use :func:`isinstance` on user-defined classes
1552 very often. If you are developing the classes yourself, a more proper
1553 object-oriented style is to define methods on the classes that encapsulate a
1554 particular behaviour, instead of checking the object's class and doing a
1555 different thing based on what class it is. For example, if you have a function
1556 that does something::
1557
1558 def search(obj):
1559 if isinstance(obj, Mailbox):
1560 ... # code to search a mailbox
1561 elif isinstance(obj, Document):
1562 ... # code to search a document
1563 elif ...
1564
1565 A better approach is to define a ``search()`` method on all the classes and just
1566 call it::
1567
1568 class Mailbox:
1569 def search(self):
1570 ... # code to search a mailbox
1571
1572 class Document:
1573 def search(self):
1574 ... # code to search a document
1575
1576 obj.search()
1577
1578
1579 What is delegation?
1580 -------------------
1581
1582 Delegation is an object-oriented technique (also called a design pattern).
1583 Let's say you have an object ``x`` and want to change the behaviour of just one
1584 of its methods. You can create a new class that provides a new implementation
1585 of the method you're interested in changing and delegates all other methods to
1586 the corresponding method of ``x``.
1587
1588 Python programmers can easily implement delegation. For example, the following
1589 class implements a class that behaves like a file but converts all written data
1590 to uppercase::
1591
1592 class UpperOut:
1593
1594 def __init__(self, outfile):
1595 self._outfile = outfile
1596
1597 def write(self, s):
1598 self._outfile.write(s.upper())
1599
1600 def __getattr__(self, name):
1601 return getattr(self._outfile, name)
1602
1603 Here the ``UpperOut`` class redefines the ``write()`` method to convert the
1604 argument string to uppercase before calling the underlying
1605 ``self._outfile.write()`` method. All other methods are delegated to the
1606 underlying ``self._outfile`` object. The delegation is accomplished via the
1607 :meth:`~object.__getattr__` method; consult :ref:`the language reference <attribute-access>`
1608 for more information about controlling attribute access.
1609
1610 Note that for more general cases delegation can get trickier. When attributes
1611 must be set as well as retrieved, the class must define a :meth:`~object.__setattr__`
1612 method too, and it must do so carefully. The basic implementation of
1613 :meth:`!__setattr__` is roughly equivalent to the following::
1614
1615 class X:
1616 ...
1617 def __setattr__(self, name, value):
1618 self.__dict__[name] = value
1619 ...
1620
1621 Many :meth:`~object.__setattr__` implementations call :meth:`!object.__setattr__` to set
1622 an attribute on self without causing infinite recursion::
1623
1624 class X:
1625 def __setattr__(self, name, value):
1626 # Custom logic here...
1627 object.__setattr__(self, name, value)
1628
1629 Alternatively, it is possible to set attributes by inserting
1630 entries into :attr:`self.__dict__ <object.__dict__>` directly.
1631
1632
1633 How do I call a method defined in a base class from a derived class that extends it?
1634 ------------------------------------------------------------------------------------
1635
1636 Use the built-in :func:`super` function::
1637
1638 class Derived(Base):
1639 def meth(self):
1640 super().meth() # calls Base.meth
1641
1642 In the example, :func:`super` will automatically determine the instance from
1643 which it was called (the ``self`` value), look up the :term:`method resolution
1644 order` (MRO) with ``type(self).__mro__``, and return the next in line after
1645 ``Derived`` in the MRO: ``Base``.
1646
1647
1648 How can I organize my code to make it easier to change the base class?
1649 ----------------------------------------------------------------------
1650
1651 You could assign the base class to an alias and derive from the alias. Then all
1652 you have to change is the value assigned to the alias. Incidentally, this trick
1653 is also handy if you want to decide dynamically (such as depending on availability
1654 of resources) which base class to use. Example::
1655
1656 class Base:
1657 ...
1658
1659 BaseAlias = Base
1660
1661 class Derived(BaseAlias):
1662 ...
1663
1664
1665 How do I create static class data and static class methods?
1666 -----------------------------------------------------------
1667
1668 Both static data and static methods (in the sense of C++ or Java) are supported
1669 in Python.
1670
1671 For static data, simply define a class attribute. To assign a new value to the
1672 attribute, you have to explicitly use the class name in the assignment::
1673
1674 class C:
1675 count = 0 # number of times C.__init__ called
1676
1677 def __init__(self):
1678 C.count = C.count + 1
1679
1680 def getcount(self):
1681 return C.count # or return self.count
1682
1683 ``c.count`` also refers to ``C.count`` for any ``c`` such that ``isinstance(c,
1684 C)`` holds, unless overridden by ``c`` itself or by some class on the base-class
1685 search path from ``c.__class__`` back to ``C``.
1686
1687 Caution: within a method of C, an assignment like ``self.count = 42`` creates a
1688 new and unrelated instance named "count" in ``self``'s own dict. Rebinding of a
1689 class-static data name must always specify the class whether inside a method or
1690 not::
1691
1692 C.count = 314
1693
1694 Static methods are possible::
1695
1696 class C:
1697 @staticmethod
1698 def static(arg1, arg2, arg3):
1699 # No 'self' parameter!
1700 ...
1701
1702 However, a far more straightforward way to get the effect of a static method is
1703 via a simple module-level function::
1704
1705 def getcount():
1706 return C.count
1707
1708 If your code is structured so as to define one class (or tightly related class
1709 hierarchy) per module, this supplies the desired encapsulation.
1710
1711
1712 How can I overload constructors (or methods) in Python?
1713 -------------------------------------------------------
1714
1715 This answer actually applies to all methods, but the question usually comes up
1716 first in the context of constructors.
1717
1718 In C++ you'd write:
1719
1720 .. code-block:: c++
1721
1722 class C {
1723 C() { cout << "No arguments\n"; }
1724 C(int i) { cout << "Argument is " << i << "\n"; }
1725 }
1726
1727 In Python you have to write a single constructor that catches all cases using
1728 default arguments. For example::
1729
1730 class C:
1731 def __init__(self, i=None):
1732 if i is None:
1733 print("No arguments")
1734 else:
1735 print("Argument is", i)
1736
1737 This is not entirely equivalent, but close enough in practice.
1738
1739 You could also try a variable-length argument list, for example::
1740
1741 def __init__(self, *args):
1742 ...
1743
1744 The same approach works for all method definitions.
1745
1746
1747 I try to use __spam and I get an error about _SomeClassName__spam.
1748 ------------------------------------------------------------------
1749
1750 Variable names with double leading underscores are "mangled" to provide a simple
1751 but effective way to define class private variables. Any identifier of the form
1752 ``__spam`` (at least two leading underscores, at most one trailing underscore)
1753 is textually replaced with ``_classname__spam``, where ``classname`` is the
1754 current class name with any leading underscores stripped.
1755
1756 The identifier can be used unchanged within the class, but to access it outside
1757 the class, the mangled name must be used:
1758
1759 .. code-block:: python
1760
1761 class A:
1762 def __one(self):
1763 return 1
1764 def two(self):
1765 return 2 * self.__one()
1766
1767 class B(A):
1768 def three(self):
1769 return 3 * self._A__one()
1770
1771 four = 4 * A()._A__one()
1772
1773 In particular, this does not guarantee privacy since an outside user can still
1774 deliberately access the private attribute; many Python programmers never bother
1775 to use private variable names at all.
1776
1777 .. seealso::
1778
1779 The :ref:`private name mangling specifications <private-name-mangling>`
1780 for details and special cases.
1781
1782
1783 My class defines __del__ but it is not called when I delete the object.
1784 -----------------------------------------------------------------------
1785
1786 There are several possible reasons for this.
1787
1788 The :keyword:`del` statement does not necessarily call :meth:`~object.__del__` -- it simply
1789 decrements the object's reference count, and if this reaches zero
1790 :meth:`!__del__` is called.
1791
1792 If your data structures contain circular links (for example, a tree where each child has
1793 a parent reference and each parent has a list of children) the reference counts
1794 will never go back to zero. Once in a while Python runs an algorithm to detect
1795 such cycles, but the garbage collector might run some time after the last
1796 reference to your data structure vanishes, so your :meth:`!__del__` method may be
1797 called at an inconvenient and random time. This is inconvenient if you're trying
1798 to reproduce a problem. Worse, the order in which object's :meth:`!__del__`
1799 methods are executed is arbitrary. You can run :func:`gc.collect` to force a
1800 collection, but there *are* pathological cases where objects will never be
1801 collected.
1802
1803 Despite the cycle collector, it's still a good idea to define an explicit
1804 ``close()`` method on objects to be called whenever you're done with them. The
1805 ``close()`` method can then remove attributes that refer to subobjects. Don't
1806 call :meth:`!__del__` directly -- :meth:`!__del__` should call ``close()`` and
1807 ``close()`` should make sure that it can be called more than once for the same
1808 object.
1809
1810 Another way to avoid cyclical references is to use the :mod:`weakref` module,
1811 which allows you to point to objects without incrementing their reference count.
1812 Tree data structures, for instance, should use weak references for their parent
1813 and sibling references (if they need them!).
1814
1815 .. XXX relevant for Python 3?
1816
1817 If the object has ever been a local variable in a function that caught an
1818 expression in an except clause, chances are that a reference to the object
1819 still exists in that function's stack frame as contained in the stack trace.
1820 Normally, calling :func:`sys.exc_clear` will take care of this by clearing
1821 the last recorded exception.
1822
1823 Finally, if your :meth:`!__del__` method raises an exception, a warning message
1824 is printed to :data:`sys.stderr`.
1825
1826
1827 How do I get a list of all instances of a given class?
1828 ------------------------------------------------------
1829
1830 Python does not keep track of all instances of a class (or of a built-in type).
1831 You can program the class's constructor to keep track of all instances by
1832 keeping a list of weak references to each instance.
1833
1834
1835 Why does the result of ``id()`` appear to be not unique?
1836 --------------------------------------------------------
1837
1838 The :func:`id` builtin returns an integer that is guaranteed to be unique during
1839 the lifetime of the object. Since in CPython, this is the object's memory
1840 address, it happens frequently that after an object is deleted from memory, the
1841 next freshly created object is allocated at the same position in memory. This
1842 is illustrated by this example:
1843
1844 >>> id(1000) # doctest: +SKIP
1845 13901272
1846 >>> id(2000) # doctest: +SKIP
1847 13901272
1848
1849 The two ids belong to different integer objects that are created before, and
1850 deleted immediately after execution of the ``id()`` call. To be sure that
1851 objects whose id you want to examine are still alive, create another reference
1852 to the object:
1853
1854 >>> a = 1000; b = 2000
1855 >>> id(a) # doctest: +SKIP
1856 13901272
1857 >>> id(b) # doctest: +SKIP
1858 13891296
1859
1860
1861 .. _faq-identity-with-is:
1862
1863 When can I rely on identity tests with the *is* operator?
1864 ---------------------------------------------------------
1865
1866 The ``is`` operator tests for object identity. The test ``a is b`` is
1867 equivalent to ``id(a) == id(b)``.
1868
1869 The most important property of an identity test is that an object is always
1870 identical to itself, ``a is a`` always returns ``True``. Identity tests are
1871 usually faster than equality tests. And unlike equality tests, identity tests
1872 are guaranteed to return a boolean ``True`` or ``False``.
1873
1874 However, identity tests can *only* be substituted for equality tests when
1875 object identity is assured. Generally, there are three circumstances where
1876 identity is guaranteed:
1877
1878 1) Assignments create new names but do not change object identity. After the
1879 assignment ``new = old``, it is guaranteed that ``new is old``.
1880
1881 2) Putting an object in a container that stores object references does not
1882 change object identity. After the list assignment ``s[0] = x``, it is
1883 guaranteed that ``s[0] is x``.
1884
1885 3) If an object is a singleton, it means that only one instance of that object
1886 can exist. After the assignments ``a = None`` and ``b = None``, it is
1887 guaranteed that ``a is b`` because ``None`` is a singleton.
1888
1889 In most other circumstances, identity tests are inadvisable and equality tests
1890 are preferred. In particular, identity tests should not be used to check
1891 constants such as :class:`int` and :class:`str` which aren't guaranteed to be
1892 singletons::
1893
1894 >>> a = 10_000_000
1895 >>> b = 5_000_000
1896 >>> c = b + 5_000_000
1897 >>> a is c
1898 False
1899
1900 >>> a = 'Python'
1901 >>> b = 'Py'
1902 >>> c = b + 'thon'
1903 >>> a is c
1904 False
1905
1906 Likewise, new instances of mutable containers are never identical::
1907
1908 >>> a = []
1909 >>> b = []
1910 >>> a is b
1911 False
1912
1913 In the standard library code, you will see several common patterns for
1914 correctly using identity tests:
1915
1916 1) As recommended by :pep:`8`, an identity test is the preferred way to check
1917 for ``None``. This reads like plain English in code and avoids confusion
1918 with other objects that may have boolean values that evaluate to false.
1919
1920 2) Detecting optional arguments can be tricky when ``None`` is a valid input
1921 value. In those situations, you can create a singleton sentinel object
1922 guaranteed to be distinct from other objects. For example, here is how
1923 to implement a method that behaves like :meth:`dict.pop`:
1924
1925 .. code-block:: python
1926
1927 _sentinel = object()
1928
1929 def pop(self, key, default=_sentinel):
1930 if key in self:
1931 value = self[key]
1932 del self[key]
1933 return value
1934 if default is _sentinel:
1935 raise KeyError(key)
1936 return default
1937
1938 3) Container implementations sometimes need to augment equality tests with
1939 identity tests. This prevents the code from being confused by objects
1940 such as ``float('NaN')`` that are not equal to themselves.
1941
1942 For example, here is the implementation of
1943 :meth:`!collections.abc.Sequence.__contains__`::
1944
1945 def __contains__(self, value):
1946 for v in self:
1947 if v is value or v == value:
1948 return True
1949 return False
1950
1951
1952 How can a subclass control what data is stored in an immutable instance?
1953 ------------------------------------------------------------------------
1954
1955 When subclassing an immutable type, override the :meth:`~object.__new__` method
1956 instead of the :meth:`~object.__init__` method. The latter only runs *after* an
1957 instance is created, which is too late to alter data in an immutable
1958 instance.
1959
1960 All of these immutable classes have a different signature than their
1961 parent class:
1962
1963 .. testcode::
1964
1965 import datetime as dt
1966
1967 class FirstOfMonthDate(dt.date):
1968 "Always choose the first day of the month"
1969 def __new__(cls, year, month, day):
1970 return super().__new__(cls, year, month, 1)
1971
1972 class NamedInt(int):
1973 "Allow text names for some numbers"
1974 xlat = {'zero': 0, 'one': 1, 'ten': 10}
1975 def __new__(cls, value):
1976 value = cls.xlat.get(value, value)
1977 return super().__new__(cls, value)
1978
1979 class TitleStr(str):
1980 "Convert str to name suitable for a URL path"
1981 def __new__(cls, s):
1982 s = s.lower().replace(' ', '-')
1983 s = ''.join([c for c in s if c.isalnum() or c == '-'])
1984 return super().__new__(cls, s)
1985
1986 The classes can be used like this:
1987
1988 .. doctest::
1989
1990 >>> FirstOfMonthDate(2012, 2, 14)
1991 FirstOfMonthDate(2012, 2, 1)
1992 >>> NamedInt('ten')
1993 10
1994 >>> NamedInt(20)
1995 20
1996 >>> TitleStr('Blog: Why Python Rocks')
1997 'blog-why-python-rocks'
1998
1999
2000 .. _faq-cache-method-calls:
2001
2002 How do I cache method calls?
2003 ----------------------------
2004
2005 The two principal tools for caching methods are
2006 :func:`functools.cached_property` and :func:`functools.lru_cache`. The
2007 former stores results at the instance level and the latter at the class
2008 level.
2009
2010 The ``cached_property`` approach only works with methods that do not take
2011 any arguments. It does not create a reference to the instance. The
2012 cached method result will be kept only as long as the instance is alive.
2013
2014 The advantage is that when an instance is no longer used, the cached
2015 method result will be released right away. The disadvantage is that if
2016 instances accumulate, so too will the accumulated method results. They
2017 can grow without bound.
2018
2019 The ``lru_cache`` approach works with methods that have :term:`hashable`
2020 arguments. It creates a reference to the instance unless special
2021 efforts are made to pass in weak references.
2022
2023 The advantage of the least recently used algorithm is that the cache is
2024 bounded by the specified *maxsize*. The disadvantage is that instances
2025 are kept alive until they age out of the cache or until the cache is
2026 cleared.
2027
2028 This example shows the various techniques::
2029
2030 class Weather:
2031 "Lookup weather information on a government website"
2032
2033 def __init__(self, station_id):
2034 self._station_id = station_id
2035 # The _station_id is private and immutable
2036
2037 def current_temperature(self):
2038 "Latest hourly observation"
2039 # Do not cache this because old results
2040 # can be out of date.
2041
2042 @cached_property
2043 def location(self):
2044 "Return the longitude/latitude coordinates of the station"
2045 # Result only depends on the station_id
2046
2047 @lru_cache(maxsize=20)
2048 def historic_rainfall(self, date, units='mm'):
2049 "Rainfall on a given date"
2050 # Depends on the station_id, date, and units.
2051
2052 The above example assumes that the *station_id* never changes. If the
2053 relevant instance attributes are mutable, the ``cached_property`` approach
2054 can't be made to work because it cannot detect changes to the
2055 attributes.
2056
2057 To make the ``lru_cache`` approach work when the *station_id* is mutable,
2058 the class needs to define the :meth:`~object.__eq__` and :meth:`~object.__hash__`
2059 methods so that the cache can detect relevant attribute updates::
2060
2061 class Weather:
2062 "Example with a mutable station identifier"
2063
2064 def __init__(self, station_id):
2065 self.station_id = station_id
2066
2067 def change_station(self, station_id):
2068 self.station_id = station_id
2069
2070 def __eq__(self, other):
2071 return self.station_id == other.station_id
2072
2073 def __hash__(self):
2074 return hash(self.station_id)
2075
2076 @lru_cache(maxsize=20)
2077 def historic_rainfall(self, date, units='cm'):
2078 'Rainfall on a given date'
2079 # Depends on the station_id, date, and units.
2080
2081
2082 Modules
2083 =======
2084
2085 How do I create a .pyc file?
2086 ----------------------------
2087
2088 When a module is imported for the first time (or when the source file has
2089 changed since the current compiled file was created) a ``.pyc`` file containing
2090 the compiled code should be created in a ``__pycache__`` subdirectory of the
2091 directory containing the ``.py`` file. The ``.pyc`` file will have a
2092 filename that starts with the same name as the ``.py`` file, and ends with
2093 ``.pyc``, with a middle component that depends on the particular ``python``
2094 binary that created it. (See :pep:`3147` for details.)
2095
2096 One reason that a ``.pyc`` file may not be created is a permissions problem
2097 with the directory containing the source file, meaning that the ``__pycache__``
2098 subdirectory cannot be created. This can happen, for example, if you develop as
2099 one user but run as another, such as if you are testing with a web server.
2100
2101 Unless the :envvar:`PYTHONDONTWRITEBYTECODE` environment variable is set,
2102 creation of a .pyc file is automatic if you're importing a module and Python
2103 has the ability (permissions, free space, and so on) to create a ``__pycache__``
2104 subdirectory and write the compiled module to that subdirectory.
2105
2106 Running Python on a top-level script is not considered an import and no
2107 ``.pyc`` will be created. For example, if you have a top-level module
2108 ``foo.py`` that imports another module ``xyz.py``, when you run ``foo`` (by
2109 typing ``python foo.py`` as a shell command), a ``.pyc`` will be created for
2110 ``xyz`` because ``xyz`` is imported, but no ``.pyc`` file will be created for
2111 ``foo`` since ``foo.py`` isn't being imported.
2112
2113 If you need to create a ``.pyc`` file for ``foo`` -- that is, to create a
2114 ``.pyc`` file for a module that is not imported -- you can, using the
2115 :mod:`py_compile` and :mod:`compileall` modules.
2116
2117 The :mod:`py_compile` module can manually compile any module. One way is to use
2118 the ``compile()`` function in that module interactively::
2119
2120 >>> import py_compile
2121 >>> py_compile.compile('foo.py') # doctest: +SKIP
2122
2123 This will write the ``.pyc`` to a ``__pycache__`` subdirectory in the same
2124 location as ``foo.py`` (or you can override that with the optional parameter
2125 *cfile*).
2126
2127 You can also automatically compile all files in a directory or directories using
2128 the :mod:`compileall` module. You can do it from the shell prompt by running
2129 ``compileall.py`` and providing the path of a directory containing Python files
2130 to compile::
2131
2132 python -m compileall .
2133
2134
2135 How do I find the current module name?
2136 --------------------------------------
2137
2138 A module can find out its own module name by looking at the predefined global
2139 variable ``__name__``. If this has the value ``'__main__'``, the program is
2140 running as a script. Many modules that are usually used by importing them also
2141 provide a command-line interface or a self-test, and only execute this code
2142 after checking ``__name__``::
2143
2144 def main():
2145 print('Running test...')
2146 ...
2147
2148 if __name__ == '__main__':
2149 main()
2150
2151
2152 How can I have modules that mutually import each other?
2153 -------------------------------------------------------
2154
2155 Suppose you have the following modules:
2156
2157 :file:`foo.py`::
2158
2159 from bar import bar_var
2160 foo_var = 1
2161
2162 :file:`bar.py`::
2163
2164 from foo import foo_var
2165 bar_var = 2
2166
2167 The problem is that the interpreter will perform the following steps:
2168
2169 * main imports ``foo``
2170 * Empty globals for ``foo`` are created
2171 * ``foo`` is compiled and starts executing
2172 * ``foo`` imports ``bar``
2173 * Empty globals for ``bar`` are created
2174 * ``bar`` is compiled and starts executing
2175 * ``bar`` imports ``foo`` (which is a no-op since there already is a module named ``foo``)
2176 * The import mechanism tries to read ``foo_var`` from ``foo`` globals, to set ``bar.foo_var = foo.foo_var``
2177
2178 The last step fails, because Python isn't done with interpreting ``foo`` yet and
2179 the global symbol dictionary for ``foo`` is still empty.
2180
2181 The same thing happens when you use ``import foo``, and then try to access
2182 ``foo.foo_var`` in global code.
2183
2184 There are (at least) three possible workarounds for this problem.
2185
2186 Guido van Rossum recommends avoiding all uses of ``from <module> import ...``,
2187 and placing all code inside functions. Initializations of global variables and
2188 class variables should use constants or built-in functions only. This means
2189 everything from an imported module is referenced as ``<module>.<name>``.
2190
2191 Jim Roskind suggests performing steps in the following order in each module:
2192
2193 * exports (globals, functions, and classes that don't need imported base
2194 classes)
2195 * ``import`` statements
2196 * active code (including globals that are initialized from imported values).
2197
2198 Van Rossum doesn't like this approach much because the imports appear in a
2199 strange place, but it does work.
2200
2201 Matthias Urlichs recommends restructuring your code so that the recursive import
2202 is not necessary in the first place.
2203
2204 These solutions are not mutually exclusive.
2205
2206
2207 __import__('x.y.z') returns <module 'x'>; how do I get z?
2208 ---------------------------------------------------------
2209
2210 Consider using the convenience function :func:`~importlib.import_module` from
2211 :mod:`importlib` instead::
2212
2213 z = importlib.import_module('x.y.z')
2214
2215
2216 When I edit an imported module and reimport it, the changes don't show up. Why does this happen?
2217 -------------------------------------------------------------------------------------------------
2218
2219 For reasons of efficiency as well as consistency, Python only reads the module
2220 file on the first time a module is imported. If it didn't, in a program
2221 consisting of many modules where each one imports the same basic module, the
2222 basic module would be parsed and re-parsed many times. To force re-reading of a
2223 changed module, do this::
2224
2225 import importlib
2226 import modname
2227 importlib.reload(modname)
2228
2229 Warning: this technique is not 100% fool-proof. In particular, modules
2230 containing statements like::
2231
2232 from modname import some_objects
2233
2234 will continue to work with the old version of the imported objects. If the
2235 module contains class definitions, existing class instances will *not* be
2236 updated to use the new class definition. This can result in the following
2237 paradoxical behaviour::
2238
2239 >>> import importlib
2240 >>> import cls
2241 >>> c = cls.C() # Create an instance of C
2242 >>> importlib.reload(cls)
2243 <module 'cls' from 'cls.py'>
2244 >>> isinstance(c, cls.C) # isinstance is false?!?
2245 False
2246
2247 The nature of the problem is made clear if you print out the "identity" of the
2248 class objects::
2249
2250 >>> hex(id(c.__class__))
2251 '0x7352a0'
2252 >>> hex(id(cls.C))
2253 '0x4198d0'