| 1 |
.. _devmode: |
| 2 |
|
| 3 |
Python Development Mode |
| 4 |
======================= |
| 5 |
|
| 6 |
.. versionadded:: 3.7 |
| 7 |
|
| 8 |
The Python Development Mode introduces additional runtime checks that are too |
| 9 |
expensive to be enabled by default. It should not be more verbose than the |
| 10 |
default if the code is correct; new warnings are only emitted when an issue is |
| 11 |
detected. |
| 12 |
|
| 13 |
It can be enabled using the :option:`-X dev <-X>` command line option or by |
| 14 |
setting the :envvar:`PYTHONDEVMODE` environment variable to ``1``. |
| 15 |
|
| 16 |
See also :ref:`Python debug build <debug-build>`. |
| 17 |
|
| 18 |
Effects of the Python Development Mode |
| 19 |
-------------------------------------- |
| 20 |
|
| 21 |
Enabling the Python Development Mode is similar to the following command, but |
| 22 |
with additional effects described below:: |
| 23 |
|
| 24 |
PYTHONMALLOC=debug PYTHONASYNCIODEBUG=1 python -W default -X faulthandler |
| 25 |
|
| 26 |
Effects of the Python Development Mode: |
| 27 |
|
| 28 |
* Add ``default`` :ref:`warning filter <describing-warning-filters>`. The |
| 29 |
following warnings are shown: |
| 30 |
|
| 31 |
* :exc:`DeprecationWarning` |
| 32 |
* :exc:`ImportWarning` |
| 33 |
* :exc:`PendingDeprecationWarning` |
| 34 |
* :exc:`ResourceWarning` |
| 35 |
|
| 36 |
Normally, the above warnings are filtered by the default :ref:`warning |
| 37 |
filters <describing-warning-filters>`. |
| 38 |
|
| 39 |
It behaves as if the :option:`-W default <-W>` command line option is used. |
| 40 |
|
| 41 |
Use the :option:`-W error <-W>` command line option or set the |
| 42 |
:envvar:`PYTHONWARNINGS` environment variable to ``error`` to treat warnings |
| 43 |
as errors. |
| 44 |
|
| 45 |
* Install debug hooks on memory allocators to check for: |
| 46 |
|
| 47 |
* Buffer underflow |
| 48 |
* Buffer overflow |
| 49 |
* Memory allocator API violation |
| 50 |
* Unsafe usage of the GIL |
| 51 |
|
| 52 |
See the :c:func:`PyMem_SetupDebugHooks` C function. |
| 53 |
|
| 54 |
It behaves as if the :envvar:`PYTHONMALLOC` environment variable is set to |
| 55 |
``debug``. |
| 56 |
|
| 57 |
To enable the Python Development Mode without installing debug hooks on |
| 58 |
memory allocators, set the :envvar:`PYTHONMALLOC` environment variable to |
| 59 |
``default``. |
| 60 |
|
| 61 |
* Call :func:`faulthandler.enable` at Python startup to install handlers for |
| 62 |
the :const:`~signal.SIGSEGV`, :const:`~signal.SIGFPE`, |
| 63 |
:const:`~signal.SIGABRT`, :const:`~signal.SIGBUS` and |
| 64 |
:const:`~signal.SIGILL` signals to dump the Python traceback on a crash. |
| 65 |
|
| 66 |
It behaves as if the :option:`-X faulthandler <-X>` command line option is |
| 67 |
used or if the :envvar:`PYTHONFAULTHANDLER` environment variable is set to |
| 68 |
``1``. |
| 69 |
|
| 70 |
* Enable :ref:`asyncio debug mode <asyncio-debug-mode>`. For example, |
| 71 |
:mod:`asyncio` checks for coroutines that were not awaited and logs them. |
| 72 |
|
| 73 |
It behaves as if the :envvar:`PYTHONASYNCIODEBUG` environment variable is set |
| 74 |
to ``1``. |
| 75 |
|
| 76 |
* Check the *encoding* and *errors* arguments for string encoding and decoding |
| 77 |
operations. Examples: :func:`open`, :meth:`str.encode` and |
| 78 |
:meth:`bytes.decode`. |
| 79 |
|
| 80 |
By default, for best performance, the *errors* argument is only checked at |
| 81 |
the first encoding/decoding error and the *encoding* argument is sometimes |
| 82 |
ignored for empty strings. |
| 83 |
|
| 84 |
* The :class:`io.IOBase` destructor logs ``close()`` exceptions. |
| 85 |
* Set the :attr:`~sys.flags.dev_mode` attribute of :data:`sys.flags` to |
| 86 |
``True``. |
| 87 |
|
| 88 |
The Python Development Mode does not enable the :mod:`tracemalloc` module by |
| 89 |
default, because the overhead cost (to performance and memory) would be too |
| 90 |
large. Enabling the :mod:`tracemalloc` module provides additional information |
| 91 |
on the origin of some errors. For example, :exc:`ResourceWarning` logs the |
| 92 |
traceback where the resource was allocated, and a buffer overflow error logs |
| 93 |
the traceback where the memory block was allocated. |
| 94 |
|
| 95 |
The Python Development Mode does not prevent the :option:`-O` command line |
| 96 |
option from removing :keyword:`assert` statements nor from setting |
| 97 |
:const:`__debug__` to ``False``. |
| 98 |
|
| 99 |
The Python Development Mode can only be enabled at the Python startup. Its |
| 100 |
value can be read from :data:`sys.flags.dev_mode <sys.flags>`. |
| 101 |
|
| 102 |
.. versionchanged:: 3.8 |
| 103 |
The :class:`io.IOBase` destructor now logs ``close()`` exceptions. |
| 104 |
|
| 105 |
.. versionchanged:: 3.9 |
| 106 |
The *encoding* and *errors* arguments are now checked for string encoding |
| 107 |
and decoding operations. |
| 108 |
|
| 109 |
|
| 110 |
ResourceWarning Example |
| 111 |
----------------------- |
| 112 |
|
| 113 |
Example of a script counting the number of lines of the text file specified in |
| 114 |
the command line:: |
| 115 |
|
| 116 |
import sys |
| 117 |
|
| 118 |
def main(): |
| 119 |
fp = open(sys.argv[1]) |
| 120 |
nlines = len(fp.readlines()) |
| 121 |
print(nlines) |
| 122 |
# The file is closed implicitly |
| 123 |
|
| 124 |
if __name__ == "__main__": |
| 125 |
main() |
| 126 |
|
| 127 |
The script does not close the file explicitly. By default, Python does not emit |
| 128 |
any warning. Example using README.txt, which has 269 lines: |
| 129 |
|
| 130 |
.. code-block:: shell-session |
| 131 |
|
| 132 |
$ python script.py README.txt |
| 133 |
269 |
| 134 |
|
| 135 |
Enabling the Python Development Mode displays a :exc:`ResourceWarning` warning: |
| 136 |
|
| 137 |
.. code-block:: shell-session |
| 138 |
|
| 139 |
$ python -X dev script.py README.txt |
| 140 |
269 |
| 141 |
script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='README.rst' mode='r' encoding='UTF-8'> |
| 142 |
main() |
| 143 |
ResourceWarning: Enable tracemalloc to get the object allocation traceback |
| 144 |
|
| 145 |
In addition, enabling :mod:`tracemalloc` shows the line where the file was |
| 146 |
opened: |
| 147 |
|
| 148 |
.. code-block:: shell-session |
| 149 |
|
| 150 |
$ python -X dev -X tracemalloc=5 script.py README.rst |
| 151 |
269 |
| 152 |
script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='README.rst' mode='r' encoding='UTF-8'> |
| 153 |
main() |
| 154 |
Object allocated at (most recent call last): |
| 155 |
File "script.py", lineno 10 |
| 156 |
main() |
| 157 |
File "script.py", lineno 4 |
| 158 |
fp = open(sys.argv[1]) |
| 159 |
|
| 160 |
The fix is to close explicitly the file. Example using a context manager:: |
| 161 |
|
| 162 |
def main(): |
| 163 |
# Close the file explicitly when exiting the with block |
| 164 |
with open(sys.argv[1]) as fp: |
| 165 |
nlines = len(fp.readlines()) |
| 166 |
print(nlines) |
| 167 |
|
| 168 |
Not closing a resource explicitly can leave a resource open for way longer than |
| 169 |
expected; it can cause severe issues upon exiting Python. It is bad in |
| 170 |
CPython, but it is even worse in PyPy. Closing resources explicitly makes an |
| 171 |
application more deterministic and more reliable. |
| 172 |
|
| 173 |
|
| 174 |
Bad file descriptor error example |
| 175 |
--------------------------------- |
| 176 |
|
| 177 |
Script displaying the first line of itself:: |
| 178 |
|
| 179 |
import os |
| 180 |
|
| 181 |
def main(): |
| 182 |
fp = open(__file__) |
| 183 |
firstline = fp.readline() |
| 184 |
print(firstline.rstrip()) |
| 185 |
os.close(fp.fileno()) |
| 186 |
# The file is closed implicitly |
| 187 |
|
| 188 |
main() |
| 189 |
|
| 190 |
By default, Python does not emit any warning: |
| 191 |
|
| 192 |
.. code-block:: shell-session |
| 193 |
|
| 194 |
$ python script.py |
| 195 |
import os |
| 196 |
|
| 197 |
The Python Development Mode shows a :exc:`ResourceWarning` and logs a "Bad file |
| 198 |
descriptor" error when finalizing the file object: |
| 199 |
|
| 200 |
.. code-block:: shell-session |
| 201 |
|
| 202 |
$ python -X dev script.py |
| 203 |
import os |
| 204 |
script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='script.py' mode='r' encoding='UTF-8'> |
| 205 |
main() |
| 206 |
ResourceWarning: Enable tracemalloc to get the object allocation traceback |
| 207 |
Exception ignored in: <_io.TextIOWrapper name='script.py' mode='r' encoding='UTF-8'> |
| 208 |
Traceback (most recent call last): |
| 209 |
File "script.py", line 10, in <module> |
| 210 |
main() |
| 211 |
OSError: [Errno 9] Bad file descriptor |
| 212 |
|
| 213 |
``os.close(fp.fileno())`` closes the file descriptor. When the file object |
| 214 |
finalizer tries to close the file descriptor again, it fails with the ``Bad |
| 215 |
file descriptor`` error. A file descriptor must be closed only once. In the |
| 216 |
worst case scenario, closing it twice can lead to a crash (see :issue:`18748` |
| 217 |
for an example). |
| 218 |
|
| 219 |
The fix is to remove the ``os.close(fp.fileno())`` line, or open the file with |
| 220 |
``closefd=False``. |