Debian Codegraph @

Doc/whatsnew/3.7.rst

raw
1 ****************************
2 What's New In Python 3.7
3 ****************************
4
5 :Editor: Elvis Pranskevichus <[email protected]>
6
7 .. Rules for maintenance:
8
9 * Anyone can add text to this document. Do not spend very much time
10 on the wording of your changes, because your text will probably
11 get rewritten to some degree.
12
13 * The maintainer will go through Misc/NEWS periodically and add
14 changes; it's therefore more important to add your changes to
15 Misc/NEWS than to this file.
16
17 * This is not a complete list of every single change; completeness
18 is the purpose of Misc/NEWS. Some changes I consider too small
19 or esoteric to include. If such a change is added to the text,
20 I'll just remove it. (This is another reason you shouldn't spend
21 too much time on writing your addition.)
22
23 * If you want to draw your new text to the attention of the
24 maintainer, add 'XXX' to the beginning of the paragraph or
25 section.
26
27 * It's OK to just add a fragmentary note about a change. For
28 example: "XXX Describe the transmogrify() function added to the
29 socket module." The maintainer will research the change and
30 write the necessary text.
31
32 * You can comment out your additions if you like, but it's not
33 necessary (especially when a final release is some months away).
34
35 * Credit the author of a patch or bugfix. Just the name is
36 sufficient; the e-mail address isn't necessary.
37
38 * It's helpful to add the bug/patch number as a comment:
39
40 XXX Describe the transmogrify() function added to the socket
41 module.
42 (Contributed by P.Y. Developer in :issue:`12345`.)
43
44 This saves the maintainer the effort of going through the Git log
45 when researching a change.
46
47 This article explains the new features in Python 3.7, compared to 3.6.
48 Python 3.7 was released on June 27, 2018.
49 For full details, see the :ref:`changelog <changelog>`.
50
51
52 Summary -- Release Highlights
53 =============================
54
55 .. This section singles out the most important changes in Python 3.7.
56 Brevity is key.
57
58 New syntax features:
59
60 * :ref:`PEP 563 <whatsnew37-pep563>`, postponed evaluation of type annotations.
61
62 Backwards incompatible syntax changes:
63
64 * :keyword:`async` and :keyword:`await` are now reserved keywords.
65
66 New library modules:
67
68 * :mod:`contextvars`: :ref:`PEP 567 -- Context Variables <whatsnew37-pep567>`
69 * :mod:`dataclasses`: :ref:`PEP 557 -- Data Classes <whatsnew37-pep557>`
70 * :ref:`whatsnew37_importlib_resources`
71
72 New built-in features:
73
74 * :ref:`PEP 553 <whatsnew37-pep553>`, the new :func:`breakpoint` function.
75
76 Python data model improvements:
77
78 * :ref:`PEP 562 <whatsnew37-pep562>`, customization of access to
79 module attributes.
80
81 * :ref:`PEP 560 <whatsnew37-pep560>`, core support for typing module and
82 generic types.
83
84 * the insertion-order preservation nature of :ref:`dict <typesmapping>`
85 objects `has been declared`_ to be an official
86 part of the Python language spec.
87
88 .. _has been declared: https://mail.python.org/pipermail/python-dev/2017-December/151283.html
89
90 Significant improvements in the standard library:
91
92 * The :mod:`asyncio` module has received new features, significant
93 :ref:`usability and performance improvements <whatsnew37_asyncio>`.
94
95 * The :mod:`time` module gained support for
96 :ref:`functions with nanosecond resolution <whatsnew37-pep564>`.
97
98 CPython implementation improvements:
99
100 * Avoiding the use of ASCII as a default text encoding:
101
102 * :ref:`PEP 538 <whatsnew37-pep538>`, legacy C locale coercion
103 * :ref:`PEP 540 <whatsnew37-pep540>`, forced UTF-8 runtime mode
104 * :ref:`PEP 552 <whatsnew37-pep552>`, deterministic .pycs
105 * :ref:`New Python Development Mode <whatsnew37-devmode>`
106 * :ref:`PEP 565 <whatsnew37-pep565>`, improved :exc:`DeprecationWarning`
107 handling
108
109 C API improvements:
110
111 * :ref:`PEP 539 <whatsnew37-pep539>`, new C API for thread-local storage
112
113 Documentation improvements:
114
115 * :ref:`PEP 545 <whatsnew37-pep545>`, Python documentation translations
116 * New documentation translations: `Japanese <https://docs.python.org/ja/>`_,
117 `French <https://docs.python.org/fr/>`_, and
118 `Korean <https://docs.python.org/ko/>`_.
119
120 This release features notable performance improvements in many areas.
121 The :ref:`whatsnew37-perf` section lists them in detail.
122
123 For a list of changes that may affect compatibility with previous Python
124 releases please refer to the :ref:`porting-to-python-37` section.
125
126
127 New Features
128 ============
129
130 .. _whatsnew37-pep563:
131
132 PEP 563: Postponed Evaluation of Annotations
133 --------------------------------------------
134
135 The advent of type hints in Python uncovered two glaring usability issues
136 with the functionality of annotations added in :pep:`3107` and refined
137 further in :pep:`526`:
138
139 * annotations could only use names which were already available in the
140 current scope, in other words they didn't support forward references
141 of any kind; and
142
143 * annotating source code had adverse effects on startup time of Python
144 programs.
145
146 Both of these issues are fixed by postponing the evaluation of
147 annotations. Instead of compiling code which executes expressions in
148 annotations at their definition time, the compiler stores the annotation
149 in a string form equivalent to the AST of the expression in question.
150 If needed, annotations can be resolved at runtime using
151 :func:`typing.get_type_hints`. In the common case where this is not
152 required, the annotations are cheaper to store (since short strings
153 are interned by the interpreter) and make startup time faster.
154
155 Usability-wise, annotations now support forward references, making the
156 following syntax valid::
157
158 class C:
159 @classmethod
160 def from_string(cls, source: str) -> C:
161 ...
162
163 def validate_b(self, obj: B) -> bool:
164 ...
165
166 class B:
167 ...
168
169 Since this change breaks compatibility, the new behavior needs to be enabled
170 on a per-module basis in Python 3.7 using a :mod:`__future__` import::
171
172 from __future__ import annotations
173
174 It will become the default in Python 3.10.
175
176 .. seealso::
177
178 :pep:`563` -- Postponed evaluation of annotations
179 PEP written and implemented by Ɓukasz Langa.
180
181
182 .. _whatsnew37-pep538:
183
184 PEP 538: Legacy C Locale Coercion
185 ---------------------------------
186
187 An ongoing challenge within the Python 3 series has been determining a sensible
188 default strategy for handling the "7-bit ASCII" text encoding assumption
189 currently implied by the use of the default C or POSIX locale on non-Windows
190 platforms.
191
192 :pep:`538` updates the default interpreter command line interface to
193 automatically coerce that locale to an available UTF-8 based locale as
194 described in the documentation of the new :envvar:`PYTHONCOERCECLOCALE`
195 environment variable. Automatically setting ``LC_CTYPE`` this way means that
196 both the core interpreter and locale-aware C extensions (such as
197 :mod:`readline`) will assume the use of UTF-8 as the default text encoding,
198 rather than ASCII.
199
200 The platform support definition in :pep:`11` has also been updated to limit
201 full text handling support to suitably configured non-ASCII based locales.
202
203 As part of this change, the default error handler for :data:`~sys.stdin` and
204 :data:`~sys.stdout` is now ``surrogateescape`` (rather than ``strict``) when
205 using any of the defined coercion target locales (currently ``C.UTF-8``,
206 ``C.utf8``, and ``UTF-8``). The default error handler for :data:`~sys.stderr`
207 continues to be ``backslashreplace``, regardless of locale.
208
209 Locale coercion is silent by default, but to assist in debugging potentially
210 locale related integration problems, explicit warnings (emitted directly on
211 :data:`~sys.stderr`) can be requested by setting ``PYTHONCOERCECLOCALE=warn``.
212 This setting will also cause the Python runtime to emit a warning if the
213 legacy C locale remains active when the core interpreter is initialized.
214
215 While :pep:`538`'s locale coercion has the benefit of also affecting extension
216 modules (such as GNU ``readline``), as well as child processes (including those
217 running non-Python applications and older versions of Python), it has the
218 downside of requiring that a suitable target locale be present on the running
219 system. To better handle the case where no suitable target locale is available
220 (as occurs on RHEL/CentOS 7, for example), Python 3.7 also implements
221 :ref:`whatsnew37-pep540`.
222
223 .. seealso::
224
225 :pep:`538` -- Coercing the legacy C locale to a UTF-8 based locale
226 PEP written and implemented by Nick Coghlan.
227
228
229 .. _whatsnew37-pep540:
230
231 PEP 540: Forced UTF-8 Runtime Mode
232 -----------------------------------
233
234 The new :option:`-X` ``utf8`` command line option and :envvar:`PYTHONUTF8`
235 environment variable can be used to enable the :ref:`Python UTF-8 Mode
236 <utf8-mode>`.
237
238 When in UTF-8 mode, CPython ignores the locale settings, and uses the
239 UTF-8 encoding by default. The error handlers for :data:`sys.stdin` and
240 :data:`sys.stdout` streams are set to ``surrogateescape``.
241
242 The forced UTF-8 mode can be used to change the text handling behavior in
243 an embedded Python interpreter without changing the locale settings of
244 an embedding application.
245
246 While :pep:`540`'s UTF-8 mode has the benefit of working regardless of which
247 locales are available on the running system, it has the downside of having no
248 effect on extension modules (such as GNU ``readline``), child processes running
249 non-Python applications, and child processes running older versions of Python.
250 To reduce the risk of corrupting text data when communicating with such
251 components, Python 3.7 also implements :ref:`whatsnew37-pep540`).
252
253 The UTF-8 mode is enabled by default when the locale is ``C`` or ``POSIX``, and
254 the :pep:`538` locale coercion feature fails to change it to a UTF-8 based
255 alternative (whether that failure is due to ``PYTHONCOERCECLOCALE=0`` being set,
256 ``LC_ALL`` being set, or the lack of a suitable target locale).
257
258 .. seealso::
259
260 :pep:`540` -- Add a new UTF-8 mode
261 PEP written and implemented by Victor Stinner
262
263
264 .. _whatsnew37-pep553:
265
266 PEP 553: Built-in ``breakpoint()``
267 ----------------------------------
268
269 Python 3.7 includes the new built-in :func:`breakpoint` function as
270 an easy and consistent way to enter the Python debugger.
271
272 Built-in ``breakpoint()`` calls :func:`sys.breakpointhook`. By default, the
273 latter imports :mod:`pdb` and then calls ``pdb.set_trace()``, but by binding
274 ``sys.breakpointhook()`` to the function of your choosing, ``breakpoint()`` can
275 enter any debugger. Additionally, the environment variable
276 :envvar:`PYTHONBREAKPOINT` can be set to the callable of your debugger of
277 choice. Set ``PYTHONBREAKPOINT=0`` to completely disable built-in
278 ``breakpoint()``.
279
280 .. seealso::
281
282 :pep:`553` -- Built-in breakpoint()
283 PEP written and implemented by Barry Warsaw
284
285
286 .. _whatsnew37-pep539:
287
288 PEP 539: New C API for Thread-Local Storage
289 -------------------------------------------
290
291 While Python provides a C API for thread-local storage support; the existing
292 :ref:`Thread Local Storage (TLS) API <thread-local-storage-api>` has used
293 :c:expr:`int` to represent TLS keys across all platforms. This has not
294 generally been a problem for officially support platforms, but that is neither
295 POSIX-compliant, nor portable in any practical sense.
296
297 :pep:`539` changes this by providing a new :ref:`Thread Specific Storage (TSS)
298 API <thread-specific-storage-api>` to CPython which supersedes use of the
299 existing TLS API within the CPython interpreter, while deprecating the existing
300 API. The TSS API uses a new type :c:type:`Py_tss_t` instead of :c:expr:`int`
301 to represent TSS keys--an opaque type the definition of which may depend on
302 the underlying TLS implementation. Therefore, this will allow to build CPython
303 on platforms where the native TLS key is defined in a way that cannot be safely
304 cast to :c:expr:`int`.
305
306 Note that on platforms where the native TLS key is defined in a way that cannot
307 be safely cast to :c:expr:`int`, all functions of the existing TLS API will be
308 no-op and immediately return failure. This indicates clearly that the old API
309 is not supported on platforms where it cannot be used reliably, and that no
310 effort will be made to add such support.
311
312 .. seealso::
313
314 :pep:`539` -- A New C-API for Thread-Local Storage in CPython
315 PEP written by Erik M. Bray; implementation by Masayuki Yamamoto.
316
317
318 .. _whatsnew37-pep562:
319
320 PEP 562: Customization of Access to Module Attributes
321 -----------------------------------------------------
322
323 Python 3.7 allows defining :meth:`~module.__getattr__` on modules and will call
324 it whenever a module attribute is otherwise not found. Defining
325 :meth:`~module.__dir__` on modules is now also allowed.
326
327 A typical example of where this may be useful is module attribute deprecation
328 and lazy loading.
329
330 .. seealso::
331
332 :pep:`562` -- Module ``__getattr__`` and ``__dir__``
333 PEP written and implemented by Ivan Levkivskyi
334
335
336 .. _whatsnew37-pep564:
337
338 PEP 564: New Time Functions With Nanosecond Resolution
339 ------------------------------------------------------
340
341 The resolution of clocks in modern systems can exceed the limited precision
342 of a floating-point number returned by the :func:`time.time` function
343 and its variants. To avoid loss of precision, :pep:`564` adds six new
344 "nanosecond" variants of the existing timer functions to the :mod:`time`
345 module:
346
347 * :func:`time.clock_gettime_ns`
348 * :func:`time.clock_settime_ns`
349 * :func:`time.monotonic_ns`
350 * :func:`time.perf_counter_ns`
351 * :func:`time.process_time_ns`
352 * :func:`time.time_ns`
353
354 The new functions return the number of nanoseconds as an integer value.
355
356 :pep:`Measurements <0564#annex-clocks-resolution-in-python>`
357 show that on Linux and Windows the resolution of :func:`time.time_ns` is
358 approximately 3 times better than that of :func:`time.time`.
359
360 .. seealso::
361
362 :pep:`564` -- Add new time functions with nanosecond resolution
363 PEP written and implemented by Victor Stinner
364
365
366 .. _whatsnew37-pep565:
367
368 PEP 565: Show DeprecationWarning in ``__main__``
369 ------------------------------------------------
370
371 The default handling of :exc:`DeprecationWarning` has been changed such that
372 these warnings are once more shown by default, but only when the code
373 triggering them is running directly in the :mod:`__main__` module. As a result,
374 developers of single file scripts and those using Python interactively should
375 once again start seeing deprecation warnings for the APIs they use, but
376 deprecation warnings triggered by imported application, library and framework
377 modules will continue to be hidden by default.
378
379 As a result of this change, the standard library now allows developers to choose
380 between three different deprecation warning behaviours:
381
382 * :exc:`FutureWarning`: always displayed by default, recommended for warnings
383 intended to be seen by application end users (e.g. for deprecated application
384 configuration settings).
385 * :exc:`DeprecationWarning`: displayed by default only in :mod:`__main__` and when
386 running tests, recommended for warnings intended to be seen by other Python
387 developers where a version upgrade may result in changed behaviour or an
388 error.
389 * :exc:`PendingDeprecationWarning`: displayed by default only when running
390 tests, intended for cases where a future version upgrade will change the
391 warning category to :exc:`DeprecationWarning` or :exc:`FutureWarning`.
392
393 Previously both :exc:`DeprecationWarning` and :exc:`PendingDeprecationWarning`
394 were only visible when running tests, which meant that developers primarily
395 writing single file scripts or using Python interactively could be surprised
396 by breaking changes in the APIs they used.
397
398 .. seealso::
399
400 :pep:`565` -- Show DeprecationWarning in ``__main__``
401 PEP written and implemented by Nick Coghlan
402
403
404 .. _whatsnew37-pep560:
405
406 PEP 560: Core Support for ``typing`` module and Generic Types
407 -------------------------------------------------------------
408
409 Initially :pep:`484` was designed in such way that it would not introduce *any*
410 changes to the core CPython interpreter. Now type hints and the :mod:`typing`
411 module are extensively used by the community, so this restriction is removed.
412 The PEP introduces two special methods :meth:`~object.__class_getitem__` and
413 :meth:`~object.__mro_entries__`, these methods are now used by most classes and special
414 constructs in :mod:`typing`. As a result, the speed of various operations
415 with types increased up to 7 times, the generic types can be used without
416 metaclass conflicts, and several long standing bugs in :mod:`typing` module are
417 fixed.
418
419 .. seealso::
420
421 :pep:`560` -- Core support for typing module and generic types
422 PEP written and implemented by Ivan Levkivskyi
423
424
425 .. _whatsnew37-pep552:
426
427 PEP 552: Hash-based .pyc Files
428 ------------------------------
429
430 Python has traditionally checked the up-to-dateness of bytecode cache files
431 (i.e., ``.pyc`` files) by comparing the source metadata (last-modified timestamp
432 and size) with source metadata saved in the cache file header when it was
433 generated. While effective, this invalidation method has its drawbacks. When
434 filesystem timestamps are too coarse, Python can miss source updates, leading to
435 user confusion. Additionally, having a timestamp in the cache file is
436 problematic for `build reproducibility <https://reproducible-builds.org/>`_ and
437 content-based build systems.
438
439 :pep:`552` extends the pyc format to allow the hash of the source file to be
440 used for invalidation instead of the source timestamp. Such ``.pyc`` files are
441 called "hash-based". By default, Python still uses timestamp-based invalidation
442 and does not generate hash-based ``.pyc`` files at runtime. Hash-based ``.pyc``
443 files may be generated with :mod:`py_compile` or :mod:`compileall`.
444
445 Hash-based ``.pyc`` files come in two variants: checked and unchecked. Python
446 validates checked hash-based ``.pyc`` files against the corresponding source
447 files at runtime but doesn't do so for unchecked hash-based pycs. Unchecked
448 hash-based ``.pyc`` files are a useful performance optimization for environments
449 where a system external to Python (e.g., the build system) is responsible for
450 keeping ``.pyc`` files up-to-date.
451
452 See :ref:`pyc-invalidation` for more information.
453
454 .. seealso::
455
456 :pep:`552` -- Deterministic pycs
457 PEP written and implemented by Benjamin Peterson
458
459
460 .. _whatsnew37-pep545:
461
462 PEP 545: Python Documentation Translations
463 ------------------------------------------
464
465 :pep:`545` describes the process of creating and maintaining Python
466 documentation translations.
467
468 Three new translations have been added:
469
470 - Japanese: https://docs.python.org/ja/
471 - French: https://docs.python.org/fr/
472 - Korean: https://docs.python.org/ko/
473
474 .. seealso::
475
476 :pep:`545` -- Python Documentation Translations
477 PEP written and implemented by Julien Palard, Inada Naoki, and
478 Victor Stinner.
479
480
481 .. _whatsnew37-devmode:
482
483 Python Development Mode (-X dev)
484 --------------------------------
485
486 The new :option:`-X` ``dev`` command line option or the new
487 :envvar:`PYTHONDEVMODE` environment variable can be used to enable
488 :ref:`Python Development Mode <devmode>`. When in development mode, Python performs
489 additional runtime checks that are too expensive to be enabled by default.
490 See :ref:`Python Development Mode <devmode>` documentation for the full
491 description.
492
493
494 Other Language Changes
495 ======================
496
497 * An :keyword:`await` expression and comprehensions containing an
498 :keyword:`async for` clause were illegal in the expressions in
499 :ref:`formatted string literals <f-strings>` due to a problem with the
500 implementation. In Python 3.7 this restriction was lifted.
501
502 * More than 255 arguments can now be passed to a function, and a function can
503 now have more than 255 parameters. (Contributed by Serhiy Storchaka in
504 :issue:`12844` and :issue:`18896`.)
505
506 * :meth:`bytes.fromhex` and :meth:`bytearray.fromhex` now ignore all ASCII
507 whitespace, not only spaces. (Contributed by Robert Xiao in :issue:`28927`.)
508
509 * :class:`str`, :class:`bytes`, and :class:`bytearray` gained support for
510 the new :meth:`isascii() <str.isascii>` method, which can be used to
511 test if a string or bytes contain only the ASCII characters.
512 (Contributed by INADA Naoki in :issue:`32677`.)
513
514 * :exc:`ImportError` now displays module name and module ``__file__`` path when
515 ``from ... import ...`` fails. (Contributed by Matthias Bussonnier in
516 :issue:`29546`.)
517
518 * Circular imports involving absolute imports with binding a submodule to
519 a name are now supported.
520 (Contributed by Serhiy Storchaka in :issue:`30024`.)
521
522 * ``object.__format__(x, '')`` is now equivalent to ``str(x)`` rather than
523 ``format(str(self), '')``.
524 (Contributed by Serhiy Storchaka in :issue:`28974`.)
525
526 * In order to better support dynamic creation of stack traces,
527 :class:`types.TracebackType` can now be instantiated from Python code, and
528 the :attr:`~traceback.tb_next` attribute on
529 :ref:`tracebacks <traceback-objects>` is now writable.
530 (Contributed by Nathaniel J. Smith in :issue:`30579`.)
531
532 * When using the :option:`-m` switch, ``sys.path[0]`` is now eagerly expanded
533 to the full starting directory path, rather than being left as the empty
534 directory (which allows imports from the *current* working directory at the
535 time when an import occurs)
536 (Contributed by Nick Coghlan in :issue:`33053`.)
537
538 * The new :option:`-X` ``importtime`` option or the
539 :envvar:`PYTHONPROFILEIMPORTTIME` environment variable can be used to show
540 the timing of each module import.
541 (Contributed by Inada Naoki in :issue:`31415`.)
542
543
544 New Modules
545 ===========
546
547 .. _whatsnew37-pep567:
548
549 contextvars
550 -----------
551
552 The new :mod:`contextvars` module and a set of
553 :ref:`new C APIs <contextvarsobjects>` introduce
554 support for *context variables*. Context variables are conceptually
555 similar to thread-local variables. Unlike TLS, context variables
556 support asynchronous code correctly.
557
558 The :mod:`asyncio` and :mod:`decimal` modules have been updated to use
559 and support context variables out of the box. Particularly the active
560 decimal context is now stored in a context variable, which allows
561 decimal operations to work with the correct context in asynchronous code.
562
563 .. seealso::
564
565 :pep:`567` -- Context Variables
566 PEP written and implemented by Yury Selivanov
567
568
569 .. _whatsnew37-pep557:
570
571 dataclasses
572 -----------
573
574 The new :func:`~dataclasses.dataclass` decorator provides a way to declare
575 *data classes*. A data class describes its attributes using class variable
576 annotations. Its constructor and other magic methods, such as
577 :meth:`~object.__repr__`, :meth:`~object.__eq__`, and
578 :meth:`~object.__hash__` are generated automatically.
579
580 Example::
581
582 @dataclass
583 class Point:
584 x: float
585 y: float
586 z: float = 0.0
587
588 p = Point(1.5, 2.5)
589 print(p) # produces "Point(x=1.5, y=2.5, z=0.0)"
590
591 .. seealso::
592
593 :pep:`557` -- Data Classes
594 PEP written and implemented by Eric V. Smith
595
596
597 .. _whatsnew37_importlib_resources:
598
599 importlib.resources
600 -------------------
601
602 The new :mod:`importlib.resources` module provides several new APIs and one
603 new ABC for access to, opening, and reading *resources* inside packages.
604 Resources are roughly similar to files inside packages, but they needn't
605 be actual files on the physical file system. Module loaders can provide a
606 :meth:`!get_resource_reader` function which returns
607 a :class:`!importlib.abc.ResourceReader` instance to support this
608 new API. Built-in file path loaders and zip file loaders both support this.
609
610 Contributed by Barry Warsaw and Brett Cannon in :issue:`32248`.
611
612 .. seealso::
613
614 `importlib_resources <https://importlib-resources.readthedocs.io/en/latest/>`_
615 -- a PyPI backport for earlier Python versions.
616
617
618 Improved Modules
619 ================
620
621
622 argparse
623 --------
624
625 The new :meth:`ArgumentParser.parse_intermixed_args()
626 <argparse.ArgumentParser.parse_intermixed_args>`
627 method allows intermixing options and positional arguments.
628 (Contributed by paul.j3 in :issue:`14191`.)
629
630
631 .. _whatsnew37_asyncio:
632
633 asyncio
634 -------
635
636 The :mod:`asyncio` module has received many new features, usability and
637 :ref:`performance improvements <whatsnew37-asyncio-perf>`. Notable changes
638 include:
639
640 * The new :term:`provisional <provisional API>` :func:`asyncio.run` function can
641 be used to run a coroutine from synchronous code by automatically creating and
642 destroying the event loop.
643 (Contributed by Yury Selivanov in :issue:`32314`.)
644
645 * asyncio gained support for :mod:`contextvars`.
646 :meth:`loop.call_soon() <asyncio.loop.call_soon>`,
647 :meth:`loop.call_soon_threadsafe() <asyncio.loop.call_soon_threadsafe>`,
648 :meth:`loop.call_later() <asyncio.loop.call_later>`,
649 :meth:`loop.call_at() <asyncio.loop.call_at>`, and
650 :meth:`Future.add_done_callback() <asyncio.Future.add_done_callback>`
651 have a new optional keyword-only *context* parameter.
652 :class:`Tasks <asyncio.Task>` now track their context automatically.
653 See :pep:`567` for more details.
654 (Contributed by Yury Selivanov in :issue:`32436`.)
655
656 * The new :func:`asyncio.create_task` function has been added as a shortcut
657 to ``asyncio.get_event_loop().create_task()``.
658 (Contributed by Andrew Svetlov in :issue:`32311`.)
659
660 * The new :meth:`loop.start_tls() <asyncio.loop.start_tls>`
661 method can be used to upgrade an existing connection to TLS.
662 (Contributed by Yury Selivanov in :issue:`23749`.)
663
664 * The new :meth:`loop.sock_recv_into() <asyncio.loop.sock_recv_into>`
665 method allows reading data from a socket directly into a provided buffer making
666 it possible to reduce data copies.
667 (Contributed by Antoine Pitrou in :issue:`31819`.)
668
669 * The new :func:`asyncio.current_task` function returns the currently running
670 :class:`~asyncio.Task` instance, and the new :func:`asyncio.all_tasks`
671 function returns a set of all existing ``Task`` instances in a given loop.
672 The :meth:`!Task.current_task` and
673 :meth:`!Task.all_tasks` methods have been deprecated.
674 (Contributed by Andrew Svetlov in :issue:`32250`.)
675
676 * The new *provisional* :class:`~asyncio.BufferedProtocol` class allows
677 implementing streaming protocols with manual control over the receive buffer.
678 (Contributed by Yury Selivanov in :issue:`32251`.)
679
680 * The new :func:`asyncio.get_running_loop` function returns the currently
681 running loop, and raises a :exc:`RuntimeError` if no loop is running.
682 This is in contrast with :func:`asyncio.get_event_loop`, which will *create*
683 a new event loop if none is running.
684 (Contributed by Yury Selivanov in :issue:`32269`.)
685
686 * The new :meth:`StreamWriter.wait_closed() <asyncio.StreamWriter.wait_closed>`
687 coroutine method allows waiting until the stream writer is closed. The new
688 :meth:`StreamWriter.is_closing() <asyncio.StreamWriter.is_closing>` method
689 can be used to determine if the writer is closing.
690 (Contributed by Andrew Svetlov in :issue:`32391`.)
691
692 * The new :meth:`loop.sock_sendfile() <asyncio.loop.sock_sendfile>`
693 coroutine method allows sending files using :mod:`os.sendfile` when possible.
694 (Contributed by Andrew Svetlov in :issue:`32410`.)
695
696 * The new :meth:`Future.get_loop() <asyncio.Future.get_loop>` and
697 ``Task.get_loop()`` methods return the instance of the loop on which a task or
698 a future were created.
699 :meth:`Server.get_loop() <asyncio.Server.get_loop>` allows doing the same for
700 :class:`asyncio.Server` objects.
701 (Contributed by Yury Selivanov in :issue:`32415` and
702 Srinivas Reddy Thatiparthy in :issue:`32418`.)
703
704 * It is now possible to control how instances of :class:`asyncio.Server` begin
705 serving. Previously, the server would start serving immediately when created.
706 The new *start_serving* keyword argument to
707 :meth:`loop.create_server() <asyncio.loop.create_server>` and
708 :meth:`loop.create_unix_server() <asyncio.loop.create_unix_server>`,
709 as well as :meth:`Server.start_serving() <asyncio.Server.start_serving>`, and
710 :meth:`Server.serve_forever() <asyncio.Server.serve_forever>`
711 can be used to decouple server instantiation and serving. The new
712 :meth:`Server.is_serving() <asyncio.Server.is_serving>` method returns ``True``
713 if the server is serving. :class:`~asyncio.Server` objects are now
714 asynchronous context managers::
715
716 srv = await loop.create_server(...)
717
718 async with srv:
719 # some code
720
721 # At this point, srv is closed and no longer accepts new connections.
722
723 (Contributed by Yury Selivanov in :issue:`32662`.)
724
725 * Callback objects returned by
726 :func:`loop.call_later() <asyncio.loop.call_later>`
727 gained the new :meth:`when() <asyncio.TimerHandle.when>` method which
728 returns an absolute scheduled callback timestamp.
729 (Contributed by Andrew Svetlov in :issue:`32741`.)
730
731 * The :meth:`loop.create_datagram_endpoint() \
732 <asyncio.loop.create_datagram_endpoint>` method
733 gained support for Unix sockets.
734 (Contributed by Quentin Dawans in :issue:`31245`.)
735
736 * The :func:`asyncio.open_connection`, :func:`asyncio.start_server` functions,
737 :meth:`loop.create_connection() <asyncio.loop.create_connection>`,
738 :meth:`loop.create_server() <asyncio.loop.create_server>`,
739 :meth:`loop.create_accepted_socket() <asyncio.loop.connect_accepted_socket>`
740 methods and their corresponding UNIX socket variants now accept the
741 *ssl_handshake_timeout* keyword argument.
742 (Contributed by Neil Aspinall in :issue:`29970`.)
743
744 * The new :meth:`Handle.cancelled() <asyncio.Handle.cancelled>` method returns
745 ``True`` if the callback was cancelled.
746 (Contributed by Marat Sharafutdinov in :issue:`31943`.)
747
748 * The asyncio source has been converted to use the
749 :keyword:`async`/:keyword:`await` syntax.
750 (Contributed by Andrew Svetlov in :issue:`32193`.)
751
752 * The new :meth:`ReadTransport.is_reading() <asyncio.ReadTransport.is_reading>`
753 method can be used to determine the reading state of the transport.
754 Additionally, calls to
755 :meth:`ReadTransport.resume_reading() <asyncio.ReadTransport.resume_reading>`
756 and :meth:`ReadTransport.pause_reading() <asyncio.ReadTransport.pause_reading>`
757 are now idempotent.
758 (Contributed by Yury Selivanov in :issue:`32356`.)
759
760 * Loop methods which accept socket paths now support passing
761 :term:`path-like objects <path-like object>`.
762 (Contributed by Yury Selivanov in :issue:`32066`.)
763
764 * In :mod:`asyncio` TCP sockets on Linux are now created with ``TCP_NODELAY``
765 flag set by default.
766 (Contributed by Yury Selivanov and Victor Stinner in :issue:`27456`.)
767
768 * Exceptions occurring in cancelled tasks are no longer logged.
769 (Contributed by Yury Selivanov in :issue:`30508`.)
770
771 * New ``WindowsSelectorEventLoopPolicy`` and
772 ``WindowsProactorEventLoopPolicy`` classes.
773 (Contributed by Yury Selivanov in :issue:`33792`.)
774
775 Several ``asyncio`` APIs have been
776 :ref:`deprecated <whatsnew37-asyncio-deprecated>`.
777
778
779 binascii
780 --------
781
782 The :func:`~binascii.b2a_uu` function now accepts an optional *backtick*
783 keyword argument. When it's true, zeros are represented by ``'`'``
784 instead of spaces. (Contributed by Xiang Zhang in :issue:`30103`.)
785
786
787 calendar
788 --------
789
790 The :class:`~calendar.HTMLCalendar` class has new class attributes which ease
791 the customization of CSS classes in the produced HTML calendar.
792 (Contributed by Oz Tiram in :issue:`30095`.)
793
794
795 collections
796 -----------
797
798 ``collections.namedtuple()`` now supports default values.
799 (Contributed by Raymond Hettinger in :issue:`32320`.)
800
801
802 compileall
803 ----------
804
805 :func:`compileall.compile_dir` learned the new *invalidation_mode* parameter,
806 which can be used to enable
807 :ref:`hash-based .pyc invalidation <whatsnew37-pep552>`. The invalidation
808 mode can also be specified on the command line using the new
809 ``--invalidation-mode`` argument.
810 (Contributed by Benjamin Peterson in :issue:`31650`.)
811
812
813 concurrent.futures
814 ------------------
815
816 :class:`ProcessPoolExecutor <concurrent.futures.ProcessPoolExecutor>` and
817 :class:`ThreadPoolExecutor <concurrent.futures.ThreadPoolExecutor>` now
818 support the new *initializer* and *initargs* constructor arguments.
819 (Contributed by Antoine Pitrou in :issue:`21423`.)
820
821 The :class:`ProcessPoolExecutor <concurrent.futures.ProcessPoolExecutor>`
822 can now take the multiprocessing context via the new *mp_context* argument.
823 (Contributed by Thomas Moreau in :issue:`31540`.)
824
825
826 contextlib
827 ----------
828
829 The new :func:`~contextlib.nullcontext` is a simpler and faster no-op
830 context manager than :class:`~contextlib.ExitStack`.
831 (Contributed by Jesse-Bakker in :issue:`10049`.)
832
833 The new :func:`~contextlib.asynccontextmanager`,
834 :class:`~contextlib.AbstractAsyncContextManager`, and
835 :class:`~contextlib.AsyncExitStack` have been added to
836 complement their synchronous counterparts. (Contributed
837 by Jelle Zijlstra in :issue:`29679` and :issue:`30241`,
838 and by Alexander Mohr and Ilya Kulakov in :issue:`29302`.)
839
840
841 cProfile
842 --------
843
844 The :mod:`cProfile` command line now accepts ``-m module_name`` as an
845 alternative to script path. (Contributed by Sanyam Khurana in :issue:`21862`.)
846
847
848 crypt
849 -----
850
851 The :mod:`!crypt` module now supports the Blowfish hashing method.
852 (Contributed by Serhiy Storchaka in :issue:`31664`.)
853
854 The :func:`!mksalt` function now allows specifying the number of rounds
855 for hashing. (Contributed by Serhiy Storchaka in :issue:`31702`.)
856
857
858 datetime
859 --------
860
861 The new :meth:`datetime.fromisoformat() <datetime.datetime.fromisoformat>`
862 method constructs a :class:`~datetime.datetime` object from a string
863 in one of the formats output by
864 :meth:`datetime.isoformat() <datetime.datetime.isoformat>`.
865 (Contributed by Paul Ganssle in :issue:`15873`.)
866
867 The :class:`tzinfo <datetime.tzinfo>` class now supports sub-minute offsets.
868 (Contributed by Alexander Belopolsky in :issue:`5288`.)
869
870
871 dbm
872 ---
873
874 :mod:`dbm.dumb` now supports reading read-only files and no longer writes the
875 index file when it is not changed.
876
877
878 decimal
879 -------
880
881 The :mod:`decimal` module now uses :ref:`context variables <whatsnew37-pep567>`
882 to store the decimal context.
883 (Contributed by Yury Selivanov in :issue:`32630`.)
884
885
886 dis
887 ---
888
889 The :func:`~dis.dis` function is now able to
890 disassemble nested code objects (the code of comprehensions, generator
891 expressions and nested functions, and the code used for building nested
892 classes). The maximum depth of disassembly recursion is controlled by
893 the new *depth* parameter.
894 (Contributed by Serhiy Storchaka in :issue:`11822`.)
895
896
897 distutils
898 ---------
899
900 ``README.rst`` is now included in the list of distutils standard READMEs and
901 therefore included in source distributions.
902 (Contributed by Ryan Gonzalez in :issue:`11913`.)
903
904
905 enum
906 ----
907
908 The :class:`Enum <enum.Enum>` learned the new ``_ignore_`` class property,
909 which allows listing the names of properties which should not become
910 enum members.
911 (Contributed by Ethan Furman in :issue:`31801`.)
912
913 In Python 3.8, attempting to check for non-Enum objects in :class:`~enum.Enum`
914 classes will raise a :exc:`TypeError` (e.g. ``1 in Color``); similarly,
915 attempting to check for non-Flag objects in a :class:`~enum.Flag` member will
916 raise :exc:`TypeError` (e.g. ``1 in Perm.RW``); currently, both operations
917 return :const:`False` instead and are deprecated.
918 (Contributed by Ethan Furman in :issue:`33217`.)
919
920
921 functools
922 ---------
923
924 :func:`functools.singledispatch` now supports registering implementations
925 using type annotations.
926 (Contributed by Ɓukasz Langa in :issue:`32227`.)
927
928
929 gc
930 --
931
932 The new :func:`gc.freeze` function allows freezing all objects tracked
933 by the garbage collector and excluding them from future collections.
934 This can be used before a POSIX ``fork()`` call to make the GC copy-on-write
935 friendly or to speed up collection. The new :func:`gc.unfreeze` functions
936 reverses this operation. Additionally, :func:`gc.get_freeze_count` can
937 be used to obtain the number of frozen objects.
938 (Contributed by Li Zekun in :issue:`31558`.)
939
940
941 hmac
942 ----
943
944 The :mod:`hmac` module now has an optimized one-shot :func:`~hmac.digest`
945 function, which is up to three times faster than :func:`~hmac.HMAC`.
946 (Contributed by Christian Heimes in :issue:`32433`.)
947
948
949 http.client
950 -----------
951
952 :class:`~http.client.HTTPConnection` and :class:`~http.client.HTTPSConnection`
953 now support the new *blocksize* argument for improved upload throughput.
954 (Contributed by Nir Soffer in :issue:`31945`.)
955
956
957 http.server
958 -----------
959
960 :class:`~http.server.SimpleHTTPRequestHandler` now supports the HTTP
961 ``If-Modified-Since`` header. The server returns the 304 response status if
962 the target file was not modified after the time specified in the header.
963 (Contributed by Pierre Quentel in :issue:`29654`.)
964
965 :class:`~http.server.SimpleHTTPRequestHandler` accepts the new *directory*
966 argument, in addition to the new ``--directory`` command line argument.
967 With this parameter, the server serves the specified directory, by default it
968 uses the current working directory.
969 (Contributed by Stéphane Wirtel and Julien Palard in :issue:`28707`.)
970
971 The new :class:`ThreadingHTTPServer <http.server.ThreadingHTTPServer>` class
972 uses threads to handle requests using :class:`~socketserver.ThreadingMixIn`.
973 It is used when ``http.server`` is run with ``-m``.
974 (Contributed by Julien Palard in :issue:`31639`.)
975
976
977 idlelib and IDLE
978 ----------------
979
980 Multiple fixes for autocompletion. (Contributed by Louie Lu in :issue:`15786`.)
981
982 Module Browser (on the File menu, formerly called Class Browser),
983 now displays nested functions and classes in addition to top-level
984 functions and classes.
985 (Contributed by Guilherme Polo, Cheryl Sabella, and Terry Jan Reedy
986 in :issue:`1612262`.)
987
988 The Settings dialog (Options, Configure IDLE) has been partly rewritten
989 to improve both appearance and function.
990 (Contributed by Cheryl Sabella and Terry Jan Reedy in multiple issues.)
991
992 The font sample now includes a selection of non-Latin characters so that
993 users can better see the effect of selecting a particular font.
994 (Contributed by Terry Jan Reedy in :issue:`13802`.)
995 The sample can be edited to include other characters.
996 (Contributed by Serhiy Storchaka in :issue:`31860`.)
997
998 The IDLE features formerly implemented as extensions have been reimplemented
999 as normal features. Their settings have been moved from the Extensions tab
1000 to other dialog tabs.
1001 (Contributed by Charles Wohlganger and Terry Jan Reedy in :issue:`27099`.)
1002
1003 Editor code context option revised. Box displays all context lines up to
1004 maxlines. Clicking on a context line jumps the editor to that line. Context
1005 colors for custom themes is added to Highlights tab of Settings dialog.
1006 (Contributed by Cheryl Sabella and Terry Jan Reedy in :issue:`33642`,
1007 :issue:`33768`, and :issue:`33679`.)
1008
1009 On Windows, a new API call tells Windows that tk scales for DPI. On Windows
1010 8.1+ or 10, with DPI compatibility properties of the Python binary
1011 unchanged, and a monitor resolution greater than 96 DPI, this should
1012 make text and lines sharper. It should otherwise have no effect.
1013 (Contributed by Terry Jan Reedy in :issue:`33656`.)
1014
1015 New in 3.7.1:
1016
1017 Output over N lines (50 by default) is squeezed down to a button.
1018 N can be changed in the PyShell section of the General page of the
1019 Settings dialog. Fewer, but possibly extra long, lines can be squeezed by
1020 right clicking on the output. Squeezed output can be expanded in place
1021 by double-clicking the button or into the clipboard or a separate window
1022 by right-clicking the button. (Contributed by Tal Einat in :issue:`1529353`.)
1023
1024 The changes above have been backported to 3.6 maintenance releases.
1025
1026 NEW in 3.7.4:
1027
1028 Add "Run Customized" to the Run menu to run a module with customized
1029 settings. Any command line arguments entered are added to sys.argv.
1030 They re-appear in the box for the next customized run. One can also
1031 suppress the normal Shell main module restart. (Contributed by Cheryl
1032 Sabella, Terry Jan Reedy, and others in :issue:`5680` and :issue:`37627`.)
1033
1034 New in 3.7.5:
1035
1036 Add optional line numbers for IDLE editor windows. Windows
1037 open without line numbers unless set otherwise in the General
1038 tab of the configuration dialog. Line numbers for an existing
1039 window are shown and hidden in the Options menu.
1040 (Contributed by Tal Einat and Saimadhav Heblikar in :issue:`17535`.)
1041
1042
1043 importlib
1044 ---------
1045
1046 The :class:`!importlib.abc.ResourceReader` ABC was introduced to
1047 support the loading of resources from packages. See also
1048 :ref:`whatsnew37_importlib_resources`.
1049 (Contributed by Barry Warsaw, Brett Cannon in :issue:`32248`.)
1050
1051 :func:`importlib.reload` now raises :exc:`ModuleNotFoundError` if the module
1052 lacks a spec.
1053 (Contributed by Garvit Khatri in :issue:`29851`.)
1054
1055 :func:`importlib.util.find_spec` now raises :exc:`ModuleNotFoundError` instead of
1056 :exc:`AttributeError` if the specified parent module is not a package (i.e.
1057 lacks a ``__path__`` attribute).
1058 (Contributed by Milan Oberkirch in :issue:`30436`.)
1059
1060 The new :func:`importlib.util.source_hash` can be used to compute the hash of
1061 the passed source. A :ref:`hash-based .pyc file <whatsnew37-pep552>`
1062 embeds the value returned by this function.
1063
1064
1065 io
1066 --
1067
1068 The new :meth:`TextIOWrapper.reconfigure() <io.TextIOWrapper.reconfigure>`
1069 method can be used to reconfigure the text stream with the new settings.
1070 (Contributed by Antoine Pitrou in :issue:`30526` and
1071 INADA Naoki in :issue:`15216`.)
1072
1073
1074 ipaddress
1075 ---------
1076
1077 The new ``subnet_of()`` and ``supernet_of()`` methods of
1078 :class:`ipaddress.IPv6Network` and :class:`ipaddress.IPv4Network` can
1079 be used for network containment tests.
1080 (Contributed by Michel Albert and Cheryl Sabella in :issue:`20825`.)
1081
1082
1083 itertools
1084 ---------
1085
1086 :func:`itertools.islice` now accepts
1087 :meth:`integer-like objects <object.__index__>` as start, stop,
1088 and slice arguments.
1089 (Contributed by Will Roberts in :issue:`30537`.)
1090
1091
1092 locale
1093 ------
1094
1095 The new *monetary* argument to :func:`locale.format_string` can be used
1096 to make the conversion use monetary thousands separators and
1097 grouping strings. (Contributed by Garvit in :issue:`10379`.)
1098
1099 The :func:`locale.getpreferredencoding` function now always returns ``'UTF-8'``
1100 on Android or when in the :ref:`forced UTF-8 mode <whatsnew37-pep540>`.
1101
1102
1103 logging
1104 -------
1105
1106 :class:`~logging.Logger` instances can now be pickled.
1107 (Contributed by Vinay Sajip in :issue:`30520`.)
1108
1109 The new :meth:`StreamHandler.setStream() <logging.StreamHandler.setStream>`
1110 method can be used to replace the logger stream after handler creation.
1111 (Contributed by Vinay Sajip in :issue:`30522`.)
1112
1113 It is now possible to specify keyword arguments to handler constructors in
1114 configuration passed to :func:`logging.config.fileConfig`.
1115 (Contributed by Preston Landers in :issue:`31080`.)
1116
1117
1118 math
1119 ----
1120
1121 The new :func:`math.remainder` function implements the IEEE 754-style remainder
1122 operation. (Contributed by Mark Dickinson in :issue:`29962`.)
1123
1124
1125 mimetypes
1126 ---------
1127
1128 The MIME type of .bmp has been changed from ``'image/x-ms-bmp'`` to
1129 ``'image/bmp'``.
1130 (Contributed by Nitish Chandra in :issue:`22589`.)
1131
1132
1133 msilib
1134 ------
1135
1136 The new :meth:`!Database.Close` method can be used
1137 to close the :abbr:`MSI` database.
1138 (Contributed by Berker Peksag in :issue:`20486`.)
1139
1140
1141 multiprocessing
1142 ---------------
1143
1144 The new :meth:`Process.close() <multiprocessing.Process.close>` method
1145 explicitly closes the process object and releases all resources associated
1146 with it. :exc:`ValueError` is raised if the underlying process is still
1147 running.
1148 (Contributed by Antoine Pitrou in :issue:`30596`.)
1149
1150 The new :meth:`Process.kill() <multiprocessing.Process.kill>` method can
1151 be used to terminate the process using the :data:`~signal.SIGKILL` signal on Unix.
1152 (Contributed by Vitor Pereira in :issue:`30794`.)
1153
1154 Non-daemonic threads created by :class:`~multiprocessing.Process` are now
1155 joined on process exit.
1156 (Contributed by Antoine Pitrou in :issue:`18966`.)
1157
1158
1159 os
1160 --
1161
1162 :func:`os.fwalk` now accepts the *path* argument as :class:`bytes`.
1163 (Contributed by Serhiy Storchaka in :issue:`28682`.)
1164
1165 :func:`os.scandir` gained support for :ref:`file descriptors <path_fd>`.
1166 (Contributed by Serhiy Storchaka in :issue:`25996`.)
1167
1168 The new :func:`~os.register_at_fork` function allows registering Python
1169 callbacks to be executed at process fork.
1170 (Contributed by Antoine Pitrou in :issue:`16500`.)
1171
1172 Added :func:`os.preadv` (combine the functionality of :func:`os.readv` and
1173 :func:`os.pread`) and :func:`os.pwritev` functions (combine the functionality
1174 of :func:`os.writev` and :func:`os.pwrite`). (Contributed by Pablo Galindo in
1175 :issue:`31368`.)
1176
1177 The mode argument of :func:`os.makedirs` no longer affects the file
1178 permission bits of newly created intermediate-level directories.
1179 (Contributed by Serhiy Storchaka in :issue:`19930`.)
1180
1181 :func:`os.dup2` now returns the new file descriptor. Previously, ``None``
1182 was always returned.
1183 (Contributed by Benjamin Peterson in :issue:`32441`.)
1184
1185 The structure returned by :func:`os.stat` now contains the
1186 :attr:`~os.stat_result.st_fstype` attribute on Solaris and its derivatives.
1187 (Contributed by JesĂșs Cea AviĂłn in :issue:`32659`.)
1188
1189
1190 pathlib
1191 -------
1192
1193 The new :meth:`Path.is_mount() <pathlib.Path.is_mount>` method is now available
1194 on POSIX systems and can be used to determine whether a path is a mount point.
1195 (Contributed by Cooper Ry Lees in :issue:`30897`.)
1196
1197
1198 pdb
1199 ---
1200
1201 :func:`pdb.set_trace` now takes an optional *header* keyword-only
1202 argument. If given, it is printed to the console just before debugging
1203 begins. (Contributed by Barry Warsaw in :issue:`31389`.)
1204
1205 :mod:`pdb` command line now accepts ``-m module_name`` as an alternative to
1206 script file. (Contributed by Mario Corchero in :issue:`32206`.)
1207
1208
1209 py_compile
1210 ----------
1211
1212 :func:`py_compile.compile` -- and by extension, :mod:`compileall` -- now
1213 respects the :envvar:`SOURCE_DATE_EPOCH` environment variable by
1214 unconditionally creating ``.pyc`` files for hash-based validation.
1215 This allows for guaranteeing
1216 `reproducible builds <https://reproducible-builds.org/>`_ of ``.pyc``
1217 files when they are created eagerly. (Contributed by Bernhard M. Wiedemann
1218 in :issue:`29708`.)
1219
1220
1221 pydoc
1222 -----
1223
1224 The pydoc server can now bind to an arbitrary hostname specified by the
1225 new ``-n`` command-line argument.
1226 (Contributed by Feanil Patel in :issue:`31128`.)
1227
1228
1229 queue
1230 -----
1231
1232 The new :class:`~queue.SimpleQueue` class is an unbounded :abbr:`FIFO` queue.
1233 (Contributed by Antoine Pitrou in :issue:`14976`.)
1234
1235
1236 re
1237 --
1238
1239 The flags :const:`re.ASCII`, :const:`re.LOCALE` and :const:`re.UNICODE`
1240 can be set within the scope of a group.
1241 (Contributed by Serhiy Storchaka in :issue:`31690`.)
1242
1243 :func:`re.split` now supports splitting on a pattern like ``r'\b'``,
1244 ``'^$'`` or ``(?=-)`` that matches an empty string.
1245 (Contributed by Serhiy Storchaka in :issue:`25054`.)
1246
1247 Regular expressions compiled with the :const:`re.LOCALE` flag no longer
1248 depend on the locale at compile time. Locale settings are applied only
1249 when the compiled regular expression is used.
1250 (Contributed by Serhiy Storchaka in :issue:`30215`.)
1251
1252 :exc:`FutureWarning` is now emitted if a regular expression contains
1253 character set constructs that will change semantically in the future,
1254 such as nested sets and set operations.
1255 (Contributed by Serhiy Storchaka in :issue:`30349`.)
1256
1257 Compiled regular expression and match objects can now be copied
1258 using :func:`copy.copy` and :func:`copy.deepcopy`.
1259 (Contributed by Serhiy Storchaka in :issue:`10076`.)
1260
1261
1262 signal
1263 ------
1264
1265 The new *warn_on_full_buffer* argument to the :func:`signal.set_wakeup_fd`
1266 function makes it possible to specify whether Python prints a warning on
1267 stderr when the wakeup buffer overflows.
1268 (Contributed by Nathaniel J. Smith in :issue:`30050`.)
1269
1270
1271 socket
1272 ------
1273
1274 The new :func:`socket.getblocking() <socket.socket.getblocking>` method
1275 returns ``True`` if the socket is in blocking mode and ``False`` otherwise.
1276 (Contributed by Yury Selivanov in :issue:`32373`.)
1277
1278 The new :func:`socket.close` function closes the passed socket file descriptor.
1279 This function should be used instead of :func:`os.close` for better
1280 compatibility across platforms.
1281 (Contributed by Christian Heimes in :issue:`32454`.)
1282
1283 The :mod:`socket` module now exposes the :ref:`socket.TCP_CONGESTION
1284 <socket-unix-constants>` (Linux 2.6.13), :ref:`socket.TCP_USER_TIMEOUT
1285 <socket-unix-constants>` (Linux 2.6.37), and :ref:`socket.TCP_NOTSENT_LOWAT
1286 <socket-unix-constants>` (Linux 3.12) constants.
1287 (Contributed by Omar Sandoval in :issue:`26273` and
1288 Nathaniel J. Smith in :issue:`29728`.)
1289
1290 Support for :const:`socket.AF_VSOCK` sockets has been added to allow
1291 communication between virtual machines and their hosts.
1292 (Contributed by Cathy Avery in :issue:`27584`.)
1293
1294 Sockets now auto-detect family, type and protocol from file descriptor
1295 by default.
1296 (Contributed by Christian Heimes in :issue:`28134`.)
1297
1298
1299 socketserver
1300 ------------
1301
1302 :meth:`socketserver.ThreadingMixIn.server_close
1303 <socketserver.BaseServer.server_close>` now waits until all non-daemon
1304 threads complete. :meth:`socketserver.ForkingMixIn.server_close
1305 <socketserver.BaseServer.server_close>` now waits
1306 until all child processes complete.
1307
1308 Add a new :attr:`socketserver.ForkingMixIn.block_on_close
1309 <socketserver.ThreadingMixIn.block_on_close>` class attribute to
1310 :class:`socketserver.ForkingMixIn` and :class:`socketserver.ThreadingMixIn`
1311 classes. Set the class attribute to ``False`` to get the pre-3.7 behaviour.
1312
1313
1314 sqlite3
1315 -------
1316
1317 :class:`sqlite3.Connection` now exposes the :meth:`~sqlite3.Connection.backup`
1318 method when the underlying SQLite library is at version 3.6.11 or higher.
1319 (Contributed by Lele Gaifax in :issue:`27645`.)
1320
1321 The *database* argument of :func:`sqlite3.connect` now accepts any
1322 :term:`path-like object`, instead of just a string.
1323 (Contributed by Anders Lorentsen in :issue:`31843`.)
1324
1325
1326 ssl
1327 ---
1328
1329 The :mod:`ssl` module now uses OpenSSL's builtin API instead of
1330 :func:`!match_hostname` to check a host name or an IP address. Values
1331 are validated during TLS handshake. Any certificate validation error
1332 including failing the host name check now raises
1333 :exc:`~ssl.SSLCertVerificationError` and aborts the handshake with a proper
1334 TLS Alert message. The new exception contains additional information.
1335 Host name validation can be customized with
1336 :attr:`SSLContext.hostname_checks_common_name <ssl.SSLContext.hostname_checks_common_name>`.
1337 (Contributed by Christian Heimes in :issue:`31399`.)
1338
1339 .. note::
1340 The improved host name check requires a *libssl* implementation compatible
1341 with OpenSSL 1.0.2 or 1.1. Consequently, OpenSSL 0.9.8 and 1.0.1 are no
1342 longer supported (see :ref:`37-platform-support-removals` for more details).
1343 The ssl module is mostly compatible with LibreSSL 2.7.2 and newer.
1344
1345 The ``ssl`` module no longer sends IP addresses in SNI TLS extension.
1346 (Contributed by Christian Heimes in :issue:`32185`.)
1347
1348 :func:`!match_hostname` no longer supports partial wildcards like
1349 ``www*.example.org``.
1350 (Contributed by Mandeep Singh in :issue:`23033` and Christian Heimes in
1351 :issue:`31399`.)
1352
1353 The default cipher suite selection of the ``ssl`` module now uses a blacklist
1354 approach rather than a hard-coded whitelist. Python no longer re-enables
1355 ciphers that have been blocked by OpenSSL security updates. Default cipher
1356 suite selection can be configured at compile time.
1357 (Contributed by Christian Heimes in :issue:`31429`.)
1358
1359 Validation of server certificates containing internationalized domain names
1360 (IDNs) is now supported. As part of this change, the
1361 :attr:`SSLSocket.server_hostname <ssl.SSLSocket.server_hostname>` attribute
1362 now stores the expected hostname in A-label form (``"xn--pythn-mua.org"``),
1363 rather than the U-label form (``"pythön.org"``). (Contributed by
1364 Nathaniel J. Smith and Christian Heimes in :issue:`28414`.)
1365
1366 The ``ssl`` module has preliminary and experimental support for TLS 1.3 and
1367 OpenSSL 1.1.1. At the time of Python 3.7.0 release, OpenSSL 1.1.1 is still
1368 under development and TLS 1.3 hasn't been finalized yet. The TLS 1.3
1369 handshake and protocol behaves slightly differently than TLS 1.2 and earlier,
1370 see :ref:`ssl-tlsv1_3`.
1371 (Contributed by Christian Heimes in :issue:`32947`, :issue:`20995`,
1372 :issue:`29136`, :issue:`30622` and :issue:`33618`)
1373
1374 :class:`~ssl.SSLSocket` and :class:`~ssl.SSLObject` no longer have a public
1375 constructor. Direct instantiation was never a documented and supported
1376 feature. Instances must be created with :class:`~ssl.SSLContext` methods
1377 :meth:`~ssl.SSLContext.wrap_socket` and :meth:`~ssl.SSLContext.wrap_bio`.
1378 (Contributed by Christian Heimes in :issue:`32951`)
1379
1380 OpenSSL 1.1 APIs for setting the minimum and maximum TLS protocol version are
1381 available as :attr:`SSLContext.minimum_version <ssl.SSLContext.minimum_version>`
1382 and :attr:`SSLContext.maximum_version <ssl.SSLContext.maximum_version>`.
1383 Supported protocols are indicated by several new flags, such as
1384 :data:`~ssl.HAS_TLSv1_1`.
1385 (Contributed by Christian Heimes in :issue:`32609`.)
1386
1387 Added :attr:`ssl.SSLContext.post_handshake_auth` to enable and
1388 :meth:`ssl.SSLSocket.verify_client_post_handshake` to initiate TLS 1.3
1389 post-handshake authentication.
1390 (Contributed by Christian Heimes in :gh:`78851`.)
1391
1392 string
1393 ------
1394
1395 :class:`string.Template` now lets you to optionally modify the regular
1396 expression pattern for braced placeholders and non-braced placeholders
1397 separately. (Contributed by Barry Warsaw in :issue:`1198569`.)
1398
1399
1400 subprocess
1401 ----------
1402
1403 The :func:`subprocess.run` function accepts the new *capture_output*
1404 keyword argument. When true, stdout and stderr will be captured.
1405 This is equivalent to passing :const:`subprocess.PIPE` as *stdout* and
1406 *stderr* arguments.
1407 (Contributed by Bo Bayles in :issue:`32102`.)
1408
1409 The ``subprocess.run`` function and the :class:`subprocess.Popen` constructor
1410 now accept the *text* keyword argument as an alias
1411 to *universal_newlines*.
1412 (Contributed by Andrew Clegg in :issue:`31756`.)
1413
1414 On Windows the default for *close_fds* was changed from ``False`` to
1415 ``True`` when redirecting the standard handles. It's now possible to set
1416 *close_fds* to true when redirecting the standard handles. See
1417 :class:`subprocess.Popen`. This means that *close_fds* now defaults to
1418 ``True`` on all supported platforms.
1419 (Contributed by Segev Finer in :issue:`19764`.)
1420
1421 The subprocess module is now more graceful when handling
1422 :exc:`KeyboardInterrupt` during :func:`subprocess.call`,
1423 :func:`subprocess.run`, or in a :class:`~subprocess.Popen`
1424 context manager. It now waits a short amount of time for the child
1425 to exit, before continuing the handling of the ``KeyboardInterrupt``
1426 exception.
1427 (Contributed by Gregory P. Smith in :issue:`25942`.)
1428
1429
1430 sys
1431 ---
1432
1433 The new :func:`sys.breakpointhook` hook function is called by the
1434 built-in :func:`breakpoint`.
1435 (Contributed by Barry Warsaw in :issue:`31353`.)
1436
1437 On Android, the new :func:`sys.getandroidapilevel` returns the build-time
1438 Android API version.
1439 (Contributed by Victor Stinner in :issue:`28740`.)
1440
1441 The new :func:`sys.get_coroutine_origin_tracking_depth` function returns
1442 the current coroutine origin tracking depth, as set by
1443 the new :func:`sys.set_coroutine_origin_tracking_depth`. :mod:`asyncio`
1444 has been converted to use this new API instead of
1445 the deprecated :func:`!sys.set_coroutine_wrapper`.
1446 (Contributed by Nathaniel J. Smith in :issue:`32591`.)
1447
1448
1449 time
1450 ----
1451
1452 :pep:`564` adds six new functions with nanosecond resolution to the
1453 :mod:`time` module:
1454
1455 * :func:`time.clock_gettime_ns`
1456 * :func:`time.clock_settime_ns`
1457 * :func:`time.monotonic_ns`
1458 * :func:`time.perf_counter_ns`
1459 * :func:`time.process_time_ns`
1460 * :func:`time.time_ns`
1461
1462 New clock identifiers have been added:
1463
1464 * :const:`time.CLOCK_BOOTTIME` (Linux): Identical to
1465 :const:`time.CLOCK_MONOTONIC`, except it also includes any time that the
1466 system is suspended.
1467 * :const:`time.CLOCK_PROF` (FreeBSD, NetBSD and OpenBSD): High-resolution
1468 per-process CPU timer.
1469 * :const:`time.CLOCK_UPTIME` (FreeBSD, OpenBSD): Time whose absolute value is
1470 the time the system has been running and not suspended, providing accurate
1471 uptime measurement.
1472
1473 The new :func:`time.thread_time` and :func:`time.thread_time_ns` functions
1474 can be used to get per-thread CPU time measurements.
1475 (Contributed by Antoine Pitrou in :issue:`32025`.)
1476
1477 The new :func:`time.pthread_getcpuclockid` function returns the clock ID
1478 of the thread-specific CPU-time clock.
1479
1480
1481 tkinter
1482 -------
1483
1484 The new :class:`tkinter.ttk.Spinbox` class is now available.
1485 (Contributed by Alan Moore in :issue:`32585`.)
1486
1487
1488 tracemalloc
1489 -----------
1490
1491 :class:`tracemalloc.Traceback` behaves more like regular tracebacks,
1492 sorting the frames from oldest to most recent.
1493 :meth:`Traceback.format() <tracemalloc.Traceback.format>`
1494 now accepts negative *limit*, truncating the result to the
1495 ``abs(limit)`` oldest frames. To get the old behaviour, use
1496 the new *most_recent_first* argument to ``Traceback.format()``.
1497 (Contributed by Jesse Bakker in :issue:`32121`.)
1498
1499
1500 types
1501 -----
1502
1503 The new :class:`~types.WrapperDescriptorType`,
1504 :class:`~types.MethodWrapperType`, :class:`~types.MethodDescriptorType`,
1505 and :class:`~types.ClassMethodDescriptorType` classes are now available.
1506 (Contributed by Manuel Krebber and Guido van Rossum in :issue:`29377`,
1507 and Serhiy Storchaka in :issue:`32265`.)
1508
1509 The new :func:`types.resolve_bases` function resolves MRO entries
1510 dynamically as specified by :pep:`560`.
1511 (Contributed by Ivan Levkivskyi in :issue:`32717`.)
1512
1513
1514 unicodedata
1515 -----------
1516
1517 The internal :mod:`unicodedata` database has been upgraded to use `Unicode 11
1518 <https://www.unicode.org/versions/Unicode11.0.0/>`_. (Contributed by Benjamin
1519 Peterson.)
1520
1521
1522 unittest
1523 --------
1524
1525 The new ``-k`` command-line option allows filtering tests by a name
1526 substring or a Unix shell-like pattern.
1527 For example, ``python -m unittest -k foo`` runs
1528 ``foo_tests.SomeTest.test_something``, ``bar_tests.SomeTest.test_foo``,
1529 but not ``bar_tests.FooTest.test_something``.
1530 (Contributed by Jonas Haag in :issue:`32071`.)
1531
1532
1533 unittest.mock
1534 -------------
1535
1536 The :const:`~unittest.mock.sentinel` attributes now preserve their identity
1537 when they are :mod:`copied <copy>` or :mod:`pickled <pickle>`. (Contributed by
1538 Serhiy Storchaka in :issue:`20804`.)
1539
1540 The new :func:`~unittest.mock.seal` function allows sealing
1541 :class:`~unittest.mock.Mock` instances, which will disallow further creation
1542 of attribute mocks. The seal is applied recursively to all attributes that
1543 are themselves mocks.
1544 (Contributed by Mario Corchero in :issue:`30541`.)
1545
1546
1547 urllib.parse
1548 ------------
1549
1550 :func:`urllib.parse.quote` has been updated from :rfc:`2396` to :rfc:`3986`,
1551 adding ``~`` to the set of characters that are never quoted by default.
1552 (Contributed by Christian Theune and Ratnadeep Debnath in :issue:`16285`.)
1553
1554
1555 uu
1556 --
1557
1558 The :func:`!uu.encode` function now accepts an optional *backtick*
1559 keyword argument. When it's true, zeros are represented by ``'`'``
1560 instead of spaces. (Contributed by Xiang Zhang in :issue:`30103`.)
1561
1562
1563 uuid
1564 ----
1565
1566 The new :attr:`UUID.is_safe <uuid.UUID.is_safe>` attribute relays information
1567 from the platform about whether generated UUIDs are generated with a
1568 multiprocessing-safe method.
1569 (Contributed by Barry Warsaw in :issue:`22807`.)
1570
1571 :func:`uuid.getnode` now prefers universally administered
1572 MAC addresses over locally administered MAC addresses.
1573 This makes a better guarantee for global uniqueness of UUIDs returned
1574 from :func:`uuid.uuid1`. If only locally administered MAC addresses are
1575 available, the first such one found is returned.
1576 (Contributed by Barry Warsaw in :issue:`32107`.)
1577
1578
1579 warnings
1580 --------
1581
1582 The initialization of the default warnings filters has changed as follows:
1583
1584 * warnings enabled via command line options (including those for :option:`-b`
1585 and the new CPython-specific :option:`-X` ``dev`` option) are always passed
1586 to the warnings machinery via the :data:`sys.warnoptions` attribute.
1587
1588 * warnings filters enabled via the command line or the environment now have the
1589 following order of precedence:
1590
1591 * the ``BytesWarning`` filter for :option:`-b` (or ``-bb``)
1592 * any filters specified with the :option:`-W` option
1593 * any filters specified with the :envvar:`PYTHONWARNINGS` environment
1594 variable
1595 * any other CPython specific filters (e.g. the ``default`` filter added
1596 for the new ``-X dev`` mode)
1597 * any implicit filters defined directly by the warnings machinery
1598
1599 * in :ref:`CPython debug builds <debug-build>`, all warnings are now displayed
1600 by default (the implicit filter list is empty)
1601
1602 (Contributed by Nick Coghlan and Victor Stinner in :issue:`20361`,
1603 :issue:`32043`, and :issue:`32230`.)
1604
1605 Deprecation warnings are once again shown by default in single-file scripts and
1606 at the interactive prompt. See :ref:`whatsnew37-pep565` for details.
1607 (Contributed by Nick Coghlan in :issue:`31975`.)
1608
1609
1610 xml
1611 ---
1612
1613 As mitigation against DTD and external entity retrieval, the
1614 :mod:`xml.dom.minidom` and :mod:`xml.sax` modules no longer process
1615 external entities by default.
1616 (Contributed by Christian Heimes in :gh:`61441`.)
1617
1618
1619 xml.etree
1620 ---------
1621
1622 :ref:`ElementPath <elementtree-xpath>` predicates in the :meth:`!find`
1623 methods can now compare text of the current node with ``[. = "text"]``,
1624 not only text in children. Predicates also allow adding spaces for
1625 better readability. (Contributed by Stefan Behnel in :issue:`31648`.)
1626
1627
1628 xmlrpc.server
1629 -------------
1630
1631 :meth:`!SimpleXMLRPCDispatcher.register_function`
1632 can now be used as a decorator. (Contributed by Xiang Zhang in
1633 :issue:`7769`.)
1634
1635
1636 zipapp
1637 ------
1638
1639 Function :func:`~zipapp.create_archive` now accepts an optional *filter*
1640 argument to allow the user to select which files should be included in the
1641 archive. (Contributed by Irmen de Jong in :issue:`31072`.)
1642
1643 Function :func:`~zipapp.create_archive` now accepts an optional *compressed*
1644 argument to generate a compressed archive. A command line option
1645 ``--compress`` has also been added to support compression.
1646 (Contributed by Zhiming Wang in :issue:`31638`.)
1647
1648
1649 zipfile
1650 -------
1651
1652 :class:`~zipfile.ZipFile` now accepts the new *compresslevel* parameter to
1653 control the compression level.
1654 (Contributed by Bo Bayles in :issue:`21417`.)
1655
1656 Subdirectories in archives created by ``ZipFile`` are now stored in
1657 alphabetical order.
1658 (Contributed by Bernhard M. Wiedemann in :issue:`30693`.)
1659
1660
1661 C API Changes
1662 =============
1663
1664 A new API for thread-local storage has been implemented. See
1665 :ref:`whatsnew37-pep539` for an overview and
1666 :ref:`thread-specific-storage-api` for a complete reference.
1667 (Contributed by Masayuki Yamamoto in :issue:`25658`.)
1668
1669 The new :ref:`context variables <whatsnew37-pep567>` functionality
1670 exposes a number of :ref:`new C APIs <contextvarsobjects>`.
1671
1672 The new :c:func:`PyImport_GetModule` function returns the previously
1673 imported module with the given name.
1674 (Contributed by Eric Snow in :issue:`28411`.)
1675
1676 The new :c:macro:`Py_RETURN_RICHCOMPARE` macro eases writing rich
1677 comparison functions.
1678 (Contributed by Petr Victorin in :issue:`23699`.)
1679
1680 The new :c:macro:`Py_UNREACHABLE` macro can be used to mark unreachable
1681 code paths.
1682 (Contributed by Barry Warsaw in :issue:`31338`.)
1683
1684 The :mod:`tracemalloc` now exposes a C API through the new
1685 :c:func:`PyTraceMalloc_Track` and :c:func:`PyTraceMalloc_Untrack`
1686 functions.
1687 (Contributed by Victor Stinner in :issue:`30054`.)
1688
1689 The new :ref:`import__find__load__start <static-markers>` and
1690 :ref:`import__find__load__done <static-markers>` static markers can be used
1691 to trace module imports.
1692 (Contributed by Christian Heimes in :issue:`31574`.)
1693
1694 The fields :c:member:`!name` and :c:member:`!doc` of structures
1695 :c:type:`PyMemberDef`, :c:type:`PyGetSetDef`,
1696 :c:type:`PyStructSequence_Field`, :c:type:`PyStructSequence_Desc`,
1697 and :c:struct:`!wrapperbase` are now of type ``const char *`` rather of
1698 ``char *``. (Contributed by Serhiy Storchaka in :issue:`28761`.)
1699
1700 The result of :c:func:`PyUnicode_AsUTF8AndSize` and :c:func:`PyUnicode_AsUTF8`
1701 is now of type ``const char *`` rather of ``char *``. (Contributed by Serhiy
1702 Storchaka in :issue:`28769`.)
1703
1704 The result of :c:func:`PyMapping_Keys`, :c:func:`PyMapping_Values` and
1705 :c:func:`PyMapping_Items` is now always a list, rather than a list or a
1706 tuple. (Contributed by Oren Milman in :issue:`28280`.)
1707
1708 Added functions :c:func:`PySlice_Unpack` and :c:func:`PySlice_AdjustIndices`.
1709 (Contributed by Serhiy Storchaka in :issue:`27867`.)
1710
1711 :c:func:`PyOS_AfterFork` is deprecated in favour of the new functions
1712 :c:func:`PyOS_BeforeFork`, :c:func:`PyOS_AfterFork_Parent` and
1713 :c:func:`PyOS_AfterFork_Child`. (Contributed by Antoine Pitrou in
1714 :issue:`16500`.)
1715
1716 The ``PyExc_RecursionErrorInst`` singleton that was part of the public API
1717 has been removed as its members being never cleared may cause a segfault
1718 during finalization of the interpreter. Contributed by Xavier de Gaye in
1719 :issue:`22898` and :issue:`30697`.
1720
1721 Added C API support for timezones with timezone constructors
1722 :c:func:`PyTimeZone_FromOffset` and :c:func:`PyTimeZone_FromOffsetAndName`,
1723 and access to the UTC singleton with :c:data:`PyDateTime_TimeZone_UTC`.
1724 Contributed by Paul Ganssle in :issue:`10381`.
1725
1726 The type of results of :c:func:`!PyThread_start_new_thread` and
1727 :c:func:`!PyThread_get_thread_ident`, and the *id* parameter of
1728 :c:func:`PyThreadState_SetAsyncExc` changed from :c:expr:`long` to
1729 :c:expr:`unsigned long`.
1730 (Contributed by Serhiy Storchaka in :issue:`6532`.)
1731
1732 :c:func:`PyUnicode_AsWideCharString` now raises a :exc:`ValueError` if the
1733 second argument is ``NULL`` and the :c:expr:`wchar_t*` string contains null
1734 characters. (Contributed by Serhiy Storchaka in :issue:`30708`.)
1735
1736 Changes to the startup sequence and the management of dynamic memory
1737 allocators mean that the long documented requirement to call
1738 :c:func:`Py_Initialize` before calling most C API functions is now
1739 relied on more heavily, and failing to abide by it may lead to segfaults in
1740 embedding applications. See the :ref:`porting-to-python-37` section in this
1741 document and the :ref:`pre-init-safe` section in the C API documentation
1742 for more details.
1743
1744 The new :c:func:`PyInterpreterState_GetID` returns the unique ID for a
1745 given interpreter.
1746 (Contributed by Eric Snow in :issue:`29102`.)
1747
1748 :c:func:`Py_DecodeLocale`, :c:func:`Py_EncodeLocale` now use the UTF-8
1749 encoding when the :ref:`UTF-8 mode <whatsnew37-pep540>` is enabled.
1750 (Contributed by Victor Stinner in :issue:`29240`.)
1751
1752 :c:func:`PyUnicode_DecodeLocaleAndSize` and :c:func:`PyUnicode_EncodeLocale`
1753 now use the current locale encoding for ``surrogateescape`` error handler.
1754 (Contributed by Victor Stinner in :issue:`29240`.)
1755
1756 The *start* and *end* parameters of :c:func:`PyUnicode_FindChar` are
1757 now adjusted to behave like string slices.
1758 (Contributed by Xiang Zhang in :issue:`28822`.)
1759
1760
1761 Build Changes
1762 =============
1763
1764 Support for building ``--without-threads`` has been removed. The
1765 :mod:`threading` module is now always available.
1766 (Contributed by Antoine Pitrou in :issue:`31370`.).
1767
1768 A full copy of libffi is no longer bundled for use when building the
1769 :mod:`_ctypes <ctypes>` module on non-OSX UNIX platforms. An installed copy
1770 of libffi is now required when building ``_ctypes`` on such platforms.
1771 (Contributed by Zachary Ware in :issue:`27979`.)
1772
1773 The Windows build process no longer depends on Subversion to pull in external
1774 sources, a Python script is used to download zipfiles from GitHub instead.
1775 If Python 3.6 is not found on the system (via ``py -3.6``), NuGet is used to
1776 download a copy of 32-bit Python for this purpose. (Contributed by Zachary
1777 Ware in :issue:`30450`.)
1778
1779 The :mod:`ssl` module requires OpenSSL 1.0.2 or 1.1 compatible libssl.
1780 OpenSSL 1.0.1 has reached end of lifetime on 2016-12-31 and is no longer
1781 supported. LibreSSL is temporarily not supported as well. LibreSSL releases
1782 up to version 2.6.4 are missing required OpenSSL 1.0.2 APIs.
1783
1784
1785 .. _whatsnew37-perf:
1786
1787 Optimizations
1788 =============
1789
1790 The overhead of calling many methods of various standard library classes
1791 implemented in C has been significantly reduced by porting more code
1792 to use the ``METH_FASTCALL`` convention.
1793 (Contributed by Victor Stinner in :issue:`29300`, :issue:`29507`,
1794 :issue:`29452`, and :issue:`29286`.)
1795
1796 Various optimizations have reduced Python startup time by 10% on Linux and
1797 up to 30% on macOS.
1798 (Contributed by Victor Stinner, INADA Naoki in :issue:`29585`, and
1799 Ivan Levkivskyi in :issue:`31333`.)
1800
1801 Method calls are now up to 20% faster due to the bytecode changes which
1802 avoid creating bound method instances.
1803 (Contributed by Yury Selivanov and INADA Naoki in :issue:`26110`.)
1804
1805 .. _whatsnew37-asyncio-perf:
1806
1807 The :mod:`asyncio` module received a number of notable optimizations for
1808 commonly used functions:
1809
1810 * The :func:`asyncio.get_event_loop` function has been reimplemented in C to
1811 make it up to 15 times faster.
1812 (Contributed by Yury Selivanov in :issue:`32296`.)
1813
1814 * :class:`asyncio.Future` callback management has been optimized.
1815 (Contributed by Yury Selivanov in :issue:`32348`.)
1816
1817 * :func:`asyncio.gather` is now up to 15% faster.
1818 (Contributed by Yury Selivanov in :issue:`32355`.)
1819
1820 * :func:`asyncio.sleep` is now up to 2 times faster when the *delay*
1821 argument is zero or negative.
1822 (Contributed by Andrew Svetlov in :issue:`32351`.)
1823
1824 * The performance overhead of asyncio debug mode has been reduced.
1825 (Contributed by Antoine Pitrou in :issue:`31970`.)
1826
1827 As a result of :ref:`PEP 560 work <whatsnew37-pep560>`, the import time
1828 of :mod:`typing` has been reduced by a factor of 7, and many typing operations
1829 are now faster.
1830 (Contributed by Ivan Levkivskyi in :issue:`32226`.)
1831
1832 :func:`sorted` and :meth:`list.sort` have been optimized for common cases
1833 to be up to 40-75% faster.
1834 (Contributed by Elliot Gorokhovsky in :issue:`28685`.)
1835
1836 :meth:`dict.copy` is now up to 5.5 times faster.
1837 (Contributed by Yury Selivanov in :issue:`31179`.)
1838
1839 :func:`hasattr` and :func:`getattr` are now about 4 times faster when
1840 *name* is not found and *obj* does not override :meth:`object.__getattr__`
1841 or :meth:`object.__getattribute__`.
1842 (Contributed by INADA Naoki in :issue:`32544`.)
1843
1844 Searching for certain Unicode characters (like Ukrainian capital "Є")
1845 in a string was up to 25 times slower than searching for other characters.
1846 It is now only 3 times slower in the worst case.
1847 (Contributed by Serhiy Storchaka in :issue:`24821`.)
1848
1849 The :func:`collections.namedtuple` factory has been reimplemented to
1850 make the creation of named tuples 4 to 6 times faster.
1851 (Contributed by Jelle Zijlstra with further improvements by INADA Naoki,
1852 Serhiy Storchaka, and Raymond Hettinger in :issue:`28638`.)
1853
1854 :meth:`datetime.date.fromordinal` and :meth:`datetime.date.fromtimestamp`
1855 are now up to 30% faster in the common case.
1856 (Contributed by Paul Ganssle in :issue:`32403`.)
1857
1858 The :func:`os.fwalk` function is now up to 2 times faster thanks to
1859 the use of :func:`os.scandir`.
1860 (Contributed by Serhiy Storchaka in :issue:`25996`.)
1861
1862 The speed of the :func:`shutil.rmtree` function has been improved by
1863 20--40% thanks to the use of the :func:`os.scandir` function.
1864 (Contributed by Serhiy Storchaka in :issue:`28564`.)
1865
1866 Optimized case-insensitive matching and searching of :mod:`regular
1867 expressions <re>`. Searching some patterns can now be up to 20 times faster.
1868 (Contributed by Serhiy Storchaka in :issue:`30285`.)
1869
1870 :func:`re.compile` now converts ``flags`` parameter to int object if
1871 it is ``RegexFlag``. It is now as fast as Python 3.5, and faster than
1872 Python 3.6 by about 10% depending on the pattern.
1873 (Contributed by INADA Naoki in :issue:`31671`.)
1874
1875 The :meth:`~selectors.BaseSelector.modify` methods of classes
1876 :class:`selectors.EpollSelector`, :class:`selectors.PollSelector`
1877 and :class:`selectors.DevpollSelector` may be around 10% faster under
1878 heavy loads. (Contributed by Giampaolo Rodola' in :issue:`30014`)
1879
1880 Constant folding has been moved from the peephole optimizer to the new AST
1881 optimizer, which is able perform optimizations more consistently.
1882 (Contributed by Eugene Toder and INADA Naoki in :issue:`29469` and
1883 :issue:`11549`.)
1884
1885 Most functions and methods in :mod:`abc` have been rewritten in C.
1886 This makes creation of abstract base classes, and calling :func:`isinstance`
1887 and :func:`issubclass` on them 1.5x faster. This also reduces Python
1888 start-up time by up to 10%. (Contributed by Ivan Levkivskyi and INADA Naoki
1889 in :issue:`31333`)
1890
1891 Significant speed improvements to alternate constructors for
1892 :class:`datetime.date` and :class:`datetime.datetime` by using fast-path
1893 constructors when not constructing subclasses. (Contributed by Paul Ganssle
1894 in :issue:`32403`)
1895
1896 The speed of comparison of :class:`array.array` instances has been
1897 improved considerably in certain cases. It is now from 10x to 70x faster
1898 when comparing arrays holding values of the same integer type.
1899 (Contributed by Adrian Wielgosik in :issue:`24700`.)
1900
1901 The :func:`math.erf` and :func:`math.erfc` functions now use the (faster)
1902 C library implementation on most platforms.
1903 (Contributed by Serhiy Storchaka in :issue:`26121`.)
1904
1905
1906 Other CPython Implementation Changes
1907 ====================================
1908
1909 * Trace hooks may now opt out of receiving the ``line`` and opt into receiving
1910 the ``opcode`` events from the interpreter by setting the corresponding new
1911 :attr:`~frame.f_trace_lines` and :attr:`~frame.f_trace_opcodes` attributes on the
1912 frame being traced. (Contributed by Nick Coghlan in :issue:`31344`.)
1913
1914 * Fixed some consistency problems with namespace package module attributes.
1915 Namespace module objects now have an ``__file__`` that is set to ``None``
1916 (previously unset), and their ``__spec__.origin`` is also set to ``None``
1917 (previously the string ``"namespace"``). See :issue:`32305`. Also, the
1918 namespace module object's ``__spec__.loader`` is set to the same value as
1919 ``__loader__`` (previously, the former was set to ``None``). See
1920 :issue:`32303`.
1921
1922 * The :func:`locals` dictionary now displays in the lexical order that
1923 variables were defined. Previously, the order was undefined.
1924 (Contributed by Raymond Hettinger in :issue:`32690`.)
1925
1926 * The ``distutils`` ``upload`` command no longer tries to change CR
1927 end-of-line characters to CRLF. This fixes a corruption issue with sdists
1928 that ended with a byte equivalent to CR.
1929 (Contributed by Bo Bayles in :issue:`32304`.)
1930
1931
1932 Deprecated Python Behavior
1933 ==========================
1934
1935 Yield expressions (both ``yield`` and ``yield from`` clauses) are now deprecated
1936 in comprehensions and generator expressions (aside from the iterable expression
1937 in the leftmost :keyword:`!for` clause). This ensures that comprehensions
1938 always immediately return a container of the appropriate type (rather than
1939 potentially returning a :term:`generator iterator` object), while generator
1940 expressions won't attempt to interleave their implicit output with the output
1941 from any explicit yield expressions. In Python 3.7, such expressions emit
1942 :exc:`DeprecationWarning` when compiled, in Python 3.8 this will be a
1943 :exc:`SyntaxError`.
1944 (Contributed by Serhiy Storchaka in :issue:`10544`.)
1945
1946 Returning a subclass of :class:`complex` from :meth:`object.__complex__` is
1947 deprecated and will be an error in future Python versions. This makes
1948 ``__complex__()`` consistent with :meth:`object.__int__` and
1949 :meth:`object.__float__`.
1950 (Contributed by Serhiy Storchaka in :issue:`28894`.)
1951
1952
1953
1954 Deprecated Python modules, functions and methods
1955 ================================================
1956
1957 aifc
1958 ----
1959
1960 :func:`!aifc.openfp` has been deprecated and will be removed in Python 3.9.
1961 Use :func:`!aifc.open` instead.
1962 (Contributed by Brian Curtin in :issue:`31985`.)
1963
1964
1965 .. _whatsnew37-asyncio-deprecated:
1966
1967 asyncio
1968 -------
1969
1970 Support for directly ``await``-ing instances of :class:`asyncio.Lock` and
1971 other asyncio synchronization primitives has been deprecated. An
1972 asynchronous context manager must be used in order to acquire and release
1973 the synchronization resource.
1974 (Contributed by Andrew Svetlov in :issue:`32253`.)
1975
1976 The :meth:`!asyncio.Task.current_task` and :meth:`!asyncio.Task.all_tasks`
1977 methods have been deprecated.
1978 (Contributed by Andrew Svetlov in :issue:`32250`.)
1979
1980
1981 collections
1982 -----------
1983
1984 In Python 3.8, the abstract base classes in :mod:`collections.abc` will no
1985 longer be exposed in the regular :mod:`collections` module. This will help
1986 create a clearer distinction between the concrete classes and the abstract
1987 base classes.
1988 (Contributed by Serhiy Storchaka in :issue:`25988`.)
1989
1990
1991 dbm
1992 ---
1993
1994 :mod:`dbm.dumb` now supports reading read-only files and no longer writes the
1995 index file when it is not changed. A deprecation warning is now emitted
1996 if the index file is missing and recreated in the ``'r'`` and ``'w'``
1997 modes (this will be an error in future Python releases).
1998 (Contributed by Serhiy Storchaka in :issue:`28847`.)
1999
2000
2001 enum
2002 ----
2003
2004 In Python 3.8, attempting to check for non-Enum objects in :class:`~enum.Enum`
2005 classes will raise a :exc:`TypeError` (e.g. ``1 in Color``); similarly,
2006 attempting to check for non-Flag objects in a :class:`~enum.Flag` member will
2007 raise :exc:`TypeError` (e.g. ``1 in Perm.RW``); currently, both operations
2008 return :const:`False` instead.
2009 (Contributed by Ethan Furman in :issue:`33217`.)
2010
2011
2012 gettext
2013 -------
2014
2015 Using non-integer value for selecting a plural form in :mod:`gettext` is
2016 now deprecated. It never correctly worked. (Contributed by Serhiy Storchaka
2017 in :issue:`28692`.)
2018
2019
2020 importlib
2021 ---------
2022
2023 Methods
2024 :meth:`!MetaPathFinder.find_module`
2025 (replaced by
2026 :meth:`MetaPathFinder.find_spec() <importlib.abc.MetaPathFinder.find_spec>`)
2027 and
2028 :meth:`!PathEntryFinder.find_loader`
2029 (replaced by
2030 :meth:`PathEntryFinder.find_spec() <importlib.abc.PathEntryFinder.find_spec>`)
2031 both deprecated in Python 3.4 now emit :exc:`DeprecationWarning`.
2032 (Contributed by Matthias Bussonnier in :issue:`29576`.)
2033
2034 The :class:`importlib.abc.ResourceLoader` ABC has been deprecated in
2035 favour of :class:`!importlib.abc.ResourceReader`.
2036
2037
2038 locale
2039 ------
2040
2041 :func:`!locale.format` has been deprecated, use :meth:`locale.format_string`
2042 instead. (Contributed by Garvit in :issue:`10379`.)
2043
2044
2045 macpath
2046 -------
2047
2048 The :mod:`!macpath` is now deprecated and will be removed in Python 3.8.
2049 (Contributed by Chi Hsuan Yen in :issue:`9850`.)
2050
2051
2052 threading
2053 ---------
2054
2055 :mod:`!dummy_threading` and :mod:`!_dummy_thread` have been deprecated. It is
2056 no longer possible to build Python with threading disabled.
2057 Use :mod:`threading` instead.
2058 (Contributed by Antoine Pitrou in :issue:`31370`.)
2059
2060
2061 socket
2062 ------
2063
2064 The silent argument value truncation in :func:`socket.htons` and
2065 :func:`socket.ntohs` has been deprecated. In future versions of Python,
2066 if the passed argument is larger than 16 bits, an exception will be raised.
2067 (Contributed by Oren Milman in :issue:`28332`.)
2068
2069
2070 ssl
2071 ---
2072
2073 :func:`!ssl.wrap_socket` is deprecated. Use
2074 :meth:`ssl.SSLContext.wrap_socket` instead.
2075 (Contributed by Christian Heimes in :issue:`28124`.)
2076
2077
2078 sunau
2079 -----
2080
2081 :func:`!sunau.openfp` has been deprecated and will be removed in Python 3.9.
2082 Use :func:`!sunau.open` instead.
2083 (Contributed by Brian Curtin in :issue:`31985`.)
2084
2085
2086 sys
2087 ---
2088
2089 Deprecated :func:`!sys.set_coroutine_wrapper` and
2090 :func:`!sys.get_coroutine_wrapper`.
2091
2092 The undocumented ``sys.callstats()`` function has been deprecated and
2093 will be removed in a future Python version.
2094 (Contributed by Victor Stinner in :issue:`28799`.)
2095
2096
2097 wave
2098 ----
2099
2100 :func:`!wave.openfp` has been deprecated and will be removed in Python 3.9.
2101 Use :func:`wave.open` instead.
2102 (Contributed by Brian Curtin in :issue:`31985`.)
2103
2104
2105 Deprecated functions and types of the C API
2106 ===========================================
2107
2108 Function :c:func:`PySlice_GetIndicesEx` is deprecated and replaced with
2109 a macro if ``Py_LIMITED_API`` is not set or set to a value in the range
2110 between ``0x03050400`` and ``0x03060000`` (not inclusive), or is ``0x03060100``
2111 or higher. (Contributed by Serhiy Storchaka in :issue:`27867`.)
2112
2113 :c:func:`PyOS_AfterFork` has been deprecated. Use :c:func:`PyOS_BeforeFork`,
2114 :c:func:`PyOS_AfterFork_Parent` or :c:func:`PyOS_AfterFork_Child()` instead.
2115 (Contributed by Antoine Pitrou in :issue:`16500`.)
2116
2117
2118 .. _37-platform-support-removals:
2119
2120 Platform Support Removals
2121 =========================
2122
2123 * FreeBSD 9 and older are no longer officially supported.
2124 * For full Unicode support, including within extension modules, \*nix platforms
2125 are now expected to provide at least one of ``C.UTF-8`` (full locale),
2126 ``C.utf8`` (full locale) or ``UTF-8`` (``LC_CTYPE``-only locale) as an
2127 alternative to the legacy ``ASCII``-based ``C`` locale.
2128 * OpenSSL 0.9.8 and 1.0.1 are no longer supported, which means building CPython
2129 3.7 with SSL/TLS support on older platforms still using these versions
2130 requires custom build options that link to a more recent version of OpenSSL.
2131
2132 Notably, this issue affects the Debian 8 (aka "jessie") and Ubuntu 14.04
2133 (aka "Trusty") LTS Linux distributions, as they still use OpenSSL 1.0.1 by
2134 default.
2135
2136 Debian 9 ("stretch") and Ubuntu 16.04 ("xenial"), as well as recent releases
2137 of other LTS Linux releases (e.g. RHEL/CentOS 7.5, SLES 12-SP3), use OpenSSL
2138 1.0.2 or later, and remain supported in the default build configuration.
2139
2140 CPython's own `CI configuration file
2141 <https://github.com/python/cpython/blob/v3.7.13/.travis.yml>`_ provides an
2142 example of using the SSL
2143 :source:`compatibility testing infrastructure <Tools/ssl/multissltests.py>` in
2144 CPython's test suite to build and link against OpenSSL 1.1.0 rather than an
2145 outdated system provided OpenSSL.
2146
2147
2148 API and Feature Removals
2149 ========================
2150
2151 The following features and APIs have been removed from Python 3.7:
2152
2153 * The ``os.stat_float_times()`` function has been removed. It was introduced in
2154 Python 2.3 for backward compatibility with Python 2.2, and was deprecated
2155 since Python 3.1.
2156
2157 * Unknown escapes consisting of ``'\'`` and an ASCII letter in replacement
2158 templates for :func:`re.sub` were deprecated in Python 3.5, and will now
2159 cause an error.
2160
2161 * Removed support of the *exclude* argument in :meth:`tarfile.TarFile.add`.
2162 It was deprecated in Python 2.7 and 3.2. Use the *filter* argument instead.
2163
2164 * The :func:`!ntpath.splitunc` function was deprecated in
2165 Python 3.1, and has now been removed. Use :func:`~os.path.splitdrive`
2166 instead.
2167
2168 * :func:`collections.namedtuple` no longer supports the *verbose* parameter
2169 or ``_source`` attribute which showed the generated source code for the
2170 named tuple class. This was part of an optimization designed to speed-up
2171 class creation. (Contributed by Jelle Zijlstra with further improvements
2172 by INADA Naoki, Serhiy Storchaka, and Raymond Hettinger in :issue:`28638`.)
2173
2174 * Functions :func:`bool`, :func:`float`, :func:`list` and :func:`tuple` no
2175 longer take keyword arguments. The first argument of :func:`int` can now
2176 be passed only as positional argument.
2177
2178 * Removed previously deprecated in Python 2.4 classes ``Plist``, ``Dict`` and
2179 ``_InternalDict`` in the :mod:`plistlib` module. Dict values in the result
2180 of functions :func:`!readPlist` and
2181 :func:`!readPlistFromBytes` are now normal dicts. You no longer
2182 can use attribute access to access items of these dictionaries.
2183
2184 * The ``asyncio.windows_utils.socketpair()`` function has been
2185 removed. Use the :func:`socket.socketpair` function instead,
2186 it is available on all platforms since Python 3.5.
2187 ``asyncio.windows_utils.socketpair`` was just an alias to
2188 ``socket.socketpair`` on Python 3.5 and newer.
2189
2190 * :mod:`asyncio` no longer exports the :mod:`selectors` and
2191 :mod:`!_overlapped` modules as ``asyncio.selectors`` and
2192 ``asyncio._overlapped``. Replace ``from asyncio import selectors`` with
2193 ``import selectors``.
2194
2195 * Direct instantiation of :class:`ssl.SSLSocket` and :class:`ssl.SSLObject`
2196 objects is now prohibited. The constructors were never documented, tested,
2197 or designed as public constructors. Users were supposed to use
2198 :func:`!ssl.wrap_socket` or :class:`ssl.SSLContext`.
2199 (Contributed by Christian Heimes in :issue:`32951`.)
2200
2201 * The unused ``distutils`` ``install_misc`` command has been removed.
2202 (Contributed by Eric N. Vander Weele in :issue:`29218`.)
2203
2204
2205 Module Removals
2206 ===============
2207
2208 The ``fpectl`` module has been removed. It was never enabled by
2209 default, never worked correctly on x86-64, and it changed the Python
2210 ABI in ways that caused unexpected breakage of C extensions.
2211 (Contributed by Nathaniel J. Smith in :issue:`29137`.)
2212
2213
2214 Windows-only Changes
2215 ====================
2216
2217 The python launcher, (py.exe), can accept 32 & 64 bit specifiers **without**
2218 having to specify a minor version as well. So ``py -3-32`` and ``py -3-64``
2219 become valid as well as ``py -3.7-32``, also the -*m*-64 and -*m.n*-64 forms
2220 are now accepted to force 64 bit python even if 32 bit would have otherwise
2221 been used. If the specified version is not available py.exe will error exit.
2222 (Contributed by Steve Barnes in :issue:`30291`.)
2223
2224 The launcher can be run as ``py -0`` to produce a list of the installed pythons,
2225 *with default marked with an asterisk*. Running ``py -0p`` will include the paths.
2226 If py is run with a version specifier that cannot be matched it will also print
2227 the *short form* list of available specifiers.
2228 (Contributed by Steve Barnes in :issue:`30362`.)
2229
2230
2231 .. _porting-to-python-37:
2232
2233 Porting to Python 3.7
2234 =====================
2235
2236 This section lists previously described changes and other bugfixes
2237 that may require changes to your code.
2238
2239
2240 Changes in Python Behavior
2241 --------------------------
2242
2243 * :keyword:`async` and :keyword:`await` names are now reserved keywords.
2244 Code using these names as identifiers will now raise a :exc:`SyntaxError`.
2245 (Contributed by Jelle Zijlstra in :issue:`30406`.)
2246
2247 * :pep:`479` is enabled for all code in Python 3.7, meaning that
2248 :exc:`StopIteration` exceptions raised directly or indirectly in
2249 coroutines and generators are transformed into :exc:`RuntimeError`
2250 exceptions.
2251 (Contributed by Yury Selivanov in :issue:`32670`.)
2252
2253 * :meth:`object.__aiter__` methods can no longer be declared as
2254 asynchronous. (Contributed by Yury Selivanov in :issue:`31709`.)
2255
2256 * Due to an oversight, earlier Python versions erroneously accepted the
2257 following syntax::
2258
2259 f(1 for x in [1],)
2260
2261 class C(1 for x in [1]):
2262 pass
2263
2264 Python 3.7 now correctly raises a :exc:`SyntaxError`, as a generator
2265 expression always needs to be directly inside a set of parentheses
2266 and cannot have a comma on either side, and the duplication of the
2267 parentheses can be omitted only on calls.
2268 (Contributed by Serhiy Storchaka in :issue:`32012` and :issue:`32023`.)
2269
2270 * When using the :option:`-m` switch, the initial working directory is now added
2271 to :data:`sys.path`, rather than an empty string (which dynamically denoted
2272 the current working directory at the time of each import). Any programs that
2273 are checking for the empty string, or otherwise relying on the previous
2274 behaviour, will need to be updated accordingly (e.g. by also checking for
2275 ``os.getcwd()`` or ``os.path.dirname(__main__.__file__)``, depending on why
2276 the code was checking for the empty string in the first place).
2277
2278
2279 Changes in the Python API
2280 -------------------------
2281
2282 * :meth:`socketserver.ThreadingMixIn.server_close
2283 <socketserver.BaseServer.server_close>` now waits until all
2284 non-daemon threads complete. Set the new
2285 :attr:`socketserver.ThreadingMixIn.block_on_close` class attribute to
2286 ``False`` to get the pre-3.7 behaviour.
2287 (Contributed by Victor Stinner in :issue:`31233` and :issue:`33540`.)
2288
2289 * :meth:`socketserver.ForkingMixIn.server_close
2290 <socketserver.BaseServer.server_close>` now waits until all
2291 child processes complete. Set the new
2292 :attr:`socketserver.ForkingMixIn.block_on_close
2293 <socketserver.ThreadingMixIn.block_on_close>` class attribute to ``False``
2294 to get the pre-3.7 behaviour.
2295 (Contributed by Victor Stinner in :issue:`31151` and :issue:`33540`.)
2296
2297 * The :func:`locale.localeconv` function now temporarily sets the ``LC_CTYPE``
2298 locale to the value of ``LC_NUMERIC`` in some cases.
2299 (Contributed by Victor Stinner in :issue:`31900`.)
2300
2301 * :meth:`pkgutil.walk_packages` now raises a :exc:`ValueError` if *path* is
2302 a string. Previously an empty list was returned.
2303 (Contributed by Sanyam Khurana in :issue:`24744`.)
2304
2305 * A format string argument for :meth:`string.Formatter.format`
2306 is now :ref:`positional-only <positional-only_parameter>`.
2307 Passing it as a keyword argument was deprecated in Python 3.5. (Contributed
2308 by Serhiy Storchaka in :issue:`29193`.)
2309
2310 * Attributes :attr:`~http.cookies.Morsel.key`,
2311 :attr:`~http.cookies.Morsel.value` and
2312 :attr:`~http.cookies.Morsel.coded_value` of class
2313 :class:`http.cookies.Morsel` are now read-only.
2314 Assigning to them was deprecated in Python 3.5.
2315 Use the :meth:`~http.cookies.Morsel.set` method for setting them.
2316 (Contributed by Serhiy Storchaka in :issue:`29192`.)
2317
2318 * The *mode* argument of :func:`os.makedirs` no longer affects the file
2319 permission bits of newly created intermediate-level directories.
2320 To set their file permission bits you can set the umask before invoking
2321 ``makedirs()``.
2322 (Contributed by Serhiy Storchaka in :issue:`19930`.)
2323
2324 * The :attr:`struct.Struct.format` type is now :class:`str` instead of
2325 :class:`bytes`. (Contributed by Victor Stinner in :issue:`21071`.)
2326
2327 * :func:`!cgi.parse_multipart` now accepts the *encoding* and *errors*
2328 arguments and returns the same results as
2329 :class:`!FieldStorage`: for non-file fields, the value associated to a key
2330 is a list of strings, not bytes.
2331 (Contributed by Pierre Quentel in :issue:`29979`.)
2332
2333 * Due to internal changes in :mod:`socket`, calling :func:`socket.fromshare`
2334 on a socket created by :func:`socket.share <socket.socket.share>` in older
2335 Python versions is not supported.
2336
2337 * ``repr`` for :exc:`BaseException` has changed to not include the trailing
2338 comma. Most exceptions are affected by this change.
2339 (Contributed by Serhiy Storchaka in :issue:`30399`.)
2340
2341 * ``repr`` for :class:`datetime.timedelta` has changed to include the keyword
2342 arguments in the output. (Contributed by Utkarsh Upadhyay in :issue:`30302`.)
2343
2344 * Because :func:`shutil.rmtree` is now implemented using the :func:`os.scandir`
2345 function, the user specified handler *onerror* is now called with the first
2346 argument ``os.scandir`` instead of ``os.listdir`` when listing the directory
2347 is failed.
2348
2349 * Support for nested sets and set operations in regular expressions as in
2350 `Unicode Technical Standard #18`_ might be added in the future. This would
2351 change the syntax. To facilitate this future change a :exc:`FutureWarning`
2352 will be raised in ambiguous cases for the time being.
2353 That include sets starting with a literal ``'['`` or containing literal
2354 character sequences ``'--'``, ``'&&'``, ``'~~'``, and ``'||'``. To
2355 avoid a warning, escape them with a backslash.
2356 (Contributed by Serhiy Storchaka in :issue:`30349`.)
2357
2358 .. _Unicode Technical Standard #18: https://unicode.org/reports/tr18/
2359
2360 * The result of splitting a string on a :mod:`regular expression <re>`
2361 that could match an empty string has been changed. For example
2362 splitting on ``r'\s*'`` will now split not only on whitespaces as it
2363 did previously, but also on empty strings before all non-whitespace
2364 characters and just before the end of the string.
2365 The previous behavior can be restored by changing the pattern
2366 to ``r'\s+'``. A :exc:`FutureWarning` was emitted for such patterns since
2367 Python 3.5.
2368
2369 For patterns that match both empty and non-empty strings, the result of
2370 searching for all matches may also be changed in other cases. For example
2371 in the string ``'a\n\n'``, the pattern ``r'(?m)^\s*?$'`` will not only
2372 match empty strings at positions 2 and 3, but also the string ``'\n'`` at
2373 positions 2--3. To match only blank lines, the pattern should be rewritten
2374 as ``r'(?m)^[^\S\n]*$'``.
2375
2376 :func:`re.sub` now replaces empty matches adjacent to a previous
2377 non-empty match. For example ``re.sub('x*', '-', 'abxd')`` returns now
2378 ``'-a-b--d-'`` instead of ``'-a-b-d-'`` (the first minus between 'b' and
2379 'd' replaces 'x', and the second minus replaces an empty string between
2380 'x' and 'd').
2381
2382 (Contributed by Serhiy Storchaka in :issue:`25054` and :issue:`32308`.)
2383
2384 * Change :func:`re.escape` to only escape regex special characters instead
2385 of escaping all characters other than ASCII letters, numbers, and ``'_'``.
2386 (Contributed by Serhiy Storchaka in :issue:`29995`.)
2387
2388 * :class:`tracemalloc.Traceback` frames are now sorted from oldest to most
2389 recent to be more consistent with :mod:`traceback`.
2390 (Contributed by Jesse Bakker in :issue:`32121`.)
2391
2392 * On OSes that support :const:`socket.SOCK_NONBLOCK` or
2393 :const:`socket.SOCK_CLOEXEC` bit flags, the
2394 :attr:`socket.type <socket.socket.type>` no longer has them applied.
2395 Therefore, checks like ``if sock.type == socket.SOCK_STREAM``
2396 work as expected on all platforms.
2397 (Contributed by Yury Selivanov in :issue:`32331`.)
2398
2399 * On Windows the default for the *close_fds* argument of
2400 :class:`subprocess.Popen` was changed from :const:`False` to :const:`True`
2401 when redirecting the standard handles. If you previously depended on handles
2402 being inherited when using :class:`subprocess.Popen` with standard io
2403 redirection, you will have to pass ``close_fds=False`` to preserve the
2404 previous behaviour, or use
2405 :attr:`STARTUPINFO.lpAttributeList <subprocess.STARTUPINFO.lpAttributeList>`.
2406
2407 * :meth:`importlib.machinery.PathFinder.invalidate_caches` -- which implicitly
2408 affects :func:`importlib.invalidate_caches` -- now deletes entries
2409 in :data:`sys.path_importer_cache` which are set to ``None``.
2410 (Contributed by Brett Cannon in :issue:`33169`.)
2411
2412 * In :mod:`asyncio`,
2413 :meth:`loop.sock_recv() <asyncio.loop.sock_recv>`,
2414 :meth:`loop.sock_sendall() <asyncio.loop.sock_sendall>`,
2415 :meth:`loop.sock_accept() <asyncio.loop.sock_accept>`,
2416 :meth:`loop.getaddrinfo() <asyncio.loop.getaddrinfo>`,
2417 :meth:`loop.getnameinfo() <asyncio.loop.getnameinfo>`
2418 have been changed to be proper coroutine methods to match their
2419 documentation. Previously, these methods returned :class:`asyncio.Future`
2420 instances.
2421 (Contributed by Yury Selivanov in :issue:`32327`.)
2422
2423 * :attr:`asyncio.Server.sockets` now returns a copy of the internal list
2424 of server sockets, instead of returning it directly.
2425 (Contributed by Yury Selivanov in :issue:`32662`.)
2426
2427 * :attr:`Struct.format <struct.Struct.format>` is now a :class:`str` instance
2428 instead of a :class:`bytes` instance.
2429 (Contributed by Victor Stinner in :issue:`21071`.)
2430
2431 * :mod:`argparse` subparsers can now be made mandatory by passing ``required=True``
2432 to :meth:`ArgumentParser.add_subparsers() <argparse.ArgumentParser.add_subparsers>`.
2433 (Contributed by Anthony Sottile in :issue:`26510`.)
2434
2435 * :meth:`ast.literal_eval` is now stricter. Addition and subtraction of
2436 arbitrary numbers are no longer allowed.
2437 (Contributed by Serhiy Storchaka in :issue:`31778`.)
2438
2439 * :meth:`Calendar.itermonthdates <calendar.Calendar.itermonthdates>`
2440 will now consistently raise an exception when a date falls outside of the
2441 ``0001-01-01`` through ``9999-12-31`` range. To support applications that
2442 cannot tolerate such exceptions, the new
2443 :meth:`Calendar.itermonthdays3 <calendar.Calendar.itermonthdays3>` and
2444 :meth:`Calendar.itermonthdays4 <calendar.Calendar.itermonthdays4>` can be used.
2445 The new methods return tuples and are not restricted by the range supported by
2446 :class:`datetime.date`.
2447 (Contributed by Alexander Belopolsky in :issue:`28292`.)
2448
2449 * :class:`collections.ChainMap` now preserves the order of the underlying
2450 mappings. (Contributed by Raymond Hettinger in :issue:`32792`.)
2451
2452 * The ``submit()`` method of :class:`concurrent.futures.ThreadPoolExecutor`
2453 and :class:`concurrent.futures.ProcessPoolExecutor` now raises
2454 a :exc:`RuntimeError` if called during interpreter shutdown.
2455 (Contributed by Mark Nemec in :issue:`33097`.)
2456
2457 * The :class:`configparser.ConfigParser` constructor now uses ``read_dict()``
2458 to process the default values, making its behavior consistent with the
2459 rest of the parser. Non-string keys and values in the defaults
2460 dictionary are now being implicitly converted to strings.
2461 (Contributed by James Tocknell in :issue:`23835`.)
2462
2463 * Several undocumented internal imports were removed.
2464 One example is that ``os.errno`` is no longer available; use ``import errno``
2465 directly instead.
2466 Note that such undocumented internal imports may be removed any time without
2467 notice, even in micro version releases.
2468
2469
2470 Changes in the C API
2471 --------------------
2472
2473 The function :c:func:`PySlice_GetIndicesEx` is considered unsafe for
2474 resizable sequences. If the slice indices are not instances of :class:`int`,
2475 but objects that implement the :meth:`!__index__` method, the sequence can be
2476 resized after passing its length to :c:func:`!PySlice_GetIndicesEx`. This
2477 can lead to returning indices out of the length of the sequence. For
2478 avoiding possible problems use new functions :c:func:`PySlice_Unpack` and
2479 :c:func:`PySlice_AdjustIndices`.
2480 (Contributed by Serhiy Storchaka in :issue:`27867`.)
2481
2482
2483 CPython bytecode changes
2484 ------------------------
2485
2486 There are two new opcodes: :opcode:`!LOAD_METHOD` and :opcode:`!CALL_METHOD`.
2487 (Contributed by Yury Selivanov and INADA Naoki in :issue:`26110`.)
2488
2489 The :opcode:`!STORE_ANNOTATION` opcode has been removed.
2490 (Contributed by Mark Shannon in :issue:`32550`.)
2491
2492
2493 Windows-only Changes
2494 --------------------
2495
2496 The file used to override :data:`sys.path` is now called
2497 ``<python-executable>._pth`` instead of ``'sys.path'``.
2498 See :ref:`windows_finding_modules` for more information.
2499 (Contributed by Steve Dower in :issue:`28137`.)
2500
2501
2502 Other CPython implementation changes
2503 ------------------------------------
2504
2505 In preparation for potential future changes to the public CPython runtime
2506 initialization API (see :pep:`432` for an initial, but somewhat outdated,
2507 draft), CPython's internal startup
2508 and configuration management logic has been significantly refactored. While
2509 these updates are intended to be entirely transparent to both embedding
2510 applications and users of the regular CPython CLI, they're being mentioned
2511 here as the refactoring changes the internal order of various operations
2512 during interpreter startup, and hence may uncover previously latent defects,
2513 either in embedding applications, or in CPython itself.
2514 (Initially contributed by Nick Coghlan and Eric Snow as part of
2515 :issue:`22257`, and further updated by Nick, Eric, and Victor Stinner in a
2516 number of other issues). Some known details affected:
2517
2518 * :c:func:`!PySys_AddWarnOptionUnicode` is not currently usable by embedding
2519 applications due to the requirement to create a Unicode object prior to
2520 calling ``Py_Initialize``. Use :c:func:`!PySys_AddWarnOption` instead.
2521
2522 * warnings filters added by an embedding application with
2523 :c:func:`!PySys_AddWarnOption` should now more consistently take precedence
2524 over the default filters set by the interpreter
2525
2526 Due to changes in the way the default warnings filters are configured,
2527 setting :c:data:`Py_BytesWarningFlag` to a value greater than one is no longer
2528 sufficient to both emit :exc:`BytesWarning` messages and have them converted
2529 to exceptions. Instead, the flag must be set (to cause the warnings to be
2530 emitted in the first place), and an explicit ``error::BytesWarning``
2531 warnings filter added to convert them to exceptions.
2532
2533 Due to a change in the way docstrings are handled by the compiler, the
2534 implicit ``return None`` in a function body consisting solely of a docstring
2535 is now marked as occurring on the same line as the docstring, not on the
2536 function's header line.
2537
2538 The current exception state has been moved from the frame object to the co-routine.
2539 This simplified the interpreter and fixed a couple of obscure bugs caused by
2540 having swap exception state when entering or exiting a generator.
2541 (Contributed by Mark Shannon in :issue:`25612`.)
2542
2543 Notable changes in Python 3.7.1
2544 ===============================
2545
2546 Starting in 3.7.1, :c:func:`Py_Initialize` now consistently reads and respects
2547 all of the same environment settings as :c:func:`Py_Main` (in earlier Python
2548 versions, it respected an ill-defined subset of those environment variables,
2549 while in Python 3.7.0 it didn't read any of them due to :issue:`34247`). If
2550 this behavior is unwanted, set :c:data:`Py_IgnoreEnvironmentFlag` to 1 before
2551 calling :c:func:`Py_Initialize`.
2552
2553 In 3.7.1 the C API for Context Variables
2554 :ref:`was updated <contextvarsobjects_pointertype_change>` to use
2555 :c:type:`PyObject` pointers. See also :issue:`34762`.
2556
2557 In 3.7.1 the :mod:`tokenize` module now implicitly emits a ``NEWLINE`` token
2558 when provided with input that does not have a trailing new line. This behavior
2559 now matches what the C tokenizer does internally.
2560 (Contributed by Ammar Askar in :issue:`33899`.)
2561
2562 Notable changes in Python 3.7.2
2563 ===============================
2564
2565 In 3.7.2, :mod:`venv` on Windows no longer copies the original binaries, but
2566 creates redirector scripts named ``python.exe`` and ``pythonw.exe`` instead.
2567 This resolves a long standing issue where all virtual environments would have
2568 to be upgraded or recreated with each Python update. However, note that this
2569 release will still require recreation of virtual environments in order to get
2570 the new scripts.
2571
2572 Notable changes in Python 3.7.6
2573 ===============================
2574
2575 Due to significant security concerns, the *reuse_address* parameter of
2576 :meth:`asyncio.loop.create_datagram_endpoint` is no longer supported. This is
2577 because of the behavior of the socket option ``SO_REUSEADDR`` in UDP. For more
2578 details, see the documentation for ``loop.create_datagram_endpoint()``.
2579 (Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in
2580 :issue:`37228`.)
2581
2582 Notable changes in Python 3.7.10
2583 ================================
2584
2585 Earlier Python versions allowed using both ``;`` and ``&`` as
2586 query parameter separators in :func:`urllib.parse.parse_qs` and
2587 :func:`urllib.parse.parse_qsl`. Due to security concerns, and to conform with
2588 newer W3C recommendations, this has been changed to allow only a single
2589 separator key, with ``&`` as the default. This change also affects
2590 :func:`!cgi.parse` and :func:`!cgi.parse_multipart` as they use the affected
2591 functions internally. For more details, please see their respective
2592 documentation.
2593 (Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in :issue:`42967`.)
2594
2595 Notable changes in Python 3.7.11
2596 ================================
2597
2598 A security fix alters the :class:`ftplib.FTP` behavior to not trust the
2599 IPv4 address sent from the remote server when setting up a passive data
2600 channel. We reuse the ftp server IP address instead. For unusual code
2601 requiring the old behavior, set a ``trust_server_pasv_ipv4_address``
2602 attribute on your FTP instance to ``True``. (See :gh:`87451`)
2603
2604
2605 The presence of newline or tab characters in parts of a URL allows for some
2606 forms of attacks. Following the WHATWG specification that updates RFC 3986,
2607 ASCII newline ``\n``, ``\r`` and tab ``\t`` characters are stripped from the
2608 URL by the parser :func:`urllib.parse` preventing such attacks. The removal
2609 characters are controlled by a new module level variable
2610 ``urllib.parse._UNSAFE_URL_BYTES_TO_REMOVE``. (See :gh:`88048`)
2611
2612 Notable security feature in 3.7.14
2613 ==================================
2614
2615 Converting between :class:`int` and :class:`str` in bases other than 2
2616 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal)
2617 now raises a :exc:`ValueError` if the number of digits in string form is
2618 above a limit to avoid potential denial of service attacks due to the
2619 algorithmic complexity. This is a mitigation for :cve:`2020-10735`.
2620 This limit can be configured or disabled by environment variable, command
2621 line flag, or :mod:`sys` APIs. See the :ref:`integer string conversion
2622 length limitation <int_max_str_digits>` documentation. The default limit
2623 is 4300 digits in string form.