Debian Codegraph @

Doc/whatsnew/3.5.rst

raw
1 ****************************
2 What's New In Python 3.5
3 ****************************
4
5 :Editors: Elvis Pranskevichus <[email protected]>, Yury Selivanov <[email protected]>
6
7 .. Rules for maintenance:
8
9 * Anyone can add text to this document. Do not spend very much time
10 on the wording of your changes, because your text will probably
11 get rewritten to some degree.
12
13 * The maintainer will go through Misc/NEWS periodically and add
14 changes; it's therefore more important to add your changes to
15 Misc/NEWS than to this file.
16
17 * This is not a complete list of every single change; completeness
18 is the purpose of Misc/NEWS. Some changes I consider too small
19 or esoteric to include. If such a change is added to the text,
20 I'll just remove it. (This is another reason you shouldn't spend
21 too much time on writing your addition.)
22
23 * If you want to draw your new text to the attention of the
24 maintainer, add 'XXX' to the beginning of the paragraph or
25 section.
26
27 * It's OK to just add a fragmentary note about a change. For
28 example: "XXX Describe the transmogrify() function added to the
29 socket module." The maintainer will research the change and
30 write the necessary text.
31
32 * You can comment out your additions if you like, but it's not
33 necessary (especially when a final release is some months away).
34
35 * Credit the author of a patch or bugfix. Just the name is
36 sufficient; the e-mail address isn't necessary.
37
38 * It's helpful to add the bug/patch number as a comment:
39
40 XXX Describe the transmogrify() function added to the socket
41 module.
42 (Contributed by P.Y. Developer in :issue:`12345`.)
43
44 This saves the maintainer the effort of going through the Mercurial log
45 when researching a change.
46
47 This article explains the new features in Python 3.5, compared to 3.4.
48 Python 3.5 was released on September 13, 2015. See the
49 `changelog <https://docs.python.org/3.5/whatsnew/changelog.html>`_ for a full
50 list of changes.
51
52 .. seealso::
53
54 :pep:`478` - Python 3.5 Release Schedule
55
56
57 Summary -- Release highlights
58 =============================
59
60 New syntax features:
61
62 * :ref:`PEP 492 <whatsnew-pep-492>`, coroutines with async and await syntax.
63 * :ref:`PEP 465 <whatsnew-pep-465>`, a new matrix multiplication operator: ``a @ b``.
64 * :ref:`PEP 448 <whatsnew-pep-448>`, additional unpacking generalizations.
65
66
67 New library modules:
68
69 * :mod:`typing`: :ref:`PEP 484 -- Type Hints <whatsnew-pep-484>`.
70 * :mod:`zipapp`: :ref:`PEP 441 Improving Python ZIP Application Support
71 <whatsnew-zipapp>`.
72
73
74 New built-in features:
75
76 * ``bytes % args``, ``bytearray % args``: :ref:`PEP 461 <whatsnew-pep-461>` --
77 Adding ``%`` formatting to bytes and bytearray.
78
79 * New :meth:`bytes.hex`, :meth:`bytearray.hex` and :meth:`memoryview.hex`
80 methods. (Contributed by Arnon Yaari in :issue:`9951`.)
81
82 * :class:`memoryview` now supports tuple indexing (including multi-dimensional).
83 (Contributed by Antoine Pitrou in :issue:`23632`.)
84
85 * Generators have a new ``gi_yieldfrom`` attribute, which returns the
86 object being iterated by ``yield from`` expressions. (Contributed
87 by Benno Leslie and Yury Selivanov in :issue:`24450`.)
88
89 * A new :exc:`RecursionError` exception is now raised when maximum
90 recursion depth is reached. (Contributed by Georg Brandl
91 in :issue:`19235`.)
92
93
94 CPython implementation improvements:
95
96 * When the ``LC_TYPE`` locale is the POSIX locale (``C`` locale),
97 :py:data:`sys.stdin` and :py:data:`sys.stdout` now use the
98 ``surrogateescape`` error handler, instead of the ``strict`` error handler.
99 (Contributed by Victor Stinner in :issue:`19977`.)
100
101 * ``.pyo`` files are no longer used and have been replaced by a more flexible
102 scheme that includes the optimization level explicitly in ``.pyc`` name.
103 (See :ref:`PEP 488 overview <whatsnew-pep-488>`.)
104
105 * Builtin and extension modules are now initialized in a multi-phase process,
106 which is similar to how Python modules are loaded.
107 (See :ref:`PEP 489 overview <whatsnew-pep-489>`.)
108
109
110 Significant improvements in the standard library:
111
112 * :class:`collections.OrderedDict` is now
113 :ref:`implemented in C <whatsnew-ordereddict>`, which makes it
114 4 to 100 times faster.
115
116 * The :mod:`ssl` module gained
117 :ref:`support for Memory BIO <whatsnew-sslmemorybio>`, which decouples SSL
118 protocol handling from network IO.
119
120 * The new :func:`os.scandir` function provides a
121 :ref:`better and significantly faster way <whatsnew-pep-471>`
122 of directory traversal.
123
124 * :func:`functools.lru_cache` has been mostly
125 :ref:`reimplemented in C <whatsnew-lrucache>`, yielding much better
126 performance.
127
128 * The new :func:`subprocess.run` function provides a
129 :ref:`streamlined way to run subprocesses <whatsnew-subprocess>`.
130
131 * The :mod:`traceback` module has been significantly
132 :ref:`enhanced <whatsnew-traceback>` for improved
133 performance and developer convenience.
134
135
136 Security improvements:
137
138 * SSLv3 is now disabled throughout the standard library.
139 It can still be enabled by instantiating a :class:`ssl.SSLContext`
140 manually. (See :issue:`22638` for more details; this change was
141 backported to CPython 3.4 and 2.7.)
142
143 * HTTP cookie parsing is now stricter, in order to protect
144 against potential injection attacks. (Contributed by Antoine Pitrou
145 in :issue:`22796`.)
146
147
148 Windows improvements:
149
150 * A new installer for Windows has replaced the old MSI.
151 See :ref:`using-on-windows` for more information.
152
153 * Windows builds now use Microsoft Visual C++ 14.0, and extension modules
154 should use the same.
155
156
157 Please read on for a comprehensive list of user-facing changes, including many
158 other smaller improvements, CPython optimizations, deprecations, and potential
159 porting issues.
160
161
162 New Features
163 ============
164
165 .. _whatsnew-pep-492:
166
167 PEP 492 - Coroutines with async and await syntax
168 ------------------------------------------------
169
170 :pep:`492` greatly improves support for asynchronous programming in Python
171 by adding :term:`awaitable objects <awaitable>`,
172 :term:`coroutine functions <coroutine function>`,
173 :term:`asynchronous iteration <asynchronous iterable>`,
174 and :term:`asynchronous context managers <asynchronous context manager>`.
175
176 Coroutine functions are declared using the new :keyword:`async def` syntax::
177
178 >>> async def coro():
179 ... return 'spam'
180
181 Inside a coroutine function, the new :keyword:`await` expression can be used
182 to suspend coroutine execution until the result is available. Any object
183 can be *awaited*, as long as it implements the :term:`awaitable` protocol by
184 defining the :meth:`~object.__await__` method.
185
186 PEP 492 also adds :keyword:`async for` statement for convenient iteration
187 over asynchronous iterables.
188
189 An example of a rudimentary HTTP client written using the new syntax::
190
191 import asyncio
192
193 async def http_get(domain):
194 reader, writer = await asyncio.open_connection(domain, 80)
195
196 writer.write(b'\r\n'.join([
197 b'GET / HTTP/1.1',
198 b'Host: %b' % domain.encode('latin-1'),
199 b'Connection: close',
200 b'', b''
201 ]))
202
203 async for line in reader:
204 print('>>>', line)
205
206 writer.close()
207
208 loop = asyncio.get_event_loop()
209 try:
210 loop.run_until_complete(http_get('example.com'))
211 finally:
212 loop.close()
213
214
215 Similarly to asynchronous iteration, there is a new syntax for asynchronous
216 context managers. The following script::
217
218 import asyncio
219
220 async def coro(name, lock):
221 print('coro {}: waiting for lock'.format(name))
222 async with lock:
223 print('coro {}: holding the lock'.format(name))
224 await asyncio.sleep(1)
225 print('coro {}: releasing the lock'.format(name))
226
227 loop = asyncio.get_event_loop()
228 lock = asyncio.Lock()
229 coros = asyncio.gather(coro(1, lock), coro(2, lock))
230 try:
231 loop.run_until_complete(coros)
232 finally:
233 loop.close()
234
235 will output::
236
237 coro 2: waiting for lock
238 coro 2: holding the lock
239 coro 1: waiting for lock
240 coro 2: releasing the lock
241 coro 1: holding the lock
242 coro 1: releasing the lock
243
244 Note that both :keyword:`async for` and :keyword:`async with` can only
245 be used inside a coroutine function declared with :keyword:`async def`.
246
247 Coroutine functions are intended to be run inside a compatible event loop,
248 such as the :ref:`asyncio loop <asyncio-event-loop>`.
249
250
251 .. note::
252
253 .. versionchanged:: 3.5.2
254 Starting with CPython 3.5.2, ``__aiter__`` can directly return
255 :term:`asynchronous iterators <asynchronous iterator>`. Returning
256 an :term:`awaitable` object will result in a
257 :exc:`PendingDeprecationWarning`.
258
259 See more details in the :ref:`async-iterators` documentation
260 section.
261
262
263 .. seealso::
264
265 :pep:`492` -- Coroutines with async and await syntax
266 PEP written and implemented by Yury Selivanov.
267
268
269 .. _whatsnew-pep-465:
270
271 PEP 465 - A dedicated infix operator for matrix multiplication
272 --------------------------------------------------------------
273
274 :pep:`465` adds the ``@`` infix operator for matrix multiplication.
275 Currently, no builtin Python types implement the new operator, however, it
276 can be implemented by defining :meth:`~object.__matmul__`,
277 :meth:`~object.__rmatmul__`, and :meth:`~object.__imatmul__` for regular,
278 reflected, and in-place matrix multiplication.
279 The semantics of these methods is similar to that of
280 methods defining other infix arithmetic operators.
281
282 Matrix multiplication is a notably common operation in many fields of
283 mathematics, science, engineering, and the addition of ``@`` allows writing
284 cleaner code::
285
286 S = (H @ beta - r).T @ inv(H @ V @ H.T) @ (H @ beta - r)
287
288 instead of::
289
290 S = dot((dot(H, beta) - r).T,
291 dot(inv(dot(dot(H, V), H.T)), dot(H, beta) - r))
292
293 NumPy 1.10 has support for the new operator::
294
295 >>> import numpy
296
297 >>> x = numpy.ones(3)
298 >>> x
299 array([ 1., 1., 1.])
300
301 >>> m = numpy.eye(3)
302 >>> m
303 array([[ 1., 0., 0.],
304 [ 0., 1., 0.],
305 [ 0., 0., 1.]])
306
307 >>> x @ m
308 array([ 1., 1., 1.])
309
310
311 .. seealso::
312
313 :pep:`465` -- A dedicated infix operator for matrix multiplication
314 PEP written by Nathaniel J. Smith; implemented by Benjamin Peterson.
315
316
317 .. _whatsnew-pep-448:
318
319 PEP 448 - Additional Unpacking Generalizations
320 ----------------------------------------------
321
322 :pep:`448` extends the allowed uses of the ``*`` iterable unpacking
323 operator and ``**`` dictionary unpacking operator. It is now possible
324 to use an arbitrary number of unpackings in :ref:`function calls <calls>`::
325
326 >>> print(*[1], *[2], 3, *[4, 5])
327 1 2 3 4 5
328
329 >>> def fn(a, b, c, d):
330 ... print(a, b, c, d)
331 ...
332
333 >>> fn(**{'a': 1, 'c': 3}, **{'b': 2, 'd': 4})
334 1 2 3 4
335
336 Similarly, tuple, list, set, and dictionary displays allow multiple
337 unpackings (see :ref:`exprlists` and :ref:`dict`)::
338
339 >>> *range(4), 4
340 (0, 1, 2, 3, 4)
341
342 >>> [*range(4), 4]
343 [0, 1, 2, 3, 4]
344
345 >>> {*range(4), 4, *(5, 6, 7)}
346 {0, 1, 2, 3, 4, 5, 6, 7}
347
348 >>> {'x': 1, **{'y': 2}}
349 {'x': 1, 'y': 2}
350
351 .. seealso::
352
353 :pep:`448` -- Additional Unpacking Generalizations
354 PEP written by Joshua Landau; implemented by Neil Girdhar,
355 Thomas Wouters, and Joshua Landau.
356
357
358 .. _whatsnew-pep-461:
359
360 PEP 461 - percent formatting support for bytes and bytearray
361 ------------------------------------------------------------
362
363 :pep:`461` adds support for the ``%``
364 :ref:`interpolation operator <bytes-formatting>` to :class:`bytes`
365 and :class:`bytearray`.
366
367 While interpolation is usually thought of as a string operation, there are
368 cases where interpolation on ``bytes`` or ``bytearrays`` makes sense, and the
369 work needed to make up for this missing functionality detracts from the
370 overall readability of the code. This issue is particularly important when
371 dealing with wire format protocols, which are often a mixture of binary
372 and ASCII compatible text.
373
374 Examples::
375
376 >>> b'Hello %b!' % b'World'
377 b'Hello World!'
378
379 >>> b'x=%i y=%f' % (1, 2.5)
380 b'x=1 y=2.500000'
381
382 Unicode is not allowed for ``%b``, but it is accepted by ``%a`` (equivalent of
383 ``repr(obj).encode('ascii', 'backslashreplace')``)::
384
385 >>> b'Hello %b!' % 'World'
386 Traceback (most recent call last):
387 File "<stdin>", line 1, in <module>
388 TypeError: %b requires bytes, or an object that implements __bytes__, not 'str'
389
390 >>> b'price: %a' % '10€'
391 b"price: '10\\u20ac'"
392
393 Note that ``%s`` and ``%r`` conversion types, although supported, should
394 only be used in codebases that need compatibility with Python 2.
395
396 .. seealso::
397
398 :pep:`461` -- Adding % formatting to bytes and bytearray
399 PEP written by Ethan Furman; implemented by Neil Schemenauer and
400 Ethan Furman.
401
402
403 .. _whatsnew-pep-484:
404
405 PEP 484 - Type Hints
406 --------------------
407
408 Function annotation syntax has been a Python feature since version 3.0
409 (:pep:`3107`), however the semantics of annotations has been left undefined.
410
411 Experience has shown that the majority of function annotation
412 uses were to provide type hints to function parameters and return values. It
413 became evident that it would be beneficial for Python users, if the
414 standard library included the base definitions and tools for type annotations.
415
416 :pep:`484` introduces a :term:`provisional module <provisional API>` to
417 provide these standard definitions and tools, along with some conventions
418 for situations where annotations are not available.
419
420 For example, here is a simple function whose argument and return type
421 are declared in the annotations::
422
423 def greeting(name: str) -> str:
424 return 'Hello ' + name
425
426 While these annotations are available at runtime through the usual
427 :attr:`~object.__annotations__` attribute, *no automatic type checking happens
428 at runtime*. Instead, it is assumed that a separate off-line type checker
429 (e.g. `mypy <https://mypy-lang.org>`_) will be used for on-demand
430 source code analysis.
431
432 The type system supports unions, generic types, and a special type
433 named :class:`~typing.Any` which is consistent with (i.e. assignable to
434 and from) all types.
435
436 .. seealso::
437
438 * :mod:`typing` module documentation
439 * :pep:`484` -- Type Hints
440 PEP written by Guido van Rossum, Jukka Lehtosalo, and Łukasz Langa;
441 implemented by Guido van Rossum.
442 * :pep:`483` -- The Theory of Type Hints
443 PEP written by Guido van Rossum
444
445
446 .. _whatsnew-pep-471:
447
448 PEP 471 - os.scandir() function -- a better and faster directory iterator
449 -------------------------------------------------------------------------
450
451 :pep:`471` adds a new directory iteration function, :func:`os.scandir`,
452 to the standard library. Additionally, :func:`os.walk` is now
453 implemented using ``scandir``, which makes it 3 to 5 times faster
454 on POSIX systems and 7 to 20 times faster on Windows systems. This is
455 largely achieved by greatly reducing the number of calls to :func:`os.stat`
456 required to walk a directory tree.
457
458 Additionally, ``scandir`` returns an iterator, as opposed to returning
459 a list of file names, which improves memory efficiency when iterating
460 over very large directories.
461
462 The following example shows a simple use of :func:`os.scandir` to display all
463 the files (excluding directories) in the given *path* that don't start with
464 ``'.'``. The :meth:`entry.is_file() <os.DirEntry.is_file>` call will generally
465 not make an additional system call::
466
467 for entry in os.scandir(path):
468 if not entry.name.startswith('.') and entry.is_file():
469 print(entry.name)
470
471 .. seealso::
472
473 :pep:`471` -- os.scandir() function -- a better and faster directory iterator
474 PEP written and implemented by Ben Hoyt with the help of Victor Stinner.
475
476
477 .. _whatsnew-pep-475:
478
479 PEP 475: Retry system calls failing with EINTR
480 ----------------------------------------------
481
482 An :py:const:`errno.EINTR` error code is returned whenever a system call, that
483 is waiting for I/O, is interrupted by a signal. Previously, Python would
484 raise :exc:`InterruptedError` in such cases. This meant that, when writing a
485 Python application, the developer had two choices:
486
487 #. Ignore the ``InterruptedError``.
488 #. Handle the ``InterruptedError`` and attempt to restart the interrupted
489 system call at every call site.
490
491 The first option makes an application fail intermittently.
492 The second option adds a large amount of boilerplate that makes the
493 code nearly unreadable. Compare::
494
495 print("Hello World")
496
497 and::
498
499 while True:
500 try:
501 print("Hello World")
502 break
503 except InterruptedError:
504 continue
505
506 :pep:`475` implements automatic retry of system calls on
507 ``EINTR``. This removes the burden of dealing with ``EINTR``
508 or :exc:`InterruptedError` in user code in most situations and makes
509 Python programs, including the standard library, more robust. Note that
510 the system call is only retried if the signal handler does not raise an
511 exception.
512
513 Below is a list of functions which are now retried when interrupted
514 by a signal:
515
516 * :func:`open` and :func:`io.open`;
517
518 * functions of the :mod:`faulthandler` module;
519
520 * :mod:`os` functions: :func:`~os.fchdir`, :func:`~os.fchmod`,
521 :func:`~os.fchown`, :func:`~os.fdatasync`, :func:`~os.fstat`,
522 :func:`~os.fstatvfs`, :func:`~os.fsync`, :func:`~os.ftruncate`,
523 :func:`~os.mkfifo`, :func:`~os.mknod`, :func:`~os.open`,
524 :func:`~os.posix_fadvise`, :func:`~os.posix_fallocate`, :func:`~os.pread`,
525 :func:`~os.pwrite`, :func:`~os.read`, :func:`~os.readv`, :func:`~os.sendfile`,
526 :func:`~os.wait3`, :func:`~os.wait4`, :func:`~os.wait`,
527 :func:`~os.waitid`, :func:`~os.waitpid`, :func:`~os.write`,
528 :func:`~os.writev`;
529
530 * special cases: :func:`os.close` and :func:`os.dup2` now ignore
531 :py:const:`~errno.EINTR` errors; the syscall is not retried (see the PEP
532 for the rationale);
533
534 * :mod:`select` functions: :func:`devpoll.poll() <select.devpoll.poll>`,
535 :func:`epoll.poll() <select.epoll.poll>`,
536 :func:`kqueue.control() <select.kqueue.control>`,
537 :func:`poll.poll() <select.poll.poll>`, :func:`~select.select`;
538
539 * methods of the :class:`~socket.socket` class: :meth:`~socket.socket.accept`,
540 :meth:`~socket.socket.connect` (except for non-blocking sockets),
541 :meth:`~socket.socket.recv`, :meth:`~socket.socket.recvfrom`,
542 :meth:`~socket.socket.recvmsg`, :meth:`~socket.socket.send`,
543 :meth:`~socket.socket.sendall`, :meth:`~socket.socket.sendmsg`,
544 :meth:`~socket.socket.sendto`;
545
546 * :func:`signal.sigtimedwait` and :func:`signal.sigwaitinfo`;
547
548 * :func:`time.sleep`.
549
550 .. seealso::
551
552 :pep:`475` -- Retry system calls failing with EINTR
553 PEP and implementation written by Charles-François Natali and
554 Victor Stinner, with the help of Antoine Pitrou (the French connection).
555
556
557 .. _whatsnew-pep-479:
558
559 PEP 479: Change StopIteration handling inside generators
560 --------------------------------------------------------
561
562 The interaction of generators and :exc:`StopIteration` in Python 3.4 and
563 earlier was sometimes surprising, and could conceal obscure bugs. Previously,
564 ``StopIteration`` raised accidentally inside a generator function was
565 interpreted as the end of the iteration by the loop construct driving the
566 generator.
567
568 :pep:`479` changes the behavior of generators: when a ``StopIteration``
569 exception is raised inside a generator, it is replaced with a
570 :exc:`RuntimeError` before it exits the generator frame. The main goal of
571 this change is to ease debugging in the situation where an unguarded
572 :func:`next` call raises ``StopIteration`` and causes the iteration controlled
573 by the generator to terminate silently. This is particularly pernicious in
574 combination with the ``yield from`` construct.
575
576 This is a backwards incompatible change, so to enable the new behavior,
577 a :term:`__future__` import is necessary::
578
579 >>> from __future__ import generator_stop
580
581 >>> def gen():
582 ... next(iter([]))
583 ... yield
584 ...
585 >>> next(gen())
586 Traceback (most recent call last):
587 File "<stdin>", line 2, in gen
588 StopIteration
589
590 The above exception was the direct cause of the following exception:
591
592 Traceback (most recent call last):
593 File "<stdin>", line 1, in <module>
594 RuntimeError: generator raised StopIteration
595
596 Without a ``__future__`` import, a :exc:`PendingDeprecationWarning` will be
597 raised whenever a :exc:`StopIteration` exception is raised inside a generator.
598
599 .. seealso::
600
601 :pep:`479` -- Change StopIteration handling inside generators
602 PEP written by Chris Angelico and Guido van Rossum. Implemented by
603 Chris Angelico, Yury Selivanov and Nick Coghlan.
604
605
606 .. _whatsnew-pep-485:
607
608 PEP 485: A function for testing approximate equality
609 ----------------------------------------------------
610
611 :pep:`485` adds the :func:`math.isclose` and :func:`cmath.isclose`
612 functions which tell whether two values are approximately equal or
613 "close" to each other. Whether or not two values are considered
614 close is determined according to given absolute and relative tolerances.
615 Relative tolerance is the maximum allowed difference between ``isclose``
616 arguments, relative to the larger absolute value::
617
618 >>> import math
619 >>> a = 5.0
620 >>> b = 4.99998
621 >>> math.isclose(a, b, rel_tol=1e-5)
622 True
623 >>> math.isclose(a, b, rel_tol=1e-6)
624 False
625
626 It is also possible to compare two values using absolute tolerance, which
627 must be a non-negative value::
628
629 >>> import math
630 >>> a = 5.0
631 >>> b = 4.99998
632 >>> math.isclose(a, b, abs_tol=0.00003)
633 True
634 >>> math.isclose(a, b, abs_tol=0.00001)
635 False
636
637 .. seealso::
638
639 :pep:`485` -- A function for testing approximate equality
640 PEP written by Christopher Barker; implemented by Chris Barker and
641 Tal Einat.
642
643
644 .. _whatsnew-pep-486:
645
646 PEP 486: Make the Python Launcher aware of virtual environments
647 ---------------------------------------------------------------
648
649 :pep:`486` makes the Windows launcher (see :pep:`397`) aware of an active
650 virtual environment. When the default interpreter would be used and the
651 ``VIRTUAL_ENV`` environment variable is set, the interpreter in the virtual
652 environment will be used.
653
654 .. seealso::
655
656 :pep:`486` -- Make the Python Launcher aware of virtual environments
657 PEP written and implemented by Paul Moore.
658
659
660 .. _whatsnew-pep-488:
661
662 PEP 488: Elimination of PYO files
663 ---------------------------------
664
665 :pep:`488` does away with the concept of ``.pyo`` files. This means that
666 ``.pyc`` files represent both unoptimized and optimized bytecode. To prevent the
667 need to constantly regenerate bytecode files, ``.pyc`` files now have an
668 optional ``opt-`` tag in their name when the bytecode is optimized. This has the
669 side-effect of no more bytecode file name clashes when running under either
670 :option:`-O` or :option:`-OO`. Consequently, bytecode files generated from
671 :option:`-O`, and :option:`-OO` may now exist simultaneously.
672 :func:`importlib.util.cache_from_source` has an updated API to help with
673 this change.
674
675 .. seealso::
676
677 :pep:`488` -- Elimination of PYO files
678 PEP written and implemented by Brett Cannon.
679
680
681 .. _whatsnew-pep-489:
682
683 PEP 489: Multi-phase extension module initialization
684 ----------------------------------------------------
685
686 :pep:`489` updates extension module initialization to take advantage of the
687 two step module loading mechanism introduced by :pep:`451` in Python 3.4.
688
689 This change brings the import semantics of extension modules that opt-in to
690 using the new mechanism much closer to those of Python source and bytecode
691 modules, including the ability to use any valid identifier as a module name,
692 rather than being restricted to ASCII.
693
694 .. seealso::
695
696 :pep:`489` -- Multi-phase extension module initialization
697 PEP written by Petr Viktorin, Stefan Behnel, and Nick Coghlan;
698 implemented by Petr Viktorin.
699
700
701 Other Language Changes
702 ======================
703
704 Some smaller changes made to the core Python language are:
705
706 * Added the ``"namereplace"`` error handlers. The ``"backslashreplace"``
707 error handlers now work with decoding and translating.
708 (Contributed by Serhiy Storchaka in :issue:`19676` and :issue:`22286`.)
709
710 * The :option:`-b` option now affects comparisons of :class:`bytes` with
711 :class:`int`. (Contributed by Serhiy Storchaka in :issue:`23681`.)
712
713 * New Kazakh ``kz1048`` and Tajik ``koi8_t`` :ref:`codecs <standard-encodings>`.
714 (Contributed by Serhiy Storchaka in :issue:`22682` and :issue:`22681`.)
715
716 * Property docstrings are now writable. This is especially useful for
717 :func:`collections.namedtuple` docstrings.
718 (Contributed by Berker Peksag in :issue:`24064`.)
719
720 * Circular imports involving relative imports are now supported.
721 (Contributed by Brett Cannon and Antoine Pitrou in :issue:`17636`.)
722
723
724 New Modules
725 ===========
726
727 typing
728 ------
729
730 The new :mod:`typing` :term:`provisional <provisional API>` module
731 provides standard definitions and tools for function type annotations.
732 See :ref:`Type Hints <whatsnew-pep-484>` for more information.
733
734 .. _whatsnew-zipapp:
735
736 zipapp
737 ------
738
739 The new :mod:`zipapp` module (specified in :pep:`441`) provides an API and
740 command line tool for creating executable Python Zip Applications, which
741 were introduced in Python 2.6 in :issue:`1739468`, but which were not well
742 publicized, either at the time or since.
743
744 With the new module, bundling your application is as simple as putting all
745 the files, including a ``__main__.py`` file, into a directory ``myapp``
746 and running:
747
748 .. code-block:: shell-session
749
750 $ python -m zipapp myapp
751 $ python myapp.pyz
752
753 The module implementation has been contributed by Paul Moore in
754 :issue:`23491`.
755
756 .. seealso::
757
758 :pep:`441` -- Improving Python ZIP Application Support
759
760
761 Improved Modules
762 ================
763
764 argparse
765 --------
766
767 The :class:`~argparse.ArgumentParser` class now allows disabling
768 :ref:`abbreviated usage <prefix-matching>` of long options by setting
769 :ref:`allow_abbrev` to ``False``. (Contributed by Jonathan Paugh,
770 Steven Bethard, paul j3 and Daniel Eriksson in :issue:`14910`.)
771
772
773 asyncio
774 -------
775
776 Since the :mod:`asyncio` module is :term:`provisional <provisional API>`,
777 all changes introduced in Python 3.5 have also been backported to Python 3.4.x.
778
779 Notable changes in the :mod:`asyncio` module since Python 3.4.0:
780
781 * New debugging APIs: :meth:`loop.set_debug() <asyncio.loop.set_debug>`
782 and :meth:`loop.get_debug() <asyncio.loop.get_debug>` methods.
783 (Contributed by Victor Stinner.)
784
785 * The proactor event loop now supports SSL.
786 (Contributed by Antoine Pitrou and Victor Stinner in :issue:`22560`.)
787
788 * A new :meth:`loop.is_closed() <asyncio.loop.is_closed>` method to
789 check if the event loop is closed.
790 (Contributed by Victor Stinner in :issue:`21326`.)
791
792 * A new :meth:`loop.create_task() <asyncio.loop.create_task>`
793 to conveniently create and schedule a new :class:`~asyncio.Task`
794 for a coroutine. The ``create_task`` method is also used by all
795 asyncio functions that wrap coroutines into tasks, such as
796 :func:`asyncio.wait`, :func:`asyncio.gather`, etc.
797 (Contributed by Victor Stinner.)
798
799 * A new :meth:`transport.get_write_buffer_limits() <asyncio.WriteTransport.get_write_buffer_limits>`
800 method to inquire for *high-* and *low-* water limits of the flow
801 control.
802 (Contributed by Victor Stinner.)
803
804 * The :func:`!async` function is deprecated in favor of
805 :func:`~asyncio.ensure_future`.
806 (Contributed by Yury Selivanov.)
807
808 * New :meth:`loop.set_task_factory()
809 <asyncio.loop.set_task_factory>` and
810 :meth:`loop.get_task_factory() <asyncio.loop.get_task_factory>`
811 methods to customize the task factory that :meth:`loop.create_task()
812 <asyncio.loop.create_task>` method uses. (Contributed by Yury
813 Selivanov.)
814
815 * New :meth:`Queue.join() <asyncio.Queue.join>` and
816 :meth:`Queue.task_done() <asyncio.Queue.task_done>` queue methods.
817 (Contributed by Victor Stinner.)
818
819 * The ``JoinableQueue`` class was removed, in favor of the
820 :class:`asyncio.Queue` class.
821 (Contributed by Victor Stinner.)
822
823 Updates in 3.5.1:
824
825 * The :func:`~asyncio.ensure_future` function and all functions that
826 use it, such as :meth:`loop.run_until_complete() <asyncio.loop.run_until_complete>`,
827 now accept all kinds of :term:`awaitable objects <awaitable>`.
828 (Contributed by Yury Selivanov.)
829
830 * New :func:`~asyncio.run_coroutine_threadsafe` function to submit
831 coroutines to event loops from other threads.
832 (Contributed by Vincent Michel.)
833
834 * New :meth:`Transport.is_closing() <asyncio.BaseTransport.is_closing>`
835 method to check if the transport is closing or closed.
836 (Contributed by Yury Selivanov.)
837
838 * The :meth:`loop.create_server() <asyncio.loop.create_server>`
839 method can now accept a list of hosts.
840 (Contributed by Yann Sionneau.)
841
842 Updates in 3.5.2:
843
844 * New :meth:`loop.create_future() <asyncio.loop.create_future>`
845 method to create Future objects. This allows alternative event
846 loop implementations, such as
847 `uvloop <https://github.com/MagicStack/uvloop>`_, to provide a faster
848 :class:`asyncio.Future` implementation.
849 (Contributed by Yury Selivanov.)
850
851 * New :meth:`loop.get_exception_handler() <asyncio.loop.get_exception_handler>`
852 method to get the current exception handler.
853 (Contributed by Yury Selivanov.)
854
855 * New :meth:`StreamReader.readuntil() <asyncio.StreamReader.readuntil>`
856 method to read data from the stream until a separator bytes
857 sequence appears.
858 (Contributed by Mark Korenberg.)
859
860 * The :meth:`loop.create_connection() <asyncio.loop.create_connection>`
861 and :meth:`loop.create_server() <asyncio.loop.create_server>`
862 methods are optimized to avoid calling the system ``getaddrinfo``
863 function if the address is already resolved.
864 (Contributed by A. Jesse Jiryu Davis.)
865
866 * The :meth:`loop.sock_connect(sock, address) <asyncio.loop.sock_connect>`
867 no longer requires the *address* to be resolved prior to the call.
868 (Contributed by A. Jesse Jiryu Davis.)
869
870
871 bz2
872 ---
873
874 The :meth:`BZ2Decompressor.decompress <bz2.BZ2Decompressor.decompress>`
875 method now accepts an optional *max_length* argument to limit the maximum
876 size of decompressed data. (Contributed by Nikolaus Rath in :issue:`15955`.)
877
878
879 cgi
880 ---
881
882 The :class:`!FieldStorage` class now supports the :term:`context manager`
883 protocol. (Contributed by Berker Peksag in :issue:`20289`.)
884
885
886 cmath
887 -----
888
889 A new function :func:`~cmath.isclose` provides a way to test for approximate
890 equality. (Contributed by Chris Barker and Tal Einat in :issue:`24270`.)
891
892
893 code
894 ----
895
896 The :func:`InteractiveInterpreter.showtraceback() <code.InteractiveInterpreter.showtraceback>`
897 method now prints the full chained traceback, just like the interactive
898 interpreter. (Contributed by Claudiu Popa in :issue:`17442`.)
899
900
901 collections
902 -----------
903
904 .. _whatsnew-ordereddict:
905
906 The :class:`~collections.OrderedDict` class is now implemented in C, which
907 makes it 4 to 100 times faster. (Contributed by Eric Snow in :issue:`16991`.)
908
909 :meth:`!OrderedDict.items`, :meth:`!OrderedDict.keys`,
910 and :meth:`!OrderedDict.values` views now support :func:`reversed` iteration.
911 (Contributed by Serhiy Storchaka in :issue:`19505`.)
912
913 The :class:`~collections.deque` class now defines
914 :meth:`~collections.deque.index`, :meth:`~collections.deque.insert`, and
915 :meth:`~collections.deque.copy`, and supports the ``+`` and ``*`` operators.
916 This allows deques to be recognized as a :class:`~collections.abc.MutableSequence`
917 and improves their substitutability for lists.
918 (Contributed by Raymond Hettinger in :issue:`23704`.)
919
920 Docstrings produced by :func:`~collections.namedtuple` can now be updated::
921
922 Point = namedtuple('Point', ['x', 'y'])
923 Point.__doc__ += ': Cartesian coordinate'
924 Point.x.__doc__ = 'abscissa'
925 Point.y.__doc__ = 'ordinate'
926
927 (Contributed by Berker Peksag in :issue:`24064`.)
928
929 The :class:`~collections.UserString` class now implements the
930 :meth:`~object.__getnewargs__`, :meth:`~object.__rmod__`, :meth:`~str.casefold`,
931 :meth:`~str.format_map`, :meth:`~str.isprintable`, and :meth:`~str.maketrans`
932 methods to match the corresponding methods of :class:`str`.
933 (Contributed by Joe Jevnik in :issue:`22189`.)
934
935
936 collections.abc
937 ---------------
938
939 The :meth:`!Sequence.index` method now
940 accepts *start* and *stop* arguments to match the corresponding methods
941 of :class:`tuple`, :class:`list`, etc.
942 (Contributed by Devin Jeanpierre in :issue:`23086`.)
943
944 A new :class:`~collections.abc.Generator` abstract base class. (Contributed
945 by Stefan Behnel in :issue:`24018`.)
946
947 New :class:`~collections.abc.Awaitable`, :class:`~collections.abc.Coroutine`,
948 :class:`~collections.abc.AsyncIterator`, and
949 :class:`~collections.abc.AsyncIterable` abstract base classes.
950 (Contributed by Yury Selivanov in :issue:`24184`.)
951
952 For earlier Python versions, a backport of the new ABCs is available in an
953 external :pypi:`PyPI package <backports_abc>`.
954
955
956 compileall
957 ----------
958
959 A new :mod:`compileall` option, :samp:`-j {N}`, allows running *N* workers
960 simultaneously to perform parallel bytecode compilation.
961 The :func:`~compileall.compile_dir` function has a corresponding ``workers``
962 parameter. (Contributed by Claudiu Popa in :issue:`16104`.)
963
964 Another new option, ``-r``, allows controlling the maximum recursion
965 level for subdirectories. (Contributed by Claudiu Popa in :issue:`19628`.)
966
967 The ``-q`` command line option can now be specified more than once, in
968 which case all output, including errors, will be suppressed. The corresponding
969 ``quiet`` parameter in :func:`~compileall.compile_dir`,
970 :func:`~compileall.compile_file`, and :func:`~compileall.compile_path` can now
971 accept an integer value indicating the level of output suppression.
972 (Contributed by Thomas Kluyver in :issue:`21338`.)
973
974
975 concurrent.futures
976 ------------------
977
978 The :meth:`Executor.map() <concurrent.futures.Executor.map>` method now accepts a
979 *chunksize* argument to allow batching of tasks to improve performance when
980 :meth:`~concurrent.futures.ProcessPoolExecutor` is used.
981 (Contributed by Dan O'Reilly in :issue:`11271`.)
982
983 The number of workers in the :class:`~concurrent.futures.ThreadPoolExecutor`
984 constructor is optional now. The default value is 5 times the number of CPUs.
985 (Contributed by Claudiu Popa in :issue:`21527`.)
986
987
988 configparser
989 ------------
990
991 :mod:`configparser` now provides a way to customize the conversion
992 of values by specifying a dictionary of converters in the
993 :class:`~configparser.ConfigParser` constructor, or by defining them
994 as methods in ``ConfigParser`` subclasses. Converters defined in
995 a parser instance are inherited by its section proxies.
996
997 Example::
998
999 >>> import configparser
1000 >>> conv = {}
1001 >>> conv['list'] = lambda v: [e.strip() for e in v.split() if e.strip()]
1002 >>> cfg = configparser.ConfigParser(converters=conv)
1003 >>> cfg.read_string("""
1004 ... [s]
1005 ... list = a b c d e f g
1006 ... """)
1007 >>> cfg.get('s', 'list')
1008 'a b c d e f g'
1009 >>> cfg.getlist('s', 'list')
1010 ['a', 'b', 'c', 'd', 'e', 'f', 'g']
1011 >>> section = cfg['s']
1012 >>> section.getlist('list')
1013 ['a', 'b', 'c', 'd', 'e', 'f', 'g']
1014
1015 (Contributed by Łukasz Langa in :issue:`18159`.)
1016
1017
1018 contextlib
1019 ----------
1020
1021 The new :func:`~contextlib.redirect_stderr` :term:`context manager` (similar to
1022 :func:`~contextlib.redirect_stdout`) makes it easier for utility scripts to
1023 handle inflexible APIs that write their output to :data:`sys.stderr` and
1024 don't provide any options to redirect it::
1025
1026 >>> import contextlib, io, logging
1027 >>> f = io.StringIO()
1028 >>> with contextlib.redirect_stderr(f):
1029 ... logging.warning('warning')
1030 ...
1031 >>> f.getvalue()
1032 'WARNING:root:warning\n'
1033
1034 (Contributed by Berker Peksag in :issue:`22389`.)
1035
1036
1037 csv
1038 ---
1039
1040 The :meth:`~csv.csvwriter.writerow` method now supports arbitrary iterables,
1041 not just sequences. (Contributed by Serhiy Storchaka in :issue:`23171`.)
1042
1043
1044 curses
1045 ------
1046
1047 The new :func:`~curses.update_lines_cols` function updates the :data:`~curses.LINES`
1048 and :data:`~curses.COLS` module variables. This is useful for detecting
1049 manual screen resizing. (Contributed by Arnon Yaari in :issue:`4254`.)
1050
1051
1052 dbm
1053 ---
1054
1055 :func:`dumb.open <dbm.dumb.open>` always creates a new database when the flag
1056 has the value ``"n"``. (Contributed by Claudiu Popa in :issue:`18039`.)
1057
1058
1059 difflib
1060 -------
1061
1062 The charset of HTML documents generated by
1063 :meth:`HtmlDiff.make_file() <difflib.HtmlDiff.make_file>`
1064 can now be customized by using a new *charset* keyword-only argument.
1065 The default charset of HTML document changed from ``"ISO-8859-1"``
1066 to ``"utf-8"``.
1067 (Contributed by Berker Peksag in :issue:`2052`.)
1068
1069 The :func:`~difflib.diff_bytes` function can now compare lists of byte
1070 strings. This fixes a regression from Python 2.
1071 (Contributed by Terry J. Reedy and Greg Ward in :issue:`17445`.)
1072
1073
1074 distutils
1075 ---------
1076
1077 Both the ``build`` and ``build_ext`` commands now accept a ``-j`` option to
1078 enable parallel building of extension modules.
1079 (Contributed by Antoine Pitrou in :issue:`5309`.)
1080
1081 The ``distutils`` module now supports ``xz`` compression, and can be
1082 enabled by passing ``xztar`` as an argument to ``bdist --format``.
1083 (Contributed by Serhiy Storchaka in :issue:`16314`.)
1084
1085
1086 doctest
1087 -------
1088
1089 The :func:`~doctest.DocTestSuite` function returns an empty
1090 :class:`unittest.TestSuite` if *module* contains no docstrings, instead of
1091 raising :exc:`ValueError`. (Contributed by Glenn Jones in :issue:`15916`.)
1092
1093
1094 email
1095 -----
1096
1097 A new policy option :attr:`Policy.mangle_from_ <email.policy.Policy.mangle_from_>`
1098 controls whether or not lines that start with ``"From "`` in email bodies are
1099 prefixed with a ``">"`` character by generators. The default is ``True`` for
1100 :attr:`~email.policy.compat32` and ``False`` for all other policies.
1101 (Contributed by Milan Oberkirch in :issue:`20098`.)
1102
1103 A new
1104 :meth:`Message.get_content_disposition() <email.message.Message.get_content_disposition>`
1105 method provides easy access to a canonical value for the
1106 :mailheader:`Content-Disposition` header.
1107 (Contributed by Abhilash Raj in :issue:`21083`.)
1108
1109 A new policy option :attr:`EmailPolicy.utf8 <email.policy.EmailPolicy.utf8>`
1110 can be set to ``True`` to encode email headers using the UTF-8 charset instead
1111 of using encoded words. This allows ``Messages`` to be formatted according to
1112 :rfc:`6532` and used with an SMTP server that supports the :rfc:`6531`
1113 ``SMTPUTF8`` extension. (Contributed by R. David Murray in
1114 :issue:`24211`.)
1115
1116 The :class:`mime.text.MIMEText <email.mime.text.MIMEText>` constructor now
1117 accepts a :class:`charset.Charset <email.charset.Charset>` instance.
1118 (Contributed by Claude Paroz and Berker Peksag in :issue:`16324`.)
1119
1120
1121 enum
1122 ----
1123
1124 The :class:`~enum.Enum` callable has a new parameter *start* to
1125 specify the initial number of enum values if only *names* are provided::
1126
1127 >>> Animal = enum.Enum('Animal', 'cat dog', start=10)
1128 >>> Animal.cat
1129 <Animal.cat: 10>
1130 >>> Animal.dog
1131 <Animal.dog: 11>
1132
1133 (Contributed by Ethan Furman in :issue:`21706`.)
1134
1135
1136 faulthandler
1137 ------------
1138
1139 The :func:`~faulthandler.enable`, :func:`~faulthandler.register`,
1140 :func:`~faulthandler.dump_traceback` and
1141 :func:`~faulthandler.dump_traceback_later` functions now accept file
1142 descriptors in addition to file-like objects.
1143 (Contributed by Wei Wu in :issue:`23566`.)
1144
1145
1146 functools
1147 ---------
1148
1149 .. _whatsnew-lrucache:
1150
1151 Most of the :func:`~functools.lru_cache` machinery is now implemented in C, making
1152 it significantly faster. (Contributed by Matt Joiner, Alexey Kachayev, and
1153 Serhiy Storchaka in :issue:`14373`.)
1154
1155
1156 glob
1157 ----
1158
1159 The :func:`~glob.iglob` and :func:`~glob.glob` functions now support recursive
1160 search in subdirectories, using the ``"**"`` pattern.
1161 (Contributed by Serhiy Storchaka in :issue:`13968`.)
1162
1163
1164 gzip
1165 ----
1166
1167 The *mode* argument of the :class:`~gzip.GzipFile` constructor now
1168 accepts ``"x"`` to request exclusive creation.
1169 (Contributed by Tim Heaney in :issue:`19222`.)
1170
1171
1172 heapq
1173 -----
1174
1175 Element comparison in :func:`~heapq.merge` can now be customized by
1176 passing a :term:`key function` in a new optional *key* keyword argument,
1177 and a new optional *reverse* keyword argument can be used to reverse element
1178 comparison::
1179
1180 >>> import heapq
1181 >>> a = ['9', '777', '55555']
1182 >>> b = ['88', '6666']
1183 >>> list(heapq.merge(a, b, key=len))
1184 ['9', '88', '777', '6666', '55555']
1185 >>> list(heapq.merge(reversed(a), reversed(b), key=len, reverse=True))
1186 ['55555', '6666', '777', '88', '9']
1187
1188 (Contributed by Raymond Hettinger in :issue:`13742`.)
1189
1190
1191 http
1192 ----
1193
1194 A new :class:`HTTPStatus <http.HTTPStatus>` enum that defines a set of
1195 HTTP status codes, reason phrases and long descriptions written in English.
1196 (Contributed by Demian Brecht in :issue:`21793`.)
1197
1198
1199 http.client
1200 -----------
1201
1202 :meth:`HTTPConnection.getresponse() <http.client.HTTPConnection.getresponse>`
1203 now raises a :exc:`~http.client.RemoteDisconnected` exception when a
1204 remote server connection is closed unexpectedly. Additionally, if a
1205 :exc:`ConnectionError` (of which ``RemoteDisconnected``
1206 is a subclass) is raised, the client socket is now closed automatically,
1207 and will reconnect on the next request::
1208
1209 import http.client
1210 conn = http.client.HTTPConnection('www.python.org')
1211 for retries in range(3):
1212 try:
1213 conn.request('GET', '/')
1214 resp = conn.getresponse()
1215 except http.client.RemoteDisconnected:
1216 pass
1217
1218 (Contributed by Martin Panter in :issue:`3566`.)
1219
1220
1221 idlelib and IDLE
1222 ----------------
1223
1224 Since idlelib implements the IDLE shell and editor and is not intended for
1225 import by other programs, it gets improvements with every release. See
1226 :file:`Lib/idlelib/NEWS.txt` for a cumulative list of changes since 3.4.0,
1227 as well as changes made in future 3.5.x releases. This file is also available
1228 from the IDLE :menuselection:`Help --> About IDLE` dialog.
1229
1230
1231 imaplib
1232 -------
1233
1234 The :class:`~imaplib.IMAP4` class now supports the :term:`context manager` protocol.
1235 When used in a :keyword:`with` statement, the IMAP4 ``LOGOUT``
1236 command will be called automatically at the end of the block.
1237 (Contributed by Tarek Ziadé and Serhiy Storchaka in :issue:`4972`.)
1238
1239 The :mod:`imaplib` module now supports :rfc:`5161` (ENABLE Extension)
1240 and :rfc:`6855` (UTF-8 Support) via the :meth:`IMAP4.enable() <imaplib.IMAP4.enable>`
1241 method. A new :attr:`IMAP4.utf8_enabled <imaplib.IMAP4.utf8_enabled>`
1242 attribute tracks whether or not :rfc:`6855` support is enabled.
1243 (Contributed by Milan Oberkirch, R. David Murray, and Maciej Szulik in
1244 :issue:`21800`.)
1245
1246 The :mod:`imaplib` module now automatically encodes non-ASCII string usernames
1247 and passwords using UTF-8, as recommended by the RFCs. (Contributed by Milan
1248 Oberkirch in :issue:`21800`.)
1249
1250
1251 imghdr
1252 ------
1253
1254 The :func:`!what` function now recognizes the
1255 `OpenEXR <https://www.openexr.com>`_ format
1256 (contributed by Martin Vignali and Claudiu Popa in :issue:`20295`),
1257 and the `WebP <https://en.wikipedia.org/wiki/WebP>`_ format
1258 (contributed by Fabrice Aneche and Claudiu Popa in :issue:`20197`.)
1259
1260
1261 importlib
1262 ---------
1263
1264 The :class:`util.LazyLoader <importlib.util.LazyLoader>` class allows for
1265 lazy loading of modules in applications where startup time is important.
1266 (Contributed by Brett Cannon in :issue:`17621`.)
1267
1268 The :func:`abc.InspectLoader.source_to_code() <importlib.abc.InspectLoader.source_to_code>`
1269 method is now a static method. This makes it easier to initialize a module
1270 object with code compiled from a string by running
1271 ``exec(code, module.__dict__)``.
1272 (Contributed by Brett Cannon in :issue:`21156`.)
1273
1274 The new :func:`util.module_from_spec() <importlib.util.module_from_spec>`
1275 function is now the preferred way to create a new module. As opposed to
1276 creating a :class:`types.ModuleType` instance directly, this new function
1277 will set the various import-controlled attributes based on the passed-in
1278 spec object. (Contributed by Brett Cannon in :issue:`20383`.)
1279
1280
1281 inspect
1282 -------
1283
1284 Both the :class:`~inspect.Signature` and :class:`~inspect.Parameter` classes are
1285 now picklable and hashable. (Contributed by Yury Selivanov in :issue:`20726`
1286 and :issue:`20334`.)
1287
1288 A new
1289 :meth:`BoundArguments.apply_defaults() <inspect.BoundArguments.apply_defaults>`
1290 method provides a way to set default values for missing arguments::
1291
1292 >>> def foo(a, b='ham', *args): pass
1293 >>> ba = inspect.signature(foo).bind('spam')
1294 >>> ba.apply_defaults()
1295 >>> ba.arguments
1296 OrderedDict([('a', 'spam'), ('b', 'ham'), ('args', ())])
1297
1298 (Contributed by Yury Selivanov in :issue:`24190`.)
1299
1300 A new class method
1301 :meth:`Signature.from_callable() <inspect.Signature.from_callable>` makes
1302 subclassing of :class:`~inspect.Signature` easier. (Contributed
1303 by Yury Selivanov and Eric Snow in :issue:`17373`.)
1304
1305 The :func:`~inspect.signature` function now accepts a *follow_wrapped*
1306 optional keyword argument, which, when set to ``False``, disables automatic
1307 following of ``__wrapped__`` links.
1308 (Contributed by Yury Selivanov in :issue:`20691`.)
1309
1310 A set of new functions to inspect
1311 :term:`coroutine functions <coroutine function>` and
1312 :term:`coroutine objects <coroutine>` has been added:
1313 :func:`~inspect.iscoroutine`, :func:`~inspect.iscoroutinefunction`,
1314 :func:`~inspect.isawaitable`, :func:`~inspect.getcoroutinelocals`,
1315 and :func:`~inspect.getcoroutinestate`.
1316 (Contributed by Yury Selivanov in :issue:`24017` and :issue:`24400`.)
1317
1318 The :func:`~inspect.stack`, :func:`~inspect.trace`,
1319 :func:`~inspect.getouterframes`, and :func:`~inspect.getinnerframes`
1320 functions now return a list of named tuples.
1321 (Contributed by Daniel Shahaf in :issue:`16808`.)
1322
1323
1324 io
1325 --
1326
1327 A new :meth:`BufferedIOBase.readinto1() <io.BufferedIOBase.readinto1>`
1328 method, that uses at most one call to the underlying raw stream's
1329 :meth:`RawIOBase.read() <io.RawIOBase.read>` or
1330 :meth:`RawIOBase.readinto() <io.RawIOBase.readinto>` methods.
1331 (Contributed by Nikolaus Rath in :issue:`20578`.)
1332
1333
1334 ipaddress
1335 ---------
1336
1337 Both the :class:`~ipaddress.IPv4Network` and :class:`~ipaddress.IPv6Network` classes
1338 now accept an ``(address, netmask)`` tuple argument, so as to easily construct
1339 network objects from existing addresses::
1340
1341 >>> import ipaddress
1342 >>> ipaddress.IPv4Network(('127.0.0.0', 8))
1343 IPv4Network('127.0.0.0/8')
1344 >>> ipaddress.IPv4Network(('127.0.0.0', '255.0.0.0'))
1345 IPv4Network('127.0.0.0/8')
1346
1347 (Contributed by Peter Moody and Antoine Pitrou in :issue:`16531`.)
1348
1349 A new :attr:`~ipaddress.IPv4Address.reverse_pointer` attribute for the
1350 :class:`~ipaddress.IPv4Address` and :class:`~ipaddress.IPv6Address` classes
1351 returns the name of the reverse DNS PTR record::
1352
1353 >>> import ipaddress
1354 >>> addr = ipaddress.IPv4Address('127.0.0.1')
1355 >>> addr.reverse_pointer
1356 '1.0.0.127.in-addr.arpa'
1357 >>> addr6 = ipaddress.IPv6Address('::1')
1358 >>> addr6.reverse_pointer
1359 '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa'
1360
1361 (Contributed by Leon Weber in :issue:`20480`.)
1362
1363
1364 json
1365 ----
1366
1367 The :mod:`json.tool` command line interface now preserves the order of keys in
1368 JSON objects passed in input. The new ``--sort-keys`` option can be used
1369 to sort the keys alphabetically. (Contributed by Berker Peksag
1370 in :issue:`21650`.)
1371
1372 JSON decoder now raises :exc:`~json.JSONDecodeError` instead of
1373 :exc:`ValueError` to provide better context information about the error.
1374 (Contributed by Serhiy Storchaka in :issue:`19361`.)
1375
1376
1377 linecache
1378 ---------
1379
1380 A new :func:`~linecache.lazycache` function can be used to capture information
1381 about a non-file-based module to permit getting its lines later via
1382 :func:`~linecache.getline`. This avoids doing I/O until a line is actually
1383 needed, without having to carry the module globals around indefinitely.
1384 (Contributed by Robert Collins in :issue:`17911`.)
1385
1386
1387 locale
1388 ------
1389
1390 A new :func:`~locale.delocalize` function can be used to convert a string into
1391 a normalized number string, taking the ``LC_NUMERIC`` settings into account::
1392
1393 >>> import locale
1394 >>> locale.setlocale(locale.LC_NUMERIC, 'de_DE.UTF-8')
1395 'de_DE.UTF-8'
1396 >>> locale.delocalize('1.234,56')
1397 '1234.56'
1398 >>> locale.setlocale(locale.LC_NUMERIC, 'en_US.UTF-8')
1399 'en_US.UTF-8'
1400 >>> locale.delocalize('1,234.56')
1401 '1234.56'
1402
1403 (Contributed by Cédric Krier in :issue:`13918`.)
1404
1405
1406 logging
1407 -------
1408
1409 All logging methods (:class:`~logging.Logger` :meth:`~logging.Logger.log`,
1410 :meth:`~logging.Logger.exception`, :meth:`~logging.Logger.critical`,
1411 :meth:`~logging.Logger.debug`, etc.), now accept exception instances
1412 as an *exc_info* argument, in addition to boolean values and exception
1413 tuples::
1414
1415 >>> import logging
1416 >>> try:
1417 ... 1/0
1418 ... except ZeroDivisionError as ex:
1419 ... logging.error('exception', exc_info=ex)
1420 ERROR:root:exception
1421
1422 (Contributed by Yury Selivanov in :issue:`20537`.)
1423
1424 The :class:`handlers.HTTPHandler <logging.handlers.HTTPHandler>` class now
1425 accepts an optional :class:`ssl.SSLContext` instance to configure SSL
1426 settings used in an HTTP connection.
1427 (Contributed by Alex Gaynor in :issue:`22788`.)
1428
1429 The :class:`handlers.QueueListener <logging.handlers.QueueListener>` class now
1430 takes a *respect_handler_level* keyword argument which, if set to ``True``,
1431 will pass messages to handlers taking handler levels into account.
1432 (Contributed by Vinay Sajip.)
1433
1434
1435 lzma
1436 ----
1437
1438 The :meth:`LZMADecompressor.decompress() <lzma.LZMADecompressor.decompress>`
1439 method now accepts an optional *max_length* argument to limit the maximum
1440 size of decompressed data.
1441 (Contributed by Martin Panter in :issue:`15955`.)
1442
1443
1444 math
1445 ----
1446
1447 Two new constants have been added to the :mod:`math` module: :data:`~math.inf`
1448 and :data:`~math.nan`. (Contributed by Mark Dickinson in :issue:`23185`.)
1449
1450 A new function :func:`~math.isclose` provides a way to test for approximate
1451 equality. (Contributed by Chris Barker and Tal Einat in :issue:`24270`.)
1452
1453 A new :func:`~math.gcd` function has been added. The :func:`!fractions.gcd`
1454 function is now deprecated. (Contributed by Mark Dickinson and Serhiy
1455 Storchaka in :issue:`22486`.)
1456
1457
1458 multiprocessing
1459 ---------------
1460
1461 :func:`sharedctypes.synchronized() <multiprocessing.sharedctypes.synchronized>`
1462 objects now support the :term:`context manager` protocol.
1463 (Contributed by Charles-François Natali in :issue:`21565`.)
1464
1465
1466 operator
1467 --------
1468
1469 :func:`~operator.attrgetter`, :func:`~operator.itemgetter`,
1470 and :func:`~operator.methodcaller` objects now support pickling.
1471 (Contributed by Josh Rosenberg and Serhiy Storchaka in :issue:`22955`.)
1472
1473 New :func:`~operator.matmul` and :func:`~operator.imatmul` functions
1474 to perform matrix multiplication.
1475 (Contributed by Benjamin Peterson in :issue:`21176`.)
1476
1477
1478 os
1479 --
1480
1481 The new :func:`~os.scandir` function returning an iterator of
1482 :class:`~os.DirEntry` objects has been added. If possible, :func:`~os.scandir`
1483 extracts file attributes while scanning a directory, removing the need to
1484 perform subsequent system calls to determine file type or attributes, which may
1485 significantly improve performance. (Contributed by Ben Hoyt with the help
1486 of Victor Stinner in :issue:`22524`.)
1487
1488 On Windows, a new
1489 :attr:`stat_result.st_file_attributes <os.stat_result.st_file_attributes>`
1490 attribute is now available. It corresponds to the ``dwFileAttributes`` member
1491 of the ``BY_HANDLE_FILE_INFORMATION`` structure returned by
1492 ``GetFileInformationByHandle()``. (Contributed by Ben Hoyt in :issue:`21719`.)
1493
1494 The :func:`~os.urandom` function now uses the ``getrandom()`` syscall on Linux 3.17
1495 or newer, and ``getentropy()`` on OpenBSD 5.6 and newer, removing the need to
1496 use ``/dev/urandom`` and avoiding failures due to potential file descriptor
1497 exhaustion. (Contributed by Victor Stinner in :issue:`22181`.)
1498
1499 New :func:`~os.get_blocking` and :func:`~os.set_blocking` functions allow
1500 getting and setting a file descriptor's blocking mode (:const:`~os.O_NONBLOCK`.)
1501 (Contributed by Victor Stinner in :issue:`22054`.)
1502
1503 The :func:`~os.truncate` and :func:`~os.ftruncate` functions are now supported
1504 on Windows. (Contributed by Steve Dower in :issue:`23668`.)
1505
1506 There is a new :func:`os.path.commonpath` function returning the longest
1507 common sub-path of each passed pathname. Unlike the
1508 :func:`os.path.commonprefix` function, it always returns a valid
1509 path::
1510
1511 >>> os.path.commonprefix(['/usr/lib', '/usr/local/lib'])
1512 '/usr/l'
1513
1514 >>> os.path.commonpath(['/usr/lib', '/usr/local/lib'])
1515 '/usr'
1516
1517 (Contributed by Rafik Draoui and Serhiy Storchaka in :issue:`10395`.)
1518
1519
1520 pathlib
1521 -------
1522
1523 The new :meth:`Path.samefile() <pathlib.Path.samefile>` method can be used
1524 to check whether the path points to the same file as another path, which can
1525 be either another :class:`~pathlib.Path` object, or a string::
1526
1527 >>> import pathlib
1528 >>> p1 = pathlib.Path('/etc/hosts')
1529 >>> p2 = pathlib.Path('/etc/../etc/hosts')
1530 >>> p1.samefile(p2)
1531 True
1532
1533 (Contributed by Vajrasky Kok and Antoine Pitrou in :issue:`19775`.)
1534
1535 The :meth:`Path.mkdir() <pathlib.Path.mkdir>` method now accepts a new optional
1536 *exist_ok* argument to match ``mkdir -p`` and :func:`os.makedirs`
1537 functionality. (Contributed by Berker Peksag in :issue:`21539`.)
1538
1539 There is a new :meth:`Path.expanduser() <pathlib.Path.expanduser>` method to
1540 expand ``~`` and ``~user`` prefixes. (Contributed by Serhiy Storchaka and
1541 Claudiu Popa in :issue:`19776`.)
1542
1543 A new :meth:`Path.home() <pathlib.Path.home>` class method can be used to get
1544 a :class:`~pathlib.Path` instance representing the user’s home
1545 directory.
1546 (Contributed by Victor Salgado and Mayank Tripathi in :issue:`19777`.)
1547
1548 New :meth:`Path.write_text() <pathlib.Path.write_text>`,
1549 :meth:`Path.read_text() <pathlib.Path.read_text>`,
1550 :meth:`Path.write_bytes() <pathlib.Path.write_bytes>`,
1551 :meth:`Path.read_bytes() <pathlib.Path.read_bytes>` methods to simplify
1552 read/write operations on files.
1553
1554 The following code snippet will create or rewrite existing file
1555 ``~/spam42``::
1556
1557 >>> import pathlib
1558 >>> p = pathlib.Path('~/spam42')
1559 >>> p.expanduser().write_text('ham')
1560 3
1561
1562 (Contributed by Christopher Welborn in :issue:`20218`.)
1563
1564
1565 pickle
1566 ------
1567
1568 Nested objects, such as unbound methods or nested classes, can now be pickled
1569 using :ref:`pickle protocols <pickle-protocols>` older than protocol version 4.
1570 Protocol version 4 already supports these cases. (Contributed by Serhiy
1571 Storchaka in :issue:`23611`.)
1572
1573
1574 poplib
1575 ------
1576
1577 A new :meth:`POP3.utf8() <poplib.POP3.utf8>` command enables :rfc:`6856`
1578 (Internationalized Email) support, if a POP server supports it.
1579 (Contributed by Milan OberKirch in :issue:`21804`.)
1580
1581
1582 re
1583 --
1584
1585 References and conditional references to groups with fixed length are now
1586 allowed in lookbehind assertions::
1587
1588 >>> import re
1589 >>> pat = re.compile(r'(a|b).(?<=\1)c')
1590 >>> pat.match('aac')
1591 <_sre.SRE_Match object; span=(0, 3), match='aac'>
1592 >>> pat.match('bbc')
1593 <_sre.SRE_Match object; span=(0, 3), match='bbc'>
1594
1595 (Contributed by Serhiy Storchaka in :issue:`9179`.)
1596
1597 The number of capturing groups in regular expressions is no longer limited to
1598 100. (Contributed by Serhiy Storchaka in :issue:`22437`.)
1599
1600 The :func:`~re.sub` and :func:`~re.subn` functions now replace unmatched
1601 groups with empty strings instead of raising an exception.
1602 (Contributed by Serhiy Storchaka in :issue:`1519638`.)
1603
1604 The :class:`re.error <re.PatternError>` exceptions have new attributes,
1605 :attr:`~re.PatternError.msg`, :attr:`~re.PatternError.pattern`,
1606 :attr:`~re.PatternError.pos`, :attr:`~re.PatternError.lineno`,
1607 and :attr:`~re.PatternError.colno`, that provide better context
1608 information about the error::
1609
1610 >>> re.compile("""
1611 ... (?x)
1612 ... .++
1613 ... """)
1614 Traceback (most recent call last):
1615 ...
1616 sre_constants.error: multiple repeat at position 16 (line 3, column 7)
1617
1618 (Contributed by Serhiy Storchaka in :issue:`22578`.)
1619
1620
1621 readline
1622 --------
1623
1624 A new :func:`~readline.append_history_file` function can be used to append
1625 the specified number of trailing elements in history to the given file.
1626 (Contributed by Bruno Cauet in :issue:`22940`.)
1627
1628
1629 selectors
1630 ---------
1631
1632 The new :class:`~selectors.DevpollSelector` supports efficient
1633 ``/dev/poll`` polling on Solaris.
1634 (Contributed by Giampaolo Rodola' in :issue:`18931`.)
1635
1636
1637 shutil
1638 ------
1639
1640 The :func:`~shutil.move` function now accepts a *copy_function* argument,
1641 allowing, for example, the :func:`~shutil.copy` function to be used instead of
1642 the default :func:`~shutil.copy2` if there is a need to ignore file metadata
1643 when moving.
1644 (Contributed by Claudiu Popa in :issue:`19840`.)
1645
1646 The :func:`~shutil.make_archive` function now supports the *xztar* format.
1647 (Contributed by Serhiy Storchaka in :issue:`5411`.)
1648
1649
1650 signal
1651 ------
1652
1653 On Windows, the :func:`~signal.set_wakeup_fd` function now also supports
1654 socket handles. (Contributed by Victor Stinner in :issue:`22018`.)
1655
1656 Various ``SIG*`` constants in the :mod:`signal` module have been converted into
1657 :mod:`Enums <enum>`. This allows meaningful names to be printed
1658 during debugging, instead of integer "magic numbers".
1659 (Contributed by Giampaolo Rodola' in :issue:`21076`.)
1660
1661
1662 smtpd
1663 -----
1664
1665 Both the :class:`!SMTPServer` and :class:`!SMTPChannel` classes now
1666 accept a *decode_data* keyword argument to determine if the ``DATA`` portion of
1667 the SMTP transaction is decoded using the ``"utf-8"`` codec or is instead
1668 provided to the
1669 :meth:`!SMTPServer.process_message`
1670 method as a byte string. The default is ``True`` for backward compatibility
1671 reasons, but will change to ``False`` in Python 3.6. If *decode_data* is set
1672 to ``False``, the ``process_message`` method must be prepared to accept keyword
1673 arguments.
1674 (Contributed by Maciej Szulik in :issue:`19662`.)
1675
1676 The :class:`!SMTPServer` class now advertises the ``8BITMIME`` extension
1677 (:rfc:`6152`) if *decode_data* has been set ``True``. If the client
1678 specifies ``BODY=8BITMIME`` on the ``MAIL`` command, it is passed to
1679 :meth:`!SMTPServer.process_message`
1680 via the *mail_options* keyword.
1681 (Contributed by Milan Oberkirch and R. David Murray in :issue:`21795`.)
1682
1683 The :class:`!SMTPServer` class now also supports the ``SMTPUTF8``
1684 extension (:rfc:`6531`: Internationalized Email). If the client specified
1685 ``SMTPUTF8 BODY=8BITMIME`` on the ``MAIL`` command, they are passed to
1686 :meth:`!SMTPServer.process_message`
1687 via the *mail_options* keyword. It is the responsibility of the
1688 ``process_message`` method to correctly handle the ``SMTPUTF8`` data.
1689 (Contributed by Milan Oberkirch in :issue:`21725`.)
1690
1691 It is now possible to provide, directly or via name resolution, IPv6
1692 addresses in the :class:`!SMTPServer` constructor, and have it
1693 successfully connect. (Contributed by Milan Oberkirch in :issue:`14758`.)
1694
1695
1696 smtplib
1697 -------
1698
1699 A new :meth:`SMTP.auth() <smtplib.SMTP.auth>` method provides a convenient way to
1700 implement custom authentication mechanisms. (Contributed by Milan
1701 Oberkirch in :issue:`15014`.)
1702
1703 The :meth:`SMTP.set_debuglevel() <smtplib.SMTP.set_debuglevel>` method now
1704 accepts an additional debuglevel (2), which enables timestamps in debug
1705 messages. (Contributed by Gavin Chappell and Maciej Szulik in :issue:`16914`.)
1706
1707 Both the :meth:`SMTP.sendmail() <smtplib.SMTP.sendmail>` and
1708 :meth:`SMTP.send_message() <smtplib.SMTP.send_message>` methods now
1709 support :rfc:`6531` (SMTPUTF8).
1710 (Contributed by Milan Oberkirch and R. David Murray in :issue:`22027`.)
1711
1712
1713 sndhdr
1714 ------
1715
1716 The :func:`!what` and :func:`!whathdr` functions now return
1717 a :func:`~collections.namedtuple`. (Contributed by Claudiu Popa in
1718 :issue:`18615`.)
1719
1720
1721 socket
1722 ------
1723
1724 Functions with timeouts now use a monotonic clock, instead of a system clock.
1725 (Contributed by Victor Stinner in :issue:`22043`.)
1726
1727 A new :meth:`socket.sendfile() <socket.socket.sendfile>` method allows
1728 sending a file over a socket by using the high-performance :func:`os.sendfile`
1729 function on UNIX, resulting in uploads being from 2 to 3 times faster than when
1730 using plain :meth:`socket.send() <socket.socket.send>`.
1731 (Contributed by Giampaolo Rodola' in :issue:`17552`.)
1732
1733 The :meth:`socket.sendall() <socket.socket.sendall>` method no longer resets the
1734 socket timeout every time bytes are received or sent. The socket timeout is
1735 now the maximum total duration to send all data.
1736 (Contributed by Victor Stinner in :issue:`23853`.)
1737
1738 The *backlog* argument of the :meth:`socket.listen() <socket.socket.listen>`
1739 method is now optional. By default it is set to
1740 :data:`SOMAXCONN <socket.SOMAXCONN>` or to ``128``, whichever is less.
1741 (Contributed by Charles-François Natali in :issue:`21455`.)
1742
1743
1744 ssl
1745 ---
1746
1747 .. _whatsnew-sslmemorybio:
1748
1749 Memory BIO Support
1750 ~~~~~~~~~~~~~~~~~~
1751
1752 (Contributed by Geert Jansen in :issue:`21965`.)
1753
1754 The new :class:`~ssl.SSLObject` class has been added to provide SSL protocol
1755 support for cases when the network I/O capabilities of :class:`~ssl.SSLSocket`
1756 are not necessary or are suboptimal. ``SSLObject`` represents
1757 an SSL protocol instance, but does not implement any network I/O methods, and
1758 instead provides a memory buffer interface. The new :class:`~ssl.MemoryBIO`
1759 class can be used to pass data between Python and an SSL protocol instance.
1760
1761 The memory BIO SSL support is primarily intended to be used in frameworks
1762 implementing asynchronous I/O for which :class:`~ssl.SSLSocket`'s readiness
1763 model ("select/poll") is inefficient.
1764
1765 A new :meth:`SSLContext.wrap_bio() <ssl.SSLContext.wrap_bio>` method can be used
1766 to create a new ``SSLObject`` instance.
1767
1768
1769 Application-Layer Protocol Negotiation Support
1770 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1771
1772 (Contributed by Benjamin Peterson in :issue:`20188`.)
1773
1774 Where OpenSSL support is present, the :mod:`ssl` module now implements
1775 the *Application-Layer Protocol Negotiation* TLS extension as described
1776 in :rfc:`7301`.
1777
1778 The new :meth:`SSLContext.set_alpn_protocols() <ssl.SSLContext.set_alpn_protocols>`
1779 can be used to specify which protocols a socket should advertise during
1780 the TLS handshake.
1781
1782 The new
1783 :meth:`SSLSocket.selected_alpn_protocol() <ssl.SSLSocket.selected_alpn_protocol>`
1784 returns the protocol that was selected during the TLS handshake.
1785 The :const:`~ssl.HAS_ALPN` flag indicates whether ALPN support is present.
1786
1787
1788 Other Changes
1789 ~~~~~~~~~~~~~
1790
1791 There is a new :meth:`SSLSocket.version() <ssl.SSLSocket.version>` method to
1792 query the actual protocol version in use.
1793 (Contributed by Antoine Pitrou in :issue:`20421`.)
1794
1795 The :class:`~ssl.SSLSocket` class now implements
1796 a :meth:`!SSLSocket.sendfile` method.
1797 (Contributed by Giampaolo Rodola' in :issue:`17552`.)
1798
1799 The :meth:`!SSLSocket.send` method now raises either
1800 the :exc:`ssl.SSLWantReadError` or :exc:`ssl.SSLWantWriteError` exception on a
1801 non-blocking socket if the operation would block. Previously, it would return
1802 ``0``. (Contributed by Nikolaus Rath in :issue:`20951`.)
1803
1804 The :func:`~ssl.cert_time_to_seconds` function now interprets the input time
1805 as UTC and not as local time, per :rfc:`5280`. Additionally, the return
1806 value is always an :class:`int`. (Contributed by Akira Li in :issue:`19940`.)
1807
1808 New :meth:`!SSLObject.shared_ciphers` and
1809 :meth:`SSLSocket.shared_ciphers() <ssl.SSLSocket.shared_ciphers>` methods return
1810 the list of ciphers sent by the client during the handshake.
1811 (Contributed by Benjamin Peterson in :issue:`23186`.)
1812
1813 The :meth:`SSLSocket.do_handshake() <ssl.SSLSocket.do_handshake>`,
1814 :meth:`SSLSocket.read() <ssl.SSLSocket.read>`,
1815 :meth:`!SSLSocket.shutdown`, and
1816 :meth:`SSLSocket.write() <ssl.SSLSocket.write>` methods of the :class:`~ssl.SSLSocket`
1817 class no longer reset the socket timeout every time bytes are received or sent.
1818 The socket timeout is now the maximum total duration of the method.
1819 (Contributed by Victor Stinner in :issue:`23853`.)
1820
1821 The :func:`!match_hostname` function now supports matching of IP addresses.
1822 (Contributed by Antoine Pitrou in :issue:`23239`.)
1823
1824
1825 sqlite3
1826 -------
1827
1828 The :class:`~sqlite3.Row` class now fully supports the sequence protocol,
1829 in particular :func:`reversed` iteration and slice indexing.
1830 (Contributed by Claudiu Popa in :issue:`10203`; by Lucas Sinclair,
1831 Jessica McKellar, and Serhiy Storchaka in :issue:`13583`.)
1832
1833
1834 .. _whatsnew-subprocess:
1835
1836 subprocess
1837 ----------
1838
1839 The new :func:`~subprocess.run` function has been added.
1840 It runs the specified command and returns a
1841 :class:`~subprocess.CompletedProcess` object, which describes a finished
1842 process. The new API is more consistent and is the recommended approach
1843 to invoking subprocesses in Python code that does not need to maintain
1844 compatibility with earlier Python versions.
1845 (Contributed by Thomas Kluyver in :issue:`23342`.)
1846
1847 Examples::
1848
1849 >>> subprocess.run(["ls", "-l"]) # doesn't capture output
1850 CompletedProcess(args=['ls', '-l'], returncode=0)
1851
1852 >>> subprocess.run("exit 1", shell=True, check=True)
1853 Traceback (most recent call last):
1854 ...
1855 subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1
1856
1857 >>> subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE)
1858 CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0,
1859 stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n')
1860
1861
1862 sys
1863 ---
1864
1865 A new :func:`!set_coroutine_wrapper` function allows setting a global
1866 hook that will be called whenever a :term:`coroutine object <coroutine>`
1867 is created by an :keyword:`async def` function. A corresponding
1868 :func:`!get_coroutine_wrapper` can be used to obtain a currently set
1869 wrapper. Both functions are :term:`provisional <provisional API>`,
1870 and are intended for debugging purposes only. (Contributed by Yury Selivanov
1871 in :issue:`24017`.)
1872
1873 A new :func:`~sys.is_finalizing` function can be used to check if the Python
1874 interpreter is :term:`shutting down <interpreter shutdown>`.
1875 (Contributed by Antoine Pitrou in :issue:`22696`.)
1876
1877
1878 sysconfig
1879 ---------
1880
1881 The name of the user scripts directory on Windows now includes the first
1882 two components of the Python version. (Contributed by Paul Moore
1883 in :issue:`23437`.)
1884
1885
1886 tarfile
1887 -------
1888
1889 The *mode* argument of the :func:`~tarfile.open` function now accepts ``"x"``
1890 to request exclusive creation. (Contributed by Berker Peksag in :issue:`21717`.)
1891
1892 The :meth:`TarFile.extractall() <tarfile.TarFile.extractall>` and
1893 :meth:`TarFile.extract() <tarfile.TarFile.extract>` methods now take a keyword
1894 argument *numeric_owner*. If set to ``True``, the extracted files and
1895 directories will be owned by the numeric ``uid`` and ``gid`` from the tarfile.
1896 If set to ``False`` (the default, and the behavior in versions prior to 3.5),
1897 they will be owned by the named user and group in the tarfile.
1898 (Contributed by Michael Vogt and Eric Smith in :issue:`23193`.)
1899
1900 The :meth:`TarFile.list() <tarfile.TarFile.list>` now accepts an optional
1901 *members* keyword argument that can be set to a subset of the list returned
1902 by :meth:`TarFile.getmembers() <tarfile.TarFile.getmembers>`.
1903 (Contributed by Serhiy Storchaka in :issue:`21549`.)
1904
1905
1906 threading
1907 ---------
1908
1909 Both the :meth:`Lock.acquire() <threading.Lock.acquire>` and
1910 :meth:`RLock.acquire() <threading.RLock.acquire>` methods
1911 now use a monotonic clock for timeout management.
1912 (Contributed by Victor Stinner in :issue:`22043`.)
1913
1914
1915 time
1916 ----
1917
1918 The :func:`~time.monotonic` function is now always available.
1919 (Contributed by Victor Stinner in :issue:`22043`.)
1920
1921
1922 timeit
1923 ------
1924
1925 A new command line option ``-u`` or :samp:`--unit={U}` can be used to specify the time
1926 unit for the timer output. Supported options are ``usec``, ``msec``,
1927 or ``sec``. (Contributed by Julian Gindi in :issue:`18983`.)
1928
1929 The :func:`~timeit.timeit` function has a new *globals* parameter for
1930 specifying the namespace in which the code will be running.
1931 (Contributed by Ben Roberts in :issue:`2527`.)
1932
1933
1934 tkinter
1935 -------
1936
1937 The :mod:`!tkinter._fix` module used for setting up the Tcl/Tk environment
1938 on Windows has been replaced by a private function in the :mod:`!_tkinter`
1939 module which makes no permanent changes to environment variables.
1940 (Contributed by Zachary Ware in :issue:`20035`.)
1941
1942
1943 .. _whatsnew-traceback:
1944
1945 traceback
1946 ---------
1947
1948 New :func:`~traceback.walk_stack` and :func:`~traceback.walk_tb`
1949 functions to conveniently traverse frame and
1950 :ref:`traceback objects <traceback-objects>`.
1951 (Contributed by Robert Collins in :issue:`17911`.)
1952
1953 New lightweight classes: :class:`~traceback.TracebackException`,
1954 :class:`~traceback.StackSummary`, and :class:`~traceback.FrameSummary`.
1955 (Contributed by Robert Collins in :issue:`17911`.)
1956
1957 Both the :func:`~traceback.print_tb` and :func:`~traceback.print_stack` functions
1958 now support negative values for the *limit* argument.
1959 (Contributed by Dmitry Kazakov in :issue:`22619`.)
1960
1961
1962 types
1963 -----
1964
1965 A new :func:`~types.coroutine` function to transform
1966 :term:`generator <generator iterator>` and
1967 :class:`generator-like <collections.abc.Generator>` objects into
1968 :term:`awaitables <awaitable>`.
1969 (Contributed by Yury Selivanov in :issue:`24017`.)
1970
1971 A new type called :class:`~types.CoroutineType`, which is used for
1972 :term:`coroutine` objects created by :keyword:`async def` functions.
1973 (Contributed by Yury Selivanov in :issue:`24400`.)
1974
1975
1976 unicodedata
1977 -----------
1978
1979 The :mod:`unicodedata` module now uses data from `Unicode 8.0.0
1980 <https://unicode.org/versions/Unicode8.0.0/>`_.
1981
1982
1983 unittest
1984 --------
1985
1986 The :meth:`TestLoader.loadTestsFromModule() <unittest.TestLoader.loadTestsFromModule>`
1987 method now accepts a keyword-only argument *pattern* which is passed to
1988 ``load_tests`` as the third argument. Found packages are now checked for
1989 ``load_tests`` regardless of whether their path matches *pattern*, because it
1990 is impossible for a package name to match the default pattern.
1991 (Contributed by Robert Collins and Barry A. Warsaw in :issue:`16662`.)
1992
1993 Unittest discovery errors now are exposed in the
1994 :data:`TestLoader.errors <unittest.TestLoader.errors>` attribute of the
1995 :class:`~unittest.TestLoader` instance.
1996 (Contributed by Robert Collins in :issue:`19746`.)
1997
1998 A new command line option ``--locals`` to show local variables in
1999 tracebacks. (Contributed by Robert Collins in :issue:`22936`.)
2000
2001
2002 unittest.mock
2003 -------------
2004
2005 The :class:`~unittest.mock.Mock` class has the following improvements:
2006
2007 * The class constructor has a new *unsafe* parameter, which causes mock
2008 objects to raise :exc:`AttributeError` on attribute names starting
2009 with ``"assert"``.
2010 (Contributed by Kushal Das in :issue:`21238`.)
2011
2012 * A new :meth:`Mock.assert_not_called() <unittest.mock.Mock.assert_not_called>`
2013 method to check if the mock object was called.
2014 (Contributed by Kushal Das in :issue:`21262`.)
2015
2016 The :class:`~unittest.mock.MagicMock` class now supports
2017 :meth:`~object.__truediv__`, :meth:`~object.__divmod__`
2018 and :meth:`~object.__matmul__` operators.
2019 (Contributed by Johannes Baiter in :issue:`20968`, and Håkan Lövdahl
2020 in :issue:`23581` and :issue:`23568`.)
2021
2022 It is no longer necessary to explicitly pass ``create=True`` to the
2023 :func:`~unittest.mock.patch` function when patching builtin names.
2024 (Contributed by Kushal Das in :issue:`17660`.)
2025
2026
2027 urllib
2028 ------
2029
2030 A new
2031 :class:`request.HTTPPasswordMgrWithPriorAuth <urllib.request.HTTPPasswordMgrWithPriorAuth>`
2032 class allows HTTP Basic Authentication credentials to be managed so as to
2033 eliminate unnecessary ``401`` response handling, or to unconditionally send
2034 credentials on the first request in order to communicate with servers that
2035 return a ``404`` response instead of a ``401`` if the ``Authorization`` header
2036 is not sent. (Contributed by Matej Cepl in :issue:`19494` and Akshit Khurana in
2037 :issue:`7159`.)
2038
2039 A new *quote_via* argument for the
2040 :func:`parse.urlencode() <urllib.parse.urlencode>`
2041 function provides a way to control the encoding of query parts if needed.
2042 (Contributed by Samwyse and Arnon Yaari in :issue:`13866`.)
2043
2044 The :func:`request.urlopen() <urllib.request.urlopen>` function accepts an
2045 :class:`ssl.SSLContext` object as a *context* argument, which will be used for
2046 the HTTPS connection. (Contributed by Alex Gaynor in :issue:`22366`.)
2047
2048 The :func:`parse.urljoin() <urllib.parse.urljoin>` was updated to use the
2049 :rfc:`3986` semantics for the resolution of relative URLs, rather than
2050 :rfc:`1808` and :rfc:`2396`.
2051 (Contributed by Demian Brecht and Senthil Kumaran in :issue:`22118`.)
2052
2053
2054 wsgiref
2055 -------
2056
2057 The *headers* argument of the :class:`headers.Headers <wsgiref.headers.Headers>`
2058 class constructor is now optional.
2059 (Contributed by Pablo Torres Navarrete and SilentGhost in :issue:`5800`.)
2060
2061
2062 xmlrpc
2063 ------
2064
2065 The :class:`client.ServerProxy <xmlrpc.client.ServerProxy>` class now supports
2066 the :term:`context manager` protocol.
2067 (Contributed by Claudiu Popa in :issue:`20627`.)
2068
2069 The :class:`client.ServerProxy <xmlrpc.client.ServerProxy>` constructor now accepts
2070 an optional :class:`ssl.SSLContext` instance.
2071 (Contributed by Alex Gaynor in :issue:`22960`.)
2072
2073
2074 xml.sax
2075 -------
2076
2077 SAX parsers now support a character stream of the
2078 :class:`xmlreader.InputSource <xml.sax.xmlreader.InputSource>` object.
2079 (Contributed by Serhiy Storchaka in :issue:`2175`.)
2080
2081 :func:`~xml.sax.parseString` now accepts a :class:`str` instance.
2082 (Contributed by Serhiy Storchaka in :issue:`10590`.)
2083
2084
2085 zipfile
2086 -------
2087
2088 ZIP output can now be written to unseekable streams.
2089 (Contributed by Serhiy Storchaka in :issue:`23252`.)
2090
2091 The *mode* argument of :meth:`ZipFile.open() <zipfile.ZipFile.open>` method now
2092 accepts ``"x"`` to request exclusive creation.
2093 (Contributed by Serhiy Storchaka in :issue:`21717`.)
2094
2095
2096 Other module-level changes
2097 ==========================
2098
2099 Many functions in the :mod:`mmap`, :mod:`!ossaudiodev`, :mod:`socket`,
2100 :mod:`ssl`, and :mod:`codecs` modules now accept writable
2101 :term:`bytes-like objects <bytes-like object>`.
2102 (Contributed by Serhiy Storchaka in :issue:`23001`.)
2103
2104
2105 Optimizations
2106 =============
2107
2108 The :func:`os.walk` function has been sped up by 3 to 5 times on POSIX systems,
2109 and by 7 to 20 times on Windows. This was done using the new :func:`os.scandir`
2110 function, which exposes file information from the underlying ``readdir`` or
2111 ``FindFirstFile``/``FindNextFile`` system calls. (Contributed by
2112 Ben Hoyt with help from Victor Stinner in :issue:`23605`.)
2113
2114 Construction of ``bytes(int)`` (filled by zero bytes) is faster and uses less
2115 memory for large objects. ``calloc()`` is used instead of ``malloc()`` to
2116 allocate memory for these objects.
2117 (Contributed by Victor Stinner in :issue:`21233`.)
2118
2119 Some operations on :mod:`ipaddress` :class:`~ipaddress.IPv4Network` and
2120 :class:`~ipaddress.IPv6Network` have been massively sped up, such as
2121 :meth:`~ipaddress.IPv4Network.subnets`, :meth:`~ipaddress.IPv4Network.supernet`,
2122 :func:`~ipaddress.summarize_address_range`, :func:`~ipaddress.collapse_addresses`.
2123 The speed up can range from 3 to 15 times.
2124 (Contributed by Antoine Pitrou, Michel Albert, and Markus in
2125 :issue:`21486`, :issue:`21487`, :issue:`20826`, :issue:`23266`.)
2126
2127 Pickling of :mod:`ipaddress` objects was optimized to produce significantly
2128 smaller output. (Contributed by Serhiy Storchaka in :issue:`23133`.)
2129
2130 Many operations on :class:`io.BytesIO` are now 50% to 100% faster.
2131 (Contributed by Serhiy Storchaka in :issue:`15381` and David Wilson in
2132 :issue:`22003`.)
2133
2134 The :func:`marshal.dumps` function is now faster: 65--85% with versions 3
2135 and 4, 20--25% with versions 0 to 2 on typical data, and up to 5 times in
2136 best cases.
2137 (Contributed by Serhiy Storchaka in :issue:`20416` and :issue:`23344`.)
2138
2139 The UTF-32 encoder is now 3 to 7 times faster.
2140 (Contributed by Serhiy Storchaka in :issue:`15027`.)
2141
2142 Regular expressions are now parsed up to 10% faster.
2143 (Contributed by Serhiy Storchaka in :issue:`19380`.)
2144
2145 The :func:`json.dumps` function was optimized to run with
2146 ``ensure_ascii=False`` as fast as with ``ensure_ascii=True``.
2147 (Contributed by Naoki Inada in :issue:`23206`.)
2148
2149 The :c:func:`PyObject_IsInstance` and :c:func:`PyObject_IsSubclass`
2150 functions have been sped up in the common case that the second argument
2151 has :class:`type` as its metaclass.
2152 (Contributed Georg Brandl by in :issue:`22540`.)
2153
2154 Method caching was slightly improved, yielding up to 5% performance
2155 improvement in some benchmarks.
2156 (Contributed by Antoine Pitrou in :issue:`22847`.)
2157
2158 Objects from the :mod:`random` module now use 50% less memory on 64-bit
2159 builds. (Contributed by Serhiy Storchaka in :issue:`23488`.)
2160
2161 The :func:`property` getter calls are up to 25% faster.
2162 (Contributed by Joe Jevnik in :issue:`23910`.)
2163
2164 Instantiation of :class:`fractions.Fraction` is now up to 30% faster.
2165 (Contributed by Stefan Behnel in :issue:`22464`.)
2166
2167 String methods :meth:`~str.find`, :meth:`~str.rfind`, :meth:`~str.split`,
2168 :meth:`~str.partition` and the :keyword:`in` string operator are now significantly
2169 faster for searching 1-character substrings.
2170 (Contributed by Serhiy Storchaka in :issue:`23573`.)
2171
2172
2173 Build and C API Changes
2174 =======================
2175
2176 New ``calloc`` functions were added:
2177
2178 * :c:func:`PyMem_RawCalloc`,
2179 * :c:func:`PyMem_Calloc`,
2180 * :c:func:`PyObject_Calloc`.
2181
2182 (Contributed by Victor Stinner in :issue:`21233`.)
2183
2184 New encoding/decoding helper functions:
2185
2186 * :c:func:`Py_DecodeLocale` (replaced ``_Py_char2wchar()``),
2187 * :c:func:`Py_EncodeLocale` (replaced ``_Py_wchar2char()``).
2188
2189 (Contributed by Victor Stinner in :issue:`18395`.)
2190
2191 A new :c:func:`PyCodec_NameReplaceErrors` function to replace the unicode
2192 encode error with ``\N{...}`` escapes.
2193 (Contributed by Serhiy Storchaka in :issue:`19676`.)
2194
2195 A new :c:func:`PyErr_FormatV` function similar to :c:func:`PyErr_Format`,
2196 but accepts a :c:type:`va_list` argument.
2197 (Contributed by Antoine Pitrou in :issue:`18711`.)
2198
2199 A new :c:data:`PyExc_RecursionError` exception.
2200 (Contributed by Georg Brandl in :issue:`19235`.)
2201
2202 New :c:func:`PyModule_FromDefAndSpec`, :c:func:`PyModule_FromDefAndSpec2`,
2203 and :c:func:`PyModule_ExecDef` functions introduced by :pep:`489` --
2204 multi-phase extension module initialization.
2205 (Contributed by Petr Viktorin in :issue:`24268`.)
2206
2207 New :c:func:`PyNumber_MatrixMultiply` and
2208 :c:func:`PyNumber_InPlaceMatrixMultiply` functions to perform matrix
2209 multiplication.
2210 (Contributed by Benjamin Peterson in :issue:`21176`. See also :pep:`465`
2211 for details.)
2212
2213 The :c:member:`PyTypeObject.tp_finalize` slot is now part of the stable ABI.
2214
2215 Windows builds now require Microsoft Visual C++ 14.0, which
2216 is available as part of `Visual Studio 2015 <https://visualstudio.microsoft.com/en/vs/older-downloads/#visual-studio-2015-and-other-products>`_.
2217
2218 Extension modules now include a platform information tag in their filename on
2219 some platforms (the tag is optional, and CPython will import extensions without
2220 it, although if the tag is present and mismatched, the extension won't be
2221 loaded):
2222
2223 * On Linux, extension module filenames end with
2224 ``.cpython-<major><minor>m-<architecture>-<os>.pyd``:
2225
2226 * ``<major>`` is the major number of the Python version;
2227 for Python 3.5 this is ``3``.
2228
2229 * ``<minor>`` is the minor number of the Python version;
2230 for Python 3.5 this is ``5``.
2231
2232 * ``<architecture>`` is the hardware architecture the extension module
2233 was built to run on. It's most commonly either ``i386`` for 32-bit Intel
2234 platforms or ``x86_64`` for 64-bit Intel (and AMD) platforms.
2235
2236 * ``<os>`` is always ``linux-gnu``, except for extensions built to
2237 talk to the 32-bit ABI on 64-bit platforms, in which case it is
2238 ``linux-gnu32`` (and ``<architecture>`` will be ``x86_64``).
2239
2240 * On Windows, extension module filenames end with
2241 ``<debug>.cp<major><minor>-<platform>.pyd``:
2242
2243 * ``<major>`` is the major number of the Python version;
2244 for Python 3.5 this is ``3``.
2245
2246 * ``<minor>`` is the minor number of the Python version;
2247 for Python 3.5 this is ``5``.
2248
2249 * ``<platform>`` is the platform the extension module was built for,
2250 either ``win32`` for Win32, ``win_amd64`` for Win64, ``win_ia64`` for
2251 Windows Itanium 64, and ``win_arm`` for Windows on ARM.
2252
2253 * If built in debug mode, ``<debug>`` will be ``_d``,
2254 otherwise it will be blank.
2255
2256 * On OS X platforms, extension module filenames now end with ``-darwin.so``.
2257
2258 * On all other platforms, extension module filenames are the same as they were
2259 with Python 3.4.
2260
2261
2262 Deprecated
2263 ==========
2264
2265 New Keywords
2266 ------------
2267
2268 ``async`` and ``await`` are not recommended to be used as variable, class,
2269 function or module names. Introduced by :pep:`492` in Python 3.5, they will
2270 become proper keywords in Python 3.7.
2271
2272
2273 Deprecated Python Behavior
2274 --------------------------
2275
2276 Raising the :exc:`StopIteration` exception inside a generator will now generate a silent
2277 :exc:`PendingDeprecationWarning`, which will become a non-silent deprecation
2278 warning in Python 3.6 and will trigger a :exc:`RuntimeError` in Python 3.7.
2279 See :ref:`PEP 479: Change StopIteration handling inside generators <whatsnew-pep-479>`
2280 for details.
2281
2282
2283 Unsupported Operating Systems
2284 -----------------------------
2285
2286 Windows XP is no longer supported by Microsoft, thus, per :PEP:`11`, CPython
2287 3.5 is no longer officially supported on this OS.
2288
2289
2290 Deprecated Python modules, functions and methods
2291 ------------------------------------------------
2292
2293 The :mod:`!formatter` module has now graduated to full deprecation and is still
2294 slated for removal in Python 3.6.
2295
2296 The :func:`!asyncio.async` function is deprecated in favor of
2297 :func:`~asyncio.ensure_future`.
2298
2299 The :mod:`!smtpd` module has in the past always decoded the DATA portion of
2300 email messages using the ``utf-8`` codec. This can now be controlled by the
2301 new *decode_data* keyword to :class:`!SMTPServer`. The default value is
2302 ``True``, but this default is deprecated. Specify the *decode_data* keyword
2303 with an appropriate value to avoid the deprecation warning.
2304
2305 Directly assigning values to the :attr:`~http.cookies.Morsel.key`,
2306 :attr:`~http.cookies.Morsel.value` and
2307 :attr:`~http.cookies.Morsel.coded_value` of :class:`http.cookies.Morsel`
2308 objects is deprecated. Use the :meth:`~http.cookies.Morsel.set` method
2309 instead. In addition, the undocumented *LegalChars* parameter of
2310 :meth:`~http.cookies.Morsel.set` is deprecated, and is now ignored.
2311
2312 Passing a format string as keyword argument *format_string* to the
2313 :meth:`~string.Formatter.format` method of the :class:`string.Formatter`
2314 class has been deprecated.
2315 (Contributed by Serhiy Storchaka in :issue:`23671`.)
2316
2317 The :func:`!platform.dist` and :func:`!platform.linux_distribution` functions
2318 are now deprecated. Linux distributions use too many different ways of
2319 describing themselves, so the functionality is left to a package.
2320 (Contributed by Vajrasky Kok and Berker Peksag in :issue:`1322`.)
2321
2322 The previously undocumented ``from_function`` and ``from_builtin`` methods of
2323 :class:`inspect.Signature` are deprecated. Use the new
2324 :meth:`Signature.from_callable() <inspect.Signature.from_callable>`
2325 method instead. (Contributed by Yury Selivanov in :issue:`24248`.)
2326
2327 The :func:`!inspect.getargspec` function is deprecated and scheduled to be
2328 removed in Python 3.6. (See :issue:`20438` for details.)
2329
2330 The :mod:`inspect` :func:`~inspect.getfullargspec`,
2331 :func:`~inspect.getcallargs`, and :func:`!formatargspec` functions are
2332 deprecated in favor of the :func:`inspect.signature` API. (Contributed by Yury
2333 Selivanov in :issue:`20438`.)
2334
2335 :func:`~inspect.getargvalues` and :func:`~inspect.formatargvalues` functions
2336 were inadvertently marked as deprecated with the release of Python 3.5.0.
2337
2338 Use of :const:`re.LOCALE` flag with str patterns or :const:`re.ASCII` is now
2339 deprecated. (Contributed by Serhiy Storchaka in :issue:`22407`.)
2340
2341 Use of unrecognized special sequences consisting of ``'\'`` and an ASCII letter
2342 in regular expression patterns and replacement patterns now raises a
2343 deprecation warning and will be forbidden in Python 3.6.
2344 (Contributed by Serhiy Storchaka in :issue:`23622`.)
2345
2346 The undocumented and unofficial *use_load_tests* default argument of the
2347 :meth:`unittest.TestLoader.loadTestsFromModule` method now is
2348 deprecated and ignored.
2349 (Contributed by Robert Collins and Barry A. Warsaw in :issue:`16662`.)
2350
2351
2352 Removed
2353 =======
2354
2355 API and Feature Removals
2356 ------------------------
2357
2358 The following obsolete and previously deprecated APIs and features have been
2359 removed:
2360
2361 * The ``__version__`` attribute has been dropped from the email package. The
2362 email code hasn't been shipped separately from the stdlib for a long time,
2363 and the ``__version__`` string was not updated in the last few releases.
2364
2365 * The internal ``Netrc`` class in the :mod:`ftplib` module was deprecated in
2366 3.4, and has now been removed.
2367 (Contributed by Matt Chaput in :issue:`6623`.)
2368
2369 * The concept of ``.pyo`` files has been removed.
2370
2371 * The JoinableQueue class in the provisional :mod:`asyncio` module was
2372 deprecated in 3.4.4 and is now removed.
2373 (Contributed by A. Jesse Jiryu Davis in :issue:`23464`.)
2374
2375
2376 Porting to Python 3.5
2377 =====================
2378
2379 This section lists previously described changes and other bugfixes
2380 that may require changes to your code.
2381
2382
2383 Changes in Python behavior
2384 --------------------------
2385
2386 * Due to an oversight, earlier Python versions erroneously accepted the
2387 following syntax::
2388
2389 f(1 for x in [1], *args)
2390 f(1 for x in [1], **kwargs)
2391
2392 Python 3.5 now correctly raises a :exc:`SyntaxError`, as generator
2393 expressions must be put in parentheses if not a sole argument to a function.
2394
2395
2396 Changes in the Python API
2397 -------------------------
2398
2399 * :pep:`475`: System calls are now retried when interrupted by a signal instead
2400 of raising :exc:`InterruptedError` if the Python signal handler does not
2401 raise an exception.
2402
2403 * Before Python 3.5, a :class:`datetime.time` object was considered to be false
2404 if it represented midnight in UTC. This behavior was considered obscure and
2405 error-prone and has been removed in Python 3.5. See :issue:`13936` for full
2406 details.
2407
2408 * The :meth:`!ssl.SSLSocket.send` method now raises either
2409 :exc:`ssl.SSLWantReadError` or :exc:`ssl.SSLWantWriteError`
2410 on a non-blocking socket if the operation would block. Previously,
2411 it would return ``0``. (Contributed by Nikolaus Rath in :issue:`20951`.)
2412
2413 * The ``__name__`` attribute of generators is now set from the function name,
2414 instead of being set from the code name. Use ``gen.gi_code.co_name`` to
2415 retrieve the code name. Generators also have a new ``__qualname__``
2416 attribute, the qualified name, which is now used for the representation
2417 of a generator (``repr(gen)``).
2418 (Contributed by Victor Stinner in :issue:`21205`.)
2419
2420 * The deprecated "strict" mode and argument of :class:`~html.parser.HTMLParser`,
2421 :meth:`!HTMLParser.error`, and the :exc:`!HTMLParserError` exception have been
2422 removed. (Contributed by Ezio Melotti in :issue:`15114`.)
2423 The *convert_charrefs* argument of :class:`~html.parser.HTMLParser` is
2424 now ``True`` by default. (Contributed by Berker Peksag in :issue:`21047`.)
2425
2426 * Although it is not formally part of the API, it is worth noting for porting
2427 purposes (ie: fixing tests) that error messages that were previously of the
2428 form "'sometype' does not support the buffer protocol" are now of the form "a
2429 :term:`bytes-like object` is required, not 'sometype'".
2430 (Contributed by Ezio Melotti in :issue:`16518`.)
2431
2432 * If the current directory is set to a directory that no longer exists then
2433 :exc:`FileNotFoundError` will no longer be raised and instead
2434 :meth:`~importlib.machinery.FileFinder.find_spec` will return ``None``
2435 **without** caching ``None`` in :data:`sys.path_importer_cache`, which is
2436 different than the typical case (:issue:`22834`).
2437
2438 * HTTP status code and messages from :mod:`http.client` and :mod:`http.server`
2439 were refactored into a common :class:`~http.HTTPStatus` enum. The values in
2440 :mod:`http.client` and :mod:`http.server` remain available for backwards
2441 compatibility. (Contributed by Demian Brecht in :issue:`21793`.)
2442
2443 * When an import loader defines :meth:`~importlib.abc.Loader.exec_module`
2444 it is now expected to also define
2445 :meth:`~importlib.abc.Loader.create_module` (raises a
2446 :exc:`DeprecationWarning` now, will be an error in Python 3.6). If the loader
2447 inherits from :class:`importlib.abc.Loader` then there is nothing to do, else
2448 simply define :meth:`~importlib.abc.Loader.create_module` to return
2449 ``None``. (Contributed by Brett Cannon in :issue:`23014`.)
2450
2451 * The :func:`re.split` function always ignored empty pattern matches, so the
2452 ``"x*"`` pattern worked the same as ``"x+"``, and the ``"\b"`` pattern never
2453 worked. Now :func:`re.split` raises a warning if the pattern could match
2454 an empty string. For compatibility, use patterns that never match an empty
2455 string (e.g. ``"x+"`` instead of ``"x*"``). Patterns that could only match
2456 an empty string (such as ``"\b"``) now raise an error.
2457 (Contributed by Serhiy Storchaka in :issue:`22818`.)
2458
2459 * The :class:`http.cookies.Morsel` dict-like interface has been made self
2460 consistent: morsel comparison now takes the :attr:`~http.cookies.Morsel.key`
2461 and :attr:`~http.cookies.Morsel.value` into account,
2462 :meth:`~http.cookies.Morsel.copy` now results in a
2463 :class:`~http.cookies.Morsel` instance rather than a :class:`dict`, and
2464 :meth:`~http.cookies.Morsel.update` will now raise an exception if any of the
2465 keys in the update dictionary are invalid. In addition, the undocumented
2466 *LegalChars* parameter of :func:`~http.cookies.Morsel.set` is deprecated and
2467 is now ignored. (Contributed by Demian Brecht in :issue:`2211`.)
2468
2469 * :pep:`488` has removed ``.pyo`` files from Python and introduced the optional
2470 ``opt-`` tag in ``.pyc`` file names. The
2471 :func:`importlib.util.cache_from_source` has gained an *optimization*
2472 parameter to help control the ``opt-`` tag. Because of this, the
2473 *debug_override* parameter of the function is now deprecated. ``.pyo`` files
2474 are also no longer supported as a file argument to the Python interpreter and
2475 thus serve no purpose when distributed on their own (i.e. sourceless code
2476 distribution). Due to the fact that the magic number for bytecode has changed
2477 in Python 3.5, all old ``.pyo`` files from previous versions of Python are
2478 invalid regardless of this PEP.
2479
2480 * The :mod:`socket` module now exports the :const:`~socket.CAN_RAW_FD_FRAMES`
2481 constant on linux 3.6 and greater.
2482
2483 * The :func:`ssl.cert_time_to_seconds` function now interprets the input time
2484 as UTC and not as local time, per :rfc:`5280`. Additionally, the return
2485 value is always an :class:`int`. (Contributed by Akira Li in :issue:`19940`.)
2486
2487 * The ``pygettext.py`` Tool now uses the standard +NNNN format for timezones in
2488 the POT-Creation-Date header.
2489
2490 * The :mod:`smtplib` module now uses :data:`sys.stderr` instead of the previous
2491 module-level :data:`!stderr` variable for debug output. If your (test)
2492 program depends on patching the module-level variable to capture the debug
2493 output, you will need to update it to capture sys.stderr instead.
2494
2495 * The :meth:`str.startswith` and :meth:`str.endswith` methods no longer return
2496 ``True`` when finding the empty string and the indexes are completely out of
2497 range. (Contributed by Serhiy Storchaka in :issue:`24284`.)
2498
2499 * The :func:`inspect.getdoc` function now returns documentation strings
2500 inherited from base classes. Documentation strings no longer need to be
2501 duplicated if the inherited documentation is appropriate. To suppress an
2502 inherited string, an empty string must be specified (or the documentation
2503 may be filled in). This change affects the output of the :mod:`pydoc`
2504 module and the :func:`help` function.
2505 (Contributed by Serhiy Storchaka in :issue:`15582`.)
2506
2507 * Nested :func:`functools.partial` calls are now flattened. If you were
2508 relying on the previous behavior, you can now either add an attribute to a
2509 :func:`functools.partial` object or you can create a subclass of
2510 :func:`functools.partial`.
2511 (Contributed by Alexander Belopolsky in :issue:`7830`.)
2512
2513 Changes in the C API
2514 --------------------
2515
2516 * The undocumented :c:member:`!format` member of the
2517 (non-public) :c:type:`!PyMemoryViewObject` structure has been removed.
2518 All extensions relying on the relevant parts in ``memoryobject.h``
2519 must be rebuilt.
2520
2521 * The :c:type:`!PyMemAllocator` structure was renamed to
2522 :c:type:`PyMemAllocatorEx` and a new ``calloc`` field was added.
2523
2524 * Removed non-documented macro :c:macro:`!PyObject_REPR()` which leaked references.
2525 Use format character ``%R`` in :c:func:`PyUnicode_FromFormat`-like functions
2526 to format the :func:`repr` of the object.
2527 (Contributed by Serhiy Storchaka in :issue:`22453`.)
2528
2529 * Because the lack of the :attr:`~type.__module__` attribute breaks pickling and
2530 introspection, a deprecation warning is now raised for builtin types without
2531 the :attr:`~type.__module__` attribute. This will be an :exc:`AttributeError` in
2532 the future.
2533 (Contributed by Serhiy Storchaka in :issue:`20204`.)
2534
2535 * As part of the :pep:`492` implementation, the ``tp_reserved`` slot of
2536 :c:type:`PyTypeObject` was replaced with a
2537 :c:member:`~PyTypeObject.tp_as_async` slot. Refer to :ref:`coro-objects` for
2538 new types, structures and functions.
2539
2540
2541 Notable changes in Python 3.5.4
2542 ===============================
2543
2544 New ``make regen-all`` build target
2545 -----------------------------------
2546
2547 To simplify cross-compilation, and to ensure that CPython can reliably be
2548 compiled without requiring an existing version of Python to already be
2549 available, the autotools-based build system no longer attempts to implicitly
2550 recompile generated files based on file modification times.
2551
2552 Instead, a new ``make regen-all`` command has been added to force regeneration
2553 of these files when desired (e.g. after an initial version of Python has
2554 already been built based on the pregenerated versions).
2555
2556 More selective regeneration targets are also defined - see
2557 :source:`Makefile.pre.in` for details.
2558
2559 (Contributed by Victor Stinner in :issue:`23404`.)
2560
2561 .. versionadded:: 3.5.4
2562
2563
2564 Removal of ``make touch`` build target
2565 --------------------------------------
2566
2567 The ``make touch`` build target previously used to request implicit regeneration
2568 of generated files by updating their modification times has been removed.
2569
2570 It has been replaced by the new ``make regen-all`` target.
2571
2572 (Contributed by Victor Stinner in :issue:`23404`.)
2573
2574 .. versionchanged:: 3.5.4
2575