Debian Codegraph @

Doc/whatsnew/3.4.rst

raw
1 ****************************
2 What's New In Python 3.4
3 ****************************
4
5 :Author: \R. David Murray <[email protected]> (Editor)
6
7 .. Rules for maintenance:
8
9 * Anyone can add text to this document, but the maintainer reserves the
10 right to rewrite any additions. In particular, for obscure or esoteric
11 features, the maintainer may reduce any addition to a simple reference to
12 the new documentation rather than explaining the feature inline.
13
14 * While the maintainer will periodically go through Misc/NEWS
15 and add changes, it's best not to rely on this. We know from experience
16 that any changes that aren't in the What's New documentation around the
17 time of the original release will remain largely unknown to the community
18 for years, even if they're added later. We also know from experience that
19 other priorities can arise, and the maintainer will run out of time to do
20 updates -- in such cases, end users will be much better served by partial
21 notifications that at least give a hint about new features to
22 investigate.
23
24 * This is not a complete list of every single change; completeness
25 is the purpose of Misc/NEWS. The What's New should focus on changes that
26 are visible to Python *users* and that *require* a feature release (i.e.
27 most bug fixes should only be recorded in Misc/NEWS)
28
29 * PEPs should not be marked Final until they have an entry in What's New.
30 A placeholder entry that is just a section header and a link to the PEP
31 (e.g ":pep:`397` has been implemented") is acceptable. If a PEP has been
32 implemented and noted in What's New, don't forget to mark it as Final!
33
34 * If you want to draw your new text to the attention of the
35 maintainer, add 'XXX' to the beginning of the paragraph or
36 section.
37
38 * It's OK to add just a very brief note about a change. For
39 example: "The :ref:`~socket.transmogrify()` function was added to the
40 :mod:`socket` module." The maintainer will research the change and
41 write the necessary text (if appropriate). The advantage of doing this
42 is that even if no more descriptive text is ever added, readers will at
43 least have a notification that the new feature exists and a link to the
44 relevant documentation.
45
46 * You can comment out your additions if you like, but it's not
47 necessary (especially when a final release is some months away).
48
49 * Credit the author of a patch or bugfix. Just the name is
50 sufficient; the e-mail address isn't necessary.
51
52 * It's helpful to add the bug/patch number as a comment:
53
54 The :ref:`~socket.transmogrify()` function was added to the
55 :mod:`socket` module. (Contributed by P.Y. Developer in :issue:`12345`.)
56
57 This saves the maintainer the effort of going through the Mercurial log
58 when researching a change.
59
60 * Cross referencing tip: :ref:`mod.attr` will display as ``mod.attr``,
61 while :ref:`~mod.attr` will display as ``attr``.
62
63 This article explains the new features in Python 3.4, compared to 3.3.
64 Python 3.4 was released on March 16, 2014. For full details, see the
65 `changelog <https://docs.python.org/3.4/whatsnew/changelog.html>`_.
66
67
68 .. seealso::
69
70 :pep:`429` -- Python 3.4 Release Schedule
71
72
73
74 Summary -- Release Highlights
75 =============================
76
77 .. This section singles out the most important changes in Python 3.4.
78 Brevity is key.
79
80 New syntax features:
81
82 * No new syntax features were added in Python 3.4.
83
84 Other new features:
85
86 * :ref:`pip should always be available <whatsnew-pep-453>` (:pep:`453`).
87 * :ref:`Newly created file descriptors are non-inheritable <whatsnew-pep-446>`
88 (:pep:`446`).
89 * command line option for :ref:`isolated mode <whatsnew-isolated-mode>`
90 (:issue:`16499`).
91 * :ref:`improvements in the handling of codecs <codec-handling-improvements>`
92 that are not text encodings (multiple issues).
93 * :ref:`A ModuleSpec Type <whatsnew-pep-451>` for the Import System
94 (:pep:`451`). (Affects importer authors.)
95 * The :mod:`marshal` format has been made :ref:`more compact and efficient
96 <whatsnew-marshal-3>` (:issue:`16475`).
97
98 New library modules:
99
100 * :mod:`asyncio`: :ref:`New provisional API for asynchronous IO
101 <whatsnew-asyncio>` (:pep:`3156`).
102 * :mod:`ensurepip`: :ref:`Bootstrapping the pip installer <whatsnew-ensurepip>`
103 (:pep:`453`).
104 * :mod:`enum`: :ref:`Support for enumeration types <whatsnew-enum>`
105 (:pep:`435`).
106 * :mod:`pathlib`: :ref:`Object-oriented filesystem paths <whatsnew-pathlib>`
107 (:pep:`428`).
108 * :mod:`selectors`: :ref:`High-level and efficient I/O multiplexing
109 <whatsnew-selectors>`, built upon the :mod:`select` module primitives (part
110 of :pep:`3156`).
111 * :mod:`statistics`: A basic :ref:`numerically stable statistics library
112 <whatsnew-statistics>` (:pep:`450`).
113 * :mod:`tracemalloc`: :ref:`Trace Python memory allocations
114 <whatsnew-tracemalloc>` (:pep:`454`).
115
116 Significantly improved library modules:
117
118 * :ref:`Single-dispatch generic functions <whatsnew-singledispatch>` in
119 :mod:`functools` (:pep:`443`).
120 * New :mod:`pickle` :ref:`protocol 4 <whatsnew-protocol-4>` (:pep:`3154`).
121 * :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork
122 on Unix <whatsnew-multiprocessing-no-fork>` (:issue:`8713`).
123 * :mod:`email` has a new submodule, :mod:`~email.contentmanager`, and
124 a new :mod:`~email.message.Message` subclass
125 (:class:`~email.message.EmailMessage`) that :ref:`simplify MIME
126 handling <whatsnew_email_contentmanager>` (:issue:`18891`).
127 * The :mod:`inspect` and :mod:`pydoc` modules are now capable of
128 correct introspection of a much wider variety of callable objects,
129 which improves the output of the Python :func:`help` system.
130 * The :mod:`ipaddress` module API has been declared stable
131
132 Security improvements:
133
134 * :ref:`Secure and interchangeable hash algorithm <whatsnew-pep-456>`
135 (:pep:`456`).
136 * :ref:`Make newly created file descriptors non-inheritable <whatsnew-pep-446>`
137 (:pep:`446`) to avoid leaking file descriptors to child processes.
138 * New command line option for :ref:`isolated mode <whatsnew-isolated-mode>`,
139 (:issue:`16499`).
140 * :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork
141 on Unix <whatsnew-multiprocessing-no-fork>`. *spawn* and *forkserver* are
142 more secure because they avoid sharing data with child processes.
143 * :mod:`multiprocessing` child processes on Windows no longer inherit
144 all of the parent's inheritable handles, only the necessary ones.
145 * A new :func:`hashlib.pbkdf2_hmac` function provides
146 the `PKCS#5 password-based key derivation function 2
147 <https://en.wikipedia.org/wiki/PBKDF2>`_.
148 * :ref:`TLSv1.1 and TLSv1.2 support <whatsnew-tls-11-12>` for :mod:`ssl`.
149 * :ref:`Retrieving certificates from the Windows system cert store support
150 <whatsnew34-win-cert-store>` for :mod:`ssl`.
151 * :ref:`Server-side SNI (Server Name Indication) support
152 <whatsnew34-sni>` for :mod:`ssl`.
153 * The :class:`ssl.SSLContext` class has a :ref:`lot of improvements
154 <whatsnew34-sslcontext>`.
155 * All modules in the standard library that support SSL now support server
156 certificate verification, including hostname matching
157 (:func:`!ssl.match_hostname`) and CRLs (Certificate Revocation lists, see
158 :func:`ssl.SSLContext.load_verify_locations`).
159
160 CPython implementation improvements:
161
162 * :ref:`Safe object finalization <whatsnew-pep-442>` (:pep:`442`).
163 * Leveraging :pep:`442`, in most cases :ref:`module globals are no longer set
164 to None during finalization <whatsnew-pep-442>` (:issue:`18214`).
165 * :ref:`Configurable memory allocators <whatsnew-pep-445>` (:pep:`445`).
166 * :ref:`Argument Clinic <whatsnew-pep-436>` (:pep:`436`).
167
168 Please read on for a comprehensive list of user-facing changes, including many
169 other smaller improvements, CPython optimizations, deprecations, and potential
170 porting issues.
171
172
173
174 New Features
175 ============
176
177 .. _whatsnew-pep-453:
178
179 PEP 453: Explicit Bootstrapping of PIP in Python Installations
180 --------------------------------------------------------------
181
182 Bootstrapping pip By Default
183 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
184
185 The new :mod:`ensurepip` module (defined in :pep:`453`) provides a standard
186 cross-platform mechanism to bootstrap the pip installer into Python
187 installations and virtual environments. The version of ``pip`` included
188 with Python 3.4.0 is ``pip`` 1.5.4, and future 3.4.x maintenance releases
189 will update the bundled version to the latest version of ``pip`` that is
190 available at the time of creating the release candidate.
191
192 By default, the commands ``pipX`` and ``pipX.Y`` will be installed on all
193 platforms (where X.Y stands for the version of the Python installation),
194 along with the ``pip`` Python package and its dependencies. On Windows and
195 in virtual environments on all platforms, the unversioned ``pip`` command
196 will also be installed. On other platforms, the system wide unversioned
197 ``pip`` command typically refers to the separately installed Python 2
198 version.
199
200 The ``pyvenv`` command line utility and the :mod:`venv`
201 module make use of the :mod:`ensurepip` module to make ``pip`` readily
202 available in virtual environments. When using the command line utility,
203 ``pip`` is installed by default, while when using the :mod:`venv` module
204 :ref:`venv-api` installation of ``pip`` must be requested explicitly.
205
206 For CPython :ref:`source builds on POSIX systems <building-python-on-unix>`,
207 the ``make install`` and ``make altinstall`` commands bootstrap ``pip`` by
208 default. This behaviour can be controlled through configure options, and
209 overridden through Makefile options.
210
211 On Windows and Mac OS X, the CPython installers now default to installing
212 ``pip`` along with CPython itself (users may opt out of installing it
213 during the installation process). Window users will need to opt in to the
214 automatic ``PATH`` modifications to have ``pip`` available from the command
215 line by default, otherwise it can still be accessed through the Python
216 launcher for Windows as ``py -m pip``.
217
218 As :pep:`discussed in the PEP <0453#recommendations-for-downstream-distributors>`
219 platform packagers may choose not to install
220 these commands by default, as long as, when invoked, they provide clear and
221 simple directions on how to install them on that platform (usually using
222 the system package manager).
223
224 .. note::
225
226 To avoid conflicts between parallel Python 2 and Python 3 installations,
227 only the versioned ``pip3`` and ``pip3.4`` commands are bootstrapped by
228 default when ``ensurepip`` is invoked directly - the ``--default-pip``
229 option is needed to also request the unversioned ``pip`` command.
230 ``pyvenv`` and the Windows installer ensure that the unqualified ``pip``
231 command is made available in those environments, and ``pip`` can always be
232 invoked via the ``-m`` switch rather than directly to avoid ambiguity on
233 systems with multiple Python installations.
234
235
236 Documentation Changes
237 ~~~~~~~~~~~~~~~~~~~~~
238
239 As part of this change, the :ref:`installing-index` and
240 :ref:`distributing-index` sections of the documentation have been
241 completely redesigned as short getting started and FAQ documents. Most
242 packaging documentation has now been moved out to the Python Packaging
243 Authority maintained `Python Packaging User Guide
244 <https://packaging.python.org>`__ and the documentation of the individual
245 projects.
246
247 However, as this migration is currently still incomplete, the legacy
248 versions of those guides remaining available as :ref:`install-index`
249 and :ref:`setuptools-index`.
250
251 .. seealso::
252
253 :pep:`453` -- Explicit bootstrapping of pip in Python installations
254 PEP written by Donald Stufft and Nick Coghlan, implemented by
255 Donald Stufft, Nick Coghlan, Martin von Löwis and Ned Deily.
256
257
258 .. _whatsnew-pep-446:
259
260 PEP 446: Newly Created File Descriptors Are Non-Inheritable
261 -----------------------------------------------------------
262
263 :pep:`446` makes newly created file descriptors :ref:`non-inheritable
264 <fd_inheritance>`. In general, this is the behavior an application will
265 want: when launching a new process, having currently open files also
266 open in the new process can lead to all sorts of hard to find bugs,
267 and potentially to security issues.
268
269 However, there are occasions when inheritance is desired. To support
270 these cases, the following new functions and methods are available:
271
272 * :func:`os.get_inheritable`, :func:`os.set_inheritable`
273 * :func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable`
274 * :meth:`socket.socket.get_inheritable`, :meth:`socket.socket.set_inheritable`
275
276 .. seealso::
277
278 :pep:`446` -- Make newly created file descriptors non-inheritable
279 PEP written and implemented by Victor Stinner.
280
281
282 .. _codec-handling-improvements:
283
284 Improvements to Codec Handling
285 ------------------------------
286
287 Since it was first introduced, the :mod:`codecs` module has always been
288 intended to operate as a type-neutral dynamic encoding and decoding
289 system. However, its close coupling with the Python text model, especially
290 the type restricted convenience methods on the builtin :class:`str`,
291 :class:`bytes` and :class:`bytearray` types, has historically obscured that
292 fact.
293
294 As a key step in clarifying the situation, the :meth:`codecs.encode` and
295 :meth:`codecs.decode` convenience functions are now properly documented in
296 Python 2.7, 3.3 and 3.4. These functions have existed in the :mod:`codecs`
297 module (and have been covered by the regression test suite) since Python 2.4,
298 but were previously only discoverable through runtime introspection.
299
300 Unlike the convenience methods on :class:`str`, :class:`bytes` and
301 :class:`bytearray`, the :mod:`codecs` convenience functions support arbitrary
302 codecs in both Python 2 and Python 3, rather than being limited to Unicode text
303 encodings (in Python 3) or ``basestring`` <-> ``basestring`` conversions (in
304 Python 2).
305
306 In Python 3.4, the interpreter is able to identify the known non-text
307 encodings provided in the standard library and direct users towards these
308 general purpose convenience functions when appropriate::
309
310 >>> b"abcdef".decode("hex")
311 Traceback (most recent call last):
312 File "<stdin>", line 1, in <module>
313 LookupError: 'hex' is not a text encoding; use codecs.decode() to handle arbitrary codecs
314
315 >>> "hello".encode("rot13")
316 Traceback (most recent call last):
317 File "<stdin>", line 1, in <module>
318 LookupError: 'rot13' is not a text encoding; use codecs.encode() to handle arbitrary codecs
319
320 >>> open("foo.txt", encoding="hex")
321 Traceback (most recent call last):
322 File "<stdin>", line 1, in <module>
323 LookupError: 'hex' is not a text encoding; use codecs.open() to handle arbitrary codecs
324
325 In a related change, whenever it is feasible without breaking backwards
326 compatibility, exceptions raised during encoding and decoding operations
327 are wrapped in a chained exception of the same type that mentions the
328 name of the codec responsible for producing the error::
329
330 >>> import codecs
331
332 >>> codecs.decode(b"abcdefgh", "hex")
333 Traceback (most recent call last):
334 File "/usr/lib/python3.4/encodings/hex_codec.py", line 20, in hex_decode
335 return (binascii.a2b_hex(input), len(input))
336 binascii.Error: Non-hexadecimal digit found
337
338 The above exception was the direct cause of the following exception:
339
340 Traceback (most recent call last):
341 File "<stdin>", line 1, in <module>
342 binascii.Error: decoding with 'hex' codec failed (Error: Non-hexadecimal digit found)
343
344 >>> codecs.encode("hello", "bz2")
345 Traceback (most recent call last):
346 File "/usr/lib/python3.4/encodings/bz2_codec.py", line 17, in bz2_encode
347 return (bz2.compress(input), len(input))
348 File "/usr/lib/python3.4/bz2.py", line 498, in compress
349 return comp.compress(data) + comp.flush()
350 TypeError: 'str' does not support the buffer interface
351
352 The above exception was the direct cause of the following exception:
353
354 Traceback (most recent call last):
355 File "<stdin>", line 1, in <module>
356 TypeError: encoding with 'bz2' codec failed (TypeError: 'str' does not support the buffer interface)
357
358 Finally, as the examples above show, these improvements have permitted
359 the restoration of the convenience aliases for the non-Unicode codecs that
360 were themselves restored in Python 3.2. This means that encoding binary data
361 to and from its hexadecimal representation (for example) can now be written
362 as::
363
364 >>> from codecs import encode, decode
365 >>> encode(b"hello", "hex")
366 b'68656c6c6f'
367 >>> decode(b"68656c6c6f", "hex")
368 b'hello'
369
370 The binary and text transforms provided in the standard library are detailed
371 in :ref:`binary-transforms` and :ref:`text-transforms`.
372
373 (Contributed by Nick Coghlan in :issue:`7475`, :issue:`17827`,
374 :issue:`17828` and :issue:`19619`.)
375
376
377 .. _whatsnew-pep-451:
378
379 PEP 451: A ModuleSpec Type for the Import System
380 ------------------------------------------------
381
382 :pep:`451` provides an encapsulation of the information about a module that the
383 import machinery will use to load it (that is, a module specification). This
384 helps simplify both the import implementation and several import-related APIs.
385 The change is also a stepping stone for `several future import-related
386 improvements`__.
387
388 __ https://mail.python.org/pipermail/python-dev/2013-November/130111.html
389
390 The public-facing changes from the PEP are entirely backward-compatible.
391 Furthermore, they should be transparent to everyone but importer authors. Key
392 finder and loader methods have been deprecated, but they will continue working.
393 New importers should use the new methods described in the PEP. Existing
394 importers should be updated to implement the new methods. See the
395 :ref:`deprecated-3.4` section for a list of methods that should be replaced and
396 their replacements.
397
398
399 Other Language Changes
400 ----------------------
401
402 Some smaller changes made to the core Python language are:
403
404 * Unicode database updated to UCD version 6.3.
405
406 * :func:`min` and :func:`max` now accept a *default* keyword-only argument that
407 can be used to specify the value they return if the iterable they are
408 evaluating has no elements. (Contributed by Julian Berman in
409 :issue:`18111`.)
410
411 * Module objects are now :ref:`weakly referenceable <mod-weakref>`.
412
413 * Module ``__file__`` attributes (and related values) should now always
414 contain absolute paths by default, with the sole exception of
415 ``__main__.__file__`` when a script has been executed directly using
416 a relative path. (Contributed by Brett Cannon in :issue:`18416`.)
417
418 * All the UTF-\* codecs (except UTF-7) now reject surrogates during both
419 encoding and decoding unless the ``surrogatepass`` error handler is used,
420 with the exception of the UTF-16 decoder (which accepts valid surrogate pairs)
421 and the UTF-16 encoder (which produces them while encoding non-BMP characters).
422 (Contributed by Victor Stinner, Kang-Hao (Kenny) Lu and Serhiy Storchaka in
423 :issue:`12892`.)
424
425 * New German EBCDIC :ref:`codec <standard-encodings>` ``cp273``. (Contributed
426 by Michael Bierenfeld and Andrew Kuchling in :issue:`1097797`.)
427
428 * New Ukrainian :ref:`codec <standard-encodings>` ``cp1125``. (Contributed by
429 Serhiy Storchaka in :issue:`19668`.)
430
431 * :class:`bytes`.join() and :class:`bytearray`.join() now accept arbitrary
432 buffer objects as arguments. (Contributed by Antoine Pitrou in
433 :issue:`15958`.)
434
435 * The :class:`int` constructor now accepts any object that has an ``__index__``
436 method for its *base* argument. (Contributed by Mark Dickinson in
437 :issue:`16772`.)
438
439 * Frame objects now have a :func:`~frame.clear` method that clears all
440 references to local variables from the frame. (Contributed by Antoine Pitrou
441 in :issue:`17934`.)
442
443 * :class:`memoryview` is now registered as a :class:`Sequence <collections.abc>`,
444 and supports the :func:`reversed` builtin. (Contributed by Nick Coghlan
445 and Claudiu Popa in :issue:`18690` and :issue:`19078`.)
446
447 * Signatures reported by :func:`help` have been modified and improved in
448 several cases as a result of the introduction of Argument Clinic and other
449 changes to the :mod:`inspect` and :mod:`pydoc` modules.
450
451 * :meth:`~object.__length_hint__` is now part of the formal language
452 specification (see :pep:`424`). (Contributed by Armin Ronacher in
453 :issue:`16148`.)
454
455
456 New Modules
457 ===========
458
459
460 .. _whatsnew-asyncio:
461
462 asyncio
463 -------
464
465 The new :mod:`asyncio` module (defined in :pep:`3156`) provides a standard
466 pluggable event loop model for Python, providing solid asynchronous IO
467 support in the standard library, and making it easier for other event loop
468 implementations to interoperate with the standard library and each other.
469
470 For Python 3.4, this module is considered a :term:`provisional API`.
471
472 .. seealso::
473
474 :pep:`3156` -- Asynchronous IO Support Rebooted: the "asyncio" Module
475 PEP written and implementation led by Guido van Rossum.
476
477
478 .. _whatsnew-ensurepip:
479
480 ensurepip
481 ---------
482
483 The new :mod:`ensurepip` module is the primary infrastructure for the
484 :pep:`453` implementation. In the normal course of events end users will not
485 need to interact with this module, but it can be used to manually bootstrap
486 ``pip`` if the automated bootstrapping into an installation or virtual
487 environment was declined.
488
489 :mod:`ensurepip` includes a bundled copy of ``pip``, up-to-date as of the first
490 release candidate of the release of CPython with which it ships (this applies
491 to both maintenance releases and feature releases). ``ensurepip`` does not
492 access the internet. If the installation has internet access, after
493 ``ensurepip`` is run the bundled ``pip`` can be used to upgrade ``pip`` to a
494 more recent release than the bundled one. (Note that such an upgraded version
495 of ``pip`` is considered to be a separately installed package and will not be
496 removed if Python is uninstalled.)
497
498 The module is named *ensure*\ pip because if called when ``pip`` is already
499 installed, it does nothing. It also has an ``--upgrade`` option that will
500 cause it to install the bundled copy of ``pip`` if the existing installed
501 version of ``pip`` is older than the bundled copy.
502
503
504 .. _whatsnew-enum:
505
506 enum
507 ----
508
509 The new :mod:`enum` module (defined in :pep:`435`) provides a standard
510 implementation of enumeration types, allowing other modules (such as
511 :mod:`socket`) to provide more informative error messages and better
512 debugging support by replacing opaque integer constants with backwards
513 compatible enumeration values.
514
515 .. seealso::
516
517 :pep:`435` -- Adding an Enum type to the Python standard library
518 PEP written by Barry Warsaw, Eli Bendersky and Ethan Furman,
519 implemented by Ethan Furman.
520
521
522 .. _whatsnew-pathlib:
523
524 pathlib
525 -------
526
527 The new :mod:`pathlib` module offers classes representing filesystem paths
528 with semantics appropriate for different operating systems. Path classes are
529 divided between *pure paths*, which provide purely computational operations
530 without I/O, and *concrete paths*, which inherit from pure paths but also
531 provide I/O operations.
532
533 For Python 3.4, this module is considered a :term:`provisional API`.
534
535 .. seealso::
536
537 :pep:`428` -- The pathlib module -- object-oriented filesystem paths
538 PEP written and implemented by Antoine Pitrou.
539
540
541 .. _whatsnew-selectors:
542
543 selectors
544 ---------
545
546 The new :mod:`selectors` module (created as part of implementing :pep:`3156`)
547 allows high-level and efficient I/O multiplexing, built upon the
548 :mod:`select` module primitives.
549
550
551 .. _whatsnew-statistics:
552
553 statistics
554 ----------
555
556 The new :mod:`statistics` module (defined in :pep:`450`) offers some core
557 statistics functionality directly in the standard library. This module
558 supports calculation of the mean, median, mode, variance and standard
559 deviation of a data series.
560
561 .. seealso::
562
563 :pep:`450` -- Adding A Statistics Module To The Standard Library
564 PEP written and implemented by Steven D'Aprano
565
566 .. _whatsnew-tracemalloc:
567
568
569 tracemalloc
570 -----------
571
572 The new :mod:`tracemalloc` module (defined in :pep:`454`) is a debug tool to
573 trace memory blocks allocated by Python. It provides the following information:
574
575 * Trace where an object was allocated
576 * Statistics on allocated memory blocks per filename and per line number:
577 total size, number and average size of allocated memory blocks
578 * Compute the differences between two snapshots to detect memory leaks
579
580 .. seealso::
581
582 :pep:`454` -- Add a new tracemalloc module to trace Python memory allocations
583 PEP written and implemented by Victor Stinner
584
585
586
587 Improved Modules
588 ================
589
590
591 abc
592 ---
593
594 New function :func:`abc.get_cache_token` can be used to know when to invalidate
595 caches that are affected by changes in the object graph. (Contributed
596 by Łukasz Langa in :issue:`16832`.)
597
598 New class :class:`~abc.ABC` has :class:`~abc.ABCMeta` as its meta class.
599 Using ``ABC`` as a base class has essentially the same effect as specifying
600 ``metaclass=abc.ABCMeta``, but is simpler to type and easier to read.
601 (Contributed by Bruno Dupuis in :issue:`16049`.)
602
603
604 aifc
605 ----
606
607 The :meth:`!getparams` method now returns a namedtuple rather than a
608 plain tuple. (Contributed by Claudiu Popa in :issue:`17818`.)
609
610 :func:`!aifc.open` now supports the context management protocol: when used in a
611 :keyword:`with` block, the :meth:`!close` method of the returned
612 object will be called automatically at the end of the block. (Contributed by
613 Serhiy Storchacha in :issue:`16486`.)
614
615 The :meth:`!writeframesraw` and :meth:`!writeframes`
616 methods now accept any :term:`bytes-like object`. (Contributed by Serhiy
617 Storchaka in :issue:`8311`.)
618
619
620 argparse
621 --------
622
623 The :class:`~argparse.FileType` class now accepts *encoding* and
624 *errors* arguments, which are passed through to :func:`open`. (Contributed
625 by Lucas Maystre in :issue:`11175`.)
626
627
628 audioop
629 -------
630
631 :mod:`!audioop` now supports 24-bit samples. (Contributed by Serhiy Storchaka
632 in :issue:`12866`.)
633
634 New :func:`!byteswap` function converts big-endian samples to
635 little-endian and vice versa. (Contributed by Serhiy Storchaka in
636 :issue:`19641`.)
637
638 All :mod:`!audioop` functions now accept any :term:`bytes-like object`. Strings
639 are not accepted: they didn't work before, now they raise an error right away.
640 (Contributed by Serhiy Storchaka in :issue:`16685`.)
641
642
643 base64
644 ------
645
646 The encoding and decoding functions in :mod:`base64` now accept any
647 :term:`bytes-like object` in cases where it previously required a
648 :class:`bytes` or :class:`bytearray` instance. (Contributed by Nick Coghlan in
649 :issue:`17839`.)
650
651 New functions :func:`~base64.a85encode`, :func:`~base64.a85decode`,
652 :func:`~base64.b85encode`, and :func:`~base64.b85decode` provide the ability to
653 encode and decode binary data from and to ``Ascii85`` and the git/mercurial
654 ``Base85`` formats, respectively. The ``a85`` functions have options that can
655 be used to make them compatible with the variants of the ``Ascii85`` encoding,
656 including the Adobe variant. (Contributed by Martin Morrison, the Mercurial
657 project, Serhiy Storchaka, and Antoine Pitrou in :issue:`17618`.)
658
659
660 collections
661 -----------
662
663 The :meth:`.ChainMap.new_child` method now accepts an *m* argument specifying
664 the child map to add to the chain. This allows an existing mapping and/or a
665 custom mapping type to be used for the child. (Contributed by Vinay Sajip in
666 :issue:`16613`.)
667
668
669 colorsys
670 --------
671
672 The number of digits in the coefficients for the RGB --- YIQ conversions have
673 been expanded so that they match the FCC NTSC versions. The change in
674 results should be less than 1% and may better match results found elsewhere.
675 (Contributed by Brian Landers and Serhiy Storchaka in :issue:`14323`.)
676
677
678 contextlib
679 ----------
680
681 The new :class:`contextlib.suppress` context manager helps to clarify the
682 intent of code that deliberately suppresses exceptions from a single
683 statement. (Contributed by Raymond Hettinger in :issue:`15806` and
684 Zero Piraeus in :issue:`19266`.)
685
686 The new :func:`contextlib.redirect_stdout` context manager makes it easier
687 for utility scripts to handle inflexible APIs that write their output to
688 :data:`sys.stdout` and don't provide any options to redirect it. Using the
689 context manager, the :data:`sys.stdout` output can be redirected to any
690 other stream or, in conjunction with :class:`io.StringIO`, to a string.
691 The latter can be especially useful, for example, to capture output
692 from a function that was written to implement a command line interface.
693 It is recommended only for utility scripts because it affects the
694 global state of :data:`sys.stdout`. (Contributed by Raymond Hettinger
695 in :issue:`15805`.)
696
697 The :mod:`contextlib` documentation has also been updated to include a
698 :ref:`discussion <single-use-reusable-and-reentrant-cms>` of the
699 differences between single use, reusable and reentrant context managers.
700
701
702 dbm
703 ---
704
705 :func:`dbm.open` objects now support the context management protocol. When
706 used in a :keyword:`with` statement, the ``close`` method of the database
707 object will be called automatically at the end of the block. (Contributed by
708 Claudiu Popa and Nick Coghlan in :issue:`19282`.)
709
710
711 dis
712 ---
713
714 Functions :func:`~dis.show_code`, :func:`~dis.dis`, :func:`~dis.distb`, and
715 :func:`~dis.disassemble` now accept a keyword-only *file* argument that
716 controls where they write their output.
717
718 The :mod:`dis` module is now built around an :class:`~dis.Instruction` class
719 that provides object oriented access to the details of each individual bytecode
720 operation.
721
722 A new method, :func:`~dis.get_instructions`, provides an iterator that emits
723 the Instruction stream for a given piece of Python code. Thus it is now
724 possible to write a program that inspects and manipulates a bytecode
725 object in ways different from those provided by the :mod:`~dis` module
726 itself. For example::
727
728 >>> import dis
729 >>> for instr in dis.get_instructions(lambda x: x + 1):
730 ... print(instr.opname)
731 LOAD_FAST
732 LOAD_CONST
733 BINARY_ADD
734 RETURN_VALUE
735
736 The various display tools in the :mod:`dis` module have been rewritten to use
737 these new components.
738
739 In addition, a new application-friendly class :class:`~dis.Bytecode` provides
740 an object-oriented API for inspecting bytecode in both in human-readable form
741 and for iterating over instructions. The :class:`~dis.Bytecode` constructor
742 takes the same arguments that :func:`~dis.get_instructions` does (plus an
743 optional *current_offset*), and the resulting object can be iterated to produce
744 :class:`~dis.Instruction` objects. But it also has a :mod:`~dis.Bytecode.dis`
745 method, equivalent to calling :mod:`~dis.dis` on the constructor argument, but
746 returned as a multi-line string::
747
748 >>> bytecode = dis.Bytecode(lambda x: x + 1, current_offset=3)
749 >>> for instr in bytecode:
750 ... print('{} ({})'.format(instr.opname, instr.opcode))
751 LOAD_FAST (124)
752 LOAD_CONST (100)
753 BINARY_ADD (23)
754 RETURN_VALUE (83)
755 >>> bytecode.dis().splitlines() # doctest: +NORMALIZE_WHITESPACE
756 [' 1 0 LOAD_FAST 0 (x)',
757 ' --> 3 LOAD_CONST 1 (1)',
758 ' 6 BINARY_ADD',
759 ' 7 RETURN_VALUE']
760
761 :class:`~dis.Bytecode` also has a class method,
762 :meth:`~dis.Bytecode.from_traceback`, that provides the ability to manipulate a
763 traceback (that is, ``print(Bytecode.from_traceback(tb).dis())`` is equivalent
764 to ``distb(tb)``).
765
766 (Contributed by Nick Coghlan, Ryan Kelly and Thomas Kluyver in :issue:`11816`
767 and Claudiu Popa in :issue:`17916`.)
768
769 New function :func:`~dis.stack_effect` computes the effect on the Python stack
770 of a given opcode and argument, information that is not otherwise available.
771 (Contributed by Larry Hastings in :issue:`19722`.)
772
773
774 doctest
775 -------
776
777 A new :ref:`option flag <doctest-options>`, :const:`~doctest.FAIL_FAST`, halts
778 test running as soon as the first failure is detected. (Contributed by R.
779 David Murray and Daniel Urban in :issue:`16522`.)
780
781 The :mod:`doctest` command line interface now uses :mod:`argparse`, and has two
782 new options, ``-o`` and ``-f``. ``-o`` allows :ref:`doctest options
783 <doctest-options>` to be specified on the command line, and ``-f`` is a
784 shorthand for ``-o FAIL_FAST`` (to parallel the similar option supported by the
785 :mod:`unittest` CLI). (Contributed by R. David Murray in :issue:`11390`.)
786
787 :mod:`doctest` will now find doctests in extension module ``__doc__`` strings.
788 (Contributed by Zachary Ware in :issue:`3158`.)
789
790
791 email
792 -----
793
794 :meth:`~email.message.Message.as_string` now accepts a *policy* argument to
795 override the default policy of the message when generating a string
796 representation of it. This means that ``as_string`` can now be used in more
797 circumstances, instead of having to create and use a :mod:`~email.generator` in
798 order to pass formatting parameters to its ``flatten`` method. (Contributed by
799 R. David Murray in :issue:`18600`.)
800
801 New method :meth:`~email.message.Message.as_bytes` added to produce a bytes
802 representation of the message in a fashion similar to how ``as_string``
803 produces a string representation. It does not accept the *maxheaderlen*
804 argument, but does accept the *unixfrom* and *policy* arguments. The
805 :class:`~email.message.Message` :meth:`~email.message.Message.__bytes__` method
806 calls it, meaning that ``bytes(mymsg)`` will now produce the intuitive
807 result: a bytes object containing the fully formatted message. (Contributed
808 by R. David Murray in :issue:`18600`.)
809
810 The :meth:`.Message.set_param` message now accepts a *replace* keyword argument.
811 When specified, the associated header will be updated without changing
812 its location in the list of headers. For backward compatibility, the default
813 is ``False``. (Contributed by R. David Murray in :issue:`18891`.)
814
815
816 .. _whatsnew_email_contentmanager:
817
818 A pair of new subclasses of :class:`~email.message.Message` have been added
819 (:class:`.EmailMessage` and :class:`.MIMEPart`), along with a new sub-module,
820 :mod:`~email.contentmanager` and a new :mod:`~email.policy` attribute
821 :attr:`~email.policy.EmailPolicy.content_manager`. All documentation is
822 currently in the new module, which is being added as part of email's new
823 :term:`provisional API`. These classes provide a number of new methods that
824 make extracting content from and inserting content into email messages much
825 easier. For details, see the :mod:`~email.contentmanager` documentation and
826 the :ref:`email-examples`. These API additions complete the
827 bulk of the work that was planned as part of the email6 project. The currently
828 provisional API is scheduled to become final in Python 3.5 (possibly with a few
829 minor additions in the area of error handling). (Contributed by R. David
830 Murray in :issue:`18891`.)
831
832
833 filecmp
834 -------
835
836 A new :func:`~filecmp.clear_cache` function provides the ability to clear the
837 :mod:`filecmp` comparison cache, which uses :func:`os.stat` information to
838 determine if the file has changed since the last compare. This can be used,
839 for example, if the file might have been changed and re-checked in less time
840 than the resolution of a particular filesystem's file modification time field.
841 (Contributed by Mark Levitt in :issue:`18149`.)
842
843 New module attribute :const:`~filecmp.DEFAULT_IGNORES` provides the list of
844 directories that are used as the default value for the *ignore* parameter of
845 the :func:`~filecmp.dircmp` function. (Contributed by Eli Bendersky in
846 :issue:`15442`.)
847
848
849 functools
850 ---------
851
852 The new :func:`~functools.partialmethod` descriptor brings partial argument
853 application to descriptors, just as :func:`~functools.partial` provides
854 for normal callables. The new descriptor also makes it easier to get
855 arbitrary callables (including :func:`~functools.partial` instances)
856 to behave like normal instance methods when included in a class definition.
857 (Contributed by Alon Horev and Nick Coghlan in :issue:`4331`.)
858
859 .. _whatsnew-singledispatch:
860
861 The new :func:`~functools.singledispatch` decorator brings support for
862 single-dispatch generic functions to the Python standard library. Where
863 object oriented programming focuses on grouping multiple operations on a
864 common set of data into a class, a generic function focuses on grouping
865 multiple implementations of an operation that allows it to work with
866 *different* kinds of data.
867
868 .. seealso::
869
870 :pep:`443` -- Single-dispatch generic functions
871 PEP written and implemented by Łukasz Langa.
872
873 :func:`~functools.total_ordering` now supports a return value of
874 :data:`NotImplemented` from the underlying comparison function. (Contributed
875 by Katie Miller in :issue:`10042`.)
876
877 A pure-python version of the :func:`~functools.partial` function is now in the
878 stdlib; in CPython it is overridden by the C accelerated version, but it is
879 available for other implementations to use. (Contributed by Brian Thorne in
880 :issue:`12428`.)
881
882
883 gc
884 --
885
886 New function :func:`~gc.get_stats` returns a list of three per-generation
887 dictionaries containing the collections statistics since interpreter startup.
888 (Contributed by Antoine Pitrou in :issue:`16351`.)
889
890
891 glob
892 ----
893
894 A new function :func:`~glob.escape` provides a way to escape special characters
895 in a filename so that they do not become part of the globbing expansion but are
896 instead matched literally. (Contributed by Serhiy Storchaka in :issue:`8402`.)
897
898
899 hashlib
900 -------
901
902 A new :func:`hashlib.pbkdf2_hmac` function provides
903 the `PKCS#5 password-based key derivation function 2
904 <https://en.wikipedia.org/wiki/PBKDF2>`_. (Contributed by Christian
905 Heimes in :issue:`18582`.)
906
907 The :attr:`~hashlib.hash.name` attribute of :mod:`hashlib` hash objects is now
908 a formally supported interface. It has always existed in CPython's
909 :mod:`hashlib` (although it did not return lower case names for all supported
910 hashes), but it was not a public interface and so some other Python
911 implementations have not previously supported it. (Contributed by Jason R.
912 Coombs in :issue:`18532`.)
913
914
915 hmac
916 ----
917
918 :mod:`hmac` now accepts ``bytearray`` as well as ``bytes`` for the *key*
919 argument to the :func:`~hmac.new` function, and the *msg* parameter to both the
920 :func:`~hmac.new` function and the :meth:`~hmac.HMAC.update` method now
921 accepts any type supported by the :mod:`hashlib` module. (Contributed
922 by Jonas Borgström in :issue:`18240`.)
923
924 The *digestmod* argument to the :func:`hmac.new` function may now be any hash
925 digest name recognized by :mod:`hashlib`. In addition, the current behavior in
926 which the value of *digestmod* defaults to ``MD5`` is deprecated: in a
927 future version of Python there will be no default value. (Contributed by
928 Christian Heimes in :issue:`17276`.)
929
930 With the addition of :attr:`~hmac.HMAC.block_size` and :attr:`~hmac.HMAC.name`
931 attributes (and the formal documentation of the :attr:`~hmac.HMAC.digest_size`
932 attribute), the :mod:`hmac` module now conforms fully to the :pep:`247` API.
933 (Contributed by Christian Heimes in :issue:`18775`.)
934
935
936 html
937 ----
938
939 New function :func:`~html.unescape` function converts HTML5 character references to
940 the corresponding Unicode characters. (Contributed by Ezio Melotti in
941 :issue:`2927`.)
942
943 :class:`~html.parser.HTMLParser` accepts a new keyword argument
944 *convert_charrefs* that, when ``True``, automatically converts all character
945 references. For backward-compatibility, its value defaults to ``False``, but
946 it will change to ``True`` in a future version of Python, so you are invited to
947 set it explicitly and update your code to use this new feature. (Contributed
948 by Ezio Melotti in :issue:`13633`.)
949
950 The *strict* argument of :class:`~html.parser.HTMLParser` is now deprecated.
951 (Contributed by Ezio Melotti in :issue:`15114`.)
952
953
954 http
955 ----
956
957 :meth:`~http.server.BaseHTTPRequestHandler.send_error` now accepts an
958 optional additional *explain* parameter which can be used to provide an
959 extended error description, overriding the hardcoded default if there is one.
960 This extended error description will be formatted using the
961 :attr:`~http.server.BaseHTTPRequestHandler.error_message_format` attribute
962 and sent as the body of the error response.
963 (Contributed by Karl Cow in :issue:`12921`.)
964
965 The :mod:`http.server` :ref:`command line interface <http-server-cli>` now has
966 a ``-b/--bind`` option that causes the server to listen on a specific address.
967 (Contributed by Malte Swart in :issue:`17764`.)
968
969
970 idlelib and IDLE
971 ----------------
972
973 Since idlelib implements the IDLE shell and editor and is not intended for
974 import by other programs, it gets improvements with every release. See
975 :file:`Lib/idlelib/NEWS.txt` for a cumulative list of changes since 3.3.0,
976 as well as changes made in future 3.4.x releases. This file is also available
977 from the IDLE :menuselection:`Help --> About IDLE` dialog.
978
979
980 importlib
981 ---------
982
983 The :class:`~importlib.abc.InspectLoader` ABC defines a new method,
984 :meth:`~importlib.abc.InspectLoader.source_to_code` that accepts source
985 data and a path and returns a code object. The default implementation
986 is equivalent to ``compile(data, path, 'exec', dont_inherit=True)``.
987 (Contributed by Eric Snow and Brett Cannon in :issue:`15627`.)
988
989 :class:`~importlib.abc.InspectLoader` also now has a default implementation
990 for the :meth:`~importlib.abc.InspectLoader.get_code` method. However,
991 it will normally be desirable to override the default implementation
992 for performance reasons. (Contributed by Brett Cannon in :issue:`18072`.)
993
994 The :func:`~importlib.reload` function has been moved from :mod:`!imp` to
995 :mod:`importlib` as part of the :mod:`!imp` module deprecation. (Contributed by
996 Berker Peksag in :issue:`18193`.)
997
998 :mod:`importlib.util` now has a :const:`~importlib.util.MAGIC_NUMBER` attribute
999 providing access to the bytecode version number. This replaces the
1000 :func:`!get_magic` function in the deprecated :mod:`!imp` module.
1001 (Contributed by Brett Cannon in :issue:`18192`.)
1002
1003 New :mod:`importlib.util` functions :func:`~importlib.util.cache_from_source`
1004 and :func:`~importlib.util.source_from_cache` replace the same-named functions
1005 in the deprecated :mod:`!imp` module. (Contributed by Brett Cannon in
1006 :issue:`18194`.)
1007
1008 The :mod:`importlib` bootstrap :class:`.NamespaceLoader` now conforms to
1009 the :class:`.InspectLoader` ABC, which means that ``runpy`` and
1010 ``python -m`` can now be used with namespace packages. (Contributed
1011 by Brett Cannon in :issue:`18058`.)
1012
1013 :mod:`importlib.util` has a new function :func:`~importlib.util.decode_source`
1014 that decodes source from bytes using universal newline processing. This is
1015 useful for implementing :meth:`.InspectLoader.get_source` methods.
1016
1017 :class:`importlib.machinery.ExtensionFileLoader` now has a
1018 :meth:`~importlib.machinery.ExtensionFileLoader.get_filename` method. This was
1019 inadvertently omitted in the original implementation. (Contributed by Eric
1020 Snow in :issue:`19152`.)
1021
1022
1023 inspect
1024 -------
1025
1026 The :mod:`inspect` module now offers a basic :ref:`command line interface
1027 <inspect-module-cli>` to quickly display source code and other
1028 information for modules, classes and functions. (Contributed by Claudiu Popa
1029 and Nick Coghlan in :issue:`18626`.)
1030
1031 :func:`~inspect.unwrap` makes it easy to unravel wrapper function chains
1032 created by :func:`functools.wraps` (and any other API that sets the
1033 ``__wrapped__`` attribute on a wrapper function). (Contributed by
1034 Daniel Urban, Aaron Iles and Nick Coghlan in :issue:`13266`.)
1035
1036 As part of the implementation of the new :mod:`enum` module, the
1037 :mod:`inspect` module now has substantially better support for custom
1038 ``__dir__`` methods and dynamic class attributes provided through
1039 metaclasses. (Contributed by Ethan Furman in :issue:`18929` and
1040 :issue:`19030`.)
1041
1042 :func:`~inspect.getfullargspec` and :func:`!getargspec`
1043 now use the :func:`~inspect.signature` API. This allows them to
1044 support a much broader range of callables, including those with
1045 ``__signature__`` attributes, those with metadata provided by argument
1046 clinic, :func:`functools.partial` objects and more. Note that, unlike
1047 :func:`~inspect.signature`, these functions still ignore ``__wrapped__``
1048 attributes, and report the already bound first argument for bound methods,
1049 so it is still necessary to update your code to use
1050 :func:`~inspect.signature` directly if those features are desired.
1051 (Contributed by Yury Selivanov in :issue:`17481`.)
1052
1053 :func:`~inspect.signature` now supports duck types of CPython functions,
1054 which adds support for functions compiled with Cython. (Contributed
1055 by Stefan Behnel and Yury Selivanov in :issue:`17159`.)
1056
1057
1058 ipaddress
1059 ---------
1060
1061 :mod:`ipaddress` was added to the standard library in Python 3.3 as a
1062 :term:`provisional API`. With the release of Python 3.4, this qualification
1063 has been removed: :mod:`ipaddress` is now considered a stable API, covered
1064 by the normal standard library requirements to maintain backwards
1065 compatibility.
1066
1067 A new :attr:`~ipaddress.IPv4Address.is_global` property is ``True`` if
1068 an address is globally routeable. (Contributed by Peter Moody in
1069 :issue:`17400`.)
1070
1071
1072 logging
1073 -------
1074
1075 The :class:`~logging.handlers.TimedRotatingFileHandler` has a new *atTime*
1076 parameter that can be used to specify the time of day when rollover should
1077 happen. (Contributed by Ronald Oussoren in :issue:`9556`.)
1078
1079 :class:`~logging.handlers.SocketHandler` and
1080 :class:`~logging.handlers.DatagramHandler` now support Unix domain sockets (by
1081 setting *port* to ``None``). (Contributed by Vinay Sajip in commit
1082 ce46195b56a9.)
1083
1084 :func:`~logging.config.fileConfig` now accepts a
1085 :class:`configparser.RawConfigParser` subclass instance for the *fname*
1086 parameter. This facilitates using a configuration file when logging
1087 configuration is just a part of the overall application configuration, or where
1088 the application modifies the configuration before passing it to
1089 :func:`~logging.config.fileConfig`. (Contributed by Vinay Sajip in
1090 :issue:`16110`.)
1091
1092 Logging configuration data received from a socket via the
1093 :func:`logging.config.listen` function can now be validated before being
1094 processed by supplying a verification function as the argument to the new
1095 *verify* keyword argument. (Contributed by Vinay Sajip in :issue:`15452`.)
1096
1097
1098 .. _whatsnew-marshal-3:
1099
1100 marshal
1101 -------
1102
1103 The default :mod:`marshal` version has been bumped to 3. The code implementing
1104 the new version restores the Python2 behavior of recording only one copy of
1105 interned strings and preserving the interning on deserialization, and extends
1106 this "one copy" ability to any object type (including handling recursive
1107 references). This reduces both the size of ``.pyc`` files and the amount of
1108 memory a module occupies in memory when it is loaded from a ``.pyc`` (or
1109 ``.pyo``) file. (Contributed by Kristján Valur Jónsson in :issue:`16475`,
1110 with additional speedups by Antoine Pitrou in :issue:`19219`.)
1111
1112
1113 mmap
1114 ----
1115
1116 mmap objects are now :ref:`weakly referenceable <mod-weakref>`.
1117 (Contributed by Valerie Lambert in :issue:`4885`.)
1118
1119
1120 multiprocessing
1121 ---------------
1122
1123 .. _whatsnew-multiprocessing-no-fork:
1124
1125 On Unix two new :ref:`start methods <multiprocessing-start-methods>`,
1126 ``spawn`` and ``forkserver``, have been added for starting processes using
1127 :mod:`multiprocessing`. These make the mixing of processes with threads more
1128 robust, and the ``spawn`` method matches the semantics that multiprocessing has
1129 always used on Windows. New function
1130 :func:`~multiprocessing.get_all_start_methods` reports all start methods
1131 available on the platform, :func:`~multiprocessing.get_start_method` reports
1132 the current start method, and :func:`~multiprocessing.set_start_method` sets
1133 the start method. (Contributed by Richard Oudkerk in :issue:`8713`.)
1134
1135 :mod:`multiprocessing` also now has the concept of a ``context``, which
1136 determines how child processes are created. New function
1137 :func:`~multiprocessing.get_context` returns a context that uses a specified
1138 start method. It has the same API as the :mod:`multiprocessing` module itself,
1139 so you can use it to create :class:`~multiprocessing.pool.Pool`\ s and other
1140 objects that will operate within that context. This allows a framework and an
1141 application or different parts of the same application to use multiprocessing
1142 without interfering with each other. (Contributed by Richard Oudkerk in
1143 :issue:`18999`.)
1144
1145 Except when using the old *fork* start method, child processes no longer
1146 inherit unneeded handles/file descriptors from their parents (part of
1147 :issue:`8713`).
1148
1149 :mod:`multiprocessing` now relies on :mod:`runpy` (which implements the
1150 ``-m`` switch) to initialise ``__main__`` appropriately in child processes
1151 when using the ``spawn`` or ``forkserver`` start methods. This resolves some
1152 edge cases where combining multiprocessing, the ``-m`` command line switch,
1153 and explicit relative imports could cause obscure failures in child
1154 processes. (Contributed by Nick Coghlan in :issue:`19946`.)
1155
1156
1157 operator
1158 --------
1159
1160 New function :func:`~operator.length_hint` provides an implementation of the
1161 specification for how the :meth:`~object.__length_hint__` special method should
1162 be used, as part of the :pep:`424` formal specification of this language
1163 feature. (Contributed by Armin Ronacher in :issue:`16148`.)
1164
1165 There is now a pure-python version of the :mod:`operator` module available for
1166 reference and for use by alternate implementations of Python. (Contributed by
1167 Zachary Ware in :issue:`16694`.)
1168
1169
1170 os
1171 --
1172
1173 There are new functions to get and set the :ref:`inheritable flag
1174 <fd_inheritance>` of a file descriptor (:func:`os.get_inheritable`,
1175 :func:`os.set_inheritable`) or a Windows handle
1176 (:func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable`).
1177
1178 New function :func:`~os.cpu_count` reports the number of CPUs available on the
1179 platform on which Python is running (or ``None`` if the count can't be
1180 determined). The :func:`multiprocessing.cpu_count` function is now implemented
1181 in terms of this function). (Contributed by Trent Nelson, Yogesh Chaudhari,
1182 Victor Stinner, and Charles-François Natali in :issue:`17914`.)
1183
1184 :func:`os.path.samestat` is now available on the Windows platform (and the
1185 :func:`os.path.samefile` implementation is now shared between Unix and
1186 Windows). (Contributed by Brian Curtin in :issue:`11939`.)
1187
1188 :func:`os.path.ismount` now recognizes volumes mounted below a drive
1189 root on Windows. (Contributed by Tim Golden in :issue:`9035`.)
1190
1191 :func:`os.open` supports two new flags on platforms that provide them,
1192 :const:`~os.O_PATH` (un-opened file descriptor), and :const:`~os.O_TMPFILE`
1193 (unnamed temporary file; as of 3.4.0 release available only on Linux systems
1194 with a kernel version of 3.11 or newer that have uapi headers). (Contributed
1195 by Christian Heimes in :issue:`18673` and Benjamin Peterson, respectively.)
1196
1197
1198 pdb
1199 ---
1200
1201 :mod:`pdb` has been enhanced to handle generators, :keyword:`yield`, and
1202 ``yield from`` in a more useful fashion. This is especially helpful when
1203 debugging :mod:`asyncio` based programs. (Contributed by Andrew Svetlov and
1204 Xavier de Gaye in :issue:`16596`.)
1205
1206 The ``print`` command has been removed from :mod:`pdb`, restoring access to the
1207 Python :func:`print` function from the pdb command line. Python2's ``pdb`` did
1208 not have a ``print`` command; instead, entering ``print`` executed the
1209 ``print`` statement. In Python3 ``print`` was mistakenly made an alias for the
1210 pdb :pdbcmd:`p` command. ``p``, however, prints the ``repr`` of its argument,
1211 not the ``str`` like the Python2 ``print`` command did. Worse, the Python3
1212 ``pdb print`` command shadowed the Python3 ``print`` function, making it
1213 inaccessible at the ``pdb`` prompt. (Contributed by Connor Osborn in
1214 :issue:`18764`.)
1215
1216
1217 .. _whatsnew-protocol-4:
1218
1219 pickle
1220 ------
1221
1222 :mod:`pickle` now supports (but does not use by default) a new pickle protocol,
1223 protocol 4. This new protocol addresses a number of issues that were present
1224 in previous protocols, such as the serialization of nested classes, very large
1225 strings and containers, and classes whose :meth:`~object.__new__` method takes
1226 keyword-only arguments. It also provides some efficiency improvements.
1227
1228 .. seealso::
1229
1230 :pep:`3154` -- Pickle protocol 4
1231 PEP written by Antoine Pitrou and implemented by Alexandre Vassalotti.
1232
1233
1234 plistlib
1235 --------
1236
1237 :mod:`plistlib` now has an API that is similar to the standard pattern for
1238 stdlib serialization protocols, with new :func:`~plistlib.load`,
1239 :func:`~plistlib.dump`, :func:`~plistlib.loads`, and :func:`~plistlib.dumps`
1240 functions. (The older API is now deprecated.) In addition to the already
1241 supported XML plist format (:const:`~plistlib.FMT_XML`), it also now supports
1242 the binary plist format (:const:`~plistlib.FMT_BINARY`). (Contributed by Ronald
1243 Oussoren and others in :issue:`14455`.)
1244
1245
1246 poplib
1247 ------
1248
1249 Two new methods have been added to :mod:`poplib`: :meth:`~poplib.POP3.capa`,
1250 which returns the list of capabilities advertised by the POP server, and
1251 :meth:`~poplib.POP3.stls`, which switches a clear-text POP3 session into an
1252 encrypted POP3 session if the POP server supports it. (Contributed by Lorenzo
1253 Catucci in :issue:`4473`.)
1254
1255
1256 pprint
1257 ------
1258
1259 The :mod:`pprint` module's :class:`~pprint.PrettyPrinter` class and its
1260 :func:`~pprint.pformat`, and :func:`~pprint.pprint` functions have a new
1261 option, *compact*, that controls how the output is formatted. Currently
1262 setting *compact* to ``True`` means that sequences will be printed with as many
1263 sequence elements as will fit within *width* on each (indented) line.
1264 (Contributed by Serhiy Storchaka in :issue:`19132`.)
1265
1266 Long strings are now wrapped using Python's normal line continuation
1267 syntax. (Contributed by Antoine Pitrou in :issue:`17150`.)
1268
1269
1270 pty
1271 ---
1272
1273 :func:`pty.spawn` now returns the status value from :func:`os.waitpid` on
1274 the child process, instead of ``None``. (Contributed by Gregory P. Smith.)
1275
1276
1277 pydoc
1278 -----
1279
1280 The :mod:`pydoc` module is now based directly on the :func:`inspect.signature`
1281 introspection API, allowing it to provide signature information for a wider
1282 variety of callable objects. This change also means that ``__wrapped__``
1283 attributes are now taken into account when displaying help information.
1284 (Contributed by Larry Hastings in :issue:`19674`.)
1285
1286 The :mod:`pydoc` module no longer displays the ``self`` parameter for
1287 already bound methods. Instead, it aims to always display the exact current
1288 signature of the supplied callable. (Contributed by Larry Hastings in
1289 :issue:`20710`.)
1290
1291 In addition to the changes that have been made to :mod:`pydoc` directly,
1292 its handling of custom ``__dir__`` methods and various descriptor
1293 behaviours has also been improved substantially by the underlying changes in
1294 the :mod:`inspect` module.
1295
1296 As the :func:`help` builtin is based on :mod:`pydoc`, the above changes also
1297 affect the behaviour of :func:`help`.
1298
1299
1300 re
1301 --
1302
1303 New :func:`~re.fullmatch` function and :meth:`.Pattern.fullmatch` method anchor
1304 the pattern at both ends of the string to match. This provides a way to be
1305 explicit about the goal of the match, which avoids a class of subtle bugs where
1306 ``$`` characters get lost during code changes or the addition of alternatives
1307 to an existing regular expression. (Contributed by Matthew Barnett in
1308 :issue:`16203`.)
1309
1310 The repr of :ref:`regex objects <re-objects>` now includes the pattern
1311 and the flags; the repr of :ref:`match objects <match-objects>` now
1312 includes the start, end, and the part of the string that matched. (Contributed
1313 by Hugo Lopes Tavares and Serhiy Storchaka in :issue:`13592` and
1314 :issue:`17087`.)
1315
1316
1317 resource
1318 --------
1319
1320 New :func:`~resource.prlimit` function, available on Linux platforms with a
1321 kernel version of 2.6.36 or later and glibc of 2.13 or later, provides the
1322 ability to query or set the resource limits for processes other than the one
1323 making the call. (Contributed by Christian Heimes in :issue:`16595`.)
1324
1325 On Linux kernel version 2.6.36 or later, there are also some new
1326 Linux specific constants: :const:`~resource.RLIMIT_MSGQUEUE`,
1327 :const:`~resource.RLIMIT_NICE`, :const:`~resource.RLIMIT_RTPRIO`,
1328 :const:`~resource.RLIMIT_RTTIME`, and :const:`~resource.RLIMIT_SIGPENDING`.
1329 (Contributed by Christian Heimes in :issue:`19324`.)
1330
1331 On FreeBSD version 9 and later, there some new FreeBSD specific constants:
1332 :const:`~resource.RLIMIT_SBSIZE`, :const:`~resource.RLIMIT_SWAP`, and
1333 :const:`~resource.RLIMIT_NPTS`. (Contributed by Claudiu Popa in
1334 :issue:`19343`.)
1335
1336
1337 select
1338 ------
1339
1340 :class:`~select.epoll` objects now support the context management protocol.
1341 When used in a :keyword:`with` statement, the :meth:`~select.epoll.close`
1342 method will be called automatically at the end of the block. (Contributed
1343 by Serhiy Storchaka in :issue:`16488`.)
1344
1345 :class:`~select.devpoll` objects now have :meth:`~select.devpoll.fileno` and
1346 :meth:`~select.devpoll.close` methods, as well as a new attribute
1347 :attr:`~select.devpoll.closed`. (Contributed by Victor Stinner in
1348 :issue:`18794`.)
1349
1350
1351 shelve
1352 ------
1353
1354 :class:`~shelve.Shelf` instances may now be used in :keyword:`with` statements,
1355 and will be automatically closed at the end of the :keyword:`!with` block.
1356 (Contributed by Filip Gruszczyński in :issue:`13896`.)
1357
1358
1359 shutil
1360 ------
1361
1362 :func:`~shutil.copyfile` now raises a specific :exc:`~shutil.Error` subclass,
1363 :exc:`~shutil.SameFileError`, when the source and destination are the same
1364 file, which allows an application to take appropriate action on this specific
1365 error. (Contributed by Atsuo Ishimoto and Hynek Schlawack in
1366 :issue:`1492704`.)
1367
1368
1369 smtpd
1370 -----
1371
1372 The :class:`!SMTPServer` and :class:`!SMTPChannel` classes now
1373 accept a *map* keyword argument which, if specified, is passed in to
1374 :class:`!asynchat.async_chat` as its *map* argument. This allows an application
1375 to avoid affecting the global socket map. (Contributed by Vinay Sajip in
1376 :issue:`11959`.)
1377
1378
1379 smtplib
1380 -------
1381
1382 :exc:`~smtplib.SMTPException` is now a subclass of :exc:`OSError`, which allows
1383 both socket level errors and SMTP protocol level errors to be caught in one
1384 try/except statement by code that only cares whether or not an error occurred.
1385 (Contributed by Ned Jackson Lovely in :issue:`2118`.)
1386
1387
1388 socket
1389 ------
1390
1391 The socket module now supports the :const:`~socket.CAN_BCM` protocol on
1392 platforms that support it. (Contributed by Brian Thorne in :issue:`15359`.)
1393
1394 Socket objects have new methods to get or set their :ref:`inheritable flag
1395 <fd_inheritance>`, :meth:`~socket.socket.get_inheritable` and
1396 :meth:`~socket.socket.set_inheritable`.
1397
1398 The ``socket.AF_*`` and ``socket.SOCK_*`` constants are now enumeration values
1399 using the new :mod:`enum` module. This allows meaningful names to be printed
1400 during debugging, instead of integer "magic numbers".
1401
1402 The :const:`~socket.AF_LINK` constant is now available on BSD and OSX.
1403
1404 :func:`~socket.inet_pton` and :func:`~socket.inet_ntop` are now supported
1405 on Windows. (Contributed by Atsuo Ishimoto in :issue:`7171`.)
1406
1407
1408 sqlite3
1409 -------
1410
1411 A new boolean parameter to the :func:`~sqlite3.connect` function, *uri*, can be
1412 used to indicate that the *database* parameter is a ``uri`` (see the `SQLite
1413 URI documentation <https://www.sqlite.org/uri.html>`_). (Contributed by poq in
1414 :issue:`13773`.)
1415
1416
1417 ssl
1418 ---
1419
1420 .. _whatsnew-tls-11-12:
1421
1422 :data:`~ssl.PROTOCOL_TLSv1_1` and :data:`~ssl.PROTOCOL_TLSv1_2` (TLSv1.1 and
1423 TLSv1.2 support) have been added; support for these protocols is only available if
1424 Python is linked with OpenSSL 1.0.1 or later. (Contributed by Michele Orrù and
1425 Antoine Pitrou in :issue:`16692`.)
1426
1427 .. _whatsnew34-sslcontext:
1428
1429 New function :func:`~ssl.create_default_context` provides a standard way to
1430 obtain an :class:`~ssl.SSLContext` whose settings are intended to be a
1431 reasonable balance between compatibility and security. These settings are
1432 more stringent than the defaults provided by the :class:`~ssl.SSLContext`
1433 constructor, and may be adjusted in the future, without prior deprecation, if
1434 best-practice security requirements change. The new recommended best
1435 practice for using stdlib libraries that support SSL is to use
1436 :func:`~ssl.create_default_context` to obtain an :class:`~ssl.SSLContext`
1437 object, modify it if needed, and then pass it as the *context* argument
1438 of the appropriate stdlib API. (Contributed by Christian Heimes
1439 in :issue:`19689`.)
1440
1441 :class:`~ssl.SSLContext` method :meth:`~ssl.SSLContext.load_verify_locations`
1442 accepts a new optional argument *cadata*, which can be used to provide PEM or
1443 DER encoded certificates directly via strings or bytes, respectively.
1444 (Contributed by Christian Heimes in :issue:`18138`.)
1445
1446 New function :func:`~ssl.get_default_verify_paths` returns
1447 a named tuple of the paths and environment variables that the
1448 :meth:`~ssl.SSLContext.set_default_verify_paths` method uses to set
1449 OpenSSL's default ``cafile`` and ``capath``. This can be an aid in
1450 debugging default verification issues. (Contributed by Christian Heimes
1451 in :issue:`18143`.)
1452
1453 :class:`~ssl.SSLContext` has a new method,
1454 :meth:`~ssl.SSLContext.cert_store_stats`, that reports the number of loaded
1455 ``X.509`` certs, ``X.509 CA`` certs, and certificate revocation lists
1456 (``crl``\ s), as well as a :meth:`~ssl.SSLContext.get_ca_certs` method that
1457 returns a list of the loaded ``CA`` certificates. (Contributed by Christian
1458 Heimes in :issue:`18147`.)
1459
1460 If OpenSSL 0.9.8 or later is available, :class:`~ssl.SSLContext` has a new
1461 attribute :attr:`~ssl.SSLContext.verify_flags` that can be used to control the
1462 certificate verification process by setting it to some combination of the new
1463 constants :const:`~ssl.VERIFY_DEFAULT`, :const:`~ssl.VERIFY_CRL_CHECK_LEAF`,
1464 :const:`~ssl.VERIFY_CRL_CHECK_CHAIN`, or :const:`~ssl.VERIFY_X509_STRICT`.
1465 OpenSSL does not do any CRL verification by default. (Contributed by
1466 Christien Heimes in :issue:`8813`.)
1467
1468 New :class:`~ssl.SSLContext` method :meth:`~ssl.SSLContext.load_default_certs`
1469 loads a set of default "certificate authority" (CA) certificates from default
1470 locations, which vary according to the platform. It can be used to load both
1471 TLS web server authentication certificates
1472 (``purpose=``:data:`~ssl.Purpose.SERVER_AUTH`) for a client to use to verify a
1473 server, and certificates for a server to use in verifying client certificates
1474 (``purpose=``:data:`~ssl.Purpose.CLIENT_AUTH`). (Contributed by Christian
1475 Heimes in :issue:`19292`.)
1476
1477 .. _whatsnew34-win-cert-store:
1478
1479 Two new windows-only functions, :func:`~ssl.enum_certificates` and
1480 :func:`~ssl.enum_crls` provide the ability to retrieve certificates,
1481 certificate information, and CRLs from the Windows cert store. (Contributed
1482 by Christian Heimes in :issue:`17134`.)
1483
1484 .. _whatsnew34-sni:
1485
1486 Support for server-side SNI (Server Name Indication) using the new
1487 :meth:`ssl.SSLContext.set_servername_callback` method.
1488 (Contributed by Daniel Black in :issue:`8109`.)
1489
1490 The dictionary returned by :meth:`.SSLSocket.getpeercert` contains additional
1491 ``X509v3`` extension items: ``crlDistributionPoints``, ``calIssuers``, and
1492 ``OCSP`` URIs. (Contributed by Christian Heimes in :issue:`18379`.)
1493
1494
1495 stat
1496 ----
1497
1498 The :mod:`stat` module is now backed by a C implementation in :mod:`!_stat`. A C
1499 implementation is required as most of the values aren't standardized and
1500 are platform-dependent. (Contributed by Christian Heimes in :issue:`11016`.)
1501
1502 The module supports new :mod:`~stat.ST_MODE` flags, :mod:`~stat.S_IFDOOR`,
1503 :const:`~stat.S_IFPORT`, and :const:`~stat.S_IFWHT`. (Contributed by
1504 Christian Hiemes in :issue:`11016`.)
1505
1506
1507 struct
1508 ------
1509
1510 New function :mod:`~struct.iter_unpack` and a new
1511 :meth:`struct.Struct.iter_unpack` method on compiled formats provide streamed
1512 unpacking of a buffer containing repeated instances of a given format of data.
1513 (Contributed by Antoine Pitrou in :issue:`17804`.)
1514
1515
1516 subprocess
1517 ----------
1518
1519 :func:`~subprocess.check_output` now accepts an *input* argument that can
1520 be used to provide the contents of ``stdin`` for the command that is run.
1521 (Contributed by Zack Weinberg in :issue:`16624`.)
1522
1523 :func:`~subprocess.getoutput` and :func:`~subprocess.getstatusoutput` now
1524 work on Windows. This change was actually inadvertently made in 3.3.4.
1525 (Contributed by Tim Golden in :issue:`10197`.)
1526
1527
1528 sunau
1529 -----
1530
1531 The :meth:`!getparams` method now returns a namedtuple rather than a
1532 plain tuple. (Contributed by Claudiu Popa in :issue:`18901`.)
1533
1534 :meth:`!sunau.open` now supports the context management protocol: when used in a
1535 :keyword:`with` block, the ``close`` method of the returned object will be
1536 called automatically at the end of the block. (Contributed by Serhiy Storchaka
1537 in :issue:`18878`.)
1538
1539 :meth:`!AU_write.setsampwidth` now supports 24 bit samples, thus adding
1540 support for writing 24 sample using the module. (Contributed by
1541 Serhiy Storchaka in :issue:`19261`.)
1542
1543 The :meth:`!writeframesraw` and
1544 :meth:`!writeframes` methods now accept any :term:`bytes-like
1545 object`. (Contributed by Serhiy Storchaka in :issue:`8311`.)
1546
1547
1548 sys
1549 ---
1550
1551 New function :func:`sys.getallocatedblocks` returns the current number of
1552 blocks allocated by the interpreter. (In CPython with the default
1553 ``--with-pymalloc`` setting, this is allocations made through the
1554 :c:func:`PyObject_Malloc` API.) This can be useful for tracking memory leaks,
1555 especially if automated via a test suite. (Contributed by Antoine Pitrou
1556 in :issue:`13390`.)
1557
1558 When the Python interpreter starts in :ref:`interactive mode
1559 <tut-interactive>`, it checks for an :data:`~sys.__interactivehook__` attribute
1560 on the :mod:`sys` module. If the attribute exists, its value is called with no
1561 arguments just before interactive mode is started. The check is made after the
1562 :envvar:`PYTHONSTARTUP` file is read, so it can be set there. The :mod:`site`
1563 module :ref:`sets it <rlcompleter-config>` to a function that enables tab
1564 completion and history saving (in :file:`~/.python-history`) if the platform
1565 supports :mod:`readline`. If you do not want this (new) behavior, you can
1566 override it in :envvar:`PYTHONSTARTUP`, :mod:`sitecustomize`, or
1567 :mod:`usercustomize` by deleting this attribute from :mod:`sys` (or setting it
1568 to some other callable). (Contributed by Éric Araujo and Antoine Pitrou in
1569 :issue:`5845`.)
1570
1571
1572 tarfile
1573 -------
1574
1575 The :mod:`tarfile` module now supports a simple :ref:`tarfile-commandline` when
1576 called as a script directly or via ``-m``. This can be used to create and
1577 extract tarfile archives. (Contributed by Berker Peksag in :issue:`13477`.)
1578
1579
1580 textwrap
1581 --------
1582
1583 The :class:`~textwrap.TextWrapper` class has two new attributes/constructor
1584 arguments: :attr:`~textwrap.TextWrapper.max_lines`, which limits the number of
1585 lines in the output, and :attr:`~textwrap.TextWrapper.placeholder`, which is a
1586 string that will appear at the end of the output if it has been truncated
1587 because of *max_lines*. Building on these capabilities, a new convenience
1588 function :func:`~textwrap.shorten` collapses all of the whitespace in the input
1589 to single spaces and produces a single line of a given *width* that ends with
1590 the *placeholder* (by default, ``[...]``). (Contributed by Antoine Pitrou and
1591 Serhiy Storchaka in :issue:`18585` and :issue:`18725`.)
1592
1593
1594 threading
1595 ---------
1596
1597 The :class:`~threading.Thread` object representing the main thread can be
1598 obtained from the new :func:`~threading.main_thread` function. In normal
1599 conditions this will be the thread from which the Python interpreter was
1600 started. (Contributed by Andrew Svetlov in :issue:`18882`.)
1601
1602
1603 traceback
1604 ---------
1605
1606 A new :func:`traceback.clear_frames` function takes a traceback object
1607 and clears the local variables in all of the frames it references,
1608 reducing the amount of memory consumed. (Contributed by Andrew Kuchling in
1609 :issue:`1565525`.)
1610
1611
1612 types
1613 -----
1614
1615 A new :func:`~types.DynamicClassAttribute` descriptor provides a way to define
1616 an attribute that acts normally when looked up through an instance object, but
1617 which is routed to the *class* ``__getattr__`` when looked up through the
1618 class. This allows one to have properties active on a class, and have virtual
1619 attributes on the class with the same name (see :mod:`enum` for an example).
1620 (Contributed by Ethan Furman in :issue:`19030`.)
1621
1622
1623 urllib
1624 ------
1625
1626 :mod:`urllib.request` now supports ``data:`` URLs via the
1627 :class:`~urllib.request.DataHandler` class. (Contributed by Mathias Panzenböck
1628 in :issue:`16423`.)
1629
1630 The http method that will be used by a :class:`~urllib.request.Request` class
1631 can now be specified by setting a :class:`~urllib.request.Request.method`
1632 class attribute on the subclass. (Contributed by Jason R Coombs in
1633 :issue:`18978`.)
1634
1635 :class:`~urllib.request.Request` objects are now reusable: if the
1636 :attr:`~urllib.request.Request.full_url` or :attr:`~urllib.request.Request.data`
1637 attributes are modified, all relevant internal properties are updated. This
1638 means, for example, that it is now possible to use the same
1639 :class:`~urllib.request.Request` object in more than one
1640 :meth:`.OpenerDirector.open` call with different *data* arguments, or to
1641 modify a :class:`~urllib.request.Request`\ 's ``url`` rather than recomputing it
1642 from scratch. There is also a new
1643 :meth:`~urllib.request.Request.remove_header` method that can be used to remove
1644 headers from a :class:`~urllib.request.Request`. (Contributed by Alexey
1645 Kachayev in :issue:`16464`, Daniel Wozniak in :issue:`17485`, and Damien Brecht
1646 and Senthil Kumaran in :issue:`17272`.)
1647
1648 :class:`~urllib.error.HTTPError` objects now have a
1649 :attr:`~urllib.error.HTTPError.headers` attribute that provides access to the
1650 HTTP response headers associated with the error. (Contributed by
1651 Berker Peksag in :issue:`15701`.)
1652
1653
1654 unittest
1655 --------
1656
1657 The :class:`~unittest.TestCase` class has a new method,
1658 :meth:`~unittest.TestCase.subTest`, that produces a context manager whose
1659 :keyword:`with` block becomes a "sub-test". This context manager allows a test
1660 method to dynamically generate subtests by, say, calling the ``subTest``
1661 context manager inside a loop. A single test method can thereby produce an
1662 indefinite number of separately identified and separately counted tests, all of
1663 which will run even if one or more of them fail. For example::
1664
1665 class NumbersTest(unittest.TestCase):
1666 def test_even(self):
1667 for i in range(6):
1668 with self.subTest(i=i):
1669 self.assertEqual(i % 2, 0)
1670
1671 will result in six subtests, each identified in the unittest verbose output
1672 with a label consisting of the variable name ``i`` and a particular value for
1673 that variable (``i=0``, ``i=1``, etc). See :ref:`subtests` for the full
1674 version of this example. (Contributed by Antoine Pitrou in :issue:`16997`.)
1675
1676 :func:`unittest.main` now accepts an iterable of test names for
1677 *defaultTest*, where previously it only accepted a single test name as a
1678 string. (Contributed by Jyrki Pulliainen in :issue:`15132`.)
1679
1680 If :class:`~unittest.SkipTest` is raised during test discovery (that is, at the
1681 module level in the test file), it is now reported as a skip instead of an
1682 error. (Contributed by Zach Ware in :issue:`16935`.)
1683
1684 :meth:`~unittest.TestLoader.discover` now sorts the discovered files to provide
1685 consistent test ordering. (Contributed by Martin Melin and Jeff Ramnani in
1686 :issue:`16709`.)
1687
1688 :class:`~unittest.TestSuite` now drops references to tests as soon as the test
1689 has been run, if the test is successful. On Python interpreters that do
1690 garbage collection, this allows the tests to be garbage collected if nothing
1691 else is holding a reference to the test. It is possible to override this
1692 behavior by creating a :class:`~unittest.TestSuite` subclass that defines a
1693 custom ``_removeTestAtIndex`` method. (Contributed by Tom Wardill, Matt
1694 McClure, and Andrew Svetlov in :issue:`11798`.)
1695
1696 A new test assertion context-manager, :meth:`~unittest.TestCase.assertLogs`,
1697 will ensure that a given block of code emits a log message using the
1698 :mod:`logging` module. By default the message can come from any logger and
1699 have a priority of ``INFO`` or higher, but both the logger name and an
1700 alternative minimum logging level may be specified. The object returned by the
1701 context manager can be queried for the :class:`~logging.LogRecord`\ s and/or
1702 formatted messages that were logged. (Contributed by Antoine Pitrou in
1703 :issue:`18937`.)
1704
1705 Test discovery now works with namespace packages (Contributed by Claudiu Popa
1706 in :issue:`17457`.)
1707
1708 :mod:`unittest.mock` objects now inspect their specification signatures when
1709 matching calls, which means an argument can now be matched by either position
1710 or name, instead of only by position. (Contributed by Antoine Pitrou in
1711 :issue:`17015`.)
1712
1713 :func:`~unittest.mock.mock_open` objects now have ``readline`` and ``readlines``
1714 methods. (Contributed by Toshio Kuratomi in :issue:`17467`.)
1715
1716
1717 venv
1718 ----
1719
1720 :mod:`venv` now includes activation scripts for the ``csh`` and ``fish``
1721 shells. (Contributed by Andrew Svetlov in :issue:`15417`.)
1722
1723 :class:`~venv.EnvBuilder` and the :func:`~venv.create` convenience function
1724 take a new keyword argument *with_pip*, which defaults to ``False``, that
1725 controls whether or not :class:`~venv.EnvBuilder` ensures that ``pip`` is
1726 installed in the virtual environment. (Contributed by Nick Coghlan in
1727 :issue:`19552` as part of the :pep:`453` implementation.)
1728
1729
1730 wave
1731 ----
1732
1733 The :meth:`~wave.Wave_read.getparams` method now returns a namedtuple rather
1734 than a plain tuple. (Contributed by Claudiu Popa in :issue:`17487`.)
1735
1736 :meth:`wave.open` now supports the context management protocol. (Contributed
1737 by Claudiu Popa in :issue:`17616`.)
1738
1739 :mod:`wave` can now :ref:`write output to unseekable files
1740 <wave-write-objects>`. (Contributed by David Jones, Guilherme Polo, and Serhiy
1741 Storchaka in :issue:`5202`.)
1742
1743 The :meth:`~wave.Wave_write.writeframesraw` and
1744 :meth:`~wave.Wave_write.writeframes` methods now accept any :term:`bytes-like
1745 object`. (Contributed by Serhiy Storchaka in :issue:`8311`.)
1746
1747
1748 weakref
1749 -------
1750
1751 New :class:`~weakref.WeakMethod` class simulates weak references to bound
1752 methods. (Contributed by Antoine Pitrou in :issue:`14631`.)
1753
1754 New :class:`~weakref.finalize` class makes it possible to register a callback
1755 to be invoked when an object is garbage collected, without needing to
1756 carefully manage the lifecycle of the weak reference itself. (Contributed by
1757 Richard Oudkerk in :issue:`15528`.)
1758
1759 The callback, if any, associated with a :class:`~weakref.ref` is now
1760 exposed via the :attr:`~weakref.ref.__callback__` attribute. (Contributed
1761 by Mark Dickinson in :issue:`17643`.)
1762
1763
1764 xml.etree
1765 ---------
1766
1767 A new parser, :class:`~xml.etree.ElementTree.XMLPullParser`, allows a
1768 non-blocking applications to parse XML documents. An example can be
1769 seen at :ref:`elementtree-pull-parsing`. (Contributed by Antoine
1770 Pitrou in :issue:`17741`.)
1771
1772 The :mod:`xml.etree.ElementTree` :func:`~xml.etree.ElementTree.tostring` and
1773 :func:`~xml.etree.ElementTree.tostringlist` functions, and the
1774 :class:`~xml.etree.ElementTree.ElementTree`
1775 :meth:`~xml.etree.ElementTree.ElementTree.write` method, now have a
1776 *short_empty_elements* :ref:`keyword-only parameter <keyword-only_parameter>`
1777 providing control over whether elements with no content are written in
1778 abbreviated (``<tag />``) or expanded (``<tag></tag>``) form. (Contributed by
1779 Ariel Poliak and Serhiy Storchaka in :issue:`14377`.)
1780
1781
1782 zipfile
1783 -------
1784
1785 The :meth:`~zipfile.PyZipFile.writepy` method of the
1786 :class:`~zipfile.PyZipFile` class has a new *filterfunc* option that can be
1787 used to control which directories and files are added to the archive. For
1788 example, this could be used to exclude test files from the archive.
1789 (Contributed by Christian Tismer in :issue:`19274`.)
1790
1791 The *allowZip64* parameter to :class:`~zipfile.ZipFile` and
1792 :class:`~zipfile.PyZipFile` is now ``True`` by default. (Contributed by
1793 William Mallard in :issue:`17201`.)
1794
1795
1796
1797 CPython Implementation Changes
1798 ==============================
1799
1800
1801 .. _whatsnew-pep-445:
1802
1803 PEP 445: Customization of CPython Memory Allocators
1804 ---------------------------------------------------
1805
1806 :pep:`445` adds new C level interfaces to customize memory allocation in
1807 the CPython interpreter.
1808
1809 .. seealso::
1810
1811 :pep:`445` -- Add new APIs to customize Python memory allocators
1812 PEP written and implemented by Victor Stinner.
1813
1814
1815 .. _whatsnew-pep-442:
1816
1817 PEP 442: Safe Object Finalization
1818 ---------------------------------
1819
1820 :pep:`442` removes the current limitations and quirks of object finalization
1821 in CPython. With it, objects with :meth:`~object.__del__` methods, as well as
1822 generators with :keyword:`finally` clauses, can be finalized when they are
1823 part of a reference cycle.
1824
1825 As part of this change, module globals are no longer forcibly set to
1826 :const:`None` during interpreter shutdown in most cases, instead relying
1827 on the normal operation of the cyclic garbage collector. This avoids a
1828 whole class of interpreter-shutdown-time errors, usually involving
1829 ``__del__`` methods, that have plagued Python since the cyclic GC
1830 was first introduced.
1831
1832 .. seealso::
1833
1834 :pep:`442` -- Safe object finalization
1835 PEP written and implemented by Antoine Pitrou.
1836
1837
1838 .. _whatsnew-pep-456:
1839
1840 PEP 456: Secure and Interchangeable Hash Algorithm
1841 --------------------------------------------------
1842
1843 :pep:`456` follows up on earlier security fix work done on Python's hash
1844 algorithm to address certain DOS attacks to which public facing APIs backed by
1845 dictionary lookups may be subject. (See :issue:`14621` for the start of the
1846 current round of improvements.) The PEP unifies CPython's hash code to make it
1847 easier for a packager to substitute a different hash algorithm, and switches
1848 Python's default implementation to a SipHash implementation on platforms that
1849 have a 64 bit data type. Any performance differences in comparison with the
1850 older FNV algorithm are trivial.
1851
1852 The PEP adds additional fields to the :data:`sys.hash_info` named tuple to
1853 describe the hash algorithm in use by the currently executing binary. Otherwise,
1854 the PEP does not alter any existing CPython APIs.
1855
1856
1857 .. _whatsnew-pep-436:
1858
1859 PEP 436: Argument Clinic
1860 ------------------------
1861
1862 "Argument Clinic" (:pep:`436`) is now part of the CPython build process
1863 and can be used to simplify the process of defining and maintaining
1864 accurate signatures for builtins and standard library extension modules
1865 implemented in C.
1866
1867 Some standard library extension modules have been converted to use Argument
1868 Clinic in Python 3.4, and :mod:`pydoc` and :mod:`inspect` have been updated
1869 accordingly.
1870
1871 It is expected that signature metadata for programmatic introspection will
1872 be added to additional callables implemented in C as part of Python 3.4
1873 maintenance releases.
1874
1875 .. note::
1876 The Argument Clinic PEP is not fully up to date with the state of the
1877 implementation. This has been deemed acceptable by the release manager
1878 and core development team in this case, as Argument Clinic will not
1879 be made available as a public API for third party use in Python 3.4.
1880
1881 .. seealso::
1882
1883 :pep:`436` -- The Argument Clinic DSL
1884 PEP written and implemented by Larry Hastings.
1885
1886
1887 Other Build and C API Changes
1888 -----------------------------
1889
1890 * The new :c:func:`PyType_GetSlot` function has been added to the stable ABI,
1891 allowing retrieval of function pointers from named type slots when using
1892 the limited API. (Contributed by Martin von Löwis in :issue:`17162`.)
1893
1894 * The new :c:func:`!Py_SetStandardStreamEncoding` pre-initialization API
1895 allows applications embedding the CPython interpreter to reliably force
1896 a particular encoding and error handler for the standard streams.
1897 (Contributed by Bastien Montagne and Nick Coghlan in :issue:`16129`.)
1898
1899 * Most Python C APIs that don't mutate string arguments are now correctly
1900 marked as accepting ``const char *`` rather than ``char *``. (Contributed
1901 by Serhiy Storchaka in :issue:`1772673`.)
1902
1903 * A new shell version of ``python-config`` can be used even when a python
1904 interpreter is not available (for example, in cross compilation scenarios).
1905
1906 * :c:func:`PyUnicode_FromFormat` now supports width and precision
1907 specifications for ``%s``, ``%A``, ``%U``, ``%V``, ``%S``, and ``%R``.
1908 (Contributed by Ysj Ray and Victor Stinner in :issue:`7330`.)
1909
1910 * New function :c:func:`PyStructSequence_InitType2` supplements the
1911 existing :c:func:`PyStructSequence_InitType` function. The difference
1912 is that it returns ``0`` on success and ``-1`` on failure.
1913
1914 * The CPython source can now be compiled using the address sanity checking
1915 features of recent versions of GCC and clang: the false alarms in the small
1916 object allocator have been silenced. (Contributed by Dhiru Kholia in
1917 :issue:`18596`.)
1918
1919 * The Windows build now uses `Address Space Layout Randomization
1920 <https://en.wikipedia.org/wiki/Address_space_layout_randomization>`_ and `Data Execution Prevention
1921 <https://en.wikipedia.org/wiki/Data_Execution_Prevention>`_. (Contributed by
1922 Christian Heimes in :issue:`16632`.)
1923
1924 * New function :c:func:`PyObject_LengthHint` is the C API equivalent
1925 of :func:`operator.length_hint`. (Contributed by Armin Ronacher in
1926 :issue:`16148`.)
1927
1928
1929 .. _other-improvements-3.4:
1930
1931 Other Improvements
1932 ------------------
1933
1934 .. _whatsnew-isolated-mode:
1935
1936 * The :ref:`python <using-on-cmdline>` command has a new :ref:`option
1937 <using-on-misc-options>`, ``-I``, which causes it to run in "isolated mode",
1938 which means that :data:`sys.path` contains neither the script's directory nor
1939 the user's ``site-packages`` directory, and all :envvar:`!PYTHON*` environment
1940 variables are ignored (it implies both ``-s`` and ``-E``). Other
1941 restrictions may also be applied in the future, with the goal being to
1942 isolate the execution of a script from the user's environment. This is
1943 appropriate, for example, when Python is used to run a system script. On
1944 most POSIX systems it can and should be used in the ``#!`` line of system
1945 scripts. (Contributed by Christian Heimes in :issue:`16499`.)
1946
1947 * Tab-completion is now enabled by default in the interactive interpreter
1948 on systems that support :mod:`readline`. History is also enabled by default,
1949 and is written to (and read from) the file :file:`~/.python-history`.
1950 (Contributed by Antoine Pitrou and Éric Araujo in :issue:`5845`.)
1951
1952 * Invoking the Python interpreter with ``--version`` now outputs the version to
1953 standard output instead of standard error (:issue:`18338`). Similar changes
1954 were made to :mod:`argparse` (:issue:`18920`) and other modules that have
1955 script-like invocation capabilities (:issue:`18922`).
1956
1957 * The CPython Windows installer now adds ``.py`` to the :envvar:`PATHEXT`
1958 variable when extensions are registered, allowing users to run a python
1959 script at the windows command prompt by just typing its name without the
1960 ``.py`` extension. (Contributed by Paul Moore in :issue:`18569`.)
1961
1962 * A new ``make`` target `coverage-report
1963 <https://devguide.python.org/coverage/#measuring-coverage-of-c-code-with-gcov-and-lcov>`_
1964 will build python, run the test suite, and generate an HTML coverage report
1965 for the C codebase using ``gcov`` and `lcov
1966 <https://github.com/linux-test-project/lcov>`_.
1967
1968 * The ``-R`` option to the :ref:`python regression test suite <regrtest>` now
1969 also checks for memory allocation leaks, using
1970 :func:`sys.getallocatedblocks`. (Contributed by Antoine Pitrou in
1971 :issue:`13390`.)
1972
1973 * ``python -m`` now works with namespace packages.
1974
1975 * The :mod:`stat` module is now implemented in C, which means it gets the
1976 values for its constants from the C header files, instead of having the
1977 values hard-coded in the python module as was previously the case.
1978
1979 * Loading multiple python modules from a single OS module (``.so``, ``.dll``)
1980 now works correctly (previously it silently returned the first python
1981 module in the file). (Contributed by Václav Šmilauer in :issue:`16421`.)
1982
1983 * A new opcode, :opcode:`!LOAD_CLASSDEREF`, has been added to fix a bug in the
1984 loading of free variables in class bodies that could be triggered by certain
1985 uses of :ref:`__prepare__ <prepare>`. (Contributed by Benjamin Peterson in
1986 :issue:`17853`.)
1987
1988 * A number of MemoryError-related crashes were identified and fixed by Victor
1989 Stinner using his :pep:`445`-based ``pyfailmalloc`` tool (:issue:`18408`,
1990 :issue:`18520`).
1991
1992 * The ``pyvenv`` command now accepts a ``--copies`` option
1993 to use copies rather than symlinks even on systems where symlinks are the
1994 default. (Contributed by Vinay Sajip in :issue:`18807`.)
1995
1996 * The ``pyvenv`` command also accepts a ``--without-pip``
1997 option to suppress the otherwise-automatic bootstrapping of pip into
1998 the virtual environment. (Contributed by Nick Coghlan in :issue:`19552`
1999 as part of the :pep:`453` implementation.)
2000
2001 * The encoding name is now optional in the value set for the
2002 :envvar:`PYTHONIOENCODING` environment variable. This makes it possible to
2003 set just the error handler, without changing the default encoding.
2004 (Contributed by Serhiy Storchaka in :issue:`18818`.)
2005
2006 * The :mod:`bz2`, :mod:`lzma`, and :mod:`gzip` module ``open`` functions now
2007 support ``x`` (exclusive creation) mode. (Contributed by Tim Heaney and
2008 Vajrasky Kok in :issue:`19201`, :issue:`19222`, and :issue:`19223`.)
2009
2010
2011 Significant Optimizations
2012 -------------------------
2013
2014 * The UTF-32 decoder is now 3x to 4x faster. (Contributed by Serhiy Storchaka
2015 in :issue:`14625`.)
2016
2017 * The cost of hash collisions for sets is now reduced. Each hash table
2018 probe now checks a series of consecutive, adjacent key/hash pairs before
2019 continuing to make random probes through the hash table. This exploits
2020 cache locality to make collision resolution less expensive.
2021 The collision resolution scheme can be described as a hybrid of linear
2022 probing and open addressing. The number of additional linear probes
2023 defaults to nine. This can be changed at compile-time by defining
2024 LINEAR_PROBES to be any value. Set LINEAR_PROBES=0 to turn-off
2025 linear probing entirely. (Contributed by Raymond Hettinger in
2026 :issue:`18771`.)
2027
2028 * The interpreter starts about 30% faster. A couple of measures lead to the
2029 speedup. The interpreter loads fewer modules on startup, e.g. the :mod:`re`,
2030 :mod:`collections` and :mod:`locale` modules and their dependencies are no
2031 longer imported by default. The marshal module has been improved to load
2032 compiled Python code faster. (Contributed by Antoine Pitrou, Christian
2033 Heimes and Victor Stinner in :issue:`19219`, :issue:`19218`, :issue:`19209`,
2034 :issue:`19205` and :issue:`9548`.)
2035
2036 * :class:`bz2.BZ2File` is now as fast or faster than the Python2 version for
2037 most cases. :class:`lzma.LZMAFile` has also been optimized. (Contributed by
2038 Serhiy Storchaka and Nadeem Vawda in :issue:`16034`.)
2039
2040 * :func:`random.getrandbits` is 20%-40% faster for small integers (the most
2041 common use case). (Contributed by Serhiy Storchaka in :issue:`16674`.)
2042
2043 * By taking advantage of the new storage format for strings, pickling of
2044 strings is now significantly faster. (Contributed by Victor Stinner and
2045 Antoine Pitrou in :issue:`15596`.)
2046
2047 * A performance issue in :meth:`!io.FileIO.readall` has been solved. This
2048 particularly affects Windows, and significantly speeds up the case of piping
2049 significant amounts of data through :mod:`subprocess`. (Contributed
2050 by Richard Oudkerk in :issue:`15758`.)
2051
2052 * :func:`html.escape` is now 10x faster. (Contributed by Matt Bryant in
2053 :issue:`18020`.)
2054
2055 * On Windows, the native ``VirtualAlloc`` is now used instead of the CRT
2056 ``malloc`` in ``obmalloc``. Artificial benchmarks show about a 3% memory
2057 savings.
2058
2059 * :func:`os.urandom` now uses a lazily opened persistent file descriptor
2060 so as to avoid using many file descriptors when run in parallel from
2061 multiple threads. (Contributed by Antoine Pitrou in :issue:`18756`.)
2062
2063
2064 .. _deprecated-3.4:
2065
2066 Deprecated
2067 ==========
2068
2069 This section covers various APIs and other features that have been deprecated
2070 in Python 3.4, and will be removed in Python 3.5 or later. In most (but not
2071 all) cases, using the deprecated APIs will produce a :exc:`DeprecationWarning`
2072 when the interpreter is run with deprecation warnings enabled (for example, by
2073 using ``-Wd``).
2074
2075
2076 Deprecations in the Python API
2077 ------------------------------
2078
2079 * As mentioned in :ref:`whatsnew-pep-451`, a number of :mod:`importlib`
2080 methods and functions are deprecated: :meth:`!importlib.find_loader` is
2081 replaced by :func:`importlib.util.find_spec`;
2082 :meth:`!importlib.machinery.PathFinder.find_module` is replaced by
2083 :meth:`importlib.machinery.PathFinder.find_spec`;
2084 :meth:`!importlib.abc.MetaPathFinder.find_module` is replaced by
2085 :meth:`importlib.abc.MetaPathFinder.find_spec`;
2086 :meth:`!importlib.abc.PathEntryFinder.find_loader` and
2087 :meth:`!find_module` are replaced by
2088 :meth:`importlib.abc.PathEntryFinder.find_spec`; all of the :samp:`{xxx}Loader` ABC
2089 ``load_module`` methods (:meth:`!importlib.abc.Loader.load_module`,
2090 :meth:`!importlib.abc.InspectLoader.load_module`,
2091 :meth:`!importlib.abc.FileLoader.load_module`,
2092 :meth:`!importlib.abc.SourceLoader.load_module`) should no longer be
2093 implemented, instead loaders should implement an
2094 ``exec_module`` method
2095 (:meth:`importlib.abc.Loader.exec_module`,
2096 :meth:`importlib.abc.InspectLoader.exec_module`
2097 :meth:`importlib.abc.SourceLoader.exec_module`) and let the import system
2098 take care of the rest; and
2099 :meth:`!importlib.abc.Loader.module_repr`,
2100 :meth:`!importlib.util.module_for_loader`, :meth:`!importlib.util.set_loader`,
2101 and :meth:`!importlib.util.set_package` are no longer needed because their
2102 functions are now handled automatically by the import system.
2103
2104 * The :mod:`!imp` module is pending deprecation. To keep compatibility with
2105 Python 2/3 code bases, the module's removal is currently not scheduled.
2106
2107 * The :mod:`!formatter` module is pending deprecation and is slated for removal
2108 in Python 3.6.
2109
2110 * ``MD5`` as the default *digestmod* for the :func:`hmac.new` function is
2111 deprecated. Python 3.6 will require an explicit digest name or constructor as
2112 *digestmod* argument.
2113
2114 * The internal ``Netrc`` class in the :mod:`ftplib` module has been documented
2115 as deprecated in its docstring for quite some time. It now emits a
2116 :exc:`DeprecationWarning` and will be removed completely in Python 3.5.
2117
2118 * The undocumented *endtime* argument to :meth:`subprocess.Popen.wait` should
2119 not have been exposed and is hopefully not in use; it is deprecated and
2120 will mostly likely be removed in Python 3.5.
2121
2122 * The *strict* argument of :class:`~html.parser.HTMLParser` is deprecated.
2123
2124 * The :mod:`plistlib` :func:`!readPlist`,
2125 :func:`!writePlist`, :func:`!readPlistFromBytes`, and
2126 :func:`!writePlistToBytes` functions are deprecated in favor of the
2127 corresponding new functions :func:`~plistlib.load`, :func:`~plistlib.dump`,
2128 :func:`~plistlib.loads`, and :func:`~plistlib.dumps`. :func:`!Data`
2129 is deprecated in favor of just using the :class:`bytes` constructor.
2130
2131 * The :mod:`sysconfig` key ``SO`` is deprecated, it has been replaced by
2132 ``EXT_SUFFIX``.
2133
2134 * The ``U`` mode accepted by various ``open`` functions is deprecated.
2135 In Python3 it does not do anything useful, and should be replaced by
2136 appropriate uses of :class:`io.TextIOWrapper` (if needed) and its *newline*
2137 argument.
2138
2139 * The *parser* argument of :func:`xml.etree.ElementTree.iterparse` has
2140 been deprecated, as has the *html* argument of
2141 :func:`~xml.etree.ElementTree.XMLParser`. To prepare for the removal of the
2142 latter, all arguments to ``XMLParser`` should be passed by keyword.
2143
2144
2145 Deprecated Features
2146 -------------------
2147
2148 * Running :ref:`idle` with the ``-n`` flag (no subprocess) is deprecated.
2149 However, the feature will not be removed until :issue:`18823` is resolved.
2150
2151 * The site module adding a "site-python" directory to sys.path, if it
2152 exists, is deprecated (:issue:`19375`).
2153
2154
2155
2156 Removed
2157 =======
2158
2159
2160 Operating Systems No Longer Supported
2161 -------------------------------------
2162
2163 Support for the following operating systems has been removed from the source
2164 and build tools:
2165
2166 * OS/2 (:issue:`16135`).
2167 * Windows 2000 (changeset e52df05b496a).
2168 * Windows systems where ``COMSPEC`` points to ``command.com`` (:issue:`14470`).
2169 * VMS (:issue:`16136`).
2170
2171
2172 API and Feature Removals
2173 ------------------------
2174
2175 The following obsolete and previously deprecated APIs and features have been
2176 removed:
2177
2178 * The unmaintained ``Misc/TextMate`` and ``Misc/vim`` directories have been
2179 removed (see the `devguide <https://devguide.python.org>`_
2180 for suggestions on what to use instead).
2181
2182 * The ``SO`` makefile macro is removed (it was replaced by the
2183 ``SHLIB_SUFFIX`` and ``EXT_SUFFIX`` macros) (:issue:`16754`).
2184
2185 * The ``PyThreadState.tick_counter`` field has been removed; its value has
2186 been meaningless since Python 3.2, when the "new GIL" was introduced
2187 (:issue:`19199`).
2188
2189 * ``PyLoader`` and ``PyPycLoader`` have been removed from :mod:`importlib`.
2190 (Contributed by Taras Lyapun in :issue:`15641`.)
2191
2192 * The *strict* argument to :class:`~http.client.HTTPConnection` and
2193 :class:`~http.client.HTTPSConnection` has been removed. HTTP 0.9-style
2194 "Simple Responses" are no longer supported.
2195
2196 * The deprecated :mod:`urllib.request.Request` getter and setter methods
2197 ``add_data``, ``has_data``, ``get_data``, ``get_type``, ``get_host``,
2198 ``get_selector``, ``set_proxy``, ``get_origin_req_host``, and
2199 ``is_unverifiable`` have been removed (use direct attribute access instead).
2200
2201 * Support for loading the deprecated ``TYPE_INT64`` has been removed from
2202 :mod:`marshal`. (Contributed by Dan Riti in :issue:`15480`.)
2203
2204 * :class:`inspect.Signature`: positional-only parameters are now required
2205 to have a valid name.
2206
2207 * :meth:`object.__format__` no longer accepts non-empty format strings, it now
2208 raises a :exc:`TypeError` instead. Using a non-empty string has been
2209 deprecated since Python 3.2. This change has been made to prevent a
2210 situation where previously working (but incorrect) code would start failing
2211 if an object gained a __format__ method, which means that your code may now
2212 raise a :exc:`TypeError` if you are using an ``'s'`` format code with objects
2213 that do not have a __format__ method that handles it. See :issue:`7994` for
2214 background.
2215
2216 * :meth:`!difflib.SequenceMatcher.isbjunk` and
2217 :meth:`!difflib.SequenceMatcher.isbpopular` were deprecated in 3.2, and have
2218 now been removed: use ``x in sm.bjunk`` and
2219 ``x in sm.bpopular``, where *sm* is a :class:`~difflib.SequenceMatcher` object
2220 (:issue:`13248`).
2221
2222
2223 Code Cleanups
2224 -------------
2225
2226 * The unused and undocumented internal ``Scanner`` class has been removed from
2227 the :mod:`pydoc` module.
2228
2229 * The private and effectively unused ``_gestalt`` module has been removed,
2230 along with the private :mod:`platform` functions ``_mac_ver_lookup``,
2231 ``_mac_ver_gstalt``, and ``_bcd2str``, which would only have ever been called
2232 on badly broken OSX systems (see :issue:`18393`).
2233
2234 * The hardcoded copies of certain :mod:`stat` constants that were included in
2235 the :mod:`tarfile` module namespace have been removed.
2236
2237
2238
2239 Porting to Python 3.4
2240 =====================
2241
2242 This section lists previously described changes and other bugfixes
2243 that may require changes to your code.
2244
2245
2246 Changes in 'python' Command Behavior
2247 ------------------------------------
2248
2249 * In a posix shell, setting the :envvar:`PATH` environment variable to
2250 an empty value is equivalent to not setting it at all. However, setting
2251 :envvar:`PYTHONPATH` to an empty value was *not* equivalent to not setting it
2252 at all: setting :envvar:`PYTHONPATH` to an empty value was equivalent to
2253 setting it to ``.``, which leads to confusion when reasoning by analogy to
2254 how :envvar:`PATH` works. The behavior now conforms to the posix convention
2255 for :envvar:`PATH`.
2256
2257 * The [X refs, Y blocks] output of a debug (``--with-pydebug``) build of the
2258 CPython interpreter is now off by default. It can be re-enabled using the
2259 ``-X showrefcount`` option. (Contributed by Ezio Melotti in :issue:`17323`.)
2260
2261 * The python command and most stdlib scripts (as well as :mod:`argparse`) now
2262 output ``--version`` information to ``stdout`` instead of ``stderr`` (for
2263 issue list see :ref:`other-improvements-3.4` above).
2264
2265
2266 Changes in the Python API
2267 -------------------------
2268
2269 * The ABCs defined in :mod:`importlib.abc` now either raise the appropriate
2270 exception or return a default value instead of raising
2271 :exc:`NotImplementedError` blindly. This will only affect code calling
2272 :func:`super` and falling through all the way to the ABCs. For compatibility,
2273 catch both :exc:`NotImplementedError` or the appropriate exception as needed.
2274
2275 * The module type now initializes the :attr:`~module.__package__` and
2276 :attr:`~module.__loader__`
2277 attributes to ``None`` by default. To determine if these attributes were set
2278 in a backwards-compatible fashion, use e.g.
2279 ``getattr(module, '__loader__', None) is not None``. (:issue:`17115`.)
2280
2281 * :meth:`!importlib.util.module_for_loader` now sets ``__loader__`` and
2282 ``__package__`` unconditionally to properly support reloading. If this is not
2283 desired then you will need to set these attributes manually. You can use
2284 :func:`!importlib.util.module_to_load` for module management.
2285
2286 * Import now resets relevant attributes (e.g. ``__name__``, ``__loader__``,
2287 ``__package__``, ``__file__``, ``__cached__``) unconditionally when reloading.
2288 Note that this restores a pre-3.3 behavior in that it means a module is
2289 re-found when re-loaded (:issue:`19413`).
2290
2291 * Frozen packages no longer set ``__path__`` to a list containing the package
2292 name, they now set it to an empty list. The previous behavior could cause
2293 the import system to do the wrong thing on submodule imports if there was
2294 also a directory with the same name as the frozen package. The correct way
2295 to determine if a module is a package or not is to use ``hasattr(module,
2296 '__path__')`` (:issue:`18065`).
2297
2298 * Frozen modules no longer define a ``__file__`` attribute. It's semantically
2299 incorrect for frozen modules to set the attribute as they are not loaded from
2300 any explicit location. If you must know that a module comes from frozen code
2301 then you can see if the module's ``__spec__.location`` is set to ``'frozen'``,
2302 check if the loader is a subclass of
2303 :class:`importlib.machinery.FrozenImporter`,
2304 or if Python 2 compatibility is necessary you can use :func:`!imp.is_frozen`.
2305
2306 * :func:`py_compile.compile` now raises :exc:`FileExistsError` if the file path
2307 it would write to is a symlink or a non-regular file. This is to act as a
2308 warning that import will overwrite those files with a regular file regardless
2309 of what type of file path they were originally.
2310
2311 * :meth:`importlib.abc.SourceLoader.get_source` no longer raises
2312 :exc:`ImportError` when the source code being loaded triggers a
2313 :exc:`SyntaxError` or :exc:`UnicodeDecodeError`. As :exc:`ImportError` is
2314 meant to be raised only when source code cannot be found but it should, it was
2315 felt to be over-reaching/overloading of that meaning when the source code is
2316 found but improperly structured. If you were catching ImportError before and
2317 wish to continue to ignore syntax or decoding issues, catch all three
2318 exceptions now.
2319
2320 * :func:`functools.update_wrapper` and :func:`functools.wraps` now correctly
2321 set the ``__wrapped__`` attribute to the function being wrapped, even if
2322 that function also had its ``__wrapped__`` attribute set. This means
2323 ``__wrapped__`` attributes now correctly link a stack of decorated
2324 functions rather than every ``__wrapped__`` attribute in the chain
2325 referring to the innermost function. Introspection libraries that
2326 assumed the previous behaviour was intentional can use
2327 :func:`inspect.unwrap` to access the first function in the chain that has
2328 no ``__wrapped__`` attribute.
2329
2330 * :func:`inspect.getfullargspec` has been reimplemented on top of
2331 :func:`inspect.signature` and hence handles a much wider variety of callable
2332 objects than it did in the past. It is expected that additional builtin and
2333 extension module callables will gain signature metadata over the course of
2334 the Python 3.4 series. Code that assumes that
2335 :func:`inspect.getfullargspec` will fail on non-Python callables may need
2336 to be adjusted accordingly.
2337
2338 * :class:`importlib.machinery.PathFinder` now passes on the current working
2339 directory to objects in :data:`sys.path_hooks` for the empty string. This
2340 results in :data:`sys.path_importer_cache` never containing ``''``, thus
2341 iterating through :data:`sys.path_importer_cache` based on :data:`sys.path`
2342 will not find all keys. A module's ``__file__`` when imported in the current
2343 working directory will also now have an absolute path, including when using
2344 ``-m`` with the interpreter (except for ``__main__.__file__`` when a script
2345 has been executed directly using a relative path) (Contributed by Brett
2346 Cannon in :issue:`18416`). is specified on the command-line)
2347 (:issue:`18416`).
2348
2349 * The removal of the *strict* argument to :class:`~http.client.HTTPConnection`
2350 and :class:`~http.client.HTTPSConnection` changes the meaning of the
2351 remaining arguments if you are specifying them positionally rather than by
2352 keyword. If you've been paying attention to deprecation warnings your code
2353 should already be specifying any additional arguments via keywords.
2354
2355 * Strings between ``from __future__ import ...`` statements now *always* raise
2356 a :exc:`SyntaxError`. Previously if there was no leading docstring, an
2357 interstitial string would sometimes be ignored. This brings CPython into
2358 compliance with the language spec; Jython and PyPy already were.
2359 (:issue:`17434`).
2360
2361 * :meth:`ssl.SSLSocket.getpeercert` and :meth:`ssl.SSLSocket.do_handshake`
2362 now raise an :exc:`OSError` with ``ENOTCONN`` when the ``SSLSocket`` is not
2363 connected, instead of the previous behavior of raising an
2364 :exc:`AttributeError`. In addition, :meth:`~ssl.SSLSocket.getpeercert`
2365 will raise a :exc:`ValueError` if the handshake has not yet been done.
2366
2367 * :func:`base64.b32decode` now raises a :exc:`binascii.Error` when the
2368 input string contains non-b32-alphabet characters, instead of a
2369 :exc:`TypeError`. This particular :exc:`TypeError` was missed when the other
2370 :exc:`TypeError`\ s were converted. (Contributed by Serhiy Storchaka in
2371 :issue:`18011`.) Note: this change was also inadvertently applied in Python
2372 3.3.3.
2373
2374 * The :attr:`!file` attribute is now automatically closed when
2375 the creating :class:`!cgi.FieldStorage` instance is garbage collected. If you
2376 were pulling the file object out separately from the :class:`!cgi.FieldStorage`
2377 instance and not keeping the instance alive, then you should either store the
2378 entire :class:`!cgi.FieldStorage` instance or read the contents of the file
2379 before the :class:`!cgi.FieldStorage` instance is garbage collected.
2380
2381 * Calling ``read`` or ``write`` on a closed SSL socket now raises an
2382 informative :exc:`ValueError` rather than the previous more mysterious
2383 :exc:`AttributeError` (:issue:`9177`).
2384
2385 * :meth:`slice.indices` no longer produces an :exc:`OverflowError` for huge
2386 values. As a consequence of this fix, :meth:`slice.indices` now raises a
2387 :exc:`ValueError` if given a negative length; previously it returned nonsense
2388 values (:issue:`14794`).
2389
2390 * The :class:`complex` constructor, unlike the :mod:`cmath` functions, was
2391 incorrectly accepting :class:`float` values if an object's ``__complex__``
2392 special method returned one. This now raises a :exc:`TypeError`.
2393 (:issue:`16290`.)
2394
2395 * The :class:`int` constructor in 3.2 and 3.3 erroneously accepts :class:`float`
2396 values for the *base* parameter. It is unlikely anyone was doing this, but
2397 if so, it will now raise a :exc:`TypeError` (:issue:`16772`).
2398
2399 * Defaults for keyword-only arguments are now evaluated *after* defaults for
2400 regular keyword arguments, instead of before. Hopefully no one wrote any
2401 code that depends on the previous buggy behavior (:issue:`16967`).
2402
2403 * Stale thread states are now cleared after :func:`~os.fork`. This may cause
2404 some system resources to be released that previously were incorrectly kept
2405 perpetually alive (for example, database connections kept in thread-local
2406 storage). (:issue:`17094`.)
2407
2408 * Parameter names in ``__annotations__`` dicts are now mangled properly,
2409 similarly to :attr:`~function.__kwdefaults__`.
2410 (Contributed by Yury Selivanov in :issue:`20625`.)
2411
2412 * :attr:`hashlib.hash.name` now always returns the identifier in lower case.
2413 Previously some builtin hashes had uppercase names, but now that it is a
2414 formal public interface the naming has been made consistent (:issue:`18532`).
2415
2416 * Because :mod:`unittest.TestSuite` now drops references to tests after they
2417 are run, test harnesses that reuse a :class:`~unittest.TestSuite` to re-run
2418 a set of tests may fail. Test suites should not be re-used in this fashion
2419 since it means state is retained between test runs, breaking the test
2420 isolation that :mod:`unittest` is designed to provide. However, if the lack
2421 of isolation is considered acceptable, the old behavior can be restored by
2422 creating a :mod:`~unittest.TestSuite` subclass that defines a
2423 ``_removeTestAtIndex`` method that does nothing (see
2424 :meth:`.TestSuite.__iter__`) (:issue:`11798`).
2425
2426 * :mod:`unittest` now uses :mod:`argparse` for command line parsing. There are
2427 certain invalid command forms that used to work that are no longer allowed;
2428 in theory this should not cause backward compatibility issues since the
2429 disallowed command forms didn't make any sense and are unlikely to be in use.
2430
2431 * The :func:`re.split`, :func:`re.findall`, and :func:`re.sub` functions, and
2432 the :meth:`~re.Match.group` and :meth:`~re.Match.groups` methods of
2433 ``match`` objects now always return a *bytes* object when the string
2434 to be matched is a :term:`bytes-like object`. Previously the return type
2435 matched the input type, so if your code was depending on the return value
2436 being, say, a ``bytearray``, you will need to change your code.
2437
2438 * :mod:`!audioop` functions now raise an error immediately if passed string
2439 input, instead of failing randomly later on (:issue:`16685`).
2440
2441 * The new *convert_charrefs* argument to :class:`~html.parser.HTMLParser`
2442 currently defaults to ``False`` for backward compatibility, but will
2443 eventually be changed to default to ``True``. It is recommended that you add
2444 this keyword, with the appropriate value, to any
2445 :class:`~html.parser.HTMLParser` calls in your code (:issue:`13633`).
2446
2447 * Since the *digestmod* argument to the :func:`hmac.new` function will in the
2448 future have no default, all calls to :func:`hmac.new` should be changed to
2449 explicitly specify a *digestmod* (:issue:`17276`).
2450
2451 * Calling :func:`sysconfig.get_config_var` with the ``SO`` key, or looking
2452 ``SO`` up in the results of a call to :func:`sysconfig.get_config_vars`
2453 is deprecated. This key should be replaced by ``EXT_SUFFIX`` or
2454 ``SHLIB_SUFFIX``, depending on the context (:issue:`19555`).
2455
2456 * Any calls to ``open`` functions that specify ``U`` should be modified.
2457 ``U`` is ineffective in Python3 and will eventually raise an error if used.
2458 Depending on the function, the equivalent of its old Python2 behavior can be
2459 achieved using either a *newline* argument, or if necessary by wrapping the
2460 stream in :mod:`~io.TextIOWrapper` to use its *newline* argument
2461 (:issue:`15204`).
2462
2463 * If you use ``pyvenv`` in a script and desire that pip
2464 *not* be installed, you must add ``--without-pip`` to your command
2465 invocation.
2466
2467 * The default behavior of :func:`json.dump` and :func:`json.dumps` when
2468 an indent is specified has changed: it no longer produces trailing
2469 spaces after the item separating commas at the ends of lines. This
2470 will matter only if you have tests that are doing white-space-sensitive
2471 comparisons of such output (:issue:`16333`).
2472
2473 * :mod:`doctest` now looks for doctests in extension module ``__doc__``
2474 strings, so if your doctest test discovery includes extension modules that
2475 have things that look like doctests in them you may see test failures you've
2476 never seen before when running your tests (:issue:`3158`).
2477
2478 * The :mod:`collections.abc` module has been slightly refactored as
2479 part of the Python startup improvements. As a consequence of this, it is no
2480 longer the case that importing :mod:`collections` automatically imports
2481 :mod:`collections.abc`. If your program depended on the (undocumented)
2482 implicit import, you will need to add an explicit ``import collections.abc``
2483 (:issue:`20784`).
2484
2485
2486 Changes in the C API
2487 --------------------
2488
2489 * :c:func:`PyEval_EvalFrameEx`, :c:func:`PyObject_Repr`, and
2490 :c:func:`PyObject_Str`, along with some other internal C APIs, now include
2491 a debugging assertion that ensures they are not used in situations where
2492 they may silently discard a currently active exception. In cases where
2493 discarding the active exception is expected and desired (for example,
2494 because it has already been saved locally with :c:func:`PyErr_Fetch` or
2495 is being deliberately replaced with a different exception), an explicit
2496 :c:func:`PyErr_Clear` call will be needed to avoid triggering the
2497 assertion when invoking these operations (directly or indirectly) and
2498 running against a version of Python that is compiled with assertions
2499 enabled.
2500
2501 * :c:func:`PyErr_SetImportError` now sets :exc:`TypeError` when its **msg**
2502 argument is not set. Previously only ``NULL`` was returned with no exception
2503 set.
2504
2505 * The result of the :c:data:`PyOS_ReadlineFunctionPointer` callback must
2506 now be a string allocated by :c:func:`PyMem_RawMalloc` or
2507 :c:func:`PyMem_RawRealloc`, or ``NULL`` if an error occurred, instead of a
2508 string allocated by :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`
2509 (:issue:`16742`)
2510
2511 * :c:func:`PyThread_set_key_value` now always set the value. In Python
2512 3.3, the function did nothing if the key already exists (if the current
2513 value is a non-``NULL`` pointer).
2514
2515 * The ``f_tstate`` (thread state) field of the :c:type:`PyFrameObject`
2516 structure has been removed to fix a bug: see :issue:`14432` for the
2517 rationale.
2518
2519 Changed in 3.4.3
2520 ================
2521
2522 .. _pep-476:
2523
2524 PEP 476: Enabling certificate verification by default for stdlib http clients
2525 -----------------------------------------------------------------------------
2526
2527 :mod:`http.client` and modules which use it, such as :mod:`urllib.request` and
2528 :mod:`xmlrpc.client`, will now verify that the server presents a certificate
2529 which is signed by a CA in the platform trust store and whose hostname matches
2530 the hostname being requested by default, significantly improving security for
2531 many applications.
2532
2533 For applications which require the old previous behavior, they can pass an
2534 alternate context::
2535
2536 import urllib.request
2537 import ssl
2538
2539 # This disables all verification
2540 context = ssl._create_unverified_context()
2541
2542 # This allows using a specific certificate for the host, which doesn't need
2543 # to be in the trust store
2544 context = ssl.create_default_context(cafile="/path/to/file.crt")
2545
2546 urllib.request.urlopen("https://invalid-cert", context=context)