Debian Codegraph @

Doc/whatsnew/3.3.rst

raw
1 ****************************
2 What's New In Python 3.3
3 ****************************
4
5 .. Rules for maintenance:
6
7 * Anyone can add text to this document. Do not spend very much time
8 on the wording of your changes, because your text will probably
9 get rewritten to some degree.
10
11 * The maintainer will go through Misc/NEWS periodically and add
12 changes; it's therefore more important to add your changes to
13 Misc/NEWS than to this file.
14
15 * This is not a complete list of every single change; completeness
16 is the purpose of Misc/NEWS. Some changes I consider too small
17 or esoteric to include. If such a change is added to the text,
18 I'll just remove it. (This is another reason you shouldn't spend
19 too much time on writing your addition.)
20
21 * If you want to draw your new text to the attention of the
22 maintainer, add 'XXX' to the beginning of the paragraph or
23 section.
24
25 * It's OK to just add a fragmentary note about a change. For
26 example: "XXX Describe the transmogrify() function added to the
27 socket module." The maintainer will research the change and
28 write the necessary text.
29
30 * You can comment out your additions if you like, but it's not
31 necessary (especially when a final release is some months away).
32
33 * Credit the author of a patch or bugfix. Just the name is
34 sufficient; the e-mail address isn't necessary.
35
36 * It's helpful to add the bug/patch number as a comment:
37
38 XXX Describe the transmogrify() function added to the socket
39 module.
40 (Contributed by P.Y. Developer in :issue:`12345`.)
41
42 This saves the maintainer the effort of going through the Mercurial log
43 when researching a change.
44
45 This article explains the new features in Python 3.3, compared to 3.2.
46 Python 3.3 was released on September 29, 2012. For full details,
47 see the `changelog <https://docs.python.org/3.3/whatsnew/changelog.html>`_.
48
49 .. seealso::
50
51 :pep:`398` - Python 3.3 Release Schedule
52
53
54 Summary -- Release highlights
55 =============================
56
57 .. This section singles out the most important changes in Python 3.3.
58 Brevity is key.
59
60 New syntax features:
61
62 * New ``yield from`` expression for :ref:`generator delegation <pep-380>`.
63 * The ``u'unicode'`` syntax is accepted again for :class:`str` objects.
64
65 New library modules:
66
67 * :mod:`faulthandler` (helps debugging low-level crashes)
68 * :mod:`ipaddress` (high-level objects representing IP addresses and masks)
69 * :mod:`lzma` (compress data using the XZ / LZMA algorithm)
70 * :mod:`unittest.mock` (replace parts of your system under test with mock objects)
71 * :mod:`venv` (Python :ref:`virtual environments <pep-405>`, as in the
72 popular ``virtualenv`` package)
73
74 New built-in features:
75
76 * Reworked :ref:`I/O exception hierarchy <pep-3151>`.
77
78 Implementation improvements:
79
80 * Rewritten :ref:`import machinery <importlib>` based on :mod:`importlib`.
81 * More compact :ref:`unicode strings <pep-393>`.
82 * More compact :ref:`attribute dictionaries <pep-412>`.
83
84 Significantly Improved Library Modules:
85
86 * C Accelerator for the :ref:`decimal <new-decimal>` module.
87 * Better unicode handling in the :ref:`email <new-email>` module
88 (:term:`provisional <provisional package>`).
89
90 Security improvements:
91
92 * Hash randomization is switched on by default.
93
94 Please read on for a comprehensive list of user-facing changes.
95
96
97 .. _pep-405:
98
99 PEP 405: Virtual Environments
100 =============================
101
102 Virtual environments help create separate Python setups while sharing a
103 system-wide base install, for ease of maintenance. Virtual environments
104 have their own set of private site packages (i.e. locally installed
105 libraries), and are optionally segregated from the system-wide site
106 packages. Their concept and implementation are inspired by the popular
107 ``virtualenv`` third-party package, but benefit from tighter integration
108 with the interpreter core.
109
110 This PEP adds the :mod:`venv` module for programmatic access, and the
111 ``pyvenv`` script for command-line access and
112 administration. The Python interpreter checks for a ``pyvenv.cfg``,
113 file whose existence signals the base of a virtual environment's directory
114 tree.
115
116 .. seealso::
117
118 :pep:`405` - Python Virtual Environments
119 PEP written by Carl Meyer; implementation by Carl Meyer and Vinay Sajip
120
121
122 PEP 420: Implicit Namespace Packages
123 ====================================
124
125 Native support for package directories that don't require ``__init__.py``
126 marker files and can automatically span multiple path segments (inspired by
127 various third party approaches to namespace packages, as described in
128 :pep:`420`)
129
130 .. seealso::
131
132 :pep:`420` - Implicit Namespace Packages
133 PEP written by Eric V. Smith; implementation by Eric V. Smith
134 and Barry Warsaw
135
136
137 .. _pep-3118-update:
138
139 PEP 3118: New memoryview implementation and buffer protocol documentation
140 =========================================================================
141
142 The implementation of :pep:`3118` has been significantly improved.
143
144 The new memoryview implementation comprehensively fixes all ownership and
145 lifetime issues of dynamically allocated fields in the Py_buffer struct
146 that led to multiple crash reports. Additionally, several functions that
147 crashed or returned incorrect results for non-contiguous or multi-dimensional
148 input have been fixed.
149
150 The memoryview object now has a PEP-3118 compliant getbufferproc()
151 that checks the consumer's request type. Many new features have been
152 added, most of them work in full generality for non-contiguous arrays
153 and arrays with suboffsets.
154
155 The documentation has been updated, clearly spelling out responsibilities
156 for both exporters and consumers. Buffer request flags are grouped into
157 basic and compound flags. The memory layout of non-contiguous and
158 multi-dimensional NumPy-style arrays is explained.
159
160 Features
161 --------
162
163 * All native single character format specifiers in struct module syntax
164 (optionally prefixed with '@') are now supported.
165
166 * With some restrictions, the cast() method allows changing of format and
167 shape of C-contiguous arrays.
168
169 * Multi-dimensional list representations are supported for any array type.
170
171 * Multi-dimensional comparisons are supported for any array type.
172
173 * One-dimensional memoryviews of hashable (read-only) types with formats B,
174 b or c are now hashable. (Contributed by Antoine Pitrou in :issue:`13411`.)
175
176 * Arbitrary slicing of any 1-D arrays type is supported. For example, it
177 is now possible to reverse a memoryview in *O*\ (1) by using a negative step.
178
179 API changes
180 -----------
181
182 * The maximum number of dimensions is officially limited to 64.
183
184 * The representation of empty shape, strides and suboffsets is now
185 an empty tuple instead of ``None``.
186
187 * Accessing a memoryview element with format 'B' (unsigned bytes)
188 now returns an integer (in accordance with the struct module syntax).
189 For returning a bytes object the view must be cast to 'c' first.
190
191 * memoryview comparisons now use the logical structure of the operands
192 and compare all array elements by value. All format strings in struct
193 module syntax are supported. Views with unrecognised format strings
194 are still permitted, but will always compare as unequal, regardless
195 of view contents.
196
197 * For further changes see `Build and C API Changes`_ and `Porting C code`_.
198
199 (Contributed by Stefan Krah in :issue:`10181`.)
200
201 .. seealso::
202
203 :pep:`3118` - Revising the Buffer Protocol
204
205
206 .. _pep-393:
207
208 PEP 393: Flexible String Representation
209 =======================================
210
211 The Unicode string type is changed to support multiple internal
212 representations, depending on the character with the largest Unicode ordinal
213 (1, 2, or 4 bytes) in the represented string. This allows a space-efficient
214 representation in common cases, but gives access to full UCS-4 on all
215 systems. For compatibility with existing APIs, several representations may
216 exist in parallel; over time, this compatibility should be phased out.
217
218 On the Python side, there should be no downside to this change.
219
220 On the C API side, :pep:`393` is fully backward compatible. The legacy API
221 should remain available at least five years. Applications using the legacy
222 API will not fully benefit of the memory reduction, or - worse - may use
223 a bit more memory, because Python may have to maintain two versions of each
224 string (in the legacy format and in the new efficient storage).
225
226 Functionality
227 -------------
228
229 Changes introduced by :pep:`393` are the following:
230
231 * Python now always supports the full range of Unicode code points, including
232 non-BMP ones (i.e. from ``U+0000`` to ``U+10FFFF``). The distinction between
233 narrow and wide builds no longer exists and Python now behaves like a wide
234 build, even under Windows.
235
236 * With the death of narrow builds, the problems specific to narrow builds have
237 also been fixed, for example:
238
239 * :func:`len` now always returns 1 for non-BMP characters,
240 so ``len('\U0010FFFF') == 1``;
241
242 * surrogate pairs are not recombined in string literals,
243 so ``'\uDBFF\uDFFF' != '\U0010FFFF'``;
244
245 * indexing or slicing non-BMP characters returns the expected value,
246 so ``'\U0010FFFF'[0]`` now returns ``'\U0010FFFF'`` and not ``'\uDBFF'``;
247
248 * all other functions in the standard library now correctly handle
249 non-BMP code points.
250
251 * The value of :data:`sys.maxunicode` is now always ``1114111`` (``0x10FFFF``
252 in hexadecimal). The :c:func:`!PyUnicode_GetMax` function still returns
253 either ``0xFFFF`` or ``0x10FFFF`` for backward compatibility, and it should
254 not be used with the new Unicode API (see :issue:`13054`).
255
256 * The :file:`./configure` flag ``--with-wide-unicode`` has been removed.
257
258 Performance and resource usage
259 ------------------------------
260
261 The storage of Unicode strings now depends on the highest code point in the string:
262
263 * pure ASCII and Latin1 strings (``U+0000-U+00FF``) use 1 byte per code point;
264
265 * BMP strings (``U+0000-U+FFFF``) use 2 bytes per code point;
266
267 * non-BMP strings (``U+10000-U+10FFFF``) use 4 bytes per code point.
268
269 The net effect is that for most applications, memory usage of string
270 storage should decrease significantly - especially compared to former
271 wide unicode builds - as, in many cases, strings will be pure ASCII
272 even in international contexts (because many strings store non-human
273 language data, such as XML fragments, HTTP headers, JSON-encoded data,
274 etc.). We also hope that it will, for the same reasons, increase CPU
275 cache efficiency on non-trivial applications. The memory usage of
276 Python 3.3 is two to three times smaller than Python 3.2, and a little
277 bit better than Python 2.7, on a Django benchmark (see the PEP for
278 details).
279
280 .. seealso::
281
282 :pep:`393` - Flexible String Representation
283 PEP written by Martin von Löwis; implementation by Torsten Becker
284 and Martin von Löwis.
285
286
287 .. _pep-397:
288
289 PEP 397: Python Launcher for Windows
290 ====================================
291
292 The Python 3.3 Windows installer now includes a ``py`` launcher application
293 that can be used to launch Python applications in a version independent
294 fashion.
295
296 This launcher is invoked implicitly when double-clicking ``*.py`` files.
297 If only a single Python version is installed on the system, that version
298 will be used to run the file. If multiple versions are installed, the most
299 recent version is used by default, but this can be overridden by including
300 a Unix-style "shebang line" in the Python script.
301
302 The launcher can also be used explicitly from the command line as the ``py``
303 application. Running ``py`` follows the same version selection rules as
304 implicitly launching scripts, but a more specific version can be selected
305 by passing appropriate arguments (such as ``-3`` to request Python 3 when
306 Python 2 is also installed, or ``-2.6`` to specifically request an earlier
307 Python version when a more recent version is installed).
308
309 In addition to the launcher, the Windows installer now includes an
310 option to add the newly installed Python to the system PATH. (Contributed
311 by Brian Curtin in :issue:`3561`.)
312
313 .. seealso::
314
315 :pep:`397` - Python Launcher for Windows
316 PEP written by Mark Hammond and Martin v. Löwis; implementation by
317 Vinay Sajip.
318
319 Launcher documentation: :ref:`launcher`
320
321 Installer PATH modification: :ref:`windows-path-mod`
322
323
324 .. _pep-3151:
325
326 PEP 3151: Reworking the OS and IO exception hierarchy
327 =====================================================
328
329 The hierarchy of exceptions raised by operating system errors is now both
330 simplified and finer-grained.
331
332 You don't have to worry anymore about choosing the appropriate exception
333 type between :exc:`OSError`, :exc:`IOError`, :exc:`EnvironmentError`,
334 :exc:`WindowsError`, :exc:`!mmap.error`, :exc:`socket.error` or
335 :exc:`select.error`. All these exception types are now only one:
336 :exc:`OSError`. The other names are kept as aliases for compatibility
337 reasons.
338
339 Also, it is now easier to catch a specific error condition. Instead of
340 inspecting the ``errno`` attribute (or ``args[0]``) for a particular
341 constant from the :mod:`errno` module, you can catch the adequate
342 :exc:`OSError` subclass. The available subclasses are the following:
343
344 * :exc:`BlockingIOError`
345 * :exc:`ChildProcessError`
346 * :exc:`ConnectionError`
347 * :exc:`FileExistsError`
348 * :exc:`FileNotFoundError`
349 * :exc:`InterruptedError`
350 * :exc:`IsADirectoryError`
351 * :exc:`NotADirectoryError`
352 * :exc:`PermissionError`
353 * :exc:`ProcessLookupError`
354 * :exc:`TimeoutError`
355
356 And the :exc:`ConnectionError` itself has finer-grained subclasses:
357
358 * :exc:`BrokenPipeError`
359 * :exc:`ConnectionAbortedError`
360 * :exc:`ConnectionRefusedError`
361 * :exc:`ConnectionResetError`
362
363 Thanks to the new exceptions, common usages of the :mod:`errno` can now be
364 avoided. For example, the following code written for Python 3.2::
365
366 from errno import ENOENT, EACCES, EPERM
367
368 try:
369 with open("document.txt") as f:
370 content = f.read()
371 except IOError as err:
372 if err.errno == ENOENT:
373 print("document.txt file is missing")
374 elif err.errno in (EACCES, EPERM):
375 print("You are not allowed to read document.txt")
376 else:
377 raise
378
379 can now be written without the :mod:`errno` import and without manual
380 inspection of exception attributes::
381
382 try:
383 with open("document.txt") as f:
384 content = f.read()
385 except FileNotFoundError:
386 print("document.txt file is missing")
387 except PermissionError:
388 print("You are not allowed to read document.txt")
389
390 .. seealso::
391
392 :pep:`3151` - Reworking the OS and IO Exception Hierarchy
393 PEP written and implemented by Antoine Pitrou
394
395
396 .. index::
397 single: yield; yield from (in What's New)
398
399 .. _pep-380:
400
401 PEP 380: Syntax for Delegating to a Subgenerator
402 ================================================
403
404 PEP 380 adds the ``yield from`` expression, allowing a :term:`generator` to
405 delegate
406 part of its operations to another generator. This allows a section of code
407 containing :keyword:`yield` to be factored out and placed in another generator.
408 Additionally, the subgenerator is allowed to return with a value, and the
409 value is made available to the delegating generator.
410
411 While designed primarily for use in delegating to a subgenerator, the ``yield
412 from`` expression actually allows delegation to arbitrary subiterators.
413
414 For simple iterators, ``yield from iterable`` is essentially just a shortened
415 form of ``for item in iterable: yield item``::
416
417 >>> def g(x):
418 ... yield from range(x, 0, -1)
419 ... yield from range(x)
420 ...
421 >>> list(g(5))
422 [5, 4, 3, 2, 1, 0, 1, 2, 3, 4]
423
424 However, unlike an ordinary loop, ``yield from`` allows subgenerators to
425 receive sent and thrown values directly from the calling scope, and
426 return a final value to the outer generator::
427
428 >>> def accumulate():
429 ... tally = 0
430 ... while 1:
431 ... next = yield
432 ... if next is None:
433 ... return tally
434 ... tally += next
435 ...
436 >>> def gather_tallies(tallies):
437 ... while 1:
438 ... tally = yield from accumulate()
439 ... tallies.append(tally)
440 ...
441 >>> tallies = []
442 >>> acc = gather_tallies(tallies)
443 >>> next(acc) # Ensure the accumulator is ready to accept values
444 >>> for i in range(4):
445 ... acc.send(i)
446 ...
447 >>> acc.send(None) # Finish the first tally
448 >>> for i in range(5):
449 ... acc.send(i)
450 ...
451 >>> acc.send(None) # Finish the second tally
452 >>> tallies
453 [6, 10]
454
455 The main principle driving this change is to allow even generators that are
456 designed to be used with the ``send`` and ``throw`` methods to be split into
457 multiple subgenerators as easily as a single large function can be split into
458 multiple subfunctions.
459
460 .. seealso::
461
462 :pep:`380` - Syntax for Delegating to a Subgenerator
463 PEP written by Greg Ewing; implementation by Greg Ewing, integrated into
464 3.3 by Renaud Blanch, Ryan Kelly and Nick Coghlan; documentation by
465 Zbigniew Jędrzejewski-Szmek and Nick Coghlan
466
467
468 PEP 409: Suppressing exception context
469 ======================================
470
471 PEP 409 introduces new syntax that allows the display of the chained
472 exception context to be disabled. This allows cleaner error messages in
473 applications that convert between exception types::
474
475 >>> class D:
476 ... def __init__(self, extra):
477 ... self._extra_attributes = extra
478 ... def __getattr__(self, attr):
479 ... try:
480 ... return self._extra_attributes[attr]
481 ... except KeyError:
482 ... raise AttributeError(attr) from None
483 ...
484 >>> D({}).x
485 Traceback (most recent call last):
486 File "<stdin>", line 1, in <module>
487 File "<stdin>", line 8, in __getattr__
488 AttributeError: x
489
490 Without the ``from None`` suffix to suppress the cause, the original
491 exception would be displayed by default::
492
493 >>> class C:
494 ... def __init__(self, extra):
495 ... self._extra_attributes = extra
496 ... def __getattr__(self, attr):
497 ... try:
498 ... return self._extra_attributes[attr]
499 ... except KeyError:
500 ... raise AttributeError(attr)
501 ...
502 >>> C({}).x
503 Traceback (most recent call last):
504 File "<stdin>", line 6, in __getattr__
505 KeyError: 'x'
506
507 During handling of the above exception, another exception occurred:
508
509 Traceback (most recent call last):
510 File "<stdin>", line 1, in <module>
511 File "<stdin>", line 8, in __getattr__
512 AttributeError: x
513
514 No debugging capability is lost, as the original exception context remains
515 available if needed (for example, if an intervening library has incorrectly
516 suppressed valuable underlying details)::
517
518 >>> try:
519 ... D({}).x
520 ... except AttributeError as exc:
521 ... print(repr(exc.__context__))
522 ...
523 KeyError('x',)
524
525 .. seealso::
526
527 :pep:`409` - Suppressing exception context
528 PEP written by Ethan Furman; implemented by Ethan Furman and Nick
529 Coghlan.
530
531
532 PEP 414: Explicit Unicode literals
533 ======================================
534
535 To ease the transition from Python 2 for Unicode aware Python applications
536 that make heavy use of Unicode literals, Python 3.3 once again supports the
537 "``u``" prefix for string literals. This prefix has no semantic significance
538 in Python 3, it is provided solely to reduce the number of purely mechanical
539 changes in migrating to Python 3, making it easier for developers to focus on
540 the more significant semantic changes (such as the stricter default
541 separation of binary and text data).
542
543 .. seealso::
544
545 :pep:`414` - Explicit Unicode literals
546 PEP written by Armin Ronacher.
547
548
549 PEP 3155: Qualified name for classes and functions
550 ==================================================
551
552 Functions and class objects have a new :attr:`~definition.__qualname__`
553 attribute representing
554 the "path" from the module top-level to their definition. For global functions
555 and classes, this is the same as :attr:`~definition.__name__`.
556 For other functions and classes,
557 it provides better information about where they were actually defined, and
558 how they might be accessible from the global scope.
559
560 Example with (non-bound) methods::
561
562 >>> class C:
563 ... def meth(self):
564 ... pass
565 ...
566 >>> C.meth.__name__
567 'meth'
568 >>> C.meth.__qualname__
569 'C.meth'
570
571 Example with nested classes::
572
573 >>> class C:
574 ... class D:
575 ... def meth(self):
576 ... pass
577 ...
578 >>> C.D.__name__
579 'D'
580 >>> C.D.__qualname__
581 'C.D'
582 >>> C.D.meth.__name__
583 'meth'
584 >>> C.D.meth.__qualname__
585 'C.D.meth'
586
587 Example with nested functions::
588
589 >>> def outer():
590 ... def inner():
591 ... pass
592 ... return inner
593 ...
594 >>> outer().__name__
595 'inner'
596 >>> outer().__qualname__
597 'outer.<locals>.inner'
598
599 The string representation of those objects is also changed to include the
600 new, more precise information::
601
602 >>> str(C.D)
603 "<class '__main__.C.D'>"
604 >>> str(C.D.meth)
605 '<function C.D.meth at 0x7f46b9fe31e0>'
606
607 .. seealso::
608
609 :pep:`3155` - Qualified name for classes and functions
610 PEP written and implemented by Antoine Pitrou.
611
612
613 .. _pep-412:
614
615 PEP 412: Key-Sharing Dictionary
616 ===============================
617
618 Dictionaries used for the storage of objects' attributes are now able to
619 share part of their internal storage between each other (namely, the part
620 which stores the keys and their respective hashes). This reduces the memory
621 consumption of programs creating many instances of non-builtin types.
622
623 .. seealso::
624
625 :pep:`412` - Key-Sharing Dictionary
626 PEP written and implemented by Mark Shannon.
627
628
629 PEP 362: Function Signature Object
630 ==================================
631
632 A new function :func:`inspect.signature` makes introspection of python
633 callables easy and straightforward. A broad range of callables is supported:
634 python functions, decorated or not, classes, and :func:`functools.partial`
635 objects. New classes :class:`inspect.Signature`, :class:`inspect.Parameter`
636 and :class:`inspect.BoundArguments` hold information about the call signatures,
637 such as, annotations, default values, parameters kinds, and bound arguments,
638 which considerably simplifies writing decorators and any code that validates
639 or amends calling signatures or arguments.
640
641 .. seealso::
642
643 :pep:`362`: - Function Signature Object
644 PEP written by Brett Cannon, Yury Selivanov, Larry Hastings, Jiwon Seo;
645 implemented by Yury Selivanov.
646
647
648 PEP 421: Adding sys.implementation
649 ==================================
650
651 A new attribute on the :mod:`sys` module exposes details specific to the
652 implementation of the currently running interpreter. The initial set of
653 attributes on :data:`sys.implementation` are ``name``, ``version``,
654 ``hexversion``, and ``cache_tag``.
655
656 The intention of ``sys.implementation`` is to consolidate into one namespace
657 the implementation-specific data used by the standard library. This allows
658 different Python implementations to share a single standard library code base
659 much more easily. In its initial state, ``sys.implementation`` holds only a
660 small portion of the implementation-specific data. Over time that ratio will
661 shift in order to make the standard library more portable.
662
663 One example of improved standard library portability is ``cache_tag``. As of
664 Python 3.3, ``sys.implementation.cache_tag`` is used by :mod:`importlib` to
665 support :pep:`3147` compliance. Any Python implementation that uses
666 ``importlib`` for its built-in import system may use ``cache_tag`` to control
667 the caching behavior for modules.
668
669 SimpleNamespace
670 ---------------
671
672 The implementation of ``sys.implementation`` also introduces a new type to
673 Python: :class:`types.SimpleNamespace`. In contrast to a mapping-based
674 namespace, like :class:`dict`, ``SimpleNamespace`` is attribute-based, like
675 :class:`object`. However, unlike ``object``, ``SimpleNamespace`` instances
676 are writable. This means that you can add, remove, and modify the namespace
677 through normal attribute access.
678
679 .. seealso::
680
681 :pep:`421` - Adding sys.implementation
682 PEP written and implemented by Eric Snow.
683
684
685 .. _importlib:
686
687 Using importlib as the Implementation of Import
688 ===============================================
689 :issue:`2377` - Replace __import__ w/ importlib.__import__
690 :issue:`13959` - Re-implement parts of :mod:`!imp` in pure Python
691 :issue:`14605` - Make import machinery explicit
692 :issue:`14646` - Require loaders set __loader__ and __package__
693
694 The :func:`__import__` function is now powered by :func:`importlib.__import__`.
695 This work leads to the completion of "phase 2" of :pep:`302`. There are
696 multiple benefits to this change. First, it has allowed for more of the
697 machinery powering import to be exposed instead of being implicit and hidden
698 within the C code. It also provides a single implementation for all Python VMs
699 supporting Python 3.3 to use, helping to end any VM-specific deviations in
700 import semantics. And finally it eases the maintenance of import, allowing for
701 future growth to occur.
702
703 For the common user, there should be no visible change in semantics. For
704 those whose code currently manipulates import or calls import
705 programmatically, the code changes that might possibly be required are covered
706 in the `Porting Python code`_ section of this document.
707
708 New APIs
709 --------
710 One of the large benefits of this work is the exposure of what goes into
711 making the import statement work. That means the various importers that were
712 once implicit are now fully exposed as part of the :mod:`importlib` package.
713
714 The abstract base classes defined in :mod:`importlib.abc` have been expanded
715 to properly delineate between :term:`meta path finders <meta path finder>`
716 and :term:`path entry finders <path entry finder>` by introducing
717 :class:`importlib.abc.MetaPathFinder` and
718 :class:`importlib.abc.PathEntryFinder`, respectively. The old ABC of
719 :class:`!importlib.abc.Finder` is now only provided for backwards-compatibility
720 and does not enforce any method requirements.
721
722 In terms of finders, :class:`importlib.machinery.FileFinder` exposes the
723 mechanism used to search for source and bytecode files of a module. Previously
724 this class was an implicit member of :data:`sys.path_hooks`.
725
726 For loaders, the new abstract base class :class:`importlib.abc.FileLoader` helps
727 write a loader that uses the file system as the storage mechanism for a module's
728 code. The loader for source files
729 (:class:`importlib.machinery.SourceFileLoader`), sourceless bytecode files
730 (:class:`importlib.machinery.SourcelessFileLoader`), and extension modules
731 (:class:`importlib.machinery.ExtensionFileLoader`) are now available for
732 direct use.
733
734 :exc:`ImportError` now has ``name`` and ``path`` attributes which are set when
735 there is relevant data to provide. The message for failed imports will also
736 provide the full name of the module now instead of just the tail end of the
737 module's name.
738
739 The :func:`importlib.invalidate_caches` function will now call the method with
740 the same name on all finders cached in :data:`sys.path_importer_cache` to help
741 clean up any stored state as necessary.
742
743 Visible Changes
744 ---------------
745
746 For potential required changes to code, see the `Porting Python code`_
747 section.
748
749 Beyond the expanse of what :mod:`importlib` now exposes, there are other
750 visible changes to import. The biggest is that :data:`sys.meta_path` and
751 :data:`sys.path_hooks` now store all of the meta path finders and path entry
752 hooks used by import. Previously the finders were implicit and hidden within
753 the C code of import instead of being directly exposed. This means that one can
754 now easily remove or change the order of the various finders to fit one's needs.
755
756 Another change is that all modules have a ``__loader__`` attribute, storing the
757 loader used to create the module. :pep:`302` has been updated to make this
758 attribute mandatory for loaders to implement, so in the future once 3rd-party
759 loaders have been updated people will be able to rely on the existence of the
760 attribute. Until such time, though, import is setting the module post-load.
761
762 Loaders are also now expected to set the ``__package__`` attribute from
763 :pep:`366`. Once again, import itself is already setting this on all loaders
764 from :mod:`importlib` and import itself is setting the attribute post-load.
765
766 ``None`` is now inserted into :data:`sys.path_importer_cache` when no finder
767 can be found on :data:`sys.path_hooks`. Since :class:`!imp.NullImporter` is not
768 directly exposed on :data:`sys.path_hooks` it could no longer be relied upon to
769 always be available to use as a value representing no finder found.
770
771 All other changes relate to semantic changes which should be taken into
772 consideration when updating code for Python 3.3, and thus should be read about
773 in the `Porting Python code`_ section of this document.
774
775 (Implementation by Brett Cannon)
776
777
778 Other Language Changes
779 ======================
780
781 Some smaller changes made to the core Python language are:
782
783 * Added support for Unicode name aliases and named sequences.
784 Both :func:`unicodedata.lookup` and ``'\N{...}'`` now resolve name aliases,
785 and :func:`unicodedata.lookup` resolves named sequences too.
786
787 (Contributed by Ezio Melotti in :issue:`12753`.)
788
789 * Unicode database updated to UCD version 6.1.0
790
791 * Equality comparisons on :func:`range` objects now return a result reflecting
792 the equality of the underlying sequences generated by those range objects.
793 (:issue:`13201`)
794
795 * The ``count()``, ``find()``, ``rfind()``, ``index()`` and ``rindex()``
796 methods of :class:`bytes` and :class:`bytearray` objects now accept an
797 integer between 0 and 255 as their first argument.
798
799 (Contributed by Petri Lehtinen in :issue:`12170`.)
800
801 * The ``rjust()``, ``ljust()``, and ``center()`` methods of :class:`bytes`
802 and :class:`bytearray` now accept a :class:`bytearray` for the ``fill``
803 argument. (Contributed by Petri Lehtinen in :issue:`12380`.)
804
805 * New methods have been added to :class:`list` and :class:`bytearray`:
806 ``copy()`` and ``clear()`` (:issue:`10516`). Consequently,
807 :class:`~collections.abc.MutableSequence` now also defines a
808 :meth:`!clear` method (:issue:`11388`).
809
810 * Raw bytes literals can now be written ``rb"..."`` as well as ``br"..."``.
811
812 (Contributed by Antoine Pitrou in :issue:`13748`.)
813
814 * :meth:`dict.setdefault` now does only one lookup for the given key, making
815 it atomic when used with built-in types.
816
817 (Contributed by Filip Gruszczyński in :issue:`13521`.)
818
819 * The error messages produced when a function call does not match the function
820 signature have been significantly improved.
821
822 (Contributed by Benjamin Peterson.)
823
824
825 A Finer-Grained Import Lock
826 ===========================
827
828 Previous versions of CPython have always relied on a global import lock.
829 This led to unexpected annoyances, such as deadlocks when importing a module
830 would trigger code execution in a different thread as a side-effect.
831 Clumsy workarounds were sometimes employed, such as the
832 :c:func:`PyImport_ImportModuleNoBlock` C API function.
833
834 In Python 3.3, importing a module takes a per-module lock. This correctly
835 serializes importation of a given module from multiple threads (preventing
836 the exposure of incompletely initialized modules), while eliminating the
837 aforementioned annoyances.
838
839 (Contributed by Antoine Pitrou in :issue:`9260`.)
840
841
842 Builtin functions and types
843 ===========================
844
845 * :func:`open` gets a new *opener* parameter: the underlying file descriptor
846 for the file object is then obtained by calling *opener* with (*file*,
847 *flags*). It can be used to use custom flags like :const:`os.O_CLOEXEC` for
848 example. The ``'x'`` mode was added: open for exclusive creation, failing if
849 the file already exists.
850 * :func:`print`: added the *flush* keyword argument. If the *flush* keyword
851 argument is true, the stream is forcibly flushed.
852 * :func:`hash`: hash randomization is enabled by default, see
853 :meth:`object.__hash__` and :envvar:`PYTHONHASHSEED`.
854 * The :class:`str` type gets a new :meth:`~str.casefold` method: return a
855 casefolded copy of the string, casefolded strings may be used for caseless
856 matching. For example, ``'ß'.casefold()`` returns ``'ss'``.
857 * The sequence documentation has been substantially rewritten to better
858 explain the binary/text sequence distinction and to provide specific
859 documentation sections for the individual builtin sequence types
860 (:issue:`4966`).
861
862
863 New Modules
864 ===========
865
866 faulthandler
867 ------------
868
869 This new debug module :mod:`faulthandler` contains functions to dump Python tracebacks explicitly,
870 on a fault (a crash like a segmentation fault), after a timeout, or on a user
871 signal. Call :func:`faulthandler.enable` to install fault handlers for the
872 :const:`~signal.SIGSEGV`, :const:`~signal.SIGFPE`, :const:`~signal.SIGABRT`,
873 :const:`~signal.SIGBUS`, and :const:`~signal.SIGILL` signals.
874 You can also enable them at startup by setting the :envvar:`PYTHONFAULTHANDLER`
875 environment variable or by using :option:`-X` ``faulthandler`` command line option.
876
877 Example of a segmentation fault on Linux:
878
879 .. code-block:: shell-session
880
881 $ python -q -X faulthandler
882 >>> import ctypes
883 >>> ctypes.string_at(0)
884 Fatal Python error: Segmentation fault
885
886 Current thread 0x00007fb899f39700:
887 File "/home/python/cpython/Lib/ctypes/__init__.py", line 486 in string_at
888 File "<stdin>", line 1 in <module>
889 Segmentation fault
890
891
892 ipaddress
893 ---------
894
895 The new :mod:`ipaddress` module provides tools for creating and manipulating
896 objects representing IPv4 and IPv6 addresses, networks and interfaces (i.e.
897 an IP address associated with a specific IP subnet).
898
899 (Contributed by Google and Peter Moody in :pep:`3144`.)
900
901 lzma
902 ----
903
904 The newly added :mod:`lzma` module provides data compression and decompression
905 using the LZMA algorithm, including support for the ``.xz`` and ``.lzma``
906 file formats.
907
908 (Contributed by Nadeem Vawda and Per Øyvind Karlsen in :issue:`6715`.)
909
910
911 Improved Modules
912 ================
913
914 abc
915 ---
916
917 Improved support for abstract base classes containing descriptors composed with
918 abstract methods. The recommended approach to declaring abstract descriptors is
919 now to provide :attr:`!__isabstractmethod__` as a dynamically updated
920 property. The built-in descriptors have been updated accordingly.
921
922 * :class:`abc.abstractproperty` has been deprecated, use :class:`property`
923 with :func:`abc.abstractmethod` instead.
924 * :class:`abc.abstractclassmethod` has been deprecated, use
925 :class:`classmethod` with :func:`abc.abstractmethod` instead.
926 * :class:`abc.abstractstaticmethod` has been deprecated, use
927 :class:`staticmethod` with :func:`abc.abstractmethod` instead.
928
929 (Contributed by Darren Dale in :issue:`11610`.)
930
931 :meth:`abc.ABCMeta.register` now returns the registered subclass, which means
932 it can now be used as a class decorator (:issue:`10868`).
933
934
935 array
936 -----
937
938 The :mod:`array` module supports the :c:expr:`long long` type using ``q`` and
939 ``Q`` type codes.
940
941 (Contributed by Oren Tirosh and Hirokazu Yamamoto in :issue:`1172711`.)
942
943
944 base64
945 ------
946
947 ASCII-only Unicode strings are now accepted by the decoding functions of the
948 :mod:`base64` modern interface. For example, ``base64.b64decode('YWJj')``
949 returns ``b'abc'``. (Contributed by Catalin Iacob in :issue:`13641`.)
950
951
952 binascii
953 --------
954
955 In addition to the binary objects they normally accept, the ``a2b_`` functions
956 now all also accept ASCII-only strings as input. (Contributed by Antoine
957 Pitrou in :issue:`13637`.)
958
959
960 bz2
961 ---
962
963 The :mod:`bz2` module has been rewritten from scratch. In the process, several
964 new features have been added:
965
966 * New :func:`bz2.open` function: open a bzip2-compressed file in binary or
967 text mode.
968
969 * :class:`bz2.BZ2File` can now read from and write to arbitrary file-like
970 objects, by means of its constructor's *fileobj* argument.
971
972 (Contributed by Nadeem Vawda in :issue:`5863`.)
973
974 * :class:`bz2.BZ2File` and :func:`bz2.decompress` can now decompress
975 multi-stream inputs (such as those produced by the :program:`pbzip2` tool).
976 :class:`bz2.BZ2File` can now also be used to create this type of file, using
977 the ``'a'`` (append) mode.
978
979 (Contributed by Nir Aides in :issue:`1625`.)
980
981 * :class:`bz2.BZ2File` now implements all of the :class:`io.BufferedIOBase` API,
982 except for the :meth:`!detach` and :meth:`!truncate` methods.
983
984
985 codecs
986 ------
987
988 The :mod:`~encodings.mbcs` codec has been rewritten to handle correctly
989 ``replace`` and ``ignore`` error handlers on all Windows versions. The
990 :mod:`~encodings.mbcs` codec now supports all error handlers, instead of only
991 ``replace`` to encode and ``ignore`` to decode.
992
993 A new Windows-only codec has been added: ``cp65001`` (:issue:`13216`). It is the
994 Windows code page 65001 (Windows UTF-8, ``CP_UTF8``). For example, it is used
995 by ``sys.stdout`` if the console output code page is set to cp65001 (e.g., using
996 ``chcp 65001`` command).
997
998 Multibyte CJK decoders now resynchronize faster. They only ignore the first
999 byte of an invalid byte sequence. For example, ``b'\xff\n'.decode('gb2312',
1000 'replace')`` now returns a ``\n`` after the replacement character.
1001
1002 (:issue:`12016`)
1003
1004 Incremental CJK codec encoders are no longer reset at each call to their
1005 encode() methods. For example::
1006
1007 >>> import codecs
1008 >>> encoder = codecs.getincrementalencoder('hz')('strict')
1009 >>> b''.join(encoder.encode(x) for x in '\u52ff\u65bd\u65bc\u4eba\u3002 Bye.')
1010 b'~{NpJ)l6HK!#~} Bye.'
1011
1012 This example gives ``b'~{Np~}~{J)~}~{l6~}~{HK~}~{!#~} Bye.'`` with older Python
1013 versions.
1014
1015 (:issue:`12100`)
1016
1017 The ``unicode_internal`` codec has been deprecated.
1018
1019
1020 collections
1021 -----------
1022
1023 Addition of a new :class:`~collections.ChainMap` class to allow treating a
1024 number of mappings as a single unit. (Written by Raymond Hettinger for
1025 :issue:`11089`, made public in :issue:`11297`.)
1026
1027 The abstract base classes have been moved in a new :mod:`collections.abc`
1028 module, to better differentiate between the abstract and the concrete
1029 collections classes. Aliases for ABCs are still present in the
1030 :mod:`collections` module to preserve existing imports. (:issue:`11085`)
1031
1032 .. XXX addition of __slots__ to ABCs not recorded here: internal detail
1033
1034 The :class:`~collections.Counter` class now supports the unary ``+`` and ``-``
1035 operators, as well as the in-place operators ``+=``, ``-=``, ``|=``, and
1036 ``&=``. (Contributed by Raymond Hettinger in :issue:`13121`.)
1037
1038
1039 contextlib
1040 ----------
1041
1042 :class:`~contextlib.ExitStack` now provides a solid foundation for
1043 programmatic manipulation of context managers and similar cleanup
1044 functionality. Unlike the previous ``contextlib.nested`` API (which was
1045 deprecated and removed), the new API is designed to work correctly
1046 regardless of whether context managers acquire their resources in
1047 their ``__init__`` method (for example, file objects) or in their
1048 ``__enter__`` method (for example, synchronisation objects from the
1049 :mod:`threading` module).
1050
1051 (:issue:`13585`)
1052
1053
1054 crypt
1055 -----
1056
1057 Addition of salt and modular crypt format (hashing method) and the :func:`!mksalt`
1058 function to the :mod:`!crypt` module.
1059
1060 (:issue:`10924`)
1061
1062 curses
1063 ------
1064
1065 * If the :mod:`curses` module is linked to the ncursesw library, use Unicode
1066 functions when Unicode strings or characters are passed (e.g.
1067 :c:func:`!waddwstr`), and bytes functions otherwise (e.g. :c:func:`!waddstr`).
1068 * Use the locale encoding instead of ``utf-8`` to encode Unicode strings.
1069 * :class:`curses.window` has a new :attr:`curses.window.encoding` attribute.
1070 * The :class:`curses.window` class has a new :meth:`~curses.window.get_wch`
1071 method to get a wide character
1072 * The :mod:`curses` module has a new :meth:`~curses.unget_wch` function to
1073 push a wide character so the next :meth:`~curses.window.get_wch` will return
1074 it
1075
1076 (Contributed by Iñigo Serna in :issue:`6755`.)
1077
1078 datetime
1079 --------
1080
1081 * Equality comparisons between naive and aware :class:`~datetime.datetime`
1082 instances now return :const:`False` instead of raising :exc:`TypeError`
1083 (:issue:`15006`).
1084 * New :meth:`datetime.datetime.timestamp` method: Return POSIX timestamp
1085 corresponding to the :class:`~datetime.datetime` instance.
1086 * The :meth:`datetime.datetime.strftime` method supports formatting years
1087 older than 1000.
1088 * The :meth:`datetime.datetime.astimezone` method can now be
1089 called without arguments to convert datetime instance to the system
1090 timezone.
1091
1092
1093 .. _new-decimal:
1094
1095 decimal
1096 -------
1097
1098 :issue:`7652` - integrate fast native decimal arithmetic.
1099 C-module and libmpdec written by Stefan Krah.
1100
1101 The new C version of the decimal module integrates the high speed libmpdec
1102 library for arbitrary precision correctly rounded decimal floating-point
1103 arithmetic. libmpdec conforms to IBM's General Decimal Arithmetic Specification.
1104
1105 Performance gains range from 10x for database applications to 100x for
1106 numerically intensive applications. These numbers are expected gains
1107 for standard precisions used in decimal floating-point arithmetic. Since
1108 the precision is user configurable, the exact figures may vary. For example,
1109 in integer bignum arithmetic the differences can be significantly higher.
1110
1111 The following table is meant as an illustration. Benchmarks are available
1112 at https://www.bytereef.org/mpdecimal/quickstart.html.
1113
1114 +---------+-------------+--------------+-------------+
1115 | | decimal.py | _decimal | speedup |
1116 +=========+=============+==============+=============+
1117 | pi | 42.02s | 0.345s | 120x |
1118 +---------+-------------+--------------+-------------+
1119 | telco | 172.19s | 5.68s | 30x |
1120 +---------+-------------+--------------+-------------+
1121 | psycopg | 3.57s | 0.29s | 12x |
1122 +---------+-------------+--------------+-------------+
1123
1124 Features
1125 ~~~~~~~~
1126
1127 * The :exc:`~decimal.FloatOperation` signal optionally enables stricter
1128 semantics for mixing floats and Decimals.
1129
1130 * If Python is compiled without threads, the C version automatically
1131 disables the expensive thread local context machinery. In this case,
1132 the variable :const:`~decimal.HAVE_THREADS` is set to ``False``.
1133
1134 API changes
1135 ~~~~~~~~~~~
1136
1137 * The C module has the following context limits, depending on the machine
1138 architecture:
1139
1140 +----------------------------+----------------+-------------------------+
1141 | | 32-bit | 64-bit |
1142 +============================+================+=========================+
1143 | :const:`~decimal.MAX_PREC` | ``425000000`` | ``999999999999999999`` |
1144 +----------------------------+----------------+-------------------------+
1145 | :const:`~decimal.MAX_EMAX` | ``425000000`` | ``999999999999999999`` |
1146 +----------------------------+----------------+-------------------------+
1147 | :const:`~decimal.MIN_EMIN` | ``-425000000`` | ``-999999999999999999`` |
1148 +----------------------------+----------------+-------------------------+
1149
1150 * In the context templates (:const:`~decimal.DefaultContext`,
1151 :const:`~decimal.BasicContext` and :const:`~decimal.ExtendedContext`)
1152 the magnitude of :attr:`~decimal.Context.Emax` and
1153 :attr:`~decimal.Context.Emin` has changed to ``999999``.
1154
1155 * The :class:`~decimal.Decimal` constructor in decimal.py does not observe
1156 the context limits and converts values with arbitrary exponents or precision
1157 exactly. Since the C version has internal limits, the following scheme is
1158 used: If possible, values are converted exactly, otherwise
1159 :exc:`~decimal.InvalidOperation` is raised and the result is NaN. In the
1160 latter case it is always possible to use :meth:`~decimal.Context.create_decimal`
1161 in order to obtain a rounded or inexact value.
1162
1163
1164 * The power function in decimal.py is always correctly rounded. In the
1165 C version, it is defined in terms of the correctly rounded
1166 :meth:`~decimal.Decimal.exp` and :meth:`~decimal.Decimal.ln` functions,
1167 but the final result is only "almost always correctly rounded".
1168
1169
1170 * In the C version, the context dictionary containing the signals is a
1171 :class:`~collections.abc.MutableMapping`. For speed reasons,
1172 :attr:`~decimal.Context.flags` and :attr:`~decimal.Context.traps` always
1173 refer to the same :class:`~collections.abc.MutableMapping` that the context
1174 was initialized with. If a new signal dictionary is assigned,
1175 :attr:`~decimal.Context.flags` and :attr:`~decimal.Context.traps`
1176 are updated with the new values, but they do not reference the RHS
1177 dictionary.
1178
1179
1180 * Pickling a :class:`~decimal.Context` produces a different output in order
1181 to have a common interchange format for the Python and C versions.
1182
1183
1184 * The order of arguments in the :class:`~decimal.Context` constructor has been
1185 changed to match the order displayed by :func:`repr`.
1186
1187
1188 * The ``watchexp`` parameter in the :meth:`~decimal.Decimal.quantize` method
1189 is deprecated.
1190
1191
1192 .. _new-email:
1193
1194 email
1195 -----
1196
1197 Policy Framework
1198 ~~~~~~~~~~~~~~~~
1199
1200 The email package now has a :mod:`~email.policy` framework. A
1201 :class:`~email.policy.Policy` is an object with several methods and properties
1202 that control how the email package behaves. The primary policy for Python 3.3
1203 is the :class:`~email.policy.Compat32` policy, which provides backward
1204 compatibility with the email package in Python 3.2. A ``policy`` can be
1205 specified when an email message is parsed by a :mod:`~email.parser`, or when a
1206 :class:`~email.message.Message` object is created, or when an email is
1207 serialized using a :mod:`~email.generator`. Unless overridden, a policy passed
1208 to a ``parser`` is inherited by all the ``Message`` object and sub-objects
1209 created by the ``parser``. By default a ``generator`` will use the policy of
1210 the ``Message`` object it is serializing. The default policy is
1211 :data:`~email.policy.compat32`.
1212
1213 The minimum set of controls implemented by all ``policy`` objects are:
1214
1215 .. tabularcolumns:: |l|L|
1216
1217 =============== =======================================================
1218 max_line_length The maximum length, excluding the linesep character(s),
1219 individual lines may have when a ``Message`` is
1220 serialized. Defaults to 78.
1221
1222 linesep The character used to separate individual lines when a
1223 ``Message`` is serialized. Defaults to ``\n``.
1224
1225 cte_type ``7bit`` or ``8bit``. ``8bit`` applies only to a
1226 ``Bytes`` ``generator``, and means that non-ASCII may
1227 be used where allowed by the protocol (or where it
1228 exists in the original input).
1229
1230 raise_on_defect Causes a ``parser`` to raise error when defects are
1231 encountered instead of adding them to the ``Message``
1232 object's ``defects`` list.
1233 =============== =======================================================
1234
1235 A new policy instance, with new settings, is created using the
1236 :meth:`~email.policy.Policy.clone` method of policy objects. ``clone`` takes
1237 any of the above controls as keyword arguments. Any control not specified in
1238 the call retains its default value. Thus you can create a policy that uses
1239 ``\r\n`` linesep characters like this::
1240
1241 mypolicy = compat32.clone(linesep='\r\n')
1242
1243 Policies can be used to make the generation of messages in the format needed by
1244 your application simpler. Instead of having to remember to specify
1245 ``linesep='\r\n'`` in all the places you call a ``generator``, you can specify
1246 it once, when you set the policy used by the ``parser`` or the ``Message``,
1247 whichever your program uses to create ``Message`` objects. On the other hand,
1248 if you need to generate messages in multiple forms, you can still specify the
1249 parameters in the appropriate ``generator`` call. Or you can have custom
1250 policy instances for your different cases, and pass those in when you create
1251 the ``generator``.
1252
1253
1254 Provisional Policy with New Header API
1255 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1256
1257 While the policy framework is worthwhile all by itself, the main motivation for
1258 introducing it is to allow the creation of new policies that implement new
1259 features for the email package in a way that maintains backward compatibility
1260 for those who do not use the new policies. Because the new policies introduce a
1261 new API, we are releasing them in Python 3.3 as a :term:`provisional policy
1262 <provisional package>`. Backwards incompatible changes (up to and including
1263 removal of the code) may occur if deemed necessary by the core developers.
1264
1265 The new policies are instances of :class:`~email.policy.EmailPolicy`,
1266 and add the following additional controls:
1267
1268 .. tabularcolumns:: |l|L|
1269
1270 =============== =======================================================
1271 refold_source Controls whether or not headers parsed by a
1272 :mod:`~email.parser` are refolded by the
1273 :mod:`~email.generator`. It can be ``none``, ``long``,
1274 or ``all``. The default is ``long``, which means that
1275 source headers with a line longer than
1276 ``max_line_length`` get refolded. ``none`` means no
1277 line get refolded, and ``all`` means that all lines
1278 get refolded.
1279
1280 header_factory A callable that take a ``name`` and ``value`` and
1281 produces a custom header object.
1282 =============== =======================================================
1283
1284 The ``header_factory`` is the key to the new features provided by the new
1285 policies. When one of the new policies is used, any header retrieved from
1286 a ``Message`` object is an object produced by the ``header_factory``, and any
1287 time you set a header on a ``Message`` it becomes an object produced by
1288 ``header_factory``. All such header objects have a ``name`` attribute equal
1289 to the header name. Address and Date headers have additional attributes
1290 that give you access to the parsed data of the header. This means you can now
1291 do things like this::
1292
1293 >>> m = Message(policy=SMTP)
1294 >>> m['To'] = 'Éric <[email protected]>'
1295 >>> m['to']
1296 'Éric <[email protected]>'
1297 >>> m['to'].addresses
1298 (Address(display_name='Éric', username='foo', domain='example.com'),)
1299 >>> m['to'].addresses[0].username
1300 'foo'
1301 >>> m['to'].addresses[0].display_name
1302 'Éric'
1303 >>> m['Date'] = email.utils.localtime()
1304 >>> m['Date'].datetime
1305 datetime.datetime(2012, 5, 25, 21, 39, 24, 465484, tzinfo=datetime.timezone(datetime.timedelta(-1, 72000), 'EDT'))
1306 >>> m['Date']
1307 'Fri, 25 May 2012 21:44:27 -0400'
1308 >>> print(m)
1309 To: =?utf-8?q?=C3=89ric?= <[email protected]>
1310 Date: Fri, 25 May 2012 21:44:27 -0400
1311
1312 You will note that the unicode display name is automatically encoded as
1313 ``utf-8`` when the message is serialized, but that when the header is accessed
1314 directly, you get the unicode version. This eliminates any need to deal with
1315 the :mod:`email.header` :meth:`~email.header.decode_header` or
1316 :meth:`~email.header.make_header` functions.
1317
1318 You can also create addresses from parts::
1319
1320 >>> m['cc'] = [Group('pals', [Address('Bob', 'bob', 'example.com'),
1321 ... Address('Sally', 'sally', 'example.com')]),
1322 ... Address('Bonzo', addr_spec='[email protected]')]
1323 >>> print(m)
1324 To: =?utf-8?q?=C3=89ric?= <[email protected]>
1325 Date: Fri, 25 May 2012 21:44:27 -0400
1326 cc: pals: Bob <[email protected]>, Sally <[email protected]>;, Bonzo <[email protected]>
1327
1328 Decoding to unicode is done automatically::
1329
1330 >>> m2 = message_from_string(str(m))
1331 >>> m2['to']
1332 'Éric <[email protected]>'
1333
1334 When you parse a message, you can use the ``addresses`` and ``groups``
1335 attributes of the header objects to access the groups and individual
1336 addresses::
1337
1338 >>> m2['cc'].addresses
1339 (Address(display_name='Bob', username='bob', domain='example.com'), Address(display_name='Sally', username='sally', domain='example.com'), Address(display_name='Bonzo', username='bonz', domain='laugh.com'))
1340 >>> m2['cc'].groups
1341 (Group(display_name='pals', addresses=(Address(display_name='Bob', username='bob', domain='example.com'), Address(display_name='Sally', username='sally', domain='example.com')), Group(display_name=None, addresses=(Address(display_name='Bonzo', username='bonz', domain='laugh.com'),))
1342
1343 In summary, if you use one of the new policies, header manipulation works the
1344 way it ought to: your application works with unicode strings, and the email
1345 package transparently encodes and decodes the unicode to and from the RFC
1346 standard Content Transfer Encodings.
1347
1348 Other API Changes
1349 ~~~~~~~~~~~~~~~~~
1350
1351 New :class:`~email.parser.BytesHeaderParser`, added to the :mod:`~email.parser`
1352 module to complement :class:`~email.parser.HeaderParser` and complete the Bytes
1353 API.
1354
1355 New utility functions:
1356
1357 * :func:`~email.utils.format_datetime`: given a :class:`~datetime.datetime`,
1358 produce a string formatted for use in an email header.
1359
1360 * :func:`~email.utils.parsedate_to_datetime`: given a date string from
1361 an email header, convert it into an aware :class:`~datetime.datetime`,
1362 or a naive :class:`~datetime.datetime` if the offset is ``-0000``.
1363
1364 * :func:`~email.utils.localtime`: With no argument, returns the
1365 current local time as an aware :class:`~datetime.datetime` using the local
1366 :class:`~datetime.timezone`. Given an aware :class:`~datetime.datetime`,
1367 converts it into an aware :class:`~datetime.datetime` using the
1368 local :class:`~datetime.timezone`.
1369
1370
1371 ftplib
1372 ------
1373
1374 * :class:`ftplib.FTP` now accepts a ``source_address`` keyword argument to
1375 specify the ``(host, port)`` to use as the source address in the bind call
1376 when creating the outgoing socket. (Contributed by Giampaolo Rodolà
1377 in :issue:`8594`.)
1378
1379 * The :class:`~ftplib.FTP_TLS` class now provides a new
1380 :func:`~ftplib.FTP_TLS.ccc` function to revert control channel back to
1381 plaintext. This can be useful to take advantage of firewalls that know how
1382 to handle NAT with non-secure FTP without opening fixed ports. (Contributed
1383 by Giampaolo Rodolà in :issue:`12139`.)
1384
1385 * Added :meth:`ftplib.FTP.mlsd` method which provides a parsable directory
1386 listing format and deprecates :meth:`ftplib.FTP.nlst` and
1387 :meth:`ftplib.FTP.dir`. (Contributed by Giampaolo Rodolà in :issue:`11072`.)
1388
1389
1390 functools
1391 ---------
1392
1393 The :func:`functools.lru_cache` decorator now accepts a ``typed`` keyword
1394 argument (that defaults to ``False`` to ensure that it caches values of
1395 different types that compare equal in separate cache slots. (Contributed
1396 by Raymond Hettinger in :issue:`13227`.)
1397
1398
1399 gc
1400 --
1401
1402 It is now possible to register callbacks invoked by the garbage collector
1403 before and after collection using the new :data:`~gc.callbacks` list.
1404
1405
1406 hmac
1407 ----
1408
1409 A new :func:`~hmac.compare_digest` function has been added to prevent side
1410 channel attacks on digests through timing analysis. (Contributed by Nick
1411 Coghlan and Christian Heimes in :issue:`15061`.)
1412
1413
1414 http
1415 ----
1416
1417 :class:`http.server.BaseHTTPRequestHandler` now buffers the headers and writes
1418 them all at once when :meth:`~http.server.BaseHTTPRequestHandler.end_headers` is
1419 called. A new method :meth:`~http.server.BaseHTTPRequestHandler.flush_headers`
1420 can be used to directly manage when the accumulated headers are sent.
1421 (Contributed by Andrew Schaaf in :issue:`3709`.)
1422
1423 :class:`http.server` now produces valid ``HTML 4.01 strict`` output.
1424 (Contributed by Ezio Melotti in :issue:`13295`.)
1425
1426 :class:`http.client.HTTPResponse` now has a
1427 :meth:`~http.client.HTTPResponse.readinto` method, which means it can be used
1428 as an :class:`io.RawIOBase` class. (Contributed by John Kuhn in
1429 :issue:`13464`.)
1430
1431
1432 html
1433 ----
1434
1435 :class:`html.parser.HTMLParser` is now able to parse broken markup without
1436 raising errors, therefore the *strict* argument of the constructor and the
1437 :exc:`!HTMLParseError` exception are now deprecated.
1438 The ability to parse broken markup is the result of a number of bug fixes that
1439 are also available on the latest bug fix releases of Python 2.7/3.2.
1440 (Contributed by Ezio Melotti in :issue:`15114`, and :issue:`14538`,
1441 :issue:`13993`, :issue:`13960`, :issue:`13358`, :issue:`1745761`,
1442 :issue:`755670`, :issue:`13357`, :issue:`12629`, :issue:`1200313`,
1443 :issue:`670664`, :issue:`13273`, :issue:`12888`, :issue:`7311`.)
1444
1445 A new :data:`~html.entities.html5` dictionary that maps HTML5 named character
1446 references to the equivalent Unicode character(s) (e.g. ``html5['gt;'] ==
1447 '>'``) has been added to the :mod:`html.entities` module. The dictionary is
1448 now also used by :class:`~html.parser.HTMLParser`. (Contributed by Ezio
1449 Melotti in :issue:`11113` and :issue:`15156`.)
1450
1451
1452 imaplib
1453 -------
1454
1455 The :class:`~imaplib.IMAP4_SSL` constructor now accepts an SSLContext
1456 parameter to control parameters of the secure channel.
1457
1458 (Contributed by Sijin Joseph in :issue:`8808`.)
1459
1460
1461 inspect
1462 -------
1463
1464 A new :func:`~inspect.getclosurevars` function has been added. This function
1465 reports the current binding of all names referenced from the function body and
1466 where those names were resolved, making it easier to verify correct internal
1467 state when testing code that relies on stateful closures.
1468
1469 (Contributed by Meador Inge and Nick Coghlan in :issue:`13062`.)
1470
1471 A new :func:`~inspect.getgeneratorlocals` function has been added. This
1472 function reports the current binding of local variables in the generator's
1473 stack frame, making it easier to verify correct internal state when testing
1474 generators.
1475
1476 (Contributed by Meador Inge in :issue:`15153`.)
1477
1478 io
1479 --
1480
1481 The :func:`~io.open` function has a new ``'x'`` mode that can be used to
1482 exclusively create a new file, and raise a :exc:`FileExistsError` if the file
1483 already exists. It is based on the C11 'x' mode to fopen().
1484
1485 (Contributed by David Townshend in :issue:`12760`.)
1486
1487 The constructor of the :class:`~io.TextIOWrapper` class has a new
1488 *write_through* optional argument. If *write_through* is ``True``, calls to
1489 :meth:`!write` are guaranteed not to be buffered: any data
1490 written on the :class:`~io.TextIOWrapper` object is immediately handled to its
1491 underlying binary buffer.
1492
1493
1494 itertools
1495 ---------
1496
1497 :func:`~itertools.accumulate` now takes an optional ``func`` argument for
1498 providing a user-supplied binary function.
1499
1500
1501 logging
1502 -------
1503
1504 The :func:`~logging.basicConfig` function now supports an optional ``handlers``
1505 argument taking an iterable of handlers to be added to the root logger.
1506
1507 A class level attribute :attr:`!append_nul` has
1508 been added to :class:`~logging.handlers.SysLogHandler` to allow control of the
1509 appending of the ``NUL`` (``\000``) byte to syslog records, since for some
1510 daemons it is required while for others it is passed through to the log.
1511
1512
1513
1514 math
1515 ----
1516
1517 The :mod:`math` module has a new function, :func:`~math.log2`, which returns
1518 the base-2 logarithm of *x*.
1519
1520 (Written by Mark Dickinson in :issue:`11888`.)
1521
1522
1523 mmap
1524 ----
1525
1526 The :meth:`~mmap.mmap.read` method is now more compatible with other file-like
1527 objects: if the argument is omitted or specified as ``None``, it returns the
1528 bytes from the current file position to the end of the mapping. (Contributed
1529 by Petri Lehtinen in :issue:`12021`.)
1530
1531
1532 multiprocessing
1533 ---------------
1534
1535 The new :func:`multiprocessing.connection.wait` function allows polling
1536 multiple objects (such as connections, sockets and pipes) with a timeout.
1537 (Contributed by Richard Oudkerk in :issue:`12328`.)
1538
1539 :class:`multiprocessing.connection.Connection` objects can now be transferred
1540 over multiprocessing connections.
1541 (Contributed by Richard Oudkerk in :issue:`4892`.)
1542
1543 :class:`multiprocessing.Process` now accepts a ``daemon`` keyword argument
1544 to override the default behavior of inheriting the ``daemon`` flag from
1545 the parent process (:issue:`6064`).
1546
1547 New attribute :data:`multiprocessing.Process.sentinel` allows a
1548 program to wait on multiple :class:`~multiprocessing.Process` objects at one
1549 time using the appropriate OS primitives (for example, :mod:`select` on
1550 posix systems).
1551
1552 New methods :meth:`multiprocessing.pool.Pool.starmap` and
1553 :meth:`~multiprocessing.pool.Pool.starmap_async` provide
1554 :func:`itertools.starmap` equivalents to the existing
1555 :meth:`multiprocessing.pool.Pool.map` and
1556 :meth:`~multiprocessing.pool.Pool.map_async` functions. (Contributed by Hynek
1557 Schlawack in :issue:`12708`.)
1558
1559
1560 nntplib
1561 -------
1562
1563 The :class:`!nntplib.NNTP` class now supports the context management protocol to
1564 unconditionally consume :exc:`socket.error` exceptions and to close the NNTP
1565 connection when done::
1566
1567 >>> from nntplib import NNTP
1568 >>> with NNTP('news.gmane.org') as n:
1569 ... n.group('gmane.comp.python.committers')
1570 ...
1571 ('211 1755 1 1755 gmane.comp.python.committers', 1755, 1, 1755, 'gmane.comp.python.committers')
1572 >>>
1573
1574 (Contributed by Giampaolo Rodolà in :issue:`9795`.)
1575
1576
1577 os
1578 --
1579
1580 * The :mod:`os` module has a new :func:`~os.pipe2` function that makes it
1581 possible to create a pipe with :const:`~os.O_CLOEXEC` or
1582 :const:`~os.O_NONBLOCK` flags set atomically. This is especially useful to
1583 avoid race conditions in multi-threaded programs.
1584
1585 * The :mod:`os` module has a new :func:`~os.sendfile` function which provides
1586 an efficient "zero-copy" way for copying data from one file (or socket)
1587 descriptor to another. The phrase "zero-copy" refers to the fact that all of
1588 the copying of data between the two descriptors is done entirely by the
1589 kernel, with no copying of data into userspace buffers. :func:`~os.sendfile`
1590 can be used to efficiently copy data from a file on disk to a network socket,
1591 e.g. for downloading a file.
1592
1593 (Patch submitted by Ross Lagerwall and Giampaolo Rodolà in :issue:`10882`.)
1594
1595 * To avoid race conditions like symlink attacks and issues with temporary
1596 files and directories, it is more reliable (and also faster) to manipulate
1597 file descriptors instead of file names. Python 3.3 enhances existing functions
1598 and introduces new functions to work on file descriptors (:issue:`4761`,
1599 :issue:`10755` and :issue:`14626`).
1600
1601 - The :mod:`os` module has a new :func:`~os.fwalk` function similar to
1602 :func:`~os.walk` except that it also yields file descriptors referring to the
1603 directories visited. This is especially useful to avoid symlink races.
1604
1605 - The following functions get new optional *dir_fd* (:ref:`paths relative to
1606 directory descriptors <dir_fd>`) and/or *follow_symlinks* (:ref:`not
1607 following symlinks <follow_symlinks>`):
1608 :func:`~os.access`, :func:`~os.chflags`, :func:`~os.chmod`, :func:`~os.chown`,
1609 :func:`~os.link`, :func:`~os.lstat`, :func:`~os.mkdir`, :func:`~os.mkfifo`,
1610 :func:`~os.mknod`, :func:`~os.open`, :func:`~os.readlink`, :func:`~os.remove`,
1611 :func:`~os.rename`, :func:`~os.replace`, :func:`~os.rmdir`, :func:`~os.stat`,
1612 :func:`~os.symlink`, :func:`~os.unlink`, :func:`~os.utime`. Platform
1613 support for using these parameters can be checked via the sets
1614 :data:`os.supports_dir_fd` and :data:`os.supports_follow_symlinks`.
1615
1616 - The following functions now support a file descriptor for their path argument:
1617 :func:`~os.chdir`, :func:`~os.chmod`, :func:`~os.chown`,
1618 :func:`~os.execve`, :func:`~os.listdir`, :func:`~os.pathconf`, :func:`~os.path.exists`,
1619 :func:`~os.stat`, :func:`~os.statvfs`, :func:`~os.utime`. Platform support
1620 for this can be checked via the :data:`os.supports_fd` set.
1621
1622 * :func:`~os.access` accepts an ``effective_ids`` keyword argument to turn on
1623 using the effective uid/gid rather than the real uid/gid in the access check.
1624 Platform support for this can be checked via the
1625 :data:`~os.supports_effective_ids` set.
1626
1627 * The :mod:`os` module has two new functions: :func:`~os.getpriority` and
1628 :func:`~os.setpriority`. They can be used to get or set process
1629 niceness/priority in a fashion similar to :func:`os.nice` but extended to all
1630 processes instead of just the current one.
1631
1632 (Patch submitted by Giampaolo Rodolà in :issue:`10784`.)
1633
1634 * The new :func:`os.replace` function allows cross-platform renaming of a
1635 file with overwriting the destination. With :func:`os.rename`, an existing
1636 destination file is overwritten under POSIX, but raises an error under
1637 Windows.
1638 (Contributed by Antoine Pitrou in :issue:`8828`.)
1639
1640 * The stat family of functions (:func:`~os.stat`, :func:`~os.fstat`,
1641 and :func:`~os.lstat`) now support reading a file's timestamps
1642 with nanosecond precision. Symmetrically, :func:`~os.utime`
1643 can now write file timestamps with nanosecond precision. (Contributed by
1644 Larry Hastings in :issue:`14127`.)
1645
1646 * The new :func:`os.get_terminal_size` function queries the size of the
1647 terminal attached to a file descriptor. See also
1648 :func:`shutil.get_terminal_size`.
1649 (Contributed by Zbigniew Jędrzejewski-Szmek in :issue:`13609`.)
1650
1651 .. XXX sort out this mess after beta1
1652
1653 * New functions to support Linux extended attributes (:issue:`12720`):
1654 :func:`~os.getxattr`, :func:`~os.listxattr`, :func:`~os.removexattr`,
1655 :func:`~os.setxattr`.
1656
1657 * New interface to the scheduler. These functions
1658 control how a process is allocated CPU time by the operating system. New
1659 functions:
1660 :func:`~os.sched_get_priority_max`, :func:`~os.sched_get_priority_min`,
1661 :func:`~os.sched_getaffinity`, :func:`~os.sched_getparam`,
1662 :func:`~os.sched_getscheduler`, :func:`~os.sched_rr_get_interval`,
1663 :func:`~os.sched_setaffinity`, :func:`~os.sched_setparam`,
1664 :func:`~os.sched_setscheduler`, :func:`~os.sched_yield`,
1665
1666 * New functions to control the file system:
1667
1668 * :func:`~os.posix_fadvise`: Announces an intention to access data in a
1669 specific pattern thus allowing the kernel to make optimizations.
1670 * :func:`~os.posix_fallocate`: Ensures that enough disk space is allocated
1671 for a file.
1672 * :func:`~os.sync`: Force write of everything to disk.
1673
1674 * Additional new posix functions:
1675
1676 * :func:`~os.lockf`: Apply, test or remove a POSIX lock on an open file descriptor.
1677 * :func:`~os.pread`: Read from a file descriptor at an offset, the file
1678 offset remains unchanged.
1679 * :func:`~os.pwrite`: Write to a file descriptor from an offset, leaving
1680 the file offset unchanged.
1681 * :func:`~os.readv`: Read from a file descriptor into a number of writable buffers.
1682 * :func:`~os.truncate`: Truncate the file corresponding to *path*, so that
1683 it is at most *length* bytes in size.
1684 * :func:`~os.waitid`: Wait for the completion of one or more child processes.
1685 * :func:`~os.writev`: Write the contents of *buffers* to a file descriptor,
1686 where *buffers* is an arbitrary sequence of buffers.
1687 * :func:`~os.getgrouplist` (:issue:`9344`): Return list of group ids that
1688 specified user belongs to.
1689
1690 * :func:`~os.times` and :func:`~os.uname`: Return type changed from a tuple to
1691 a tuple-like object with named attributes.
1692
1693 * Some platforms now support additional constants for the :func:`~os.lseek`
1694 function, such as ``os.SEEK_HOLE`` and ``os.SEEK_DATA``.
1695
1696 * New constants :const:`~os.RTLD_LAZY`, :const:`~os.RTLD_NOW`,
1697 :const:`~os.RTLD_GLOBAL`, :const:`~os.RTLD_LOCAL`, :const:`~os.RTLD_NODELETE`,
1698 :const:`~os.RTLD_NOLOAD`, and :const:`~os.RTLD_DEEPBIND` are available on
1699 platforms that support them. These are for use with the
1700 :func:`sys.setdlopenflags` function, and supersede the similar constants
1701 defined in :mod:`ctypes` and :mod:`!DLFCN`. (Contributed by Victor Stinner
1702 in :issue:`13226`.)
1703
1704 * :func:`os.symlink` now accepts (and ignores) the ``target_is_directory``
1705 keyword argument on non-Windows platforms, to ease cross-platform support.
1706
1707
1708 pdb
1709 ---
1710
1711 Tab-completion is now available not only for command names, but also their
1712 arguments. For example, for the ``break`` command, function and file names
1713 are completed.
1714
1715 (Contributed by Georg Brandl in :issue:`14210`)
1716
1717
1718 pickle
1719 ------
1720
1721 :class:`pickle.Pickler` objects now have an optional
1722 :attr:`~pickle.Pickler.dispatch_table` attribute allowing per-pickler
1723 reduction functions to be set.
1724
1725 (Contributed by Richard Oudkerk in :issue:`14166`.)
1726
1727
1728 pydoc
1729 -----
1730
1731 The Tk GUI and the :func:`!serve` function have been removed from the
1732 :mod:`pydoc` module: ``pydoc -g`` and :func:`!serve` have been deprecated
1733 in Python 3.2.
1734
1735
1736 re
1737 --
1738
1739 :class:`str` regular expressions now support ``\u`` and ``\U`` escapes.
1740
1741 (Contributed by Serhiy Storchaka in :issue:`3665`.)
1742
1743
1744 sched
1745 -----
1746
1747 * :meth:`~sched.scheduler.run` now accepts a *blocking* parameter which when
1748 set to false makes the method execute the scheduled events due to expire
1749 soonest (if any) and then return immediately.
1750 This is useful in case you want to use the :class:`~sched.scheduler` in
1751 non-blocking applications. (Contributed by Giampaolo Rodolà in :issue:`13449`.)
1752
1753 * :class:`~sched.scheduler` class can now be safely used in multi-threaded
1754 environments. (Contributed by Josiah Carlson and Giampaolo Rodolà in
1755 :issue:`8684`.)
1756
1757 * *timefunc* and *delayfunct* parameters of :class:`~sched.scheduler` class
1758 constructor are now optional and defaults to :func:`time.time` and
1759 :func:`time.sleep` respectively. (Contributed by Chris Clark in
1760 :issue:`13245`.)
1761
1762 * :meth:`~sched.scheduler.enter` and :meth:`~sched.scheduler.enterabs`
1763 *argument* parameter is now optional. (Contributed by Chris Clark in
1764 :issue:`13245`.)
1765
1766 * :meth:`~sched.scheduler.enter` and :meth:`~sched.scheduler.enterabs`
1767 now accept a *kwargs* parameter. (Contributed by Chris Clark in
1768 :issue:`13245`.)
1769
1770
1771 select
1772 ------
1773
1774 Solaris and derivative platforms have a new class :class:`select.devpoll`
1775 for high performance asynchronous sockets via :file:`/dev/poll`.
1776 (Contributed by Jesús Cea Avión in :issue:`6397`.)
1777
1778
1779 shlex
1780 -----
1781
1782 The previously undocumented helper function ``quote`` from the
1783 :mod:`!pipes` modules has been moved to the :mod:`shlex` module and
1784 documented. :func:`~shlex.quote` properly escapes all characters in a string
1785 that might be otherwise given special meaning by the shell.
1786
1787
1788 shutil
1789 ------
1790
1791 * New functions:
1792
1793 * :func:`~shutil.disk_usage`: provides total, used and free disk space
1794 statistics. (Contributed by Giampaolo Rodolà in :issue:`12442`.)
1795 * :func:`~shutil.chown`: allows one to change user and/or group of the given
1796 path also specifying the user/group names and not only their numeric
1797 ids. (Contributed by Sandro Tosi in :issue:`12191`.)
1798 * :func:`shutil.get_terminal_size`: returns the size of the terminal window
1799 to which the interpreter is attached. (Contributed by Zbigniew
1800 Jędrzejewski-Szmek in :issue:`13609`.)
1801
1802 * :func:`~shutil.copy2` and :func:`~shutil.copystat` now preserve file
1803 timestamps with nanosecond precision on platforms that support it.
1804 They also preserve file "extended attributes" on Linux. (Contributed
1805 by Larry Hastings in :issue:`14127` and :issue:`15238`.)
1806
1807 * Several functions now take an optional ``symlinks`` argument: when that
1808 parameter is true, symlinks aren't dereferenced and the operation instead
1809 acts on the symlink itself (or creates one, if relevant).
1810 (Contributed by Hynek Schlawack in :issue:`12715`.)
1811
1812 * When copying files to a different file system, :func:`~shutil.move` now
1813 handles symlinks the way the posix ``mv`` command does, recreating the
1814 symlink rather than copying the target file contents. (Contributed by
1815 Jonathan Niehof in :issue:`9993`.) :func:`~shutil.move` now also returns
1816 the ``dst`` argument as its result.
1817
1818 * :func:`~shutil.rmtree` is now resistant to symlink attacks on platforms
1819 which support the new ``dir_fd`` parameter in :func:`os.open` and
1820 :func:`os.unlink`. (Contributed by Martin von Löwis and Hynek Schlawack
1821 in :issue:`4489`.)
1822
1823
1824 signal
1825 ------
1826
1827 * The :mod:`signal` module has new functions:
1828
1829 * :func:`~signal.pthread_sigmask`: fetch and/or change the signal mask of the
1830 calling thread (Contributed by Jean-Paul Calderone in :issue:`8407`);
1831 * :func:`~signal.pthread_kill`: send a signal to a thread;
1832 * :func:`~signal.sigpending`: examine pending functions;
1833 * :func:`~signal.sigwait`: wait a signal;
1834 * :func:`~signal.sigwaitinfo`: wait for a signal, returning detailed
1835 information about it;
1836 * :func:`~signal.sigtimedwait`: like :func:`~signal.sigwaitinfo` but with a
1837 timeout.
1838
1839 * The signal handler writes the signal number as a single byte instead of
1840 a nul byte into the wakeup file descriptor. So it is possible to wait more
1841 than one signal and know which signals were raised.
1842
1843 * :func:`signal.signal` and :func:`signal.siginterrupt` raise an OSError,
1844 instead of a RuntimeError: OSError has an errno attribute.
1845
1846
1847 smtpd
1848 -----
1849
1850 The :mod:`!smtpd` module now supports :rfc:`5321` (extended SMTP) and :rfc:`1870`
1851 (size extension). Per the standard, these extensions are enabled if and only
1852 if the client initiates the session with an ``EHLO`` command.
1853
1854 (Initial ``ELHO`` support by Alberto Trevino. Size extension by Juhana
1855 Jauhiainen. Substantial additional work on the patch contributed by Michele
1856 Orrù and Dan Boswell. :issue:`8739`)
1857
1858
1859 smtplib
1860 -------
1861
1862 The :class:`~smtplib.SMTP`, :class:`~smtplib.SMTP_SSL`, and
1863 :class:`~smtplib.LMTP` classes now accept a ``source_address`` keyword argument
1864 to specify the ``(host, port)`` to use as the source address in the bind call
1865 when creating the outgoing socket. (Contributed by Paulo Scardine in
1866 :issue:`11281`.)
1867
1868 :class:`~smtplib.SMTP` now supports the context management protocol, allowing an
1869 ``SMTP`` instance to be used in a ``with`` statement. (Contributed
1870 by Giampaolo Rodolà in :issue:`11289`.)
1871
1872 The :class:`~smtplib.SMTP_SSL` constructor and the :meth:`~smtplib.SMTP.starttls`
1873 method now accept an SSLContext parameter to control parameters of the secure
1874 channel. (Contributed by Kasun Herath in :issue:`8809`.)
1875
1876
1877 socket
1878 ------
1879
1880 * The :class:`~socket.socket` class now exposes additional methods to process
1881 ancillary data when supported by the underlying platform:
1882
1883 * :func:`~socket.socket.sendmsg`
1884 * :func:`~socket.socket.recvmsg`
1885 * :func:`~socket.socket.recvmsg_into`
1886
1887 (Contributed by David Watson in :issue:`6560`, based on an earlier patch by
1888 Heiko Wundram)
1889
1890 * The :class:`~socket.socket` class now supports the PF_CAN protocol family
1891 (https://en.wikipedia.org/wiki/Socketcan), on Linux
1892 (https://lwn.net/Articles/253425).
1893
1894 (Contributed by Matthias Fuchs, updated by Tiago Gonçalves in :issue:`10141`.)
1895
1896 * The :class:`~socket.socket` class now supports the PF_RDS protocol family
1897 (https://en.wikipedia.org/wiki/Reliable_Datagram_Sockets and
1898 `https://oss.oracle.com/projects/rds <https://web.archive.org/web/20130115155505/https://oss.oracle.com/projects/rds/>`__).
1899
1900 * The :class:`~socket.socket` class now supports the ``PF_SYSTEM`` protocol
1901 family on OS X. (Contributed by Michael Goderbauer in :issue:`13777`.)
1902
1903 * New function :func:`~socket.sethostname` allows the hostname to be set
1904 on Unix systems if the calling process has sufficient privileges.
1905 (Contributed by Ross Lagerwall in :issue:`10866`.)
1906
1907
1908 socketserver
1909 ------------
1910
1911 :class:`~socketserver.BaseServer` now has an overridable method
1912 :meth:`~socketserver.BaseServer.service_actions` that is called by the
1913 :meth:`~socketserver.BaseServer.serve_forever` method in the service loop.
1914 :class:`~socketserver.ForkingMixIn` now uses this to clean up zombie
1915 child processes. (Contributed by Justin Warkentin in :issue:`11109`.)
1916
1917
1918 sqlite3
1919 -------
1920
1921 New :class:`sqlite3.Connection` method
1922 :meth:`~sqlite3.Connection.set_trace_callback` can be used to capture a trace of
1923 all sql commands processed by sqlite. (Contributed by Torsten Landschoff
1924 in :issue:`11688`.)
1925
1926
1927 ssl
1928 ---
1929
1930 * The :mod:`ssl` module has two new random generation functions:
1931
1932 * :func:`~ssl.RAND_bytes`: generate cryptographically strong
1933 pseudo-random bytes.
1934 * :func:`!RAND_pseudo_bytes`: generate pseudo-random bytes.
1935
1936 (Contributed by Victor Stinner in :issue:`12049`.)
1937
1938 * The :mod:`ssl` module now exposes a finer-grained exception hierarchy
1939 in order to make it easier to inspect the various kinds of errors.
1940 (Contributed by Antoine Pitrou in :issue:`11183`.)
1941
1942 * :meth:`~ssl.SSLContext.load_cert_chain` now accepts a *password* argument
1943 to be used if the private key is encrypted.
1944 (Contributed by Adam Simpkins in :issue:`12803`.)
1945
1946 * Diffie-Hellman key exchange, both regular and Elliptic Curve-based, is
1947 now supported through the :meth:`~ssl.SSLContext.load_dh_params` and
1948 :meth:`~ssl.SSLContext.set_ecdh_curve` methods.
1949 (Contributed by Antoine Pitrou in :issue:`13626` and :issue:`13627`.)
1950
1951 * SSL sockets have a new :meth:`~ssl.SSLSocket.get_channel_binding` method
1952 allowing the implementation of certain authentication mechanisms such as
1953 SCRAM-SHA-1-PLUS. (Contributed by Jacek Konieczny in :issue:`12551`.)
1954
1955 * You can query the SSL compression algorithm used by an SSL socket, thanks
1956 to its new :meth:`~ssl.SSLSocket.compression` method. The new attribute
1957 :const:`~ssl.OP_NO_COMPRESSION` can be used to disable compression.
1958 (Contributed by Antoine Pitrou in :issue:`13634`.)
1959
1960 * Support has been added for the Next Protocol Negotiation extension using
1961 the :meth:`ssl.SSLContext.set_npn_protocols` method.
1962 (Contributed by Colin Marc in :issue:`14204`.)
1963
1964 * SSL errors can now be introspected more easily thanks to
1965 :attr:`~ssl.SSLError.library` and :attr:`~ssl.SSLError.reason` attributes.
1966 (Contributed by Antoine Pitrou in :issue:`14837`.)
1967
1968 * The :func:`~ssl.get_server_certificate` function now supports IPv6.
1969 (Contributed by Charles-François Natali in :issue:`11811`.)
1970
1971 * New attribute :const:`~ssl.OP_CIPHER_SERVER_PREFERENCE` allows setting
1972 SSLv3 server sockets to use the server's cipher ordering preference rather
1973 than the client's (:issue:`13635`).
1974
1975
1976 stat
1977 ----
1978
1979 The undocumented tarfile.filemode function has been moved to
1980 :func:`stat.filemode`. It can be used to convert a file's mode to a string of
1981 the form '-rwxrwxrwx'.
1982
1983 (Contributed by Giampaolo Rodolà in :issue:`14807`.)
1984
1985
1986 struct
1987 ------
1988
1989 The :mod:`struct` module now supports :c:type:`ssize_t` and :c:type:`size_t` via the
1990 new codes ``n`` and ``N``, respectively. (Contributed by Antoine Pitrou
1991 in :issue:`3163`.)
1992
1993
1994 subprocess
1995 ----------
1996
1997 Command strings can now be bytes objects on posix platforms. (Contributed by
1998 Victor Stinner in :issue:`8513`.)
1999
2000 A new constant :const:`~subprocess.DEVNULL` allows suppressing output in a
2001 platform-independent fashion. (Contributed by Ross Lagerwall in
2002 :issue:`5870`.)
2003
2004
2005 sys
2006 ---
2007
2008 The :mod:`sys` module has a new :data:`~sys.thread_info` :term:`named
2009 tuple` holding information about the thread implementation
2010 (:issue:`11223`).
2011
2012
2013 tarfile
2014 -------
2015
2016 :mod:`tarfile` now supports ``lzma`` encoding via the :mod:`lzma` module.
2017 (Contributed by Lars Gustäbel in :issue:`5689`.)
2018
2019
2020 tempfile
2021 --------
2022
2023 :class:`tempfile.SpooledTemporaryFile`\'s :meth:`!truncate` method now accepts
2024 a ``size`` parameter. (Contributed by Ryan Kelly in :issue:`9957`.)
2025
2026
2027 textwrap
2028 --------
2029
2030 The :mod:`textwrap` module has a new :func:`~textwrap.indent` that makes
2031 it straightforward to add a common prefix to selected lines in a block
2032 of text (:issue:`13857`).
2033
2034
2035 threading
2036 ---------
2037
2038 :class:`threading.Condition`, :class:`threading.Semaphore`,
2039 :class:`threading.BoundedSemaphore`, :class:`threading.Event`, and
2040 :class:`threading.Timer`, all of which used to be factory functions returning a
2041 class instance, are now classes and may be subclassed. (Contributed by Éric
2042 Araujo in :issue:`10968`.)
2043
2044 The :class:`threading.Thread` constructor now accepts a ``daemon`` keyword
2045 argument to override the default behavior of inheriting the ``daemon`` flag
2046 value from the parent thread (:issue:`6064`).
2047
2048 The formerly private function ``_thread.get_ident`` is now available as the
2049 public function :func:`threading.get_ident`. This eliminates several cases of
2050 direct access to the ``_thread`` module in the stdlib. Third party code that
2051 used ``_thread.get_ident`` should likewise be changed to use the new public
2052 interface.
2053
2054
2055 time
2056 ----
2057
2058 The :pep:`418` added new functions to the :mod:`time` module:
2059
2060 * :func:`~time.get_clock_info`: Get information on a clock.
2061 * :func:`~time.monotonic`: Monotonic clock (cannot go backward), not affected
2062 by system clock updates.
2063 * :func:`~time.perf_counter`: Performance counter with the highest available
2064 resolution to measure a short duration.
2065 * :func:`~time.process_time`: Sum of the system and user CPU time of the
2066 current process.
2067
2068 Other new functions:
2069
2070 * :func:`~time.clock_getres`, :func:`~time.clock_gettime` and
2071 :func:`~time.clock_settime` functions with :samp:`CLOCK_{xxx}` constants.
2072 (Contributed by Victor Stinner in :issue:`10278`.)
2073
2074 To improve cross platform consistency, :func:`~time.sleep` now raises a
2075 :exc:`ValueError` when passed a negative sleep value. Previously this was an
2076 error on posix, but produced an infinite sleep on Windows.
2077
2078
2079 types
2080 -----
2081
2082 Add a new :class:`types.MappingProxyType` class: Read-only proxy of a mapping.
2083 (:issue:`14386`)
2084
2085
2086 The new functions :func:`types.new_class` and :func:`types.prepare_class` provide support
2087 for :pep:`3115` compliant dynamic type creation. (:issue:`14588`)
2088
2089
2090 unittest
2091 --------
2092
2093 :meth:`.assertRaises`, :meth:`.assertRaisesRegex`, :meth:`.assertWarns`, and
2094 :meth:`.assertWarnsRegex` now accept a keyword argument *msg* when used as
2095 context managers. (Contributed by Ezio Melotti and Winston Ewert in
2096 :issue:`10775`.)
2097
2098 :meth:`unittest.TestCase.run` now returns the :class:`~unittest.TestResult`
2099 object.
2100
2101
2102 urllib
2103 ------
2104
2105 The :class:`~urllib.request.Request` class, now accepts a *method* argument
2106 used by :meth:`~urllib.request.Request.get_method` to determine what HTTP method
2107 should be used. For example, this will send a ``'HEAD'`` request::
2108
2109 >>> urlopen(Request('https://www.python.org', method='HEAD'))
2110
2111 (:issue:`1673007`)
2112
2113
2114 webbrowser
2115 ----------
2116
2117 The :mod:`webbrowser` module supports more "browsers": Google Chrome (named
2118 :program:`chrome`, :program:`chromium`, :program:`chrome-browser` or
2119 :program:`chromium-browser` depending on the version and operating system),
2120 and the generic launchers :program:`xdg-open`, from the FreeDesktop.org
2121 project, and :program:`gvfs-open`, which is the default URI handler for GNOME
2122 3. (The former contributed by Arnaud Calmettes in :issue:`13620`, the latter
2123 by Matthias Klose in :issue:`14493`.)
2124
2125
2126 xml.etree.ElementTree
2127 ---------------------
2128
2129 The :mod:`xml.etree.ElementTree` module now imports its C accelerator by
2130 default; there is no longer a need to explicitly import
2131 :mod:`!xml.etree.cElementTree` (this module stays for backwards compatibility,
2132 but is now deprecated). In addition, the ``iter`` family of methods of
2133 :class:`~xml.etree.ElementTree.Element` has been optimized (rewritten in C).
2134 The module's documentation has also been greatly improved with added examples
2135 and a more detailed reference.
2136
2137
2138 zlib
2139 ----
2140
2141 New attribute :attr:`zlib.Decompress.eof` makes it possible to distinguish
2142 between a properly formed compressed stream and an incomplete or truncated one.
2143 (Contributed by Nadeem Vawda in :issue:`12646`.)
2144
2145 New attribute :const:`zlib.ZLIB_RUNTIME_VERSION` reports the version string of
2146 the underlying ``zlib`` library that is loaded at runtime. (Contributed by
2147 Torsten Landschoff in :issue:`12306`.)
2148
2149
2150 Optimizations
2151 =============
2152
2153 Major performance enhancements have been added:
2154
2155 * Thanks to :pep:`393`, some operations on Unicode strings have been optimized:
2156
2157 * the memory footprint is divided by 2 to 4 depending on the text
2158 * encode an ASCII string to UTF-8 doesn't need to encode characters anymore,
2159 the UTF-8 representation is shared with the ASCII representation
2160 * the UTF-8 encoder has been optimized
2161 * repeating a single ASCII letter and getting a substring of an ASCII string
2162 is 4 times faster
2163
2164 * UTF-8 is now 2x to 4x faster. UTF-16 encoding is now up to 10x faster.
2165
2166 (Contributed by Serhiy Storchaka, :issue:`14624`, :issue:`14738` and
2167 :issue:`15026`.)
2168
2169
2170 Build and C API Changes
2171 =======================
2172
2173 Changes to Python's build process and to the C API include:
2174
2175 * New :pep:`3118` related function:
2176
2177 * :c:func:`PyMemoryView_FromMemory`
2178
2179 * :pep:`393` added new Unicode types, macros and functions:
2180
2181 * High-level API:
2182
2183 * :c:func:`PyUnicode_CopyCharacters`
2184 * :c:func:`PyUnicode_FindChar`
2185 * :c:func:`PyUnicode_GetLength`, :c:macro:`PyUnicode_GET_LENGTH`
2186 * :c:func:`PyUnicode_New`
2187 * :c:func:`PyUnicode_Substring`
2188 * :c:func:`PyUnicode_ReadChar`, :c:func:`PyUnicode_WriteChar`
2189
2190 * Low-level API:
2191
2192 * :c:type:`Py_UCS1`, :c:type:`Py_UCS2`, :c:type:`Py_UCS4` types
2193 * :c:type:`PyASCIIObject` and :c:type:`PyCompactUnicodeObject` structures
2194 * :c:macro:`PyUnicode_READY`
2195 * :c:func:`PyUnicode_FromKindAndData`
2196 * :c:func:`PyUnicode_AsUCS4`, :c:func:`PyUnicode_AsUCS4Copy`
2197 * :c:macro:`PyUnicode_DATA`, :c:macro:`PyUnicode_1BYTE_DATA`,
2198 :c:macro:`PyUnicode_2BYTE_DATA`, :c:macro:`PyUnicode_4BYTE_DATA`
2199 * :c:macro:`PyUnicode_KIND` with :c:enum:`!PyUnicode_Kind` enum:
2200 :c:data:`!PyUnicode_WCHAR_KIND`, :c:data:`PyUnicode_1BYTE_KIND`,
2201 :c:data:`PyUnicode_2BYTE_KIND`, :c:data:`PyUnicode_4BYTE_KIND`
2202 * :c:macro:`PyUnicode_READ`, :c:macro:`PyUnicode_READ_CHAR`, :c:macro:`PyUnicode_WRITE`
2203 * :c:macro:`PyUnicode_MAX_CHAR_VALUE`
2204
2205 * :c:macro:`PyArg_ParseTuple` now accepts a :class:`bytearray` for the ``c``
2206 format (:issue:`12380`).
2207
2208
2209
2210 Deprecated
2211 ==========
2212
2213 Unsupported Operating Systems
2214 -----------------------------
2215
2216 OS/2 and VMS are no longer supported due to the lack of a maintainer.
2217
2218 Windows 2000 and Windows platforms which set ``COMSPEC`` to ``command.com``
2219 are no longer supported due to maintenance burden.
2220
2221 OSF support, which was deprecated in 3.2, has been completely removed.
2222
2223
2224 Deprecated Python modules, functions and methods
2225 ------------------------------------------------
2226
2227 * Passing a non-empty string to ``object.__format__()`` is deprecated, and
2228 will produce a :exc:`TypeError` in Python 3.4 (:issue:`9856`).
2229 * The ``unicode_internal`` codec has been deprecated because of the
2230 :pep:`393`, use UTF-8, UTF-16 (``utf-16-le`` or ``utf-16-be``), or UTF-32
2231 (``utf-32-le`` or ``utf-32-be``)
2232 * :meth:`ftplib.FTP.nlst` and :meth:`ftplib.FTP.dir`: use
2233 :meth:`ftplib.FTP.mlsd`
2234 * :func:`!platform.popen`: use the :mod:`subprocess` module. Check especially
2235 the :ref:`subprocess-replacements` section (:issue:`11377`).
2236 * :issue:`13374`: The Windows bytes API has been deprecated in the :mod:`os`
2237 module. Use Unicode filenames, instead of bytes filenames, to not depend on
2238 the ANSI code page anymore and to support any filename.
2239 * :issue:`13988`: The :mod:`!xml.etree.cElementTree` module is deprecated. The
2240 accelerator is used automatically whenever available.
2241 * The behaviour of :func:`!time.clock` depends on the platform: use the new
2242 :func:`time.perf_counter` or :func:`time.process_time` function instead,
2243 depending on your requirements, to have a well defined behaviour.
2244 * The :func:`!os.stat_float_times` function is deprecated.
2245 * :mod:`abc` module:
2246
2247 * :class:`abc.abstractproperty` has been deprecated, use :class:`property`
2248 with :func:`abc.abstractmethod` instead.
2249 * :class:`abc.abstractclassmethod` has been deprecated, use
2250 :class:`classmethod` with :func:`abc.abstractmethod` instead.
2251 * :class:`abc.abstractstaticmethod` has been deprecated, use
2252 :class:`staticmethod` with :func:`abc.abstractmethod` instead.
2253
2254 * :mod:`importlib` package:
2255
2256 * :meth:`importlib.abc.SourceLoader.path_mtime` is now deprecated in favour of
2257 :meth:`importlib.abc.SourceLoader.path_stats` as bytecode files now store
2258 both the modification time and size of the source file the bytecode file was
2259 compiled from.
2260
2261
2262
2263
2264
2265 Deprecated functions and types of the C API
2266 -------------------------------------------
2267
2268 The :c:type:`Py_UNICODE` has been deprecated by :pep:`393` and will be
2269 removed in Python 4. All functions using this type are deprecated:
2270
2271 Unicode functions and methods using :c:type:`Py_UNICODE` and
2272 :c:expr:`Py_UNICODE*` types:
2273
2274 * :c:macro:`!PyUnicode_FromUnicode`: use :c:func:`PyUnicode_FromWideChar` or
2275 :c:func:`PyUnicode_FromKindAndData`
2276 * :c:macro:`!PyUnicode_AS_UNICODE`, :c:func:`!PyUnicode_AsUnicode`,
2277 :c:func:`!PyUnicode_AsUnicodeAndSize`: use :c:func:`PyUnicode_AsWideCharString`
2278 * :c:macro:`!PyUnicode_AS_DATA`: use :c:macro:`PyUnicode_DATA` with
2279 :c:macro:`PyUnicode_READ` and :c:macro:`PyUnicode_WRITE`
2280 * :c:macro:`!PyUnicode_GET_SIZE`, :c:func:`!PyUnicode_GetSize`: use
2281 :c:macro:`PyUnicode_GET_LENGTH` or :c:func:`PyUnicode_GetLength`
2282 * :c:macro:`!PyUnicode_GET_DATA_SIZE`: use
2283 ``PyUnicode_GET_LENGTH(str) * PyUnicode_KIND(str)`` (only work on ready
2284 strings)
2285 * :c:func:`!PyUnicode_AsUnicodeCopy`: use :c:func:`PyUnicode_AsUCS4Copy` or
2286 :c:func:`PyUnicode_AsWideCharString`
2287 * :c:func:`!PyUnicode_GetMax`
2288
2289
2290 Functions and macros manipulating Py_UNICODE* strings:
2291
2292 * :c:macro:`!Py_UNICODE_strlen()`: use :c:func:`PyUnicode_GetLength` or
2293 :c:macro:`PyUnicode_GET_LENGTH`
2294 * :c:macro:`!Py_UNICODE_strcat()`: use :c:func:`PyUnicode_CopyCharacters` or
2295 :c:func:`PyUnicode_FromFormat`
2296 * :c:macro:`!Py_UNICODE_strcpy()`, :c:macro:`!Py_UNICODE_strncpy()`,
2297 :c:macro:`!Py_UNICODE_COPY()`: use :c:func:`PyUnicode_CopyCharacters` or
2298 :c:func:`PyUnicode_Substring`
2299 * :c:macro:`!Py_UNICODE_strcmp()`: use :c:func:`PyUnicode_Compare`
2300 * :c:macro:`!Py_UNICODE_strncmp()`: use :c:func:`PyUnicode_Tailmatch`
2301 * :c:macro:`!Py_UNICODE_strchr()`, :c:macro:`!Py_UNICODE_strrchr()`: use
2302 :c:func:`PyUnicode_FindChar`
2303 * :c:macro:`!Py_UNICODE_FILL()`: use :c:func:`PyUnicode_Fill`
2304 * :c:macro:`!Py_UNICODE_MATCH`
2305
2306 Encoders:
2307
2308 * :c:func:`!PyUnicode_Encode`: use :c:func:`!PyUnicode_AsEncodedObject`
2309 * :c:func:`!PyUnicode_EncodeUTF7`
2310 * :c:func:`!PyUnicode_EncodeUTF8`: use :c:func:`PyUnicode_AsUTF8` or
2311 :c:func:`PyUnicode_AsUTF8String`
2312 * :c:func:`!PyUnicode_EncodeUTF32`
2313 * :c:func:`!PyUnicode_EncodeUTF16`
2314 * :c:func:`!PyUnicode_EncodeUnicodeEscape` use
2315 :c:func:`PyUnicode_AsUnicodeEscapeString`
2316 * :c:func:`!PyUnicode_EncodeRawUnicodeEscape` use
2317 :c:func:`PyUnicode_AsRawUnicodeEscapeString`
2318 * :c:func:`!PyUnicode_EncodeLatin1`: use :c:func:`PyUnicode_AsLatin1String`
2319 * :c:func:`!PyUnicode_EncodeASCII`: use :c:func:`PyUnicode_AsASCIIString`
2320 * :c:func:`!PyUnicode_EncodeCharmap`
2321 * :c:func:`!PyUnicode_TranslateCharmap`
2322 * :c:func:`!PyUnicode_EncodeMBCS`: use :c:func:`PyUnicode_AsMBCSString` or
2323 :c:func:`PyUnicode_EncodeCodePage` (with ``CP_ACP`` code_page)
2324 * :c:func:`!PyUnicode_EncodeDecimal`,
2325 :c:func:`!PyUnicode_TransformDecimalToASCII`
2326
2327
2328 Deprecated features
2329 -------------------
2330
2331 The :mod:`array` module's ``'u'`` format code is now deprecated and will be
2332 removed in Python 4 together with the rest of the (:c:type:`Py_UNICODE`) API.
2333
2334
2335 Porting to Python 3.3
2336 =====================
2337
2338 This section lists previously described changes and other bugfixes
2339 that may require changes to your code.
2340
2341 .. _portingpythoncode:
2342
2343 Porting Python code
2344 -------------------
2345
2346 * Hash randomization is enabled by default. Set the :envvar:`PYTHONHASHSEED`
2347 environment variable to ``0`` to disable hash randomization. See also the
2348 :meth:`object.__hash__` method.
2349
2350 * :issue:`12326`: On Linux, sys.platform doesn't contain the major version
2351 anymore. It is now always 'linux', instead of 'linux2' or 'linux3' depending
2352 on the Linux version used to build Python. Replace sys.platform == 'linux2'
2353 with sys.platform.startswith('linux'), or directly sys.platform == 'linux' if
2354 you don't need to support older Python versions.
2355
2356 * :issue:`13847`, :issue:`14180`: :mod:`time` and :mod:`datetime`:
2357 :exc:`OverflowError` is now raised instead of :exc:`ValueError` if a
2358 timestamp is out of range. :exc:`OSError` is now raised if C functions
2359 :c:func:`gmtime` or :c:func:`localtime` failed.
2360
2361 * The default finders used by import now utilize a cache of what is contained
2362 within a specific directory. If you create a Python source file or sourceless
2363 bytecode file, make sure to call :func:`importlib.invalidate_caches` to clear
2364 out the cache for the finders to notice the new file.
2365
2366 * :exc:`ImportError` now uses the full name of the module that was attempted to
2367 be imported. Doctests that check ImportErrors' message will need to be
2368 updated to use the full name of the module instead of just the tail of the
2369 name.
2370
2371 * The *index* argument to :func:`__import__` now defaults to 0 instead of -1
2372 and no longer support negative values. It was an oversight when :pep:`328` was
2373 implemented that the default value remained -1. If you need to continue to
2374 perform a relative import followed by an absolute import, then perform the
2375 relative import using an index of 1, followed by another import using an
2376 index of 0. It is preferred, though, that you use
2377 :func:`importlib.import_module` rather than call :func:`__import__` directly.
2378
2379 * :func:`__import__` no longer allows one to use an index value other than 0
2380 for top-level modules. E.g. ``__import__('sys', level=1)`` is now an error.
2381
2382 * Because :data:`sys.meta_path` and :data:`sys.path_hooks` now have finders on
2383 them by default, you will most likely want to use :meth:`list.insert` instead
2384 of :meth:`list.append` to add to those lists.
2385
2386 * Because ``None`` is now inserted into :data:`sys.path_importer_cache`, if you
2387 are clearing out entries in the dictionary of paths that do not have a
2388 finder, you will need to remove keys paired with values of ``None`` **and**
2389 :class:`!imp.NullImporter` to be backwards-compatible. This will lead to extra
2390 overhead on older versions of Python that re-insert ``None`` into
2391 :data:`sys.path_importer_cache` where it represents the use of implicit
2392 finders, but semantically it should not change anything.
2393
2394 * :class:`!importlib.abc.Finder` no longer specifies a ``find_module()`` abstract
2395 method that must be implemented. If you were relying on subclasses to
2396 implement that method, make sure to check for the method's existence first.
2397 You will probably want to check for ``find_loader()`` first, though, in the
2398 case of working with :term:`path entry finders <path entry finder>`.
2399
2400 * :mod:`pkgutil` has been converted to use :mod:`importlib` internally. This
2401 eliminates many edge cases where the old behaviour of the :pep:`302` import
2402 emulation failed to match the behaviour of the real import system. The
2403 import emulation itself is still present, but is now deprecated. The
2404 :func:`pkgutil.iter_importers` and :func:`pkgutil.walk_packages` functions
2405 special case the standard import hooks so they are still supported even
2406 though they do not provide the non-standard ``iter_modules()`` method.
2407
2408 * A longstanding RFC-compliance bug (:issue:`1079`) in the parsing done by
2409 :func:`email.header.decode_header` has been fixed. Code that uses the
2410 standard idiom to convert encoded headers into unicode
2411 (``str(make_header(decode_header(h))``) will see no change, but code that
2412 looks at the individual tuples returned by decode_header will see that
2413 whitespace that precedes or follows ``ASCII`` sections is now included in the
2414 ``ASCII`` section. Code that builds headers using ``make_header`` should
2415 also continue to work without change, since ``make_header`` continues to add
2416 whitespace between ``ASCII`` and non-``ASCII`` sections if it is not already
2417 present in the input strings.
2418
2419 * :func:`email.utils.formataddr` now does the correct content transfer
2420 encoding when passed non-``ASCII`` display names. Any code that depended on
2421 the previous buggy behavior that preserved the non-``ASCII`` unicode in the
2422 formatted output string will need to be changed (:issue:`1690608`).
2423
2424 * :meth:`poplib.POP3.quit` may now raise protocol errors like all other
2425 ``poplib`` methods. Code that assumes ``quit`` does not raise
2426 :exc:`poplib.error_proto` errors may need to be changed if errors on ``quit``
2427 are encountered by a particular application (:issue:`11291`).
2428
2429 * The ``strict`` argument to :class:`email.parser.Parser`, deprecated since
2430 Python 2.4, has finally been removed.
2431
2432 * The deprecated method ``unittest.TestCase.assertSameElements`` has been
2433 removed.
2434
2435 * The deprecated variable ``time.accept2dyear`` has been removed.
2436
2437 * The deprecated ``Context._clamp`` attribute has been removed from the
2438 :mod:`decimal` module. It was previously replaced by the public attribute
2439 :attr:`~decimal.Context.clamp`. (See :issue:`8540`.)
2440
2441 * The undocumented internal helper class ``SSLFakeFile`` has been removed
2442 from :mod:`smtplib`, since its functionality has long been provided directly
2443 by :meth:`socket.socket.makefile`.
2444
2445 * Passing a negative value to :func:`time.sleep` on Windows now raises an
2446 error instead of sleeping forever. It has always raised an error on posix.
2447
2448 * The ``ast.__version__`` constant has been removed. If you need to
2449 make decisions affected by the AST version, use :data:`sys.version_info`
2450 to make the decision.
2451
2452 * Code that used to work around the fact that the :mod:`threading` module used
2453 factory functions by subclassing the private classes will need to change to
2454 subclass the now-public classes.
2455
2456 * The undocumented debugging machinery in the threading module has been
2457 removed, simplifying the code. This should have no effect on production
2458 code, but is mentioned here in case any application debug frameworks were
2459 interacting with it (:issue:`13550`).
2460
2461
2462 Porting C code
2463 --------------
2464
2465 * In the course of changes to the buffer API the undocumented
2466 :c:member:`!smalltable` member of the
2467 :c:type:`Py_buffer` structure has been removed and the
2468 layout of the :c:type:`!PyMemoryViewObject` has changed.
2469
2470 All extensions relying on the relevant parts in ``memoryobject.h``
2471 or ``object.h`` must be rebuilt.
2472
2473 * Due to :ref:`PEP 393 <pep-393>`, the :c:type:`Py_UNICODE` type and all
2474 functions using this type are deprecated (but will stay available for
2475 at least five years). If you were using low-level Unicode APIs to
2476 construct and access unicode objects and you want to benefit of the
2477 memory footprint reduction provided by :pep:`393`, you have to convert
2478 your code to the new :doc:`Unicode API <../c-api/unicode>`.
2479
2480 However, if you only have been using high-level functions such as
2481 :c:func:`PyUnicode_Concat()`, :c:func:`PyUnicode_Join` or
2482 :c:func:`PyUnicode_FromFormat()`, your code will automatically take
2483 advantage of the new unicode representations.
2484
2485 * :c:func:`PyImport_GetMagicNumber` now returns ``-1`` upon failure.
2486
2487 * As a negative value for the *level* argument to :func:`__import__` is no
2488 longer valid, the same now holds for :c:func:`PyImport_ImportModuleLevel`.
2489 This also means that the value of *level* used by
2490 :c:func:`PyImport_ImportModuleEx` is now ``0`` instead of ``-1``.
2491
2492
2493 Building C extensions
2494 ---------------------
2495
2496 * The range of possible file names for C extensions has been narrowed.
2497 Very rarely used spellings have been suppressed: under POSIX, files
2498 named ``xxxmodule.so``, ``xxxmodule.abi3.so`` and
2499 ``xxxmodule.cpython-*.so`` are no longer recognized as implementing
2500 the ``xxx`` module. If you had been generating such files, you have
2501 to switch to the other spellings (i.e., remove the ``module`` string
2502 from the file names).
2503
2504 (implemented in :issue:`14040`.)
2505
2506
2507 Command Line Switch Changes
2508 ---------------------------
2509
2510 * The -Q command-line flag and related artifacts have been removed. Code
2511 checking sys.flags.division_warning will need updating.
2512
2513 (:issue:`10998`, contributed by Éric Araujo.)
2514
2515 * When :program:`python` is started with :option:`-S`, ``import site``
2516 will no longer add site-specific paths to the module search paths. In
2517 previous versions, it did.
2518
2519 (:issue:`11591`, contributed by Carl Meyer with editions by Éric Araujo.)