Debian Codegraph @

Doc/whatsnew/3.2.rst

raw
1 ****************************
2 What's New In Python 3.2
3 ****************************
4
5 :Author: Raymond Hettinger
6
7 .. $Id$
8 Rules for maintenance:
9
10 * Anyone can add text to this document. Do not spend very much time
11 on the wording of your changes, because your text will probably
12 get rewritten. (Note, during release candidate phase or just before
13 a beta release, please use the tracker instead -- this helps avoid
14 merge conflicts. If you must add a suggested entry directly,
15 please put it in an XXX comment and the maintainer will take notice).
16
17 * The maintainer will go through Misc/NEWS periodically and add
18 changes; it's therefore more important to add your changes to
19 Misc/NEWS than to this file.
20
21 * This is not a complete list of every single change; completeness
22 is the purpose of Misc/NEWS. Some changes I consider too small
23 or esoteric to include. If such a change is added to the text,
24 I'll just remove it. (This is another reason you shouldn't spend
25 too much time on writing your addition.)
26
27 * If you want to draw your new text to the attention of the
28 maintainer, add 'XXX' to the beginning of the paragraph or
29 section.
30
31 * It's OK to just add a fragmentary note about a change. For
32 example: "XXX Describe the transmogrify() function added to the
33 socket module." The maintainer will research the change and
34 write the necessary text.
35
36 * You can comment out your additions if you like, but it's not
37 necessary (especially when a final release is some months away).
38
39 * Credit the author of a patch or bugfix. Just the name is
40 sufficient; the e-mail address isn't necessary. It's helpful to
41 add the issue number:
42
43 XXX Describe the transmogrify() function added to the socket
44 module.
45
46 (Contributed by P.Y. Developer; :issue:`12345`.)
47
48 This saves the maintainer the effort of going through the SVN log
49 when researching a change.
50
51 This article explains the new features in Python 3.2 as compared to 3.1.
52 Python 3.2 was released on February 20, 2011. It
53 focuses on a few highlights and gives a few examples. For full details, see the
54 `Misc/NEWS
55 <https://github.com/python/cpython/blob/076ca6c3c8df3030307e548d9be792ce3c1c6eea/Misc/NEWS>`__
56 file.
57
58 .. seealso::
59
60 :pep:`392` - Python 3.2 Release Schedule
61
62
63 PEP 384: Defining a Stable ABI
64 ==============================
65
66 In the past, extension modules built for one Python version were often
67 not usable with other Python versions. Particularly on Windows, every
68 feature release of Python required rebuilding all extension modules that
69 one wanted to use. This requirement was the result of the free access to
70 Python interpreter internals that extension modules could use.
71
72 With Python 3.2, an alternative approach becomes available: extension
73 modules which restrict themselves to a limited API (by defining
74 Py_LIMITED_API) cannot use many of the internals, but are constrained
75 to a set of API functions that are promised to be stable for several
76 releases. As a consequence, extension modules built for 3.2 in that
77 mode will also work with 3.3, 3.4, and so on. Extension modules that
78 make use of details of memory structures can still be built, but will
79 need to be recompiled for every feature release.
80
81 .. seealso::
82
83 :pep:`384` - Defining a Stable ABI
84 PEP written by Martin von Löwis.
85
86
87 PEP 389: Argparse Command Line Parsing Module
88 =============================================
89
90 A new module for command line parsing, :mod:`argparse`, was introduced to
91 overcome the limitations of :mod:`optparse` which did not provide support for
92 positional arguments (not just options), subcommands, required options and other
93 common patterns of specifying and validating options.
94
95 This module has already had widespread success in the community as a
96 third-party module. Being more fully featured than its predecessor, the
97 :mod:`argparse` module is now the preferred module for command-line processing.
98 The older module is still being kept available because of the substantial amount
99 of legacy code that depends on it.
100
101 Here's an annotated example parser showing features like limiting results to a
102 set of choices, specifying a *metavar* in the help screen, validating that one
103 or more positional arguments is present, and making a required option::
104
105 import argparse
106 parser = argparse.ArgumentParser(
107 description = 'Manage servers', # main description for help
108 epilog = 'Tested on Solaris and Linux') # displayed after help
109 parser.add_argument('action', # argument name
110 choices = ['deploy', 'start', 'stop'], # three allowed values
111 help = 'action on each target') # help msg
112 parser.add_argument('targets',
113 metavar = 'HOSTNAME', # var name used in help msg
114 nargs = '+', # require one or more targets
115 help = 'url for target machines') # help msg explanation
116 parser.add_argument('-u', '--user', # -u or --user option
117 required = True, # make it a required argument
118 help = 'login as user')
119
120 Example of calling the parser on a command string::
121
122 >>> cmd = 'deploy sneezy.example.com sleepy.example.com -u skycaptain'
123 >>> result = parser.parse_args(cmd.split())
124 >>> result.action
125 'deploy'
126 >>> result.targets
127 ['sneezy.example.com', 'sleepy.example.com']
128 >>> result.user
129 'skycaptain'
130
131 Example of the parser's automatically generated help::
132
133 >>> parser.parse_args('-h'.split())
134
135 usage: manage_cloud.py [-h] -u USER
136 {deploy,start,stop} HOSTNAME [HOSTNAME ...]
137
138 Manage servers
139
140 positional arguments:
141 {deploy,start,stop} action on each target
142 HOSTNAME url for target machines
143
144 optional arguments:
145 -h, --help show this help message and exit
146 -u USER, --user USER login as user
147
148 Tested on Solaris and Linux
149
150 An especially nice :mod:`argparse` feature is the ability to define subparsers,
151 each with their own argument patterns and help displays::
152
153 import argparse
154 parser = argparse.ArgumentParser(prog='HELM')
155 subparsers = parser.add_subparsers()
156
157 parser_l = subparsers.add_parser('launch', help='Launch Control') # first subgroup
158 parser_l.add_argument('-m', '--missiles', action='store_true')
159 parser_l.add_argument('-t', '--torpedos', action='store_true')
160
161 parser_m = subparsers.add_parser('move', help='Move Vessel', # second subgroup
162 aliases=('steer', 'turn')) # equivalent names
163 parser_m.add_argument('-c', '--course', type=int, required=True)
164 parser_m.add_argument('-s', '--speed', type=int, default=0)
165
166 .. code-block:: shell-session
167
168 $ ./helm.py --help # top level help (launch and move)
169 $ ./helm.py launch --help # help for launch options
170 $ ./helm.py launch --missiles # set missiles=True and torpedos=False
171 $ ./helm.py steer --course 180 --speed 5 # set movement parameters
172
173 .. seealso::
174
175 :pep:`389` - New Command Line Parsing Module
176 PEP written by Steven Bethard.
177
178 :ref:`upgrading-optparse-code` for details on the differences from :mod:`optparse`.
179
180
181 PEP 391: Dictionary Based Configuration for Logging
182 ====================================================
183
184 The :mod:`logging` module provided two kinds of configuration, one style with
185 function calls for each option or another style driven by an external file saved
186 in a :mod:`configparser` format. Those options did not provide the flexibility
187 to create configurations from JSON or YAML files, nor did they support
188 incremental configuration, which is needed for specifying logger options from a
189 command line.
190
191 To support a more flexible style, the module now offers
192 :func:`logging.config.dictConfig` for specifying logging configuration with
193 plain Python dictionaries. The configuration options include formatters,
194 handlers, filters, and loggers. Here's a working example of a configuration
195 dictionary::
196
197 {"version": 1,
198 "formatters": {"brief": {"format": "%(levelname)-8s: %(name)-15s: %(message)s"},
199 "full": {"format": "%(asctime)s %(name)-15s %(levelname)-8s %(message)s"}
200 },
201 "handlers": {"console": {
202 "class": "logging.StreamHandler",
203 "formatter": "brief",
204 "level": "INFO",
205 "stream": "ext://sys.stdout"},
206 "console_priority": {
207 "class": "logging.StreamHandler",
208 "formatter": "full",
209 "level": "ERROR",
210 "stream": "ext://sys.stderr"}
211 },
212 "root": {"level": "DEBUG", "handlers": ["console", "console_priority"]}}
213
214
215 If that dictionary is stored in a file called :file:`conf.json`, it can be
216 loaded and called with code like this::
217
218 >>> import json, logging.config
219 >>> with open('conf.json') as f:
220 ... conf = json.load(f)
221 ...
222 >>> logging.config.dictConfig(conf)
223 >>> logging.info("Transaction completed normally")
224 INFO : root : Transaction completed normally
225 >>> logging.critical("Abnormal termination")
226 2011-02-17 11:14:36,694 root CRITICAL Abnormal termination
227
228 .. seealso::
229
230 :pep:`391` - Dictionary Based Configuration for Logging
231 PEP written by Vinay Sajip.
232
233
234 PEP 3148: The ``concurrent.futures`` module
235 ============================================
236
237 Code for creating and managing concurrency is being collected in a new top-level
238 namespace, *concurrent*. Its first member is a *futures* package which provides
239 a uniform high-level interface for managing threads and processes.
240
241 The design for :mod:`concurrent.futures` was inspired by the
242 *java.util.concurrent* package. In that model, a running call and its result
243 are represented by a :class:`~concurrent.futures.Future` object that abstracts
244 features common to threads, processes, and remote procedure calls. That object
245 supports status checks (running or done), timeouts, cancellations, adding
246 callbacks, and access to results or exceptions.
247
248 The primary offering of the new module is a pair of executor classes for
249 launching and managing calls. The goal of the executors is to make it easier to
250 use existing tools for making parallel calls. They save the effort needed to
251 setup a pool of resources, launch the calls, create a results queue, add
252 time-out handling, and limit the total number of threads, processes, or remote
253 procedure calls.
254
255 Ideally, each application should share a single executor across multiple
256 components so that process and thread limits can be centrally managed. This
257 solves the design challenge that arises when each component has its own
258 competing strategy for resource management.
259
260 Both classes share a common interface with three methods:
261 :meth:`~concurrent.futures.Executor.submit` for scheduling a callable and
262 returning a :class:`~concurrent.futures.Future` object;
263 :meth:`~concurrent.futures.Executor.map` for scheduling many asynchronous calls
264 at a time, and :meth:`~concurrent.futures.Executor.shutdown` for freeing
265 resources. The class is a :term:`context manager` and can be used in a
266 :keyword:`with` statement to assure that resources are automatically released
267 when currently pending futures are done executing.
268
269 A simple of example of :class:`~concurrent.futures.ThreadPoolExecutor` is a
270 launch of four parallel threads for copying files::
271
272 import concurrent.futures, shutil
273 with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e:
274 e.submit(shutil.copy, 'src1.txt', 'dest1.txt')
275 e.submit(shutil.copy, 'src2.txt', 'dest2.txt')
276 e.submit(shutil.copy, 'src3.txt', 'dest3.txt')
277 e.submit(shutil.copy, 'src3.txt', 'dest4.txt')
278
279 .. seealso::
280
281 :pep:`3148` - Futures -- Execute Computations Asynchronously
282 PEP written by Brian Quinlan.
283
284 :ref:`Code for Threaded Parallel URL reads<threadpoolexecutor-example>`, an
285 example using threads to fetch multiple web pages in parallel.
286
287 :ref:`Code for computing prime numbers in
288 parallel<processpoolexecutor-example>`, an example demonstrating
289 :class:`~concurrent.futures.ProcessPoolExecutor`.
290
291
292 PEP 3147: PYC Repository Directories
293 =====================================
294
295 Python's scheme for caching bytecode in *.pyc* files did not work well in
296 environments with multiple Python interpreters. If one interpreter encountered
297 a cached file created by another interpreter, it would recompile the source and
298 overwrite the cached file, thus losing the benefits of caching.
299
300 The issue of "pyc fights" has become more pronounced as it has become
301 commonplace for Linux distributions to ship with multiple versions of Python.
302 These conflicts also arise with CPython alternatives such as Unladen Swallow.
303
304 To solve this problem, Python's import machinery has been extended to use
305 distinct filenames for each interpreter. Instead of Python 3.2 and Python 3.3 and
306 Unladen Swallow each competing for a file called "mymodule.pyc", they will now
307 look for "mymodule.cpython-32.pyc", "mymodule.cpython-33.pyc", and
308 "mymodule.unladen10.pyc". And to prevent all of these new files from
309 cluttering source directories, the *pyc* files are now collected in a
310 "__pycache__" directory stored under the package directory.
311
312 Aside from the filenames and target directories, the new scheme has a few
313 aspects that are visible to the programmer:
314
315 * Imported modules now have a :attr:`~module.__cached__` attribute which stores
316 the name of the actual file that was imported:
317
318 >>> import collections
319 >>> collections.__cached__ # doctest: +SKIP
320 'c:/py32/lib/__pycache__/collections.cpython-32.pyc'
321
322 * The tag that is unique to each interpreter is accessible from the :mod:`!imp`
323 module:
324
325 >>> import imp # doctest: +SKIP
326 >>> imp.get_tag() # doctest: +SKIP
327 'cpython-32'
328
329 * Scripts that try to deduce source filename from the imported file now need to
330 be smarter. It is no longer sufficient to simply strip the "c" from a ".pyc"
331 filename. Instead, use the new functions in the :mod:`!imp` module:
332
333 >>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc') # doctest: +SKIP
334 'c:/py32/lib/collections.py'
335 >>> imp.cache_from_source('c:/py32/lib/collections.py') # doctest: +SKIP
336 'c:/py32/lib/__pycache__/collections.cpython-32.pyc'
337
338 * The :mod:`py_compile` and :mod:`compileall` modules have been updated to
339 reflect the new naming convention and target directory. The command-line
340 invocation of *compileall* has new options: ``-i`` for
341 specifying a list of files and directories to compile and ``-b`` which causes
342 bytecode files to be written to their legacy location rather than
343 *__pycache__*.
344
345 * The :mod:`importlib.abc` module has been updated with new :term:`abstract base
346 classes <abstract base class>` for loading bytecode files. The obsolete
347 ABCs, :class:`!PyLoader` and
348 :class:`!PyPycLoader`, have been deprecated (instructions on how
349 to stay Python 3.1 compatible are included with the documentation).
350
351 .. seealso::
352
353 :pep:`3147` - PYC Repository Directories
354 PEP written by Barry Warsaw.
355
356
357 PEP 3149: ABI Version Tagged .so Files
358 ======================================
359
360 The PYC repository directory allows multiple bytecode cache files to be
361 co-located. This PEP implements a similar mechanism for shared object files by
362 giving them a common directory and distinct names for each version.
363
364 The common directory is "pyshared" and the file names are made distinct by
365 identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the
366 major and minor version numbers, and optional build flags (such as "d" for
367 debug, "m" for pymalloc, "u" for wide-unicode). For an arbitrary package "foo",
368 you may see these files when the distribution package is installed::
369
370 /usr/share/pyshared/foo.cpython-32m.so
371 /usr/share/pyshared/foo.cpython-33md.so
372
373 In Python itself, the tags are accessible from functions in the :mod:`sysconfig`
374 module::
375
376 >>> import sysconfig
377 >>> sysconfig.get_config_var('SOABI') # find the version tag
378 'cpython-32mu'
379 >>> sysconfig.get_config_var('EXT_SUFFIX') # find the full filename extension
380 '.cpython-32mu.so'
381
382 .. seealso::
383
384 :pep:`3149` - ABI Version Tagged .so Files
385 PEP written by Barry Warsaw.
386
387
388 PEP 3333: Python Web Server Gateway Interface v1.0.1
389 =====================================================
390
391 This informational PEP clarifies how bytes/text issues are to be handled by the
392 WSGI protocol. The challenge is that string handling in Python 3 is most
393 conveniently handled with the :class:`str` type even though the HTTP protocol
394 is itself bytes oriented.
395
396 The PEP differentiates so-called *native strings* that are used for
397 request/response headers and metadata versus *byte strings* which are used for
398 the bodies of requests and responses.
399
400 The *native strings* are always of type :class:`str` but are restricted to code
401 points between *U+0000* through *U+00FF* which are translatable to bytes using
402 *Latin-1* encoding. These strings are used for the keys and values in the
403 environment dictionary and for response headers and statuses in the
404 :func:`!start_response` function. They must follow :rfc:`2616` with respect to
405 encoding. That is, they must either be *ISO-8859-1* characters or use
406 :rfc:`2047` MIME encoding.
407
408 For developers porting WSGI applications from Python 2, here are the salient
409 points:
410
411 * If the app already used strings for headers in Python 2, no change is needed.
412
413 * If instead, the app encoded output headers or decoded input headers, then the
414 headers will need to be re-encoded to Latin-1. For example, an output header
415 encoded in utf-8 was using ``h.encode('utf-8')`` now needs to convert from
416 bytes to native strings using ``h.encode('utf-8').decode('latin-1')``.
417
418 * Values yielded by an application or sent using the :meth:`!write` method
419 must be byte strings. The :func:`!start_response` function and environ
420 must use native strings. The two cannot be mixed.
421
422 For server implementers writing CGI-to-WSGI pathways or other CGI-style
423 protocols, the users must to be able access the environment using native strings
424 even though the underlying platform may have a different convention. To bridge
425 this gap, the :mod:`wsgiref` module has a new function,
426 :func:`wsgiref.handlers.read_environ` for transcoding CGI variables from
427 :data:`os.environ` into native strings and returning a new dictionary.
428
429 .. seealso::
430
431 :pep:`3333` - Python Web Server Gateway Interface v1.0.1
432 PEP written by Phillip Eby.
433
434
435 Other Language Changes
436 ======================
437
438 Some smaller changes made to the core Python language are:
439
440 * String formatting for :func:`format` and :meth:`str.format` gained new
441 capabilities for the format character **#**. Previously, for integers in
442 binary, octal, or hexadecimal, it caused the output to be prefixed with '0b',
443 '0o', or '0x' respectively. Now it can also handle floats, complex, and
444 Decimal, causing the output to always have a decimal point even when no digits
445 follow it.
446
447 >>> format(20, '#o')
448 '0o24'
449 >>> format(12.34, '#5.0f')
450 ' 12.'
451
452 (Suggested by Mark Dickinson and implemented by Eric Smith in :issue:`7094`.)
453
454 * There is also a new :meth:`str.format_map` method that extends the
455 capabilities of the existing :meth:`str.format` method by accepting arbitrary
456 :term:`mapping` objects. This new method makes it possible to use string
457 formatting with any of Python's many dictionary-like objects such as
458 :class:`~collections.defaultdict`, :class:`~shelve.Shelf`,
459 :class:`~configparser.ConfigParser`, or :mod:`dbm`. It is also useful with
460 custom :class:`dict` subclasses that normalize keys before look-up or that
461 supply a :meth:`~object.__missing__` method for unknown keys::
462
463 >>> import shelve
464 >>> d = shelve.open('tmp.shl')
465 >>> 'The {project_name} status is {status} as of {date}'.format_map(d)
466 'The testing project status is green as of February 15, 2011'
467
468 >>> class LowerCasedDict(dict):
469 ... def __getitem__(self, key):
470 ... return dict.__getitem__(self, key.lower())
471 ...
472 >>> lcd = LowerCasedDict(part='widgets', quantity=10)
473 >>> 'There are {QUANTITY} {Part} in stock'.format_map(lcd)
474 'There are 10 widgets in stock'
475
476 >>> class PlaceholderDict(dict):
477 ... def __missing__(self, key):
478 ... return '<{}>'.format(key)
479 ...
480 >>> 'Hello {name}, welcome to {location}'.format_map(PlaceholderDict())
481 'Hello <name>, welcome to <location>'
482
483 (Suggested by Raymond Hettinger and implemented by Eric Smith in
484 :issue:`6081`.)
485
486 * The interpreter can now be started with a quiet option, ``-q``, to prevent
487 the copyright and version information from being displayed in the interactive
488 mode. The option can be introspected using the :data:`sys.flags` attribute:
489
490 .. code-block:: shell-session
491
492 $ python -q
493 >>> sys.flags
494 sys.flags(debug=0, division_warning=0, inspect=0, interactive=0,
495 optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0,
496 ignore_environment=0, verbose=0, bytes_warning=0, quiet=1)
497
498 (Contributed by Marcin Wojdyr in :issue:`1772833`).
499
500 * The :func:`hasattr` function works by calling :func:`getattr` and detecting
501 whether an exception is raised. This technique allows it to detect methods
502 created dynamically by :meth:`~object.__getattr__` or :meth:`~object.__getattribute__` which
503 would otherwise be absent from the class dictionary. Formerly, *hasattr*
504 would catch any exception, possibly masking genuine errors. Now, *hasattr*
505 has been tightened to only catch :exc:`AttributeError` and let other
506 exceptions pass through::
507
508 >>> class A:
509 ... @property
510 ... def f(self):
511 ... return 1 // 0
512 ...
513 >>> a = A()
514 >>> hasattr(a, 'f')
515 Traceback (most recent call last):
516 ...
517 ZeroDivisionError: integer division or modulo by zero
518
519 (Discovered by Yury Selivanov and fixed by Benjamin Peterson; :issue:`9666`.)
520
521 * The :func:`str` of a float or complex number is now the same as its
522 :func:`repr`. Previously, the :func:`str` form was shorter but that just
523 caused confusion and is no longer needed now that the shortest possible
524 :func:`repr` is displayed by default:
525
526 >>> import math
527 >>> repr(math.pi)
528 '3.141592653589793'
529 >>> str(math.pi)
530 '3.141592653589793'
531
532 (Proposed and implemented by Mark Dickinson; :issue:`9337`.)
533
534 * :class:`memoryview` objects now have a :meth:`~memoryview.release` method
535 and they also now support the context management protocol. This allows timely
536 release of any resources that were acquired when requesting a buffer from the
537 original object.
538
539 >>> with memoryview(b'abcdefgh') as v:
540 ... print(v.tolist())
541 [97, 98, 99, 100, 101, 102, 103, 104]
542
543 (Added by Antoine Pitrou; :issue:`9757`.)
544
545 * Previously it was illegal to delete a name from the local namespace if it
546 occurs as a free variable in a nested block::
547
548 def outer(x):
549 def inner():
550 return x
551 inner()
552 del x
553
554 This is now allowed. Remember that the target of an :keyword:`except` clause
555 is cleared, so this code which used to work with Python 2.6, raised a
556 :exc:`SyntaxError` with Python 3.1 and now works again::
557
558 def f():
559 def print_error():
560 print(e)
561 try:
562 something
563 except Exception as e:
564 print_error()
565 # implicit "del e" here
566
567 (See :issue:`4617`.)
568
569 * :ref:`Struct sequence types <struct-sequence-objects>` are now subclasses of tuple.
570 This means that C structures like those returned by :func:`os.stat`,
571 :func:`time.gmtime`, and :data:`sys.version_info` now work like a
572 :term:`named tuple` and now work with functions and methods that
573 expect a tuple as an argument. This is a big step forward in making the C
574 structures as flexible as their pure Python counterparts:
575
576 >>> import sys
577 >>> isinstance(sys.version_info, tuple)
578 True
579 >>> 'Version %d.%d.%d %s(%d)' % sys.version_info # doctest: +SKIP
580 'Version 3.2.0 final(0)'
581
582 (Suggested by Arfrever Frehtes Taifersar Arahesis and implemented
583 by Benjamin Peterson in :issue:`8413`.)
584
585 * Warnings are now easier to control using the :envvar:`PYTHONWARNINGS`
586 environment variable as an alternative to using ``-W`` at the command line:
587
588 .. code-block:: shell-session
589
590 $ export PYTHONWARNINGS='ignore::RuntimeWarning::,once::UnicodeWarning::'
591
592 (Suggested by Barry Warsaw and implemented by Philip Jenvey in :issue:`7301`.)
593
594 * A new warning category, :exc:`ResourceWarning`, has been added. It is
595 emitted when potential issues with resource consumption or cleanup
596 are detected. It is silenced by default in normal release builds but
597 can be enabled through the means provided by the :mod:`warnings`
598 module, or on the command line.
599
600 A :exc:`ResourceWarning` is issued at interpreter shutdown if the
601 :data:`gc.garbage` list isn't empty, and if :const:`gc.DEBUG_UNCOLLECTABLE` is
602 set, all uncollectable objects are printed. This is meant to make the
603 programmer aware that their code contains object finalization issues.
604
605 A :exc:`ResourceWarning` is also issued when a :term:`file object` is destroyed
606 without having been explicitly closed. While the deallocator for such
607 object ensures it closes the underlying operating system resource
608 (usually, a file descriptor), the delay in deallocating the object could
609 produce various issues, especially under Windows. Here is an example
610 of enabling the warning from the command line:
611
612 .. code-block:: shell-session
613
614 $ python -q -Wdefault
615 >>> f = open("foo", "wb")
616 >>> del f
617 __main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'>
618
619 (Added by Antoine Pitrou and Georg Brandl in :issue:`10093` and :issue:`477863`.)
620
621 * :class:`range` objects now support *index* and *count* methods. This is part
622 of an effort to make more objects fully implement the
623 :class:`collections.Sequence <collections.abc.Sequence>` :term:`abstract base class`. As a result, the
624 language will have a more uniform API. In addition, :class:`range` objects
625 now support slicing and negative indices, even with values larger than
626 :data:`sys.maxsize`. This makes *range* more interoperable with lists::
627
628 >>> range(0, 100, 2).count(10)
629 1
630 >>> range(0, 100, 2).index(10)
631 5
632 >>> range(0, 100, 2)[5]
633 10
634 >>> range(0, 100, 2)[0:5]
635 range(0, 10, 2)
636
637 (Contributed by Daniel Stutzbach in :issue:`9213`, by Alexander Belopolsky
638 in :issue:`2690`, and by Nick Coghlan in :issue:`10889`.)
639
640 * The :func:`callable` builtin function from Py2.x was resurrected. It provides
641 a concise, readable alternative to using an :term:`abstract base class` in an
642 expression like ``isinstance(x, collections.Callable)``:
643
644 >>> callable(max)
645 True
646 >>> callable(20)
647 False
648
649 (See :issue:`10518`.)
650
651 * Python's import mechanism can now load modules installed in directories with
652 non-ASCII characters in the path name. This solved an aggravating problem
653 with home directories for users with non-ASCII characters in their usernames.
654
655 (Required extensive work by Victor Stinner in :issue:`9425`.)
656
657
658 New, Improved, and Deprecated Modules
659 =====================================
660
661 Python's standard library has undergone significant maintenance efforts and
662 quality improvements.
663
664 The biggest news for Python 3.2 is that the :mod:`email` package, :mod:`mailbox`
665 module, and :mod:`!nntplib` modules now work correctly with the bytes/text model
666 in Python 3. For the first time, there is correct handling of messages with
667 mixed encodings.
668
669 Throughout the standard library, there has been more careful attention to
670 encodings and text versus bytes issues. In particular, interactions with the
671 operating system are now better able to exchange non-ASCII data using the
672 Windows MBCS encoding, locale-aware encodings, or UTF-8.
673
674 Another significant win is the addition of substantially better support for
675 *SSL* connections and security certificates.
676
677 In addition, more classes now implement a :term:`context manager` to support
678 convenient and reliable resource clean-up using a :keyword:`with` statement.
679
680 email
681 -----
682
683 The usability of the :mod:`email` package in Python 3 has been mostly fixed by
684 the extensive efforts of R. David Murray. The problem was that emails are
685 typically read and stored in the form of :class:`bytes` rather than :class:`str`
686 text, and they may contain multiple encodings within a single email. So, the
687 email package had to be extended to parse and generate email messages in bytes
688 format.
689
690 * New functions :func:`~email.message_from_bytes` and
691 :func:`~email.message_from_binary_file`, and new classes
692 :class:`~email.parser.BytesFeedParser` and :class:`~email.parser.BytesParser`
693 allow binary message data to be parsed into model objects.
694
695 * Given bytes input to the model, :meth:`~email.message.Message.get_payload`
696 will by default decode a message body that has a
697 :mailheader:`Content-Transfer-Encoding` of *8bit* using the charset
698 specified in the MIME headers and return the resulting string.
699
700 * Given bytes input to the model, :class:`~email.generator.Generator` will
701 convert message bodies that have a :mailheader:`Content-Transfer-Encoding` of
702 *8bit* to instead have a *7bit* :mailheader:`Content-Transfer-Encoding`.
703
704 Headers with unencoded non-ASCII bytes are deemed to be :rfc:`2047`\ -encoded
705 using the *unknown-8bit* character set.
706
707 * A new class :class:`~email.generator.BytesGenerator` produces bytes as output,
708 preserving any unchanged non-ASCII data that was present in the input used to
709 build the model, including message bodies with a
710 :mailheader:`Content-Transfer-Encoding` of *8bit*.
711
712 * The :mod:`smtplib` :class:`~smtplib.SMTP` class now accepts a byte string
713 for the *msg* argument to the :meth:`~smtplib.SMTP.sendmail` method,
714 and a new method, :meth:`~smtplib.SMTP.send_message` accepts a
715 :class:`~email.message.Message` object and can optionally obtain the
716 *from_addr* and *to_addrs* addresses directly from the object.
717
718 (Proposed and implemented by R. David Murray, :issue:`4661` and :issue:`10321`.)
719
720 elementtree
721 -----------
722
723 The :mod:`xml.etree.ElementTree` package and its :mod:`!xml.etree.cElementTree`
724 counterpart have been updated to version 1.3.
725
726 Several new and useful functions and methods have been added:
727
728 * :func:`xml.etree.ElementTree.fromstringlist` which builds an XML document
729 from a sequence of fragments
730 * :func:`xml.etree.ElementTree.register_namespace` for registering a global
731 namespace prefix
732 * :func:`xml.etree.ElementTree.tostringlist` for string representation
733 including all sublists
734 * :meth:`xml.etree.ElementTree.Element.extend` for appending a sequence of zero
735 or more elements
736 * :meth:`xml.etree.ElementTree.Element.iterfind` searches an element and
737 subelements
738 * :meth:`xml.etree.ElementTree.Element.itertext` creates a text iterator over
739 an element and its subelements
740 * :meth:`xml.etree.ElementTree.TreeBuilder.end` closes the current element
741 * :meth:`xml.etree.ElementTree.TreeBuilder.doctype` handles a doctype
742 declaration
743
744 Two methods have been deprecated:
745
746 * :meth:`!xml.etree.ElementTree.getchildren` use ``list(elem)`` instead.
747 * :meth:`!xml.etree.ElementTree.getiterator` use ``Element.iter`` instead.
748
749 For details of the update, see `Introducing ElementTree
750 <https://web.archive.org/web/20200703234532/http://effbot.org/zone/elementtree-13-intro.htm>`_
751 on Fredrik Lundh's website.
752
753 (Contributed by Florent Xicluna and Fredrik Lundh, :issue:`6472`.)
754
755 functools
756 ---------
757
758 * The :mod:`functools` module includes a new decorator for caching function
759 calls. :func:`functools.lru_cache` can save repeated queries to an external
760 resource whenever the results are expected to be the same.
761
762 For example, adding a caching decorator to a database query function can save
763 database accesses for popular searches:
764
765 >>> import functools
766 >>> @functools.lru_cache(maxsize=300)
767 ... def get_phone_number(name):
768 ... c = conn.cursor()
769 ... c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,))
770 ... return c.fetchone()[0]
771
772 >>> for name in user_requests: # doctest: +SKIP
773 ... get_phone_number(name) # cached lookup
774
775 To help with choosing an effective cache size, the wrapped function is
776 instrumented for tracking cache statistics:
777
778 >>> get_phone_number.cache_info() # doctest: +SKIP
779 CacheInfo(hits=4805, misses=980, maxsize=300, currsize=300)
780
781 If the phonelist table gets updated, the outdated contents of the cache can be
782 cleared with:
783
784 >>> get_phone_number.cache_clear()
785
786 (Contributed by Raymond Hettinger and incorporating design ideas from Jim
787 Baker, Miki Tebeka, and Nick Coghlan; see `recipe 498245
788 <https://code.activestate.com/recipes/498245-lru-and-lfu-cache-decorators/>`_\, `recipe 577479
789 <https://code.activestate.com/recipes/577479-simple-caching-decorator/>`_\, :issue:`10586`, and
790 :issue:`10593`.)
791
792 * The :func:`functools.wraps` decorator now adds a :attr:`__wrapped__` attribute
793 pointing to the original callable function. This allows wrapped functions to
794 be introspected. It also copies :attr:`~function.__annotations__` if
795 defined. And now it also gracefully skips over missing attributes such as
796 :attr:`~function.__doc__` which
797 might not be defined for the wrapped callable.
798
799 In the above example, the cache can be removed by recovering the original
800 function:
801
802 >>> get_phone_number = get_phone_number.__wrapped__ # uncached function
803
804 (By Nick Coghlan and Terrence Cole; :issue:`9567`, :issue:`3445`, and
805 :issue:`8814`.)
806
807 * To help write classes with rich comparison methods, a new decorator
808 :func:`functools.total_ordering` will use existing equality and inequality
809 methods to fill in the remaining methods.
810
811 For example, supplying *__eq__* and *__lt__* will enable
812 :func:`~functools.total_ordering` to fill-in *__le__*, *__gt__* and *__ge__*::
813
814 @total_ordering
815 class Student:
816 def __eq__(self, other):
817 return ((self.lastname.lower(), self.firstname.lower()) ==
818 (other.lastname.lower(), other.firstname.lower()))
819
820 def __lt__(self, other):
821 return ((self.lastname.lower(), self.firstname.lower()) <
822 (other.lastname.lower(), other.firstname.lower()))
823
824 With the *total_ordering* decorator, the remaining comparison methods
825 are filled in automatically.
826
827 (Contributed by Raymond Hettinger.)
828
829 * To aid in porting programs from Python 2, the :func:`functools.cmp_to_key`
830 function converts an old-style comparison function to
831 modern :term:`key function`:
832
833 >>> # locale-aware sort order
834 >>> sorted(iterable, key=cmp_to_key(locale.strcoll)) # doctest: +SKIP
835
836 For sorting examples and a brief sorting tutorial, see the `Sorting HowTo
837 <https://wiki.python.org/moin/HowTo/Sorting/>`_ tutorial.
838
839 (Contributed by Raymond Hettinger.)
840
841 itertools
842 ---------
843
844 * The :mod:`itertools` module has a new :func:`~itertools.accumulate` function
845 modeled on APL's *scan* operator and Numpy's *accumulate* function:
846
847 >>> from itertools import accumulate
848 >>> list(accumulate([8, 2, 50]))
849 [8, 10, 60]
850
851 >>> prob_dist = [0.1, 0.4, 0.2, 0.3]
852 >>> list(accumulate(prob_dist)) # cumulative probability distribution
853 [0.1, 0.5, 0.7, 1.0]
854
855 For an example using :func:`~itertools.accumulate`, see the :ref:`examples for
856 the random module <random-examples>`.
857
858 (Contributed by Raymond Hettinger and incorporating design suggestions
859 from Mark Dickinson.)
860
861 collections
862 -----------
863
864 * The :class:`collections.Counter` class now has two forms of in-place
865 subtraction, the existing *-=* operator for `saturating subtraction
866 <https://en.wikipedia.org/wiki/Saturation_arithmetic>`_ and the new
867 :meth:`~collections.Counter.subtract` method for regular subtraction. The
868 former is suitable for `multisets <https://en.wikipedia.org/wiki/Multiset>`_
869 which only have positive counts, and the latter is more suitable for use cases
870 that allow negative counts:
871
872 >>> from collections import Counter
873 >>> tally = Counter(dogs=5, cats=3)
874 >>> tally -= Counter(dogs=2, cats=8) # saturating subtraction
875 >>> tally
876 Counter({'dogs': 3})
877
878 >>> tally = Counter(dogs=5, cats=3)
879 >>> tally.subtract(dogs=2, cats=8) # regular subtraction
880 >>> tally
881 Counter({'dogs': 3, 'cats': -5})
882
883 (Contributed by Raymond Hettinger.)
884
885 * The :class:`collections.OrderedDict` class has a new method
886 :meth:`~collections.OrderedDict.move_to_end` which takes an existing key and
887 moves it to either the first or last position in the ordered sequence.
888
889 The default is to move an item to the last position. This is equivalent of
890 renewing an entry with ``od[k] = od.pop(k)``.
891
892 A fast move-to-end operation is useful for resequencing entries. For example,
893 an ordered dictionary can be used to track order of access by aging entries
894 from the oldest to the most recently accessed.
895
896 >>> from collections import OrderedDict
897 >>> d = OrderedDict.fromkeys(['a', 'b', 'X', 'd', 'e'])
898 >>> list(d)
899 ['a', 'b', 'X', 'd', 'e']
900 >>> d.move_to_end('X')
901 >>> list(d)
902 ['a', 'b', 'd', 'e', 'X']
903
904 (Contributed by Raymond Hettinger.)
905
906 * The :class:`collections.deque` class grew two new methods
907 :meth:`~collections.deque.count` and :meth:`~collections.deque.reverse` that
908 make them more substitutable for :class:`list` objects:
909
910 >>> from collections import deque
911 >>> d = deque('simsalabim')
912 >>> d.count('s')
913 2
914 >>> d.reverse()
915 >>> d
916 deque(['m', 'i', 'b', 'a', 'l', 'a', 's', 'm', 'i', 's'])
917
918 (Contributed by Raymond Hettinger.)
919
920 threading
921 ---------
922
923 The :mod:`threading` module has a new :class:`~threading.Barrier`
924 synchronization class for making multiple threads wait until all of them have
925 reached a common barrier point. Barriers are useful for making sure that a task
926 with multiple preconditions does not run until all of the predecessor tasks are
927 complete.
928
929 Barriers can work with an arbitrary number of threads. This is a generalization
930 of a `Rendezvous <https://en.wikipedia.org/wiki/Synchronous_rendezvous>`_ which
931 is defined for only two threads.
932
933 Implemented as a two-phase cyclic barrier, :class:`~threading.Barrier` objects
934 are suitable for use in loops. The separate *filling* and *draining* phases
935 assure that all threads get released (drained) before any one of them can loop
936 back and re-enter the barrier. The barrier fully resets after each cycle.
937
938 Example of using barriers::
939
940 from threading import Barrier, Thread
941
942 def get_votes(site):
943 ballots = conduct_election(site)
944 all_polls_closed.wait() # do not count until all polls are closed
945 totals = summarize(ballots)
946 publish(site, totals)
947
948 all_polls_closed = Barrier(len(sites))
949 for site in sites:
950 Thread(target=get_votes, args=(site,)).start()
951
952 In this example, the barrier enforces a rule that votes cannot be counted at any
953 polling site until all polls are closed. Notice how a solution with a barrier
954 is similar to one with :meth:`threading.Thread.join`, but the threads stay alive
955 and continue to do work (summarizing ballots) after the barrier point is
956 crossed.
957
958 If any of the predecessor tasks can hang or be delayed, a barrier can be created
959 with an optional *timeout* parameter. Then if the timeout period elapses before
960 all the predecessor tasks reach the barrier point, all waiting threads are
961 released and a :exc:`~threading.BrokenBarrierError` exception is raised::
962
963 def get_votes(site):
964 ballots = conduct_election(site)
965 try:
966 all_polls_closed.wait(timeout=midnight - time.now())
967 except BrokenBarrierError:
968 lockbox = seal_ballots(ballots)
969 queue.put(lockbox)
970 else:
971 totals = summarize(ballots)
972 publish(site, totals)
973
974 In this example, the barrier enforces a more robust rule. If some election
975 sites do not finish before midnight, the barrier times-out and the ballots are
976 sealed and deposited in a queue for later handling.
977
978 See `Barrier Synchronization Patterns
979 <https://osl.cs.illinois.edu/media/papers/karmani-2009-barrier_synchronization_pattern.pdf>`_
980 for more examples of how barriers can be used in parallel computing. Also, there is
981 a simple but thorough explanation of barriers in `The Little Book of Semaphores
982 <https://greenteapress.com/semaphores/LittleBookOfSemaphores.pdf>`_, *section 3.6*.
983
984 (Contributed by Kristján Valur Jónsson with an API review by Jeffrey Yasskin in
985 :issue:`8777`.)
986
987 datetime and time
988 -----------------
989
990 * The :mod:`datetime` module has a new type :class:`~datetime.timezone` that
991 implements the :class:`~datetime.tzinfo` interface by returning a fixed UTC
992 offset and timezone name. This makes it easier to create timezone-aware
993 datetime objects::
994
995 >>> import datetime as dt
996
997 >>> dt.datetime.now(dt.timezone.utc)
998 datetime.datetime(2010, 12, 8, 21, 4, 2, 923754, tzinfo=datetime.timezone.utc)
999
1000 >>> dt.datetime.strptime("01/01/2000 12:00 +0000", "%m/%d/%Y %H:%M %z")
1001 datetime.datetime(2000, 1, 1, 12, 0, tzinfo=datetime.timezone.utc)
1002
1003 * Also, :class:`~datetime.timedelta` objects can now be multiplied by
1004 :class:`float` and divided by :class:`float` and :class:`int` objects.
1005 And :class:`~datetime.timedelta` objects can now divide one another.
1006
1007 * The :meth:`datetime.date.strftime` method is no longer restricted to years
1008 after 1900. The new supported year range is from 1000 to 9999 inclusive.
1009
1010 * Whenever a two-digit year is used in a time tuple, the interpretation has been
1011 governed by :data:`!time.accept2dyear`. The default is ``True`` which means that
1012 for a two-digit year, the century is guessed according to the POSIX rules
1013 governing the ``%y`` strptime format.
1014
1015 Starting with Py3.2, use of the century guessing heuristic will emit a
1016 :exc:`DeprecationWarning`. Instead, it is recommended that
1017 :data:`!time.accept2dyear` be set to ``False`` so that large date ranges
1018 can be used without guesswork::
1019
1020 >>> import time, warnings
1021 >>> warnings.resetwarnings() # remove the default warning filters
1022
1023 >>> time.accept2dyear = True # guess whether 11 means 11 or 2011
1024 >>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0))
1025 Warning (from warnings module):
1026 ...
1027 DeprecationWarning: Century info guessed for a 2-digit year.
1028 'Fri Jan 1 12:34:56 2011'
1029
1030 >>> time.accept2dyear = False # use the full range of allowable dates
1031 >>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0))
1032 'Fri Jan 1 12:34:56 11'
1033
1034 Several functions now have significantly expanded date ranges. When
1035 :data:`!time.accept2dyear` is false, the :func:`time.asctime` function will
1036 accept any year that fits in a C int, while the :func:`time.mktime` and
1037 :func:`time.strftime` functions will accept the full range supported by the
1038 corresponding operating system functions.
1039
1040 (Contributed by Alexander Belopolsky and Victor Stinner in :issue:`1289118`,
1041 :issue:`5094`, :issue:`6641`, :issue:`2706`, :issue:`1777412`, :issue:`8013`,
1042 and :issue:`10827`.)
1043
1044 .. XXX https://bugs.python.org/issue?%40search_text=datetime&%40sort=-activity
1045
1046 math
1047 ----
1048
1049 The :mod:`math` module has been updated with six new functions inspired by the
1050 C99 standard.
1051
1052 The :func:`~math.isfinite` function provides a reliable and fast way to detect
1053 special values. It returns ``True`` for regular numbers and ``False`` for *Nan* or
1054 *Infinity*:
1055
1056 >>> from math import isfinite
1057 >>> [isfinite(x) for x in (123, 4.56, float('Nan'), float('Inf'))]
1058 [True, True, False, False]
1059
1060 The :func:`~math.expm1` function computes ``e**x-1`` for small values of *x*
1061 without incurring the loss of precision that usually accompanies the subtraction
1062 of nearly equal quantities:
1063
1064 >>> from math import expm1
1065 >>> expm1(0.013671875) # more accurate way to compute e**x-1 for a small x
1066 0.013765762467652909
1067
1068 The :func:`~math.erf` function computes a probability integral or `Gaussian
1069 error function <https://en.wikipedia.org/wiki/Error_function>`_. The
1070 complementary error function, :func:`~math.erfc`, is ``1 - erf(x)``:
1071
1072 .. doctest::
1073 :options: +SKIP
1074
1075 >>> from math import erf, erfc, sqrt
1076 >>> erf(1.0/sqrt(2.0)) # portion of normal distribution within 1 standard deviation
1077 0.682689492137086
1078 >>> erfc(1.0/sqrt(2.0)) # portion of normal distribution outside 1 standard deviation
1079 0.31731050786291404
1080 >>> erf(1.0/sqrt(2.0)) + erfc(1.0/sqrt(2.0))
1081 1.0
1082
1083 The :func:`~math.gamma` function is a continuous extension of the factorial
1084 function. See https://en.wikipedia.org/wiki/Gamma_function for details. Because
1085 the function is related to factorials, it grows large even for small values of
1086 *x*, so there is also a :func:`~math.lgamma` function for computing the natural
1087 logarithm of the gamma function:
1088
1089 >>> from math import gamma, lgamma
1090 >>> gamma(7.0) # six factorial
1091 720.0
1092 >>> lgamma(801.0) # log(800 factorial)
1093 4551.950730698041
1094
1095 (Contributed by Mark Dickinson.)
1096
1097 abc
1098 ---
1099
1100 The :mod:`abc` module now supports :func:`~abc.abstractclassmethod` and
1101 :func:`~abc.abstractstaticmethod`.
1102
1103 These tools make it possible to define an :term:`abstract base class` that
1104 requires a particular :func:`classmethod` or :func:`staticmethod` to be
1105 implemented::
1106
1107 class Temperature(metaclass=abc.ABCMeta):
1108 @abc.abstractclassmethod
1109 def from_fahrenheit(cls, t):
1110 ...
1111 @abc.abstractclassmethod
1112 def from_celsius(cls, t):
1113 ...
1114
1115 (Patch submitted by Daniel Urban; :issue:`5867`.)
1116
1117 io
1118 --
1119
1120 The :class:`io.BytesIO` has a new method, :meth:`~io.BytesIO.getbuffer`, which
1121 provides functionality similar to :func:`memoryview`. It creates an editable
1122 view of the data without making a copy. The buffer's random access and support
1123 for slice notation are well-suited to in-place editing::
1124
1125 >>> REC_LEN, LOC_START, LOC_LEN = 34, 7, 11
1126
1127 >>> def change_location(buffer, record_number, location):
1128 ... start = record_number * REC_LEN + LOC_START
1129 ... buffer[start: start+LOC_LEN] = location
1130
1131 >>> import io
1132
1133 >>> byte_stream = io.BytesIO(
1134 ... b'G3805 storeroom Main chassis '
1135 ... b'X7899 shipping Reserve cog '
1136 ... b'L6988 receiving Primary sprocket'
1137 ... )
1138 >>> buffer = byte_stream.getbuffer()
1139 >>> change_location(buffer, 1, b'warehouse ')
1140 >>> change_location(buffer, 0, b'showroom ')
1141 >>> print(byte_stream.getvalue())
1142 b'G3805 showroom Main chassis '
1143 b'X7899 warehouse Reserve cog '
1144 b'L6988 receiving Primary sprocket'
1145
1146 (Contributed by Antoine Pitrou in :issue:`5506`.)
1147
1148 reprlib
1149 -------
1150
1151 When writing a :meth:`~object.__repr__` method for a custom container, it is easy to
1152 forget to handle the case where a member refers back to the container itself.
1153 Python's builtin objects such as :class:`list` and :class:`set` handle
1154 self-reference by displaying "..." in the recursive part of the representation
1155 string.
1156
1157 To help write such :meth:`~object.__repr__` methods, the :mod:`reprlib` module has a new
1158 decorator, :func:`~reprlib.recursive_repr`, for detecting recursive calls to
1159 :meth:`!__repr__` and substituting a placeholder string instead::
1160
1161 >>> class MyList(list):
1162 ... @recursive_repr()
1163 ... def __repr__(self):
1164 ... return '<' + '|'.join(map(repr, self)) + '>'
1165 ...
1166 >>> m = MyList('abc')
1167 >>> m.append(m)
1168 >>> m.append('x')
1169 >>> print(m)
1170 <'a'|'b'|'c'|...|'x'>
1171
1172 (Contributed by Raymond Hettinger in :issue:`9826` and :issue:`9840`.)
1173
1174 logging
1175 -------
1176
1177 In addition to dictionary-based configuration described above, the
1178 :mod:`logging` package has many other improvements.
1179
1180 The logging documentation has been augmented by a :ref:`basic tutorial
1181 <logging-basic-tutorial>`\, an :ref:`advanced tutorial
1182 <logging-advanced-tutorial>`\, and a :ref:`cookbook <logging-cookbook>` of
1183 logging recipes. These documents are the fastest way to learn about logging.
1184
1185 The :func:`logging.basicConfig` set-up function gained a *style* argument to
1186 support three different types of string formatting. It defaults to "%" for
1187 traditional %-formatting, can be set to "{" for the new :meth:`str.format` style, or
1188 can be set to "$" for the shell-style formatting provided by
1189 :class:`string.Template`. The following three configurations are equivalent::
1190
1191 >>> from logging import basicConfig
1192 >>> basicConfig(style='%', format="%(name)s -> %(levelname)s: %(message)s")
1193 >>> basicConfig(style='{', format="{name} -> {levelname} {message}")
1194 >>> basicConfig(style='$', format="$name -> $levelname: $message")
1195
1196 If no configuration is set-up before a logging event occurs, there is now a
1197 default configuration using a :class:`~logging.StreamHandler` directed to
1198 :data:`sys.stderr` for events of ``WARNING`` level or higher. Formerly, an
1199 event occurring before a configuration was set-up would either raise an
1200 exception or silently drop the event depending on the value of
1201 :data:`logging.raiseExceptions`. The new default handler is stored in
1202 :data:`logging.lastResort`.
1203
1204 The use of filters has been simplified. Instead of creating a
1205 :class:`~logging.Filter` object, the predicate can be any Python callable that
1206 returns ``True`` or ``False``.
1207
1208 There were a number of other improvements that add flexibility and simplify
1209 configuration. See the module documentation for a full listing of changes in
1210 Python 3.2.
1211
1212 csv
1213 ---
1214
1215 The :mod:`csv` module now supports a new dialect, :class:`~csv.unix_dialect`,
1216 which applies quoting for all fields and a traditional Unix style with ``'\n'`` as
1217 the line terminator. The registered dialect name is ``unix``.
1218
1219 The :class:`csv.DictWriter` has a new method,
1220 :meth:`~csv.DictWriter.writeheader` for writing-out an initial row to document
1221 the field names::
1222
1223 >>> import csv, sys
1224 >>> w = csv.DictWriter(sys.stdout, ['name', 'dept'], dialect='unix')
1225 >>> w.writeheader()
1226 "name","dept"
1227 >>> w.writerows([
1228 ... {'name': 'tom', 'dept': 'accounting'},
1229 ... {'name': 'susan', 'dept': 'Salesl'}])
1230 "tom","accounting"
1231 "susan","sales"
1232
1233 (New dialect suggested by Jay Talbot in :issue:`5975`, and the new method
1234 suggested by Ed Abraham in :issue:`1537721`.)
1235
1236 contextlib
1237 ----------
1238
1239 There is a new and slightly mind-blowing tool
1240 :class:`~contextlib.ContextDecorator` that is helpful for creating a
1241 :term:`context manager` that does double duty as a function decorator.
1242
1243 As a convenience, this new functionality is used by
1244 :func:`~contextlib.contextmanager` so that no extra effort is needed to support
1245 both roles.
1246
1247 The basic idea is that both context managers and function decorators can be used
1248 for pre-action and post-action wrappers. Context managers wrap a group of
1249 statements using a :keyword:`with` statement, and function decorators wrap a
1250 group of statements enclosed in a function. So, occasionally there is a need to
1251 write a pre-action or post-action wrapper that can be used in either role.
1252
1253 For example, it is sometimes useful to wrap functions or groups of statements
1254 with a logger that can track the time of entry and time of exit. Rather than
1255 writing both a function decorator and a context manager for the task, the
1256 :func:`~contextlib.contextmanager` provides both capabilities in a single
1257 definition::
1258
1259 from contextlib import contextmanager
1260 import logging
1261
1262 logging.basicConfig(level=logging.INFO)
1263
1264 @contextmanager
1265 def track_entry_and_exit(name):
1266 logging.info('Entering: %s', name)
1267 yield
1268 logging.info('Exiting: %s', name)
1269
1270 Formerly, this would have only been usable as a context manager::
1271
1272 with track_entry_and_exit('widget loader'):
1273 print('Some time consuming activity goes here')
1274 load_widget()
1275
1276 Now, it can be used as a decorator as well::
1277
1278 @track_entry_and_exit('widget loader')
1279 def activity():
1280 print('Some time consuming activity goes here')
1281 load_widget()
1282
1283 Trying to fulfill two roles at once places some limitations on the technique.
1284 Context managers normally have the flexibility to return an argument usable by
1285 a :keyword:`with` statement, but there is no parallel for function decorators.
1286
1287 In the above example, there is not a clean way for the *track_entry_and_exit*
1288 context manager to return a logging instance for use in the body of enclosed
1289 statements.
1290
1291 (Contributed by Michael Foord in :issue:`9110`.)
1292
1293 decimal and fractions
1294 ---------------------
1295
1296 Mark Dickinson crafted an elegant and efficient scheme for assuring that
1297 different numeric datatypes will have the same hash value whenever their actual
1298 values are equal (:issue:`8188`)::
1299
1300 assert hash(Fraction(3, 2)) == hash(1.5) == \
1301 hash(Decimal("1.5")) == hash(complex(1.5, 0))
1302
1303 Some of the hashing details are exposed through a new attribute,
1304 :data:`sys.hash_info`, which describes the bit width of the hash value, the
1305 prime modulus, the hash values for *infinity* and *nan*, and the multiplier
1306 used for the imaginary part of a number:
1307
1308 >>> sys.hash_info # doctest: +SKIP
1309 sys.hash_info(width=64, modulus=2305843009213693951, inf=314159, nan=0, imag=1000003)
1310
1311 An early decision to limit the interoperability of various numeric types has
1312 been relaxed. It is still unsupported (and ill-advised) to have implicit
1313 mixing in arithmetic expressions such as ``Decimal('1.1') + float('1.1')``
1314 because the latter loses information in the process of constructing the binary
1315 float. However, since existing floating-point value can be converted losslessly
1316 to either a decimal or rational representation, it makes sense to add them to
1317 the constructor and to support mixed-type comparisons.
1318
1319 * The :class:`decimal.Decimal` constructor now accepts :class:`float` objects
1320 directly so there in no longer a need to use the :meth:`~decimal.Decimal.from_float`
1321 method (:issue:`8257`).
1322
1323 * Mixed type comparisons are now fully supported so that
1324 :class:`~decimal.Decimal` objects can be directly compared with :class:`float`
1325 and :class:`fractions.Fraction` (:issue:`2531` and :issue:`8188`).
1326
1327 Similar changes were made to :class:`fractions.Fraction` so that the
1328 :meth:`~fractions.Fraction.from_float` and :meth:`~fractions.Fraction.from_decimal`
1329 methods are no longer needed (:issue:`8294`):
1330
1331 >>> from decimal import Decimal
1332 >>> from fractions import Fraction
1333 >>> Decimal(1.1)
1334 Decimal('1.100000000000000088817841970012523233890533447265625')
1335 >>> Fraction(1.1)
1336 Fraction(2476979795053773, 2251799813685248)
1337
1338 Another useful change for the :mod:`decimal` module is that the
1339 :attr:`Context.clamp <decimal.Context.clamp>` attribute is now public. This is useful in creating
1340 contexts that correspond to the decimal interchange formats specified in IEEE
1341 754 (see :issue:`8540`).
1342
1343 (Contributed by Mark Dickinson and Raymond Hettinger.)
1344
1345 ftp
1346 ---
1347
1348 The :class:`ftplib.FTP` class now supports the context management protocol to
1349 unconditionally consume :exc:`socket.error` exceptions and to close the FTP
1350 connection when done::
1351
1352 >>> from ftplib import FTP
1353 >>> with FTP("ftp1.at.proftpd.org") as ftp:
1354 ftp.login()
1355 ftp.dir()
1356
1357 '230 Anonymous login ok, restrictions apply.'
1358 dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 .
1359 dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 ..
1360 dr-xr-xr-x 5 ftp ftp 4096 May 6 10:43 CentOS
1361 dr-xr-xr-x 3 ftp ftp 18 Jul 10 2008 Fedora
1362
1363 Other file-like objects such as :class:`mmap.mmap` and :func:`fileinput.input`
1364 also grew auto-closing context managers::
1365
1366 with fileinput.input(files=('log1.txt', 'log2.txt')) as f:
1367 for line in f:
1368 process(line)
1369
1370 (Contributed by Tarek Ziadé and Giampaolo Rodolà in :issue:`4972`, and
1371 by Georg Brandl in :issue:`8046` and :issue:`1286`.)
1372
1373 The :class:`~ftplib.FTP_TLS` class now accepts a *context* parameter, which is a
1374 :class:`ssl.SSLContext` object allowing bundling SSL configuration options,
1375 certificates and private keys into a single (potentially long-lived) structure.
1376
1377 (Contributed by Giampaolo Rodolà; :issue:`8806`.)
1378
1379 popen
1380 -----
1381
1382 The :func:`os.popen` and :func:`subprocess.Popen` functions now support
1383 :keyword:`with` statements for auto-closing of the file descriptors.
1384
1385 (Contributed by Antoine Pitrou and Brian Curtin in :issue:`7461` and
1386 :issue:`10554`.)
1387
1388 select
1389 ------
1390
1391 The :mod:`select` module now exposes a new, constant attribute,
1392 :const:`~select.PIPE_BUF`, which gives the minimum number of bytes which are
1393 guaranteed not to block when :func:`select.select` says a pipe is ready
1394 for writing.
1395
1396 >>> import select
1397 >>> select.PIPE_BUF # doctest: +SKIP
1398 512
1399
1400 (Available on Unix systems. Patch by Sébastien Sablé in :issue:`9862`)
1401
1402 gzip and zipfile
1403 ----------------
1404
1405 :class:`gzip.GzipFile` now implements the :class:`io.BufferedIOBase`
1406 :term:`abstract base class` (except for ``truncate()``). It also has a
1407 :meth:`~gzip.GzipFile.peek` method and supports unseekable as well as
1408 zero-padded file objects.
1409
1410 The :mod:`gzip` module also gains the :func:`~gzip.compress` and
1411 :func:`~gzip.decompress` functions for easier in-memory compression and
1412 decompression. Keep in mind that text needs to be encoded as :class:`bytes`
1413 before compressing and decompressing:
1414
1415 >>> import gzip
1416 >>> s = 'Three shall be the number thou shalt count, '
1417 >>> s += 'and the number of the counting shall be three'
1418 >>> b = s.encode() # convert to utf-8
1419 >>> len(b)
1420 89
1421 >>> c = gzip.compress(b)
1422 >>> len(c)
1423 77
1424 >>> gzip.decompress(c).decode()[:42] # decompress and convert to text
1425 'Three shall be the number thou shalt count'
1426
1427 (Contributed by Anand B. Pillai in :issue:`3488`; and by Antoine Pitrou, Nir
1428 Aides and Brian Curtin in :issue:`9962`, :issue:`1675951`, :issue:`7471` and
1429 :issue:`2846`.)
1430
1431 Also, the :class:`zipfile.ZipExtFile <zipfile.ZipFile.open>` class was reworked internally to represent
1432 files stored inside an archive. The new implementation is significantly faster
1433 and can be wrapped in an :class:`io.BufferedReader` object for more speedups. It
1434 also solves an issue where interleaved calls to *read* and *readline* gave the
1435 wrong results.
1436
1437 (Patch submitted by Nir Aides in :issue:`7610`.)
1438
1439 tarfile
1440 -------
1441
1442 The :class:`~tarfile.TarFile` class can now be used as a context manager. In
1443 addition, its :meth:`~tarfile.TarFile.add` method has a new option, *filter*,
1444 that controls which files are added to the archive and allows the file metadata
1445 to be edited.
1446
1447 The new *filter* option replaces the older, less flexible *exclude* parameter
1448 which is now deprecated. If specified, the optional *filter* parameter needs to
1449 be a :term:`keyword argument`. The user-supplied filter function accepts a
1450 :class:`~tarfile.TarInfo` object and returns an updated
1451 :class:`~tarfile.TarInfo` object, or if it wants the file to be excluded, the
1452 function can return ``None``::
1453
1454 >>> import tarfile, glob
1455
1456 >>> def myfilter(tarinfo):
1457 ... if tarinfo.isfile(): # only save real files
1458 ... tarinfo.uname = 'monty' # redact the user name
1459 ... return tarinfo
1460
1461 >>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf:
1462 ... for filename in glob.glob('*.txt'):
1463 ... tf.add(filename, filter=myfilter)
1464 ... tf.list()
1465 -rw-r--r-- monty/501 902 2011-01-26 17:59:11 annotations.txt
1466 -rw-r--r-- monty/501 123 2011-01-26 17:59:11 general_questions.txt
1467 -rw-r--r-- monty/501 3514 2011-01-26 17:59:11 prion.txt
1468 -rw-r--r-- monty/501 124 2011-01-26 17:59:11 py_todo.txt
1469 -rw-r--r-- monty/501 1399 2011-01-26 17:59:11 semaphore_notes.txt
1470
1471 (Proposed by Tarek Ziadé and implemented by Lars Gustäbel in :issue:`6856`.)
1472
1473 hashlib
1474 -------
1475
1476 The :mod:`hashlib` module has two new constant attributes listing the hashing
1477 algorithms guaranteed to be present in all implementations and those available
1478 on the current implementation::
1479
1480 >>> import hashlib
1481
1482 >>> hashlib.algorithms_guaranteed
1483 {'sha1', 'sha224', 'sha384', 'sha256', 'sha512', 'md5'}
1484
1485 >>> hashlib.algorithms_available
1486 {'md2', 'SHA256', 'SHA512', 'dsaWithSHA', 'mdc2', 'SHA224', 'MD4', 'sha256',
1487 'sha512', 'ripemd160', 'SHA1', 'MDC2', 'SHA', 'SHA384', 'MD2',
1488 'ecdsa-with-SHA1','md4', 'md5', 'sha1', 'DSA-SHA', 'sha224',
1489 'dsaEncryption', 'DSA', 'RIPEMD160', 'sha', 'MD5', 'sha384'}
1490
1491 (Suggested by Carl Chenet in :issue:`7418`.)
1492
1493 ast
1494 ---
1495
1496 The :mod:`ast` module has a wonderful a general-purpose tool for safely
1497 evaluating expression strings using the Python literal
1498 syntax. The :func:`ast.literal_eval` function serves as a secure alternative to
1499 the builtin :func:`eval` function which is easily abused. Python 3.2 adds
1500 :class:`bytes` and :class:`set` literals to the list of supported types:
1501 strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and ``None``.
1502
1503 ::
1504
1505 >>> from ast import literal_eval
1506
1507 >>> request = "{'req': 3, 'func': 'pow', 'args': (2, 0.5)}"
1508 >>> literal_eval(request)
1509 {'args': (2, 0.5), 'req': 3, 'func': 'pow'}
1510
1511 >>> request = "os.system('do something harmful')"
1512 >>> literal_eval(request)
1513 Traceback (most recent call last):
1514 ...
1515 ValueError: malformed node or string: <_ast.Call object at 0x101739a10>
1516
1517 (Implemented by Benjamin Peterson and Georg Brandl.)
1518
1519 os
1520 --
1521
1522 Different operating systems use various encodings for filenames and environment
1523 variables. The :mod:`os` module provides two new functions,
1524 :func:`~os.fsencode` and :func:`~os.fsdecode`, for encoding and decoding
1525 filenames:
1526
1527 >>> import os
1528 >>> filename = 'Sehenswürdigkeiten'
1529 >>> os.fsencode(filename)
1530 b'Sehensw\xc3\xbcrdigkeiten'
1531
1532 Some operating systems allow direct access to encoded bytes in the
1533 environment. If so, the :const:`os.supports_bytes_environ` constant will be
1534 true.
1535
1536 For direct access to encoded environment variables (if available),
1537 use the new :func:`os.getenvb` function or use :data:`os.environb`
1538 which is a bytes version of :data:`os.environ`.
1539
1540 (Contributed by Victor Stinner.)
1541
1542 shutil
1543 ------
1544
1545 The :func:`shutil.copytree` function has two new options:
1546
1547 * *ignore_dangling_symlinks*: when ``symlinks=False`` so that the function
1548 copies a file pointed to by a symlink, not the symlink itself. This option
1549 will silence the error raised if the file doesn't exist.
1550
1551 * *copy_function*: is a callable that will be used to copy files.
1552 :func:`shutil.copy2` is used by default.
1553
1554 (Contributed by Tarek Ziadé.)
1555
1556 In addition, the :mod:`shutil` module now supports :ref:`archiving operations
1557 <archiving-operations>` for zipfiles, uncompressed tarfiles, gzipped tarfiles,
1558 and bzipped tarfiles. And there are functions for registering additional
1559 archiving file formats (such as xz compressed tarfiles or custom formats).
1560
1561 The principal functions are :func:`~shutil.make_archive` and
1562 :func:`~shutil.unpack_archive`. By default, both operate on the current
1563 directory (which can be set by :func:`os.chdir`) and on any sub-directories.
1564 The archive filename needs to be specified with a full pathname. The archiving
1565 step is non-destructive (the original files are left unchanged).
1566
1567 ::
1568
1569 >>> import shutil, pprint
1570
1571 >>> os.chdir('mydata') # change to the source directory
1572 >>> f = shutil.make_archive('/var/backup/mydata',
1573 ... 'zip') # archive the current directory
1574 >>> f # show the name of archive
1575 '/var/backup/mydata.zip'
1576 >>> os.chdir('tmp') # change to an unpacking
1577 >>> shutil.unpack_archive('/var/backup/mydata.zip') # recover the data
1578
1579 >>> pprint.pprint(shutil.get_archive_formats()) # display known formats
1580 [('bztar', "bzip2'ed tar-file"),
1581 ('gztar', "gzip'ed tar-file"),
1582 ('tar', 'uncompressed tar file'),
1583 ('zip', 'ZIP file')]
1584
1585 >>> shutil.register_archive_format( # register a new archive format
1586 ... name='xz',
1587 ... function=xz.compress, # callable archiving function
1588 ... extra_args=[('level', 8)], # arguments to the function
1589 ... description='xz compression'
1590 ... )
1591
1592 (Contributed by Tarek Ziadé.)
1593
1594 sqlite3
1595 -------
1596
1597 The :mod:`sqlite3` module was updated to pysqlite version 2.6.0. It has two new capabilities.
1598
1599 * The :attr:`!sqlite3.Connection.in_transit` attribute is true if there is an
1600 active transaction for uncommitted changes.
1601
1602 * The :meth:`sqlite3.Connection.enable_load_extension` and
1603 :meth:`sqlite3.Connection.load_extension` methods allows you to load SQLite
1604 extensions from ".so" files. One well-known extension is the fulltext-search
1605 extension distributed with SQLite.
1606
1607 (Contributed by R. David Murray and Shashwat Anand; :issue:`8845`.)
1608
1609 html
1610 ----
1611
1612 A new :mod:`html` module was introduced with only a single function,
1613 :func:`~html.escape`, which is used for escaping reserved characters from HTML
1614 markup:
1615
1616 >>> import html
1617 >>> html.escape('x > 2 && x < 7')
1618 'x &gt; 2 &amp;&amp; x &lt; 7'
1619
1620 socket
1621 ------
1622
1623 The :mod:`socket` module has two new improvements.
1624
1625 * Socket objects now have a :meth:`~socket.socket.detach` method which puts
1626 the socket into closed state without actually closing the underlying file
1627 descriptor. The latter can then be reused for other purposes.
1628 (Added by Antoine Pitrou; :issue:`8524`.)
1629
1630 * :func:`socket.create_connection` now supports the context management protocol
1631 to unconditionally consume :exc:`socket.error` exceptions and to close the
1632 socket when done.
1633 (Contributed by Giampaolo Rodolà; :issue:`9794`.)
1634
1635 ssl
1636 ---
1637
1638 The :mod:`ssl` module added a number of features to satisfy common requirements
1639 for secure (encrypted, authenticated) internet connections:
1640
1641 * A new class, :class:`~ssl.SSLContext`, serves as a container for persistent
1642 SSL data, such as protocol settings, certificates, private keys, and various
1643 other options. It includes a :meth:`~ssl.SSLContext.wrap_socket` for creating
1644 an SSL socket from an SSL context.
1645
1646 * A new function, :func:`!ssl.match_hostname`, supports server identity
1647 verification for higher-level protocols by implementing the rules of HTTPS
1648 (from :rfc:`2818`) which are also suitable for other protocols.
1649
1650 * The :func:`ssl.wrap_socket() <ssl.SSLContext.wrap_socket>` constructor function now takes a *ciphers*
1651 argument. The *ciphers* string lists the allowed encryption algorithms using
1652 the format described in the `OpenSSL documentation
1653 <https://docs.openssl.org/1.0.2/man1/ciphers/#cipher-list-format>`__.
1654
1655 * When linked against recent versions of OpenSSL, the :mod:`ssl` module now
1656 supports the Server Name Indication extension to the TLS protocol, allowing
1657 multiple "virtual hosts" using different certificates on a single IP port.
1658 This extension is only supported in client mode, and is activated by passing
1659 the *server_hostname* argument to :meth:`ssl.SSLContext.wrap_socket`.
1660
1661 * Various options have been added to the :mod:`ssl` module, such as
1662 :data:`~ssl.OP_NO_SSLv2` which disables the insecure and obsolete SSLv2
1663 protocol.
1664
1665 * The extension now loads all the OpenSSL ciphers and digest algorithms. If
1666 some SSL certificates cannot be verified, they are reported as an "unknown
1667 algorithm" error.
1668
1669 * The version of OpenSSL being used is now accessible using the module
1670 attributes :const:`ssl.OPENSSL_VERSION` (a string),
1671 :const:`ssl.OPENSSL_VERSION_INFO` (a 5-tuple), and
1672 :const:`ssl.OPENSSL_VERSION_NUMBER` (an integer).
1673
1674 (Contributed by Antoine Pitrou in :issue:`8850`, :issue:`1589`, :issue:`8322`,
1675 :issue:`5639`, :issue:`4870`, :issue:`8484`, and :issue:`8321`.)
1676
1677 nntp
1678 ----
1679
1680 The :mod:`!nntplib` module has a revamped implementation with better bytes and
1681 text semantics as well as more practical APIs. These improvements break
1682 compatibility with the nntplib version in Python 3.1, which was partly
1683 dysfunctional in itself.
1684
1685 Support for secure connections through both implicit (using
1686 :class:`!nntplib.NNTP_SSL`) and explicit (using :meth:`!nntplib.NNTP.starttls`)
1687 TLS has also been added.
1688
1689 (Contributed by Antoine Pitrou in :issue:`9360` and Andrew Vant in :issue:`1926`.)
1690
1691 certificates
1692 ------------
1693
1694 :class:`http.client.HTTPSConnection`, :class:`urllib.request.HTTPSHandler`
1695 and :func:`urllib.request.urlopen` now take optional arguments to allow for
1696 server certificate checking against a set of Certificate Authorities,
1697 as recommended in public uses of HTTPS.
1698
1699 (Added by Antoine Pitrou, :issue:`9003`.)
1700
1701 imaplib
1702 -------
1703
1704 Support for explicit TLS on standard IMAP4 connections has been added through
1705 the new :mod:`imaplib.IMAP4.starttls` method.
1706
1707 (Contributed by Lorenzo M. Catucci and Antoine Pitrou, :issue:`4471`.)
1708
1709 http.client
1710 -----------
1711
1712 There were a number of small API improvements in the :mod:`http.client` module.
1713 The old-style HTTP 0.9 simple responses are no longer supported and the *strict*
1714 parameter is deprecated in all classes.
1715
1716 The :class:`~http.client.HTTPConnection` and
1717 :class:`~http.client.HTTPSConnection` classes now have a *source_address*
1718 parameter for a (host, port) tuple indicating where the HTTP connection is made
1719 from.
1720
1721 Support for certificate checking and HTTPS virtual hosts were added to
1722 :class:`~http.client.HTTPSConnection`.
1723
1724 The :meth:`~http.client.HTTPConnection.request` method on connection objects
1725 allowed an optional *body* argument so that a :term:`file object` could be used
1726 to supply the content of the request. Conveniently, the *body* argument now
1727 also accepts an :term:`iterable` object so long as it includes an explicit
1728 ``Content-Length`` header. This extended interface is much more flexible than
1729 before.
1730
1731 To establish an HTTPS connection through a proxy server, there is a new
1732 :meth:`~http.client.HTTPConnection.set_tunnel` method that sets the host and
1733 port for HTTP Connect tunneling.
1734
1735 To match the behavior of :mod:`http.server`, the HTTP client library now also
1736 encodes headers with ISO-8859-1 (Latin-1) encoding. It was already doing that
1737 for incoming headers, so now the behavior is consistent for both incoming and
1738 outgoing traffic. (See work by Armin Ronacher in :issue:`10980`.)
1739
1740 unittest
1741 --------
1742
1743 The unittest module has a number of improvements supporting test discovery for
1744 packages, easier experimentation at the interactive prompt, new testcase
1745 methods, improved diagnostic messages for test failures, and better method
1746 names.
1747
1748 * The command-line call ``python -m unittest`` can now accept file paths
1749 instead of module names for running specific tests (:issue:`10620`). The new
1750 test discovery can find tests within packages, locating any test importable
1751 from the top-level directory. The top-level directory can be specified with
1752 the ``-t`` option, a pattern for matching files with ``-p``, and a directory to
1753 start discovery with ``-s``:
1754
1755 .. code-block:: shell-session
1756
1757 $ python -m unittest discover -s my_proj_dir -p _test.py
1758
1759 (Contributed by Michael Foord.)
1760
1761 * Experimentation at the interactive prompt is now easier because the
1762 :class:`unittest.TestCase` class can now be instantiated without
1763 arguments:
1764
1765 >>> from unittest import TestCase
1766 >>> TestCase().assertEqual(pow(2, 3), 8)
1767
1768 (Contributed by Michael Foord.)
1769
1770 * The :mod:`unittest` module has two new methods,
1771 :meth:`~unittest.TestCase.assertWarns` and
1772 :meth:`~unittest.TestCase.assertWarnsRegex` to verify that a given warning type
1773 is triggered by the code under test::
1774
1775 with self.assertWarns(DeprecationWarning):
1776 legacy_function('XYZ')
1777
1778 (Contributed by Antoine Pitrou, :issue:`9754`.)
1779
1780 Another new method, :meth:`~unittest.TestCase.assertCountEqual` is used to
1781 compare two iterables to determine if their element counts are equal (whether
1782 the same elements are present with the same number of occurrences regardless
1783 of order)::
1784
1785 def test_anagram(self):
1786 self.assertCountEqual('algorithm', 'logarithm')
1787
1788 (Contributed by Raymond Hettinger.)
1789
1790 * A principal feature of the unittest module is an effort to produce meaningful
1791 diagnostics when a test fails. When possible, the failure is recorded along
1792 with a diff of the output. This is especially helpful for analyzing log files
1793 of failed test runs. However, since diffs can sometime be voluminous, there is
1794 a new :attr:`~unittest.TestCase.maxDiff` attribute that sets maximum length of
1795 diffs displayed.
1796
1797 * In addition, the method names in the module have undergone a number of clean-ups.
1798
1799 For example, :meth:`~unittest.TestCase.assertRegex` is the new name for
1800 :meth:`!assertRegexpMatches` which was misnamed because the
1801 test uses :func:`re.search`, not :func:`re.match`. Other methods using
1802 regular expressions are now named using short form "Regex" in preference to
1803 "Regexp" -- this matches the names used in other unittest implementations,
1804 matches Python's old name for the :mod:`re` module, and it has unambiguous
1805 camel-casing.
1806
1807 (Contributed by Raymond Hettinger and implemented by Ezio Melotti.)
1808
1809 * To improve consistency, some long-standing method aliases are being
1810 deprecated in favor of the preferred names:
1811
1812 =============================== ==============================
1813 Old Name Preferred Name
1814 =============================== ==============================
1815 :meth:`!assert_` :meth:`.assertTrue`
1816 :meth:`!assertEquals` :meth:`.assertEqual`
1817 :meth:`!assertNotEquals` :meth:`.assertNotEqual`
1818 :meth:`!assertAlmostEquals` :meth:`.assertAlmostEqual`
1819 :meth:`!assertNotAlmostEquals` :meth:`.assertNotAlmostEqual`
1820 =============================== ==============================
1821
1822 Likewise, the ``TestCase.fail*`` methods deprecated in Python 3.1 are expected
1823 to be removed in Python 3.3.
1824
1825 (Contributed by Ezio Melotti; :issue:`9424`.)
1826
1827 * The :meth:`!assertDictContainsSubset` method was deprecated
1828 because it was misimplemented with the arguments in the wrong order. This
1829 created hard-to-debug optical illusions where tests like
1830 ``TestCase().assertDictContainsSubset({'a':1, 'b':2}, {'a':1})`` would fail.
1831
1832 (Contributed by Raymond Hettinger.)
1833
1834 random
1835 ------
1836
1837 The integer methods in the :mod:`random` module now do a better job of producing
1838 uniform distributions. Previously, they computed selections with
1839 ``int(n*random())`` which had a slight bias whenever *n* was not a power of two.
1840 Now, multiple selections are made from a range up to the next power of two and a
1841 selection is kept only when it falls within the range ``0 <= x < n``. The
1842 functions and methods affected are :func:`~random.randrange`,
1843 :func:`~random.randint`, :func:`~random.choice`, :func:`~random.shuffle` and
1844 :func:`~random.sample`.
1845
1846 (Contributed by Raymond Hettinger; :issue:`9025`.)
1847
1848 poplib
1849 ------
1850
1851 :class:`~poplib.POP3_SSL` class now accepts a *context* parameter, which is a
1852 :class:`ssl.SSLContext` object allowing bundling SSL configuration options,
1853 certificates and private keys into a single (potentially long-lived)
1854 structure.
1855
1856 (Contributed by Giampaolo Rodolà; :issue:`8807`.)
1857
1858 asyncore
1859 --------
1860
1861 :class:`!asyncore.dispatcher` now provides a
1862 :meth:`!handle_accepted` method
1863 returning a ``(sock, addr)`` pair which is called when a connection has actually
1864 been established with a new remote endpoint. This is supposed to be used as a
1865 replacement for old :meth:`!handle_accept` and avoids
1866 the user to call :meth:`!accept` directly.
1867
1868 (Contributed by Giampaolo Rodolà; :issue:`6706`.)
1869
1870 tempfile
1871 --------
1872
1873 The :mod:`tempfile` module has a new context manager,
1874 :class:`~tempfile.TemporaryDirectory` which provides easy deterministic
1875 cleanup of temporary directories::
1876
1877 with tempfile.TemporaryDirectory() as tmpdirname:
1878 print('created temporary dir:', tmpdirname)
1879
1880 (Contributed by Neil Schemenauer and Nick Coghlan; :issue:`5178`.)
1881
1882 inspect
1883 -------
1884
1885 * The :mod:`inspect` module has a new function
1886 :func:`~inspect.getgeneratorstate` to easily identify the current state of a
1887 generator-iterator::
1888
1889 >>> from inspect import getgeneratorstate
1890 >>> def gen():
1891 ... yield 'demo'
1892 ...
1893 >>> g = gen()
1894 >>> getgeneratorstate(g)
1895 'GEN_CREATED'
1896 >>> next(g)
1897 'demo'
1898 >>> getgeneratorstate(g)
1899 'GEN_SUSPENDED'
1900 >>> next(g, None)
1901 >>> getgeneratorstate(g)
1902 'GEN_CLOSED'
1903
1904 (Contributed by Rodolpho Eckhardt and Nick Coghlan, :issue:`10220`.)
1905
1906 * To support lookups without the possibility of activating a dynamic attribute,
1907 the :mod:`inspect` module has a new function, :func:`~inspect.getattr_static`.
1908 Unlike :func:`hasattr`, this is a true read-only search, guaranteed not to
1909 change state while it is searching::
1910
1911 >>> class A:
1912 ... @property
1913 ... def f(self):
1914 ... print('Running')
1915 ... return 10
1916 ...
1917 >>> a = A()
1918 >>> getattr(a, 'f')
1919 Running
1920 10
1921 >>> inspect.getattr_static(a, 'f')
1922 <property object at 0x1022bd788>
1923
1924 (Contributed by Michael Foord.)
1925
1926 pydoc
1927 -----
1928
1929 The :mod:`pydoc` module now provides a much-improved web server interface, as
1930 well as a new command-line option ``-b`` to automatically open a browser window
1931 to display that server:
1932
1933 .. code-block:: shell-session
1934
1935 $ pydoc3.2 -b
1936
1937 (Contributed by Ron Adam; :issue:`2001`.)
1938
1939 dis
1940 ---
1941
1942 The :mod:`dis` module gained two new functions for inspecting code,
1943 :func:`~dis.code_info` and :func:`~dis.show_code`. Both provide detailed code
1944 object information for the supplied function, method, source code string or code
1945 object. The former returns a string and the latter prints it::
1946
1947 >>> import dis, random
1948 >>> dis.show_code(random.choice)
1949 Name: choice
1950 Filename: /Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/random.py
1951 Argument count: 2
1952 Kw-only arguments: 0
1953 Number of locals: 3
1954 Stack size: 11
1955 Flags: OPTIMIZED, NEWLOCALS, NOFREE
1956 Constants:
1957 0: 'Choose a random element from a non-empty sequence.'
1958 1: 'Cannot choose from an empty sequence'
1959 Names:
1960 0: _randbelow
1961 1: len
1962 2: ValueError
1963 3: IndexError
1964 Variable names:
1965 0: self
1966 1: seq
1967 2: i
1968
1969 In addition, the :func:`~dis.dis` function now accepts string arguments
1970 so that the common idiom ``dis(compile(s, '', 'eval'))`` can be shortened
1971 to ``dis(s)``::
1972
1973 >>> dis('3*x+1 if x%2==1 else x//2')
1974 1 0 LOAD_NAME 0 (x)
1975 3 LOAD_CONST 0 (2)
1976 6 BINARY_MODULO
1977 7 LOAD_CONST 1 (1)
1978 10 COMPARE_OP 2 (==)
1979 13 POP_JUMP_IF_FALSE 28
1980 16 LOAD_CONST 2 (3)
1981 19 LOAD_NAME 0 (x)
1982 22 BINARY_MULTIPLY
1983 23 LOAD_CONST 1 (1)
1984 26 BINARY_ADD
1985 27 RETURN_VALUE
1986 >> 28 LOAD_NAME 0 (x)
1987 31 LOAD_CONST 0 (2)
1988 34 BINARY_FLOOR_DIVIDE
1989 35 RETURN_VALUE
1990
1991 Taken together, these improvements make it easier to explore how CPython is
1992 implemented and to see for yourself what the language syntax does
1993 under-the-hood.
1994
1995 (Contributed by Nick Coghlan in :issue:`9147`.)
1996
1997 dbm
1998 ---
1999
2000 All database modules now support the :meth:`!get` and :meth:`!setdefault` methods.
2001
2002 (Suggested by Ray Allen in :issue:`9523`.)
2003
2004 ctypes
2005 ------
2006
2007 A new type, :class:`ctypes.c_ssize_t` represents the C :c:type:`ssize_t` datatype.
2008
2009 site
2010 ----
2011
2012 The :mod:`site` module has three new functions useful for reporting on the
2013 details of a given Python installation.
2014
2015 * :func:`~site.getsitepackages` lists all global site-packages directories.
2016
2017 * :func:`~site.getuserbase` reports on the user's base directory where data can
2018 be stored.
2019
2020 * :func:`~site.getusersitepackages` reveals the user-specific site-packages
2021 directory path.
2022
2023 ::
2024
2025 >>> import site
2026 >>> site.getsitepackages()
2027 ['/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages',
2028 '/Library/Frameworks/Python.framework/Versions/3.2/lib/site-python',
2029 '/Library/Python/3.2/site-packages']
2030 >>> site.getuserbase()
2031 '/Users/raymondhettinger/Library/Python/3.2'
2032 >>> site.getusersitepackages()
2033 '/Users/raymondhettinger/Library/Python/3.2/lib/python/site-packages'
2034
2035 Conveniently, some of site's functionality is accessible directly from the
2036 command-line:
2037
2038 .. code-block:: shell-session
2039
2040 $ python -m site --user-base
2041 /Users/raymondhettinger/.local
2042 $ python -m site --user-site
2043 /Users/raymondhettinger/.local/lib/python3.2/site-packages
2044
2045 (Contributed by Tarek Ziadé in :issue:`6693`.)
2046
2047 sysconfig
2048 ---------
2049
2050 The new :mod:`sysconfig` module makes it straightforward to discover
2051 installation paths and configuration variables that vary across platforms and
2052 installations.
2053
2054 The module offers access simple access functions for platform and version
2055 information:
2056
2057 * :func:`~sysconfig.get_platform` returning values like *linux-i586* or
2058 *macosx-10.6-ppc*.
2059 * :func:`~sysconfig.get_python_version` returns a Python version string
2060 such as "3.2".
2061
2062 It also provides access to the paths and variables corresponding to one of
2063 seven named schemes used by ``distutils``. Those include *posix_prefix*,
2064 *posix_home*, *posix_user*, *nt*, *nt_user*, *os2*, *os2_home*:
2065
2066 * :func:`~sysconfig.get_paths` makes a dictionary containing installation paths
2067 for the current installation scheme.
2068 * :func:`~sysconfig.get_config_vars` returns a dictionary of platform specific
2069 variables.
2070
2071 There is also a convenient command-line interface:
2072
2073 .. code-block:: doscon
2074
2075 C:\Python32>python -m sysconfig
2076 Platform: "win32"
2077 Python version: "3.2"
2078 Current installation scheme: "nt"
2079
2080 Paths:
2081 data = "C:\Python32"
2082 include = "C:\Python32\Include"
2083 platinclude = "C:\Python32\Include"
2084 platlib = "C:\Python32\Lib\site-packages"
2085 platstdlib = "C:\Python32\Lib"
2086 purelib = "C:\Python32\Lib\site-packages"
2087 scripts = "C:\Python32\Scripts"
2088 stdlib = "C:\Python32\Lib"
2089
2090 Variables:
2091 BINDIR = "C:\Python32"
2092 BINLIBDEST = "C:\Python32\Lib"
2093 EXE = ".exe"
2094 INCLUDEPY = "C:\Python32\Include"
2095 LIBDEST = "C:\Python32\Lib"
2096 SO = ".pyd"
2097 VERSION = "32"
2098 abiflags = ""
2099 base = "C:\Python32"
2100 exec_prefix = "C:\Python32"
2101 platbase = "C:\Python32"
2102 prefix = "C:\Python32"
2103 projectbase = "C:\Python32"
2104 py_version = "3.2"
2105 py_version_nodot = "32"
2106 py_version_short = "3.2"
2107 srcdir = "C:\Python32"
2108 userbase = "C:\Documents and Settings\Raymond\Application Data\Python"
2109
2110 (Moved out of Distutils by Tarek Ziadé.)
2111
2112 pdb
2113 ---
2114
2115 The :mod:`pdb` debugger module gained a number of usability improvements:
2116
2117 * :file:`pdb.py` now has a ``-c`` option that executes commands as given in a
2118 :file:`.pdbrc` script file.
2119 * A :file:`.pdbrc` script file can contain ``continue`` and ``next`` commands
2120 that continue debugging.
2121 * The :class:`~pdb.Pdb` class constructor now accepts a *nosigint* argument.
2122 * New commands: ``l(list)``, ``ll(long list)`` and ``source`` for
2123 listing source code.
2124 * New commands: ``display`` and ``undisplay`` for showing or hiding
2125 the value of an expression if it has changed.
2126 * New command: ``interact`` for starting an interactive interpreter containing
2127 the global and local names found in the current scope.
2128 * Breakpoints can be cleared by breakpoint number.
2129
2130 (Contributed by Georg Brandl, Antonio Cuni and Ilya Sandler.)
2131
2132 configparser
2133 ------------
2134
2135 The :mod:`configparser` module was modified to improve usability and
2136 predictability of the default parser and its supported INI syntax. The old
2137 :class:`!ConfigParser` class was removed in favor of :class:`!SafeConfigParser`
2138 which has in turn been renamed to :class:`~configparser.ConfigParser`. Support
2139 for inline comments is now turned off by default and section or option
2140 duplicates are not allowed in a single configuration source.
2141
2142 Config parsers gained a new API based on the mapping protocol::
2143
2144 >>> parser = ConfigParser()
2145 >>> parser.read_string("""
2146 ... [DEFAULT]
2147 ... location = upper left
2148 ... visible = yes
2149 ... editable = no
2150 ... color = blue
2151 ...
2152 ... [main]
2153 ... title = Main Menu
2154 ... color = green
2155 ...
2156 ... [options]
2157 ... title = Options
2158 ... """)
2159 >>> parser['main']['color']
2160 'green'
2161 >>> parser['main']['editable']
2162 'no'
2163 >>> section = parser['options']
2164 >>> section['title']
2165 'Options'
2166 >>> section['title'] = 'Options (editable: %(editable)s)'
2167 >>> section['title']
2168 'Options (editable: no)'
2169
2170 The new API is implemented on top of the classical API, so custom parser
2171 subclasses should be able to use it without modifications.
2172
2173 The INI file structure accepted by config parsers can now be customized. Users
2174 can specify alternative option/value delimiters and comment prefixes, change the
2175 name of the *DEFAULT* section or switch the interpolation syntax.
2176
2177 There is support for pluggable interpolation including an additional interpolation
2178 handler :class:`~configparser.ExtendedInterpolation`::
2179
2180 >>> parser = ConfigParser(interpolation=ExtendedInterpolation())
2181 >>> parser.read_dict({'buildout': {'directory': '/home/ambv/zope9'},
2182 ... 'custom': {'prefix': '/usr/local'}})
2183 >>> parser.read_string("""
2184 ... [buildout]
2185 ... parts =
2186 ... zope9
2187 ... instance
2188 ... find-links =
2189 ... ${buildout:directory}/downloads/dist
2190 ...
2191 ... [zope9]
2192 ... recipe = plone.recipe.zope9install
2193 ... location = /opt/zope
2194 ...
2195 ... [instance]
2196 ... recipe = plone.recipe.zope9instance
2197 ... zope9-location = ${zope9:location}
2198 ... zope-conf = ${custom:prefix}/etc/zope.conf
2199 ... """)
2200 >>> parser['buildout']['find-links']
2201 '\n/home/ambv/zope9/downloads/dist'
2202 >>> parser['instance']['zope-conf']
2203 '/usr/local/etc/zope.conf'
2204 >>> instance = parser['instance']
2205 >>> instance['zope-conf']
2206 '/usr/local/etc/zope.conf'
2207 >>> instance['zope9-location']
2208 '/opt/zope'
2209
2210 A number of smaller features were also introduced, like support for specifying
2211 encoding in read operations, specifying fallback values for get-functions, or
2212 reading directly from dictionaries and strings.
2213
2214 (All changes contributed by Łukasz Langa.)
2215
2216 .. XXX consider showing a difflib example
2217
2218 urllib.parse
2219 ------------
2220
2221 A number of usability improvements were made for the :mod:`urllib.parse` module.
2222
2223 The :func:`~urllib.parse.urlparse` function now supports `IPv6
2224 <https://en.wikipedia.org/wiki/IPv6>`_ addresses as described in :rfc:`2732`:
2225
2226 >>> import urllib.parse
2227 >>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') # doctest: +NORMALIZE_WHITESPACE
2228 ParseResult(scheme='http',
2229 netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]',
2230 path='/foo/',
2231 params='',
2232 query='',
2233 fragment='')
2234
2235 The :func:`~urllib.parse.urldefrag` function now returns a :term:`named tuple`::
2236
2237 >>> r = urllib.parse.urldefrag('http://python.org/about/#target')
2238 >>> r
2239 DefragResult(url='http://python.org/about/', fragment='target')
2240 >>> r[0]
2241 'http://python.org/about/'
2242 >>> r.fragment
2243 'target'
2244
2245 And, the :func:`~urllib.parse.urlencode` function is now much more flexible,
2246 accepting either a string or bytes type for the *query* argument. If it is a
2247 string, then the *safe*, *encoding*, and *error* parameters are sent to
2248 :func:`~urllib.parse.quote_plus` for encoding::
2249
2250 >>> urllib.parse.urlencode([
2251 ... ('type', 'telenovela'),
2252 ... ('name', '¿Dónde Está Elisa?')],
2253 ... encoding='latin-1')
2254 'type=telenovela&name=%BFD%F3nde+Est%E1+Elisa%3F'
2255
2256 As detailed in :ref:`parsing-ascii-encoded-bytes`, all the :mod:`urllib.parse`
2257 functions now accept ASCII-encoded byte strings as input, so long as they are
2258 not mixed with regular strings. If ASCII-encoded byte strings are given as
2259 parameters, the return types will also be an ASCII-encoded byte strings:
2260
2261 >>> urllib.parse.urlparse(b'http://www.python.org:80/about/') # doctest: +NORMALIZE_WHITESPACE
2262 ParseResultBytes(scheme=b'http', netloc=b'www.python.org:80',
2263 path=b'/about/', params=b'', query=b'', fragment=b'')
2264
2265 (Work by Nick Coghlan, Dan Mahn, and Senthil Kumaran in :issue:`2987`,
2266 :issue:`5468`, and :issue:`9873`.)
2267
2268 mailbox
2269 -------
2270
2271 Thanks to a concerted effort by R. David Murray, the :mod:`mailbox` module has
2272 been fixed for Python 3.2. The challenge was that mailbox had been originally
2273 designed with a text interface, but email messages are best represented with
2274 :class:`bytes` because various parts of a message may have different encodings.
2275
2276 The solution harnessed the :mod:`email` package's binary support for parsing
2277 arbitrary email messages. In addition, the solution required a number of API
2278 changes.
2279
2280 As expected, the :meth:`~mailbox.Mailbox.add` method for
2281 :class:`mailbox.Mailbox` objects now accepts binary input.
2282
2283 :class:`~io.StringIO` and text file input are deprecated. Also, string input
2284 will fail early if non-ASCII characters are used. Previously it would fail when
2285 the email was processed in a later step.
2286
2287 There is also support for binary output. The :meth:`~mailbox.Mailbox.get_file`
2288 method now returns a file in the binary mode (where it used to incorrectly set
2289 the file to text-mode). There is also a new :meth:`~mailbox.Mailbox.get_bytes`
2290 method that returns a :class:`bytes` representation of a message corresponding
2291 to a given *key*.
2292
2293 It is still possible to get non-binary output using the old API's
2294 :meth:`~mailbox.Mailbox.get_string` method, but that approach
2295 is not very useful. Instead, it is best to extract messages from
2296 a :class:`~mailbox.Message` object or to load them from binary input.
2297
2298 (Contributed by R. David Murray, with efforts from Steffen Daode Nurpmeso and an
2299 initial patch by Victor Stinner in :issue:`9124`.)
2300
2301 turtledemo
2302 ----------
2303
2304 The demonstration code for the :mod:`turtle` module was moved from the *Demo*
2305 directory to main library. It includes over a dozen sample scripts with
2306 lively displays. Being on :data:`sys.path`, it can now be run directly
2307 from the command-line:
2308
2309 .. code-block:: shell-session
2310
2311 $ python -m turtledemo
2312
2313 (Moved from the Demo directory by Alexander Belopolsky in :issue:`10199`.)
2314
2315 Multi-threading
2316 ===============
2317
2318 * The mechanism for serializing execution of concurrently running Python threads
2319 (generally known as the :term:`GIL` or Global Interpreter Lock) has
2320 been rewritten. Among the objectives were more predictable switching
2321 intervals and reduced overhead due to lock contention and the number of
2322 ensuing system calls. The notion of a "check interval" to allow thread
2323 switches has been abandoned and replaced by an absolute duration expressed in
2324 seconds. This parameter is tunable through :func:`sys.setswitchinterval`.
2325 It currently defaults to 5 milliseconds.
2326
2327 Additional details about the implementation can be read from a `python-dev
2328 mailing-list message
2329 <https://mail.python.org/pipermail/python-dev/2009-October/093321.html>`_
2330 (however, "priority requests" as exposed in this message have not been kept
2331 for inclusion).
2332
2333 (Contributed by Antoine Pitrou.)
2334
2335 * Regular and recursive locks now accept an optional *timeout* argument to their
2336 :meth:`~threading.Lock.acquire` method. (Contributed by Antoine Pitrou;
2337 :issue:`7316`.)
2338
2339 * Similarly, :meth:`threading.Semaphore.acquire` also gained a *timeout*
2340 argument. (Contributed by Torsten Landschoff; :issue:`850728`.)
2341
2342 * Regular and recursive lock acquisitions can now be interrupted by signals on
2343 platforms using Pthreads. This means that Python programs that deadlock while
2344 acquiring locks can be successfully killed by repeatedly sending SIGINT to the
2345 process (by pressing :kbd:`Ctrl+C` in most shells).
2346 (Contributed by Reid Kleckner; :issue:`8844`.)
2347
2348
2349 Optimizations
2350 =============
2351
2352 A number of small performance enhancements have been added:
2353
2354 * Python's peephole optimizer now recognizes patterns such ``x in {1, 2, 3}`` as
2355 being a test for membership in a set of constants. The optimizer recasts the
2356 :class:`set` as a :class:`frozenset` and stores the pre-built constant.
2357
2358 Now that the speed penalty is gone, it is practical to start writing
2359 membership tests using set-notation. This style is both semantically clear
2360 and operationally fast::
2361
2362 extension = name.rpartition('.')[2]
2363 if extension in {'xml', 'html', 'xhtml', 'css'}:
2364 handle(name)
2365
2366 (Patch and additional tests contributed by Dave Malcolm; :issue:`6690`).
2367
2368 * Serializing and unserializing data using the :mod:`pickle` module is now
2369 several times faster.
2370
2371 (Contributed by Alexandre Vassalotti, Antoine Pitrou
2372 and the Unladen Swallow team in :issue:`9410` and :issue:`3873`.)
2373
2374 * The `Timsort algorithm <https://en.wikipedia.org/wiki/Timsort>`_ used in
2375 :meth:`list.sort` and :func:`sorted` now runs faster and uses less memory
2376 when called with a :term:`key function`. Previously, every element of
2377 a list was wrapped with a temporary object that remembered the key value
2378 associated with each element. Now, two arrays of keys and values are
2379 sorted in parallel. This saves the memory consumed by the sort wrappers,
2380 and it saves time lost to delegating comparisons.
2381
2382 (Patch by Daniel Stutzbach in :issue:`9915`.)
2383
2384 * JSON decoding performance is improved and memory consumption is reduced
2385 whenever the same string is repeated for multiple keys. Also, JSON encoding
2386 now uses the C speedups when the ``sort_keys`` argument is true.
2387
2388 (Contributed by Antoine Pitrou in :issue:`7451` and by Raymond Hettinger and
2389 Antoine Pitrou in :issue:`10314`.)
2390
2391 * Recursive locks (created with the :func:`threading.RLock` API) now benefit
2392 from a C implementation which makes them as fast as regular locks, and between
2393 10x and 15x faster than their previous pure Python implementation.
2394
2395 (Contributed by Antoine Pitrou; :issue:`3001`.)
2396
2397 * The fast-search algorithm in stringlib is now used by the :meth:`~str.split`,
2398 :meth:`~str.rsplit`, :meth:`~str.splitlines` and :meth:`~str.replace` methods on
2399 :class:`bytes`, :class:`bytearray` and :class:`str` objects. Likewise, the
2400 algorithm is also used by :meth:`~str.rfind`, :meth:`~str.rindex`, :meth:`~str.rsplit` and
2401 :meth:`~str.rpartition`.
2402
2403 (Patch by Florent Xicluna in :issue:`7622` and :issue:`7462`.)
2404
2405
2406 * Integer to string conversions now work two "digits" at a time, reducing the
2407 number of division and modulo operations.
2408
2409 (:issue:`6713` by Gawain Bolton, Mark Dickinson, and Victor Stinner.)
2410
2411 There were several other minor optimizations. Set differencing now runs faster
2412 when one operand is much larger than the other (patch by Andress Bennetts in
2413 :issue:`8685`). The :meth:`!array.repeat` method has a faster implementation
2414 (:issue:`1569291` by Alexander Belopolsky). The :class:`~http.server.BaseHTTPRequestHandler`
2415 has more efficient buffering (:issue:`3709` by Andrew Schaaf). The
2416 :func:`operator.attrgetter` function has been sped-up (:issue:`10160` by
2417 Christos Georgiou). And :class:`~configparser.ConfigParser` loads multi-line arguments a bit
2418 faster (:issue:`7113` by Łukasz Langa).
2419
2420
2421 Unicode
2422 =======
2423
2424 Python has been updated to `Unicode 6.0.0
2425 <https://unicode.org/versions/Unicode6.0.0/>`_. The update to the standard adds
2426 over 2,000 new characters including `emoji <https://en.wikipedia.org/wiki/Emoji>`_
2427 symbols which are important for mobile phones.
2428
2429 In addition, the updated standard has altered the character properties for two
2430 Kannada characters (U+0CF1, U+0CF2) and one New Tai Lue numeric character
2431 (U+19DA), making the former eligible for use in identifiers while disqualifying
2432 the latter. For more information, see `Unicode Character Database Changes
2433 <https://www.unicode.org/versions/Unicode6.0.0/#Database_Changes>`_.
2434
2435
2436 Codecs
2437 ======
2438
2439 Support was added for *cp720* Arabic DOS encoding (:issue:`1616979`).
2440
2441 MBCS encoding no longer ignores the error handler argument. In the default
2442 strict mode, it raises an :exc:`UnicodeDecodeError` when it encounters an
2443 undecodable byte sequence and an :exc:`UnicodeEncodeError` for an unencodable
2444 character.
2445
2446 The MBCS codec supports ``'strict'`` and ``'ignore'`` error handlers for
2447 decoding, and ``'strict'`` and ``'replace'`` for encoding.
2448
2449 To emulate Python3.1 MBCS encoding, select the ``'ignore'`` handler for decoding
2450 and the ``'replace'`` handler for encoding.
2451
2452 On Mac OS X, Python decodes command line arguments with ``'utf-8'`` rather than
2453 the locale encoding.
2454
2455 By default, :mod:`tarfile` uses ``'utf-8'`` encoding on Windows (instead of
2456 ``'mbcs'``) and the ``'surrogateescape'`` error handler on all operating
2457 systems.
2458
2459
2460 Documentation
2461 =============
2462
2463 The documentation continues to be improved.
2464
2465 * A table of quick links has been added to the top of lengthy sections such as
2466 :ref:`built-in-funcs`. In the case of :mod:`itertools`, the links are
2467 accompanied by tables of cheatsheet-style summaries to provide an overview and
2468 memory jog without having to read all of the docs.
2469
2470 * In some cases, the pure Python source code can be a helpful adjunct to the
2471 documentation, so now many modules now feature quick links to the latest
2472 version of the source code. For example, the :mod:`functools` module
2473 documentation has a quick link at the top labeled:
2474
2475 **Source code** :source:`Lib/functools.py`.
2476
2477 (Contributed by Raymond Hettinger; see
2478 `rationale <https://rhettinger.wordpress.com/2011/01/28/open-your-source-more/>`_.)
2479
2480 * The docs now contain more examples and recipes. In particular, :mod:`re`
2481 module has an extensive section, :ref:`re-examples`. Likewise, the
2482 :mod:`itertools` module continues to be updated with new
2483 :ref:`itertools-recipes`.
2484
2485 * The :mod:`datetime` module now has an auxiliary implementation in pure Python.
2486 No functionality was changed. This just provides an easier-to-read alternate
2487 implementation.
2488
2489 (Contributed by Alexander Belopolsky in :issue:`9528`.)
2490
2491 * The unmaintained :file:`Demo` directory has been removed. Some demos were
2492 integrated into the documentation, some were moved to the :file:`Tools/demo`
2493 directory, and others were removed altogether.
2494
2495 (Contributed by Georg Brandl in :issue:`7962`.)
2496
2497
2498 IDLE
2499 ====
2500
2501 * The format menu now has an option to clean source files by stripping
2502 trailing whitespace.
2503
2504 (Contributed by Raymond Hettinger; :issue:`5150`.)
2505
2506 * IDLE on Mac OS X now works with both Carbon AquaTk and Cocoa AquaTk.
2507
2508 (Contributed by Kevin Walzer, Ned Deily, and Ronald Oussoren; :issue:`6075`.)
2509
2510 Code Repository
2511 ===============
2512
2513 In addition to the existing Subversion code repository at https://svn.python.org
2514 there is now a `Mercurial <https://www.mercurial-scm.org/>`_ repository at
2515 https://hg.python.org/\ .
2516
2517 After the 3.2 release, there are plans to switch to Mercurial as the primary
2518 repository. This distributed version control system should make it easier for
2519 members of the community to create and share external changesets. See
2520 :pep:`385` for details.
2521
2522 To learn to use the new version control system, see the `Quick Start
2523 <https://www.mercurial-scm.org/wiki/QuickStart>`_ or the `Guide to
2524 Mercurial Workflows <https://www.mercurial-scm.org/guide>`_.
2525
2526
2527 Build and C API Changes
2528 =======================
2529
2530 Changes to Python's build process and to the C API include:
2531
2532 * The *idle*, *pydoc* and *2to3* scripts are now installed with a
2533 version-specific suffix on ``make altinstall`` (:issue:`10679`).
2534
2535 * The C functions that access the Unicode Database now accept and return
2536 characters from the full Unicode range, even on narrow unicode builds
2537 (Py_UNICODE_TOLOWER, Py_UNICODE_ISDECIMAL, and others). A visible difference
2538 in Python is that :func:`unicodedata.numeric` now returns the correct value
2539 for large code points, and :func:`repr` may consider more characters as
2540 printable.
2541
2542 (Reported by Bupjoe Lee and fixed by Amaury Forgeot D'Arc; :issue:`5127`.)
2543
2544 * Computed gotos are now enabled by default on supported compilers (which are
2545 detected by the configure script). They can still be disabled selectively by
2546 specifying ``--without-computed-gotos``.
2547
2548 (Contributed by Antoine Pitrou; :issue:`9203`.)
2549
2550 * The option ``--with-wctype-functions`` was removed. The built-in unicode
2551 database is now used for all functions.
2552
2553 (Contributed by Amaury Forgeot D'Arc; :issue:`9210`.)
2554
2555 * Hash values are now values of a new type, :c:type:`Py_hash_t`, which is
2556 defined to be the same size as a pointer. Previously they were of type long,
2557 which on some 64-bit operating systems is still only 32 bits long. As a
2558 result of this fix, :class:`set` and :class:`dict` can now hold more than
2559 ``2**32`` entries on builds with 64-bit pointers (previously, they could grow
2560 to that size but their performance degraded catastrophically).
2561
2562 (Suggested by Raymond Hettinger and implemented by Benjamin Peterson;
2563 :issue:`9778`.)
2564
2565 * A new macro :c:macro:`!Py_VA_COPY` copies the state of the variable argument
2566 list. It is equivalent to C99 *va_copy* but available on all Python platforms
2567 (:issue:`2443`).
2568
2569 * A new C API function :c:func:`!PySys_SetArgvEx` allows an embedded interpreter
2570 to set :data:`sys.argv` without also modifying :data:`sys.path`
2571 (:issue:`5753`).
2572
2573 * :c:func:`!PyEval_CallObject` is now only available in macro form. The
2574 function declaration, which was kept for backwards compatibility reasons, is
2575 now removed -- the macro was introduced in 1997 (:issue:`8276`).
2576
2577 * There is a new function :c:func:`PyLong_AsLongLongAndOverflow` which
2578 is analogous to :c:func:`PyLong_AsLongAndOverflow`. They both serve to
2579 convert Python :class:`int` into a native fixed-width type while providing
2580 detection of cases where the conversion won't fit (:issue:`7767`).
2581
2582 * The :c:func:`PyUnicode_CompareWithASCIIString` function now returns *not
2583 equal* if the Python string is *NUL* terminated.
2584
2585 * There is a new function :c:func:`PyErr_NewExceptionWithDoc` that is
2586 like :c:func:`PyErr_NewException` but allows a docstring to be specified.
2587 This lets C exceptions have the same self-documenting capabilities as
2588 their pure Python counterparts (:issue:`7033`).
2589
2590 * When compiled with the ``--with-valgrind`` option, the pymalloc
2591 allocator will be automatically disabled when running under Valgrind. This
2592 gives improved memory leak detection when running under Valgrind, while taking
2593 advantage of pymalloc at other times (:issue:`2422`).
2594
2595 * Removed the ``O?`` format from the *PyArg_Parse* functions. The format is no
2596 longer used and it had never been documented (:issue:`8837`).
2597
2598 There were a number of other small changes to the C-API. See the
2599 `Misc/NEWS <https://github.com/python/cpython/blob/v3.2.6/Misc/NEWS>`__
2600 file for a complete list.
2601
2602 Also, there were a number of updates to the Mac OS X build, see
2603 `Mac/BuildScript/README.txt <https://github.com/python/cpython/blob/v3.2.6/Mac/BuildScript/README.txt>`_
2604 for details. For users running a 32/64-bit
2605 build, there is a known problem with the default Tcl/Tk on Mac OS X 10.6.
2606 Accordingly, we recommend installing an updated alternative such as
2607 `ActiveState Tcl/Tk 8.5.9 <https://web.archive.org/web/20101208191259/https://www.activestate.com/activetcl/downloads>`_\.
2608 See https://www.python.org/download/mac/tcltk/ for additional details.
2609
2610 Porting to Python 3.2
2611 =====================
2612
2613 This section lists previously described changes and other bugfixes that may
2614 require changes to your code:
2615
2616 * The :mod:`configparser` module has a number of clean-ups. The major change is
2617 to replace the old :class:`!ConfigParser` class with long-standing preferred
2618 alternative :class:`!SafeConfigParser`. In addition there are a number of
2619 smaller incompatibilities:
2620
2621 * The interpolation syntax is now validated on
2622 :meth:`~configparser.ConfigParser.get` and
2623 :meth:`~configparser.ConfigParser.set` operations. In the default
2624 interpolation scheme, only two tokens with percent signs are valid: ``%(name)s``
2625 and ``%%``, the latter being an escaped percent sign.
2626
2627 * The :meth:`~configparser.ConfigParser.set` and
2628 :meth:`~configparser.ConfigParser.add_section` methods now verify that
2629 values are actual strings. Formerly, unsupported types could be introduced
2630 unintentionally.
2631
2632 * Duplicate sections or options from a single source now raise either
2633 :exc:`~configparser.DuplicateSectionError` or
2634 :exc:`~configparser.DuplicateOptionError`. Formerly, duplicates would
2635 silently overwrite a previous entry.
2636
2637 * Inline comments are now disabled by default so now the **;** character
2638 can be safely used in values.
2639
2640 * Comments now can be indented. Consequently, for **;** or **#** to appear at
2641 the start of a line in multiline values, it has to be interpolated. This
2642 keeps comment prefix characters in values from being mistaken as comments.
2643
2644 * ``""`` is now a valid value and is no longer automatically converted to an
2645 empty string. For empty strings, use ``"option ="`` in a line.
2646
2647 * The :mod:`!nntplib` module was reworked extensively, meaning that its APIs
2648 are often incompatible with the 3.1 APIs.
2649
2650 * :class:`bytearray` objects can no longer be used as filenames; instead,
2651 they should be converted to :class:`bytes`.
2652
2653 * The :meth:`!array.tostring` and :meth:`!array.fromstring` have been renamed to
2654 :meth:`array.tobytes() <array.array.tobytes>` and
2655 :meth:`array.frombytes() <array.array.frombytes>` for clarity. The old names
2656 have been deprecated. (See :issue:`8990`.)
2657
2658 * ``PyArg_Parse*()`` functions:
2659
2660 * "t#" format has been removed: use "s#" or "s*" instead
2661 * "w" and "w#" formats has been removed: use "w*" instead
2662
2663 * The :c:type:`!PyCObject` type, deprecated in 3.1, has been removed. To wrap
2664 opaque C pointers in Python objects, the :c:type:`PyCapsule` API should be used
2665 instead; the new type has a well-defined interface for passing typing safety
2666 information and a less complicated signature for calling a destructor.
2667
2668 * The :func:`!sys.setfilesystemencoding` function was removed because
2669 it had a flawed design.
2670
2671 * The :func:`random.seed` function and method now salt string seeds with an
2672 sha512 hash function. To access the previous version of *seed* in order to
2673 reproduce Python 3.1 sequences, set the *version* argument to *1*,
2674 ``random.seed(s, version=1)``.
2675
2676 * The previously deprecated :func:`!string.maketrans` function has been removed
2677 in favor of the static methods :meth:`bytes.maketrans` and
2678 :meth:`bytearray.maketrans`. This change solves the confusion around which
2679 types were supported by the :mod:`string` module. Now, :class:`str`,
2680 :class:`bytes`, and :class:`bytearray` each have their own **maketrans** and
2681 **translate** methods with intermediate translation tables of the appropriate
2682 type.
2683
2684 (Contributed by Georg Brandl; :issue:`5675`.)
2685
2686 * The previously deprecated :func:`!contextlib.nested` function has been removed
2687 in favor of a plain :keyword:`with` statement which can accept multiple
2688 context managers. The latter technique is faster (because it is built-in),
2689 and it does a better job finalizing multiple context managers when one of them
2690 raises an exception::
2691
2692 with open('mylog.txt') as infile, open('a.out', 'w') as outfile:
2693 for line in infile:
2694 if '<critical>' in line:
2695 outfile.write(line)
2696
2697 (Contributed by Georg Brandl and Mattias Brändström;
2698 `appspot issue 53094 <https://codereview.appspot.com/53094>`_.)
2699
2700 * :func:`struct.pack` now only allows bytes for the ``s`` string pack code.
2701 Formerly, it would accept text arguments and implicitly encode them to bytes
2702 using UTF-8. This was problematic because it made assumptions about the
2703 correct encoding and because a variable-length encoding can fail when writing
2704 to fixed length segment of a structure.
2705
2706 Code such as ``struct.pack('<6sHHBBB', 'GIF87a', x, y)`` should be rewritten
2707 with to use bytes instead of text, ``struct.pack('<6sHHBBB', b'GIF87a', x, y)``.
2708
2709 (Discovered by David Beazley and fixed by Victor Stinner; :issue:`10783`.)
2710
2711 * The :class:`xml.etree.ElementTree` class now raises an
2712 :exc:`xml.etree.ElementTree.ParseError` when a parse fails. Previously it
2713 raised an :exc:`xml.parsers.expat.ExpatError`.
2714
2715 * The new, longer :func:`str` value on floats may break doctests which rely on
2716 the old output format.
2717
2718 * In :class:`subprocess.Popen`, the default value for *close_fds* is now
2719 ``True`` under Unix; under Windows, it is ``True`` if the three standard
2720 streams are set to ``None``, ``False`` otherwise. Previously, *close_fds*
2721 was always ``False`` by default, which produced difficult to solve bugs
2722 or race conditions when open file descriptors would leak into the child
2723 process.
2724
2725 * Support for legacy HTTP 0.9 has been removed from :mod:`urllib.request`
2726 and :mod:`http.client`. Such support is still present on the server side
2727 (in :mod:`http.server`).
2728
2729 (Contributed by Antoine Pitrou, :issue:`10711`.)
2730
2731 * SSL sockets in timeout mode now raise :exc:`socket.timeout` when a timeout
2732 occurs, rather than a generic :exc:`~ssl.SSLError`.
2733
2734 (Contributed by Antoine Pitrou, :issue:`10272`.)
2735
2736 * The misleading functions :c:func:`!PyEval_AcquireLock` and
2737 :c:func:`!PyEval_ReleaseLock` have been officially deprecated. The
2738 thread-state aware APIs (such as :c:func:`PyEval_SaveThread`
2739 and :c:func:`PyEval_RestoreThread`) should be used instead.
2740
2741 * Due to security risks, :func:`!asyncore.handle_accept` has been deprecated, and
2742 a new function, :func:`!asyncore.handle_accepted`, was added to replace it.
2743
2744 (Contributed by Giampaolo Rodola in :issue:`6706`.)
2745
2746 * Due to the new :term:`GIL` implementation, :c:func:`!PyEval_InitThreads`
2747 cannot be called before :c:func:`Py_Initialize` anymore.