Debian Codegraph @

Doc/whatsnew/3.1.rst

raw
1 ****************************
2 What's New In Python 3.1
3 ****************************
4
5 :Author: Raymond Hettinger
6
7 .. $Id$
8 Rules for maintenance:
9
10 * Anyone can add text to this document. Do not spend very much time
11 on the wording of your changes, because your text will probably
12 get rewritten to some degree.
13
14 * The maintainer will go through Misc/NEWS periodically and add
15 changes; it's therefore more important to add your changes to
16 Misc/NEWS than to this file.
17
18 * This is not a complete list of every single change; completeness
19 is the purpose of Misc/NEWS. Some changes I consider too small
20 or esoteric to include. If such a change is added to the text,
21 I'll just remove it. (This is another reason you shouldn't spend
22 too much time on writing your addition.)
23
24 * If you want to draw your new text to the attention of the
25 maintainer, add 'XXX' to the beginning of the paragraph or
26 section.
27
28 * It's OK to just add a fragmentary note about a change. For
29 example: "XXX Describe the transmogrify() function added to the
30 socket module." The maintainer will research the change and
31 write the necessary text.
32
33 * You can comment out your additions if you like, but it's not
34 necessary (especially when a final release is some months away).
35
36 * Credit the author of a patch or bugfix. Just the name is
37 sufficient; the e-mail address isn't necessary.
38
39 * It's helpful to add the bug/patch number as a comment:
40
41 % Patch 12345
42 XXX Describe the transmogrify() function added to the socket
43 module.
44 (Contributed by P.Y. Developer.)
45
46 This saves the maintainer the effort of going through the SVN log
47 when researching a change.
48
49 This article explains the new features in Python 3.1, compared to 3.0.
50 Python 3.1 was released on June 27, 2009.
51
52
53 PEP 372: Ordered Dictionaries
54 =============================
55
56 Regular Python dictionaries iterate over key/value pairs in arbitrary order.
57 Over the years, a number of authors have written alternative implementations
58 that remember the order that the keys were originally inserted. Based on
59 the experiences from those implementations, a new
60 :class:`collections.OrderedDict` class has been introduced.
61
62 The OrderedDict API is substantially the same as regular dictionaries
63 but will iterate over keys and values in a guaranteed order depending on
64 when a key was first inserted. If a new entry overwrites an existing entry,
65 the original insertion position is left unchanged. Deleting an entry and
66 reinserting it will move it to the end.
67
68 The standard library now supports use of ordered dictionaries in several
69 modules. The :mod:`configparser` module uses them by default. This lets
70 configuration files be read, modified, and then written back in their original
71 order. The *_asdict()* method for :func:`collections.namedtuple` now
72 returns an ordered dictionary with the values appearing in the same order as
73 the underlying tuple indices. The :mod:`json` module is being built-out with
74 an *object_pairs_hook* to allow OrderedDicts to be built by the decoder.
75 Support was also added for third-party tools like `PyYAML <https://pyyaml.org/>`_.
76
77 .. seealso::
78
79 :pep:`372` - Ordered Dictionaries
80 PEP written by Armin Ronacher and Raymond Hettinger. Implementation
81 written by Raymond Hettinger.
82
83 Since an ordered dictionary remembers its insertion order, it can be used
84 in conjunction with sorting to make a sorted dictionary::
85
86 >>> # regular unsorted dictionary
87 >>> d = {'banana': 3, 'apple':4, 'pear': 1, 'orange': 2}
88
89 >>> # dictionary sorted by key
90 >>> OrderedDict(sorted(d.items(), key=lambda t: t[0]))
91 OrderedDict([('apple', 4), ('banana', 3), ('orange', 2), ('pear', 1)])
92
93 >>> # dictionary sorted by value
94 >>> OrderedDict(sorted(d.items(), key=lambda t: t[1]))
95 OrderedDict([('pear', 1), ('orange', 2), ('banana', 3), ('apple', 4)])
96
97 >>> # dictionary sorted by length of the key string
98 >>> OrderedDict(sorted(d.items(), key=lambda t: len(t[0])))
99 OrderedDict([('pear', 1), ('apple', 4), ('orange', 2), ('banana', 3)])
100
101 The new sorted dictionaries maintain their sort order when entries
102 are deleted. But when new keys are added, the keys are appended
103 to the end and the sort is not maintained.
104
105
106 PEP 378: Format Specifier for Thousands Separator
107 =================================================
108
109 The built-in :func:`format` function and the :meth:`str.format` method use
110 a mini-language that now includes a simple, non-locale aware way to format
111 a number with a thousands separator. That provides a way to humanize a
112 program's output, improving its professional appearance and readability::
113
114 >>> format(1234567, ',d')
115 '1,234,567'
116 >>> format(1234567.89, ',.2f')
117 '1,234,567.89'
118 >>> format(12345.6 + 8901234.12j, ',f')
119 '12,345.600000+8,901,234.120000j'
120 >>> format(Decimal('1234567.89'), ',f')
121 '1,234,567.89'
122
123 The supported types are :class:`int`, :class:`float`, :class:`complex`
124 and :class:`decimal.Decimal`.
125
126 Discussions are underway about how to specify alternative separators
127 like dots, spaces, apostrophes, or underscores. Locale-aware applications
128 should use the existing *n* format specifier which already has some support
129 for thousands separators.
130
131 .. seealso::
132
133 :pep:`378` - Format Specifier for Thousands Separator
134 PEP written by Raymond Hettinger and implemented by Eric Smith and
135 Mark Dickinson.
136
137
138 Other Language Changes
139 ======================
140
141 Some smaller changes made to the core Python language are:
142
143 * Directories and zip archives containing a :file:`__main__.py`
144 file can now be executed directly by passing their name to the
145 interpreter. The directory/zipfile is automatically inserted as the
146 first entry in sys.path. (Suggestion and initial patch by Andy Chu;
147 revised patch by Phillip J. Eby and Nick Coghlan; :issue:`1739468`.)
148
149 * The :func:`int` type gained a ``bit_length`` method that returns the
150 number of bits necessary to represent its argument in binary::
151
152 >>> n = 37
153 >>> bin(37)
154 '0b100101'
155 >>> n.bit_length()
156 6
157 >>> n = 2**123-1
158 >>> n.bit_length()
159 123
160 >>> (n+1).bit_length()
161 124
162
163 (Contributed by Fredrik Johansson, Victor Stinner, Raymond Hettinger,
164 and Mark Dickinson; :issue:`3439`.)
165
166 * The fields in :func:`format` strings can now be automatically
167 numbered::
168
169 >>> 'Sir {} of {}'.format('Gallahad', 'Camelot')
170 'Sir Gallahad of Camelot'
171
172 Formerly, the string would have required numbered fields such as:
173 ``'Sir {0} of {1}'``.
174
175 (Contributed by Eric Smith; :issue:`5237`.)
176
177 * The :func:`!string.maketrans` function is deprecated and is replaced by new
178 static methods, :meth:`bytes.maketrans` and :meth:`bytearray.maketrans`.
179 This change solves the confusion around which types were supported by the
180 :mod:`string` module. Now, :class:`str`, :class:`bytes`, and
181 :class:`bytearray` each have their own **maketrans** and **translate**
182 methods with intermediate translation tables of the appropriate type.
183
184 (Contributed by Georg Brandl; :issue:`5675`.)
185
186 * The syntax of the :keyword:`with` statement now allows multiple context
187 managers in a single statement::
188
189 >>> with open('mylog.txt') as infile, open('a.out', 'w') as outfile:
190 ... for line in infile:
191 ... if '<critical>' in line:
192 ... outfile.write(line)
193
194 With the new syntax, the :func:`!contextlib.nested` function is no longer
195 needed and is now deprecated.
196
197 (Contributed by Georg Brandl and Mattias Brändström;
198 `appspot issue 53094 <https://codereview.appspot.com/53094>`_.)
199
200 * ``round(x, n)`` now returns an integer if *x* is an integer.
201 Previously it returned a float::
202
203 >>> round(1123, -2)
204 1100
205
206 (Contributed by Mark Dickinson; :issue:`4707`.)
207
208 * Python now uses David Gay's algorithm for finding the shortest floating-point
209 representation that doesn't change its value. This should help
210 mitigate some of the confusion surrounding binary floating-point
211 numbers.
212
213 The significance is easily seen with a number like ``1.1`` which does not
214 have an exact equivalent in binary floating point. Since there is no exact
215 equivalent, an expression like ``float('1.1')`` evaluates to the nearest
216 representable value which is ``0x1.199999999999ap+0`` in hex or
217 ``1.100000000000000088817841970012523233890533447265625`` in decimal. That
218 nearest value was and still is used in subsequent floating-point
219 calculations.
220
221 What is new is how the number gets displayed. Formerly, Python used a
222 simple approach. The value of ``repr(1.1)`` was computed as ``format(1.1,
223 '.17g')`` which evaluated to ``'1.1000000000000001'``. The advantage of
224 using 17 digits was that it relied on IEEE-754 guarantees to assure that
225 ``eval(repr(1.1))`` would round-trip exactly to its original value. The
226 disadvantage is that many people found the output to be confusing (mistaking
227 intrinsic limitations of binary floating-point representation as being a
228 problem with Python itself).
229
230 The new algorithm for ``repr(1.1)`` is smarter and returns ``'1.1'``.
231 Effectively, it searches all equivalent string representations (ones that
232 get stored with the same underlying float value) and returns the shortest
233 representation.
234
235 The new algorithm tends to emit cleaner representations when possible, but
236 it does not change the underlying values. So, it is still the case that
237 ``1.1 + 2.2 != 3.3`` even though the representations may suggest otherwise.
238
239 The new algorithm depends on certain features in the underlying floating-point
240 implementation. If the required features are not found, the old
241 algorithm will continue to be used. Also, the text pickle protocols
242 assure cross-platform portability by using the old algorithm.
243
244 (Contributed by Eric Smith and Mark Dickinson; :issue:`1580`)
245
246 New, Improved, and Deprecated Modules
247 =====================================
248
249 * Added a :class:`collections.Counter` class to support convenient
250 counting of unique items in a sequence or iterable::
251
252 >>> Counter(['red', 'blue', 'red', 'green', 'blue', 'blue'])
253 Counter({'blue': 3, 'red': 2, 'green': 1})
254
255 (Contributed by Raymond Hettinger; :issue:`1696199`.)
256
257 * Added a new module, :mod:`tkinter.ttk` for access to the Tk themed widget set.
258 The basic idea of ttk is to separate, to the extent possible, the code
259 implementing a widget's behavior from the code implementing its appearance.
260
261 (Contributed by Guilherme Polo; :issue:`2983`.)
262
263 * The :class:`gzip.GzipFile` and :class:`bz2.BZ2File` classes now support
264 the context management protocol::
265
266 >>> # Automatically close file after writing
267 >>> with gzip.GzipFile(filename, "wb") as f:
268 ... f.write(b"xxx")
269
270 (Contributed by Antoine Pitrou.)
271
272 * The :mod:`decimal` module now supports methods for creating a
273 decimal object from a binary :class:`float`. The conversion is
274 exact but can sometimes be surprising::
275
276 >>> Decimal.from_float(1.1)
277 Decimal('1.100000000000000088817841970012523233890533447265625')
278
279 The long decimal result shows the actual binary fraction being
280 stored for *1.1*. The fraction has many digits because *1.1* cannot
281 be exactly represented in binary.
282
283 (Contributed by Raymond Hettinger and Mark Dickinson.)
284
285 * The :mod:`itertools` module grew two new functions. The
286 :func:`itertools.combinations_with_replacement` function is one of
287 four for generating combinatorics including permutations and Cartesian
288 products. The :func:`itertools.compress` function mimics its namesake
289 from APL. Also, the existing :func:`itertools.count` function now has
290 an optional *step* argument and can accept any type of counting
291 sequence including :class:`fractions.Fraction` and
292 :class:`decimal.Decimal`::
293
294 >>> [p+q for p,q in combinations_with_replacement('LOVE', 2)]
295 ['LL', 'LO', 'LV', 'LE', 'OO', 'OV', 'OE', 'VV', 'VE', 'EE']
296
297 >>> list(compress(data=range(10), selectors=[0,0,1,1,0,1,0,1,0,0]))
298 [2, 3, 5, 7]
299
300 >>> c = count(start=Fraction(1,2), step=Fraction(1,6))
301 >>> [next(c), next(c), next(c), next(c)]
302 [Fraction(1, 2), Fraction(2, 3), Fraction(5, 6), Fraction(1, 1)]
303
304 (Contributed by Raymond Hettinger.)
305
306 * :func:`collections.namedtuple` now supports a keyword argument
307 *rename* which lets invalid fieldnames be automatically converted to
308 positional names in the form _0, _1, etc. This is useful when
309 the field names are being created by an external source such as a
310 CSV header, SQL field list, or user input::
311
312 >>> query = input()
313 SELECT region, dept, count(*) FROM main GROUPBY region, dept
314
315 >>> cursor.execute(query)
316 >>> query_fields = [desc[0] for desc in cursor.description]
317 >>> UserQuery = namedtuple('UserQuery', query_fields, rename=True)
318 >>> pprint.pprint([UserQuery(*row) for row in cursor])
319 [UserQuery(region='South', dept='Shipping', _2=185),
320 UserQuery(region='North', dept='Accounting', _2=37),
321 UserQuery(region='West', dept='Sales', _2=419)]
322
323 (Contributed by Raymond Hettinger; :issue:`1818`.)
324
325 * The :func:`re.sub`, :func:`re.subn` and :func:`re.split` functions now
326 accept a flags parameter.
327
328 (Contributed by Gregory Smith.)
329
330 * The :mod:`logging` module now implements a simple :class:`logging.NullHandler`
331 class for applications that are not using logging but are calling
332 library code that does. Setting-up a null handler will suppress
333 spurious warnings such as "No handlers could be found for logger foo"::
334
335 >>> h = logging.NullHandler()
336 >>> logging.getLogger("foo").addHandler(h)
337
338 (Contributed by Vinay Sajip; :issue:`4384`).
339
340 * The :mod:`runpy` module which supports the ``-m`` command line switch
341 now supports the execution of packages by looking for and executing
342 a ``__main__`` submodule when a package name is supplied.
343
344 (Contributed by Andi Vajda; :issue:`4195`.)
345
346 * The :mod:`pdb` module can now access and display source code loaded via
347 :mod:`zipimport` (or any other conformant :pep:`302` loader).
348
349 (Contributed by Alexander Belopolsky; :issue:`4201`.)
350
351 * :class:`functools.partial` objects can now be pickled.
352
353 (Suggested by Antoine Pitrou and Jesse Noller. Implemented by
354 Jack Diederich; :issue:`5228`.)
355
356 * Add :mod:`pydoc` help topics for symbols so that ``help('@')``
357 works as expected in the interactive environment.
358
359 (Contributed by David Laban; :issue:`4739`.)
360
361 * The :mod:`unittest` module now supports skipping individual tests or classes
362 of tests. And it supports marking a test as an expected failure, a test that
363 is known to be broken, but shouldn't be counted as a failure on a
364 TestResult::
365
366 class TestGizmo(unittest.TestCase):
367
368 @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows")
369 def test_gizmo_on_windows(self):
370 ...
371
372 @unittest.expectedFailure
373 def test_gimzo_without_required_library(self):
374 ...
375
376 Also, tests for exceptions have been builtout to work with context managers
377 using the :keyword:`with` statement::
378
379 def test_division_by_zero(self):
380 with self.assertRaises(ZeroDivisionError):
381 x / 0
382
383 In addition, several new assertion methods were added including
384 :meth:`~unittest.TestCase.assertSetEqual`,
385 :meth:`~unittest.TestCase.assertDictEqual`,
386 :meth:`!assertDictContainsSubset`,
387 :meth:`~unittest.TestCase.assertListEqual`,
388 :meth:`~unittest.TestCase.assertTupleEqual`,
389 :meth:`~unittest.TestCase.assertSequenceEqual`,
390 :meth:`assertRaisesRegexp() <unittest.TestCase.assertRaisesRegex>`,
391 :meth:`~unittest.TestCase.assertIsNone`,
392 and :meth:`~unittest.TestCase.assertIsNotNone`.
393
394 (Contributed by Benjamin Peterson and Antoine Pitrou.)
395
396 * The :mod:`io` module has three new constants for the :meth:`~io.IOBase.seek`
397 method: :data:`~os.SEEK_SET`, :data:`~os.SEEK_CUR`, and :data:`~os.SEEK_END`.
398
399 * The :data:`sys.version_info` tuple is now a named tuple::
400
401 >>> sys.version_info
402 sys.version_info(major=3, minor=1, micro=0, releaselevel='alpha', serial=2)
403
404 (Contributed by Ross Light; :issue:`4285`.)
405
406 * The :mod:`!nntplib` and :mod:`imaplib` modules now support IPv6.
407
408 (Contributed by Derek Morr; :issue:`1655` and :issue:`1664`.)
409
410 * The :mod:`pickle` module has been adapted for better interoperability with
411 Python 2.x when used with protocol 2 or lower. The reorganization of the
412 standard library changed the formal reference for many objects. For
413 example, ``__builtin__.set`` in Python 2 is called ``builtins.set`` in Python
414 3. This change confounded efforts to share data between different versions of
415 Python. But now when protocol 2 or lower is selected, the pickler will
416 automatically use the old Python 2 names for both loading and dumping. This
417 remapping is turned-on by default but can be disabled with the *fix_imports*
418 option::
419
420 >>> s = {1, 2, 3}
421 >>> pickle.dumps(s, protocol=0)
422 b'c__builtin__\nset\np0\n((lp1\nL1L\naL2L\naL3L\natp2\nRp3\n.'
423 >>> pickle.dumps(s, protocol=0, fix_imports=False)
424 b'cbuiltins\nset\np0\n((lp1\nL1L\naL2L\naL3L\natp2\nRp3\n.'
425
426 An unfortunate but unavoidable side-effect of this change is that protocol 2
427 pickles produced by Python 3.1 won't be readable with Python 3.0. The latest
428 pickle protocol, protocol 3, should be used when migrating data between
429 Python 3.x implementations, as it doesn't attempt to remain compatible with
430 Python 2.x.
431
432 (Contributed by Alexandre Vassalotti and Antoine Pitrou, :issue:`6137`.)
433
434 * A new module, :mod:`importlib` was added. It provides a complete, portable,
435 pure Python reference implementation of the :keyword:`import` statement and its
436 counterpart, the :func:`__import__` function. It represents a substantial
437 step forward in documenting and defining the actions that take place during
438 imports.
439
440 (Contributed by Brett Cannon.)
441
442 Optimizations
443 =============
444
445 Major performance enhancements have been added:
446
447 * The new I/O library (as defined in :pep:`3116`) was mostly written in
448 Python and quickly proved to be a problematic bottleneck in Python 3.0.
449 In Python 3.1, the I/O library has been entirely rewritten in C and is
450 2 to 20 times faster depending on the task at hand. The pure Python
451 version is still available for experimentation purposes through
452 the ``_pyio`` module.
453
454 (Contributed by Amaury Forgeot d'Arc and Antoine Pitrou.)
455
456 * Added a heuristic so that tuples and dicts containing only untrackable objects
457 are not tracked by the garbage collector. This can reduce the size of
458 collections and therefore the garbage collection overhead on long-running
459 programs, depending on their particular use of datatypes.
460
461 (Contributed by Antoine Pitrou, :issue:`4688`.)
462
463 * Enabling a configure option named ``--with-computed-gotos``
464 on compilers that support it (notably: gcc, SunPro, icc), the bytecode
465 evaluation loop is compiled with a new dispatch mechanism which gives
466 speedups of up to 20%, depending on the system, the compiler, and
467 the benchmark.
468
469 (Contributed by Antoine Pitrou along with a number of other participants,
470 :issue:`4753`).
471
472 * The decoding of UTF-8, UTF-16 and LATIN-1 is now two to four times
473 faster.
474
475 (Contributed by Antoine Pitrou and Amaury Forgeot d'Arc, :issue:`4868`.)
476
477 * The :mod:`json` module now has a C extension to substantially improve
478 its performance. In addition, the API was modified so that json works
479 only with :class:`str`, not with :class:`bytes`. That change makes the
480 module closely match the `JSON specification <https://json.org/>`_
481 which is defined in terms of Unicode.
482
483 (Contributed by Bob Ippolito and converted to Py3.1 by Antoine Pitrou
484 and Benjamin Peterson; :issue:`4136`.)
485
486 * Unpickling now interns the attribute names of pickled objects. This saves
487 memory and allows pickles to be smaller.
488
489 (Contributed by Jake McGuire and Antoine Pitrou; :issue:`5084`.)
490
491 IDLE
492 ====
493
494 * IDLE's format menu now provides an option to strip trailing whitespace
495 from a source file.
496
497 (Contributed by Roger D. Serwy; :issue:`5150`.)
498
499 Build and C API Changes
500 =======================
501
502 Changes to Python's build process and to the C API include:
503
504 * Integers are now stored internally either in base ``2**15`` or in base
505 ``2**30``, the base being determined at build time. Previously, they
506 were always stored in base ``2**15``. Using base ``2**30`` gives
507 significant performance improvements on 64-bit machines, but
508 benchmark results on 32-bit machines have been mixed. Therefore,
509 the default is to use base ``2**30`` on 64-bit machines and base ``2**15``
510 on 32-bit machines; on Unix, there's a new configure option
511 ``--enable-big-digits`` that can be used to override this default.
512
513 Apart from the performance improvements this change should be invisible to
514 end users, with one exception: for testing and debugging purposes there's a
515 new :data:`sys.int_info` that provides information about the
516 internal format, giving the number of bits per digit and the size in bytes
517 of the C type used to store each digit::
518
519 >>> import sys
520 >>> sys.int_info
521 sys.int_info(bits_per_digit=30, sizeof_digit=4)
522
523 (Contributed by Mark Dickinson; :issue:`4258`.)
524
525 * The :c:func:`PyLong_AsUnsignedLongLong()` function now handles a negative
526 *pylong* by raising :exc:`OverflowError` instead of :exc:`TypeError`.
527
528 (Contributed by Mark Dickinson and Lisandro Dalcrin; :issue:`5175`.)
529
530 * Deprecated :c:func:`!PyNumber_Int`. Use :c:func:`PyNumber_Long` instead.
531
532 (Contributed by Mark Dickinson; :issue:`4910`.)
533
534 * Added a new :c:func:`PyOS_string_to_double` function to replace the
535 deprecated functions :c:func:`!PyOS_ascii_strtod` and :c:func:`!PyOS_ascii_atof`.
536
537 (Contributed by Mark Dickinson; :issue:`5914`.)
538
539 * Added :c:type:`PyCapsule` as a replacement for the :c:type:`!PyCObject` API.
540 The principal difference is that the new type has a well defined interface
541 for passing typing safety information and a less complicated signature
542 for calling a destructor. The old type had a problematic API and is now
543 deprecated.
544
545 (Contributed by Larry Hastings; :issue:`5630`.)
546
547 Porting to Python 3.1
548 =====================
549
550 This section lists previously described changes and other bugfixes
551 that may require changes to your code:
552
553 * The new floating-point string representations can break existing doctests.
554 For example::
555
556 def e():
557 '''Compute the base of natural logarithms.
558
559 >>> e()
560 2.7182818284590451
561
562 '''
563 return sum(1/math.factorial(x) for x in reversed(range(30)))
564
565 doctest.testmod()
566
567 **********************************************************************
568 Failed example:
569 e()
570 Expected:
571 2.7182818284590451
572 Got:
573 2.718281828459045
574 **********************************************************************
575
576 * The automatic name remapping in the pickle module for protocol 2 or lower can
577 make Python 3.1 pickles unreadable in Python 3.0. One solution is to use
578 protocol 3. Another solution is to set the *fix_imports* option to ``False``.
579 See the discussion above for more details.