Debian Codegraph @

Doc/whatsnew/2.4.rst

raw
1 ****************************
2 What's New in Python 2.4
3 ****************************
4
5 :Author: A.M. Kuchling
6
7 .. |release| replace:: 1.02
8
9 .. $Id: whatsnew24.tex 54632 2007-03-31 11:59:54Z georg.brandl $
10 .. Don't write extensive text for new sections; I'll do that.
11 .. Feel free to add commented-out reminders of things that need
12 .. to be covered. --amk
13
14 This article explains the new features in Python 2.4.1, released on March 30,
15 2005.
16
17 Python 2.4 is a medium-sized release. It doesn't introduce as many changes as
18 the radical Python 2.2, but introduces more features than the conservative 2.3
19 release. The most significant new language features are function decorators and
20 generator expressions; most other changes are to the standard library.
21
22 According to the CVS change logs, there were 481 patches applied and 502 bugs
23 fixed between Python 2.3 and 2.4. Both figures are likely to be underestimates.
24
25 This article doesn't attempt to provide a complete specification of every single
26 new feature, but instead provides a brief introduction to each feature. For
27 full details, you should refer to the documentation for Python 2.4, such as the
28 Python Library Reference and the Python Reference Manual. Often you will be
29 referred to the PEP for a particular new feature for explanations of the
30 implementation and design rationale.
31
32 .. ======================================================================
33
34
35 PEP 218: Built-In Set Objects
36 =============================
37
38 Python 2.3 introduced the :mod:`sets` module. C implementations of set data
39 types have now been added to the Python core as two new built-in types,
40 ``set(iterable)`` and ``frozenset(iterable)``. They provide high speed
41 operations for membership testing, for eliminating duplicates from sequences,
42 and for mathematical operations like unions, intersections, differences, and
43 symmetric differences. ::
44
45 >>> a = set('abracadabra') # form a set from a string
46 >>> 'z' in a # fast membership testing
47 False
48 >>> a # unique letters in a
49 set(['a', 'r', 'b', 'c', 'd'])
50 >>> ''.join(a) # convert back into a string
51 'arbcd'
52
53 >>> b = set('alacazam') # form a second set
54 >>> a - b # letters in a but not in b
55 set(['r', 'd', 'b'])
56 >>> a | b # letters in either a or b
57 set(['a', 'c', 'r', 'd', 'b', 'm', 'z', 'l'])
58 >>> a & b # letters in both a and b
59 set(['a', 'c'])
60 >>> a ^ b # letters in a or b but not both
61 set(['r', 'd', 'b', 'm', 'z', 'l'])
62
63 >>> a.add('z') # add a new element
64 >>> a.update('wxy') # add multiple new elements
65 >>> a
66 set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'x', 'z'])
67 >>> a.remove('x') # take one element out
68 >>> a
69 set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'z'])
70
71 The :func:`frozenset` type is an immutable version of :func:`set`. Since it is
72 immutable and hashable, it may be used as a dictionary key or as a member of
73 another set.
74
75 The :mod:`sets` module remains in the standard library, and may be useful if you
76 wish to subclass the :class:`Set` or :class:`ImmutableSet` classes. There are
77 currently no plans to deprecate the module.
78
79
80 .. seealso::
81
82 :pep:`218` - Adding a Built-In Set Object Type
83 Originally proposed by Greg Wilson and ultimately implemented by Raymond
84 Hettinger.
85
86 .. ======================================================================
87
88
89 PEP 237: Unifying Long Integers and Integers
90 ============================================
91
92 The lengthy transition process for this PEP, begun in Python 2.2, takes another
93 step forward in Python 2.4. In 2.3, certain integer operations that would
94 behave differently after int/long unification triggered :exc:`FutureWarning`
95 warnings and returned values limited to 32 or 64 bits (depending on your
96 platform). In 2.4, these expressions no longer produce a warning and instead
97 produce a different result that's usually a long integer.
98
99 The problematic expressions are primarily left shifts and lengthy hexadecimal
100 and octal constants. For example, ``2 << 32`` results in a warning in 2.3,
101 evaluating to 0 on 32-bit platforms. In Python 2.4, this expression now returns
102 the correct answer, 8589934592.
103
104
105 .. seealso::
106
107 :pep:`237` - Unifying Long Integers and Integers
108 Original PEP written by Moshe Zadka and GvR. The changes for 2.4 were
109 implemented by Kalle Svensson.
110
111 .. ======================================================================
112
113
114 PEP 289: Generator Expressions
115 ==============================
116
117 The iterator feature introduced in Python 2.2 and the :mod:`itertools` module
118 make it easier to write programs that loop through large data sets without
119 having the entire data set in memory at one time. List comprehensions don't fit
120 into this picture very well because they produce a Python list object containing
121 all of the items. This unavoidably pulls all of the objects into memory, which
122 can be a problem if your data set is very large. When trying to write a
123 functionally styled program, it would be natural to write something like::
124
125 links = [link for link in get_all_links() if not link.followed]
126 for link in links:
127 ...
128
129 instead of ::
130
131 for link in get_all_links():
132 if link.followed:
133 continue
134 ...
135
136 The first form is more concise and perhaps more readable, but if you're dealing
137 with a large number of link objects you'd have to write the second form to avoid
138 having all link objects in memory at the same time.
139
140 Generator expressions work similarly to list comprehensions but don't
141 materialize the entire list; instead they create a generator that will return
142 elements one by one. The above example could be written as::
143
144 links = (link for link in get_all_links() if not link.followed)
145 for link in links:
146 ...
147
148 Generator expressions always have to be written inside parentheses, as in the
149 above example. The parentheses signalling a function call also count, so if you
150 want to create an iterator that will be immediately passed to a function you
151 could write::
152
153 print sum(obj.count for obj in list_all_objects())
154
155 Generator expressions differ from list comprehensions in various small ways.
156 Most notably, the loop variable (*obj* in the above example) is not accessible
157 outside of the generator expression. List comprehensions leave the variable
158 assigned to its last value; future versions of Python will change this, making
159 list comprehensions match generator expressions in this respect.
160
161
162 .. seealso::
163
164 :pep:`289` - Generator Expressions
165 Proposed by Raymond Hettinger and implemented by Jiwon Seo with early efforts
166 steered by Hye-Shik Chang.
167
168 .. ======================================================================
169
170
171 PEP 292: Simpler String Substitutions
172 =====================================
173
174 Some new classes in the standard library provide an alternative mechanism for
175 substituting variables into strings; this style of substitution may be better
176 for applications where untrained users need to edit templates.
177
178 The usual way of substituting variables by name is the ``%`` operator::
179
180 >>> '%(page)i: %(title)s' % {'page':2, 'title': 'The Best of Times'}
181 '2: The Best of Times'
182
183 When writing the template string, it can be easy to forget the ``i`` or ``s``
184 after the closing parenthesis. This isn't a big problem if the template is in a
185 Python module, because you run the code, get an "Unsupported format character"
186 :exc:`ValueError`, and fix the problem. However, consider an application such
187 as Mailman where template strings or translations are being edited by users who
188 aren't aware of the Python language. The format string's syntax is complicated
189 to explain to such users, and if they make a mistake, it's difficult to provide
190 helpful feedback to them.
191
192 PEP 292 adds a :class:`Template` class to the :mod:`string` module that uses
193 ``$`` to indicate a substitution::
194
195 >>> import string
196 >>> t = string.Template('$page: $title')
197 >>> t.substitute({'page':2, 'title': 'The Best of Times'})
198 '2: The Best of Times'
199
200 If a key is missing from the dictionary, the :meth:`substitute` method will
201 raise a :exc:`KeyError`. There's also a :meth:`safe_substitute` method that
202 ignores missing keys::
203
204 >>> t = string.Template('$page: $title')
205 >>> t.safe_substitute({'page':3})
206 '3: $title'
207
208
209 .. seealso::
210
211 :pep:`292` - Simpler String Substitutions
212 Written and implemented by Barry Warsaw.
213
214 .. ======================================================================
215
216
217 PEP 318: Decorators for Functions and Methods
218 =============================================
219
220 Python 2.2 extended Python's object model by adding static methods and class
221 methods, but it didn't extend Python's syntax to provide any new way of defining
222 static or class methods. Instead, you had to write a :keyword:`def` statement
223 in the usual way, and pass the resulting method to a :func:`staticmethod` or
224 :func:`classmethod` function that would wrap up the function as a method of the
225 new type. Your code would look like this::
226
227 class C:
228 def meth (cls):
229 ...
230
231 meth = classmethod(meth) # Rebind name to wrapped-up class method
232
233 If the method was very long, it would be easy to miss or forget the
234 :func:`classmethod` invocation after the function body.
235
236 The intention was always to add some syntax to make such definitions more
237 readable, but at the time of 2.2's release a good syntax was not obvious. Today
238 a good syntax *still* isn't obvious but users are asking for easier access to
239 the feature; a new syntactic feature has been added to meet this need.
240
241 The new feature is called "function decorators". The name comes from the idea
242 that :func:`classmethod`, :func:`staticmethod`, and friends are storing
243 additional information on a function object; they're *decorating* functions with
244 more details.
245
246 The notation borrows from Java and uses the ``'@'`` character as an indicator.
247 Using the new syntax, the example above would be written::
248
249 class C:
250
251 @classmethod
252 def meth (cls):
253 ...
254
255
256 The ``@classmethod`` is shorthand for the ``meth=classmethod(meth)`` assignment.
257 More generally, if you have the following::
258
259 @A
260 @B
261 @C
262 def f ():
263 ...
264
265 It's equivalent to the following pre-decorator code::
266
267 def f(): ...
268 f = A(B(C(f)))
269
270 Decorators must come on the line before a function definition, one decorator per
271 line, and can't be on the same line as the def statement, meaning that ``@A def
272 f(): ...`` is illegal. You can only decorate function definitions, either at
273 the module level or inside a class; you can't decorate class definitions.
274
275 A decorator is just a function that takes the function to be decorated as an
276 argument and returns either the same function or some new object. The return
277 value of the decorator need not be callable (though it typically is), unless
278 further decorators will be applied to the result. It's easy to write your own
279 decorators. The following simple example just sets an attribute on the function
280 object::
281
282 >>> def deco(func):
283 ... func.attr = 'decorated'
284 ... return func
285 ...
286 >>> @deco
287 ... def f(): pass
288 ...
289 >>> f
290 <function f at 0x402ef0d4>
291 >>> f.attr
292 'decorated'
293 >>>
294
295 As a slightly more realistic example, the following decorator checks that the
296 supplied argument is an integer::
297
298 def require_int (func):
299 def wrapper (arg):
300 assert isinstance(arg, int)
301 return func(arg)
302
303 return wrapper
304
305 @require_int
306 def p1 (arg):
307 print arg
308
309 @require_int
310 def p2(arg):
311 print arg*2
312
313 An example in :pep:`318` contains a fancier version of this idea that lets you
314 both specify the required type and check the returned type.
315
316 Decorator functions can take arguments. If arguments are supplied, your
317 decorator function is called with only those arguments and must return a new
318 decorator function; this function must take a single function and return a
319 function, as previously described. In other words, ``@A @B @C(args)`` becomes::
320
321 def f(): ...
322 _deco = C(args)
323 f = A(B(_deco(f)))
324
325 Getting this right can be slightly brain-bending, but it's not too difficult.
326
327 A small related change makes the :attr:`func_name <function.__name__>`
328 attribute of functions
329 writable. This attribute is used to display function names in tracebacks, so
330 decorators should change the name of any new function that's constructed and
331 returned.
332
333
334 .. seealso::
335
336 :pep:`318` - Decorators for Functions, Methods and Classes
337 Written by Kevin D. Smith, Jim Jewett, and Skip Montanaro. Several people
338 wrote patches implementing function decorators, but the one that was actually
339 checked in was patch #979728, written by Mark Russell.
340
341 https://wiki.python.org/moin/PythonDecoratorLibrary
342 This Wiki page contains several examples of decorators.
343
344 .. ======================================================================
345
346
347 PEP 322: Reverse Iteration
348 ==========================
349
350 A new built-in function, ``reversed(seq)``, takes a sequence and returns an
351 iterator that loops over the elements of the sequence in reverse order. ::
352
353 >>> for i in reversed(xrange(1,4)):
354 ... print i
355 ...
356 3
357 2
358 1
359
360 Compared to extended slicing, such as ``range(1,4)[::-1]``, :func:`reversed` is
361 easier to read, runs faster, and uses substantially less memory.
362
363 Note that :func:`reversed` only accepts sequences, not arbitrary iterators. If
364 you want to reverse an iterator, first convert it to a list with :func:`list`.
365 ::
366
367 >>> input = open('/etc/passwd', 'r')
368 >>> for line in reversed(list(input)):
369 ... print line
370 ...
371 root:*:0:0:System Administrator:/var/root:/bin/tcsh
372 ...
373
374
375 .. seealso::
376
377 :pep:`322` - Reverse Iteration
378 Written and implemented by Raymond Hettinger.
379
380 .. ======================================================================
381
382
383 PEP 324: New subprocess Module
384 ==============================
385
386 The standard library provides a number of ways to execute a subprocess, offering
387 different features and different levels of complexity.
388 ``os.system(command)`` is easy to use, but slow (it runs a shell process
389 which executes the command) and dangerous (you have to be careful about escaping
390 the shell's metacharacters). The :mod:`!popen2` module offers classes that can
391 capture standard output and standard error from the subprocess, but the naming
392 is confusing. The :mod:`subprocess` module cleans this up, providing a unified
393 interface that offers all the features you might need.
394
395 Instead of :mod:`!popen2`'s collection of classes, :mod:`subprocess` contains a
396 single class called :class:`subprocess.Popen` whose constructor supports a number of
397 different keyword arguments. ::
398
399 class Popen(args, bufsize=0, executable=None,
400 stdin=None, stdout=None, stderr=None,
401 preexec_fn=None, close_fds=False, shell=False,
402 cwd=None, env=None, universal_newlines=False,
403 startupinfo=None, creationflags=0):
404
405 *args* is commonly a sequence of strings that will be the arguments to the
406 program executed as the subprocess. (If the *shell* argument is true, *args*
407 can be a string which will then be passed on to the shell for interpretation,
408 just as :func:`os.system` does.)
409
410 *stdin*, *stdout*, and *stderr* specify what the subprocess's input, output, and
411 error streams will be. You can provide a file object or a file descriptor, or
412 you can use the constant ``subprocess.PIPE`` to create a pipe between the
413 subprocess and the parent.
414
415 .. index::
416 single: universal newlines; What's new
417
418 The constructor has a number of handy options:
419
420 * *close_fds* requests that all file descriptors be closed before running the
421 subprocess.
422
423 * *cwd* specifies the working directory in which the subprocess will be executed
424 (defaulting to whatever the parent's working directory is).
425
426 * *env* is a dictionary specifying environment variables.
427
428 * *preexec_fn* is a function that gets called before the child is started.
429
430 * *universal_newlines* opens the child's input and output using Python's
431 :term:`universal newlines` feature.
432
433 Once you've created the :class:`Popen` instance, you can call its :meth:`wait`
434 method to pause until the subprocess has exited, :meth:`poll` to check if it's
435 exited without pausing, or ``communicate(data)`` to send the string *data*
436 to the subprocess's standard input. ``communicate(data)`` then reads any
437 data that the subprocess has sent to its standard output or standard error,
438 returning a tuple ``(stdout_data, stderr_data)``.
439
440 :func:`call` is a shortcut that passes its arguments along to the :class:`Popen`
441 constructor, waits for the command to complete, and returns the status code of
442 the subprocess. It can serve as a safer analog to :func:`os.system`::
443
444 sts = subprocess.call(['dpkg', '-i', '/tmp/new-package.deb'])
445 if sts == 0:
446 # Success
447 ...
448 else:
449 # dpkg returned an error
450 ...
451
452 The command is invoked without use of the shell. If you really do want to use
453 the shell, you can add ``shell=True`` as a keyword argument and provide a string
454 instead of a sequence::
455
456 sts = subprocess.call('dpkg -i /tmp/new-package.deb', shell=True)
457
458 The PEP takes various examples of shell and Python code and shows how they'd be
459 translated into Python code that uses :mod:`subprocess`. Reading this section
460 of the PEP is highly recommended.
461
462
463 .. seealso::
464
465 :pep:`324` - subprocess - New process module
466 Written and implemented by Peter Åstrand, with assistance from Fredrik Lundh and
467 others.
468
469 .. ======================================================================
470
471
472 PEP 327: Decimal Data Type
473 ==========================
474
475 Python has always supported floating-point (FP) numbers, based on the underlying
476 C :c:expr:`double` type, as a data type. However, while most programming
477 languages provide a floating-point type, many people (even programmers) are
478 unaware that floating-point numbers don't represent certain decimal fractions
479 accurately. The new :class:`Decimal` type can represent these fractions
480 accurately, up to a user-specified precision limit.
481
482
483 Why is Decimal needed?
484 ----------------------
485
486 The limitations arise from the representation used for floating-point numbers.
487 FP numbers are made up of three components:
488
489 * The sign, which is positive or negative.
490
491 * The mantissa, which is a single-digit binary number followed by a fractional
492 part. For example, ``1.01`` in base-2 notation is ``1 + 0/2 + 1/4``, or 1.25 in
493 decimal notation.
494
495 * The exponent, which tells where the decimal point is located in the number
496 represented.
497
498 For example, the number 1.25 has positive sign, a mantissa value of 1.01 (in
499 binary), and an exponent of 0 (the decimal point doesn't need to be shifted).
500 The number 5 has the same sign and mantissa, but the exponent is 2 because the
501 mantissa is multiplied by 4 (2 to the power of the exponent 2); 1.25 \* 4 equals
502 5.
503
504 Modern systems usually provide floating-point support that conforms to a
505 standard called IEEE 754. C's :c:expr:`double` type is usually implemented as a
506 64-bit IEEE 754 number, which uses 52 bits of space for the mantissa. This
507 means that numbers can only be specified to 52 bits of precision. If you're
508 trying to represent numbers whose expansion repeats endlessly, the expansion is
509 cut off after 52 bits. Unfortunately, most software needs to produce output in
510 base 10, and common fractions in base 10 are often repeating decimals in binary.
511 For example, 1.1 decimal is binary ``1.0001100110011 ...``; .1 = 1/16 + 1/32 +
512 1/256 plus an infinite number of additional terms. IEEE 754 has to chop off
513 that infinitely repeated decimal after 52 digits, so the representation is
514 slightly inaccurate.
515
516 Sometimes you can see this inaccuracy when the number is printed::
517
518 >>> 1.1
519 1.1000000000000001
520
521 The inaccuracy isn't always visible when you print the number because the
522 FP-to-decimal-string conversion is provided by the C library, and most C libraries try
523 to produce sensible output. Even if it's not displayed, however, the inaccuracy
524 is still there and subsequent operations can magnify the error.
525
526 For many applications this doesn't matter. If I'm plotting points and
527 displaying them on my monitor, the difference between 1.1 and 1.1000000000000001
528 is too small to be visible. Reports often limit output to a certain number of
529 decimal places, and if you round the number to two or three or even eight
530 decimal places, the error is never apparent. However, for applications where it
531 does matter, it's a lot of work to implement your own custom arithmetic
532 routines.
533
534 Hence, the :class:`Decimal` type was created.
535
536
537 The :class:`Decimal` type
538 -------------------------
539
540 A new module, :mod:`decimal`, was added to Python's standard library. It
541 contains two classes, :class:`Decimal` and :class:`Context`. :class:`Decimal`
542 instances represent numbers, and :class:`Context` instances are used to wrap up
543 various settings such as the precision and default rounding mode.
544
545 :class:`Decimal` instances are immutable, like regular Python integers and FP
546 numbers; once it's been created, you can't change the value an instance
547 represents. :class:`Decimal` instances can be created from integers or
548 strings::
549
550 >>> import decimal
551 >>> decimal.Decimal(1972)
552 Decimal("1972")
553 >>> decimal.Decimal("1.1")
554 Decimal("1.1")
555
556 You can also provide tuples containing the sign, the mantissa represented as a
557 tuple of decimal digits, and the exponent::
558
559 >>> decimal.Decimal((1, (1, 4, 7, 5), -2))
560 Decimal("-14.75")
561
562 Cautionary note: the sign bit is a Boolean value, so 0 is positive and 1 is
563 negative.
564
565 Converting from floating-point numbers poses a bit of a problem: should the FP
566 number representing 1.1 turn into the decimal number for exactly 1.1, or for 1.1
567 plus whatever inaccuracies are introduced? The decision was to dodge the issue
568 and leave such a conversion out of the API. Instead, you should convert the
569 floating-point number into a string using the desired precision and pass the
570 string to the :class:`Decimal` constructor::
571
572 >>> f = 1.1
573 >>> decimal.Decimal(str(f))
574 Decimal("1.1")
575 >>> decimal.Decimal('%.12f' % f)
576 Decimal("1.100000000000")
577
578 Once you have :class:`Decimal` instances, you can perform the usual mathematical
579 operations on them. One limitation: exponentiation requires an integer
580 exponent::
581
582 >>> a = decimal.Decimal('35.72')
583 >>> b = decimal.Decimal('1.73')
584 >>> a+b
585 Decimal("37.45")
586 >>> a-b
587 Decimal("33.99")
588 >>> a*b
589 Decimal("61.7956")
590 >>> a/b
591 Decimal("20.64739884393063583815028902")
592 >>> a ** 2
593 Decimal("1275.9184")
594 >>> a**b
595 Traceback (most recent call last):
596 ...
597 decimal.InvalidOperation: x ** (non-integer)
598
599 You can combine :class:`Decimal` instances with integers, but not with
600 floating-point numbers::
601
602 >>> a + 4
603 Decimal("39.72")
604 >>> a + 4.5
605 Traceback (most recent call last):
606 ...
607 TypeError: You can interact Decimal only with int, long or Decimal data types.
608 >>>
609
610 :class:`Decimal` numbers can be used with the :mod:`math` and :mod:`cmath`
611 modules, but note that they'll be immediately converted to floating-point
612 numbers before the operation is performed, resulting in a possible loss of
613 precision and accuracy. You'll also get back a regular floating-point number
614 and not a :class:`Decimal`. ::
615
616 >>> import math, cmath
617 >>> d = decimal.Decimal('123456789012.345')
618 >>> math.sqrt(d)
619 351364.18288201344
620 >>> cmath.sqrt(-d)
621 351364.18288201344j
622
623 :class:`Decimal` instances have a :meth:`sqrt` method that returns a
624 :class:`Decimal`, but if you need other things such as trigonometric functions
625 you'll have to implement them. ::
626
627 >>> d.sqrt()
628 Decimal("351364.1828820134592177245001")
629
630
631 The :class:`Context` type
632 -------------------------
633
634 Instances of the :class:`Context` class encapsulate several settings for
635 decimal operations:
636
637 * :attr:`prec` is the precision, the number of decimal places.
638
639 * :attr:`rounding` specifies the rounding mode. The :mod:`decimal` module has
640 constants for the various possibilities: :const:`ROUND_DOWN`,
641 :const:`ROUND_CEILING`, :const:`ROUND_HALF_EVEN`, and various others.
642
643 * :attr:`traps` is a dictionary specifying what happens on encountering certain
644 error conditions: either an exception is raised or a value is returned. Some
645 examples of error conditions are division by zero, loss of precision, and
646 overflow.
647
648 There's a thread-local default context available by calling :func:`getcontext`;
649 you can change the properties of this context to alter the default precision,
650 rounding, or trap handling. The following example shows the effect of changing
651 the precision of the default context::
652
653 >>> decimal.getcontext().prec
654 28
655 >>> decimal.Decimal(1) / decimal.Decimal(7)
656 Decimal("0.1428571428571428571428571429")
657 >>> decimal.getcontext().prec = 9
658 >>> decimal.Decimal(1) / decimal.Decimal(7)
659 Decimal("0.142857143")
660
661 The default action for error conditions is selectable; the module can either
662 return a special value such as infinity or not-a-number, or exceptions can be
663 raised::
664
665 >>> decimal.Decimal(1) / decimal.Decimal(0)
666 Traceback (most recent call last):
667 ...
668 decimal.DivisionByZero: x / 0
669 >>> decimal.getcontext().traps[decimal.DivisionByZero] = False
670 >>> decimal.Decimal(1) / decimal.Decimal(0)
671 Decimal("Infinity")
672 >>>
673
674 The :class:`Context` instance also has various methods for formatting numbers
675 such as :meth:`to_eng_string` and :meth:`to_sci_string`.
676
677 For more information, see the documentation for the :mod:`decimal` module, which
678 includes a quick-start tutorial and a reference.
679
680
681 .. seealso::
682
683 :pep:`327` - Decimal Data Type
684 Written by Facundo Batista and implemented by Facundo Batista, Eric Price,
685 Raymond Hettinger, Aahz, and Tim Peters.
686
687 `http://www.lahey.com/float.htm <https://web.archive.org/web/20230604072523/http://www.lahey.com/float.htm>`__
688 The article uses Fortran code to illustrate many of the problems that
689 floating-point inaccuracy can cause.
690
691 https://speleotrove.com/decimal/
692 A description of a decimal-based representation. This representation is being
693 proposed as a standard, and underlies the new Python decimal type. Much of this
694 material was written by Mike Cowlishaw, designer of the Rexx language.
695
696 .. ======================================================================
697
698
699 PEP 328: Multi-line Imports
700 ===========================
701
702 One language change is a small syntactic tweak aimed at making it easier to
703 import many names from a module. In a ``from module import names`` statement,
704 *names* is a sequence of names separated by commas. If the sequence is very
705 long, you can either write multiple imports from the same module, or you can use
706 backslashes to escape the line endings like this::
707
708 from SimpleXMLRPCServer import SimpleXMLRPCServer,\
709 SimpleXMLRPCRequestHandler,\
710 CGIXMLRPCRequestHandler,\
711 resolve_dotted_attribute
712
713 The syntactic change in Python 2.4 simply allows putting the names within
714 parentheses. Python ignores newlines within a parenthesized expression, so the
715 backslashes are no longer needed::
716
717 from SimpleXMLRPCServer import (SimpleXMLRPCServer,
718 SimpleXMLRPCRequestHandler,
719 CGIXMLRPCRequestHandler,
720 resolve_dotted_attribute)
721
722 The PEP also proposes that all :keyword:`import` statements be absolute imports,
723 with a leading ``.`` character to indicate a relative import. This part of the
724 PEP was not implemented for Python 2.4, but was completed for Python 2.5.
725
726
727 .. seealso::
728
729 :pep:`328` - Imports: Multi-Line and Absolute/Relative
730 Written by Aahz. Multi-line imports were implemented by Dima Dorfman.
731
732 .. ======================================================================
733
734
735 PEP 331: Locale-Independent Float/String Conversions
736 ====================================================
737
738 The :mod:`locale` modules lets Python software select various conversions and
739 display conventions that are localized to a particular country or language.
740 However, the module was careful to not change the numeric locale because various
741 functions in Python's implementation required that the numeric locale remain set
742 to the ``'C'`` locale. Often this was because the code was using the C
743 library's :c:func:`atof` function.
744
745 Not setting the numeric locale caused trouble for extensions that used third-party
746 C libraries, however, because they wouldn't have the correct locale set.
747 The motivating example was GTK+, whose user interface widgets weren't displaying
748 numbers in the current locale.
749
750 The solution described in the PEP is to add three new functions to the Python
751 API that perform ASCII-only conversions, ignoring the locale setting:
752
753 * ``PyOS_ascii_strtod(str, ptr)`` and ``PyOS_ascii_atof(str, ptr)``
754 both convert a string to a C :c:expr:`double`.
755
756 * ``PyOS_ascii_formatd(buffer, buf_len, format, d)`` converts a
757 :c:expr:`double` to an ASCII string.
758
759 The code for these functions came from the GLib library
760 (`https://developer-old.gnome.org/glib/2.26/ <http://web.archive.org/web/20210306104320/https://developer.gnome.org/glib/2.26/>`__), whose developers kindly
761 relicensed the relevant functions and donated them to the Python Software
762 Foundation. The :mod:`locale` module can now change the numeric locale,
763 letting extensions such as GTK+ produce the correct results.
764
765
766 .. seealso::
767
768 :pep:`331` - Locale-Independent Float/String Conversions
769 Written by Christian R. Reis, and implemented by Gustavo Carneiro.
770
771 .. ======================================================================
772
773
774 Other Language Changes
775 ======================
776
777 Here are all of the changes that Python 2.4 makes to the core Python language.
778
779 * Decorators for functions and methods were added (:pep:`318`).
780
781 * Built-in :func:`set` and :func:`frozenset` types were added (:pep:`218`).
782 Other new built-ins include the ``reversed(seq)`` function (:pep:`322`).
783
784 * Generator expressions were added (:pep:`289`).
785
786 * Certain numeric expressions no longer return values restricted to 32 or 64
787 bits (:pep:`237`).
788
789 * You can now put parentheses around the list of names in a ``from module import
790 names`` statement (:pep:`328`).
791
792 * The :meth:`dict.update` method now accepts the same argument forms as the
793 :class:`dict` constructor. This includes any mapping, any iterable of key/value
794 pairs, and keyword arguments. (Contributed by Raymond Hettinger.)
795
796 * The string methods :meth:`ljust`, :meth:`rjust`, and :meth:`center` now take
797 an optional argument for specifying a fill character other than a space.
798 (Contributed by Raymond Hettinger.)
799
800 * Strings also gained an :meth:`rsplit` method that works like the :meth:`split`
801 method but splits from the end of the string. (Contributed by Sean
802 Reifschneider.) ::
803
804 >>> 'www.python.org'.split('.', 1)
805 ['www', 'python.org']
806 'www.python.org'.rsplit('.', 1)
807 ['www.python', 'org']
808
809 * Three keyword parameters, *cmp*, *key*, and *reverse*, were added to the
810 :meth:`sort` method of lists. These parameters make some common usages of
811 :meth:`sort` simpler. All of these parameters are optional.
812
813 For the *cmp* parameter, the value should be a comparison function that takes
814 two parameters and returns -1, 0, or +1 depending on how the parameters compare.
815 This function will then be used to sort the list. Previously this was the only
816 parameter that could be provided to :meth:`sort`.
817
818 *key* should be a single-parameter function that takes a list element and
819 returns a comparison key for the element. The list is then sorted using the
820 comparison keys. The following example sorts a list case-insensitively::
821
822 >>> L = ['A', 'b', 'c', 'D']
823 >>> L.sort() # Case-sensitive sort
824 >>> L
825 ['A', 'D', 'b', 'c']
826 >>> # Using 'key' parameter to sort list
827 >>> L.sort(key=lambda x: x.lower())
828 >>> L
829 ['A', 'b', 'c', 'D']
830 >>> # Old-fashioned way
831 >>> L.sort(cmp=lambda x,y: cmp(x.lower(), y.lower()))
832 >>> L
833 ['A', 'b', 'c', 'D']
834
835 The last example, which uses the *cmp* parameter, is the old way to perform a
836 case-insensitive sort. It works but is slower than using a *key* parameter.
837 Using *key* calls :meth:`lower` method once for each element in the list while
838 using *cmp* will call it twice for each comparison, so using *key* saves on
839 invocations of the :meth:`lower` method.
840
841 For simple key functions and comparison functions, it is often possible to avoid
842 a :keyword:`lambda` expression by using an unbound method instead. For example,
843 the above case-insensitive sort is best written as::
844
845 >>> L.sort(key=str.lower)
846 >>> L
847 ['A', 'b', 'c', 'D']
848
849 Finally, the *reverse* parameter takes a Boolean value. If the value is true,
850 the list will be sorted into reverse order. Instead of ``L.sort();
851 L.reverse()``, you can now write ``L.sort(reverse=True)``.
852
853 The results of sorting are now guaranteed to be stable. This means that two
854 entries with equal keys will be returned in the same order as they were input.
855 For example, you can sort a list of people by name, and then sort the list by
856 age, resulting in a list sorted by age where people with the same age are in
857 name-sorted order.
858
859 (All changes to :meth:`sort` contributed by Raymond Hettinger.)
860
861 * There is a new built-in function ``sorted(iterable)`` that works like the
862 in-place :meth:`list.sort` method but can be used in expressions. The
863 differences are:
864
865 * the input may be any iterable;
866
867 * a newly formed copy is sorted, leaving the original intact; and
868
869 * the expression returns the new sorted copy
870
871 ::
872
873 >>> L = [9,7,8,3,2,4,1,6,5]
874 >>> [10+i for i in sorted(L)] # usable in a list comprehension
875 [11, 12, 13, 14, 15, 16, 17, 18, 19]
876 >>> L # original is left unchanged
877 [9,7,8,3,2,4,1,6,5]
878 >>> sorted('Monty Python') # any iterable may be an input
879 [' ', 'M', 'P', 'h', 'n', 'n', 'o', 'o', 't', 't', 'y', 'y']
880
881 >>> # List the contents of a dict sorted by key values
882 >>> colormap = dict(red=1, blue=2, green=3, black=4, yellow=5)
883 >>> for k, v in sorted(colormap.iteritems()):
884 ... print k, v
885 ...
886 black 4
887 blue 2
888 green 3
889 red 1
890 yellow 5
891
892 (Contributed by Raymond Hettinger.)
893
894 * Integer operations will no longer trigger an :exc:`OverflowWarning`. The
895 :exc:`OverflowWarning` warning will disappear in Python 2.5.
896
897 * The interpreter gained a new switch, :option:`-m`, that takes a name, searches
898 for the corresponding module on ``sys.path``, and runs the module as a script.
899 For example, you can now run the Python profiler with ``python -m profile``.
900 (Contributed by Nick Coghlan.)
901
902 * The ``eval(expr, globals, locals)`` and ``execfile(filename, globals,
903 locals)`` functions and the ``exec`` statement now accept any mapping type
904 for the *locals* parameter. Previously this had to be a regular Python
905 dictionary. (Contributed by Raymond Hettinger.)
906
907 * The :func:`zip` built-in function and :func:`itertools.izip` now return an
908 empty list if called with no arguments. Previously they raised a
909 :exc:`TypeError` exception. This makes them more suitable for use with variable
910 length argument lists::
911
912 >>> def transpose(array):
913 ... return zip(*array)
914 ...
915 >>> transpose([(1,2,3), (4,5,6)])
916 [(1, 4), (2, 5), (3, 6)]
917 >>> transpose([])
918 []
919
920 (Contributed by Raymond Hettinger.)
921
922 * Encountering a failure while importing a module no longer leaves a partially initialized
923 module object in ``sys.modules``. The incomplete module object left
924 behind would fool further imports of the same module into succeeding, leading to
925 confusing errors. (Fixed by Tim Peters.)
926
927 * :const:`None` is now a constant; code that binds a new value to the name
928 ``None`` is now a syntax error. (Contributed by Raymond Hettinger.)
929
930 .. ======================================================================
931
932
933 Optimizations
934 -------------
935
936 * The inner loops for list and tuple slicing were optimized and now run about
937 one-third faster. The inner loops for dictionaries were also optimized,
938 resulting in performance boosts for :meth:`keys`, :meth:`values`, :meth:`items`,
939 :meth:`iterkeys`, :meth:`itervalues`, and :meth:`iteritems`. (Contributed by
940 Raymond Hettinger.)
941
942 * The machinery for growing and shrinking lists was optimized for speed and for
943 space efficiency. Appending and popping from lists now runs faster due to more
944 efficient code paths and less frequent use of the underlying system
945 :c:func:`realloc`. List comprehensions also benefit. :meth:`list.extend` was
946 also optimized and no longer converts its argument into a temporary list before
947 extending the base list. (Contributed by Raymond Hettinger.)
948
949 * :func:`list`, :func:`tuple`, :func:`map`, :func:`filter`, and :func:`zip` now
950 run several times faster with non-sequence arguments that supply a
951 :meth:`__len__` method. (Contributed by Raymond Hettinger.)
952
953 * The methods :meth:`list.__getitem__`, :meth:`dict.__getitem__`, and
954 :meth:`dict.__contains__` are now implemented as :class:`method_descriptor`
955 objects rather than :class:`wrapper_descriptor` objects. This form of access
956 doubles their performance and makes them more suitable for use as arguments to
957 functionals: ``map(mydict.__getitem__, keylist)``. (Contributed by Raymond
958 Hettinger.)
959
960 * Added a new opcode, ``LIST_APPEND``, that simplifies the generated bytecode
961 for list comprehensions and speeds them up by about a third. (Contributed by
962 Raymond Hettinger.)
963
964 * The peephole bytecode optimizer has been improved to produce shorter, faster
965 bytecode; remarkably, the resulting bytecode is more readable. (Enhanced by
966 Raymond Hettinger.)
967
968 * String concatenations in statements of the form ``s = s + "abc"`` and ``s +=
969 "abc"`` are now performed more efficiently in certain circumstances. This
970 optimization won't be present in other Python implementations such as Jython, so
971 you shouldn't rely on it; using the :meth:`join` method of strings is still
972 recommended when you want to efficiently glue a large number of strings
973 together. (Contributed by Armin Rigo.)
974
975 The net result of the 2.4 optimizations is that Python 2.4 runs the pystone
976 benchmark around 5% faster than Python 2.3 and 35% faster than Python 2.2.
977 (pystone is not a particularly good benchmark, but it's the most commonly used
978 measurement of Python's performance. Your own applications may show greater or
979 smaller benefits from Python 2.4.)
980
981 .. pystone is almost useless for comparing different versions of Python;
982 instead, it excels at predicting relative Python performance on different
983 machines. So, this section would be more informative if it used other tools
984 such as pybench and parrotbench. For a more application oriented benchmark,
985 try comparing the timings of test_decimal.py under 2.3 and 2.4.
986
987 .. ======================================================================
988
989
990 New, Improved, and Deprecated Modules
991 =====================================
992
993 As usual, Python's standard library received a number of enhancements and bug
994 fixes. Here's a partial list of the most notable changes, sorted alphabetically
995 by module name. Consult the :file:`Misc/NEWS` file in the source tree for a more
996 complete list of changes, or look through the CVS logs for all the details.
997
998 * The :mod:`!asyncore` module's :func:`!loop` function now has a *count* parameter
999 that lets you perform a limited number of passes through the polling loop. The
1000 default is still to loop forever.
1001
1002 * The :mod:`base64` module now has more complete :rfc:`3548` support for Base64,
1003 Base32, and Base16 encoding and decoding, including optional case folding and
1004 optional alternative alphabets. (Contributed by Barry Warsaw.)
1005
1006 * The :mod:`bisect` module now has an underlying C implementation for improved
1007 performance. (Contributed by Dmitry Vasiliev.)
1008
1009 * The CJKCodecs collections of East Asian codecs, maintained by Hye-Shik Chang,
1010 was integrated into 2.4. The new encodings are:
1011
1012 * Chinese (PRC): gb2312, gbk, gb18030, big5hkscs, hz
1013
1014 * Chinese (ROC): big5, cp950
1015
1016 * Japanese: cp932, euc-jis-2004, euc-jp, euc-jisx0213, iso-2022-jp,
1017 iso-2022-jp-1, iso-2022-jp-2, iso-2022-jp-3, iso-2022-jp-ext, iso-2022-jp-2004,
1018 shift-jis, shift-jisx0213, shift-jis-2004
1019
1020 * Korean: cp949, euc-kr, johab, iso-2022-kr
1021
1022 * Some other new encodings were added: HP Roman8, ISO_8859-11, ISO_8859-16,
1023 PCTP-154, and TIS-620.
1024
1025 * The UTF-8 and UTF-16 codecs now cope better with receiving partial input.
1026 Previously the :class:`StreamReader` class would try to read more data, making
1027 it impossible to resume decoding from the stream. The :meth:`read` method will
1028 now return as much data as it can and future calls will resume decoding where
1029 previous ones left off. (Implemented by Walter Dörwald.)
1030
1031 * There is a new :mod:`collections` module for various specialized collection
1032 datatypes. Currently it contains just one type, :class:`deque`, a double-ended
1033 queue that supports efficiently adding and removing elements from either
1034 end::
1035
1036 >>> from collections import deque
1037 >>> d = deque('ghi') # make a new deque with three items
1038 >>> d.append('j') # add a new entry to the right side
1039 >>> d.appendleft('f') # add a new entry to the left side
1040 >>> d # show the representation of the deque
1041 deque(['f', 'g', 'h', 'i', 'j'])
1042 >>> d.pop() # return and remove the rightmost item
1043 'j'
1044 >>> d.popleft() # return and remove the leftmost item
1045 'f'
1046 >>> list(d) # list the contents of the deque
1047 ['g', 'h', 'i']
1048 >>> 'h' in d # search the deque
1049 True
1050
1051 Several modules, such as the :mod:`Queue` and :mod:`threading` modules, now take
1052 advantage of :class:`collections.deque` for improved performance. (Contributed
1053 by Raymond Hettinger.)
1054
1055 * The :mod:`ConfigParser <configparser>` classes have been enhanced slightly. The :meth:`~configparser.ConfigParser.read`
1056 method now returns a list of the files that were successfully parsed, and the
1057 :meth:`~configparser.ConfigParser.set` method raises :exc:`TypeError` if passed a *value* argument that
1058 isn't a string. (Contributed by John Belmonte and David Goodger.)
1059
1060 * The :mod:`curses` module now supports the ncurses extension
1061 :func:`use_default_colors`. On platforms where the terminal supports
1062 transparency, this makes it possible to use a transparent background.
1063 (Contributed by Jörg Lehmann.)
1064
1065 * The :mod:`difflib` module now includes an :class:`HtmlDiff` class that creates
1066 an HTML table showing a side by side comparison of two versions of a text.
1067 (Contributed by Dan Gass.)
1068
1069 * The :mod:`email` package was updated to version 3.0, which dropped various
1070 deprecated APIs and removes support for Python versions earlier than 2.3. The
1071 3.0 version of the package uses a new incremental parser for MIME messages,
1072 available in the :mod:`email.FeedParser` module. The new parser doesn't require
1073 reading the entire message into memory, and doesn't raise exceptions if a
1074 message is malformed; instead it records any problems in the :attr:`defect`
1075 attribute of the message. (Developed by Anthony Baxter, Barry Warsaw, Thomas
1076 Wouters, and others.)
1077
1078 * The :mod:`heapq` module has been converted to C. The resulting tenfold
1079 improvement in speed makes the module suitable for handling high volumes of
1080 data. In addition, the module has two new functions :func:`nlargest` and
1081 :func:`nsmallest` that use heaps to find the N largest or smallest values in a
1082 dataset without the expense of a full sort. (Contributed by Raymond Hettinger.)
1083
1084 * The :mod:`httplib <http>` module now contains constants for HTTP status codes defined
1085 in various HTTP-related RFC documents. Constants have names such as
1086 :const:`OK`, :const:`CREATED`, :const:`CONTINUE`, and
1087 :const:`MOVED_PERMANENTLY`; use pydoc to get a full list. (Contributed by
1088 Andrew Eland.)
1089
1090 * The :mod:`imaplib` module now supports IMAP's THREAD command (contributed by
1091 Yves Dionne) and new :meth:`deleteacl` and :meth:`myrights` methods (contributed
1092 by Arnaud Mazin).
1093
1094 * The :mod:`itertools` module gained a ``groupby(iterable[, *func*])``
1095 function. *iterable* is something that can be iterated over to return a stream
1096 of elements, and the optional *func* parameter is a function that takes an
1097 element and returns a key value; if omitted, the key is simply the element
1098 itself. :func:`groupby` then groups the elements into subsequences which have
1099 matching values of the key, and returns a series of 2-tuples containing the key
1100 value and an iterator over the subsequence.
1101
1102 Here's an example to make this clearer. The *key* function simply returns
1103 whether a number is even or odd, so the result of :func:`groupby` is to return
1104 consecutive runs of odd or even numbers. ::
1105
1106 >>> import itertools
1107 >>> L = [2, 4, 6, 7, 8, 9, 11, 12, 14]
1108 >>> for key_val, it in itertools.groupby(L, lambda x: x % 2):
1109 ... print key_val, list(it)
1110 ...
1111 0 [2, 4, 6]
1112 1 [7]
1113 0 [8]
1114 1 [9, 11]
1115 0 [12, 14]
1116 >>>
1117
1118 :func:`groupby` is typically used with sorted input. The logic for
1119 :func:`groupby` is similar to the Unix ``uniq`` filter which makes it handy for
1120 eliminating, counting, or identifying duplicate elements::
1121
1122 >>> word = 'abracadabra'
1123 >>> letters = sorted(word) # Turn string into a sorted list of letters
1124 >>> letters
1125 ['a', 'a', 'a', 'a', 'a', 'b', 'b', 'c', 'd', 'r', 'r']
1126 >>> for k, g in itertools.groupby(letters):
1127 ... print k, list(g)
1128 ...
1129 a ['a', 'a', 'a', 'a', 'a']
1130 b ['b', 'b']
1131 c ['c']
1132 d ['d']
1133 r ['r', 'r']
1134 >>> # List unique letters
1135 >>> [k for k, g in groupby(letters)]
1136 ['a', 'b', 'c', 'd', 'r']
1137 >>> # Count letter occurrences
1138 >>> [(k, len(list(g))) for k, g in groupby(letters)]
1139 [('a', 5), ('b', 2), ('c', 1), ('d', 1), ('r', 2)]
1140
1141 (Contributed by Hye-Shik Chang.)
1142
1143 * :mod:`itertools` also gained a function named ``tee(iterator, N)`` that
1144 returns *N* independent iterators that replicate *iterator*. If *N* is omitted,
1145 the default is 2. ::
1146
1147 >>> L = [1,2,3]
1148 >>> i1, i2 = itertools.tee(L)
1149 >>> i1,i2
1150 (<itertools.tee object at 0x402c2080>, <itertools.tee object at 0x402c2090>)
1151 >>> list(i1) # Run the first iterator to exhaustion
1152 [1, 2, 3]
1153 >>> list(i2) # Run the second iterator to exhaustion
1154 [1, 2, 3]
1155
1156 Note that :func:`tee` has to keep copies of the values returned by the
1157 iterator; in the worst case, it may need to keep all of them. This should
1158 therefore be used carefully if the leading iterator can run far ahead of the
1159 trailing iterator in a long stream of inputs. If the separation is large, then
1160 you might as well use :func:`list` instead. When the iterators track closely
1161 with one another, :func:`tee` is ideal. Possible applications include
1162 bookmarking, windowing, or lookahead iterators. (Contributed by Raymond
1163 Hettinger.)
1164
1165 * A number of functions were added to the :mod:`locale` module, such as
1166 :func:`bind_textdomain_codeset` to specify a particular encoding and a family of
1167 :func:`!l\*gettext` functions that return messages in the chosen encoding.
1168 (Contributed by Gustavo Niemeyer.)
1169
1170 * Some keyword arguments were added to the :mod:`logging` package's
1171 :func:`basicConfig` function to simplify log configuration. The default
1172 behavior is to log messages to standard error, but various keyword arguments can
1173 be specified to log to a particular file, change the logging format, or set the
1174 logging level. For example::
1175
1176 import logging
1177 logging.basicConfig(filename='/var/log/application.log',
1178 level=0, # Log all messages
1179 format='%(levelname):%(process):%(thread):%(message)')
1180
1181 Other additions to the :mod:`logging` package include a ``log(level, msg)``
1182 convenience method, as well as a :class:`TimedRotatingFileHandler` class that
1183 rotates its log files at a timed interval. The module already had
1184 :class:`RotatingFileHandler`, which rotated logs once the file exceeded a
1185 certain size. Both classes derive from a new :class:`BaseRotatingHandler` class
1186 that can be used to implement other rotating handlers.
1187
1188 (Changes implemented by Vinay Sajip.)
1189
1190 * The :mod:`marshal` module now shares interned strings on unpacking a data
1191 structure. This may shrink the size of certain pickle strings, but the primary
1192 effect is to make :file:`.pyc` files significantly smaller. (Contributed by
1193 Martin von Löwis.)
1194
1195 * The :mod:`!nntplib` module's :class:`NNTP` class gained :meth:`description` and
1196 :meth:`descriptions` methods to retrieve newsgroup descriptions for a single
1197 group or for a range of groups. (Contributed by Jürgen A. Erhard.)
1198
1199 * Two new functions were added to the :mod:`operator` module,
1200 ``attrgetter(attr)`` and ``itemgetter(index)``. Both functions return
1201 callables that take a single argument and return the corresponding attribute or
1202 item; these callables make excellent data extractors when used with :func:`map`
1203 or :func:`sorted`. For example::
1204
1205 >>> L = [('c', 2), ('d', 1), ('a', 4), ('b', 3)]
1206 >>> map(operator.itemgetter(0), L)
1207 ['c', 'd', 'a', 'b']
1208 >>> map(operator.itemgetter(1), L)
1209 [2, 1, 4, 3]
1210 >>> sorted(L, key=operator.itemgetter(1)) # Sort list by second tuple item
1211 [('d', 1), ('c', 2), ('b', 3), ('a', 4)]
1212
1213 (Contributed by Raymond Hettinger.)
1214
1215 * The :mod:`optparse` module was updated in various ways. The module now passes
1216 its messages through :func:`gettext.gettext`, making it possible to
1217 internationalize Optik's help and error messages. Help messages for options can
1218 now include the string ``'%default'``, which will be replaced by the option's
1219 default value. (Contributed by Greg Ward.)
1220
1221 * The long-term plan is to deprecate the :mod:`!rfc822` module in some future
1222 Python release in favor of the :mod:`email` package. To this end, the
1223 :func:`email.Utils.formatdate <email.utils.formatdate>` function has been changed to make it usable as a
1224 replacement for :func:`!rfc822.formatdate`. You may want to write new e-mail
1225 processing code with this in mind. (Change implemented by Anthony Baxter.)
1226
1227 * A new ``urandom(n)`` function was added to the :mod:`os` module, returning
1228 a string containing *n* bytes of random data. This function provides access to
1229 platform-specific sources of randomness such as :file:`/dev/urandom` on Linux or
1230 the Windows CryptoAPI. (Contributed by Trevor Perrin.)
1231
1232 * Another new function: ``os.path.lexists(path)`` returns true if the file
1233 specified by *path* exists, whether or not it's a symbolic link. This differs
1234 from the existing ``os.path.exists(path)`` function, which returns false if
1235 *path* is a symlink that points to a destination that doesn't exist.
1236 (Contributed by Beni Cherniavsky.)
1237
1238 * A new :func:`getsid` function was added to the :mod:`posix` module that
1239 underlies the :mod:`os` module. (Contributed by J. Raynor.)
1240
1241 * The :mod:`poplib` module now supports POP over SSL. (Contributed by Hector
1242 Urtubia.)
1243
1244 * The :mod:`profile` module can now profile C extension functions. (Contributed
1245 by Nick Bastin.)
1246
1247 * The :mod:`random` module has a new method called ``getrandbits(N)`` that
1248 returns a long integer *N* bits in length. The existing :meth:`randrange`
1249 method now uses :meth:`getrandbits` where appropriate, making generation of
1250 arbitrarily large random numbers more efficient. (Contributed by Raymond
1251 Hettinger.)
1252
1253 * The regular expression language accepted by the :mod:`re` module was extended
1254 with simple conditional expressions, written as ``(?(group)A|B)``. *group* is
1255 either a numeric group ID or a group name defined with ``(?P<group>...)``
1256 earlier in the expression. If the specified group matched, the regular
1257 expression pattern *A* will be tested against the string; if the group didn't
1258 match, the pattern *B* will be used instead. (Contributed by Gustavo Niemeyer.)
1259
1260 * The :mod:`re` module is also no longer recursive, thanks to a massive amount
1261 of work by Gustavo Niemeyer. In a recursive regular expression engine, certain
1262 patterns result in a large amount of C stack space being consumed, and it was
1263 possible to overflow the stack. For example, if you matched a 30000-byte string
1264 of ``a`` characters against the expression ``(a|b)+``, one stack frame was
1265 consumed per character. Python 2.3 tried to check for stack overflow and raise
1266 a :exc:`RuntimeError` exception, but certain patterns could sidestep the
1267 checking and if you were unlucky Python could segfault. Python 2.4's regular
1268 expression engine can match this pattern without problems.
1269
1270 * The :mod:`signal` module now performs tighter error-checking on the parameters
1271 to the :func:`signal.signal` function. For example, you can't set a handler on
1272 the :const:`SIGKILL` signal; previous versions of Python would quietly accept
1273 this, but 2.4 will raise a :exc:`RuntimeError` exception.
1274
1275 * Two new functions were added to the :mod:`socket` module. :func:`socketpair`
1276 returns a pair of connected sockets and ``getservbyport(port)`` looks up the
1277 service name for a given port number. (Contributed by Dave Cole and Barry
1278 Warsaw.)
1279
1280 * The :func:`sys.exitfunc` function has been deprecated. Code should be using
1281 the existing :mod:`atexit` module, which correctly handles calling multiple exit
1282 functions. Eventually :func:`sys.exitfunc` will become a purely internal
1283 interface, accessed only by :mod:`atexit`.
1284
1285 * The :mod:`tarfile` module now generates GNU-format tar files by default.
1286 (Contributed by Lars Gustäbel.)
1287
1288 * The :mod:`threading` module now has an elegantly simple way to support
1289 thread-local data. The module contains a :class:`local` class whose attribute
1290 values are local to different threads. ::
1291
1292 import threading
1293
1294 data = threading.local()
1295 data.number = 42
1296 data.url = ('www.python.org', 80)
1297
1298 Other threads can assign and retrieve their own values for the :attr:`number`
1299 and :attr:`url` attributes. You can subclass :class:`local` to initialize
1300 attributes or to add methods. (Contributed by Jim Fulton.)
1301
1302 * The :mod:`timeit` module now automatically disables periodic garbage
1303 collection during the timing loop. This change makes consecutive timings more
1304 comparable. (Contributed by Raymond Hettinger.)
1305
1306 * The :mod:`weakref` module now supports a wider variety of objects including
1307 Python functions, class instances, sets, frozensets, deques, arrays, files,
1308 sockets, and regular expression pattern objects. (Contributed by Raymond
1309 Hettinger.)
1310
1311 * The :mod:`xmlrpclib <xmlrpc.client>` module now supports a multi-call extension for
1312 transmitting multiple XML-RPC calls in a single HTTP operation. (Contributed by
1313 Brian Quinlan.)
1314
1315 * The :mod:`mpz`, :mod:`rotor`, and :mod:`xreadlines` modules have been
1316 removed.
1317
1318 .. ======================================================================
1319 .. whole new modules get described in subsections here
1320 .. =====================
1321
1322
1323 cookielib
1324 ---------
1325
1326 The :mod:`cookielib <http.cookiejar>` library supports client-side handling for HTTP cookies,
1327 mirroring the :mod:`Cookie <http.cookies>` module's server-side cookie support. Cookies are
1328 stored in cookie jars; the library transparently stores cookies offered by the
1329 web server in the cookie jar, and fetches the cookie from the jar when
1330 connecting to the server. As in web browsers, policy objects control whether
1331 cookies are accepted or not.
1332
1333 In order to store cookies across sessions, two implementations of cookie jars
1334 are provided: one that stores cookies in the Netscape format so applications can
1335 use the Mozilla or Lynx cookie files, and one that stores cookies in the same
1336 format as the Perl libwww library.
1337
1338 :mod:`urllib2 <urllib.request>` has been changed to interact with :mod:`cookielib <http.cookiejar>`:
1339 :class:`HTTPCookieProcessor` manages a cookie jar that is used when accessing
1340 URLs.
1341
1342 This module was contributed by John J. Lee.
1343
1344 .. ==================
1345
1346
1347 doctest
1348 -------
1349
1350 The :mod:`doctest` module underwent considerable refactoring thanks to Edward
1351 Loper and Tim Peters. Testing can still be as simple as running
1352 :func:`doctest.testmod`, but the refactorings allow customizing the module's
1353 operation in various ways
1354
1355 The new :class:`DocTestFinder` class extracts the tests from a given object's
1356 docstrings::
1357
1358 def f (x, y):
1359 """>>> f(2,2)
1360 4
1361 >>> f(3,2)
1362 6
1363 """
1364 return x*y
1365
1366 finder = doctest.DocTestFinder()
1367
1368 # Get list of DocTest instances
1369 tests = finder.find(f)
1370
1371 The new :class:`DocTestRunner` class then runs individual tests and can produce
1372 a summary of the results::
1373
1374 runner = doctest.DocTestRunner()
1375 for t in tests:
1376 tried, failed = runner.run(t)
1377
1378 runner.summarize(verbose=1)
1379
1380 The above example produces the following output::
1381
1382 1 items passed all tests:
1383 2 tests in f
1384 2 tests in 1 items.
1385 2 passed and 0 failed.
1386 Test passed.
1387
1388 :class:`DocTestRunner` uses an instance of the :class:`OutputChecker` class to
1389 compare the expected output with the actual output. This class takes a number
1390 of different flags that customize its behaviour; ambitious users can also write
1391 a completely new subclass of :class:`OutputChecker`.
1392
1393 The default output checker provides a number of handy features. For example,
1394 with the :const:`doctest.ELLIPSIS` option flag, an ellipsis (``...``) in the
1395 expected output matches any substring, making it easier to accommodate outputs
1396 that vary in minor ways::
1397
1398 def o (n):
1399 """>>> o(1)
1400 <__main__.C instance at 0x...>
1401 >>>
1402 """
1403
1404 Another special string, ``<BLANKLINE>``, matches a blank line::
1405
1406 def p (n):
1407 """>>> p(1)
1408 <BLANKLINE>
1409 >>>
1410 """
1411
1412 Another new capability is producing a diff-style display of the output by
1413 specifying the :const:`doctest.REPORT_UDIFF` (unified diffs),
1414 :const:`doctest.REPORT_CDIFF` (context diffs), or :const:`doctest.REPORT_NDIFF`
1415 (delta-style) option flags. For example::
1416
1417 def g (n):
1418 """>>> g(4)
1419 here
1420 is
1421 a
1422 lengthy
1423 >>>"""
1424 L = 'here is a rather lengthy list of words'.split()
1425 for word in L[:n]:
1426 print word
1427
1428 Running the above function's tests with :const:`doctest.REPORT_UDIFF` specified,
1429 you get the following output:
1430
1431 .. code-block:: none
1432
1433 **********************************************************************
1434 File "t.py", line 15, in g
1435 Failed example:
1436 g(4)
1437 Differences (unified diff with -expected +actual):
1438 @@ -2,3 +2,3 @@
1439 is
1440 a
1441 -lengthy
1442 +rather
1443 **********************************************************************
1444
1445 .. ======================================================================
1446
1447
1448 Build and C API Changes
1449 =======================
1450
1451 Some of the changes to Python's build process and to the C API are:
1452
1453 * Three new convenience macros were added for common return values from
1454 extension functions: :c:macro:`Py_RETURN_NONE`, :c:macro:`Py_RETURN_TRUE`, and
1455 :c:macro:`Py_RETURN_FALSE`. (Contributed by Brett Cannon.)
1456
1457 * Another new macro, :c:macro:`Py_CLEAR`, decreases the reference count of
1458 *obj* and sets *obj* to the null pointer. (Contributed by Jim Fulton.)
1459
1460 * A new function, ``PyTuple_Pack(N, obj1, obj2, ..., objN)``, constructs
1461 tuples from a variable length argument list of Python objects. (Contributed by
1462 Raymond Hettinger.)
1463
1464 * A new function, ``PyDict_Contains(d, k)``, implements fast dictionary
1465 lookups without masking exceptions raised during the look-up process.
1466 (Contributed by Raymond Hettinger.)
1467
1468 * The :c:expr:`Py_IS_NAN(X)` macro returns 1 if its float or double argument
1469 *X* is a NaN. (Contributed by Tim Peters.)
1470
1471 * C code can avoid unnecessary locking by using the new
1472 :c:func:`!PyEval_ThreadsInitialized` function to tell if any thread operations
1473 have been performed. If this function returns false, no lock operations are
1474 needed. (Contributed by Nick Coghlan.)
1475
1476 * A new function, :c:func:`PyArg_VaParseTupleAndKeywords`, is the same as
1477 :c:func:`PyArg_ParseTupleAndKeywords` but takes a :c:type:`va_list` instead of a
1478 number of arguments. (Contributed by Greg Chapman.)
1479
1480 * A new method flag, :c:macro:`METH_COEXIST`, allows a function defined in slots
1481 to co-exist with a :c:type:`PyCFunction` having the same name. This can halve
1482 the access time for a method such as :meth:`set.__contains__`. (Contributed by
1483 Raymond Hettinger.)
1484
1485 * Python can now be built with additional profiling for the interpreter itself,
1486 intended as an aid to people developing the Python core. Providing
1487 :option:`!--enable-profiling` to the :program:`configure` script will let you
1488 profile the interpreter with :program:`gprof`, and providing the
1489 :option:`!--with-tsc` switch enables profiling using the Pentium's
1490 Time-Stamp-Counter register. Note that the :option:`!--with-tsc` switch is slightly
1491 misnamed, because the profiling feature also works on the PowerPC platform,
1492 though that processor architecture doesn't call that register "the TSC
1493 register". (Contributed by Jeremy Hylton.)
1494
1495 * The :c:type:`!tracebackobject` type has been renamed to
1496 :c:type:`PyTracebackObject`.
1497
1498 .. ======================================================================
1499
1500
1501 Port-Specific Changes
1502 ---------------------
1503
1504 * The Windows port now builds under MSVC++ 7.1 as well as version 6.
1505 (Contributed by Martin von Löwis.)
1506
1507 .. ======================================================================
1508
1509
1510 Porting to Python 2.4
1511 =====================
1512
1513 This section lists previously described changes that may require changes to your
1514 code:
1515
1516 * Left shifts and hexadecimal/octal constants that are too large no longer
1517 trigger a :exc:`FutureWarning` and return a value limited to 32 or 64 bits;
1518 instead they return a long integer.
1519
1520 * Integer operations will no longer trigger an :exc:`OverflowWarning`. The
1521 :exc:`OverflowWarning` warning will disappear in Python 2.5.
1522
1523 * The :func:`zip` built-in function and :func:`itertools.izip` now return an
1524 empty list instead of raising a :exc:`TypeError` exception if called with no
1525 arguments.
1526
1527 * You can no longer compare the :class:`date` and :class:`~datetime.datetime` instances
1528 provided by the :mod:`datetime` module. Two instances of different classes
1529 will now always be unequal, and relative comparisons (``<``, ``>``) will raise
1530 a :exc:`TypeError`.
1531
1532 * :func:`!dircache.listdir` now passes exceptions to the caller instead of
1533 returning empty lists.
1534
1535 * :func:`LexicalHandler.startDTD` used to receive the public and system IDs in
1536 the wrong order. This has been corrected; applications relying on the wrong
1537 order need to be fixed.
1538
1539 * :func:`fcntl.ioctl` now warns if the *mutate* argument is omitted and
1540 relevant.
1541
1542 * The :mod:`tarfile` module now generates GNU-format tar files by default.
1543
1544 * Encountering a failure while importing a module no longer leaves a
1545 partially initialized module object in ``sys.modules``.
1546
1547 * :const:`None` is now a constant; code that binds a new value to the name
1548 ``None`` is now a syntax error.
1549
1550 * The :func:`signals.signal` function now raises a :exc:`RuntimeError` exception
1551 for certain illegal values; previously these errors would pass silently. For
1552 example, you can no longer set a handler on the :const:`SIGKILL` signal.
1553
1554 .. ======================================================================
1555
1556
1557 .. _24acks:
1558
1559 Acknowledgements
1560 ================
1561
1562 The author would like to thank the following people for offering suggestions,
1563 corrections and assistance with various drafts of this article: Koray Can,
1564 Hye-Shik Chang, Michael Dyck, Raymond Hettinger, Brian Hurt, Hamish Lawson,
1565 Fredrik Lundh, Sean Reifschneider, Sadruddin Rejeb.
1566