| 1 |
.. _tut-brieftourtwo: |
| 2 |
|
| 3 |
********************************************** |
| 4 |
Brief tour of the standard library --- part II |
| 5 |
********************************************** |
| 6 |
|
| 7 |
This second tour covers more advanced modules that support professional |
| 8 |
programming needs. These modules rarely occur in small scripts. |
| 9 |
|
| 10 |
|
| 11 |
.. _tut-output-formatting: |
| 12 |
|
| 13 |
Output formatting |
| 14 |
================= |
| 15 |
|
| 16 |
The :mod:`reprlib` module provides a version of :func:`repr` customized for |
| 17 |
abbreviated displays of large or deeply nested containers:: |
| 18 |
|
| 19 |
>>> import reprlib |
| 20 |
>>> reprlib.repr(set('supercalifragilisticexpialidocious')) |
| 21 |
"{'a', 'c', 'd', 'e', 'f', 'g', ...}" |
| 22 |
|
| 23 |
The :mod:`pprint` module offers more sophisticated control over printing both |
| 24 |
built-in and user defined objects in a way that is readable by the interpreter. |
| 25 |
When the result is longer than one line, the "pretty printer" adds line breaks |
| 26 |
and indentation to more clearly reveal data structure:: |
| 27 |
|
| 28 |
>>> import pprint |
| 29 |
>>> t = [[[['black', 'cyan'], 'white', ['green', 'red']], [['magenta', |
| 30 |
... 'yellow'], 'blue']]] |
| 31 |
... |
| 32 |
>>> pprint.pprint(t, width=30) |
| 33 |
[[[['black', 'cyan'], |
| 34 |
'white', |
| 35 |
['green', 'red']], |
| 36 |
[['magenta', 'yellow'], |
| 37 |
'blue']]] |
| 38 |
|
| 39 |
The :mod:`textwrap` module formats paragraphs of text to fit a given screen |
| 40 |
width:: |
| 41 |
|
| 42 |
>>> import textwrap |
| 43 |
>>> doc = """The wrap() method is just like fill() except that it returns |
| 44 |
... a list of strings instead of one big string with newlines to separate |
| 45 |
... the wrapped lines.""" |
| 46 |
... |
| 47 |
>>> print(textwrap.fill(doc, width=40)) |
| 48 |
The wrap() method is just like fill() |
| 49 |
except that it returns a list of strings |
| 50 |
instead of one big string with newlines |
| 51 |
to separate the wrapped lines. |
| 52 |
|
| 53 |
The :mod:`locale` module accesses a database of culture specific data formats. |
| 54 |
The grouping attribute of locale's format function provides a direct way of |
| 55 |
formatting numbers with group separators:: |
| 56 |
|
| 57 |
>>> import locale |
| 58 |
>>> locale.setlocale(locale.LC_ALL, 'English_United States.1252') |
| 59 |
'English_United States.1252' |
| 60 |
>>> conv = locale.localeconv() # get a mapping of conventions |
| 61 |
>>> x = 1234567.8 |
| 62 |
>>> locale.format_string("%d", x, grouping=True) |
| 63 |
'1,234,567' |
| 64 |
>>> locale.format_string("%s%.*f", (conv['currency_symbol'], |
| 65 |
... conv['frac_digits'], x), grouping=True) |
| 66 |
'$1,234,567.80' |
| 67 |
|
| 68 |
|
| 69 |
.. _tut-templating: |
| 70 |
|
| 71 |
Templating |
| 72 |
========== |
| 73 |
|
| 74 |
The :mod:`string` module includes a versatile :class:`~string.Template` class |
| 75 |
with a simplified syntax suitable for editing by end-users. This allows users |
| 76 |
to customize their applications without having to alter the application. |
| 77 |
|
| 78 |
The format uses placeholder names formed by ``$`` with valid Python identifiers |
| 79 |
(alphanumeric characters and underscores). Surrounding the placeholder with |
| 80 |
braces allows it to be followed by more alphanumeric letters with no intervening |
| 81 |
spaces. Writing ``$$`` creates a single escaped ``$``:: |
| 82 |
|
| 83 |
>>> from string import Template |
| 84 |
>>> t = Template('${village}folk send $$10 to $cause.') |
| 85 |
>>> t.substitute(village='Nottingham', cause='the ditch fund') |
| 86 |
'Nottinghamfolk send $10 to the ditch fund.' |
| 87 |
|
| 88 |
The :meth:`~string.Template.substitute` method raises a :exc:`KeyError` when a |
| 89 |
placeholder is not supplied in a dictionary or a keyword argument. For |
| 90 |
mail-merge style applications, user supplied data may be incomplete and the |
| 91 |
:meth:`~string.Template.safe_substitute` method may be more appropriate --- |
| 92 |
it will leave placeholders unchanged if data is missing:: |
| 93 |
|
| 94 |
>>> t = Template('Return the $item to $owner.') |
| 95 |
>>> d = dict(item='unladen swallow') |
| 96 |
>>> t.substitute(d) |
| 97 |
Traceback (most recent call last): |
| 98 |
... |
| 99 |
KeyError: 'owner' |
| 100 |
>>> t.safe_substitute(d) |
| 101 |
'Return the unladen swallow to $owner.' |
| 102 |
|
| 103 |
Template subclasses can specify a custom delimiter. For example, a batch |
| 104 |
renaming utility for a photo browser may elect to use percent signs for |
| 105 |
placeholders such as the current date, image sequence number, or file format:: |
| 106 |
|
| 107 |
>>> import time, os.path |
| 108 |
>>> photofiles = ['img_1074.jpg', 'img_1076.jpg', 'img_1077.jpg'] |
| 109 |
>>> class BatchRename(Template): |
| 110 |
... delimiter = '%' |
| 111 |
... |
| 112 |
>>> fmt = input('Enter rename style (%d-date %n-seqnum %f-format): ') |
| 113 |
Enter rename style (%d-date %n-seqnum %f-format): Ashley_%n%f |
| 114 |
|
| 115 |
>>> t = BatchRename(fmt) |
| 116 |
>>> date = time.strftime('%d%b%y') |
| 117 |
>>> for i, filename in enumerate(photofiles): |
| 118 |
... base, ext = os.path.splitext(filename) |
| 119 |
... newname = t.substitute(d=date, n=i, f=ext) |
| 120 |
... print('{0} --> {1}'.format(filename, newname)) |
| 121 |
|
| 122 |
img_1074.jpg --> Ashley_0.jpg |
| 123 |
img_1076.jpg --> Ashley_1.jpg |
| 124 |
img_1077.jpg --> Ashley_2.jpg |
| 125 |
|
| 126 |
Another application for templating is separating program logic from the details |
| 127 |
of multiple output formats. This makes it possible to substitute custom |
| 128 |
templates for XML files, plain text reports, and HTML web reports. |
| 129 |
|
| 130 |
|
| 131 |
.. _tut-binary-formats: |
| 132 |
|
| 133 |
Working with binary data record layouts |
| 134 |
======================================= |
| 135 |
|
| 136 |
The :mod:`struct` module provides :func:`~struct.pack` and |
| 137 |
:func:`~struct.unpack` functions for working with variable length binary |
| 138 |
record formats. The following example shows |
| 139 |
how to loop through header information in a ZIP file without using the |
| 140 |
:mod:`zipfile` module. Pack codes ``"H"`` and ``"I"`` represent two and four |
| 141 |
byte unsigned numbers respectively. The ``"<"`` indicates that they are |
| 142 |
standard size and in little-endian byte order:: |
| 143 |
|
| 144 |
import struct |
| 145 |
|
| 146 |
with open('myfile.zip', 'rb') as f: |
| 147 |
data = f.read() |
| 148 |
|
| 149 |
start = 0 |
| 150 |
for i in range(3): # show the first 3 file headers |
| 151 |
start += 14 |
| 152 |
fields = struct.unpack('<IIIHH', data[start:start+16]) |
| 153 |
crc32, comp_size, uncomp_size, filenamesize, extra_size = fields |
| 154 |
|
| 155 |
start += 16 |
| 156 |
filename = data[start:start+filenamesize] |
| 157 |
start += filenamesize |
| 158 |
extra = data[start:start+extra_size] |
| 159 |
print(filename, hex(crc32), comp_size, uncomp_size) |
| 160 |
|
| 161 |
start += extra_size + comp_size # skip to the next header |
| 162 |
|
| 163 |
|
| 164 |
.. _tut-multi-threading: |
| 165 |
|
| 166 |
Multi-threading |
| 167 |
=============== |
| 168 |
|
| 169 |
Threading is a technique for decoupling tasks which are not sequentially |
| 170 |
dependent. Threads can be used to improve the responsiveness of applications |
| 171 |
that accept user input while other tasks run in the background. A related use |
| 172 |
case is running I/O in parallel with computations in another thread. |
| 173 |
|
| 174 |
The following code shows how the high level :mod:`threading` module can run |
| 175 |
tasks in background while the main program continues to run:: |
| 176 |
|
| 177 |
import threading, zipfile |
| 178 |
|
| 179 |
class AsyncZip(threading.Thread): |
| 180 |
def __init__(self, infile, outfile): |
| 181 |
super().__init__() |
| 182 |
self.infile = infile |
| 183 |
self.outfile = outfile |
| 184 |
|
| 185 |
def run(self): |
| 186 |
with zipfile.ZipFile(self.outfile, 'w', zipfile.ZIP_DEFLATED) as f: |
| 187 |
f.write(self.infile) |
| 188 |
print('Finished background zip of:', self.infile) |
| 189 |
|
| 190 |
background = AsyncZip('mydata.txt', 'myarchive.zip') |
| 191 |
background.start() |
| 192 |
print('The main program continues to run in foreground.') |
| 193 |
|
| 194 |
background.join() # Wait for the background task to finish |
| 195 |
print('Main program waited until background was done.') |
| 196 |
|
| 197 |
The principal challenge of multi-threaded applications is coordinating threads |
| 198 |
that share data or other resources. To that end, the threading module provides |
| 199 |
a number of synchronization primitives including locks, events, condition |
| 200 |
variables, and semaphores. |
| 201 |
|
| 202 |
While those tools are powerful, minor design errors can result in problems that |
| 203 |
are difficult to reproduce. So, the preferred approach to task coordination is |
| 204 |
to concentrate all access to a resource in a single thread and then use the |
| 205 |
:mod:`queue` module to feed that thread with requests from other threads. |
| 206 |
Applications using :class:`~queue.Queue` objects for inter-thread communication and |
| 207 |
coordination are easier to design, more readable, and more reliable. |
| 208 |
|
| 209 |
|
| 210 |
.. _tut-logging: |
| 211 |
|
| 212 |
Logging |
| 213 |
======= |
| 214 |
|
| 215 |
The :mod:`logging` module offers a full featured and flexible logging system. |
| 216 |
At its simplest, log messages are sent to a file or to ``sys.stderr``:: |
| 217 |
|
| 218 |
import logging |
| 219 |
logging.debug('Debugging information') |
| 220 |
logging.info('Informational message') |
| 221 |
logging.warning('Warning:config file %s not found', 'server.conf') |
| 222 |
logging.error('Error occurred') |
| 223 |
logging.critical('Critical error -- shutting down') |
| 224 |
|
| 225 |
This produces the following output: |
| 226 |
|
| 227 |
.. code-block:: none |
| 228 |
|
| 229 |
WARNING:root:Warning:config file server.conf not found |
| 230 |
ERROR:root:Error occurred |
| 231 |
CRITICAL:root:Critical error -- shutting down |
| 232 |
|
| 233 |
By default, informational and debugging messages are suppressed and the output |
| 234 |
is sent to standard error. Other output options include routing messages |
| 235 |
through email, datagrams, sockets, or to an HTTP Server. New filters can select |
| 236 |
different routing based on message priority: :const:`~logging.DEBUG`, |
| 237 |
:const:`~logging.INFO`, :const:`~logging.WARNING`, :const:`~logging.ERROR`, |
| 238 |
and :const:`~logging.CRITICAL`. |
| 239 |
|
| 240 |
The logging system can be configured directly from Python or can be loaded from |
| 241 |
a user editable configuration file for customized logging without altering the |
| 242 |
application. |
| 243 |
|
| 244 |
|
| 245 |
.. _tut-weak-references: |
| 246 |
|
| 247 |
Weak references |
| 248 |
=============== |
| 249 |
|
| 250 |
Python does automatic memory management (reference counting for most objects and |
| 251 |
:term:`garbage collection` to eliminate cycles). The memory is freed shortly |
| 252 |
after the last reference to it has been eliminated. |
| 253 |
|
| 254 |
This approach works fine for most applications but occasionally there is a need |
| 255 |
to track objects only as long as they are being used by something else. |
| 256 |
Unfortunately, just tracking them creates a reference that makes them permanent. |
| 257 |
The :mod:`weakref` module provides tools for tracking objects without creating a |
| 258 |
reference. When the object is no longer needed, it is automatically removed |
| 259 |
from a weakref table and a callback is triggered for weakref objects. Typical |
| 260 |
applications include caching objects that are expensive to create:: |
| 261 |
|
| 262 |
>>> import weakref, gc |
| 263 |
>>> class A: |
| 264 |
... def __init__(self, value): |
| 265 |
... self.value = value |
| 266 |
... def __repr__(self): |
| 267 |
... return str(self.value) |
| 268 |
... |
| 269 |
>>> a = A(10) # create a reference |
| 270 |
>>> d = weakref.WeakValueDictionary() |
| 271 |
>>> d['primary'] = a # does not create a reference |
| 272 |
>>> d['primary'] # fetch the object if it is still alive |
| 273 |
10 |
| 274 |
>>> del a # remove the one reference |
| 275 |
>>> gc.collect() # run garbage collection right away |
| 276 |
0 |
| 277 |
>>> d['primary'] # entry was automatically removed |
| 278 |
Traceback (most recent call last): |
| 279 |
File "<stdin>", line 1, in <module> |
| 280 |
d['primary'] # entry was automatically removed |
| 281 |
File "C:/python314/lib/weakref.py", line 46, in __getitem__ |
| 282 |
o = self.data[key]() |
| 283 |
KeyError: 'primary' |
| 284 |
|
| 285 |
|
| 286 |
.. _tut-list-tools: |
| 287 |
|
| 288 |
Tools for working with lists |
| 289 |
============================ |
| 290 |
|
| 291 |
Many data structure needs can be met with the built-in list type. However, |
| 292 |
sometimes there is a need for alternative implementations with different |
| 293 |
performance trade-offs. |
| 294 |
|
| 295 |
The :mod:`array` module provides an :class:`~array.array` object that is like |
| 296 |
a list that stores only homogeneous data and stores it more compactly. The |
| 297 |
following example shows an array of numbers stored as two byte unsigned binary |
| 298 |
numbers (typecode ``"H"``) rather than the usual 16 bytes per entry for regular |
| 299 |
lists of Python int objects:: |
| 300 |
|
| 301 |
>>> from array import array |
| 302 |
>>> a = array('H', [4000, 10, 700, 22222]) |
| 303 |
>>> sum(a) |
| 304 |
26932 |
| 305 |
>>> a[1:3] |
| 306 |
array('H', [10, 700]) |
| 307 |
|
| 308 |
The :mod:`collections` module provides a :class:`~collections.deque` object |
| 309 |
that is like a list with faster appends and pops from the left side but slower |
| 310 |
lookups in the middle. These objects are well suited for implementing queues |
| 311 |
and breadth first tree searches:: |
| 312 |
|
| 313 |
>>> from collections import deque |
| 314 |
>>> d = deque(["task1", "task2", "task3"]) |
| 315 |
>>> d.append("task4") |
| 316 |
>>> print("Handling", d.popleft()) |
| 317 |
Handling task1 |
| 318 |
|
| 319 |
:: |
| 320 |
|
| 321 |
unsearched = deque([starting_node]) |
| 322 |
def breadth_first_search(unsearched): |
| 323 |
node = unsearched.popleft() |
| 324 |
for m in gen_moves(node): |
| 325 |
if is_goal(m): |
| 326 |
return m |
| 327 |
unsearched.append(m) |
| 328 |
|
| 329 |
In addition to alternative list implementations, the library also offers other |
| 330 |
tools such as the :mod:`bisect` module with functions for manipulating sorted |
| 331 |
lists:: |
| 332 |
|
| 333 |
>>> import bisect |
| 334 |
>>> scores = [(100, 'perl'), (200, 'tcl'), (400, 'lua'), (500, 'python')] |
| 335 |
>>> bisect.insort(scores, (300, 'ruby')) |
| 336 |
>>> scores |
| 337 |
[(100, 'perl'), (200, 'tcl'), (300, 'ruby'), (400, 'lua'), (500, 'python')] |
| 338 |
|
| 339 |
The :mod:`heapq` module provides functions for implementing heaps based on |
| 340 |
regular lists. The lowest valued entry is always kept at position zero. This |
| 341 |
is useful for applications which repeatedly access the smallest element but do |
| 342 |
not want to run a full list sort:: |
| 343 |
|
| 344 |
>>> from heapq import heapify, heappop, heappush |
| 345 |
>>> data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0] |
| 346 |
>>> heapify(data) # rearrange the list into heap order |
| 347 |
>>> heappush(data, -5) # add a new entry |
| 348 |
>>> [heappop(data) for i in range(3)] # fetch the three smallest entries |
| 349 |
[-5, 0, 1] |
| 350 |
|
| 351 |
|
| 352 |
.. _tut-decimal-fp: |
| 353 |
|
| 354 |
Decimal floating-point arithmetic |
| 355 |
================================= |
| 356 |
|
| 357 |
The :mod:`decimal` module offers a :class:`~decimal.Decimal` datatype for |
| 358 |
decimal floating-point arithmetic. Compared to the built-in :class:`float` |
| 359 |
implementation of binary floating point, the class is especially helpful for |
| 360 |
|
| 361 |
* financial applications and other uses which require exact decimal |
| 362 |
representation, |
| 363 |
* control over precision, |
| 364 |
* control over rounding to meet legal or regulatory requirements, |
| 365 |
* tracking of significant decimal places, or |
| 366 |
* applications where the user expects the results to match calculations done by |
| 367 |
hand. |
| 368 |
|
| 369 |
For example, calculating a 5% tax on a 70 cent phone charge gives different |
| 370 |
results in decimal floating point and binary floating point. The difference |
| 371 |
becomes significant if the results are rounded to the nearest cent:: |
| 372 |
|
| 373 |
>>> from decimal import * |
| 374 |
>>> round(Decimal('0.70') * Decimal('1.05'), 2) |
| 375 |
Decimal('0.74') |
| 376 |
>>> round(.70 * 1.05, 2) |
| 377 |
0.73 |
| 378 |
|
| 379 |
The :class:`~decimal.Decimal` result keeps a trailing zero, automatically |
| 380 |
inferring four place significance from multiplicands with two place |
| 381 |
significance. Decimal reproduces mathematics as done by hand and avoids |
| 382 |
issues that can arise when binary floating point cannot exactly represent |
| 383 |
decimal quantities. |
| 384 |
|
| 385 |
Exact representation enables the :class:`~decimal.Decimal` class to perform |
| 386 |
modulo calculations and equality tests that are unsuitable for binary floating |
| 387 |
point:: |
| 388 |
|
| 389 |
>>> Decimal('1.00') % Decimal('.10') |
| 390 |
Decimal('0.00') |
| 391 |
>>> 1.00 % 0.10 |
| 392 |
0.09999999999999995 |
| 393 |
|
| 394 |
>>> sum([Decimal('0.1')]*10) == Decimal('1.0') |
| 395 |
True |
| 396 |
>>> 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 == 1.0 |
| 397 |
False |
| 398 |
|
| 399 |
The :mod:`decimal` module provides arithmetic with as much precision as needed:: |
| 400 |
|
| 401 |
>>> getcontext().prec = 36 |
| 402 |
>>> Decimal(1) / Decimal(7) |
| 403 |
Decimal('0.142857142857142857142857142857142857') |
| 404 |
|
| 405 |
|