Debian Codegraph @

Doc/tutorial/stdlib2.rst

raw
1 .. _tut-brieftourtwo:
2
3 **********************************************
4 Brief tour of the standard library --- part II
5 **********************************************
6
7 This second tour covers more advanced modules that support professional
8 programming needs. These modules rarely occur in small scripts.
9
10
11 .. _tut-output-formatting:
12
13 Output formatting
14 =================
15
16 The :mod:`reprlib` module provides a version of :func:`repr` customized for
17 abbreviated displays of large or deeply nested containers::
18
19 >>> import reprlib
20 >>> reprlib.repr(set('supercalifragilisticexpialidocious'))
21 "{'a', 'c', 'd', 'e', 'f', 'g', ...}"
22
23 The :mod:`pprint` module offers more sophisticated control over printing both
24 built-in and user defined objects in a way that is readable by the interpreter.
25 When the result is longer than one line, the "pretty printer" adds line breaks
26 and indentation to more clearly reveal data structure::
27
28 >>> import pprint
29 >>> t = [[[['black', 'cyan'], 'white', ['green', 'red']], [['magenta',
30 ... 'yellow'], 'blue']]]
31 ...
32 >>> pprint.pprint(t, width=30)
33 [[[['black', 'cyan'],
34 'white',
35 ['green', 'red']],
36 [['magenta', 'yellow'],
37 'blue']]]
38
39 The :mod:`textwrap` module formats paragraphs of text to fit a given screen
40 width::
41
42 >>> import textwrap
43 >>> doc = """The wrap() method is just like fill() except that it returns
44 ... a list of strings instead of one big string with newlines to separate
45 ... the wrapped lines."""
46 ...
47 >>> print(textwrap.fill(doc, width=40))
48 The wrap() method is just like fill()
49 except that it returns a list of strings
50 instead of one big string with newlines
51 to separate the wrapped lines.
52
53 The :mod:`locale` module accesses a database of culture specific data formats.
54 The grouping attribute of locale's format function provides a direct way of
55 formatting numbers with group separators::
56
57 >>> import locale
58 >>> locale.setlocale(locale.LC_ALL, 'English_United States.1252')
59 'English_United States.1252'
60 >>> conv = locale.localeconv() # get a mapping of conventions
61 >>> x = 1234567.8
62 >>> locale.format_string("%d", x, grouping=True)
63 '1,234,567'
64 >>> locale.format_string("%s%.*f", (conv['currency_symbol'],
65 ... conv['frac_digits'], x), grouping=True)
66 '$1,234,567.80'
67
68
69 .. _tut-templating:
70
71 Templating
72 ==========
73
74 The :mod:`string` module includes a versatile :class:`~string.Template` class
75 with a simplified syntax suitable for editing by end-users. This allows users
76 to customize their applications without having to alter the application.
77
78 The format uses placeholder names formed by ``$`` with valid Python identifiers
79 (alphanumeric characters and underscores). Surrounding the placeholder with
80 braces allows it to be followed by more alphanumeric letters with no intervening
81 spaces. Writing ``$$`` creates a single escaped ``$``::
82
83 >>> from string import Template
84 >>> t = Template('${village}folk send $$10 to $cause.')
85 >>> t.substitute(village='Nottingham', cause='the ditch fund')
86 'Nottinghamfolk send $10 to the ditch fund.'
87
88 The :meth:`~string.Template.substitute` method raises a :exc:`KeyError` when a
89 placeholder is not supplied in a dictionary or a keyword argument. For
90 mail-merge style applications, user supplied data may be incomplete and the
91 :meth:`~string.Template.safe_substitute` method may be more appropriate ---
92 it will leave placeholders unchanged if data is missing::
93
94 >>> t = Template('Return the $item to $owner.')
95 >>> d = dict(item='unladen swallow')
96 >>> t.substitute(d)
97 Traceback (most recent call last):
98 ...
99 KeyError: 'owner'
100 >>> t.safe_substitute(d)
101 'Return the unladen swallow to $owner.'
102
103 Template subclasses can specify a custom delimiter. For example, a batch
104 renaming utility for a photo browser may elect to use percent signs for
105 placeholders such as the current date, image sequence number, or file format::
106
107 >>> import time, os.path
108 >>> photofiles = ['img_1074.jpg', 'img_1076.jpg', 'img_1077.jpg']
109 >>> class BatchRename(Template):
110 ... delimiter = '%'
111 ...
112 >>> fmt = input('Enter rename style (%d-date %n-seqnum %f-format): ')
113 Enter rename style (%d-date %n-seqnum %f-format): Ashley_%n%f
114
115 >>> t = BatchRename(fmt)
116 >>> date = time.strftime('%d%b%y')
117 >>> for i, filename in enumerate(photofiles):
118 ... base, ext = os.path.splitext(filename)
119 ... newname = t.substitute(d=date, n=i, f=ext)
120 ... print('{0} --> {1}'.format(filename, newname))
121
122 img_1074.jpg --> Ashley_0.jpg
123 img_1076.jpg --> Ashley_1.jpg
124 img_1077.jpg --> Ashley_2.jpg
125
126 Another application for templating is separating program logic from the details
127 of multiple output formats. This makes it possible to substitute custom
128 templates for XML files, plain text reports, and HTML web reports.
129
130
131 .. _tut-binary-formats:
132
133 Working with binary data record layouts
134 =======================================
135
136 The :mod:`struct` module provides :func:`~struct.pack` and
137 :func:`~struct.unpack` functions for working with variable length binary
138 record formats. The following example shows
139 how to loop through header information in a ZIP file without using the
140 :mod:`zipfile` module. Pack codes ``"H"`` and ``"I"`` represent two and four
141 byte unsigned numbers respectively. The ``"<"`` indicates that they are
142 standard size and in little-endian byte order::
143
144 import struct
145
146 with open('myfile.zip', 'rb') as f:
147 data = f.read()
148
149 start = 0
150 for i in range(3): # show the first 3 file headers
151 start += 14
152 fields = struct.unpack('<IIIHH', data[start:start+16])
153 crc32, comp_size, uncomp_size, filenamesize, extra_size = fields
154
155 start += 16
156 filename = data[start:start+filenamesize]
157 start += filenamesize
158 extra = data[start:start+extra_size]
159 print(filename, hex(crc32), comp_size, uncomp_size)
160
161 start += extra_size + comp_size # skip to the next header
162
163
164 .. _tut-multi-threading:
165
166 Multi-threading
167 ===============
168
169 Threading is a technique for decoupling tasks which are not sequentially
170 dependent. Threads can be used to improve the responsiveness of applications
171 that accept user input while other tasks run in the background. A related use
172 case is running I/O in parallel with computations in another thread.
173
174 The following code shows how the high level :mod:`threading` module can run
175 tasks in background while the main program continues to run::
176
177 import threading, zipfile
178
179 class AsyncZip(threading.Thread):
180 def __init__(self, infile, outfile):
181 super().__init__()
182 self.infile = infile
183 self.outfile = outfile
184
185 def run(self):
186 with zipfile.ZipFile(self.outfile, 'w', zipfile.ZIP_DEFLATED) as f:
187 f.write(self.infile)
188 print('Finished background zip of:', self.infile)
189
190 background = AsyncZip('mydata.txt', 'myarchive.zip')
191 background.start()
192 print('The main program continues to run in foreground.')
193
194 background.join() # Wait for the background task to finish
195 print('Main program waited until background was done.')
196
197 The principal challenge of multi-threaded applications is coordinating threads
198 that share data or other resources. To that end, the threading module provides
199 a number of synchronization primitives including locks, events, condition
200 variables, and semaphores.
201
202 While those tools are powerful, minor design errors can result in problems that
203 are difficult to reproduce. So, the preferred approach to task coordination is
204 to concentrate all access to a resource in a single thread and then use the
205 :mod:`queue` module to feed that thread with requests from other threads.
206 Applications using :class:`~queue.Queue` objects for inter-thread communication and
207 coordination are easier to design, more readable, and more reliable.
208
209
210 .. _tut-logging:
211
212 Logging
213 =======
214
215 The :mod:`logging` module offers a full featured and flexible logging system.
216 At its simplest, log messages are sent to a file or to ``sys.stderr``::
217
218 import logging
219 logging.debug('Debugging information')
220 logging.info('Informational message')
221 logging.warning('Warning:config file %s not found', 'server.conf')
222 logging.error('Error occurred')
223 logging.critical('Critical error -- shutting down')
224
225 This produces the following output:
226
227 .. code-block:: none
228
229 WARNING:root:Warning:config file server.conf not found
230 ERROR:root:Error occurred
231 CRITICAL:root:Critical error -- shutting down
232
233 By default, informational and debugging messages are suppressed and the output
234 is sent to standard error. Other output options include routing messages
235 through email, datagrams, sockets, or to an HTTP Server. New filters can select
236 different routing based on message priority: :const:`~logging.DEBUG`,
237 :const:`~logging.INFO`, :const:`~logging.WARNING`, :const:`~logging.ERROR`,
238 and :const:`~logging.CRITICAL`.
239
240 The logging system can be configured directly from Python or can be loaded from
241 a user editable configuration file for customized logging without altering the
242 application.
243
244
245 .. _tut-weak-references:
246
247 Weak references
248 ===============
249
250 Python does automatic memory management (reference counting for most objects and
251 :term:`garbage collection` to eliminate cycles). The memory is freed shortly
252 after the last reference to it has been eliminated.
253
254 This approach works fine for most applications but occasionally there is a need
255 to track objects only as long as they are being used by something else.
256 Unfortunately, just tracking them creates a reference that makes them permanent.
257 The :mod:`weakref` module provides tools for tracking objects without creating a
258 reference. When the object is no longer needed, it is automatically removed
259 from a weakref table and a callback is triggered for weakref objects. Typical
260 applications include caching objects that are expensive to create::
261
262 >>> import weakref, gc
263 >>> class A:
264 ... def __init__(self, value):
265 ... self.value = value
266 ... def __repr__(self):
267 ... return str(self.value)
268 ...
269 >>> a = A(10) # create a reference
270 >>> d = weakref.WeakValueDictionary()
271 >>> d['primary'] = a # does not create a reference
272 >>> d['primary'] # fetch the object if it is still alive
273 10
274 >>> del a # remove the one reference
275 >>> gc.collect() # run garbage collection right away
276 0
277 >>> d['primary'] # entry was automatically removed
278 Traceback (most recent call last):
279 File "<stdin>", line 1, in <module>
280 d['primary'] # entry was automatically removed
281 File "C:/python314/lib/weakref.py", line 46, in __getitem__
282 o = self.data[key]()
283 KeyError: 'primary'
284
285
286 .. _tut-list-tools:
287
288 Tools for working with lists
289 ============================
290
291 Many data structure needs can be met with the built-in list type. However,
292 sometimes there is a need for alternative implementations with different
293 performance trade-offs.
294
295 The :mod:`array` module provides an :class:`~array.array` object that is like
296 a list that stores only homogeneous data and stores it more compactly. The
297 following example shows an array of numbers stored as two byte unsigned binary
298 numbers (typecode ``"H"``) rather than the usual 16 bytes per entry for regular
299 lists of Python int objects::
300
301 >>> from array import array
302 >>> a = array('H', [4000, 10, 700, 22222])
303 >>> sum(a)
304 26932
305 >>> a[1:3]
306 array('H', [10, 700])
307
308 The :mod:`collections` module provides a :class:`~collections.deque` object
309 that is like a list with faster appends and pops from the left side but slower
310 lookups in the middle. These objects are well suited for implementing queues
311 and breadth first tree searches::
312
313 >>> from collections import deque
314 >>> d = deque(["task1", "task2", "task3"])
315 >>> d.append("task4")
316 >>> print("Handling", d.popleft())
317 Handling task1
318
319 ::
320
321 unsearched = deque([starting_node])
322 def breadth_first_search(unsearched):
323 node = unsearched.popleft()
324 for m in gen_moves(node):
325 if is_goal(m):
326 return m
327 unsearched.append(m)
328
329 In addition to alternative list implementations, the library also offers other
330 tools such as the :mod:`bisect` module with functions for manipulating sorted
331 lists::
332
333 >>> import bisect
334 >>> scores = [(100, 'perl'), (200, 'tcl'), (400, 'lua'), (500, 'python')]
335 >>> bisect.insort(scores, (300, 'ruby'))
336 >>> scores
337 [(100, 'perl'), (200, 'tcl'), (300, 'ruby'), (400, 'lua'), (500, 'python')]
338
339 The :mod:`heapq` module provides functions for implementing heaps based on
340 regular lists. The lowest valued entry is always kept at position zero. This
341 is useful for applications which repeatedly access the smallest element but do
342 not want to run a full list sort::
343
344 >>> from heapq import heapify, heappop, heappush
345 >>> data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0]
346 >>> heapify(data) # rearrange the list into heap order
347 >>> heappush(data, -5) # add a new entry
348 >>> [heappop(data) for i in range(3)] # fetch the three smallest entries
349 [-5, 0, 1]
350
351
352 .. _tut-decimal-fp:
353
354 Decimal floating-point arithmetic
355 =================================
356
357 The :mod:`decimal` module offers a :class:`~decimal.Decimal` datatype for
358 decimal floating-point arithmetic. Compared to the built-in :class:`float`
359 implementation of binary floating point, the class is especially helpful for
360
361 * financial applications and other uses which require exact decimal
362 representation,
363 * control over precision,
364 * control over rounding to meet legal or regulatory requirements,
365 * tracking of significant decimal places, or
366 * applications where the user expects the results to match calculations done by
367 hand.
368
369 For example, calculating a 5% tax on a 70 cent phone charge gives different
370 results in decimal floating point and binary floating point. The difference
371 becomes significant if the results are rounded to the nearest cent::
372
373 >>> from decimal import *
374 >>> round(Decimal('0.70') * Decimal('1.05'), 2)
375 Decimal('0.74')
376 >>> round(.70 * 1.05, 2)
377 0.73
378
379 The :class:`~decimal.Decimal` result keeps a trailing zero, automatically
380 inferring four place significance from multiplicands with two place
381 significance. Decimal reproduces mathematics as done by hand and avoids
382 issues that can arise when binary floating point cannot exactly represent
383 decimal quantities.
384
385 Exact representation enables the :class:`~decimal.Decimal` class to perform
386 modulo calculations and equality tests that are unsuitable for binary floating
387 point::
388
389 >>> Decimal('1.00') % Decimal('.10')
390 Decimal('0.00')
391 >>> 1.00 % 0.10
392 0.09999999999999995
393
394 >>> sum([Decimal('0.1')]*10) == Decimal('1.0')
395 True
396 >>> 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 == 1.0
397 False
398
399 The :mod:`decimal` module provides arithmetic with as much precision as needed::
400
401 >>> getcontext().prec = 36
402 >>> Decimal(1) / Decimal(7)
403 Decimal('0.142857142857142857142857142857142857')
404
405