| 1 |
:tocdepth: 2 |
| 2 |
|
| 3 |
========================= |
| 4 |
Library and Extension FAQ |
| 5 |
========================= |
| 6 |
|
| 7 |
.. only:: html |
| 8 |
|
| 9 |
.. contents:: |
| 10 |
|
| 11 |
General Library Questions |
| 12 |
========================= |
| 13 |
|
| 14 |
How do I find a module or application to perform task X? |
| 15 |
-------------------------------------------------------- |
| 16 |
|
| 17 |
Check :ref:`the Library Reference <library-index>` to see if there's a relevant |
| 18 |
standard library module. (Eventually you'll learn what's in the standard |
| 19 |
library and will be able to skip this step.) |
| 20 |
|
| 21 |
For third-party packages, search the `Python Package Index |
| 22 |
<https://pypi.org>`_ or try `Google <https://www.google.com>`_ or |
| 23 |
another web search engine. Searching for "Python" plus a keyword or two for |
| 24 |
your topic of interest will usually find something helpful. |
| 25 |
|
| 26 |
|
| 27 |
Where is the math.py (socket.py, regex.py, etc.) source file? |
| 28 |
------------------------------------------------------------- |
| 29 |
|
| 30 |
If you can't find a source file for a module it may be a built-in or |
| 31 |
dynamically loaded module implemented in C, C++ or other compiled language. |
| 32 |
In this case you may not have the source file or it may be something like |
| 33 |
:file:`mathmodule.c`, somewhere in a C source directory (not on the Python Path). |
| 34 |
|
| 35 |
There are (at least) three kinds of modules in Python: |
| 36 |
|
| 37 |
1) modules written in Python (.py); |
| 38 |
2) modules written in C and dynamically loaded (.dll, .pyd, .so, .sl, etc); |
| 39 |
3) modules written in C and linked with the interpreter; to get a list of these, |
| 40 |
type:: |
| 41 |
|
| 42 |
import sys |
| 43 |
print(sys.builtin_module_names) |
| 44 |
|
| 45 |
|
| 46 |
How do I make a Python script executable on Unix? |
| 47 |
------------------------------------------------- |
| 48 |
|
| 49 |
You need to do two things: the script file's mode must be executable and the |
| 50 |
first line must begin with ``#!`` followed by the path of the Python |
| 51 |
interpreter. |
| 52 |
|
| 53 |
The first is done by executing ``chmod +x scriptfile`` or perhaps ``chmod 755 |
| 54 |
scriptfile``. |
| 55 |
|
| 56 |
The second can be done in a number of ways. The most straightforward way is to |
| 57 |
write :: |
| 58 |
|
| 59 |
#!/usr/local/bin/python |
| 60 |
|
| 61 |
as the very first line of your file, using the pathname for where the Python |
| 62 |
interpreter is installed on your platform. |
| 63 |
|
| 64 |
If you would like the script to be independent of where the Python interpreter |
| 65 |
lives, you can use the :program:`env` program. Almost all Unix variants support |
| 66 |
the following, assuming the Python interpreter is in a directory on the user's |
| 67 |
:envvar:`PATH`:: |
| 68 |
|
| 69 |
#!/usr/bin/env python |
| 70 |
|
| 71 |
*Don't* do this for CGI scripts. The :envvar:`PATH` variable for CGI scripts is |
| 72 |
often very minimal, so you need to use the actual absolute pathname of the |
| 73 |
interpreter. |
| 74 |
|
| 75 |
Occasionally, a user's environment is so full that the :program:`/usr/bin/env` |
| 76 |
program fails; or there's no env program at all. In that case, you can try the |
| 77 |
following hack (due to Alex Rezinsky): |
| 78 |
|
| 79 |
.. code-block:: sh |
| 80 |
|
| 81 |
#! /bin/sh |
| 82 |
""":" |
| 83 |
exec python $0 ${1+"$@"} |
| 84 |
""" |
| 85 |
|
| 86 |
The minor disadvantage is that this defines the script's __doc__ string. |
| 87 |
However, you can fix that by adding :: |
| 88 |
|
| 89 |
__doc__ = """...Whatever...""" |
| 90 |
|
| 91 |
|
| 92 |
|
| 93 |
Is there a curses/termcap package for Python? |
| 94 |
--------------------------------------------- |
| 95 |
|
| 96 |
.. XXX curses *is* built by default, isn't it? |
| 97 |
|
| 98 |
For Unix variants: The standard Python source distribution comes with a curses |
| 99 |
module in the :source:`Modules` subdirectory, though it's not compiled by default. |
| 100 |
(Note that this is not available in the Windows distribution -- there is no |
| 101 |
curses module for Windows.) |
| 102 |
|
| 103 |
The :mod:`curses` module supports basic curses features as well as many additional |
| 104 |
functions from ncurses and SYSV curses such as colour, alternative character set |
| 105 |
support, pads, and mouse support. This means the module isn't compatible with |
| 106 |
operating systems that only have BSD curses, but there don't seem to be any |
| 107 |
currently maintained OSes that fall into this category. |
| 108 |
|
| 109 |
|
| 110 |
Is there an equivalent to C's onexit() in Python? |
| 111 |
------------------------------------------------- |
| 112 |
|
| 113 |
The :mod:`atexit` module provides a register function that is similar to C's |
| 114 |
:c:func:`!onexit`. |
| 115 |
|
| 116 |
|
| 117 |
Why don't my signal handlers work? |
| 118 |
---------------------------------- |
| 119 |
|
| 120 |
The most common problem is that the signal handler is declared with the wrong |
| 121 |
argument list. It is called as :: |
| 122 |
|
| 123 |
handler(signum, frame) |
| 124 |
|
| 125 |
so it should be declared with two parameters:: |
| 126 |
|
| 127 |
def handler(signum, frame): |
| 128 |
... |
| 129 |
|
| 130 |
|
| 131 |
Common tasks |
| 132 |
============ |
| 133 |
|
| 134 |
How do I test a Python program or component? |
| 135 |
-------------------------------------------- |
| 136 |
|
| 137 |
Python comes with two testing frameworks. The :mod:`doctest` module finds |
| 138 |
examples in the docstrings for a module and runs them, comparing the output with |
| 139 |
the expected output given in the docstring. |
| 140 |
|
| 141 |
The :mod:`unittest` module is a fancier testing framework modelled on Java and |
| 142 |
Smalltalk testing frameworks. |
| 143 |
|
| 144 |
To make testing easier, you should use good modular design in your program. |
| 145 |
Your program should have almost all functionality |
| 146 |
encapsulated in either functions or class methods -- and this sometimes has the |
| 147 |
surprising and delightful effect of making the program run faster (because local |
| 148 |
variable accesses are faster than global accesses). Furthermore the program |
| 149 |
should avoid depending on mutating global variables, since this makes testing |
| 150 |
much more difficult to do. |
| 151 |
|
| 152 |
The "global main logic" of your program may be as simple as :: |
| 153 |
|
| 154 |
if __name__ == "__main__": |
| 155 |
main_logic() |
| 156 |
|
| 157 |
at the bottom of the main module of your program. |
| 158 |
|
| 159 |
Once your program is organized as a tractable collection of function and class |
| 160 |
behaviours, you should write test functions that exercise the behaviours. A |
| 161 |
test suite that automates a sequence of tests can be associated with each module. |
| 162 |
This sounds like a lot of work, but since Python is so terse and flexible it's |
| 163 |
surprisingly easy. You can make coding much more pleasant and fun by writing |
| 164 |
your test functions in parallel with the "production code", since this makes it |
| 165 |
easy to find bugs and even design flaws earlier. |
| 166 |
|
| 167 |
"Support modules" that are not intended to be the main module of a program may |
| 168 |
include a self-test of the module. :: |
| 169 |
|
| 170 |
if __name__ == "__main__": |
| 171 |
self_test() |
| 172 |
|
| 173 |
Even programs that interact with complex external interfaces may be tested when |
| 174 |
the external interfaces are unavailable by using "fake" interfaces implemented |
| 175 |
in Python. |
| 176 |
|
| 177 |
|
| 178 |
How do I create documentation from doc strings? |
| 179 |
----------------------------------------------- |
| 180 |
|
| 181 |
The :mod:`pydoc` module can create HTML from the doc strings in your Python |
| 182 |
source code. An alternative for creating API documentation purely from |
| 183 |
docstrings is `epydoc <https://epydoc.sourceforge.net/>`_. `Sphinx |
| 184 |
<https://www.sphinx-doc.org>`_ can also include docstring content. |
| 185 |
|
| 186 |
|
| 187 |
How do I get a single keypress at a time? |
| 188 |
----------------------------------------- |
| 189 |
|
| 190 |
For Unix variants there are several solutions. It's straightforward to do this |
| 191 |
using curses, but curses is a fairly large module to learn. |
| 192 |
|
| 193 |
.. XXX this doesn't work out of the box, some IO expert needs to check why |
| 194 |
|
| 195 |
Here's a solution without curses:: |
| 196 |
|
| 197 |
import termios, fcntl, sys, os |
| 198 |
fd = sys.stdin.fileno() |
| 199 |
|
| 200 |
oldterm = termios.tcgetattr(fd) |
| 201 |
newattr = termios.tcgetattr(fd) |
| 202 |
newattr[3] = newattr[3] & ~termios.ICANON & ~termios.ECHO |
| 203 |
termios.tcsetattr(fd, termios.TCSANOW, newattr) |
| 204 |
|
| 205 |
oldflags = fcntl.fcntl(fd, fcntl.F_GETFL) |
| 206 |
fcntl.fcntl(fd, fcntl.F_SETFL, oldflags | os.O_NONBLOCK) |
| 207 |
|
| 208 |
try: |
| 209 |
while True: |
| 210 |
try: |
| 211 |
c = sys.stdin.read(1) |
| 212 |
print("Got character", repr(c)) |
| 213 |
except OSError: |
| 214 |
pass |
| 215 |
finally: |
| 216 |
termios.tcsetattr(fd, termios.TCSAFLUSH, oldterm) |
| 217 |
fcntl.fcntl(fd, fcntl.F_SETFL, oldflags) |
| 218 |
|
| 219 |
You need the :mod:`termios` and the :mod:`fcntl` module for any of this to |
| 220 |
work, and I've only tried it on Linux, though it should work elsewhere. In |
| 221 |
this code, characters are read and printed one at a time. |
| 222 |
|
| 223 |
:func:`termios.tcsetattr` turns off stdin's echoing and disables canonical |
| 224 |
mode. :func:`fcntl.fnctl` is used to obtain stdin's file descriptor flags |
| 225 |
and modify them for non-blocking mode. Since reading stdin when it is empty |
| 226 |
results in an :exc:`OSError`, this error is caught and ignored. |
| 227 |
|
| 228 |
.. versionchanged:: 3.3 |
| 229 |
*sys.stdin.read* used to raise :exc:`IOError`. Starting from Python 3.3 |
| 230 |
:exc:`IOError` is alias for :exc:`OSError`. |
| 231 |
|
| 232 |
|
| 233 |
Threads |
| 234 |
======= |
| 235 |
|
| 236 |
How do I program using threads? |
| 237 |
------------------------------- |
| 238 |
|
| 239 |
Be sure to use the :mod:`threading` module and not the :mod:`_thread` module. |
| 240 |
The :mod:`threading` module builds convenient abstractions on top of the |
| 241 |
low-level primitives provided by the :mod:`_thread` module. |
| 242 |
|
| 243 |
|
| 244 |
None of my threads seem to run: why? |
| 245 |
------------------------------------ |
| 246 |
|
| 247 |
As soon as the main thread exits, all threads are killed. Your main thread is |
| 248 |
running too quickly, giving the threads no time to do any work. |
| 249 |
|
| 250 |
A simple fix is to add a sleep to the end of the program that's long enough for |
| 251 |
all the threads to finish:: |
| 252 |
|
| 253 |
import threading, time |
| 254 |
|
| 255 |
def thread_task(name, n): |
| 256 |
for i in range(n): |
| 257 |
print(name, i) |
| 258 |
|
| 259 |
for i in range(10): |
| 260 |
T = threading.Thread(target=thread_task, args=(str(i), i)) |
| 261 |
T.start() |
| 262 |
|
| 263 |
time.sleep(10) # <---------------------------! |
| 264 |
|
| 265 |
But now (on many platforms) the threads don't run in parallel, but appear to run |
| 266 |
sequentially, one at a time! The reason is that the OS thread scheduler doesn't |
| 267 |
start a new thread until the previous thread is blocked. |
| 268 |
|
| 269 |
A simple fix is to add a tiny sleep to the start of the run function:: |
| 270 |
|
| 271 |
def thread_task(name, n): |
| 272 |
time.sleep(0.001) # <--------------------! |
| 273 |
for i in range(n): |
| 274 |
print(name, i) |
| 275 |
|
| 276 |
for i in range(10): |
| 277 |
T = threading.Thread(target=thread_task, args=(str(i), i)) |
| 278 |
T.start() |
| 279 |
|
| 280 |
time.sleep(10) |
| 281 |
|
| 282 |
Instead of trying to guess a good delay value for :func:`time.sleep`, |
| 283 |
it's better to use some kind of semaphore mechanism. One idea is to use the |
| 284 |
:mod:`queue` module to create a queue object, let each thread append a token to |
| 285 |
the queue when it finishes, and let the main thread read as many tokens from the |
| 286 |
queue as there are threads. |
| 287 |
|
| 288 |
|
| 289 |
How do I parcel out work among a bunch of worker threads? |
| 290 |
--------------------------------------------------------- |
| 291 |
|
| 292 |
The easiest way is to use the :mod:`concurrent.futures` module, |
| 293 |
especially the :mod:`~concurrent.futures.ThreadPoolExecutor` class. |
| 294 |
|
| 295 |
Or, if you want fine control over the dispatching algorithm, you can write |
| 296 |
your own logic manually. Use the :mod:`queue` module to create a queue |
| 297 |
containing a list of jobs. The :class:`~queue.Queue` class maintains a |
| 298 |
list of objects and has a ``.put(obj)`` method that adds items to the queue and |
| 299 |
a ``.get()`` method to return them. The class will take care of the locking |
| 300 |
necessary to ensure that each job is handed out exactly once. |
| 301 |
|
| 302 |
Here's a trivial example:: |
| 303 |
|
| 304 |
import threading, queue, time |
| 305 |
|
| 306 |
# The worker thread gets jobs off the queue. When the queue is empty, it |
| 307 |
# assumes there will be no more work and exits. |
| 308 |
# (Realistically workers will run until terminated.) |
| 309 |
def worker(): |
| 310 |
print('Running worker') |
| 311 |
time.sleep(0.1) |
| 312 |
while True: |
| 313 |
try: |
| 314 |
arg = q.get(block=False) |
| 315 |
except queue.Empty: |
| 316 |
print('Worker', threading.current_thread(), end=' ') |
| 317 |
print('queue empty') |
| 318 |
break |
| 319 |
else: |
| 320 |
print('Worker', threading.current_thread(), end=' ') |
| 321 |
print('running with argument', arg) |
| 322 |
time.sleep(0.5) |
| 323 |
|
| 324 |
# Create queue |
| 325 |
q = queue.Queue() |
| 326 |
|
| 327 |
# Start a pool of 5 workers |
| 328 |
for i in range(5): |
| 329 |
t = threading.Thread(target=worker, name='worker %i' % (i+1)) |
| 330 |
t.start() |
| 331 |
|
| 332 |
# Begin adding work to the queue |
| 333 |
for i in range(50): |
| 334 |
q.put(i) |
| 335 |
|
| 336 |
# Give threads time to run |
| 337 |
print('Main thread sleeping') |
| 338 |
time.sleep(5) |
| 339 |
|
| 340 |
When run, this will produce the following output: |
| 341 |
|
| 342 |
.. code-block:: none |
| 343 |
|
| 344 |
Running worker |
| 345 |
Running worker |
| 346 |
Running worker |
| 347 |
Running worker |
| 348 |
Running worker |
| 349 |
Main thread sleeping |
| 350 |
Worker <Thread(worker 1, started 130283832797456)> running with argument 0 |
| 351 |
Worker <Thread(worker 2, started 130283824404752)> running with argument 1 |
| 352 |
Worker <Thread(worker 3, started 130283816012048)> running with argument 2 |
| 353 |
Worker <Thread(worker 4, started 130283807619344)> running with argument 3 |
| 354 |
Worker <Thread(worker 5, started 130283799226640)> running with argument 4 |
| 355 |
Worker <Thread(worker 1, started 130283832797456)> running with argument 5 |
| 356 |
... |
| 357 |
|
| 358 |
Consult the module's documentation for more details; the :class:`~queue.Queue` |
| 359 |
class provides a featureful interface. |
| 360 |
|
| 361 |
|
| 362 |
What kinds of global value mutation are thread-safe? |
| 363 |
---------------------------------------------------- |
| 364 |
|
| 365 |
A :term:`global interpreter lock` (GIL) is used internally to ensure that only one |
| 366 |
thread runs in the Python VM at a time. In general, Python offers to switch |
| 367 |
among threads only between bytecode instructions; how frequently it switches can |
| 368 |
be set via :func:`sys.setswitchinterval`. Each bytecode instruction and |
| 369 |
therefore all the C implementation code reached from each instruction is |
| 370 |
therefore atomic from the point of view of a Python program. |
| 371 |
|
| 372 |
In theory, this means an exact accounting requires an exact understanding of the |
| 373 |
PVM bytecode implementation. In practice, it means that operations on shared |
| 374 |
variables of built-in data types (ints, lists, dicts, etc) that "look atomic" |
| 375 |
really are. |
| 376 |
|
| 377 |
For example, the following operations are all atomic (L, L1, L2 are lists, D, |
| 378 |
D1, D2 are dicts, x, y are objects, i, j are ints):: |
| 379 |
|
| 380 |
L.append(x) |
| 381 |
L1.extend(L2) |
| 382 |
x = L[i] |
| 383 |
x = L.pop() |
| 384 |
L1[i:j] = L2 |
| 385 |
L.sort() |
| 386 |
x = y |
| 387 |
x.field = y |
| 388 |
D[x] = y |
| 389 |
D1.update(D2) |
| 390 |
D.keys() |
| 391 |
|
| 392 |
These aren't:: |
| 393 |
|
| 394 |
i = i+1 |
| 395 |
L.append(L[-1]) |
| 396 |
L[i] = L[j] |
| 397 |
D[x] = D[x] + 1 |
| 398 |
|
| 399 |
Operations that replace other objects may invoke those other objects' |
| 400 |
:meth:`~object.__del__` method when their reference count reaches zero, and that can |
| 401 |
affect things. This is especially true for the mass updates to dictionaries and |
| 402 |
lists. When in doubt, use a mutex! |
| 403 |
|
| 404 |
|
| 405 |
Can't we get rid of the Global Interpreter Lock? |
| 406 |
------------------------------------------------ |
| 407 |
|
| 408 |
The :term:`global interpreter lock` (GIL) is often seen as a hindrance to Python's |
| 409 |
deployment on high-end multiprocessor server machines, because a multi-threaded |
| 410 |
Python program effectively only uses one CPU, due to the insistence that |
| 411 |
(almost) all Python code can only run while the GIL is held. |
| 412 |
|
| 413 |
With the approval of :pep:`703` work is now underway to remove the GIL from the |
| 414 |
CPython implementation of Python. Initially it will be implemented as an |
| 415 |
optional compiler flag when building the interpreter, and so separate |
| 416 |
builds will be available with and without the GIL. Long-term, the hope is |
| 417 |
to settle on a single build, once the performance implications of removing the |
| 418 |
GIL are fully understood. Python 3.13 is likely to be the first release |
| 419 |
containing this work, although it may not be completely functional in this |
| 420 |
release. |
| 421 |
|
| 422 |
The current work to remove the GIL is based on a |
| 423 |
`fork of Python 3.9 with the GIL removed <https://github.com/colesbury/nogil>`_ |
| 424 |
by Sam Gross. |
| 425 |
Prior to that, |
| 426 |
in the days of Python 1.5, Greg Stein actually implemented a comprehensive |
| 427 |
patch set (the "free threading" patches) that removed the GIL and replaced it |
| 428 |
with fine-grained locking. Adam Olsen did a similar experiment |
| 429 |
in his `python-safethread <https://code.google.com/archive/p/python-safethread>`_ |
| 430 |
project. Unfortunately, both of these earlier experiments exhibited a sharp |
| 431 |
drop in single-thread |
| 432 |
performance (at least 30% slower), due to the amount of fine-grained locking |
| 433 |
necessary to compensate for the removal of the GIL. The Python 3.9 fork |
| 434 |
is the first attempt at removing the GIL with an acceptable performance |
| 435 |
impact. |
| 436 |
|
| 437 |
The presence of the GIL in current Python releases |
| 438 |
doesn't mean that you can't make good use of Python on multi-CPU machines! |
| 439 |
You just have to be creative with dividing the work up between multiple |
| 440 |
*processes* rather than multiple *threads*. The |
| 441 |
:class:`~concurrent.futures.ProcessPoolExecutor` class in the new |
| 442 |
:mod:`concurrent.futures` module provides an easy way of doing so; the |
| 443 |
:mod:`multiprocessing` module provides a lower-level API in case you want |
| 444 |
more control over dispatching of tasks. |
| 445 |
|
| 446 |
Judicious use of C extensions will also help; if you use a C extension to |
| 447 |
perform a time-consuming task, the extension can release the GIL while the |
| 448 |
thread of execution is in the C code and allow other threads to get some work |
| 449 |
done. Some standard library modules such as :mod:`zlib` and :mod:`hashlib` |
| 450 |
already do this. |
| 451 |
|
| 452 |
An alternative approach to reducing the impact of the GIL is |
| 453 |
to make the GIL a per-interpreter-state lock rather than truly global. |
| 454 |
This was :ref:`first implemented in Python 3.12 <whatsnew312-pep684>` and is |
| 455 |
available in the C API. A Python interface to it is expected in Python 3.13. |
| 456 |
The main limitation to it at the moment is likely to be 3rd party extension |
| 457 |
modules, since these must be written with multiple interpreters in mind in |
| 458 |
order to be usable, so many older extension modules will not be usable. |
| 459 |
|
| 460 |
|
| 461 |
Input and Output |
| 462 |
================ |
| 463 |
|
| 464 |
How do I delete a file? (And other file questions...) |
| 465 |
----------------------------------------------------- |
| 466 |
|
| 467 |
Use ``os.remove(filename)`` or ``os.unlink(filename)``; for documentation, see |
| 468 |
the :mod:`os` module. The two functions are identical; :func:`~os.unlink` is simply |
| 469 |
the name of the Unix system call for this function. |
| 470 |
|
| 471 |
To remove a directory, use :func:`os.rmdir`; use :func:`os.mkdir` to create one. |
| 472 |
``os.makedirs(path)`` will create any intermediate directories in ``path`` that |
| 473 |
don't exist. ``os.removedirs(path)`` will remove intermediate directories as |
| 474 |
long as they're empty; if you want to delete an entire directory tree and its |
| 475 |
contents, use :func:`shutil.rmtree`. |
| 476 |
|
| 477 |
To rename a file, use ``os.rename(old_path, new_path)``. |
| 478 |
|
| 479 |
To truncate a file, open it using ``f = open(filename, "rb+")``, and use |
| 480 |
``f.truncate(offset)``; offset defaults to the current seek position. There's |
| 481 |
also ``os.ftruncate(fd, offset)`` for files opened with :func:`os.open`, where |
| 482 |
*fd* is the file descriptor (a small integer). |
| 483 |
|
| 484 |
The :mod:`shutil` module also contains a number of functions to work on files |
| 485 |
including :func:`~shutil.copyfile`, :func:`~shutil.copytree`, and |
| 486 |
:func:`~shutil.rmtree`. |
| 487 |
|
| 488 |
|
| 489 |
How do I copy a file? |
| 490 |
--------------------- |
| 491 |
|
| 492 |
The :mod:`shutil` module contains a :func:`~shutil.copyfile` function. |
| 493 |
Note that on Windows NTFS volumes, it does not copy |
| 494 |
`alternate data streams |
| 495 |
<https://en.wikipedia.org/wiki/NTFS#Alternate_data_stream_(ADS)>`_ |
| 496 |
nor `resource forks <https://en.wikipedia.org/wiki/Resource_fork>`__ |
| 497 |
on macOS HFS+ volumes, though both are now rarely used. |
| 498 |
It also doesn't copy file permissions and metadata, though using |
| 499 |
:func:`shutil.copy2` instead will preserve most (though not all) of it. |
| 500 |
|
| 501 |
|
| 502 |
How do I read (or write) binary data? |
| 503 |
------------------------------------- |
| 504 |
|
| 505 |
To read or write complex binary data formats, it's best to use the :mod:`struct` |
| 506 |
module. It allows you to take a string containing binary data (usually numbers) |
| 507 |
and convert it to Python objects; and vice versa. |
| 508 |
|
| 509 |
For example, the following code reads two 2-byte integers and one 4-byte integer |
| 510 |
in big-endian format from a file:: |
| 511 |
|
| 512 |
import struct |
| 513 |
|
| 514 |
with open(filename, "rb") as f: |
| 515 |
s = f.read(8) |
| 516 |
x, y, z = struct.unpack(">hhl", s) |
| 517 |
|
| 518 |
The '>' in the format string forces big-endian data; the letter 'h' reads one |
| 519 |
"short integer" (2 bytes), and 'l' reads one "long integer" (4 bytes) from the |
| 520 |
string. |
| 521 |
|
| 522 |
For data that is more regular (e.g. a homogeneous list of ints or floats), |
| 523 |
you can also use the :mod:`array` module. |
| 524 |
|
| 525 |
.. note:: |
| 526 |
|
| 527 |
To read and write binary data, it is mandatory to open the file in |
| 528 |
binary mode (here, passing ``"rb"`` to :func:`open`). If you use |
| 529 |
``"r"`` instead (the default), the file will be open in text mode |
| 530 |
and ``f.read()`` will return :class:`str` objects rather than |
| 531 |
:class:`bytes` objects. |
| 532 |
|
| 533 |
|
| 534 |
I can't seem to use os.read() on a pipe created with os.popen(); why? |
| 535 |
--------------------------------------------------------------------- |
| 536 |
|
| 537 |
:func:`os.read` is a low-level function which takes a file descriptor, a small |
| 538 |
integer representing the opened file. :func:`os.popen` creates a high-level |
| 539 |
file object, the same type returned by the built-in :func:`open` function. |
| 540 |
Thus, to read *n* bytes from a pipe *p* created with :func:`os.popen`, you need to |
| 541 |
use ``p.read(n)``. |
| 542 |
|
| 543 |
|
| 544 |
How do I access the serial (RS232) port? |
| 545 |
---------------------------------------- |
| 546 |
|
| 547 |
For Win32, OSX, Linux, BSD, Jython, IronPython: |
| 548 |
|
| 549 |
:pypi:`pyserial` |
| 550 |
|
| 551 |
For Unix, see a Usenet post by Mitch Chapman: |
| 552 |
|
| 553 |
https://groups.google.com/[email protected] |
| 554 |
|
| 555 |
|
| 556 |
Why doesn't closing sys.stdout (stdin, stderr) really close it? |
| 557 |
--------------------------------------------------------------- |
| 558 |
|
| 559 |
Python :term:`file objects <file object>` are a high-level layer of |
| 560 |
abstraction on low-level C file descriptors. |
| 561 |
|
| 562 |
For most file objects you create in Python via the built-in :func:`open` |
| 563 |
function, ``f.close()`` marks the Python file object as being closed from |
| 564 |
Python's point of view, and also arranges to close the underlying C file |
| 565 |
descriptor. This also happens automatically in ``f``'s destructor, when |
| 566 |
``f`` becomes garbage. |
| 567 |
|
| 568 |
But stdin, stdout and stderr are treated specially by Python, because of the |
| 569 |
special status also given to them by C. Running ``sys.stdout.close()`` marks |
| 570 |
the Python-level file object as being closed, but does *not* close the |
| 571 |
associated C file descriptor. |
| 572 |
|
| 573 |
To close the underlying C file descriptor for one of these three, you should |
| 574 |
first be sure that's what you really want to do (e.g., you may confuse |
| 575 |
extension modules trying to do I/O). If it is, use :func:`os.close`:: |
| 576 |
|
| 577 |
os.close(stdin.fileno()) |
| 578 |
os.close(stdout.fileno()) |
| 579 |
os.close(stderr.fileno()) |
| 580 |
|
| 581 |
Or you can use the numeric constants 0, 1 and 2, respectively. |
| 582 |
|
| 583 |
|
| 584 |
Network/Internet Programming |
| 585 |
============================ |
| 586 |
|
| 587 |
What WWW tools are there for Python? |
| 588 |
------------------------------------ |
| 589 |
|
| 590 |
See the chapters titled :ref:`internet` and :ref:`netdata` in the Library |
| 591 |
Reference Manual. Python has many modules that will help you build server-side |
| 592 |
and client-side web systems. |
| 593 |
|
| 594 |
.. XXX check if wiki page is still up to date |
| 595 |
|
| 596 |
A summary of available frameworks is maintained by Paul Boddie at |
| 597 |
https://wiki.python.org/moin/WebProgramming\ . |
| 598 |
|
| 599 |
|
| 600 |
What module should I use to help with generating HTML? |
| 601 |
------------------------------------------------------ |
| 602 |
|
| 603 |
.. XXX add modern template languages |
| 604 |
|
| 605 |
You can find a collection of useful links on the `Web Programming wiki page |
| 606 |
<https://wiki.python.org/moin/WebProgramming>`_. |
| 607 |
|
| 608 |
|
| 609 |
How do I send mail from a Python script? |
| 610 |
---------------------------------------- |
| 611 |
|
| 612 |
Use the standard library module :mod:`smtplib`. |
| 613 |
|
| 614 |
Here's a very simple interactive mail sender that uses it. This method will |
| 615 |
work on any host that supports an SMTP listener. :: |
| 616 |
|
| 617 |
import sys, smtplib |
| 618 |
|
| 619 |
fromaddr = input("From: ") |
| 620 |
toaddrs = input("To: ").split(',') |
| 621 |
print("Enter message, end with ^D:") |
| 622 |
msg = '' |
| 623 |
while True: |
| 624 |
line = sys.stdin.readline() |
| 625 |
if not line: |
| 626 |
break |
| 627 |
msg += line |
| 628 |
|
| 629 |
# The actual mail send |
| 630 |
server = smtplib.SMTP('localhost') |
| 631 |
server.sendmail(fromaddr, toaddrs, msg) |
| 632 |
server.quit() |
| 633 |
|
| 634 |
A Unix-only alternative uses sendmail. The location of the sendmail program |
| 635 |
varies between systems; sometimes it is ``/usr/lib/sendmail``, sometimes |
| 636 |
``/usr/sbin/sendmail``. The sendmail manual page will help you out. Here's |
| 637 |
some sample code:: |
| 638 |
|
| 639 |
import os |
| 640 |
|
| 641 |
SENDMAIL = "/usr/sbin/sendmail" # sendmail location |
| 642 |
p = os.popen("%s -t -i" % SENDMAIL, "w") |
| 643 |
p.write("To: [email protected]\n") |
| 644 |
p.write("Subject: test\n") |
| 645 |
p.write("\n") # blank line separating headers from body |
| 646 |
p.write("Some text\n") |
| 647 |
p.write("some more text\n") |
| 648 |
sts = p.close() |
| 649 |
if sts != 0: |
| 650 |
print("Sendmail exit status", sts) |
| 651 |
|
| 652 |
|
| 653 |
How do I avoid blocking in the connect() method of a socket? |
| 654 |
------------------------------------------------------------ |
| 655 |
|
| 656 |
The :mod:`select` module is commonly used to help with asynchronous I/O on |
| 657 |
sockets. |
| 658 |
|
| 659 |
To prevent the TCP connect from blocking, you can set the socket to non-blocking |
| 660 |
mode. Then when you do the :meth:`~socket.socket.connect`, |
| 661 |
you will either connect immediately |
| 662 |
(unlikely) or get an exception that contains the error number as ``.errno``. |
| 663 |
``errno.EINPROGRESS`` indicates that the connection is in progress, but hasn't |
| 664 |
finished yet. Different OSes will return different values, so you're going to |
| 665 |
have to check what's returned on your system. |
| 666 |
|
| 667 |
You can use the :meth:`~socket.socket.connect_ex` method |
| 668 |
to avoid creating an exception. |
| 669 |
It will just return the errno value. |
| 670 |
To poll, you can call :meth:`~socket.socket.connect_ex` again later |
| 671 |
-- ``0`` or ``errno.EISCONN`` indicate that you're connected -- or you can pass this |
| 672 |
socket to :meth:`select.select` to check if it's writable. |
| 673 |
|
| 674 |
.. note:: |
| 675 |
The :mod:`asyncio` module provides a general purpose single-threaded and |
| 676 |
concurrent asynchronous library, which can be used for writing non-blocking |
| 677 |
network code. |
| 678 |
The third-party `Twisted <https://twisted.org/>`_ library is |
| 679 |
a popular and feature-rich alternative. |
| 680 |
|
| 681 |
|
| 682 |
Databases |
| 683 |
========= |
| 684 |
|
| 685 |
Are there any interfaces to database packages in Python? |
| 686 |
-------------------------------------------------------- |
| 687 |
|
| 688 |
Yes. |
| 689 |
|
| 690 |
Interfaces to disk-based hashes such as :mod:`DBM <dbm.ndbm>` and :mod:`GDBM |
| 691 |
<dbm.gnu>` are also included with standard Python. There is also the |
| 692 |
:mod:`sqlite3` module, which provides a lightweight disk-based relational |
| 693 |
database. |
| 694 |
|
| 695 |
Support for most relational databases is available. See the |
| 696 |
`DatabaseProgramming wiki page |
| 697 |
<https://wiki.python.org/moin/DatabaseProgramming>`_ for details. |
| 698 |
|
| 699 |
|
| 700 |
How do you implement persistent objects in Python? |
| 701 |
-------------------------------------------------- |
| 702 |
|
| 703 |
The :mod:`pickle` library module solves this in a very general way (though you |
| 704 |
still can't store things like open files, sockets or windows), and the |
| 705 |
:mod:`shelve` library module uses pickle and (g)dbm to create persistent |
| 706 |
mappings containing arbitrary Python objects. |
| 707 |
|
| 708 |
|
| 709 |
Mathematics and Numerics |
| 710 |
======================== |
| 711 |
|
| 712 |
How do I generate random numbers in Python? |
| 713 |
------------------------------------------- |
| 714 |
|
| 715 |
The standard module :mod:`random` implements a random number generator. Usage |
| 716 |
is simple:: |
| 717 |
|
| 718 |
import random |
| 719 |
random.random() |
| 720 |
|
| 721 |
This returns a random floating-point number in the range [0, 1). |
| 722 |
|
| 723 |
There are also many other specialized generators in this module, such as: |
| 724 |
|
| 725 |
* ``randrange(a, b)`` chooses an integer in the range [a, b). |
| 726 |
* ``uniform(a, b)`` chooses a floating-point number in the range [a, b). |
| 727 |
* ``normalvariate(mean, sdev)`` samples the normal (Gaussian) distribution. |
| 728 |
|
| 729 |
Some higher-level functions operate on sequences directly, such as: |
| 730 |
|
| 731 |
* ``choice(S)`` chooses a random element from a given sequence. |
| 732 |
* ``shuffle(L)`` shuffles a list in-place, i.e. permutes it randomly. |
| 733 |
|
| 734 |
There's also a ``Random`` class you can instantiate to create independent |
| 735 |
multiple random number generators. |