Debian Codegraph @

Misc/README.valgrind

raw
1 This document describes some caveats about the use of Valgrind with
2 Python. Valgrind is used periodically by Python developers to try
3 to ensure there are no memory leaks or invalid memory reads/writes.
4
5 If you want to enable valgrind support in Python, you will need to
6 configure Python --with-valgrind option or an older option
7 --without-pymalloc.
8
9 UPDATE: Python 3.6 now supports PYTHONMALLOC=malloc environment variable which
10 can be used to force the usage of the malloc() allocator of the C library.
11
12 If you don't want to read about the details of using Valgrind, there
13 are still two things you must do to suppress the warnings. First,
14 you must use a suppressions file. One is supplied in
15 Misc/valgrind-python.supp. Second, you must uncomment the lines in
16 Misc/valgrind-python.supp that suppress the warnings for PyObject_Free and
17 PyObject_Realloc.
18
19 If you want to use Valgrind more effectively and catch even more
20 memory leaks, you will need to configure python --without-pymalloc.
21 PyMalloc allocates a few blocks in big chunks and most object
22 allocations don't call malloc, they use chunks doled about by PyMalloc
23 from the big blocks. This means Valgrind can't detect
24 many allocations (and frees), except for those that are forwarded
25 to the system malloc. Note: configuring python --without-pymalloc
26 makes Python run much slower, especially when running under Valgrind.
27 You may need to run the tests in batches under Valgrind to keep
28 the memory usage down to allow the tests to complete. It seems to take
29 about 5 times longer to run --without-pymalloc.
30
31 Apr 15, 2006:
32 test_ctypes causes Valgrind 3.1.1 to fail (crash).
33 test_socket_ssl should be skipped when running valgrind.
34 The reason is that it purposely uses uninitialized memory.
35 This causes many spurious warnings, so it's easier to just skip it.
36
37
38 Details:
39 --------
40 Python uses its own small-object allocation scheme on top of malloc,
41 called PyMalloc.
42
43 Valgrind may show some unexpected results when PyMalloc is used.
44 Starting with Python 2.3, PyMalloc is used by default. You can disable
45 PyMalloc when configuring python by adding the --without-pymalloc option.
46 If you disable PyMalloc, most of the information in this document and
47 the supplied suppressions file will not be useful. As discussed above,
48 disabling PyMalloc can catch more problems.
49
50 PyMalloc uses 256KB chunks of memory, so it can't detect anything
51 wrong within these blocks. For that reason, compiling Python
52 --without-pymalloc usually increases the usefulness of other tools.
53
54 If you use valgrind on a default build of Python, you will see
55 many errors like:
56
57 ==6399== Use of uninitialised value of size 4
58 ==6399== at 0x4A9BDE7E: PyObject_Free (obmalloc.c:711)
59 ==6399== by 0x4A9B8198: dictresize (dictobject.c:477)
60
61 These are expected and not a problem. Tim Peters explains
62 the situation:
63
64 PyMalloc needs to know whether an arbitrary address is one
65 that's managed by it, or is managed by the system malloc.
66 The current scheme allows this to be determined in constant
67 time, regardless of how many memory areas are under pymalloc's
68 control.
69
70 The memory pymalloc manages itself is in one or more "arenas",
71 each a large contiguous memory area obtained from malloc.
72 The base address of each arena is saved by pymalloc
73 in a vector. Each arena is carved into "pools", and a field at
74 the start of each pool contains the index of that pool's arena's
75 base address in that vector.
76
77 Given an arbitrary address, pymalloc computes the pool base
78 address corresponding to it, then looks at "the index" stored
79 near there. If the index read up is out of bounds for the
80 vector of arena base addresses pymalloc maintains, then
81 pymalloc knows for certain that this address is not under
82 pymalloc's control. Otherwise the index is in bounds, and
83 pymalloc compares
84
85 the arena base address stored at that index in the vector
86
87 to
88
89 the arbitrary address pymalloc is investigating
90
91 pymalloc controls this arbitrary address if and only if it lies
92 in the arena the address's pool's index claims it lies in.
93
94 It doesn't matter whether the memory pymalloc reads up ("the
95 index") is initialized. If it's not initialized, then
96 whatever trash gets read up will lead pymalloc to conclude
97 (correctly) that the address isn't controlled by it, either
98 because the index is out of bounds, or the index is in bounds
99 but the arena it represents doesn't contain the address.
100
101 This determination has to be made on every call to one of
102 pymalloc's free/realloc entry points, so its speed is critical
103 (Python allocates and frees dynamic memory at a ferocious rate
104 -- everything in Python, from integers to "stack frames",
105 lives in the heap).