Debian Codegraph @

Doc/howto/gdb_helpers.rst

raw
1 .. _gdb:
2
3 =========================================================
4 Debugging C API extensions and CPython Internals with GDB
5 =========================================================
6
7 .. highlight:: none
8
9 This document explains how the Python GDB extension, ``python-gdb.py``, can
10 be used with the GDB debugger to debug CPython extensions and the
11 CPython interpreter itself.
12
13 When debugging low-level problems such as crashes or deadlocks, a low-level
14 debugger, such as GDB, is useful to diagnose and correct the issue.
15 By default, GDB (or any of its front-ends) doesn't support high-level
16 information specific to the CPython interpreter.
17
18 The ``python-gdb.py`` extension adds CPython interpreter information to GDB.
19 The extension helps introspect the stack of currently executing Python functions.
20 Given a Python object represented by a :c:expr:`PyObject *` pointer,
21 the extension surfaces the type and value of the object.
22
23 Developers who are working on CPython extensions or tinkering with parts
24 of CPython that are written in C can use this document to learn how to use the
25 ``python-gdb.py`` extension with GDB.
26
27 .. note::
28
29 This document assumes that you are familiar with the basics of GDB and the
30 CPython C API. It consolidates guidance from the
31 `devguide <https://devguide.python.org>`_ and the
32 `Python wiki <https://wiki.python.org/moin/DebuggingWithGdb>`_.
33
34
35 Prerequisites
36 =============
37
38 You need to have:
39
40 - GDB 7 or later. (For earlier versions of GDB, see ``Misc/gdbinit`` in the
41 sources of Python 3.11 or earlier.)
42 - GDB-compatible debugging information for Python and any extension you are
43 debugging.
44 - The ``python-gdb.py`` extension.
45
46 The extension is built with Python, but might be distributed separately or
47 not at all. Below, we include tips for a few common systems as examples.
48 Note that even if the instructions match your system, they might be outdated.
49
50
51 Setup with Python built from source
52 -----------------------------------
53
54 When you build CPython from source, debugging information should be available,
55 and the build should add a ``python-gdb.py`` file to the root directory of
56 your repository.
57
58 To activate support, you must add the directory containing ``python-gdb.py``
59 to GDB's "auto-load-safe-path".
60 If you haven't done this, recent versions of GDB will print out a warning
61 with instructions on how to do this.
62
63 .. note::
64
65 If you do not see instructions for your version of GDB, put this in your
66 configuration file (``~/.gdbinit`` or ``~/.config/gdb/gdbinit``)::
67
68 add-auto-load-safe-path /path/to/cpython
69
70 You can also add multiple paths, separated by ``:``.
71
72
73 Setup for Python from a Linux distro
74 ------------------------------------
75
76 Most Linux systems provide debug information for the system Python
77 in a package called ``python-debuginfo``, ``python-dbg`` or similar.
78 For example:
79
80 - Fedora:
81
82 .. code-block:: shell
83
84 sudo dnf install gdb
85 sudo dnf debuginfo-install python3
86
87 - Ubuntu:
88
89 .. code-block:: shell
90
91 sudo apt install gdb python3-dbg
92
93 On several recent Linux systems, GDB can download debugging symbols
94 automatically using *debuginfod*.
95 However, this will not install the ``python-gdb.py`` extension;
96 you generally do need to install the debug info package separately.
97
98
99 Using the Debug build and Development mode
100 ==========================================
101
102 For easier debugging, you might want to:
103
104 - Use a :ref:`debug build <debug-build>` of Python. (When building from source,
105 use ``configure --with-pydebug``. On Linux distros, install and run a package
106 like ``python-debug`` or ``python-dbg``, if available.)
107 - Use the runtime :ref:`development mode <devmode>` (``-X dev``).
108
109 Both enable extra assertions and disable some optimizations.
110 Sometimes this hides the bug you are trying to find, but in most cases they
111 make the process easier.
112
113
114 Using the ``python-gdb`` extension
115 ==================================
116
117 When the extension is loaded, it provides two main features:
118 pretty printers for Python values, and additional commands.
119
120 Pretty-printers
121 ---------------
122
123 This is what a GDB backtrace looks like (truncated) when this extension is
124 enabled::
125
126 #0 0x000000000041a6b1 in PyObject_Malloc (nbytes=Cannot access memory at address 0x7fffff7fefe8
127 ) at Objects/obmalloc.c:748
128 #1 0x000000000041b7c0 in _PyObject_DebugMallocApi (id=111 'o', nbytes=24) at Objects/obmalloc.c:1445
129 #2 0x000000000041b717 in _PyObject_DebugMalloc (nbytes=24) at Objects/obmalloc.c:1412
130 #3 0x000000000044060a in _PyUnicode_New (length=11) at Objects/unicodeobject.c:346
131 #4 0x00000000004466aa in PyUnicodeUCS2_DecodeUTF8Stateful (s=0x5c2b8d "__lltrace__", size=11, errors=0x0, consumed=
132 0x0) at Objects/unicodeobject.c:2531
133 #5 0x0000000000446647 in PyUnicodeUCS2_DecodeUTF8 (s=0x5c2b8d "__lltrace__", size=11, errors=0x0)
134 at Objects/unicodeobject.c:2495
135 #6 0x0000000000440d1b in PyUnicodeUCS2_FromStringAndSize (u=0x5c2b8d "__lltrace__", size=11)
136 at Objects/unicodeobject.c:551
137 #7 0x0000000000440d94 in PyUnicodeUCS2_FromString (u=0x5c2b8d "__lltrace__") at Objects/unicodeobject.c:569
138 #8 0x0000000000584abd in PyDict_GetItemString (v=
139 {'Yuck': <type at remote 0xad4730>, '__builtins__': <module at remote 0x7ffff7fd5ee8>, '__file__': 'Lib/test/crashers/nasty_eq_vs_dict.py', '__package__': None, 'y': <Yuck(i=0) at remote 0xaacd80>, 'dict': {0: 0, 1: 1, 2: 2, 3: 3}, '__cached__': None, '__name__': '__main__', 'z': <Yuck(i=0) at remote 0xaace60>, '__doc__': None}, key=
140 0x5c2b8d "__lltrace__") at Objects/dictobject.c:2171
141
142 Notice how the dictionary argument to ``PyDict_GetItemString`` is displayed
143 as its ``repr()``, rather than an opaque ``PyObject *`` pointer.
144
145 The extension works by supplying a custom printing routine for values of type
146 ``PyObject *``. If you need to access lower-level details of an object, then
147 cast the value to a pointer of the appropriate type. For example::
148
149 (gdb) p globals
150 $1 = {'__builtins__': <module at remote 0x7ffff7fb1868>, '__name__':
151 '__main__', 'ctypes': <module at remote 0x7ffff7f14360>, '__doc__': None,
152 '__package__': None}
153
154 (gdb) p *(PyDictObject*)globals
155 $2 = {ob_refcnt = 3, ob_type = 0x3dbdf85820, ma_fill = 5, ma_used = 5,
156 ma_mask = 7, ma_table = 0x63d0f8, ma_lookup = 0x3dbdc7ea70
157 <lookdict_string>, ma_smalltable = {{me_hash = 7065186196740147912,
158 me_key = '__builtins__', me_value = <module at remote 0x7ffff7fb1868>},
159 {me_hash = -368181376027291943, me_key = '__name__',
160 me_value ='__main__'}, {me_hash = 0, me_key = 0x0, me_value = 0x0},
161 {me_hash = 0, me_key = 0x0, me_value = 0x0},
162 {me_hash = -9177857982131165996, me_key = 'ctypes',
163 me_value = <module at remote 0x7ffff7f14360>},
164 {me_hash = -8518757509529533123, me_key = '__doc__', me_value = None},
165 {me_hash = 0, me_key = 0x0, me_value = 0x0}, {
166 me_hash = 6614918939584953775, me_key = '__package__', me_value = None}}}
167
168 Note that the pretty-printers do not actually call ``repr()``.
169 For basic types, they try to match its result closely.
170
171 An area that can be confusing is that the custom printer for some types look a
172 lot like GDB's built-in printer for standard types. For example, the
173 pretty-printer for a Python ``int`` (:c:expr:`PyLongObject *`)
174 gives a representation that is not distinguishable from one of a
175 regular machine-level integer::
176
177 (gdb) p some_machine_integer
178 $3 = 42
179
180 (gdb) p some_python_integer
181 $4 = 42
182
183 The internal structure can be revealed with a cast to :c:expr:`PyLongObject *`::
184
185 (gdb) p *(PyLongObject*)some_python_integer
186 $5 = {ob_base = {ob_base = {ob_refcnt = 8, ob_type = 0x3dad39f5e0}, ob_size = 1},
187 ob_digit = {42}}
188
189 A similar confusion can arise with the ``str`` type, where the output looks a
190 lot like gdb's built-in printer for ``char *``::
191
192 (gdb) p ptr_to_python_str
193 $6 = '__builtins__'
194
195 The pretty-printer for ``str`` instances defaults to using single-quotes (as
196 does Python's ``repr`` for strings) whereas the standard printer for ``char *``
197 values uses double-quotes and contains a hexadecimal address::
198
199 (gdb) p ptr_to_char_star
200 $7 = 0x6d72c0 "hello world"
201
202 Again, the implementation details can be revealed with a cast to
203 :c:expr:`PyUnicodeObject *`::
204
205 (gdb) p *(PyUnicodeObject*)$6
206 $8 = {ob_base = {ob_refcnt = 33, ob_type = 0x3dad3a95a0}, length = 12,
207 str = 0x7ffff2128500, hash = 7065186196740147912, state = 1, defenc = 0x0}
208
209 ``py-list``
210 -----------
211
212 The extension adds a ``py-list`` command, which
213 lists the Python source code (if any) for the current frame in the selected
214 thread. The current line is marked with a ">"::
215
216 (gdb) py-list
217 901 if options.profile:
218 902 options.profile = False
219 903 profile_me()
220 904 return
221 905
222 >906 u = UI()
223 907 if not u.quit:
224 908 try:
225 909 gtk.main()
226 910 except KeyboardInterrupt:
227 911 # properly quit on a keyboard interrupt...
228
229 Use ``py-list START`` to list at a different line number within the Python
230 source, and ``py-list START,END`` to list a specific range of lines within
231 the Python source.
232
233 ``py-up`` and ``py-down``
234 -------------------------
235
236 The ``py-up`` and ``py-down`` commands are analogous to GDB's regular ``up``
237 and ``down`` commands, but try to move at the level of CPython frames, rather
238 than C frames.
239
240 GDB is not always able to read the relevant frame information, depending on
241 the optimization level with which CPython was compiled. Internally, the
242 commands look for C frames that are executing the default frame evaluation
243 function (that is, the core bytecode interpreter loop within CPython) and
244 look up the value of the related ``PyFrameObject *``.
245
246 They emit the frame number (at the C level) within the thread.
247
248 For example::
249
250 (gdb) py-up
251 #37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/
252 gnome_sudoku/main.py, line 906, in start_game ()
253 u = UI()
254 (gdb) py-up
255 #40 Frame 0x948e82c, for file /usr/lib/python2.6/site-packages/
256 gnome_sudoku/gnome_sudoku.py, line 22, in start_game(main=<module at remote 0xb771b7f4>)
257 main.start_game()
258 (gdb) py-up
259 Unable to find an older python frame
260
261 so we're at the top of the Python stack.
262
263 The frame numbers correspond to those displayed by GDB's standard
264 ``backtrace`` command.
265 The command skips C frames which are not executing Python code.
266
267 Going back down::
268
269 (gdb) py-down
270 #37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/gnome_sudoku/main.py, line 906, in start_game ()
271 u = UI()
272 (gdb) py-down
273 #34 (unable to read python frame information)
274 (gdb) py-down
275 #23 (unable to read python frame information)
276 (gdb) py-down
277 #19 (unable to read python frame information)
278 (gdb) py-down
279 #14 Frame 0x99262ac, for file /usr/lib/python2.6/site-packages/gnome_sudoku/game_selector.py, line 201, in run_swallowed_dialog (self=<NewOrSavedGameSelector(new_game_model=<gtk.ListStore at remote 0x98fab44>, puzzle=None, saved_games=[{'gsd.auto_fills': 0, 'tracking': {}, 'trackers': {}, 'notes': [], 'saved_at': 1270084485, 'game': '7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 0 0 0 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5\n7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 1 8 3 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5', 'gsd.impossible_hints': 0, 'timer.__absolute_start_time__': <float at remote 0x984b474>, 'gsd.hints': 0, 'timer.active_time': <float at remote 0x984b494>, 'timer.total_time': <float at remote 0x984b464>}], dialog=<gtk.Dialog at remote 0x98faaa4>, saved_game_model=<gtk.ListStore at remote 0x98fad24>, sudoku_maker=<SudokuMaker(terminated=False, played=[], batch_siz...(truncated)
280 swallower.run_dialog(self.dialog)
281 (gdb) py-down
282 #11 Frame 0x9aead74, for file /usr/lib/python2.6/site-packages/gnome_sudoku/dialog_swallower.py, line 48, in run_dialog (self=<SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>, main_page=0) at remote 0x98fa6e4>, d=<gtk.Dialog at remote 0x98faaa4>)
283 gtk.main()
284 (gdb) py-down
285 #8 (unable to read python frame information)
286 (gdb) py-down
287 Unable to find a newer python frame
288
289 and we're at the bottom of the Python stack.
290
291 Note that in Python 3.12 and newer, the same C stack frame can be used for
292 multiple Python stack frames. This means that ``py-up`` and ``py-down``
293 may move multiple Python frames at once. For example::
294
295 (gdb) py-up
296 #6 Frame 0x7ffff7fb62b0, for file /tmp/rec.py, line 5, in recursive_function (n=0)
297 time.sleep(5)
298 #6 Frame 0x7ffff7fb6240, for file /tmp/rec.py, line 7, in recursive_function (n=1)
299 recursive_function(n-1)
300 #6 Frame 0x7ffff7fb61d0, for file /tmp/rec.py, line 7, in recursive_function (n=2)
301 recursive_function(n-1)
302 #6 Frame 0x7ffff7fb6160, for file /tmp/rec.py, line 7, in recursive_function (n=3)
303 recursive_function(n-1)
304 #6 Frame 0x7ffff7fb60f0, for file /tmp/rec.py, line 7, in recursive_function (n=4)
305 recursive_function(n-1)
306 #6 Frame 0x7ffff7fb6080, for file /tmp/rec.py, line 7, in recursive_function (n=5)
307 recursive_function(n-1)
308 #6 Frame 0x7ffff7fb6020, for file /tmp/rec.py, line 9, in <module> ()
309 recursive_function(5)
310 (gdb) py-up
311 Unable to find an older python frame
312
313
314 ``py-bt``
315 ---------
316
317 The ``py-bt`` command attempts to display a Python-level backtrace of the
318 current thread.
319
320 For example::
321
322 (gdb) py-bt
323 #8 (unable to read python frame information)
324 #11 Frame 0x9aead74, for file /usr/lib/python2.6/site-packages/gnome_sudoku/dialog_swallower.py, line 48, in run_dialog (self=<SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>, main_page=0) at remote 0x98fa6e4>, d=<gtk.Dialog at remote 0x98faaa4>)
325 gtk.main()
326 #14 Frame 0x99262ac, for file /usr/lib/python2.6/site-packages/gnome_sudoku/game_selector.py, line 201, in run_swallowed_dialog (self=<NewOrSavedGameSelector(new_game_model=<gtk.ListStore at remote 0x98fab44>, puzzle=None, saved_games=[{'gsd.auto_fills': 0, 'tracking': {}, 'trackers': {}, 'notes': [], 'saved_at': 1270084485, 'game': '7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 0 0 0 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5\n7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 1 8 3 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5', 'gsd.impossible_hints': 0, 'timer.__absolute_start_time__': <float at remote 0x984b474>, 'gsd.hints': 0, 'timer.active_time': <float at remote 0x984b494>, 'timer.total_time': <float at remote 0x984b464>}], dialog=<gtk.Dialog at remote 0x98faaa4>, saved_game_model=<gtk.ListStore at remote 0x98fad24>, sudoku_maker=<SudokuMaker(terminated=False, played=[], batch_siz...(truncated)
327 swallower.run_dialog(self.dialog)
328 #19 (unable to read python frame information)
329 #23 (unable to read python frame information)
330 #34 (unable to read python frame information)
331 #37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/gnome_sudoku/main.py, line 906, in start_game ()
332 u = UI()
333 #40 Frame 0x948e82c, for file /usr/lib/python2.6/site-packages/gnome_sudoku/gnome_sudoku.py, line 22, in start_game (main=<module at remote 0xb771b7f4>)
334 main.start_game()
335
336 The frame numbers correspond to those displayed by GDB's standard
337 ``backtrace`` command.
338
339 ``py-print``
340 ------------
341
342 The ``py-print`` command looks up a Python name and tries to print it.
343 It looks in locals within the current thread, then globals, then finally
344 builtins::
345
346 (gdb) py-print self
347 local 'self' = <SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>,
348 main_page=0) at remote 0x98fa6e4>
349 (gdb) py-print __name__
350 global '__name__' = 'gnome_sudoku.dialog_swallower'
351 (gdb) py-print len
352 builtin 'len' = <built-in function len>
353 (gdb) py-print scarlet_pimpernel
354 'scarlet_pimpernel' not found
355
356 If the current C frame corresponds to multiple Python frames, ``py-print``
357 only considers the first one.
358
359 ``py-locals``
360 -------------
361
362 The ``py-locals`` command looks up all Python locals within the current
363 Python frame in the selected thread, and prints their representations::
364
365 (gdb) py-locals
366 self = <SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>,
367 main_page=0) at remote 0x98fa6e4>
368 d = <gtk.Dialog at remote 0x98faaa4>
369
370 If the current C frame corresponds to multiple Python frames, locals from
371 all of them will be shown::
372
373 (gdb) py-locals
374 Locals for recursive_function
375 n = 0
376 Locals for recursive_function
377 n = 1
378 Locals for recursive_function
379 n = 2
380 Locals for recursive_function
381 n = 3
382 Locals for recursive_function
383 n = 4
384 Locals for recursive_function
385 n = 5
386 Locals for <module>
387
388
389 Use with GDB commands
390 =====================
391
392 The extension commands complement GDB's built-in commands.
393 For example, you can use a frame numbers shown by ``py-bt`` with the ``frame``
394 command to go a specific frame within the selected thread, like this::
395
396 (gdb) py-bt
397 (output snipped)
398 #68 Frame 0xaa4560, for file Lib/test/regrtest.py, line 1548, in <module> ()
399 main()
400 (gdb) frame 68
401 #68 0x00000000004cd1e6 in PyEval_EvalFrameEx (f=Frame 0xaa4560, for file Lib/test/regrtest.py, line 1548, in <module> (), throwflag=0) at Python/ceval.c:2665
402 2665 x = call_function(&sp, oparg);
403 (gdb) py-list
404 1543 # Run the tests in a context manager that temporary changes the CWD to a
405 1544 # temporary and writable directory. If it's not possible to create or
406 1545 # change the CWD, the original CWD will be used. The original CWD is
407 1546 # available from test_support.SAVEDCWD.
408 1547 with test_support.temp_cwd(TESTCWD, quiet=True):
409 >1548 main()
410
411 The ``info threads`` command will give you a list of the threads within the
412 process, and you can use the ``thread`` command to select a different one::
413
414 (gdb) info threads
415 105 Thread 0x7fffefa18710 (LWP 10260) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
416 104 Thread 0x7fffdf5fe710 (LWP 10259) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
417 * 1 Thread 0x7ffff7fe2700 (LWP 10145) 0x00000038e46d73e3 in select () at ../sysdeps/unix/syscall-template.S:82
418
419 You can use ``thread apply all COMMAND`` or (``t a a COMMAND`` for short) to run
420 a command on all threads. With ``py-bt``, this lets you see what every
421 thread is doing at the Python level::
422
423 (gdb) t a a py-bt
424
425 Thread 105 (Thread 0x7fffefa18710 (LWP 10260)):
426 #5 Frame 0x7fffd00019d0, for file /home/david/coding/python-svn/Lib/threading.py, line 155, in _acquire_restore (self=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, count_owner=(1, 140737213728528), count=1, owner=140737213728528)
427 self.__block.acquire()
428 #8 Frame 0x7fffac001640, for file /home/david/coding/python-svn/Lib/threading.py, line 269, in wait (self=<_Condition(_Condition__lock=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, acquire=<instancemethod at remote 0xd80260>, _is_owned=<instancemethod at remote 0xd80160>, _release_save=<instancemethod at remote 0xd803e0>, release=<instancemethod at remote 0xd802e0>, _acquire_restore=<instancemethod at remote 0xd7ee60>, _Verbose__verbose=False, _Condition__waiters=[]) at remote 0xd7fd10>, timeout=None, waiter=<thread.lock at remote 0x858a90>, saved_state=(1, 140737213728528))
429 self._acquire_restore(saved_state)
430 #12 Frame 0x7fffb8001a10, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 348, in f ()
431 cond.wait()
432 #16 Frame 0x7fffb8001c40, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 37, in task (tid=140737213728528)
433 f()
434
435 Thread 104 (Thread 0x7fffdf5fe710 (LWP 10259)):
436 #5 Frame 0x7fffe4001580, for file /home/david/coding/python-svn/Lib/threading.py, line 155, in _acquire_restore (self=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, count_owner=(1, 140736940992272), count=1, owner=140736940992272)
437 self.__block.acquire()
438 #8 Frame 0x7fffc8002090, for file /home/david/coding/python-svn/Lib/threading.py, line 269, in wait (self=<_Condition(_Condition__lock=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, acquire=<instancemethod at remote 0xd80260>, _is_owned=<instancemethod at remote 0xd80160>, _release_save=<instancemethod at remote 0xd803e0>, release=<instancemethod at remote 0xd802e0>, _acquire_restore=<instancemethod at remote 0xd7ee60>, _Verbose__verbose=False, _Condition__waiters=[]) at remote 0xd7fd10>, timeout=None, waiter=<thread.lock at remote 0x858860>, saved_state=(1, 140736940992272))
439 self._acquire_restore(saved_state)
440 #12 Frame 0x7fffac001c90, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 348, in f ()
441 cond.wait()
442 #16 Frame 0x7fffac0011c0, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 37, in task (tid=140736940992272)
443 f()
444
445 Thread 1 (Thread 0x7ffff7fe2700 (LWP 10145)):
446 #5 Frame 0xcb5380, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 16, in _wait ()
447 time.sleep(0.01)
448 #8 Frame 0x7fffd00024a0, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 378, in _check_notify (self=<ConditionTests(_testMethodName='test_notify', _resultForDoCleanups=<TestResult(_original_stdout=<cStringIO.StringO at remote 0xc191e0>, skipped=[], _mirrorOutput=False, testsRun=39, buffer=False, _original_stderr=<file at remote 0x7ffff7fc6340>, _stdout_buffer=<cStringIO.StringO at remote 0xc9c7f8>, _stderr_buffer=<cStringIO.StringO at remote 0xc9c790>, _moduleSetUpFailed=False, expectedFailures=[], errors=[], _previousTestClass=<type at remote 0x928310>, unexpectedSuccesses=[], failures=[], shouldStop=False, failfast=False) at remote 0xc185a0>, _threads=(0,), _cleanups=[], _type_equality_funcs={<type at remote 0x7eba00>: <instancemethod at remote 0xd750e0>, <type at remote 0x7e7820>: <instancemethod at remote 0xd75160>, <type at remote 0x7e30e0>: <instancemethod at remote 0xd75060>, <type at remote 0x7e7d20>: <instancemethod at remote 0xd751e0>, <type at remote 0x7f19e0...(truncated)
449 _wait()