| 1 |
# Debugging Garbage Collector Related Problems |
| 2 |
|
| 3 |
This page contains some hints on debugging issues specific to the |
| 4 |
Boehm-Demers-Weiser conservative garbage collector. It applies both |
| 5 |
to debugging issues in client code that manifest themselves as collector |
| 6 |
misbehavior, and to debugging the collector itself. |
| 7 |
|
| 8 |
If you suspect a bug in the collector itself, it is strongly recommended that |
| 9 |
you try the latest collector release before proceeding. |
| 10 |
|
| 11 |
## Bus Errors and Segmentation Violations |
| 12 |
|
| 13 |
If the fault occurred in `GC_find_limit`, or with incremental collection |
| 14 |
enabled, this is probably normal. The collector installs handlers to take care |
| 15 |
of these. You will not see these unless you are using a debugger. Your |
| 16 |
debugger _should_ allow you to continue. It's often preferable to tell the |
| 17 |
debugger to ignore SIGBUS and SIGSEGV ("handle SIGSEGV SIGBUS nostop noprint" |
| 18 |
in gdb, "ignore SIGSEGV SIGBUS" in most versions of dbx) and set a breakpoint |
| 19 |
in `abort`. The collector will call abort if the signal had another cause, and |
| 20 |
there was not other handler previously installed. |
| 21 |
|
| 22 |
We recommend debugging without incremental collection if possible. (This |
| 23 |
applies directly to UNIX systems. Debugging with incremental collection under |
| 24 |
Win32 is worse. See README.win32.) |
| 25 |
|
| 26 |
If the application generates an unhandled SIGSEGV or equivalent, it may often |
| 27 |
be easiest to set the environment variable `GC_LOOP_ON_ABORT`. On many |
| 28 |
platforms, this will cause the collector to loop in a handler when the SIGSEGV |
| 29 |
is encountered (or when the collector aborts for some other reason), and |
| 30 |
a debugger can then be attached to the looping process. This sidesteps common |
| 31 |
operating system problems related to incomplete core files for multi-threaded |
| 32 |
applications, etc. |
| 33 |
|
| 34 |
## Other Signals |
| 35 |
|
| 36 |
On most platforms, the multi-threaded version of the collector needs one or |
| 37 |
two other signals for internal use by the collector in stopping threads. It is |
| 38 |
normally wise to tell the debugger to ignore these. On Linux, the collector |
| 39 |
currently uses SIGPWR and SIGXCPU by default. |
| 40 |
|
| 41 |
## Warning Messages About Needing to Allocate Blacklisted Blocks |
| 42 |
|
| 43 |
The garbage collector generates warning messages of the form: |
| 44 |
|
| 45 |
|
| 46 |
Repeated allocation of very large block ... |
| 47 |
May lead to memory leak and poor performance |
| 48 |
|
| 49 |
|
| 50 |
when it needs to allocate a block at a location that it knows to be referenced |
| 51 |
by a false pointer. These false pointers can be either permanent (e.g. |
| 52 |
a static integer variable that never changes) or temporary. In the latter |
| 53 |
case, the warning is largely spurious, and the block will eventually |
| 54 |
be reclaimed normally. In the former case, the program will still run |
| 55 |
correctly, but the block will never be reclaimed. Unless the block is intended |
| 56 |
to be permanent, the warning indicates a memory leak. |
| 57 |
|
| 58 |
1. Ignore these warnings while you are using GC_DEBUG. Some of the routines |
| 59 |
mentioned below don't have debugging equivalents. (Alternatively, write the |
| 60 |
missing routines and send them to me.) |
| 61 |
2. Replace allocator calls that request large blocks with calls to |
| 62 |
`GC_malloc_ignore_off_page` or `GC_malloc_atomic_ignore_off_page`. You may |
| 63 |
want to set a breakpoint in `GC_default_warn_proc` to help you identify such |
| 64 |
calls. Make sure that a pointer to somewhere near the beginning of the |
| 65 |
resulting block is maintained in a (preferably volatile) variable as long |
| 66 |
as the block is needed. |
| 67 |
3. If the large blocks are allocated with realloc, we suggest instead |
| 68 |
allocating them with something like the following. Note that the realloc |
| 69 |
size increment should be fairly large (e.g. a factor of 3/2) for this to |
| 70 |
exhibit reasonable performance. But we all know we should do that anyway. |
| 71 |
|
| 72 |
|
| 73 |
void * big_realloc(void *p, size_t new_size) { |
| 74 |
size_t old_size = GC_size(p); |
| 75 |
void * result; |
| 76 |
if (new_size <= 10000) return(GC_realloc(p, new_size)); |
| 77 |
if (new_size <= old_size) return(p); |
| 78 |
result = GC_malloc_ignore_off_page(new_size); |
| 79 |
if (result == 0) return(0); |
| 80 |
memcpy(result,p,old_size); |
| 81 |
GC_free(p); |
| 82 |
return(result); |
| 83 |
} |
| 84 |
|
| 85 |
|
| 86 |
4. In the unlikely case that even relatively small object (<20 KB) |
| 87 |
allocations are triggering these warnings, then your address space contains |
| 88 |
lots of "bogus pointers", i.e. values that appear to be pointers but aren't. |
| 89 |
Usually this can be solved by using `GC_malloc_atomic` or the routines |
| 90 |
in `gc_typed.h` to allocate large pointer-free regions of bitmaps, etc. |
| 91 |
Sometimes the problem can be solved with trivial changes of encoding |
| 92 |
in certain values. It is possible, to identify the source of the bogus |
| 93 |
pointers by building the collector with `-DPRINT_BLACK_LIST`, which will |
| 94 |
cause it to print the "bogus pointers", along with their location. |
| 95 |
5. If you get only a fixed number of these warnings, you are probably only |
| 96 |
introducing a bounded leak by ignoring them. If the data structures being |
| 97 |
allocated are intended to be permanent, then it is also safe to ignore them. |
| 98 |
The warnings can be turned off by calling `GC_set_warn_proc` with |
| 99 |
a procedure that ignores these warnings (e.g. by doing absolutely nothing). |
| 100 |
|
| 101 |
## The Collector References a Bad Address in GC_malloc |
| 102 |
|
| 103 |
This typically happens while the collector is trying to remove an entry from |
| 104 |
its free list, and the free list pointer is bad because the free list link |
| 105 |
in the last allocated object was bad. |
| 106 |
|
| 107 |
With >99% probability, you wrote past the end of an allocated object. Try |
| 108 |
setting `GC_DEBUG` before including `gc.h` and allocating with `GC_MALLOC`. |
| 109 |
This will try to detect such overwrite errors. |
| 110 |
|
| 111 |
## Unexpectedly Large Heap |
| 112 |
|
| 113 |
Unexpected heap growth can be due to one of the following: |
| 114 |
|
| 115 |
1. Data structures that are being unintentionally retained. This is commonly |
| 116 |
caused by data structures that are no longer being used, but were not |
| 117 |
cleared, or by caches growing without bounds. |
| 118 |
2. Pointer misidentification. The garbage collector is interpreting integers |
| 119 |
or other data as pointers and retaining the "referenced" objects. A common |
| 120 |
symptom is that GC_dump() shows much of the heap as black-listed. |
| 121 |
3. Heap fragmentation. This should never result in unbounded growth, but |
| 122 |
it may account for larger heaps. This is most commonly caused by allocation |
| 123 |
of large objects. |
| 124 |
4. Per object overhead. This is usually a relatively minor effect, but |
| 125 |
it may be worth considering. If the collector recognizes interior pointers, |
| 126 |
object sizes are increased, so that one-past-the-end pointers are correctly |
| 127 |
recognized. The collector can be configured not to do this |
| 128 |
(`-DDONT_ADD_BYTE_AT_END`). |
| 129 |
|
| 130 |
The collector rounds up object sizes so the result fits well into the chunk |
| 131 |
size (`HBLKSIZE`, normally 4 KB on 32-bit machines, 8 KB on 64-bit ones) used |
| 132 |
by the collector. Thus it may be worth avoiding objects of size 2K + 1 bytes |
| 133 |
(or exactly 2 KB if a byte is being added at the end.) The last two cases can |
| 134 |
often be identified by looking at the output of a call to `GC_dump`. Among |
| 135 |
other things, it will print the list of free heap blocks, and a very brief |
| 136 |
description of all chunks in the heap, the object sizes they correspond to, |
| 137 |
and how many live objects were found in the chunk at the last collection. |
| 138 |
|
| 139 |
Growing data structures can usually be identified by: |
| 140 |
|
| 141 |
1. Building the collector with `-DKEEP_BACK_PTRS`, |
| 142 |
2. Preferably using debugging allocation (defining `GC_DEBUG` before |
| 143 |
including `gc.h` and allocating with `GC_MALLOC`), so that objects will |
| 144 |
be identified by their allocation site, |
| 145 |
3. Running the application long enough so that most of the heap is composed |
| 146 |
of "leaked" memory, and |
| 147 |
4. Then calling `GC_generate_random_backtrace` from gc_backptr.h a few times |
| 148 |
to determine why some randomly sampled objects in the heap are being |
| 149 |
retained. |
| 150 |
|
| 151 |
The same technique can often be used to identify problems with false pointers, |
| 152 |
by noting whether the reference chains printed |
| 153 |
by `GC_generate_random_backtrace` involve any misidentified pointers. |
| 154 |
An alternate technique is to build the collector with `-DPRINT_BLACK_LIST` |
| 155 |
which will cause it to report values that are almost, but not quite, look like |
| 156 |
heap pointers. It is very likely that actual false pointers will come from |
| 157 |
similar sources. |
| 158 |
|
| 159 |
In the unlikely case that false pointers are an issue, it can usually |
| 160 |
be resolved using one or more of the following techniques: |
| 161 |
|
| 162 |
1. Use `GC_malloc_atomic` for objects containing no pointers. This is |
| 163 |
especially important for large arrays containing compressed data, |
| 164 |
pseudo-random numbers, and the like. It is also likely to improve GC |
| 165 |
performance, perhaps drastically so if the application is paging. |
| 166 |
2. If you allocate large objects containing only one or two pointers at the |
| 167 |
beginning, either try the typed allocation primitives in `gc_typed.h`, |
| 168 |
or separate out the pointer-free component. |
| 169 |
3. Consider using `GC_malloc_ignore_off_page` to allocate large objects. |
| 170 |
(See `gc.h` and above for details. Large means more than 100 KB in most |
| 171 |
environments.) |
| 172 |
4. If your heap size is larger than 100 MB or so, build the collector with |
| 173 |
`-DLARGE_CONFIG`. This allows the collector to keep more precise black-list |
| 174 |
information. |
| 175 |
5. If you are using heaps close to, or larger than, a gigabyte on a 32-bit |
| 176 |
machine, you may want to consider moving to a platform with 64-bit pointers. |
| 177 |
This is very likely to resolve any false pointer issues. |
| 178 |
|
| 179 |
## Prematurely Reclaimed Objects |
| 180 |
|
| 181 |
The usual symptom of this is a segmentation fault, or an obviously overwritten |
| 182 |
value in a heap object. This should, of course, be impossible. In practice, |
| 183 |
it may happen for reasons like the following: |
| 184 |
|
| 185 |
1. The collector did not intercept the creation of threads correctly |
| 186 |
in a multi-threaded application, e.g. because the client called |
| 187 |
`pthread_create` without including `gc.h`, which redefines it. |
| 188 |
2. The last pointer to an object in the garbage collected heap was stored |
| 189 |
somewhere were the collector could not see it, e.g. in an object allocated |
| 190 |
with system `malloc`, in certain types of `mmap`ed files, or in some data |
| 191 |
structure visible only to the OS. (On some platforms, thread-local storage |
| 192 |
is one of these.) |
| 193 |
3. The last pointer to an object was somehow disguised, e.g. by XORing |
| 194 |
it with another pointer. |
| 195 |
4. Incorrect use of `GC_malloc_atomic` or typed allocation. |
| 196 |
5. An incorrect `GC_free` call. |
| 197 |
6. The client program overwrote an internal garbage collector data |
| 198 |
structure. |
| 199 |
7. A garbage collector bug. |
| 200 |
8. (Empirically less likely than any of the above.) A compiler optimization |
| 201 |
that disguised the last pointer. |
| 202 |
|
| 203 |
The following relatively simple techniques should be tried first to narrow |
| 204 |
down the problem: |
| 205 |
|
| 206 |
1. If you are using the incremental collector try turning it off for |
| 207 |
debugging. |
| 208 |
2. If you are using shared libraries, try linking statically. If that works, |
| 209 |
ensure that DYNAMIC_LOADING is defined on your platform. |
| 210 |
3. Try to reproduce the problem with fully debuggable unoptimized code. This |
| 211 |
will eliminate the last possibility, as well as making debugging easier. |
| 212 |
4. Try replacing any suspect typed allocation and `GC_malloc_atomic` calls |
| 213 |
with calls to `GC_malloc`. |
| 214 |
5. Try removing any `GC_free` calls (e.g. with a suitable `#define`). |
| 215 |
6. Rebuild the collector with `-DGC_ASSERTIONS`. |
| 216 |
7. If the following works on your platform (i.e. if gctest still works if |
| 217 |
you do this), try building the collector with |
| 218 |
`-DREDIRECT_MALLOC=GC_malloc_uncollectable`. This will cause the collector |
| 219 |
to scan memory allocated with malloc. |
| 220 |
|
| 221 |
If all else fails, you will have to attack this with a debugger. The suggested |
| 222 |
steps are: |
| 223 |
|
| 224 |
1. Call `GC_dump` from the debugger around the time of the failure. Verify |
| 225 |
that the collectors idea of the root set (i.e. static data regions which |
| 226 |
it should scan for pointers) looks plausible. If not, i.e. if it does not |
| 227 |
include some static variables, report this as a collector bug. Be sure |
| 228 |
to describe your platform precisely, since this sort of problem is nearly |
| 229 |
always very platform dependent. |
| 230 |
2. Especially if the failure is not deterministic, try to isolate |
| 231 |
it to a relatively small test case. |
| 232 |
3. Set a break point in `GC_finish_collection`. This is a good point |
| 233 |
to examine what has been marked, i.e. found reachable, by the collector. |
| 234 |
4. If the failure is deterministic, run the process up to the last |
| 235 |
collection before the failure. Note that the variable `GC_gc_no` counts |
| 236 |
collections and can be used to set a conditional breakpoint in the right |
| 237 |
one. It is incremented just before the call to `GC_finish_collection`. |
| 238 |
If object `p` was prematurely recycled, it may be helpful to look |
| 239 |
at `*GC_find_header(p)` at the failure point. The `hb_last_reclaimed` field |
| 240 |
will identify the collection number during which its block was last swept. |
| 241 |
5. Verify that the offending object still has its correct contents at this |
| 242 |
point. Then call `GC_is_marked(p)` from the debugger to verify that the |
| 243 |
object has not been marked, and is about to be reclaimed. Note that |
| 244 |
`GC_is_marked(p)` expects the real address of an object (the address of the |
| 245 |
debug header if there is one), and thus it may be more appropriate to call |
| 246 |
`GC_is_marked(GC_base(p))` instead. |
| 247 |
6. Determine a path from a root, i.e. static variable, stack, or register |
| 248 |
variable, to the reclaimed object. Call `GC_is_marked(q)` for each object |
| 249 |
`q` along the path, trying to locate the first unmarked object, say `r`. |
| 250 |
7. If `r` is pointed to by a static root, verify that the location pointing |
| 251 |
to it is part of the root set printed by `GC_dump`. If it is on the stack |
| 252 |
in the main (or only) thread, verify that `GC_stackbottom` is set correctly |
| 253 |
to the base of the stack. If it is in another thread stack, check the |
| 254 |
collector's thread data structure (`GC_thread[]` on several platforms) |
| 255 |
to make sure that stack bounds are set correctly. |
| 256 |
8. If `r` is pointed to by heap object `s`, check that the collector's |
| 257 |
layout description for `s` is such that the pointer field will be scanned. |
| 258 |
Call `*GC_find_header(s)` to look at the descriptor for the heap chunk. |
| 259 |
The `hb_descr` field specifies the layout of objects in that chunk. |
| 260 |
See `gc_mark.h` for the meaning of the descriptor. (If its low order 2 bits |
| 261 |
are zero, then it is just the length of the object prefix to be scanned. |
| 262 |
This form is always used for objects allocated with `GC_malloc` or |
| 263 |
`GC_malloc_atomic`.) |
| 264 |
9. If the failure is not deterministic, you may still be able to apply some |
| 265 |
of the above technique at the point of failure. But remember that objects |
| 266 |
allocated since the last collection will not have been marked, even if the |
| 267 |
collector is functioning properly. On some platforms, the collector can |
| 268 |
be configured to save call chains in objects for debugging. Enabling this |
| 269 |
feature will also cause it to save the call stack at the point of the last |
| 270 |
GC in `GC_arrays._last_stack`. |
| 271 |
10. When looking at GC internal data structures remember that a number |
| 272 |
of `GC_xxx` variables are really macro defined to `GC_arrays._xxx`, so that |
| 273 |
the collector can avoid scanning them. |