Debian Codegraph @

HOWTOs/GNU-APL-Memory-Management.asciidoc

raw
1 = Memory Management in GNU APL
2 :Author: Jürgen Sauermann, GNU APL
3 :page-width: 80em
4
5 == Abstract
6
7 This document describes how memory is being managed in GNU APL and what
8 the consequences are for a GNU APL programmer.
9
10 == Fundamentals
11
12 * The GNU APL interpreter is an executable that runs in a single process.
13 Inside the interpreter run a handful of threads; the memory overhead of
14 these threads can be neglected.
15
16 * Depending on configuration details and user preferences, the interpreter may
17 fork additional processes (APserver, GTK_server) whose sizes are so small
18 they can also be neglected.
19
20 The sizes of the binaries mentioned (at the time of this writing) are:
21
22 .Binary Sizes
23 [options="header"]
24 [width="60%"]
25 |===============
26 | Binary | Size (bytes)
27 |GNU APL Interpreter | 44,620,848
28 |APserver | 1,540,944
29 |Gtk_server | 530,376
30 |===============
31
32 * After the interpreter has started it requests additional memory from the
33 operating system on behalf of APL commands and APL code executed. That
34 memory is allocated by malloc(), either directly or indirectly (via Cpp
35 new()). When the memory is no longer needed, it is returned to malloc()
36 memory management (for later re-use). The memory kept in the malloc()
37 memory management is only returned to the operating system when the
38 GNU APL interpreter exits.
39
40 * GNU APL tries to allocate memory statically as much as possible. The
41 static allocation has already taken place when the interpreter outputs its
42 first APL prompt. After that dynamic memory allocation via malloc()/new()
43 is performed for objects whose lifetime is potentially infinite (i.e. whose
44 destruction is triggered by APL code. These objects are:
45 ** APL variables
46 ** APL values
47 ** defined functions
48 ** call stack (aka. *)SI* stack) entries of defined functions
49
50 In the following the sizes of these objects are given. The sizes differ
51 slightly between 32-bit and 64-bit systems because 32-bit systems use 4-byte
52 pointers and 64-bit systems use 8-byte pointers. In addition to the indicated
53 sizes, there is a (very) small overhead in the malloc() library for every
54 object. Each of the objects is significantly larger than its malloc overhead
55 and therefore the overhead can normally be ignored. A typical APL workspace is
56 assumed to have fewer than 1000 APL symbols (i.e. variables or defined
57 functions) and an )SI stack depth of less than 100. A typical computer on which
58 GNU APL runs is assumed to have 4 GB of memory available to the interpreter and
59 it should definitely have at least 1 GB.
60
61 .Sizes of Dynamic Objects
62 [options="header"]
63 [width="60%"]
64 |===============
65 | Object | 32-bit system | 64-bit system
66 | ValueStackItem | 16 | 24
67 | Token | 20 | 24
68 | Ravel Cell | 20 | 24
69 | UCS_string | 20 + X | 24 + X
70 | APL Symbol | 56 + X | 80 + X
71 | Defined Function | 292 + X | 400 + X
72 | Call Stack Entry | 2952 | 3952
73 | APL Value | 372 + X | 456 + X
74 |===============
75
76 In the table above, the term + X means that the object is a structure that
77 contains other objects. For example, the size of a defined function depends
78 on the number of function lines and their lengths. Likewise, the size of an
79 APL value depends primarily on the number of its ravel elements. The term X
80 for the different objects is explained below.
81
82 === X for a Unicode string (aka. UCS string)
83
84 X is 4 bytes per Unicode character. For performance reasons at least 16
85 characters are allocated so that X ≥ 64.
86
87 === X for an APL Symbol
88
89 An APL symbol has two variable parts: a UCS string containing the name of the
90 symbol (e.g. a variable name or the name of a defined function) and a value
91 stack. The value stack of a function is initially empty, but is increased
92 when a defined function is called and the symbol is a local variable or a
93 label in the defined function and decreased if that function returns. The
94 amount of memory by which the stack is increased is the size of a
95 *ValueStackItem* in the table above.
96
97 === X for a Defined Function
98
99 When a defined function is created, its textual representation (i.e. the
100 argument for ⎕FX or the result produced by the ∇-editor) is stored in the
101 function object for the purpose of displaying error information in case the
102 execution of the function fails.
103
104 The textual representation is then converted into an internal representation
105 of fixed-sized Tokens with one token for each lexical unit. A lexical unit is:
106
107 * one user-defined name, or
108 * one distinguished name, or
109 * one primitive function, or
110 * a special symbol (◊ [ ] etc.)
111
112 Finally, a jump table to the different function lines is added to the
113 function. The X for the function is therefore Utx + N×To + L×Jt, where
114
115 * Utx is the size of the UCS_string containing the function text,
116 * N is the number of Tokens,
117 * To is the size of a Token as above,
118 * L is the number of function lines
119 * Jt is the size of a jump table entry (4 bytes)
120
121 === X for an APL Value
122
123 GNU APL distinguishes between short values and long values. A short value is a
124 value with up to 12 ravel elements while a long value is a value with more
125 than 12 ravel elements. The split point between long and short values is 12 by
126 default, but can be changed via ./configure.
127
128 The rationale for distinguishing short and long values is performance. A short
129 value can be allocated in a single new() call while a long value needs two
130 new() calls: one for the value header (which contains the shape of the value),
131 and one for the ravel cells of the value. Looking at the details of malloc()
132 one can see that the allocation or deallocation of smaller memory areas is more
133 efficient than the allocation or deallocation of larger areas, Finally, short
134 APL values occur more frequently in APL programs than long values, and they
135 are also more frequently overridden as a whole (which allocates a new value
136 and discards the old) as opposed to being updated via indexed assignment
137 (which changes an existing value but does not create a newly allocated
138 value (unless nesting is involved)).
139
140 * If a value is short then X = 0
141 * otherwise X = N × Ce, where
142 ** N is the number of ravel elements (i.e. ⍴,VALUE), and
143 ** Ce is the size of a ravel Cell as above.
144
145 *Note:* In the classical APL world of ISO 8485, all ravel elements of an APL
146 value had the same type such as bit, 8-bit char, 32-bit integer, or float and
147 the ravel was densely packed. Accessing a particular element V[n] of a value V
148 was a fast constant time operation, which was internally performed very
149 frequently. Many APL primitive functions (with the noteworthy exception of
150 scalar functions) need to perform multiple computations of V[n] for one
151 of their arguments V.
152
153 In ISO 13751 the concept of mixed values was introduced. In a mixed value,
154 different ravel elements can have different types, so that the type becomes a
155 property of each ravel element of a value instead of a property of the value
156 itself. The implementer of ISO 13751 then had different alternatives (and
157 possibly combinations of them) to deal with mixed values:
158
159 * Leave the existing bit-, byte-, integer- and float-arrays as they
160 are and add a new kind of mixed-arrays. This was a reasonable approach for
161 those who had a code base from ISO 8485 already and who wanted to extend
162 that code base to ISO 13751.
163
164 * Make the ravel elements pointers to a different location that contains the
165 real ravel elements. Since all pointers have the same size, the constant-time
166 access to arbitrary ravel elements would remain. The performance cost
167 would be somewhat higher due to:
168 ** the additional indirection via the pointers, and
169 ** possibly one more memory allocation per ravel element.
170 While the cost for the additional indirection would probably be more than
171 compensated by the reduced access time, the cost for additional memory
172 allocations could have become a major headache.
173
174 * Make the ravel elements virtual objects that all have the same size. That
175 means instead of primitive functions working with pointers to ravel
176 elements, the ravel elements would point to the functions that are
177 appropriate for their type. On one hand, this approach wastes quite some
178 memory because the largest possible type (i.e. complex numbers) determines
179 the size of all ravel cells including those that would fit in far fewer
180 bytes. On the other hand, there are quite a few advantages:
181 ** The same size of all elements causes V[n] to take constant time,
182 ** The indexed assignment of e.g. a single element (of a different type) to a
183 non-mixed array would also take (constant time (in option 1 above the entire
184 array would have needed to be converted from non-mixed to mixed)).
185 ** The number of different array types would be reduced to 1 instead of being
186 increased from 5 to 6 (bit, byte, integer, float, complex, and mixed). This
187 in turn makes a huge number of type checks in the dyadic APL primitives
188 obsolete. Since a virtual cell knows its own type, it only needs to check
189 the type of the other cell(s) in a dyadic operation.
190 ** The "wasted" space could be used for other features at no extra cost such as:
191 *** 64-bit integers instead of 32-bit Integers,
192 *** rational number arithmetic for floating point numbers
193
194 Since GNU APL was to be designed from scratch without having a prior code base,
195 the first option was ruled out almost immediately. The second option was
196 briefly considered, but then ruled out as well because code size for the second
197 option was expected to be significantly larger than for the third option.
198
199 == Summary
200
201 The discussion of sizes so far can be summarized as:
202
203 *Corollary:* For a typical APL program on a typical computer, the only items
204 of concern in the context of memory management are APL values.
205
206 *Note:* The default values of system variable ⎕SYL (aka. system limits)
207 guarantee to some extent that an APL program behaves like a typical APL
208 program,
209
210 == How the Available Memory is Estimated
211
212 === History
213
214 In the good old days of CP/M and friends, say 1975, memory management was
215 rather simple. The process running an APL interpreter would know how much
216 memory was reserved (and therefore available for it) and it could use that
217 memory without restrictions. The amount of memory (say *initial-⎕WA*) that
218 the process could use was simply:
219
220 * *initial-⎕WA = top-of-memory - top-of-used-memory*
221
222 This value was determined at the start-up of the interpreter (where the malloc()
223 pool was essentially empty) and the interpreter would keep track of the memory
224 that it allocated after that, say *memory-used*. So printing ⎕WA at any time
225 would essentially show *initial-⎕WA - memory-used*. And that was it.
226
227 === Present Memory Management
228
229 Since then a number of things have changed. The changes are in general
230 advantageous, but also cause trouble in very specific situations. The main
231 differences, as seen by an application like GNU APL are:
232
233 * Change from physical to virtual memory. These days the memory allocated by
234 malloc() is virtual. Under *normal* circumstances the application need not
235 care about this difference, but in some special cases it is negatively
236 affected (and can then do very little about it).
237
238 * A direct consequence of virtual memory is that the size of it is no longer
239 determined by the physical memory available in a computer, but by other
240 properties like the number of bits in the virtual memory addresses. This
241 makes it impossible to determine initial-⎕WA (more precisely: ⎕WA is no
242 longer a constant but becomes dependent on what happens not only in the APL
243 interpreter itself but also in other processes and/or in the kernel.
244
245 * Over-commitment: Modern operating systems return far more virtual memory to
246 applications than they really have in terms of physical memory. The idea is
247 that not all applications reach their maximum need for memory at the same
248 time, so that the same piece of physical memory can be used by different
249 applications (or by the kernel for that matter) at different times.
250
251 *Corollary 1:* In the old days, the memory that an application has obtained
252 from the kernel was a *guarantee*. These days it is merely a *promise* which
253 is *normally* kept but may fail with a small though non-zero probability.
254
255 *Corollary 2:* As far as GNU APL is concerned, GNU APL does not (and actually
256 cannot) guarantee proper operation if the kernel reaches the limit of its
257 physical memory. Some error handling mechanisms, in particular WS FULL errors,
258 may fail in a non-graceful fashion - including immediate termination of the
259 GNU APL process without any warnings.
260
261 Although a crash of GNU APL cannot be prevented in general, one can decrease
262 the probability of such crashes by taking countermeasures beforehand. Most GNU
263 APL users will not need this (see *normal* above), but those facing problems
264 with improper WS FULL handling should read on.
265
266 == Improving the GNU APL WS FULL handling
267
268 === The OOM handler
269
270 One enemy of GNU APL is the kernel's OOM (out-of-memory) handler. The OOM
271 handler is invoked when the kernel needs (physical) memory and has none. It
272 then kills one or more running processes in order to claim their memory back.
273 The processes that are killed (ungracefully) may or may not be the process(es)
274 that have caused the shortage of memory. That is:
275
276 * GNU APL may be killed by some other (unrelated) process or interrupt that
277 has requested memory, or
278 * some other (unrelated) process may be killed by a memory request from GNU
279 APL, or
280 * GNU APL is killed as a result of its own memory request.
281
282 It should be clear that the first two cases are very difficult to reproduce.
283 The third case is much easier by creating a very large APL value. It is also
284 the case that has been observed in reality.
285
286 In GNU/Linux, one can protect individual processes from being killed by the
287 OOM handler with the following command (as root; <PID> is the process ID of
288 the process to be protected:
289
290 ----
291 echo -17 > /proc/<PID>/oom_adj
292 ----
293
294 The OOM handler can also be disabled entirely with
295
296 ----
297 sysctl vm.overcommit_memory=2
298 ----
299
300 or:
301
302 ----
303 echo "vm.overcommit_memory=2" >> /etc/sysctl.conf
304 ----
305
306 *Warning:* These settings can seriously impact the stability of the operating
307 system and should not be used on machines whose availability is a concern.
308 There is no point in sacrificing the stability of a system for the stability of
309 a process running on that system.
310
311 === Using System Limits
312
313 Instead of disabling the OOM handler, one should use methods that prevent GNU
314 APL from requesting too much memory, so that the OOM handler is not invoked
315 in the first place (at least not from GNU APL). There are several ⎕SYL (aka.
316 system limit) entries that can be set by an APL program to prevent GNU APL
317 from requesting too much memory. Some of them have to be combined with others
318 to achieve full protection.
319
320 * ⎕SYL[1;] limits the depth of the )SI stack and therefore the amount of
321 memory spent for function calls. Referring to Table 2 above, *⎕SYL[1;2]←1000*
322 will throw a LIMIT ERROR if the )SI stack grows above 3 Megabytes (32-bit
323 systems) or 4 Megabytes (64-bit systems).
324 * ⎕SYL[2;] limits the number of APL values. *⎕SYL[2;2]←1000* will throw a
325 LIMIT ERROR when more than 1000 APL values (variables as well as
326 intermediate results) are created. Referring to Table 2 above, this
327 corresponds to 372 kByte on 32-bit systems, or 456 kByte on 64-bit systems.
328 Note that the space for localized variables of defined functions belongs to
329 ⎕SYL[2;] and not to ⎕SYL[1;].
330 * ⎕SYL[3;] limits the number of ravel elements in APL values.
331 *⎕SYL[3;2]←1000000* will throw a LIMIT ERROR before more than 20 MByte on
332 32-bit systems, or more than 24 MByte on 64-bit systems is requested.
333 * Finally, ⎕SYL[31 32;] define a safety margin that is is explained below.
334
335 In short, ⎕SYL[1;] protects primarily against too deep recursion, which is most
336 likely caused by a programming fault. ⎕SYL[2;] protects against too many APL
337 values, and ⎕SYL[3;] protects against APL values becoming too large.
338
339 Like WS FULL and other APL errors, hitting a system limit brings the
340 interpreter back to immediate execution (interactive) mode. If one wants to
341 handle the situation programmatically, one can catch the limit error with ⎕EA,
342 ⎕EB, or ⎕EC.
343
344
345 === Analyzing /proc/meminfo
346
347 On a GNU/Linux machine, the file */proc/meminfo* provides quite useful
348 information about the usage of memory at the point in time when /proc/meminfo
349 is read. Before setting system limits as described above, one should consult
350 /proc/meminfo to see how the memory is distributed and then set the limits
351 accordingly.
352
353 Ideally */proc/meminfo* shows something like:
354
355 ----
356 MemTotal: 185736 kB
357 MemFree: 12660 kB
358 MemAvailable: 126264 kB
359 Buffers: 74704 kB
360 Cached: 54732 kB
361 ----
362
363 In that case *MemAvailable:* is the value on which the system limits settings
364 should be based. On older GNU/Linux machines *MemAvailable:* may not be
365 displayed. In that case *MemFree: + Cached:* can be used as a workaround.
366
367 The actual implementation of *MemAvailable:* by the kernel is quite complicated
368 (more than simply adding *MemFree:* and *Cached:*). For that reason, even if
369 *MemAvailable:* were available on all GNU/Linux machines it would still be
370 impossible to call it before each creation of an APL value.
371
372 Instead, GNU APL uses the following simple algorithm:
373
374 * After the interpreter has started, compute *total_memory* as follows:
375 ** If the process that runs the interpreter has a limit set on the amount of
376 its virtual memory, then the interpreter assumes that amount of memory
377 will be available and sets *total_memory* to that limit. In this case
378 */proc/meminfo* is not used.
379 ** Otherwise (i.e. the virtual memory for the process is unlimited) the
380 interpreter consults */proc/meminfo* and sets *total_memory* to either
381 *MemAvailable:* (if present) or else to *MemFree: + Cached:*.
382 * As new APL values are created and destroyed in the course of running APL
383 programs, the amount of memory allocated for them is tracked in, say,
384 *used_memory*.
385 * Before requesting new memory of, say, size *new*, GNU APL checks that
386
387 ----
388 total_memory_1 ≥ used_memory + new + ⎕WA-margin
389 ----
390 * In this check *total_memory_1* is *⎕SYL[32;2]* percent of *total_memory*
391 above, and reduced by *⎕SYL[31;2]*. The reason for the scaling by
392 *⎕SYL[32;2]* is that malloc() usually requests slightly more memory from
393 the kernel than the application had requested from malloc().
394
395 ** If that check fails then WS FULL is raised,
396 ** otherwise the request is forwarded to malloc() and from there possibly
397 to the kernel.
398
399 The ⎕WA-margin currently has a default value of 0, but can be changed via
400
401 === Hints for GNU APL Users
402
403 ==== Setting a memory limit
404
405 If stability of a system is a concern, then setting memory limits for
406 processes is normally a good idea. The command for doing that differs
407 between shells; in *bash* you can start GNU APL with a virtual memory limit
408 of like this:
409
410 ----
411 ulimit -v 1000000
412 apl
413 ----
414 The unit is kB, so that the command above will set the limit to 1 GB. The limit
415 remains in force for all subsequent commands and can be removed like this:
416
417 ----
418 ulimit -v unlimited
419 apl
420 ----
421
422 *Note:* setting memory limits via *ulimit* as such does not directly solve the
423 problem that a process may have when it reaches the limit. However, it helps a
424 process to predict when the limit will be reached.
425
426 === 32-bit GNU/Linux
427
428 On 32-bit machines, there is a per process memory limit of about 3 GByte. If
429 the memory of a machine is, say, 8 GB or more, then one can have several
430 processes with each process allocating up to 3 GByte as long as the memory
431 shown in */proc/meminfo* is not exceeded.
432
433 If GNU APL sees a *total_memory* of more than 3 GB in the analysis of
434 */proc/meminfo*, then it reduces *total_memory* to about 3 GB. if it is
435 running on a 32-bit kernel. 64-bit machines have a similar limit, but the
436 limit is so high that it is not relevant in practice.
437
438 The safety margin *(total_memory_1* - *total_memory)* that results from
439 *⎕SYL[31 32;2]* remains in effect so that the total memory that an APL program
440 can obtain is somewhere below 3 GB.
441
442 ==== Troubleshooting WS FULL problems
443
444 For debugging purposes the safety margin can be disabled in APL like this:
445
446 ----
447 ⎕SYL[31;2] ← 1000000 ⍝ smallest margin (1 MB)
448 ⎕SYL[32;2] ← 200 ⍝ far above available
449 ----
450
451 WS FULL can occur very many places within the interpreter. To find the exact
452 position where a WS FULL was generated, some related logging facilities can be
453 turned on:
454
455 ----
456 ]log 25
457 ]log 26
458 ]log 45
459 ----
460 On a 32-bit machine a WS FULL can be reliably produced like this (the example
461 was kindly provided by Christian Robert):
462
463 ----
464 N←5000000 ⍝ 32-bit; use a larger value for 64-bit
465 A←⊂'0123456789ABCDEF'
466 ↑↑ N ⍴ A ⍝ trigger WS FULL
467 ----
468 This example is of particular interest because, depending upon variable *N*
469 it creates two different challenges for the kernel. One is a big allocation
470 when the process is already near starvation and one is a small allocation
471 that fails.
472
473 In both cases the WS FULL is caused by *N ⍴ A*. Because A is a scalar (due to
474 ⊂), all ravel elements of the result are nested values. Therefore the
475 computation of *N ⍴ A* creates exactly 1+N values: a large value for the result
476 itself and N small nested values - the ravel of the result.
477
478 Referring to Table 2, the variable *A* (and therefore each nested value of the
479 result needs *372 + 16×20 = 692* bytes. The top-level of the result requires
480 *372 + N×20* bytes. Let *F* be the amount of (free) memory that could be
481 obtained from malloc(). Then:
482
483 1. if *F < 372 + N×20* then the allocation of the top-level value fails,
484
485 2. else if *F < 372 + N×20 + N×692* then the allocation of the top-level value
486 succeeds, but the allocation of one of the nested sub-values fails,
487
488 3. else all allocations succeed and *N ⍴ A* succeeds as expected.
489
490 The less than obvious difference between the first and the second case is the
491 amount of free memory at the point in time where the allocation failed:
492
493 1. if the first (large) allocation fails, then the free memory is still F,
494
495 2. if one of the subsequent (small) allocations fails then (since the previous
496 allocations have succeeded) the free memory is now less than the requested
497 size, regardless how large F was initially. In our example F would be less
498 than 692 bytes (in theory) or less than about 1024 (in practice because
499 malloc allocates the next power of 2).
500
501 Exception handling with only 1024 bytes left can fail as can be demonstrated
502 on a 32-bit machine quite easily. The following examples show both cases (with
503 safety margin disabled in order to allow near-memory-starvation.
504
505 ==== Example 1: WS FULL with sufficient free memory left
506
507 ----
508
509 ⍝ enable relevant logging facilities
510 ]log 25
511 Log facility 'more verbose errors ' is now ON
512 ]log 26
513 Log facility 'details of error throwing ' is now ON
514 ]log 45
515 Log facility 'details of Value allocation ' is now ON
516
517 ⍝ disable safety margin
518 ⎕SYL[31 32;2]←1000000 200
519 ⎕SYL[31 32;]
520 ⎕WA safety margin (bytes) 1000000
521 ⎕WA memory scale (%) 200
522
523 A←⊂'0123456789ABCDEF' ⍝ 692 bytes
524 N←200000000 ⍝ for 200 Mio ravel Cells
525 ↑↑ N⍴A ⍝ 4 GB top-level + 138.4 GB nested sub-values
526
527 throwing WS FULL at PrimitiveFunction.cc:230
528
529 ----------------------------------------
530 -- Stack trace at Error.cc:184
531 ----------------------------------------
532 0xB7160AF3 __libc_start_main
533 0x8092998 main
534 0x8247A9D Workspace::immediate_execution(bool)
535 0x80EF330 Command::process_line()
536 0x80EFCAB Command::do_APL_expression(UCS_string&)
537 0x80EF459 Command::finish_context()
538 0x80FB64D Executable::execute_body() const
539 0x81DBE31 StateIndicator::run()
540 0x8149E2C Prefix::reduce_statements()
541 0x8146913 Prefix::reduce_A_F_B_()
542 0x815579D Bif_F12_RHO::eval_AB(Value_P, Value_P)
543 0x8155463 Bif_F12_RHO::do_reshape(Shape const&, Value const&)
544 0x80B5EE6 Value_P::Value_P(Shape const&, char const*)
545 0x8245B9A Value::init_ravel()
546 0x80FA35B throw_apl_error(ErrorCode, char const*)
547 ========================================
548 WS FULL+
549 ↑↑N⍴A
550 ^ ^
551 Ravel allocation failed
552
553 )MORE
554 new Value(PrimitiveFunction.cc:230) failed (APL error in ravel allocation)
555 throwing WS FULL at Value_P.icc:227
556 ----
557
558 ==== Example 2: WS FULL with very little free memory left
559
560 ----
561 ⍝ enable relevant logging facilities
562 ]log 25
563 Log facility 'more verbose errors ' is now ON
564 ]log 26
565 Log facility 'details of error throwing ' is now ON
566 ]log 45
567 Log facility 'details of Value allocation ' is now ON
568
569 ⍝ disable safety margin
570 ⎕SYL[31 32;2]←1000000 200
571 ⎕SYL[31 32;]
572 ⎕WA safety margin (bytes) 1000000
573 ⎕WA memory scale (%) 200
574
575 A←⊂'0123456789ABCDEF' ⍝ 692 bytes
576 N←20000000 ⍝ for 20 Mio ravel Cells
577 ↑↑ N⍴A ⍝ 0,4 GB top-level + 13.84 GB nested sub-values
578 Value_P::Value_P(const Shape & shape, const char * loc) failed at
579 Value_P.icc:235 (caller: PrimitiveFunction.cc:240)
580 what: std::bad_alloc
581 initial sbrk(): 0xa28a000
582 current sbrk(): 0x9eaec000
583 alloc_size: 0x140 (320)
584 used memory: 0xbbb495c0 (3149174208)
585
586 throwing WS FULL at Value.cc:233
587
588 ----------------------------------------
589 -- Stack trace at Error.cc:184
590 ----------------------------------------
591 backtrace_symbols() failed. Using backtrace_symbols_fd() instead...
592
593 ./apl[0x809a380]
594 /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb70c0af3]
595 ./apl(main+0x48)[0x8092998]
596 ./apl(_ZN9Workspace19immediate_executionEb+0x1d)[0x8247a9d]
597 ./apl(_ZN7Command12process_lineEv+0x460)[0x80ef330]
598 ./apl(_ZN7Command17do_APL_expressionER10UCS_string+0x6b)[0x80efcab]
599 ./apl(_ZN7Command14finish_contextEv+0x29)[0x80ef459]
600 ./apl(_ZNK10Executable12execute_bodyEv+0x1d)[0x80fb64d]
601 ./apl(_ZN14StateIndicator3runEv+0x21)[0x81dbe31]
602 ./apl(_ZN6Prefix17reduce_statementsEv+0x22c)[0x8149e2c]
603 ./apl(_ZN6Prefix13reduce_A_F_B_Ev+0xd3)[0x8146913]
604 ./apl(_ZN11Bif_F12_RHO7eval_ABE7Value_PS0_+0xad)[0x815579d]
605 ./apl(_ZN11Bif_F12_RHO10do_reshapeERK5ShapeRK5Value+0x258)[0x8155678]
606 ./apl(_ZNK11PointerCell10init_otherEPvR5ValuePKc+0x49)[0x814c6e9]
607 ./apl(_ZNK5Value5cloneEPKc+0x30)[0x82438a0]
608 ./apl(_ZN7Value_PC1ERK5ShapePKc+0x13e)[0x80b5fae]
609 ./apl(_ZN5Value15catch_exceptionERKSt9exceptionPKcS4_S4_+0x31e)[0x823c93e]
610 ./apl(_Z15throw_apl_error9ErrorCodePKc+0x6b)[0x80fa35b]
611 ========================================
612 WS FULL+
613 ↑↑N⍴A
614 ^ ^
615
616 )MORE
617 new Value(const Shape & shape, const char * loc) failed (std::bad_alloc)
618 throwing WS FULL at Value_P.icc:227
619 ----
620
621 ==== Differences between Example 1 and Example 2
622
623 * Execution time: the second example uses noticeably more time before the WS
624 FULL occurs. In this time the interpreter creates a large number of
625 sub-values until the memory is exhausted.
626 * The stack dumps differ. The pretty-printing of Cpp function names requires
627 extra memory, in example 2 that extra memory is not available so that
628 backtrace_symbols() fails (and the interpreter then uses a function that
629 needs less memory):
630
631 ----
632 ----------------------------------------
633 -- Stack trace at Error.cc:184
634 ----------------------------------------
635 backtrace_symbols() failed. Using backtrace_symbols_fd() instead...
636
637 ----
638
639 * The WS FULL is thrown from different locations. PrimitiveFunction.cc:230 is
640 in the implementation of dyadic ⍴, while Value.cc:233 is the exception
641 handler for *std::bad_alloc* (see also *what:* in the debug printout),
642
643 ----
644 throwing WS FULL at PrimitiveFunction.cc:230
645 ----
646 ----
647 throwing WS FULL at Value.cc:233
648 ----
649
650 * The )MORE information in the two examples differ. The term *Ravel
651 allocation* in the printout of example 1 refers to the allocation of the
652 (large) top-level ravel.
653
654 === Final Remarks
655
656 The exact behavior of processes with too little memory is generally difficult
657 to predict. While the A⍴N example given above is understood to some extent,
658 there are other cases such as out-of-memory when the process stack is
659 exhausted that were not yet reported.
660
661 We are trying our best to fix such cases, but when reporting them as errors
662 on bug-apl please consider the following.
663
664 * It helps a lot if you can provide a reliable way to reproduce the problem.
665 Most of the time for fixing an error is spent on reproducing it.
666 * Check that the problem cannot be fixed by setting memory limits. As explained
667 above there are some boundary conditions from the operating system that
668 cannot be fixed (or even properly handled) by a process. Remember the
669 existing tools for that:
670 ** ulimit to protect processes from each other,
671 ** ⎕SYL[1 2 3;2] to limit what the GNU APL interpreter may try to allocate, and
672 ** ⎕SYL[31 32;2] to avoid near-memory-starvation situations in the WS FULL
673 exception handling.
674
675 Thanks for using GNU APL.
676