= Trouble-Shooting GNU APL // vim: et:ts=4:sw=4:tw=78 :Author: Dr. Jürgen Sauermann, GNU APL :page-width: 120em :toc: :toclevels: 4 :iconsdir: /usr/share/asciidoc/icons :icons: == Abstract This document contains hints for *GNU APL* users who would like to report *GNU APL* problems and who want to make the life of the *GNU APL* maintainer(s) a little easier by localizing the root cause of a problem. == Introduction *GNU APL* is, in terms of source code size, a fairly large program. As of this writing the source code for the interpreter, not counting code in sub-directories, has more than 110,000 lines of code. Finding faults in code of that quantity is not as trivial as finding faults in a short program of only a few thousand lines. Past experience shows, that after a new fault was observed, the time for reproducing and localizing the fault typically takes more than 90% of the total time to fix it. For this reason, *GNU APL* comes with a handful of built-in trouble-shooting capabilities that makes locating a fault easier. Some of these capabilities require that *GNU APL* is built and run in a specific way as described in this document. Almost every error that might occur in *GNU APL* falls into one of the following 4 categories of increasing severities: * Errors in User Programs (aka. APL errors), * Incorrect APL results, * Failed Assertions, and * Crashes of the *GNU APL* interpreter. Each category requires different means and preparations for locating the root cause of the error. == Errors in User Programs (aka. APL errors), Errors in user programs are reported by the APL intepreter when APL code is being executed. Before the interpreter starts its internal computation(s) it checks the arguments. These checks depend on the computation required and usually occur in the following order: . Syntax Check (may raise SYNTAX ERROR) . Valence Check (may raise VALENCE ERROR or AXIS ERROR) . Value Check (may raise VALUE ERROR) . Rank Check (may raise RANK ERROR) . Shape Check (may raise LENGTH ERROR) . Domain Checks (may raise DOMAIN ERROR) That is, a VALUE error can normally only occur in the absence of SYNTAX and VALENCE ERRORs, a LENGTH ERROR can only occur in the absence of SYNTAX, VALENCE, AXIS, VALUE, and RANK ERRORs, and so on. An APL error is displayed immediately and (provided that output coloring is enabled) in a different color than the APL output. The first line of the error display is the type of error, possibly followed by +. The +, if present, indicates that there is more information available about the error (and that the additional information can be obtained with command )MORE. Example: +/[1;2] 3 ⍝ ; is not allowed in function axes. AXIS ERROR+ +/[1;2]3 ^ ^ )MORE illegal ; in axis APL errors shall, of course, not be reported to the *GNU APL* maintainers, with the following exceptions: * An error was raised incorrectly, i.e. for valid APL input, or * the display of )MORE is misleading or could be improved. Many errors create no )MORE information and that is on purpose. If you believe, however, that a particular error case is difficult to grasp and would benefit from a )MORE information, then feel free to propose that in an email to bug-apl@gnu.org == Incorrect APL results If *GNU APL* produces an incorrect result, then you should report it in an email to bug-apl@gnu.org. People on that mailing list are gentle and helpful, so don't be afraid to report an error. However, please note the following: Even though APL as a language is standardized, (ISO Standard 13751 "Programming Language APL, Extended") there exist considerable differences (and also errors) between: * the APL standard, and * the APL documentation of different vendors, * and the interpreter implementations of different vendors (including *GNU APL*). In the case of such differences, *GNU APL* usually resolves the issue in the following order: * IBM APL2 PC implementation (highest priority) * IBM APL2 language reference manual * ISO Standard 13751 * Other vendors (lowest priority). Before sending an email to bug-apl@gnu.org it may make sense to check if the issue was reported already (and often caused by such differences). However, don't be afraid to report the same error twice if you are not sure. == Failed Assertions An assertion is an expression in the source code that the programmer assumes to be *true* (aka. ≠ 0 in C/Cpp). It can be very hard to find the origin (root cause) of an error, in particular when many CPU instructions were performed between the root cause and error being detected. By making assertions all over place in the source code, the programmer introduces checks along the execution path in order to reduce the number of CPU instructions between a root cause and the point in time where it is being detected. In the C/Cpp code, assertions are typically made in the following places: * near the beginning of a function (also known as pre-condition) to check the arguments passed to the function, * after calling a reasonably complex sub-function, and * near the end of a function (post-condition) to check the result returned by the function. Assertions are implemented as one or the other of two C/Cpp macros called Assert() and Assert1(). As of this writing there are more than 530 Assert() and more than 110 Assert1() macros spread across the *GNU APL* source code. The reason for having different Assert macros is their impact on performance. Assert1() macros are intended to test more trivial conditions while Assert() macros test more complex conditions. Every test performed by a single Assert() or Assert1() macro has a (relatively mild) performance impact. However, the large number of Assert() macros increases the total performance impact which may be undesirable. To deal with this, the user may *./configure* one of 3 different assertion levels as follows (see also *README-2-configure*): * *ASSERT_LEVEL_WANTED = 0*: this disables both the Assert1() and the Assert() macro so that the interpreter will have the maximal performance. * *ASSERT_LEVEL_WANTED = 1* (the default): this disables the Assert1() macro and enables and the Assert() macro. * *ASSERT_LEVEL_WANTED = 2*: this enables both the Assert1() and the Assert() macro. We kindly ask the user to: make develop which, besides other settings, does *./configure* with ASSERT_LEVEL_WANTED=2, thus enabling both Assert macros and possibly making it easier to find the root cause of a fault. As confidence with the source code increases over time, Assert() macros will eventually be converted to Assert1() macros while new code will initially be protected with Assert() macros. === Emulating Failed Assertions Fortunately we can emulate a failed Assert() or Assert1() macro with ⎕FIO: ---- ⎕FIO ¯16 ⍝ emulate Assert() ⎕FIO ¯17 ⍝ emulate Assert1() ---- If the respective macro is enabled with ASSERT_LEVEL_WANTED as explained above, then it produces a stack trace like this: ---- ⎕FIO ¯16 ============================================================================== Assertion failed: 0 && "Simulated Assert() (aka. ⎕FIO ¯16)" in Function: eval_B in file: Quad_FIO.cc:1085 C/Cpp call stack: ---------------------------------------- -- Stack trace at Assert.cc:72 ---------------------------------------- 0x7FBFD71CBBF7 __libc_start_main 0x563D2320B04A main 0x563D233E822F Workspace::immediate_execution(bool) 0x563D2327411A Command::process_line() 0x563D232742DC Command::process_line(UCS_string&) 0x563D232768A2 Command::do_APL_expression(UCS_string&) 0x563D23276C73 Command::finish_context() 0x563D2329B5D2 Executable::execute_body() const 0x563D2338199C StateIndicator::run() 0x563D232DAF86 Prefix::reduce_statements() 0x563D232DBE92 Prefix::reduce_MISC_F_B_() 0x563D2332D0CF Quad_FIO::eval_B(Value_P) const 0x563D2322C2A0 do_Assert(char const*, char const*, char const*, int) ======================================== SI stack: Depth: 0 Exec: 0x563d23f9fde0 Safe exec: 0 Pmode: ◊ ⎕FIO ¯16 PC: 3 (4) RETURN_STATS Stat: ⎕FIO ¯16 err_code: 0x0 ============================================================================== ---- The C/Cpp stack above tells us which assertion has failed and where the assertion is located in the C/Cpp source code (i.e. file Quad_FIO.cc line 1085), while the )SI stack tells us where that location is in the APL code. This brings us closer to the root cause of a problem. Unfortunately the hex addresses numbers on the left side of the C/Cpp stack dump are process specific, i.e. the same *⎕FIO ¯16* will produce different numbers in a different process of the operating system. === Line numbers in Stack Traces If the functions on the right side of the C/Cpp stack are relatively small (as most functions in the source code of *GNU APL* are) then we can usually find the exact source line for each stack entry rather easily using the function name displayed. For larger functions, the same function may be called from different source code lines in the same function. This can be improved by converting the hex addresses on the left into source line numbers as exlained in the following. First of all the Cpp compiler must be instructed to not discard line numbers. Older g++ versions would always keep line numbers in the object files, but newer versions do not. The trick is this: ---- CXXFLAGS="-rdynamic -gdwarf-2" ./configure ... ---- *make develop* also does that for you. The CXXFLAGS above tell the *g++* compiler to use an older object format in which line numbers are not discarded. If your compiler does not accept *-gdwarf-2* then it probably uses it anyway. The second step is to create a file named *apl.lines*. If *GNU APL* finds this file then it uses it to map hex addresses to line numbers. In the src directory, the make target *apl.lines* creates this file (which takes quite a while and is therefore not created in a standard build). *make apl.lines* essentially does: ---- objdump --section=.text --line-numbers --disassemble apl > apl.lines ---- which extracts the line numbers from the apl binary (provided that the compiler has inserted them - see above). If *GNU APL* finds apl.lines, then the stack dump looks a little different: ---- ⎕FIO ¯16 ============================================================================== Assertion failed: 0 && "Simulated Assert() (aka. ⎕FIO ¯16)" in Function: eval_B in file: Quad_FIO.cc:1085 C/Cpp call stack: total_lines in apl.lines: 618610 assembler lines in apl.lines: 86495 source line numbers found: 86495 ---------------------------------------- -- Stack trace at Assert.cc:72 ---------------------------------------- 0x29465EE76BF7 __libc_start_main 0x14CCD2 main at main.cc:635 0x3166E1 Workspace::immediate_execution(bool) at Workspace.cc:273 0x1B1004 Command::process_line() at Command.cc:103 0x1B11C6 Command::process_line(UCS_string&) at Command.cc:139 0x1B378C Command::do_APL_expression(UCS_string&) at Command.cc:329 0x1B3B5D Command::finish_context() at Command.cc:347 0x1D6806 Executable::execute_body() const at Executable.cc:261 0x2B564F StateIndicator::run() at StateIndicator.cc:362 0x211DA4 Prefix::reduce_statements() at Prefix.cc:723 (discriminator 4) 0x212A79 Prefix::reduce_MISC_F_B_() at Prefix.cc:990 0x261E7F Quad_FIO::eval_B(Value_P) const at Quad_FIO.cc:1090 0x16CC03 do_Assert(char const*, char const*, char const*, int) at Assert.cc:74 ======================================== SI stack: Depth: 0 Exec: 0x561c865a4da0 Safe exec: 0 Pmode: ◊ ⎕FIO ¯16 PC: 3 (4) RETURN_STATS Stat: ⎕FIO ¯16 err_code: 0x0 ============================================================================== ---- The difference is twofold: . the hex numbers are now relative to the location of the *main()* function of *GNU APL*, which makes them not only smaller but also identical even in different processes. . The source line numbers are now displayed for most hex numbers. Some addresses can not be resolved, this could be caused by C/Cpp function inlining or calls through function pointers etc. Needless to say that *GNU APL* trouble-shooters prefer stack traces with hex numbers resolved to line numbers. Occasionally you may get this message: ---- ... C/Cpp call stack: Cannot show line numbers, since apl.lines is older than apl. Consider running 'make apl.lines' in directory /home/eedjsa/apl-1.8 to fix this ... ---- This happens if you do *make apl* without a subsequent *make apl.lines*. === Summary concerning Assertions The *GNU APL* configuration for troubleshooting purposes is obtained like this: ---- cd make develop cd src make clean make apl.lines ---- where *top-level-directory* is the directory that contains the *README* files. Additional troubleshooting options may be used as well. Consider the *make* target *develop:* in file Makefile.incl (around line 41) only as a template that can be augmented by other troubleshooting options (see also *README-2-configure*). == The )CHECK command *GNU APL* provides a command *)CHECK* which is somewhat similar to IBM APL2's command *)CHECK WS*. This command performs an internal check of the internal data structures of the interpreter. Under normal circumstances, the output of the *)CHECK* command should be: ---- OK - no stale functions OK - no stale values OK - no stale indices OK - no duplicate parents ---- If not, then there were inconsistencies (primarily memory leaks) that need to be fixed and should therefore be reported to bug-apl@gnu.org. For example, a *stale value* is an APL value that has memory allocated but is not reachable any longer via either the symbol table or the interpreter (like an APL variable) or the body of a defined function (like an APL literal). If a )CHECK command fails, then it is important to remember, or better write down, which sequence of APL operations has led to the state of the workspace. *GNU APL* has a command *)HISTORY* which displays the last lines entered. An assertion checks the immediate state of the interpreter and can therefore not check the entire lifetime of an APL value or function. For this purpose, *GNU APL* povides a different debugging tool called *Value history*. The Value history tracks the locations in the *GNU APL* source code that have manipulated a value from its creation (memory allocation) to its destruction (memory deallocation). This tracking, when enabled, has performance impacts and is therefore disabled by default. If the )CHECK command indicates stale values, but also under some other error conditions, the history of a value at fault is being displayed and helps the *GNU APL* maintainers to locate the problem. The value history is enabled in *./configure*: ---- ./configure VALUE_HISTORY_WANTED=yes ... ---- and therefore requires recompilation of the interpreter. == Logging facilities == Crashes of the GNU APL interpreter Some programming errors bypass all built-in debugging tools of GNU and crash the interpreter rather than returning to the interactive immediate execution mode of APL. The crash may be silent or accompanied by an error message and it may or may not create a core file. A core file is an important file if no other information is printed as to what kind of fault has occurred. In the good old times core files were always produced when a binary executable crashed. These days the creation of core files may need to be enabled before the crashing binary is started. In GNU/Linux with its *bash* shell that means: ---- ulimit -c unlimited # enable core files ---- Instead of *unlimited* you may also use a different file size limit. After a core file was produced (we always assume the interpreter was started in directory src below the top--level-directory) you can start the debugger *gdb* in the src directory with the core file as its second argument: ---- gdb ./apl core ---- After that, you can use the gdb command *bt* (for 'backtrace') to show a stack dump similar to the one discussed earlier. The *GNU APL* maintainer(s) will appreciate a copy of that stack dump for problems that are difficult to reproduce. Normally the file is named '*core*' and found in the same directory as the binary, but some GNU/Linux distributions use a different directory for storing core files. See also *man 5 core*. Finally, try to nail down the root cause with logging facilities (see below). == Non-reproducible Errors The most difficult to fix errors are those that occur on one machine (usually your's) and not on others. The first thing to locate their root cause it to *./configure* with either no options or via *make develop* to see if the error is related to specific *./configure* options. Also, look for compiler warnings when *GNU APL* is built. In the default build, compiler warnings can be overlooked easiy. *make develop* turns most warnings into errors so you will see at the end if the build was successful. === Optimization Recently we have seen faults that may have been caused by an overly aggressive optimization of the *GNU APL* Cpp source code. These faults were observed for the same source code but only when compiled with newer versions of the same g++ compiler. By default *GNU APL* compiles with optimization level -O2. If a fault is suspected to be caused by optimization, then it could make sense to also test the same source code of *GNU APL* without any optimization. To do that: ---- cd src find and remove or replace the string -O2 in the relevant Makefile(s) make clean all ---- == Logging facilities The source code of *GNU APL* is populated with more than 380 Log() macros. A Log() macro may or may not print some additional information when the execution of the APL interpreter reaches them. If all these Log() macros would output something, then even a trivial APL statement would cause a lot of output in which the relevant information gets easily lost. For that reason, each Log () macro belongs to one of currently over 45 Groups called *logging facilities*. Each logging facility can be turned ON or OFF which causes Log() macros belonging the facility to print or not to print its information. The large (and increasing) number of logging facilities cause the tests, whether Log() macro or its respective logging facility is ON or OFF, to have significant performance impacts. For this reason, *GNU APL* provides two different methods to control how logging facilities are turned ON or OFF: . static logging . dynamic logging The method is selected via *./configure* at compile time (more precisely: at *./configure* time), see also *README-2-configure*. === Static Logging Static logging is used with: ---- ./configure DYNAMIC_LOG_WANTED=no ---- Static logging is also the default, therefore *DYNAMIC_LOG_WANTED=no* is the same as not using DYNAMIC_LOG_WANTED= in *./configure*. With Static logging, the selection of which logging facility shall be ON is made by the first argument of the *log_def()* macros in file *src/Logging.def. A value of 0 means OFF while 1 means ON for the respective logging facility. With static logging, the log_def() macro defines an *enum* value for each log_def() macro which is a compile-time constant. The compiler will therefore remove all code (including the ON/OFF for all log_def(0, ...) macros and also all tests for log_def(1, ...) in all Log() macros. That is: * Advantage: Static Logging has little if any performance impacts (none if all log_def macros are off, and * Disadvantage: changes of log_def() macros require re-compilation of GNU APL. === Dynamic Logging Dynamic logging is used with: ---- ./configure DYNAMIC_LOG_WANTED=yes ---- With dynamic logging, the log_def() macro defines the initial value of a variable (rather than an enum value) for each logging facility. After the interpreter was started, the value of each variable can be changed from OFF to ON (and vice versa) with the *GNU APL* debug command *]LOG* (which toggles the state of the respective variable). * Disadvantage: Dynamic Logging has a small performance impact (even if all variables are set to OFF) which will accumulate for all Log() macros, * Advantage: no re-compilation of *GNU APL* is required. ==== The ]LOG command GNU APL has a few debug commands, which are like standard APL commands but start with ] instead of ). One of these debug commands is *]LOG* which controls the logging facilities. If static logging was *./configure*'d then *]LOG* displays an error message because changing the logging facilities at run-time requires dynamic logging. ---- ]LOG Command ]LOG is not available, since dynamic logging was not configured for this APL interpreter. To enable dynamic logging (which will slightly decrease performance), recompile the interpreter as follows: ./configure DYNAMIC_LOG_WANTED=yes (... other configure options) make make install (or try: src/apl) above the src directory. ---- If dynamic logging was ./configure'd then *]LOG* without an argument displays the current state (i.e. ON or OFF) of all logging facilities: ---- ]LOG 1: (OFF) AV details 2: (OFF) new and delete calls 3: (OFF) input from user or testcase file 4: (OFF) parser: parsing 5: (OFF) ... function find_closing() 6: (OFF) ... tokenization 7: (OFF) ... function collect_constants() 8: (OFF) ... create value() 9: (OFF) defined function: fix() 10: (OFF) ... set_line() 11: (OFF) ... load() 12: (OFF) ... execute() 13: (OFF) ... enter/leave 14: (OFF) State indicator: enter/leave 15: (OFF) ... push/pop 16: (OFF) Symbol: lookup 17: (OFF) ... push/pop and )ERASE 18: (OFF) ... resolve 19: (OFF) Value: glue() 20: (OFF) ... erase_stale() 21: (OFF) APL primitive function format 22: (OFF) character conversions 23: (OFF) APL system function Quad-FX 24: (OFF) commands )LOAD, )SAVE, )IN, and )OUT 25: (OFF) more verbose errors 26: (OFF) details of error throwing 27: (OFF) nabla editor 28: (OFF) execute(): state changes 29: (OFF) PrintBuffer: align() function 30: (OFF) Output: cork() functions 31: (OFF) Details of test execution 32: (OFF) Prefix parser 33: (OFF) ... location information 34: (OFF) FunOper1 and FunOper2 functions 35: (OFF) Shared Variable operations 36: (OFF) command line arguments (argc/argv) 37: (OFF) interpreter start-up messages 38: (OFF) optimization messages 39: (OFF) )LOAD and )SAVE details 40: (OFF) Svar_DB signals 41: (OFF) Parallel (multi-core) execution 42: (OFF) EOC handler functionality 43: (OFF) ⎕DLX details 44: (OFF) command ]DOXY 45: (OFF) details of Value allocation 46: (OFF) ⎕TF details 47: (OFF) ⎕PLOT details ---- Otherwise *]LOG N* toggles the state of logging facility N, e.g.: ---- ]log 25 Logging facility 'more verbose errors' is now ON ]log 26 Logging facility 'details of error throwing' is now ON ---- === Logging Facilities of Interest Many of the currently over 40 logging facilities serve a particular purpose for the maintainers or have become obsolete and are therefore not too useful for the trouble-shooting performed by the user. The others mentioned below. ==== ]Log 2 : new and delete calls For tracking the allocation and de-allocation of memory. To find memo leaks. ==== ]Log 25 and 26 : APL Errors An APL ERROR provides enough information as to where an error has occurred in the APL code, but none as to where that error was detected on the C/Cpp code. These logging facilities add this information. ==== ]Log 32 Prefix Parser Parsing of APL commands and Expressions. To analyze SYNTAX ERRORs. ==== ]Log 37 : Interpreter Startup Information when the *GNU APL* interpreter starts. This facility can also be enabled from the command line because the output of interest occurs before the interpreter accepts commands like *]LOG*. == Intrusive Troubleshooting If all the above fails, then it is time for intrusive troubleshooting. By intrusive troubleshooting we mean that the source code is modified in such a way that it prints troubleshooting information at source code locations where no Log() macros have been inserted. The top-level algorithm for intrusive troubleshooting is this: ---- loop: 1. modify the source code by adding printouts 2. recompile GNU APL 3. run apl and reproduce the problem 4. repeat until the error (or a least a source code file or function responsible for the error) was found. ---- All this is usually done in the *src* directory where most of the source files are located and where the compile result will be stored. Alternatively one could use a debugger like gdb for the same purpose, setting breakpoints and printing data. However, the author prefers to modify the source code, in particular if more than one printout is needed to find a fault. To simplify the source code for printouts, *GNU APL* has defined two powerful macros *LOC* and *Q()*. The *Q()* macro is used exclusively for intrusive troubleshooting while the LOC macro is also used for other purposes in the source code (with more than 1700 invocations as of this writing). Understanding the LOC macro is therefore a prerequisite for understanding the *GNU APL* code. === The LOC Macro The LOC macro is a short but somewhat tricky construction based on two other macros *Loc* and *STR*. The LOC macro takes no arguments and is therefore used without (), *STR()* takes one argument and *Loc(x) takes two arguments. All three macros *LOC*, *Loc()*, and *STR()* are *#defined* in file *Common.hh* which makes them available in practically every source file. The definitions are: ---- #define STR(x) #x #define LOC Loc(__FILE__, __LINE__) #define Loc(f, l) f ":" STR(l) ---- How does this work? Suppose we have a 13-line Cpp file *test.cc* like this: ---- #include // for cout #define LOC Loc(__FILE__, __LINE__) #define Loc(f, l) f ":" STR(l) #define STR(x) #x int main() { std::cout << LOC << std::endl; // line 10 of test.cc std::cout << LOC << std::endl; // line 11 of test.cc std::cout << LOC << std::endl; // line 12 of test.cc } ---- If we compile and run this file then it produces the following output: ---- test.cc:10 test.cc:11 test.cc:12 ---- That is, LOC expands to a string literal that consists of the filename in which LOC was used, followed by a colon, followed by the line number in which LOC was used. String literals have the advantage that they are produced at compile time and can be passed as function arguments at almost no performance cost. Now lets see how *LOC* works. Consider the first *LOC* on line *10* of file *test.cc*. Then: . macro *LOC* expands to *Loc(\_\_FILE\_\_, \_\_LINE__)* . *\_\_FILE__* expands to literal *"test.cc"* . *\_\_LINE__* expands to the line number *10*, so that (so far) . *LOC* expands to *Loc("test.cc", 10)* . *Loc("test.cc", 10)* expands to *"test.cc" ":" STR(10)* . *STR(10)* expands to *"10"* so that (so far) LOC expands to *"test.cc" ":" "10"* . the compiler combines the 3 adjacent string literals *"test.cc"*, *":"*, and *"10"* into the single string literal *"test.cc:10"* which is then printed. This is quite some work for the compiler, but the result is such that the run-time cost for *GNU APL* is only the loading of the string literal when *GNU APL* starts. === The Q Macro The second trick is the Q() macro which is defined as ---- #define Q(x) get_CERR() << std::left << setw(20) << #x ":" << " '" << x << "' at " LOC << endl; ---- In this definition *get_CERR()* is a function which returns an output channel such as *std::cerr* (or *stderr* in C). The operator << is overloaded a lot and in such a way that commonly used data types in *GNU APL* are being printed nicely. A consequence of the overloading is that *Q()* macro can be used with many different data types without the burdon to properly format them for the debug output. *Q(x)* uses the *LOC* macro to get the source file name and line number of its invocation and essentially prints *x* in a suitable format and followed by its source code location. For example: ---- #include // for cout #include // for setw using namespace std; #define STR(x) #x #define LOC Loc(__FILE__, __LINE__) #define Loc(f, l) f ":" STR(l) #define Q(x) cerr << std::left << std::setw(20) << #x ":" << " '" << x << "' at " LOC << std::endl; int main() { Q("Hello") // line 15 Q(10) Q(LOC) } ---- which, when compiled, produces: ---- "Hello": 'Hello' at test.cc:15 10: '10' at test.cc:16 LOC: 'test.cc:17' at test.cc:17 ---- With these prerequisites intrusive troubleshooting boils down to spreading *Q()* macros in different source files until the fault is found. A frequent question is not the value of a Cpp variable, but rather if (and when) a particular source code line of code is reached. To answer this question just insert *Q(LOC)* or *Q(0)* on the line of interest. Another useful helper is *subversion* (svn). After having made multiple insertions of *Q()* macros in various source files (and hopefully having located the fault this way), simply do ---- svn revert * ---- in the *src* directory which undoes all changes you made. == Keep It Short and Simple The first thing to check before reporting an error is to make sure that you have compiled and installed the latest GNU APL version from subversion (SVN) or git. From a maintainer's perspective, the trouble reports that are appreciated the most are the short ones. Locating a fault usually requires multiple repetitions of the sequence that causes the fault. Consider two extremes: A. The fault can be reproduced by executing a handful of APL lines in immediate execution mode, possibly accompanied by one or two ∇-defined functions or lambdas. B. A tar file contains a workspace, possibly )COPYing other workspaces. In order to reproduce the fault, the workspace has to be )LOADed and started in a specific way in order to reproduce the error. In case A. the maintainer can simply copy the lines into a small apl script created for fixing the fault. That script can then be repeated again and again by starting the script from the command line of a shell. In case B. the maintainer must first install the workspaces that come with the fault. Then start apl, )LOAD the workspace and figure out: * how the workspace shall be started, * how the defined functions in the workspace appear, how they interact, and which ones are related to the fault, * and so on. Case B. creates a lot of overhead for the maintainer that does very little to find the fault. Although sometimes a complex workspace is needed to reproduce a fault, this is rather rare and almost all errors reported so far could be reduced to a small number of APL lines. Last but not least, please have a look here: https://www.gnu.org/software/libc/manual/html_node/Reporting-Bugs.html