| 1 |
\chapter{What To Do When \bacula{} Crashes (Kaboom)} |
| 2 |
\label{blb:KaboomChapter} |
| 3 |
\index[general]{Kaboom!What To Do When \bacula{} Crashes } |
| 4 |
\index[general]{What To Do When \bacula{} Crashes (Kaboom) } |
| 5 |
|
| 6 |
If you are running on a Linux system, and you have a set of working |
| 7 |
configuration files, it is very unlikely that \textbf{\bacula{}} will crash. As with |
| 8 |
all software, however, it is inevitable that someday, it may crash, |
| 9 |
particularly if you are running on another operating system or using a new or |
| 10 |
unusual feature. |
| 11 |
|
| 12 |
This chapter explains what you should do if one of the three \textbf{\bacula{}} |
| 13 |
daemons (Director, File, Storage) crashes. When we speak of crashing, we |
| 14 |
mean that the daemon terminates abnormally because of an error. There are |
| 15 |
many cases where \bacula{} detects errors (such as PIPE errors) and will fail |
| 16 |
a job. These are not considered crashes. In addition, under certain |
| 17 |
conditions, \bacula{} will detect a fatal in the configuration, such as |
| 18 |
lack of permission to read/write the working directory. In that case, |
| 19 |
\bacula{} will force itself to crash with a SEGFAULT. However, before |
| 20 |
crashing, \bacula{} will normally display a message indicating why. |
| 21 |
For more details, please read on. |
| 22 |
|
| 23 |
|
| 24 |
\section{Traceback} |
| 25 |
\index[general]{Traceback} |
| 26 |
|
| 27 |
Each of the three \bacula{} daemons has a built-in exception handler which, in |
| 28 |
case of an error, will attempt to produce a traceback. If successful the |
| 29 |
traceback will be emailed to you. |
| 30 |
|
| 31 |
For this to work, you need to ensure that a few things are setup correctly on |
| 32 |
your system: |
| 33 |
|
| 34 |
\begin{benumerate} |
| 35 |
\item You must have a version of \bacula{} built with debug information turned |
| 36 |
on and not stripped of debugging symbols. |
| 37 |
|
| 38 |
\item You must have an installed copy of \btool{gdb} (the \acs{GNU} debugger), and it |
| 39 |
must be on \textbf{\bacula{}'s} path. On some systems such as Solaris, |
| 40 |
\btool{gdb} may be replaced by \btool{dbx}. |
| 41 |
|
| 42 |
\item The \bacula{} installed script file \btool{btraceback} must be in the same |
| 43 |
directory as the daemon which dies, and it must be marked as executable. |
| 44 |
|
| 45 |
\item The script file \bfilename{btraceback.gdb} must have the correct path to it |
| 46 |
specified in the \btool{btraceback} file. |
| 47 |
|
| 48 |
\item You must have a \btool{mail} program which is on \textbf{\bacula{}'s} path. |
| 49 |
By default, this \btool{mail} program is set to \btool{bsmtp}, so it must |
| 50 |
be correctly configured. |
| 51 |
|
| 52 |
\item If you run either the Director or Storage daemon under a non-root |
| 53 |
userid, you will most likely need to modify the \btool{btraceback} file |
| 54 |
to do something like \btool{sudo} (raise to root priority) for the |
| 55 |
call to \btool{gdb} so that it has the proper permissions to debug |
| 56 |
\bacula{}. |
| 57 |
\end{benumerate} |
| 58 |
|
| 59 |
If all the above conditions are met, the daemon that crashes will produce a |
| 60 |
traceback report and email it to you. If the above conditions are not true, |
| 61 |
you can either run the debugger by hand as described below, or you may be able |
| 62 |
to correct the problems by editing the \btool{btraceback} file. I recommend not |
| 63 |
spending too much time on trying to get the traceback to work as it can be |
| 64 |
very difficult. |
| 65 |
|
| 66 |
The changes that might be needed are to add a correct path to the \btool{gdb} |
| 67 |
program, correct the path to the \bfilename{btraceback.gdb} file, change the \btool{mail} |
| 68 |
program or its path, or change your email address. The key line in the |
| 69 |
\btool{btraceback} file is: |
| 70 |
|
| 71 |
\begin{bVerbatim} |
| 72 |
gdb -quiet -batch -x /home/kern/bacula/bin/btraceback.gdb \ |
| 73 |
$1 $2 2>\&1 | bsmtp -s "Bacula traceback" [email protected] |
| 74 |
\end{bVerbatim} |
| 75 |
|
| 76 |
Since each daemon has the same traceback code, a single btraceback file is |
| 77 |
sufficient if you are running more than one daemon on a machine. |
| 78 |
|
| 79 |
\section{Testing The Traceback} |
| 80 |
\index[general]{Traceback!Testing The } |
| 81 |
\index[general]{Testing The Traceback } |
| 82 |
|
| 83 |
To ``manually'' test the traceback feature, you simply start \textbf{\bacula{}} then |
| 84 |
obtain the \textbf{PID} of the main daemon thread (there are multiple threads). |
| 85 |
The output produced here will look different depending on what OS and what |
| 86 |
version of the kernel you are running. |
| 87 |
Unfortunately, the output had to be split to fit on this page: |
| 88 |
|
| 89 |
\begin{bVerbatim} |
| 90 |
[kern@rufus kern]$ ps fax --columns 132 | grep bacula-dir |
| 91 |
2103 ? S 0:00 /home/kern/bacula/k/src/dird/bacula-dir -c |
| 92 |
/home/kern/bacula/k/src/dird/dird.conf |
| 93 |
2104 ? S 0:00 \_ /home/kern/bacula/k/src/dird/bacula-dir -c |
| 94 |
/home/kern/bacula/k/src/dird/dird.conf |
| 95 |
2106 ? S 0:00 \_ /home/kern/bacula/k/src/dird/bacula-dir -c |
| 96 |
/home/kern/bacula/k/src/dird/dird.conf |
| 97 |
2105 ? S 0:00 \_ /home/kern/bacula/k/src/dird/bacula-dir -c |
| 98 |
/home/kern/bacula/k/src/dird/dird.conf |
| 99 |
\end{bVerbatim} |
| 100 |
|
| 101 |
which in this case is 2103. Then while \bacula{} is running, you call the program |
| 102 |
giving it the path to the \bacula{} executable and the \textbf{PID}. In this case, |
| 103 |
it is: |
| 104 |
|
| 105 |
\begin{bVerbatim} |
| 106 |
./btraceback /home/kern/bacula/k/src/dird 2103 |
| 107 |
\end{bVerbatim} |
| 108 |
|
| 109 |
It should produce an email showing you the current state of the daemon (in |
| 110 |
this case the Director), and then exit leaving \textbf{\bacula{}} running as if |
| 111 |
nothing happened. If this is not the case, you will need to correct the |
| 112 |
problem by modifying the \btool{btraceback} script. |
| 113 |
|
| 114 |
Typical problems might be that \btool{gdb} or \btool{dbx} for Solaris is not on |
| 115 |
the default path. Fix this by specifying the full path to it in the |
| 116 |
\btool{btraceback} file. Another common problem is that you haven't modified the |
| 117 |
script so that the \btool{bsmtp} program has an appropriate smtp server or |
| 118 |
the proper syntax for your \acs{SMTP} server. If you use the \btool{mail} program |
| 119 |
and it is not on the default path, it will also fail. On some systems, it |
| 120 |
is preferable to use \btool{Mail} rather than \btool{mail}. |
| 121 |
|
| 122 |
\section{Getting A Traceback On Other Systems} |
| 123 |
\index[general]{Getting A Traceback On Other Systems} |
| 124 |
\index[general]{Systems!Getting A Traceback On Other} |
| 125 |
|
| 126 |
It should be possible to produce a similar traceback on systems other than |
| 127 |
Linux, either using \btool{gdb} or some other debugger. Solaris with \btool{dbx} |
| 128 |
loaded works quite fine. On other systems, you will need to modify the |
| 129 |
\btool{btraceback} program to invoke the correct debugger, and possibly correct the |
| 130 |
\bfilename{btraceback.gdb} script to have appropriate commands for your debugger. If |
| 131 |
anyone succeeds in making this work with another debugger, please send us a |
| 132 |
copy of what you modified. Please keep in mind that for any debugger to |
| 133 |
work, it will most likely need to run as root, so you may need to modify |
| 134 |
the \btool{btraceback} script accordingly. |
| 135 |
|
| 136 |
\label{blb:ManuallyDebugging} |
| 137 |
\section{Manually Running \bacula{} Under The Debugger} |
| 138 |
\index[general]{Manually Running \bacula{} Under The Debugger} |
| 139 |
\index[general]{Debugger!Manually Running \bacula{} Under The} |
| 140 |
|
| 141 |
If for some reason you cannot get the automatic traceback, or if you want to |
| 142 |
interactively examine the variable contents after a crash, you can run \bacula{} |
| 143 |
under the debugger. Assuming you want to run the Storage daemon under the |
| 144 |
debugger (the technique is the same for the other daemons, only the name |
| 145 |
changes), you would do the following: |
| 146 |
|
| 147 |
\begin{benumerate} |
| 148 |
\item Start the Director and the File daemon. If the Storage daemon also |
| 149 |
starts, you will need to find its PID as shown above (\btool{ps fax | grep bacula-sd}) |
| 150 |
and kill it with a command like the following: |
| 151 |
|
| 152 |
\begin{bVerbatim} |
| 153 |
kill -15 PID |
| 154 |
\end{bVerbatim} |
| 155 |
|
| 156 |
where you replace \textbf{PID} by the actual value. |
| 157 |
|
| 158 |
\item At this point, the Director and the File daemon should be running but |
| 159 |
the Storage daemon should not. |
| 160 |
|
| 161 |
\item \btool{cd} to the directory containing the Storage daemon |
| 162 |
|
| 163 |
\item Start the Storage daemon under the debugger: |
| 164 |
|
| 165 |
\begin{bVerbatim} |
| 166 |
gdb ./bacula-sd |
| 167 |
\end{bVerbatim} |
| 168 |
|
| 169 |
\item Run the Storage daemon: |
| 170 |
|
| 171 |
\begin{bVerbatim} |
| 172 |
run -s -f -c ./bacula-sd.conf |
| 173 |
\end{bVerbatim} |
| 174 |
|
| 175 |
You may replace the \bfilename{./bacula-sd.conf} with the full path to the Storage |
| 176 |
daemon's configuration file. |
| 177 |
|
| 178 |
\item At this point, \bacula{} will be fully operational. |
| 179 |
|
| 180 |
\item In another shell command window, start the Console program and do what |
| 181 |
is necessary to cause \bacula{} to die. |
| 182 |
|
| 183 |
\item When \bacula{} crashes, the \btool{gdb} shell window will become active and |
| 184 |
\btool{gdb} will show you the error that occurred. |
| 185 |
|
| 186 |
\item To get a general traceback of all threads, issue the following command: |
| 187 |
|
| 188 |
\begin{bVerbatim} |
| 189 |
thread apply all bt |
| 190 |
\end{bVerbatim} |
| 191 |
|
| 192 |
After that you can issue any debugging command. |
| 193 |
\end{benumerate} |
| 194 |
|
| 195 |
\section{Getting Debug Output from \bacula{}} |
| 196 |
\index[general]{Getting Debug Output from \bacula{} } |
| 197 |
Each of the daemons normally has debug compiled into the program, but |
| 198 |
disabled. There are two ways to enable the debug output. One is to add the |
| 199 |
\textbf{-d nnn} option on the command line when starting the debugger. |
| 200 |
The \textbf{nnn} is the debug level, and generally anything between 50 and 200 is |
| 201 |
reasonable. The higher the number, the more output is produced. The output is |
| 202 |
written to standard output. |
| 203 |
|
| 204 |
The second way of getting debug output is to dynamically turn it on using the |
| 205 |
Console using the \bcommandname{setdebug} command. The full syntax of the command is: |
| 206 |
|
| 207 |
\begin{bVerbatim} |
| 208 |
setdebug level=nnn client=client-name storage=storage-name dir |
| 209 |
\end{bVerbatim} |
| 210 |
|
| 211 |
If none of the options are given, the command will prompt you. You can |
| 212 |
selectively turn on/off debugging in any or all the daemons (i.e. it is not |
| 213 |
necessary to specify all the components of the above command). |