Debian Codegraph @

manuals/en/problems/kaboom.tex

raw
1 \chapter{What To Do When \bacula{} Crashes (Kaboom)}
2 \label{blb:KaboomChapter}
3 \index[general]{Kaboom!What To Do When \bacula{} Crashes }
4 \index[general]{What To Do When \bacula{} Crashes (Kaboom) }
5
6 If you are running on a Linux system, and you have a set of working
7 configuration files, it is very unlikely that \textbf{\bacula{}} will crash. As with
8 all software, however, it is inevitable that someday, it may crash,
9 particularly if you are running on another operating system or using a new or
10 unusual feature.
11
12 This chapter explains what you should do if one of the three \textbf{\bacula{}}
13 daemons (Director, File, Storage) crashes. When we speak of crashing, we
14 mean that the daemon terminates abnormally because of an error. There are
15 many cases where \bacula{} detects errors (such as PIPE errors) and will fail
16 a job. These are not considered crashes. In addition, under certain
17 conditions, \bacula{} will detect a fatal in the configuration, such as
18 lack of permission to read/write the working directory. In that case,
19 \bacula{} will force itself to crash with a SEGFAULT. However, before
20 crashing, \bacula{} will normally display a message indicating why.
21 For more details, please read on.
22
23
24 \section{Traceback}
25 \index[general]{Traceback}
26
27 Each of the three \bacula{} daemons has a built-in exception handler which, in
28 case of an error, will attempt to produce a traceback. If successful the
29 traceback will be emailed to you.
30
31 For this to work, you need to ensure that a few things are setup correctly on
32 your system:
33
34 \begin{benumerate}
35 \item You must have a version of \bacula{} built with debug information turned
36 on and not stripped of debugging symbols.
37
38 \item You must have an installed copy of \btool{gdb} (the \acs{GNU} debugger), and it
39 must be on \textbf{\bacula{}'s} path. On some systems such as Solaris,
40 \btool{gdb} may be replaced by \btool{dbx}.
41
42 \item The \bacula{} installed script file \btool{btraceback} must be in the same
43 directory as the daemon which dies, and it must be marked as executable.
44
45 \item The script file \bfilename{btraceback.gdb} must have the correct path to it
46 specified in the \btool{btraceback} file.
47
48 \item You must have a \btool{mail} program which is on \textbf{\bacula{}'s} path.
49 By default, this \btool{mail} program is set to \btool{bsmtp}, so it must
50 be correctly configured.
51
52 \item If you run either the Director or Storage daemon under a non-root
53 userid, you will most likely need to modify the \btool{btraceback} file
54 to do something like \btool{sudo} (raise to root priority) for the
55 call to \btool{gdb} so that it has the proper permissions to debug
56 \bacula{}.
57 \end{benumerate}
58
59 If all the above conditions are met, the daemon that crashes will produce a
60 traceback report and email it to you. If the above conditions are not true,
61 you can either run the debugger by hand as described below, or you may be able
62 to correct the problems by editing the \btool{btraceback} file. I recommend not
63 spending too much time on trying to get the traceback to work as it can be
64 very difficult.
65
66 The changes that might be needed are to add a correct path to the \btool{gdb}
67 program, correct the path to the \bfilename{btraceback.gdb} file, change the \btool{mail}
68 program or its path, or change your email address. The key line in the
69 \btool{btraceback} file is:
70
71 \begin{bVerbatim}
72 gdb -quiet -batch -x /home/kern/bacula/bin/btraceback.gdb \
73 $1 $2 2>\&1 | bsmtp -s "Bacula traceback" [email protected]
74 \end{bVerbatim}
75
76 Since each daemon has the same traceback code, a single btraceback file is
77 sufficient if you are running more than one daemon on a machine.
78
79 \section{Testing The Traceback}
80 \index[general]{Traceback!Testing The }
81 \index[general]{Testing The Traceback }
82
83 To ``manually'' test the traceback feature, you simply start \textbf{\bacula{}} then
84 obtain the \textbf{PID} of the main daemon thread (there are multiple threads).
85 The output produced here will look different depending on what OS and what
86 version of the kernel you are running.
87 Unfortunately, the output had to be split to fit on this page:
88
89 \begin{bVerbatim}
90 [kern@rufus kern]$ ps fax --columns 132 | grep bacula-dir
91 2103 ? S 0:00 /home/kern/bacula/k/src/dird/bacula-dir -c
92 /home/kern/bacula/k/src/dird/dird.conf
93 2104 ? S 0:00 \_ /home/kern/bacula/k/src/dird/bacula-dir -c
94 /home/kern/bacula/k/src/dird/dird.conf
95 2106 ? S 0:00 \_ /home/kern/bacula/k/src/dird/bacula-dir -c
96 /home/kern/bacula/k/src/dird/dird.conf
97 2105 ? S 0:00 \_ /home/kern/bacula/k/src/dird/bacula-dir -c
98 /home/kern/bacula/k/src/dird/dird.conf
99 \end{bVerbatim}
100
101 which in this case is 2103. Then while \bacula{} is running, you call the program
102 giving it the path to the \bacula{} executable and the \textbf{PID}. In this case,
103 it is:
104
105 \begin{bVerbatim}
106 ./btraceback /home/kern/bacula/k/src/dird 2103
107 \end{bVerbatim}
108
109 It should produce an email showing you the current state of the daemon (in
110 this case the Director), and then exit leaving \textbf{\bacula{}} running as if
111 nothing happened. If this is not the case, you will need to correct the
112 problem by modifying the \btool{btraceback} script.
113
114 Typical problems might be that \btool{gdb} or \btool{dbx} for Solaris is not on
115 the default path. Fix this by specifying the full path to it in the
116 \btool{btraceback} file. Another common problem is that you haven't modified the
117 script so that the \btool{bsmtp} program has an appropriate smtp server or
118 the proper syntax for your \acs{SMTP} server. If you use the \btool{mail} program
119 and it is not on the default path, it will also fail. On some systems, it
120 is preferable to use \btool{Mail} rather than \btool{mail}.
121
122 \section{Getting A Traceback On Other Systems}
123 \index[general]{Getting A Traceback On Other Systems}
124 \index[general]{Systems!Getting A Traceback On Other}
125
126 It should be possible to produce a similar traceback on systems other than
127 Linux, either using \btool{gdb} or some other debugger. Solaris with \btool{dbx}
128 loaded works quite fine. On other systems, you will need to modify the
129 \btool{btraceback} program to invoke the correct debugger, and possibly correct the
130 \bfilename{btraceback.gdb} script to have appropriate commands for your debugger. If
131 anyone succeeds in making this work with another debugger, please send us a
132 copy of what you modified. Please keep in mind that for any debugger to
133 work, it will most likely need to run as root, so you may need to modify
134 the \btool{btraceback} script accordingly.
135
136 \label{blb:ManuallyDebugging}
137 \section{Manually Running \bacula{} Under The Debugger}
138 \index[general]{Manually Running \bacula{} Under The Debugger}
139 \index[general]{Debugger!Manually Running \bacula{} Under The}
140
141 If for some reason you cannot get the automatic traceback, or if you want to
142 interactively examine the variable contents after a crash, you can run \bacula{}
143 under the debugger. Assuming you want to run the Storage daemon under the
144 debugger (the technique is the same for the other daemons, only the name
145 changes), you would do the following:
146
147 \begin{benumerate}
148 \item Start the Director and the File daemon. If the Storage daemon also
149 starts, you will need to find its PID as shown above (\btool{ps fax | grep bacula-sd})
150 and kill it with a command like the following:
151
152 \begin{bVerbatim}
153 kill -15 PID
154 \end{bVerbatim}
155
156 where you replace \textbf{PID} by the actual value.
157
158 \item At this point, the Director and the File daemon should be running but
159 the Storage daemon should not.
160
161 \item \btool{cd} to the directory containing the Storage daemon
162
163 \item Start the Storage daemon under the debugger:
164
165 \begin{bVerbatim}
166 gdb ./bacula-sd
167 \end{bVerbatim}
168
169 \item Run the Storage daemon:
170
171 \begin{bVerbatim}
172 run -s -f -c ./bacula-sd.conf
173 \end{bVerbatim}
174
175 You may replace the \bfilename{./bacula-sd.conf} with the full path to the Storage
176 daemon's configuration file.
177
178 \item At this point, \bacula{} will be fully operational.
179
180 \item In another shell command window, start the Console program and do what
181 is necessary to cause \bacula{} to die.
182
183 \item When \bacula{} crashes, the \btool{gdb} shell window will become active and
184 \btool{gdb} will show you the error that occurred.
185
186 \item To get a general traceback of all threads, issue the following command:
187
188 \begin{bVerbatim}
189 thread apply all bt
190 \end{bVerbatim}
191
192 After that you can issue any debugging command.
193 \end{benumerate}
194
195 \section{Getting Debug Output from \bacula{}}
196 \index[general]{Getting Debug Output from \bacula{} }
197 Each of the daemons normally has debug compiled into the program, but
198 disabled. There are two ways to enable the debug output. One is to add the
199 \textbf{-d nnn} option on the command line when starting the debugger.
200 The \textbf{nnn} is the debug level, and generally anything between 50 and 200 is
201 reasonable. The higher the number, the more output is produced. The output is
202 written to standard output.
203
204 The second way of getting debug output is to dynamically turn it on using the
205 Console using the \bcommandname{setdebug} command. The full syntax of the command is:
206
207 \begin{bVerbatim}
208 setdebug level=nnn client=client-name storage=storage-name dir
209 \end{bVerbatim}
210
211 If none of the options are given, the command will prompt you. You can
212 selectively turn on/off debugging in any or all the daemons (i.e. it is not
213 necessary to specify all the components of the above command).