| 1 |
\chapter{\bacula{} Developer Notes} |
| 2 |
\label{blb:TheChapterStart10} |
| 3 |
\index{\bacula{} Developer Notes} |
| 4 |
\index{Notes!\bacula{} Developer} |
| 5 |
|
| 6 |
This document is intended mostly for developers and describes how you can |
| 7 |
contribute to the \bacula{} project and the the general framework of making |
| 8 |
\bacula{} source changes. |
| 9 |
|
| 10 |
\section{Contributions} |
| 11 |
\index{Contributions} |
| 12 |
|
| 13 |
Contributions to the \bacula{} project come in many forms: ideas, |
| 14 |
participation in helping people on the bacula-users email list, packaging |
| 15 |
\bacula{} binaries for the community, helping improve the documentation, and |
| 16 |
submitting code. |
| 17 |
|
| 18 |
Contributions in the form of submissions for inclusion in the project are |
| 19 |
broken into two groups. The first are contributions that are aids and not |
| 20 |
essential to \bacula{}. In general, these will be scripts or will go into the |
| 21 |
\bdirectoryname{bacula/examples} directory. For these kinds of non-essential |
| 22 |
contributions there is no obligation to do a copyright assignment as |
| 23 |
described below. However, a copyright assignment would still be |
| 24 |
appreciated. |
| 25 |
|
| 26 |
The second class of contributions are those which will be integrated with |
| 27 |
\bacula{} and become an essential part (code, scripts, documentation, \ldots{}) |
| 28 |
Within this class of contributions, there are two hurdles to surmount. One |
| 29 |
is getting your patch accepted, and two is dealing with copyright issues. |
| 30 |
The following text describes some of the requirements for such code. |
| 31 |
|
| 32 |
\section{Patches} |
| 33 |
\index{Patches} |
| 34 |
|
| 35 |
Subject to the copyright assignment described below, your patches should be |
| 36 |
sent in \btool{git format-patch} format relative to the current contents of the |
| 37 |
master branch of the Source Forge \btool{git} repository. Please attach the |
| 38 |
output file or files generated by the \btool{git format-patch} to the email |
| 39 |
rather than include them directory to avoid wrapping of the lines |
| 40 |
in the patch. Please be sure to use the \bacula{} |
| 41 |
indenting standard (see below) for source code. If you have checked out |
| 42 |
the source with \btool{git}, you can get a \bcommandname{diff} using. |
| 43 |
|
| 44 |
\begin{bVerbatim} |
| 45 |
git pull |
| 46 |
git format-patch -M |
| 47 |
\end{bVerbatim} |
| 48 |
|
| 49 |
If you plan on doing significant development work over a period of time, |
| 50 |
after having your first patch reviewed and approved, you will be eligible |
| 51 |
for having developer \btool{git} write access so that you can commit your changes |
| 52 |
directly to the \btool{git} repository. To do so, you will need a userid on Source |
| 53 |
Forge. |
| 54 |
|
| 55 |
\section{Copyrights} |
| 56 |
\index{Copyrights} |
| 57 |
|
| 58 |
To avoid future problems concerning changing licensing or |
| 59 |
copyrights, all code contributions more than a hand full of lines |
| 60 |
must be in the Public Domain or have the copyright transferred to |
| 61 |
the Free Software Foundation Europe e.V. with a \ac{FLA} |
| 62 |
as the case for all the current code. |
| 63 |
|
| 64 |
Prior to November 2004, all the code was copyrighted by Kern Sibbald and |
| 65 |
John Walker. After November 2004, the code was copyrighted by Kern |
| 66 |
Sibbald, then on the 15th of November 2006, Kern transferred the copyright |
| 67 |
to the Free Software Foundation Europe e.V. In signing the \ac{FLA} and |
| 68 |
transferring the copyright, you retain the right to use the code you have |
| 69 |
submitted as you want, and you ensure that \bacula{} will always remain Free |
| 70 |
and Open Source. |
| 71 |
|
| 72 |
Your name should be clearly indicated as the author of the code, and you |
| 73 |
must be extremely careful not to violate any copyrights or patents or use |
| 74 |
other people's code without acknowledging it. The purpose of this |
| 75 |
requirement is to avoid future copyright, patent, or intellectual property |
| 76 |
problems. Please read the \bfilename{LICENSE} agreement in the main \bacula{} source code |
| 77 |
directory. When you sign the \acl{FLA} (\acs{FLA}) and send it |
| 78 |
in, you are agreeing to the terms of that \bfilename{LICENSE} file. |
| 79 |
|
| 80 |
If you don't understand what we mean by future problems, please |
| 81 |
examine the difficulties Mozilla was having finding |
| 82 |
previous contributors at \bref{http://www.mozilla.org/MPL/missing.html} |
| 83 |
{www.mozilla.org/MPL/missing.html}. The other important issue is to |
| 84 |
avoid copyright, patent, or intellectual property violations as was |
| 85 |
(May 2003) claimed by SCO against IBM. |
| 86 |
|
| 87 |
Although the copyright will be held by the Free Software |
| 88 |
Foundation Europe e.V., each developer is expected to indicate |
| 89 |
that he wrote and/or modified a particular module (or file) and |
| 90 |
any other sources. The copyright assignment may seem a bit |
| 91 |
unusual, but in reality, it is not. Most large projects require |
| 92 |
this. |
| 93 |
|
| 94 |
If you have any doubts about this, please don't hesitate to ask. The |
| 95 |
objective is to assure the long term survival of the \bacula{} project. |
| 96 |
|
| 97 |
Items not needing a copyright assignment are: most small changes, |
| 98 |
enhancements, or bug fixes of 5-10 lines of code, which amount to |
| 99 |
less than 20\% of any particular file. |
| 100 |
|
| 101 |
\section{Copyright Assignment -- Fiduciary License Agreement} |
| 102 |
\index{Copyright Assignment} |
| 103 |
\index{Assignment!Copyright} |
| 104 |
|
| 105 |
Since this is not a commercial enterprise, and we prefer to believe in |
| 106 |
everyone's good faith, previously developers could assign the copyright by |
| 107 |
explicitly acknowledging that they do so in their first submission. This |
| 108 |
was sufficient if the developer is independent, or an employee of a |
| 109 |
not-for-profit organization or a university. However, in an effort to |
| 110 |
ensure that the \bacula{} code is really clean, beginning in August 2006, all |
| 111 |
previous and future developers with \acs{SVN} write access will be asked to submit a |
| 112 |
copyright assignment (or \acl{FLA} -- \acs{FLA}), |
| 113 |
which means you agree to the \bfilename{LICENSE} in the main source |
| 114 |
directory. It also means that you receive back the right to use |
| 115 |
the code that you have submitted. |
| 116 |
|
| 117 |
Any developer who wants to contribute and is employed by a company should |
| 118 |
either list the employer as the owner of the code, or get explicit |
| 119 |
permission from him to sign the copyright assignment. This is because in |
| 120 |
many countries, all work that an employee does whether on company time or |
| 121 |
in the employee's free time is considered to be Intellectual Property of |
| 122 |
the company. Obtaining official approval or an \ac{FLA} from the company will |
| 123 |
avoid misunderstandings between the employee, the company, and the \bacula{} |
| 124 |
project. A good number of companies have already followed this procedure. |
| 125 |
|
| 126 |
The \acl{FLA} is posted on the \bacula{} web site at: |
| 127 |
\bref{http://www.bacula.org/en/FLA-bacula.en.pdf}{www.bacula.org/en/FLA-bacula.en.pdf} |
| 128 |
|
| 129 |
The instructions for filling out this agreement are also at: |
| 130 |
\bref{http://www.bacula.org/?page=fsfe}{www.bacula.org/?page=fsfe} |
| 131 |
|
| 132 |
It should be filled out, then sent to: |
| 133 |
|
| 134 |
\begin{bVerbatim} |
| 135 |
Kern Sibbald |
| 136 |
Cotes-de-Montmoiret 9 |
| 137 |
1012 Lausanne |
| 138 |
Switzerland |
| 139 |
\end{bVerbatim} |
| 140 |
|
| 141 |
Please note that the above address is different from the officially |
| 142 |
registered office mentioned in the document. When you send in such a |
| 143 |
complete document, please notify me: \bbracket{kern at sibbald dot com}, and |
| 144 |
please add your email address to the \ac{FLA} so that I can contact you |
| 145 |
to confirm reception of the signed \ac{FLA}. |
| 146 |
|
| 147 |
|
| 148 |
\section{The Development Cycle} |
| 149 |
\index{Developement Cycle} |
| 150 |
\index{Cycle!Developement} |
| 151 |
|
| 152 |
As discussed on the email lists, the number of contributions are |
| 153 |
increasing significantly. We expect this positive trend |
| 154 |
will continue. As a consequence, we have modified how we do |
| 155 |
development, and instead of making a list of all the features that we will |
| 156 |
implement in the next version, each developer signs up for one (maybe |
| 157 |
two) projects at a time, and when they are complete, and the code |
| 158 |
is stable, we will release a new version. The release cycle will probably |
| 159 |
be roughly six months. |
| 160 |
|
| 161 |
The difference is that with a shorter release cycle and fewer released |
| 162 |
feature, we will have more time to review the new code that is being |
| 163 |
contributed, and will be able to devote more time to a smaller number of |
| 164 |
projects (some prior versions had too many new features for us to handle |
| 165 |
correctly). |
| 166 |
|
| 167 |
Future release schedules will be much the same, and the |
| 168 |
number of new features will also be much the same providing that the |
| 169 |
contributions continue to come -- and they show no signs of let up :-) |
| 170 |
|
| 171 |
\index{Feature Requests} |
| 172 |
\subsection{Feature Requests} |
| 173 |
In addition, we have ``formalizee'' the feature requests a bit. |
| 174 |
|
| 175 |
Instead of me maintaining an informal list of everything I run into |
| 176 |
(\bfilename{kernstodo}), we now maintain a ``formal'' list of projects. This |
| 177 |
means that all new feature requests, including those recently discussed on |
| 178 |
the email lists, must be formally submitted and approved. |
| 179 |
|
| 180 |
Formal submission of feature requests will take two forms: |
| 181 |
\begin{benumerate} |
| 182 |
\item non-mandatory, but highly recommended is to discuss proposed new features |
| 183 |
on the mailing list. |
| 184 |
\item Formal submission of an Feature Request in a special format. We'll |
| 185 |
give an example of this below, but you can also find it on the web site |
| 186 |
under Support \textrightarrow{} Feature Requests. Since it takes a bit of time to |
| 187 |
properly fill out a Feature Request form, you probably should check on the |
| 188 |
email list first. |
| 189 |
\end{benumerate} |
| 190 |
|
| 191 |
Once the Feature Request is received by the keeper of the projects list, it |
| 192 |
will be sent to the \bacula{} project manager (Kern), and he will either |
| 193 |
accept it (90\% of the time), send it back asking for clarification (10\% of |
| 194 |
the time), send it to the email list asking for opinions, or reject it |
| 195 |
(very few cases). |
| 196 |
|
| 197 |
If it is accepted, it will go in the \bfilename{projects} file (a simple \acs{ASCII} file) |
| 198 |
maintained in the main \bacula{} source directory. |
| 199 |
|
| 200 |
\subsubsection{Implementation of Feature Requests} |
| 201 |
Any qualified developer can sign up for a project. The project must have |
| 202 |
an entry in the projects file, and the developer's name will appear in the |
| 203 |
Status field. |
| 204 |
|
| 205 |
\subsubsection{How Feature Requests are accepted} |
| 206 |
Acceptance of Feature Requests depends on several things: |
| 207 |
\begin{benumerate} |
| 208 |
\item feedback from users. If it is negative, the Feature Request will probably not be accepted. \\ |
| 209 |
\item the difficulty of the project. A project that is so |
| 210 |
difficult that we cannot imagine finding someone to implement probably won't |
| 211 |
be accepted. Obviously if you know how to implement it, don't hesitate |
| 212 |
to put it in your Feature Request |
| 213 |
\item whether or not the Feature Request fits within the current strategy of |
| 214 |
\bacula{} (for example an Feature Request that requests changing the tape to |
| 215 |
\tar{} format probably would not be accepted, \ldots{}). |
| 216 |
\end{benumerate} |
| 217 |
|
| 218 |
\subsubsection{How Feature Requests are prioritized} |
| 219 |
Once an Feature Request is accepted, it needs to be implemented. If you |
| 220 |
can find a developer for it, or one signs up for implementing it, then the |
| 221 |
Feature Request becomes top priority (at least for that developer). |
| 222 |
|
| 223 |
Between releases of \bacula{}, we will generally solicit Feature Request input |
| 224 |
for the next version, and by way of this email, we suggest that you send |
| 225 |
discuss and send in your Feature Requests for the next release. Please |
| 226 |
verify that the Feature Request is not in the current list (attached to this email). |
| 227 |
|
| 228 |
Once users have had several weeks to submit Feature Requests, the keeper of |
| 229 |
the projects list will organize them, and request users to vote on them. |
| 230 |
This will allow fixing prioritizing the Feature Requests. Having a |
| 231 |
priority is one thing, but getting it implement is another thing -- we are |
| 232 |
hoping that the \bacula{} community will take more responsibility for assuring |
| 233 |
the implementation of accepted Feature Requests. |
| 234 |
|
| 235 |
Feature Request format: |
| 236 |
\begin{bVerbatim} |
| 237 |
============= Empty Feature Request form =========== |
| 238 |
Item n: One line summary ... |
| 239 |
Date: Date submitted |
| 240 |
Origin: Name and email of originator. |
| 241 |
Status: |
| 242 |
|
| 243 |
What: More detailed explanation ... |
| 244 |
|
| 245 |
Why: Why it is important ... |
| 246 |
|
| 247 |
Notes: Additional notes or features (omit if not used) |
| 248 |
============== End Feature Request form ============== |
| 249 |
\end{bVerbatim} |
| 250 |
|
| 251 |
\begin{bVerbatim} |
| 252 |
============= Example Completed Feature Request form =========== |
| 253 |
Item 1: Implement a Migration job type that will move the job |
| 254 |
data from one device to another. |
| 255 |
Origin: Sponsored by Riege Sofware International GmbH. Contact: |
| 256 |
Daniel Holtkamp <holtkamp at riege dot com> |
| 257 |
Date: 28 October 2005 |
| 258 |
Status: Partially coded in 1.37 -- much more to do. Assigned to |
| 259 |
Kern. |
| 260 |
|
| 261 |
What: The ability to copy, move, or archive data that is on a |
| 262 |
device to another device is very important. |
| 263 |
|
| 264 |
Why: An ISP might want to backup to disk, but after 30 days |
| 265 |
migrate the data to tape backup and delete it from |
| 266 |
disk. Bacula should be able to handle this |
| 267 |
automatically. It needs to know what was put where, |
| 268 |
and when, and what to migrate -- it is a bit like |
| 269 |
retention periods. Doing so would allow space to be |
| 270 |
freed up for current backups while maintaining older |
| 271 |
data on tape drives. |
| 272 |
|
| 273 |
Notes: Migration could be triggered by: |
| 274 |
Number of Jobs |
| 275 |
Number of Volumes |
| 276 |
Age of Jobs |
| 277 |
Highwater size (keep total size) |
| 278 |
Lowwater mark |
| 279 |
================================================= |
| 280 |
\end{bVerbatim} |
| 281 |
|
| 282 |
|
| 283 |
\section{\bacula{} Code Submissions and Projects} |
| 284 |
\index{Submissions and Projects} |
| 285 |
|
| 286 |
Getting code implemented in \bacula{} works roughly as follows: |
| 287 |
|
| 288 |
\begin{bitemize} |
| 289 |
|
| 290 |
\item Kern is the project manager, but prefers not to be a ``gate keeper''. |
| 291 |
This means that the developers are expected to be self-motivated, |
| 292 |
and once they have experience submit directly to the \btool{git} |
| 293 |
repositories. However, |
| 294 |
it is a good idea to have your patches reviewed prior to submitting, |
| 295 |
and it is a bad idea to submit monster patches because no one will |
| 296 |
be able to properly review them. See below for more details on this. |
| 297 |
|
| 298 |
\item There are growing numbers of contributions (very good). |
| 299 |
|
| 300 |
\item Some contributions come in the form of relatively small patches, |
| 301 |
which Kern reviews, integrates, documents, tests, and maintains. |
| 302 |
|
| 303 |
\item All \bacula{} developers take full |
| 304 |
responsibility for writing the code, posting as patches so that we can |
| 305 |
review it as time permits, integrating it at an appropriate time, |
| 306 |
responding to our requests for tweaking it (name changes, \ldots{}), |
| 307 |
document it in the code, document it in the manual (even though |
| 308 |
their mother tongue is not English), test it, develop and commit |
| 309 |
regression scripts, and answer in a timely fashion all bug reports -- |
| 310 |
even occasionally accepting additional bugs :-) |
| 311 |
|
| 312 |
This is a sustainable way of going forward with \bacula{}, and the |
| 313 |
direction that the project will be taking more and more. For |
| 314 |
example, in the past, we have had some very dedicated programmers |
| 315 |
who did major projects. However, some of these |
| 316 |
programmers due to outside obligations (job responsibilities change of |
| 317 |
job, school duties, \ldots{}) could not continue to maintain the code. In |
| 318 |
those cases, the code suffers from lack of maintenance, sometimes we |
| 319 |
patch it, sometimes not. In the end, if the code is not maintained, the |
| 320 |
code gets dropped from the project (there are two such contributions |
| 321 |
that are heading in that direction). When ever possible, we would like |
| 322 |
to avoid this, and ensure a continuation of the code and a sharing of |
| 323 |
the development, debugging, documentation, and maintenance |
| 324 |
responsibilities. |
| 325 |
\end{bitemize} |
| 326 |
|
| 327 |
\section{Patches for Released Versions} |
| 328 |
\index{Patches for Released Versions} |
| 329 |
If you fix a bug in a released version, you should, unless it is |
| 330 |
an absolutely trivial bug, create and release a patch file for the |
| 331 |
bug. The procedure is as follows: |
| 332 |
|
| 333 |
Fix the bug in the released branch and in the develpment master branch. |
| 334 |
|
| 335 |
Make a patch file for the branch and add the branch patch to |
| 336 |
the patches directory in both the branch and the trunk. |
| 337 |
The name should be \texttt{2.2.4-xxx.patch} where \texttt{xxx} is unique, in this case it can |
| 338 |
be ``restore'', e.g. \texttt{2.2.4-restore.patch}. Add to the top of the |
| 339 |
file a brief description and instructions for applying it -- see for example |
| 340 |
\texttt{2.2.4-poll-mount.patch}. The best way to create the patch file is as |
| 341 |
follows: |
| 342 |
|
| 343 |
\begin{bVerbatim} |
| 344 |
(edit) 2.2.4-restore.patch |
| 345 |
(input description) |
| 346 |
(end edit) |
| 347 |
|
| 348 |
git format-patch -M |
| 349 |
mv 0001-xxx 2.2.4-restore.patch |
| 350 |
\end{bVerbatim} |
| 351 |
|
| 352 |
check to make sure no extra junk got put into the patch file (i.e. |
| 353 |
it should have the patch for that bug only). |
| 354 |
|
| 355 |
If there is not a bug report on the problem, create one, then add the |
| 356 |
patch to the bug report. |
| 357 |
|
| 358 |
Then upload it to the 2.2.x release of bacula-patches. |
| 359 |
|
| 360 |
So, end the end, the patch file is: |
| 361 |
\begin{bitemize} |
| 362 |
\item Attached to the bug report |
| 363 |
|
| 364 |
\item In \bfilename{Branch-2.2/bacula/patches/...} |
| 365 |
|
| 366 |
\item In the trunk |
| 367 |
|
| 368 |
\item Loaded on Source Forge bacula-patches 2.2.x release. When |
| 369 |
you add it, click on the check box to send an Email so that all the |
| 370 |
users that are monitoring SF patches get notified. |
| 371 |
\end{bitemize} |
| 372 |
|
| 373 |
|
| 374 |
\section{Developing \bacula{}} |
| 375 |
\index{Developing \bacula{}} |
| 376 |
\index{\bacula{}!Developing} |
| 377 |
|
| 378 |
Typically the simplest way to develop \bacula{} is to open one \texttt{xterm} window |
| 379 |
pointing to the source directory you wish to update; a second \texttt{xterm} window at |
| 380 |
the top source directory level, and a third xterm window at the bacula |
| 381 |
directory \bdirectoryname{\bbracket{top}/src/bacula}. After making source changes in one of the |
| 382 |
directories, in the top source directory \texttt{xterm}, build the source, and start |
| 383 |
the daemons by entering: |
| 384 |
|
| 385 |
\begin{bVerbatim} |
| 386 |
make |
| 387 |
\end{bVerbatim} |
| 388 |
and |
| 389 |
\begin{bVerbatim} |
| 390 |
./startit |
| 391 |
\end{bVerbatim} |
| 392 |
then in the enter: |
| 393 |
\begin{bVerbatim} |
| 394 |
./console |
| 395 |
\end{bVerbatim} |
| 396 |
or |
| 397 |
\begin{bVerbatim} |
| 398 |
./gnome-console |
| 399 |
\end{bVerbatim} |
| 400 |
to start the Console program. Enter any commands for testing. |
| 401 |
For example: \bcommandname{run kernsverify full}. |
| 402 |
|
| 403 |
Note, the instructions here to use \btool{./startit} are different from using a |
| 404 |
production system where the administrator starts \bacula{} by entering |
| 405 |
\btool{./bacula start}. This difference allows a development version of {\bf \bacula{}} |
| 406 |
to be run on a computer at the same time that a production system is running. |
| 407 |
The \btool{./startit} script starts \textbf{\bacula{}} using a different set of |
| 408 |
configuration files, and thus permits avoiding conflicts with any production |
| 409 |
system. |
| 410 |
|
| 411 |
To make additional source changes, exit from the Console program, and in the |
| 412 |
top source directory, stop the daemons by entering: |
| 413 |
|
| 414 |
\begin{bVerbatim} |
| 415 |
./stopit |
| 416 |
\end{bVerbatim} |
| 417 |
then repeat the process. |
| 418 |
|
| 419 |
\section{Debugging} |
| 420 |
\index{Debugging} |
| 421 |
|
| 422 |
Probably the first thing to do is to turn on debug output. |
| 423 |
|
| 424 |
A good place to start is with a debug level of 20 as in \btool{./startit -d20}. |
| 425 |
The \btool{startit} command starts all the daemons with the same debug level. |
| 426 |
Alternatively, you can start the appropriate daemon with the debug level you |
| 427 |
want. If you really need more info, a debug level of 60 is not bad, and for |
| 428 |
just about everything a level of 200. |
| 429 |
|
| 430 |
\section{Using a Debugger} |
| 431 |
\index{Using a Debugger} |
| 432 |
\index{Debugger!Using a} |
| 433 |
|
| 434 |
If you have a serious problem such as a segmentation fault, it can usually be |
| 435 |
found quickly using a good multiple thread debugger such as \btool{gdb}. For |
| 436 |
example, suppose you get a segmentation violation in \btool{bacula-dir}. You |
| 437 |
might use the following to find the problem: |
| 438 |
|
| 439 |
\bbracket{start the Storage and File daemons} |
| 440 |
\begin{bVerbatim} |
| 441 |
cd dird |
| 442 |
gdb ./bacula-dir |
| 443 |
run -f -s -c ./dird.conf |
| 444 |
\end{bVerbatim} |
| 445 |
\bbracket{it dies with a segmentation fault} |
| 446 |
where |
| 447 |
The \texttt{-f} option is specified on the \bcommandname{run} command to inhibit |
| 448 |
\btool{dird} from going into the background. You may also want to add the \texttt{-s} |
| 449 |
option to the run command to disable signals which can potentially interfere |
| 450 |
with the debugging. |
| 451 |
|
| 452 |
As an alternative to using the debugger, each {\bf \bacula{}} daemon has a built |
| 453 |
in back trace feature when a serious error is encountered. It calls the |
| 454 |
debugger on itself, produces a back trace, and emails the report to the |
| 455 |
developer. For more details on this, please see the chapter in the main \bacula{} |
| 456 |
manual entitled ``What To Do When \bacula{} Crashes (Kaboom)''. |
| 457 |
|
| 458 |
\section{Memory Leaks} |
| 459 |
\index{Leaks!Memory} |
| 460 |
\index{Memory Leaks} |
| 461 |
|
| 462 |
Because \bacula{} runs routinely and unattended on client and server machines, it |
| 463 |
may run for a long time. As a consequence, from the very beginning, \bacula{} |
| 464 |
uses \texttt{SmartAlloc} to ensure that there are no memory leaks. To make detection of |
| 465 |
memory leaks effective, all \bacula{} code that dynamically allocates memory \textbf{must} |
| 466 |
have a way to release it. In general when the memory is no longer needed, it |
| 467 |
should be immediately released, but in some cases, the memory will be held |
| 468 |
during the entire time that \bacula{} is executing. In that case, there \textbf{must} be a |
| 469 |
routine that can be called at termination time that releases the memory. In |
| 470 |
this way, we will be able to detect memory leaks. Be sure to immediately |
| 471 |
correct any and all memory leaks that are printed at the termination of the |
| 472 |
daemons. |
| 473 |
|
| 474 |
\section{Special Files} |
| 475 |
\index{Files!Special} |
| 476 |
\index{Special Files} |
| 477 |
|
| 478 |
Kern uses files named 1, 2, \ldots{} 9 with any extension as scratch files. Thus |
| 479 |
any files with these names are subject to being rudely deleted at any time. |
| 480 |
|
| 481 |
\section{When Implementing Incomplete Code} |
| 482 |
\index{Code!When Implementing Incomplete} |
| 483 |
\index{When Implementing Incomplete Code} |
| 484 |
|
| 485 |
Please identify all incomplete code with a comment that contains |
| 486 |
|
| 487 |
\begin{bVerbatim} |
| 488 |
***FIXME*** |
| 489 |
\end{bVerbatim} |
| 490 |
|
| 491 |
where there are three asterisks (*) before and after the word |
| 492 |
FIXME (in capitals) and no intervening spaces. This is important as it allows |
| 493 |
new programmers to easily recognize where things are partially implemented. |
| 494 |
|
| 495 |
\subsection{\bacula{} Source File Structure} |
| 496 |
\index{Structure!\bacula{} Source File} |
| 497 |
\index{\bacula{} Source File Structure} |
| 498 |
|
| 499 |
The distribution generally comes as a \tar{} file of the form |
| 500 |
\bfilename{bacula.x.y.z.tar.gz} where x, y, and z are the version, release, and update |
| 501 |
numbers respectively. |
| 502 |
|
| 503 |
Once you detar this file, you will have a directory structure as follows: |
| 504 |
|
| 505 |
\begin{bVerbatim} |
| 506 |
| |
| 507 |
Tar file: |
| 508 |
|- depkgs |
| 509 |
|- mtx (autochanger control program + tape drive info) |
| 510 |
|- sqlite (SQLite database program) |
| 511 |
|
| 512 |
Tar file: |
| 513 |
|- depkgs-win32 |
| 514 |
|- pthreads (Native win32 pthreads library -- dll) |
| 515 |
|- zlib (Native win32 zlib library) |
| 516 |
|- wx (wxWidgets source code) |
| 517 |
|
| 518 |
Project bacula: |
| 519 |
|- bacula (main source directory containing configuration |
| 520 |
| and installation files) |
| 521 |
|- autoconf (automatic configuration files, not normally used |
| 522 |
| by users) |
| 523 |
|- intl (programs used to translate) |
| 524 |
|- platforms (OS specific installation files) |
| 525 |
|- redhat (Red Hat installation) |
| 526 |
|- solaris (Sun installation) |
| 527 |
|- freebsd (FreeBSD installation) |
| 528 |
|- irix (Irix installation -- not tested) |
| 529 |
|- unknown (Default if system not identified) |
| 530 |
|- po (translations of source strings) |
| 531 |
|- src (source directory; contains global header files) |
| 532 |
|- plugins (plugins for FD/SD/DIR) |
| 533 |
|-fd (FileDaemon plugins) |
| 534 |
|-sd (Storage Daemon plugins) |
| 535 |
|-dir (Director Daemon plugins) |
| 536 |
|- cats (SQL catalog database interface directory) |
| 537 |
|- console (bacula user agent directory) |
| 538 |
|- dird (Director daemon) |
| 539 |
|- filed (Unix File daemon) |
| 540 |
|- findlib (Unix file find library for File daemon) |
| 541 |
|- lib (General Bacula library) |
| 542 |
|- stored (Storage daemon) |
| 543 |
|- tools (Various tool programs) |
| 544 |
|- win32 (Native Win32 File daemon) |
| 545 |
|- libwin32 (Win32 files to make bacula-fd be a service) |
| 546 |
|- compat (compatibility interface library) |
| 547 |
|- filed (links to src/filed) |
| 548 |
|- findlib (links to src/findlib) |
| 549 |
|- lib (links to src/lib) |
| 550 |
|- console (beginning of native console program) |
| 551 |
|- wx-console (wxWidget console Win32 specific parts) |
| 552 |
|- win32-installer (Makensis installer for 32bit) |
| 553 |
|- win64-installer (Makensis installer for 64bit) |
| 554 |
|
| 555 |
|
| 556 |
Project regress: |
| 557 |
|- regress (Regression scripts) |
| 558 |
|- bin (temporary directory to hold Bacula installed binaries) |
| 559 |
|- build (temporary directory to hold Bacula source) |
| 560 |
|- scripts (scripts and .conf files) |
| 561 |
|- tests (test scripts) |
| 562 |
|- tmp (temporary directory for temp files) |
| 563 |
|- working (temporary working directory for Bacula daemons) |
| 564 |
|
| 565 |
Project docs: |
| 566 |
|- docs (documentation directory) |
| 567 |
|- developers (Developer's guide) |
| 568 |
|- home-page (Bacula's home page source) |
| 569 |
|- manual (html document directory) |
| 570 |
|- manual-fr (French translation) |
| 571 |
|- manual-de (German translation) |
| 572 |
|- techlogs (Technical development notes); |
| 573 |
|
| 574 |
Project gui: |
| 575 |
|- gui (Bacula GUI projects) |
| 576 |
|- bacula-web (Bacula web php management code) |
| 577 |
|- bimagemgr (Web application for burning CDROMs) |
| 578 |
\end{bVerbatim} |
| 579 |
|
| 580 |
\section{Header Files} |
| 581 |
\index{Header Files} |
| 582 |
\index{Files!Header} |
| 583 |
|
| 584 |
Please carefully follow the scheme defined below as it permits in general only |
| 585 |
two header file includes per \texttt{C} file, and thus vastly simplifies programming. |
| 586 |
With a large complex project like \bacula{}, it isn't always easy to ensure that |
| 587 |
the right headers are invoked in the right order (there are a few kludges to |
| 588 |
make this happen -- i.e. in a few include files because of the chicken and egg |
| 589 |
problem, certain references to typedefs had to be replaced with \textbf{void}). |
| 590 |
|
| 591 |
Every file should include \bfilename{bacula.h}. It pulls in just about everything, |
| 592 |
with very few exceptions. If you have system dependent ifdefing, please do it |
| 593 |
in \bfilename{baconfig.h}. The version number and date are kept in \bfilename{version.h}. |
| 594 |
|
| 595 |
Each of the subdirectories (\bdirectoryname{console}, \bdirectoryname{cats}, |
| 596 |
\bdirectoryname{dird}, \bfilename{filed}, \bfilename{findlib}, \bfilename{lib}, |
| 597 |
\bfilename{stored}, \ldots{}) contains a single directory dependent include file |
| 598 |
generally the name of the directory, which should be included just after the include of |
| 599 |
\bfilename{bacula.h}. This file (for example, for the dird directory, it is \bfilename{dird.h}) |
| 600 |
contains either definitions of things generally needed in this directory, or |
| 601 |
it includes the appropriate header files. It always includes \bfilename{protos.h}. |
| 602 |
See below. |
| 603 |
|
| 604 |
Each subdirectory contains a header file named \bfilename{protos.h}, which contains |
| 605 |
the prototypes for subroutines exported by files in that directory. |
| 606 |
\bfilename{protos.h} is always included by the main directory dependent include file. |
| 607 |
|
| 608 |
\section{Programming Standards} |
| 609 |
\index{Standards!Programming} |
| 610 |
\index{Programming Standards} |
| 611 |
|
| 612 |
For the most part, all code should be written in \texttt{C} unless there is a burning |
| 613 |
reason to use \texttt{C++}, and then only the simplest \texttt{C++} constructs will be used. |
| 614 |
Note, \bacula{} is slowly evolving to use more and more \texttt{C++}. |
| 615 |
|
| 616 |
Code should have some documentation -- not a lot, but enough so that I can |
| 617 |
understand it. Look at the current code, and you will see that I document more |
| 618 |
than most, but am definitely not a fanatic. |
| 619 |
|
| 620 |
We prefer simple linear code where possible. Gotos are strongly discouraged |
| 621 |
except for handling an error to either bail out or to retry some code, and |
| 622 |
such use of gotos can vastly simplify the program. |
| 623 |
|
| 624 |
Remember this is a \texttt{C} program that is migrating to a \textbf{tiny} subset of \texttt{C++}, |
| 625 |
so be conservative in your use of \texttt{C++} features. |
| 626 |
|
| 627 |
\section{Do Not Use} |
| 628 |
\index{Use!Do Not} |
| 629 |
\index{Do Not Use} |
| 630 |
|
| 631 |
\begin{bitemize} |
| 632 |
\item STL -- it is totally incomprehensible. |
| 633 |
\end{bitemize} |
| 634 |
|
| 635 |
\section{Avoid if Possible} |
| 636 |
\index{Possible!Avoid if} |
| 637 |
\index{Avoid if Possible} |
| 638 |
|
| 639 |
\begin{bitemize} |
| 640 |
\item Using \textbf{void *} because this generally means that one must |
| 641 |
using casting, and in \texttt{C++} casting is rather ugly. It is OK to use |
| 642 |
\texttt{void *} to pass structure address where the structure is not known |
| 643 |
to the routines accepting the packet (typically callback routines). |
| 644 |
However, declaring \texttt{void *buf} is a bad idea. Please use the |
| 645 |
correct types whenever possible. |
| 646 |
|
| 647 |
\item Using undefined storage specifications such as (\texttt{short}, \texttt{int}, |
| 648 |
\texttt{long}, \texttt{long long}, \texttt{size\_t \ldots{}}). |
| 649 |
The problem with all these is that the number of bytes |
| 650 |
they allocate depends on the compiler and the system. Instead use |
| 651 |
\bacula{}'s types (\texttt{int8\_t}, \texttt{uint8\_t}, \texttt{int32\_t}, |
| 652 |
\texttt{uint32\_t}, \texttt{int64\_t}, and \texttt{uint64\_t}). |
| 653 |
This guarantees that the variables are given exactly the |
| 654 |
size you want. Please try at all possible to avoid using \texttt{size\_t ssize\_t} |
| 655 |
and the such. They are very system dependent. However, some system |
| 656 |
routines may need them, so their use is often unavoidable. |
| 657 |
|
| 658 |
\item Returning a malloc'ed buffer from a subroutine -- someone will forget |
| 659 |
to release it. |
| 660 |
|
| 661 |
\item Heap allocation (malloc) unless needed -- it is expensive. Use |
| 662 |
\texttt{POOL\_MEM} instead. |
| 663 |
|
| 664 |
\item Templates -- they can create portability problems. |
| 665 |
|
| 666 |
\item Fancy or tricky \texttt{C} or \texttt{C++} code, unless you give a good explanation of |
| 667 |
why you used it. |
| 668 |
|
| 669 |
\item Too much inheritance -- it can complicate the code, and make reading it |
| 670 |
difficult (unless you are in love with colons) |
| 671 |
\end{bitemize} |
| 672 |
|
| 673 |
\section{Do Use Whenever Possible} |
| 674 |
\index{Possible!Do Use Whenever} |
| 675 |
\index{Do Use Whenever Possible} |
| 676 |
|
| 677 |
\begin{bitemize} |
| 678 |
\item Locking and unlocking within a single subroutine. |
| 679 |
|
| 680 |
\item A single point of exit from all subroutines. A goto is |
| 681 |
perfectly OK to use to get out early, but only to a label |
| 682 |
named \texttt{bail\_out}, and possibly an \texttt{ok\_out}. See current code |
| 683 |
examples. |
| 684 |
|
| 685 |
\item \texttt{malloc} and \texttt{free} within a single subroutine. |
| 686 |
|
| 687 |
\item Comments and global explanations on what your code or algorithm does. |
| 688 |
|
| 689 |
\item When committing a fix for a bug, make the comment of the |
| 690 |
following form: |
| 691 |
|
| 692 |
\begin{bVerbatim} |
| 693 |
Reason for bug fix or other message. Fixes bug #1234 |
| 694 |
\end{bVerbatim} |
| 695 |
|
| 696 |
It is important to write the \textbf{ bug \#1234} like |
| 697 |
that because our program that automatically pulls messages |
| 698 |
from the \btool{git} repository to make the ChangeLog looks for that |
| 699 |
pattern. Obviously the \textbf{1234} should be replaced with |
| 700 |
the number of the bug you actually fixed. |
| 701 |
|
| 702 |
Providing the commit comment line has one of the following |
| 703 |
keywords (or phrases), it will be ignored: |
| 704 |
|
| 705 |
\begin{bVerbatim} |
| 706 |
tweak |
| 707 |
typo |
| 708 |
cleanup |
| 709 |
bweb: |
| 710 |
regress: |
| 711 |
again |
| 712 |
.gitignore |
| 713 |
fix compilation |
| 714 |
technotes |
| 715 |
update version |
| 716 |
update technotes |
| 717 |
update kernstodo |
| 718 |
update projects |
| 719 |
update releasenotes |
| 720 |
update version |
| 721 |
update home |
| 722 |
update release |
| 723 |
update todo |
| 724 |
update notes |
| 725 |
update changelog |
| 726 |
\end{bVerbatim} |
| 727 |
|
| 728 |
\item Use the following keywords at the beginning of a \btool{git} commit message |
| 729 |
\end{bitemize} |
| 730 |
|
| 731 |
\section{Indenting Standards} |
| 732 |
\index{Standards!Indenting} |
| 733 |
\index{Indenting Standards} |
| 734 |
|
| 735 |
We find it very hard to read code indented 8 columns at a time. |
| 736 |
Even 4 at a time uses a lot of space, so we have adopted indenting |
| 737 |
3 spaces at every level. Note, indention is the visual appearance of the |
| 738 |
source on the page, while tabbing is replacing a series of up to 8 spaces from |
| 739 |
a tab character. |
| 740 |
|
| 741 |
The closest set of parameters for the Linux \textbf{indent} program that will |
| 742 |
produce reasonably indented code are: |
| 743 |
|
| 744 |
\begin{bVerbatim} |
| 745 |
-nbad -bap -bbo -nbc -br -brs -c36 -cd36 -ncdb -ce -ci3 -cli0 |
| 746 |
-cp36 -d0 -di1 -ndj -nfc1 -nfca -hnl -i3 -ip0 -l85 -lp -npcs |
| 747 |
-nprs -npsl -saf -sai -saw -nsob -nss -nbc -ncs -nbfda |
| 748 |
\end{bVerbatim} |
| 749 |
|
| 750 |
You can put the above in your \bfilename{.indent.pro} file, and then just invoke indent on |
| 751 |
your file. However, be warned. This does not produce perfect indenting, and it |
| 752 |
will mess up \texttt{C++} class statements pretty badly. |
| 753 |
|
| 754 |
Braces are required in all if statements (missing in some very old code). To |
| 755 |
avoid generating too many lines, the first brace appears on the first line |
| 756 |
(e.g. of an \texttt{if}), and the closing brace is on a line by itself. E.g. |
| 757 |
|
| 758 |
\begin{bVerbatim} |
| 759 |
if (abc) { |
| 760 |
some_code; |
| 761 |
} |
| 762 |
\end{bVerbatim} |
| 763 |
|
| 764 |
Just follow the convention in the code. For example we I prefer non-indented cases. |
| 765 |
|
| 766 |
\begin{bVerbatim} |
| 767 |
switch (code) { |
| 768 |
case 'A': |
| 769 |
do something |
| 770 |
break; |
| 771 |
case 'B': |
| 772 |
again(); |
| 773 |
break; |
| 774 |
default: |
| 775 |
break; |
| 776 |
} |
| 777 |
\end{bVerbatim} |
| 778 |
|
| 779 |
Avoid using \texttt{//} style comments except for temporary code or turning off debug |
| 780 |
code. Standard \texttt{C} comments are preferred (this also keeps the code closer to \texttt{C}). |
| 781 |
|
| 782 |
Attempt to keep all lines less than 85 characters long so that the whole line |
| 783 |
of code is readable at one time. This is not a rigid requirement. |
| 784 |
|
| 785 |
Always put a brief description at the top of any new file created describing |
| 786 |
what it does and including your name and the date it was first written. Please |
| 787 |
don't forget any Copyrights and acknowledgments if it isn't 100\% your code. |
| 788 |
Also, include the \bacula{} copyright notice that is in \bdirectoryname{src/c}. |
| 789 |
|
| 790 |
In general you should have two includes at the top of the an include for the |
| 791 |
particular directory the code is in, for includes are needed, but this should |
| 792 |
be rare. |
| 793 |
|
| 794 |
In general (except for self-contained packages), prototypes should all be put |
| 795 |
in \bfilename{protos.h} in each directory. |
| 796 |
|
| 797 |
Always put space around assignment and comparison operators. |
| 798 |
|
| 799 |
\begin{bVerbatim} |
| 800 |
a = 1; |
| 801 |
if (b >= 2) { |
| 802 |
cleanup(); |
| 803 |
} |
| 804 |
\end{bVerbatim} |
| 805 |
|
| 806 |
but your can compress things in a \textbf{for} statement: |
| 807 |
|
| 808 |
\begin{bVerbatim} |
| 809 |
for (i=0; i < del.num_ids; i++) { |
| 810 |
... |
| 811 |
\end{bVerbatim} |
| 812 |
|
| 813 |
Don't overuse the inline if (\texttt{?:}). A full \texttt{if} is preferred, except in a |
| 814 |
print statement, e.g.: |
| 815 |
|
| 816 |
\begin{bVerbatim} |
| 817 |
if (ua->verbose && del.num_del != 0) { |
| 818 |
bsendmsg(ua, _("Pruned %d %s on Volume %s from catalog.\n"), del.num_del, |
| 819 |
del.num_del == 1 ? "Job" : "Jobs", mr->VolumeName); |
| 820 |
} |
| 821 |
\end{bVerbatim} |
| 822 |
|
| 823 |
Leave a certain amount of debug code (Dmsg) in code you submit, so that future |
| 824 |
problems can be identified. This is particularly true for complicated code |
| 825 |
likely to break. However, try to keep the debug code to a minimum to avoid |
| 826 |
bloating the program and above all to keep the code readable. |
| 827 |
|
| 828 |
Please keep the same style in all new code you develop. If you include code |
| 829 |
previously written, you have the option of leaving it with the old indenting |
| 830 |
or re-indenting it. If the old code is indented with 8 spaces, then please |
| 831 |
re-indent it to \bacula{} standards. |
| 832 |
|
| 833 |
If you are using \btool{vim}, simply set your tabstop to 8 and your shiftwidth |
| 834 |
to 3. |
| 835 |
|
| 836 |
\section{Naming Convention} |
| 837 |
|
| 838 |
A fairly big amount of functions and variable are using the ``snake\_case'' |
| 839 |
format, although you may find some classes that use the ``CamelFormat''. |
| 840 |
|
| 841 |
\smallskip |
| 842 |
|
| 843 |
We strongly recommend to stick to the ``snake\_case'' format to ease the |
| 844 |
reading of the code flow. Switching from one to the other is difficult |
| 845 |
and requires extra brain power. |
| 846 |
|
| 847 |
\smallskip |
| 848 |
|
| 849 |
Functions associated with a concept or a module and be prefixed with the name |
| 850 |
of the concept. For example, all Bacula Virtual FileSystem (bvfs) functions |
| 851 |
are prefixed with \texttt{bvfs\_}. |
| 852 |
|
| 853 |
\smallskip |
| 854 |
|
| 855 |
Function and variable names must be readable, in general it is a bad idea to |
| 856 |
use striped down version of a function or a variable. In general you will find |
| 857 |
the following variables, functions or label in the code: |
| 858 |
|
| 859 |
\begin{itemize} |
| 860 |
\item \texttt{ret} - int/bool return code for a function |
| 861 |
\item \texttt{buf} - buffer |
| 862 |
\item \texttt{bail\_out} - cleanup label for a function |
| 863 |
\item \texttt{ed1, ed2, ed3} - temporary buffer used to convert integers (50c |
| 864 |
long) |
| 865 |
\item \texttt{len} - length of a buffer |
| 866 |
\item \texttt{src} - source |
| 867 |
\item \texttt{dest} - destination |
| 868 |
\item \texttt{argc} - argument count |
| 869 |
\item \texttt{argk} - argument key (key=val) |
| 870 |
\item \texttt{argv} - argument value (key=val) |
| 871 |
\item \texttt{ua} - User Agent object |
| 872 |
\item \texttt{bvfs} - Bacula Virtual File System |
| 873 |
\item \texttt{lmgr} - Lock Manager |
| 874 |
\item \texttt{DCR} - Device Control Resource |
| 875 |
\item \texttt{ctx} - Context |
| 876 |
\end{itemize} |
| 877 |
|
| 878 |
\section{Locks and Threads} |
| 879 |
|
| 880 |
Bacula has a builtin lock manager called \texttt{lmgr}. This lock manager is a |
| 881 |
wrapper for common pthread or mutex operation. The lock manager will overwrite |
| 882 |
the different POSIX thread functions via \texttt{src/lockmgr.h}. |
| 883 |
|
| 884 |
\smallskip{} |
| 885 |
It can detect deadlock situation during the run time. \texttt{Found a deadlock !!!!} |
| 886 |
|
| 887 |
\smallskip{} |
| 888 |
|
| 889 |
The lock manager can prevent deadlock and help developers to design with the mutex |
| 890 |
list (\texttt{lib/mutex\_list.h}). If all mutex are acquired/released with some predefined |
| 891 |
order, deadlocks are not possible. |
| 892 |
|
| 893 |
\begin{bVerbatim} |
| 894 |
ERROR: V out of order lock=%p %s:%i dumping lock |
| 895 |
\end{bVerbatim} |
| 896 |
|
| 897 |
The lock manager will also dump all the mutex map during a backtrace. It can be analyzed easily |
| 898 |
to find the incorrect lock path. In the following example, we can see that one thread (0x7f67abe5f700) |
| 899 |
has requested to lock the same lock (0x6a8b40) two times from two different location. |
| 900 |
|
| 901 |
\begin{bVerbatim} |
| 902 |
threadid=0x7f67abe5f700 max=4 current=1 |
| 903 |
lock=0x6a8b40 state=Granted priority=0 jobq.c:476 |
| 904 |
lock=0x6a8b40 state=Wanted priority=0 jobq.c:324 |
| 905 |
\end{bVerbatim} |
| 906 |
|
| 907 |
Using the thread id, we can locate in the backtrace what the thread was doing |
| 908 |
\begin{bVerbatim} |
| 909 |
Thread 982 (Thread 0x7f67abe5f700 (LWP 6211)): |
| 910 |
#0 0x00007f6d1d8684ed in __lll_lock_wait () from /lib64/libpthread.so.0 |
| 911 |
#1 0x00007f6d1d863dcb in _L_lock_883 () from /lib64/libpthread.so.0 |
| 912 |
#2 0x00007f6d1d863c98 in pthread_mutex_lock () from /lib64/libpthread.so.0 |
| 913 |
#3 0x00007f6d1dcebdcb in lmgr_p (m=m@entry=0x6a8b40 <job_queue>) at lockmgr.c:105 |
| 914 |
#4 0x00007f6d1dced2ed in bthread_mutex_lock_p ("jobq.c", 324) at lockmgr.c:1026 |
| 915 |
#5 0x0000000000430d82 in jobq_remove (jcr=jcr@entry=0x7f6cfcb542c8) at jobq.c:324 |
| 916 |
#6 0x000000000042ec9e in cancel_job (ua=ua@entry=0x7f6be807a7f8, jcr=jcr@entry=0x7f6cfcb542c8, |
| 917 |
wait=wait@entry=60, cancel=cancel@entry=true) at job.c:746 |
| 918 |
#7 0x000000000042f10a in allow_duplicate_job (jcr=jcr@entry=0x7f6cfc361838) at job.c:1155 |
| 919 |
#8 0x00000000004317e5 in reschedule_job (je=0x7f6cfcac48d8, jq=0x6a8b40, jcr=0x7f6cfc361838) at jobq.c:675 |
| 920 |
#9 jobq_server (arg=arg@entry=0x6a8b40 <job_queue>) at jobq.c:494 |
| 921 |
#10 0x00007f6d1dcece25 in lmgr_thread_launcher (x=0x7f6cfc369578) at lockmgr.c:1184 |
| 922 |
#11 0x00007f6d1d861dd5 in start_thread () from /lib64/libpthread.so.0 |
| 923 |
#12 0x00007f6d1c289ead in clone () from /lib64/libc.so.6 |
| 924 |
\end{bVerbatim} |
| 925 |
|
| 926 |
We can also have some information about the job 0x7f67abe5f700 looking the jcr dump list. |
| 927 |
|
| 928 |
\begin{bVerbatim} |
| 929 |
threadid=0x7f67abe5f700 JobId=936628 JobStatus=t jcr=0x7f6cfc361838 |
| 930 |
name=XXX-SQL.2019-08-17_21.30.00_07 |
| 931 |
use_count=1 killable=0 |
| 932 |
JobType=B JobLevel=F |
| 933 |
sched_time=18-Aug-2019 03:13 start_time=18-Aug-2019 00:18 |
| 934 |
end_time=18-Aug-2019 02:43 wait_time=01-Jan-1970 01:00 |
| 935 |
db=0x7f69b0921e28 db_batch=0x7f6bb0033d38 batch_started=0 |
| 936 |
wstore=0x7f6b246262e8 rstore=(nil) wjcr=(nil) client=0x7f6b240c5438 |
| 937 |
reschedule_count=1 SD_msg_chan_started=0 |
| 938 |
|
| 939 |
BDB=0x7f69b0921e28 db_name=bacula db_user=bacula connected=true |
| 940 |
cmd="SELECT MediaId,VolumeName,VolJobs,... |
| 941 |
RWLOCK=0x7f69b0921e40 w_active=0 w_wait=0 |
| 942 |
\end{bVerbatim} |
| 943 |
|
| 944 |
The lock manager can also handle threads, by default on Linux, it is not safe |
| 945 |
to call \texttt{pthread\_kill} on a non existing thread. On linux, pthread\_t |
| 946 |
is a pointer to a struct. As detached threads are released automatically, trying |
| 947 |
to kill an old thread will raise a segmentation fault. With the lockmanager, the |
| 948 |
replacement of the \texttt{pthread\_kill} will check if the thread is registered |
| 949 |
in the lock manager before to kill it. |
| 950 |
|
| 951 |
\smallskip{} |
| 952 |
|
| 953 |
The lock manager also implements a ring of events. This list can be displayed |
| 954 |
in the backtrace file. It can be used to analyze the life of a mutex for |
| 955 |
example. (the backtrace is a fixed picture at a given time). |
| 956 |
|
| 957 |
\section{Tabbing} |
| 958 |
\index{Tabbing} |
| 959 |
|
| 960 |
Tabbing (inserting the tab character in place of spaces) is as normal on all |
| 961 |
Unix systems -- a tab is converted space up to the next column multiple of 8. |
| 962 |
My editor converts strings of spaces to tabs automatically -- this results in |
| 963 |
significant compression of the files. Thus, you can remove tabs by replacing |
| 964 |
them with spaces if you wish. Please don't confuse tabbing (use of tab |
| 965 |
characters) with indenting (visual alignment of the code). |
| 966 |
|
| 967 |
\section{Don'ts} |
| 968 |
\index{Don'ts} |
| 969 |
|
| 970 |
Please don't use: |
| 971 |
|
| 972 |
\begin{bVerbatim} |
| 973 |
strcpy() |
| 974 |
strcat() |
| 975 |
strncpy() |
| 976 |
strncat(); |
| 977 |
sprintf() |
| 978 |
snprintf() |
| 979 |
\end{bVerbatim} |
| 980 |
|
| 981 |
They are system dependent and un-safe. These should be replaced by the \bacula{} |
| 982 |
safe equivalents: |
| 983 |
|
| 984 |
\begin{bVerbatim} |
| 985 |
char *bstrncpy(char *dest, char *source, int dest_size); |
| 986 |
char *bstrncat(char *dest, char *source, int dest_size); |
| 987 |
int bsnprintf(char *buf, int32_t buf_len, const char *fmt, ...); |
| 988 |
int bvsnprintf(char *str, int32_t size, const char *format, va_list ap); |
| 989 |
\end{bVerbatim} |
| 990 |
|
| 991 |
See \bfilename{src/lib/bsys.c} for more details on these routines. |
| 992 |
|
| 993 |
Don't use the \textbf{\%lld} or the \textbf{\%q} printf format editing types to edit |
| 994 |
64 bit integers -- they are not portable. Instead, use \textbf{\%s} with \textbf{edit\_uint64()}. |
| 995 |
For example: |
| 996 |
|
| 997 |
\begin{bVerbatim} |
| 998 |
char buf[100]; |
| 999 |
uint64_t num = something; |
| 1000 |
char ed1[50]; |
| 1001 |
bsnprintf(buf, sizeof(buf), "Num=%s\n", edit_uint64(num, ed1)); |
| 1002 |
\end{bVerbatim} |
| 1003 |
|
| 1004 |
\paragraph{Note}: \textbf{\%lld} is now permitted in \bacula{} code -- we have our |
| 1005 |
own \texttt{printf} routines which handle it correctly. The \texttt{edit\_uint64()} subroutine |
| 1006 |
can still be used if you wish, but over time, most of that old style will |
| 1007 |
be removed. |
| 1008 |
|
| 1009 |
The edit buffer \textbf{ed1} must be at least 27 bytes long to avoid overflow. |
| 1010 |
See \bfilename{src/lib/edit.c} for more details. If you look at the code, don't start |
| 1011 |
screaming that I use \textbf{lld}. I actually use subtle trick taught to me by |
| 1012 |
John Walker. The \textbf{lld} that appears in the editing routine is actually |
| 1013 |
\textbf{\#define} to a what is needed on your OS (usually ``\texttt{lld}'' or ``\texttt{q}'') and |
| 1014 |
is defined in \bfilename{autoconf/configure.in} for each OS. \texttt{C} string concatenation causes |
| 1015 |
the appropriate string to be concatenated to the ``\%''. |
| 1016 |
|
| 1017 |
Also please don't use the STL or Templates or any complicated \texttt{C++} code. |
| 1018 |
|
| 1019 |
\section{Message Classes} |
| 1020 |
\index{Classes!Message} |
| 1021 |
\index{Message Classes} |
| 1022 |
|
| 1023 |
Currently, there are five classes of messages: Debug, Error, Job, Memory, |
| 1024 |
and Queued. |
| 1025 |
|
| 1026 |
\section{Debug Messages} |
| 1027 |
\index{Messages!Debug} |
| 1028 |
\index{Debug Messages} |
| 1029 |
|
| 1030 |
Debug messages are designed to be turned on at a specified debug level and are |
| 1031 |
always sent to \texttt{STDOUT}. There are designed to only be used in the development |
| 1032 |
debug process. They are coded as: |
| 1033 |
|
| 1034 |
\begin{bVerbatim} |
| 1035 |
DmsgN(level, message, arg1, ...) |
| 1036 |
\end{bVerbatim} |
| 1037 |
where the N is a number indicating how many arguments are to be substituted into the message (i.e. it is a count of the |
| 1038 |
number arguments you have in your message -- generally the number of percent |
| 1039 |
signs (\%)). \textbf{level} is the debug level at which you wish the message to |
| 1040 |
be printed. message is the debug message to be printed, and arg1, \ldots{} are the |
| 1041 |
arguments to be substituted. Since not all compilers support \texttt{\#defines} with |
| 1042 |
varargs, you must explicitly specify how many arguments you have. |
| 1043 |
|
| 1044 |
When the debug message is printed, it will automatically be prefixed by the |
| 1045 |
name of the daemon which is running, the filename where the Dmsg is, and the |
| 1046 |
line number within the file. |
| 1047 |
|
| 1048 |
Some actual examples are: |
| 1049 |
\begin{bVerbatim} |
| 1050 |
Dmsg2(20, "MD5len=\%d MD5=\%s\n", strlen(buf), buf); |
| 1051 |
\end{bVerbatim} |
| 1052 |
\begin{bVerbatim} |
| 1053 |
Dmsg1(9, "Created client \%s record\n", client->hdr.name); |
| 1054 |
\end{bVerbatim} |
| 1055 |
|
| 1056 |
|
| 1057 |
\subsection{Debug Tags} |
| 1058 |
|
| 1059 |
It is possible to use tags to organize debug messages. When using a tag, it is possible |
| 1060 |
to hide/show an entire class of messages. The message tags are defined in \texttt{lib/messages.c/h} |
| 1061 |
|
| 1062 |
\begin{bVerbatim} |
| 1063 |
... |
| 1064 |
#define DT_SCHEDULER (1<<23) /* scheduler */ |
| 1065 |
#define DT_PROTOCOL (1<<22) /* protocol */ |
| 1066 |
#define DT_DEDUP (1<<21) /* BEEF deduplication */ |
| 1067 |
#define DT_DDE (1<<20) /* BEEF dedup engine */ |
| 1068 |
#define DT_SNAPSHOT (1<<19) /* Snapshot */ |
| 1069 |
#define DT_RECORD (1<<18) /* Record/block */ |
| 1070 |
#define DT_CLOUD (1<<17) /* cloud */ |
| 1071 |
#define DT_ALL (0x7FFF0000) /* all (up to debug_level 65635, 15 flags available) */ |
| 1072 |
|
| 1073 |
/* setdebug tag=all,-plugin */ |
| 1074 |
static struct debugtags debug_tags[] = { |
| 1075 |
... |
| 1076 |
{ NT_("scheduler"), DT_SCHEDULER,_("Debug scheduler information")}, |
| 1077 |
{ NT_("protocol"), DT_PROTOCOL, _("Debug protocol information")}, |
| 1078 |
{ NT_("snapshot"), DT_SNAPSHOT, _("Debug snapshots")}, |
| 1079 |
{ NT_("record"), DT_RECORD, _("Debug records")}, |
| 1080 |
{ NT_("all"), DT_ALL, _("Debug all information")}, |
| 1081 |
{ NULL, 0, NULL} |
| 1082 |
}; |
| 1083 |
\end{bVerbatim} |
| 1084 |
|
| 1085 |
In the code, the tag can be used like in the following example: |
| 1086 |
|
| 1087 |
\begin{bVerbatim} |
| 1088 |
Dmsg0(DT_CLOUD|50, "This is a message about cloud\n"); |
| 1089 |
\end{bVerbatim} |
| 1090 |
|
| 1091 |
The message will be displayed if the \texttt{cloud} tag is set AND if the debug level is at least 50. |
| 1092 |
|
| 1093 |
\begin{bVerbatim} |
| 1094 |
setdebug level=50 tags=cloud storage |
| 1095 |
\end{bVerbatim} |
| 1096 |
|
| 1097 |
\section{Error Messages} |
| 1098 |
\index{Messages!Error} |
| 1099 |
\index{Error Messages} |
| 1100 |
|
| 1101 |
Error messages are messages that are related to the daemon as a whole rather |
| 1102 |
than a particular job. For example, an out of memory condition my generate an |
| 1103 |
error message. They should be very rarely needed. In general, you should be |
| 1104 |
using Job and Job Queued messages (Jmsg and Qmsg). They are coded as: |
| 1105 |
|
| 1106 |
EmsgN(error-code, level, message, arg1, \ldots{}) As with debug messages, you |
| 1107 |
must explicitly code the of arguments to be substituted in the message. |
| 1108 |
Error-code indicates the severity or class of error, and it may be one of the |
| 1109 |
following (See table \vref{blb:tabdev:errorcodes}: |
| 1110 |
|
| 1111 |
\LTXtable{\linewidth}{t-dev-errorcodes} |
| 1112 |
There are other error message classes, but they are in a state of being |
| 1113 |
redesigned or deprecated, so please do not use them. Some actual examples are: |
| 1114 |
\begin{bVerbatim} |
| 1115 |
Emsg1(M_ABORT, 0, "Cannot create message thread: \%s\n",strerror(status)); |
| 1116 |
\end{bVerbatim} |
| 1117 |
|
| 1118 |
\begin{bVerbatim} |
| 1119 |
Emsg3(M_WARNING, 0, "Connect to File daemon %s at %s:%d failed. Retrying...\n", |
| 1120 |
client->hdr.name, client->address,client->port); |
| 1121 |
\end{bVerbatim} |
| 1122 |
|
| 1123 |
\begin{bVerbatim} |
| 1124 |
Emsg3(M_FATAL, 0, "bdird<filed: bad response from Filed to %s command: |
| 1125 |
%d %s\n", cmd, n, strerror(errno)); |
| 1126 |
\end{bVerbatim} |
| 1127 |
|
| 1128 |
\section{Job Messages} |
| 1129 |
\index{Job Messages} |
| 1130 |
\index{Messages!Job} |
| 1131 |
|
| 1132 |
Job messages are messages that pertain to a particular job such as a file that |
| 1133 |
could not be saved, or the number of files and bytes that were saved. They |
| 1134 |
Are coded as: |
| 1135 |
\begin{bVerbatim} |
| 1136 |
Jmsg(jcr, M\_FATAL, 0, "Text of message"); |
| 1137 |
\end{bVerbatim} |
| 1138 |
A Jmsg with \texttt{M\_FATAL} will fail the job. The Jmsg() takes varargs so can |
| 1139 |
have any number of arguments for substituted in a printf like format. |
| 1140 |
Output from the Jmsg() will go to the Job report. |
| 1141 |
|
| 1142 |
If the Jmsg is followed with a number such as Jmsg1(\ldots{}), the number |
| 1143 |
indicates the number of arguments to be substituted (varargs is not |
| 1144 |
standard for \#defines), and what is more important is that the file and |
| 1145 |
line number will be prefixed to the message. This permits a sort of debug |
| 1146 |
from user's output. |
| 1147 |
|
| 1148 |
\section{Queued Job Messages} |
| 1149 |
\index{Queued Job Messages} |
| 1150 |
\index{Messages!Queued Job} |
| 1151 |
Queued Job messages are similar to Jmsg()s except that the message is |
| 1152 |
Queued rather than immediately dispatched. This is necessary within the |
| 1153 |
network subroutines and in the message editing routines. This is to prevent |
| 1154 |
recursive loops, and to ensure that messages can be delivered even in the |
| 1155 |
event of a network error. |
| 1156 |
|
| 1157 |
|
| 1158 |
\section{Memory Messages} |
| 1159 |
\index{Messages!Memory} |
| 1160 |
\index{Memory Messages} |
| 1161 |
|
| 1162 |
Memory messages are messages that are edited into a memory buffer. Generally |
| 1163 |
they are used in low level routines such as the low level device file \bfilename{dev.c} in |
| 1164 |
the Storage daemon or in the low level Catalog routines. These routines do not |
| 1165 |
generally have access to the Job Control Record and so they return error |
| 1166 |
essages reformatted in a memory buffer. Mmsg() is the way to do this. |
| 1167 |
|
| 1168 |
\section{Bugs Database} |
| 1169 |
\index{Database!Bugs} |
| 1170 |
\index{Bugs Database} |
| 1171 |
We have a bugs database which is at: |
| 1172 |
\bref{http://bugs.bacula.org}{bugs.bacula.org}, and as |
| 1173 |
a developer you will need to respond to bugs, perhaps bugs in general |
| 1174 |
if you have time, otherwise just bugs that correspond to code that |
| 1175 |
you wrote. |
| 1176 |
|
| 1177 |
If you need to answer bugs, please be sure to ask the Project Manager |
| 1178 |
(currently Kern) to give you Developer access to the bugs database. This |
| 1179 |
allows you to modify statuses and close bugs. |
| 1180 |
|
| 1181 |
The first thing is if you want to take over a bug, rather than just make a |
| 1182 |
note, you should assign the bug to yourself. This helps other developers |
| 1183 |
know that you are the principal person to deal with the bug. You can do so |
| 1184 |
by going into the bug and clicking on the \textbf{Update Issue} button. Then |
| 1185 |
you simply go to the \textbf{Assigned To} box and select your name from the |
| 1186 |
drop down box. To actually update it you must click on the |
| 1187 |
\textbf{Update Information} button a bit further down on the screen, but if you have other |
| 1188 |
things to do such as add a Note, you might wait before clicking on the |
| 1189 |
\textbf{Update Information} button. |
| 1190 |
|
| 1191 |
Generally, we set the \textbf{Status} field to either acknowledged, confirmed, |
| 1192 |
or feedback when we first start working on the bug. Feedback is set when |
| 1193 |
we expect that the user should give us more information. |
| 1194 |
|
| 1195 |
Normally, once you are reasonably sure that the bug is fixed, and a patch |
| 1196 |
is made and attached to the bug report, and/or in the SVN, you can close |
| 1197 |
the bug. If you want the user to test the patch, then leave the bug open, |
| 1198 |
otherwise close it and set \textbf{Resolution} to \textbf{Fixed}. We generally |
| 1199 |
close bug reports rather quickly, even without confirmation, especially if |
| 1200 |
we have run tests and can see that for us the problem is fixed. However, |
| 1201 |
in doing so, it avoids misunderstandings if you leave a note while you are |
| 1202 |
closing the bug that says something to the following effect: |
| 1203 |
We are closing this bug because \ldots{} If for some reason, it does not fix |
| 1204 |
your problem, please feel free to reopen it, or to open a new bug report |
| 1205 |
describing the problem''. |
| 1206 |
|
| 1207 |
We do not recommend that you attempt to edit any of the bug notes that have |
| 1208 |
been submitted, nor to delete them or make them private. In fact, if |
| 1209 |
someone accidentally makes a bug note private, you should ask the reason |
| 1210 |
and if at all possible (with his agreement) make the bug note public. |
| 1211 |
|
| 1212 |
If the user has not properly filled in most of the important fields |
| 1213 |
(platorm, OS, Product Version, \ldots{}) please do not hesitate to politely ask |
| 1214 |
him. Also, if the bug report is a request for a new feature, please |
| 1215 |
politely send the user to the Feature Request menu item on \bref{http://www.bacula.org}{www.bacula.org}. |
| 1216 |
The same applies to a support request (we answer only bugs), you might give |
| 1217 |
the user a tip, but please politely refer him to the manual and the |
| 1218 |
Getting Support page of \bref{http://www.bacula.org}{www.bacula.org}. |