\chapter{\bacula{} Developer Notes} \label{blb:TheChapterStart10} \index{\bacula{} Developer Notes} \index{Notes!\bacula{} Developer} This document is intended mostly for developers and describes how you can contribute to the \bacula{} project and the the general framework of making \bacula{} source changes. \section{Contributions} \index{Contributions} Contributions to the \bacula{} project come in many forms: ideas, participation in helping people on the bacula-users email list, packaging \bacula{} binaries for the community, helping improve the documentation, and submitting code. Contributions in the form of submissions for inclusion in the project are broken into two groups. The first are contributions that are aids and not essential to \bacula{}. In general, these will be scripts or will go into the \bdirectoryname{bacula/examples} directory. For these kinds of non-essential contributions there is no obligation to do a copyright assignment as described below. However, a copyright assignment would still be appreciated. The second class of contributions are those which will be integrated with \bacula{} and become an essential part (code, scripts, documentation, \ldots{}) Within this class of contributions, there are two hurdles to surmount. One is getting your patch accepted, and two is dealing with copyright issues. The following text describes some of the requirements for such code. \section{Patches} \index{Patches} Subject to the copyright assignment described below, your patches should be sent in \btool{git format-patch} format relative to the current contents of the master branch of the Source Forge \btool{git} repository. Please attach the output file or files generated by the \btool{git format-patch} to the email rather than include them directory to avoid wrapping of the lines in the patch. Please be sure to use the \bacula{} indenting standard (see below) for source code. If you have checked out the source with \btool{git}, you can get a \bcommandname{diff} using. \begin{bVerbatim} git pull git format-patch -M \end{bVerbatim} If you plan on doing significant development work over a period of time, after having your first patch reviewed and approved, you will be eligible for having developer \btool{git} write access so that you can commit your changes directly to the \btool{git} repository. To do so, you will need a userid on Source Forge. \section{Copyrights} \index{Copyrights} To avoid future problems concerning changing licensing or copyrights, all code contributions more than a hand full of lines must be in the Public Domain or have the copyright transferred to the Free Software Foundation Europe e.V. with a \ac{FLA} as the case for all the current code. Prior to November 2004, all the code was copyrighted by Kern Sibbald and John Walker. After November 2004, the code was copyrighted by Kern Sibbald, then on the 15th of November 2006, Kern transferred the copyright to the Free Software Foundation Europe e.V. In signing the \ac{FLA} and transferring the copyright, you retain the right to use the code you have submitted as you want, and you ensure that \bacula{} will always remain Free and Open Source. Your name should be clearly indicated as the author of the code, and you must be extremely careful not to violate any copyrights or patents or use other people's code without acknowledging it. The purpose of this requirement is to avoid future copyright, patent, or intellectual property problems. Please read the \bfilename{LICENSE} agreement in the main \bacula{} source code directory. When you sign the \acl{FLA} (\acs{FLA}) and send it in, you are agreeing to the terms of that \bfilename{LICENSE} file. If you don't understand what we mean by future problems, please examine the difficulties Mozilla was having finding previous contributors at \bref{http://www.mozilla.org/MPL/missing.html} {www.mozilla.org/MPL/missing.html}. The other important issue is to avoid copyright, patent, or intellectual property violations as was (May 2003) claimed by SCO against IBM. Although the copyright will be held by the Free Software Foundation Europe e.V., each developer is expected to indicate that he wrote and/or modified a particular module (or file) and any other sources. The copyright assignment may seem a bit unusual, but in reality, it is not. Most large projects require this. If you have any doubts about this, please don't hesitate to ask. The objective is to assure the long term survival of the \bacula{} project. Items not needing a copyright assignment are: most small changes, enhancements, or bug fixes of 5-10 lines of code, which amount to less than 20\% of any particular file. \section{Copyright Assignment -- Fiduciary License Agreement} \index{Copyright Assignment} \index{Assignment!Copyright} Since this is not a commercial enterprise, and we prefer to believe in everyone's good faith, previously developers could assign the copyright by explicitly acknowledging that they do so in their first submission. This was sufficient if the developer is independent, or an employee of a not-for-profit organization or a university. However, in an effort to ensure that the \bacula{} code is really clean, beginning in August 2006, all previous and future developers with \acs{SVN} write access will be asked to submit a copyright assignment (or \acl{FLA} -- \acs{FLA}), which means you agree to the \bfilename{LICENSE} in the main source directory. It also means that you receive back the right to use the code that you have submitted. Any developer who wants to contribute and is employed by a company should either list the employer as the owner of the code, or get explicit permission from him to sign the copyright assignment. This is because in many countries, all work that an employee does whether on company time or in the employee's free time is considered to be Intellectual Property of the company. Obtaining official approval or an \ac{FLA} from the company will avoid misunderstandings between the employee, the company, and the \bacula{} project. A good number of companies have already followed this procedure. The \acl{FLA} is posted on the \bacula{} web site at: \bref{http://www.bacula.org/en/FLA-bacula.en.pdf}{www.bacula.org/en/FLA-bacula.en.pdf} The instructions for filling out this agreement are also at: \bref{http://www.bacula.org/?page=fsfe}{www.bacula.org/?page=fsfe} It should be filled out, then sent to: \begin{bVerbatim} Kern Sibbald Cotes-de-Montmoiret 9 1012 Lausanne Switzerland \end{bVerbatim} Please note that the above address is different from the officially registered office mentioned in the document. When you send in such a complete document, please notify me: \bbracket{kern at sibbald dot com}, and please add your email address to the \ac{FLA} so that I can contact you to confirm reception of the signed \ac{FLA}. \section{The Development Cycle} \index{Developement Cycle} \index{Cycle!Developement} As discussed on the email lists, the number of contributions are increasing significantly. We expect this positive trend will continue. As a consequence, we have modified how we do development, and instead of making a list of all the features that we will implement in the next version, each developer signs up for one (maybe two) projects at a time, and when they are complete, and the code is stable, we will release a new version. The release cycle will probably be roughly six months. The difference is that with a shorter release cycle and fewer released feature, we will have more time to review the new code that is being contributed, and will be able to devote more time to a smaller number of projects (some prior versions had too many new features for us to handle correctly). Future release schedules will be much the same, and the number of new features will also be much the same providing that the contributions continue to come -- and they show no signs of let up :-) \index{Feature Requests} \subsection{Feature Requests} In addition, we have ``formalizee'' the feature requests a bit. Instead of me maintaining an informal list of everything I run into (\bfilename{kernstodo}), we now maintain a ``formal'' list of projects. This means that all new feature requests, including those recently discussed on the email lists, must be formally submitted and approved. Formal submission of feature requests will take two forms: \begin{benumerate} \item non-mandatory, but highly recommended is to discuss proposed new features on the mailing list. \item Formal submission of an Feature Request in a special format. We'll give an example of this below, but you can also find it on the web site under Support \textrightarrow{} Feature Requests. Since it takes a bit of time to properly fill out a Feature Request form, you probably should check on the email list first. \end{benumerate} Once the Feature Request is received by the keeper of the projects list, it will be sent to the \bacula{} project manager (Kern), and he will either accept it (90\% of the time), send it back asking for clarification (10\% of the time), send it to the email list asking for opinions, or reject it (very few cases). If it is accepted, it will go in the \bfilename{projects} file (a simple \acs{ASCII} file) maintained in the main \bacula{} source directory. \subsubsection{Implementation of Feature Requests} Any qualified developer can sign up for a project. The project must have an entry in the projects file, and the developer's name will appear in the Status field. \subsubsection{How Feature Requests are accepted} Acceptance of Feature Requests depends on several things: \begin{benumerate} \item feedback from users. If it is negative, the Feature Request will probably not be accepted. \\ \item the difficulty of the project. A project that is so difficult that we cannot imagine finding someone to implement probably won't be accepted. Obviously if you know how to implement it, don't hesitate to put it in your Feature Request \item whether or not the Feature Request fits within the current strategy of \bacula{} (for example an Feature Request that requests changing the tape to \tar{} format probably would not be accepted, \ldots{}). \end{benumerate} \subsubsection{How Feature Requests are prioritized} Once an Feature Request is accepted, it needs to be implemented. If you can find a developer for it, or one signs up for implementing it, then the Feature Request becomes top priority (at least for that developer). Between releases of \bacula{}, we will generally solicit Feature Request input for the next version, and by way of this email, we suggest that you send discuss and send in your Feature Requests for the next release. Please verify that the Feature Request is not in the current list (attached to this email). Once users have had several weeks to submit Feature Requests, the keeper of the projects list will organize them, and request users to vote on them. This will allow fixing prioritizing the Feature Requests. Having a priority is one thing, but getting it implement is another thing -- we are hoping that the \bacula{} community will take more responsibility for assuring the implementation of accepted Feature Requests. Feature Request format: \begin{bVerbatim} ============= Empty Feature Request form =========== Item n: One line summary ... Date: Date submitted Origin: Name and email of originator. Status: What: More detailed explanation ... Why: Why it is important ... Notes: Additional notes or features (omit if not used) ============== End Feature Request form ============== \end{bVerbatim} \begin{bVerbatim} ============= Example Completed Feature Request form =========== Item 1: Implement a Migration job type that will move the job data from one device to another. Origin: Sponsored by Riege Sofware International GmbH. Contact: Daniel Holtkamp Date: 28 October 2005 Status: Partially coded in 1.37 -- much more to do. Assigned to Kern. What: The ability to copy, move, or archive data that is on a device to another device is very important. Why: An ISP might want to backup to disk, but after 30 days migrate the data to tape backup and delete it from disk. Bacula should be able to handle this automatically. It needs to know what was put where, and when, and what to migrate -- it is a bit like retention periods. Doing so would allow space to be freed up for current backups while maintaining older data on tape drives. Notes: Migration could be triggered by: Number of Jobs Number of Volumes Age of Jobs Highwater size (keep total size) Lowwater mark ================================================= \end{bVerbatim} \section{\bacula{} Code Submissions and Projects} \index{Submissions and Projects} Getting code implemented in \bacula{} works roughly as follows: \begin{bitemize} \item Kern is the project manager, but prefers not to be a ``gate keeper''. This means that the developers are expected to be self-motivated, and once they have experience submit directly to the \btool{git} repositories. However, it is a good idea to have your patches reviewed prior to submitting, and it is a bad idea to submit monster patches because no one will be able to properly review them. See below for more details on this. \item There are growing numbers of contributions (very good). \item Some contributions come in the form of relatively small patches, which Kern reviews, integrates, documents, tests, and maintains. \item All \bacula{} developers take full responsibility for writing the code, posting as patches so that we can review it as time permits, integrating it at an appropriate time, responding to our requests for tweaking it (name changes, \ldots{}), document it in the code, document it in the manual (even though their mother tongue is not English), test it, develop and commit regression scripts, and answer in a timely fashion all bug reports -- even occasionally accepting additional bugs :-) This is a sustainable way of going forward with \bacula{}, and the direction that the project will be taking more and more. For example, in the past, we have had some very dedicated programmers who did major projects. However, some of these programmers due to outside obligations (job responsibilities change of job, school duties, \ldots{}) could not continue to maintain the code. In those cases, the code suffers from lack of maintenance, sometimes we patch it, sometimes not. In the end, if the code is not maintained, the code gets dropped from the project (there are two such contributions that are heading in that direction). When ever possible, we would like to avoid this, and ensure a continuation of the code and a sharing of the development, debugging, documentation, and maintenance responsibilities. \end{bitemize} \section{Patches for Released Versions} \index{Patches for Released Versions} If you fix a bug in a released version, you should, unless it is an absolutely trivial bug, create and release a patch file for the bug. The procedure is as follows: Fix the bug in the released branch and in the develpment master branch. Make a patch file for the branch and add the branch patch to the patches directory in both the branch and the trunk. The name should be \texttt{2.2.4-xxx.patch} where \texttt{xxx} is unique, in this case it can be ``restore'', e.g. \texttt{2.2.4-restore.patch}. Add to the top of the file a brief description and instructions for applying it -- see for example \texttt{2.2.4-poll-mount.patch}. The best way to create the patch file is as follows: \begin{bVerbatim} (edit) 2.2.4-restore.patch (input description) (end edit) git format-patch -M mv 0001-xxx 2.2.4-restore.patch \end{bVerbatim} check to make sure no extra junk got put into the patch file (i.e. it should have the patch for that bug only). If there is not a bug report on the problem, create one, then add the patch to the bug report. Then upload it to the 2.2.x release of bacula-patches. So, end the end, the patch file is: \begin{bitemize} \item Attached to the bug report \item In \bfilename{Branch-2.2/bacula/patches/...} \item In the trunk \item Loaded on Source Forge bacula-patches 2.2.x release. When you add it, click on the check box to send an Email so that all the users that are monitoring SF patches get notified. \end{bitemize} \section{Developing \bacula{}} \index{Developing \bacula{}} \index{\bacula{}!Developing} Typically the simplest way to develop \bacula{} is to open one \texttt{xterm} window pointing to the source directory you wish to update; a second \texttt{xterm} window at the top source directory level, and a third xterm window at the bacula directory \bdirectoryname{\bbracket{top}/src/bacula}. After making source changes in one of the directories, in the top source directory \texttt{xterm}, build the source, and start the daemons by entering: \begin{bVerbatim} make \end{bVerbatim} and \begin{bVerbatim} ./startit \end{bVerbatim} then in the enter: \begin{bVerbatim} ./console \end{bVerbatim} or \begin{bVerbatim} ./gnome-console \end{bVerbatim} to start the Console program. Enter any commands for testing. For example: \bcommandname{run kernsverify full}. Note, the instructions here to use \btool{./startit} are different from using a production system where the administrator starts \bacula{} by entering \btool{./bacula start}. This difference allows a development version of {\bf \bacula{}} to be run on a computer at the same time that a production system is running. The \btool{./startit} script starts \textbf{\bacula{}} using a different set of configuration files, and thus permits avoiding conflicts with any production system. To make additional source changes, exit from the Console program, and in the top source directory, stop the daemons by entering: \begin{bVerbatim} ./stopit \end{bVerbatim} then repeat the process. \section{Debugging} \index{Debugging} Probably the first thing to do is to turn on debug output. A good place to start is with a debug level of 20 as in \btool{./startit -d20}. The \btool{startit} command starts all the daemons with the same debug level. Alternatively, you can start the appropriate daemon with the debug level you want. If you really need more info, a debug level of 60 is not bad, and for just about everything a level of 200. \section{Using a Debugger} \index{Using a Debugger} \index{Debugger!Using a} If you have a serious problem such as a segmentation fault, it can usually be found quickly using a good multiple thread debugger such as \btool{gdb}. For example, suppose you get a segmentation violation in \btool{bacula-dir}. You might use the following to find the problem: \bbracket{start the Storage and File daemons} \begin{bVerbatim} cd dird gdb ./bacula-dir run -f -s -c ./dird.conf \end{bVerbatim} \bbracket{it dies with a segmentation fault} where The \texttt{-f} option is specified on the \bcommandname{run} command to inhibit \btool{dird} from going into the background. You may also want to add the \texttt{-s} option to the run command to disable signals which can potentially interfere with the debugging. As an alternative to using the debugger, each {\bf \bacula{}} daemon has a built in back trace feature when a serious error is encountered. It calls the debugger on itself, produces a back trace, and emails the report to the developer. For more details on this, please see the chapter in the main \bacula{} manual entitled ``What To Do When \bacula{} Crashes (Kaboom)''. \section{Memory Leaks} \index{Leaks!Memory} \index{Memory Leaks} Because \bacula{} runs routinely and unattended on client and server machines, it may run for a long time. As a consequence, from the very beginning, \bacula{} uses \texttt{SmartAlloc} to ensure that there are no memory leaks. To make detection of memory leaks effective, all \bacula{} code that dynamically allocates memory \textbf{must} have a way to release it. In general when the memory is no longer needed, it should be immediately released, but in some cases, the memory will be held during the entire time that \bacula{} is executing. In that case, there \textbf{must} be a routine that can be called at termination time that releases the memory. In this way, we will be able to detect memory leaks. Be sure to immediately correct any and all memory leaks that are printed at the termination of the daemons. \section{Special Files} \index{Files!Special} \index{Special Files} Kern uses files named 1, 2, \ldots{} 9 with any extension as scratch files. Thus any files with these names are subject to being rudely deleted at any time. \section{When Implementing Incomplete Code} \index{Code!When Implementing Incomplete} \index{When Implementing Incomplete Code} Please identify all incomplete code with a comment that contains \begin{bVerbatim} ***FIXME*** \end{bVerbatim} where there are three asterisks (*) before and after the word FIXME (in capitals) and no intervening spaces. This is important as it allows new programmers to easily recognize where things are partially implemented. \subsection{\bacula{} Source File Structure} \index{Structure!\bacula{} Source File} \index{\bacula{} Source File Structure} The distribution generally comes as a \tar{} file of the form \bfilename{bacula.x.y.z.tar.gz} where x, y, and z are the version, release, and update numbers respectively. Once you detar this file, you will have a directory structure as follows: \begin{bVerbatim} | Tar file: |- depkgs |- mtx (autochanger control program + tape drive info) |- sqlite (SQLite database program) Tar file: |- depkgs-win32 |- pthreads (Native win32 pthreads library -- dll) |- zlib (Native win32 zlib library) |- wx (wxWidgets source code) Project bacula: |- bacula (main source directory containing configuration | and installation files) |- autoconf (automatic configuration files, not normally used | by users) |- intl (programs used to translate) |- platforms (OS specific installation files) |- redhat (Red Hat installation) |- solaris (Sun installation) |- freebsd (FreeBSD installation) |- irix (Irix installation -- not tested) |- unknown (Default if system not identified) |- po (translations of source strings) |- src (source directory; contains global header files) |- plugins (plugins for FD/SD/DIR) |-fd (FileDaemon plugins) |-sd (Storage Daemon plugins) |-dir (Director Daemon plugins) |- cats (SQL catalog database interface directory) |- console (bacula user agent directory) |- dird (Director daemon) |- filed (Unix File daemon) |- findlib (Unix file find library for File daemon) |- lib (General Bacula library) |- stored (Storage daemon) |- tools (Various tool programs) |- win32 (Native Win32 File daemon) |- libwin32 (Win32 files to make bacula-fd be a service) |- compat (compatibility interface library) |- filed (links to src/filed) |- findlib (links to src/findlib) |- lib (links to src/lib) |- console (beginning of native console program) |- wx-console (wxWidget console Win32 specific parts) |- win32-installer (Makensis installer for 32bit) |- win64-installer (Makensis installer for 64bit) Project regress: |- regress (Regression scripts) |- bin (temporary directory to hold Bacula installed binaries) |- build (temporary directory to hold Bacula source) |- scripts (scripts and .conf files) |- tests (test scripts) |- tmp (temporary directory for temp files) |- working (temporary working directory for Bacula daemons) Project docs: |- docs (documentation directory) |- developers (Developer's guide) |- home-page (Bacula's home page source) |- manual (html document directory) |- manual-fr (French translation) |- manual-de (German translation) |- techlogs (Technical development notes); Project gui: |- gui (Bacula GUI projects) |- bacula-web (Bacula web php management code) |- bimagemgr (Web application for burning CDROMs) \end{bVerbatim} \section{Header Files} \index{Header Files} \index{Files!Header} Please carefully follow the scheme defined below as it permits in general only two header file includes per \texttt{C} file, and thus vastly simplifies programming. With a large complex project like \bacula{}, it isn't always easy to ensure that the right headers are invoked in the right order (there are a few kludges to make this happen -- i.e. in a few include files because of the chicken and egg problem, certain references to typedefs had to be replaced with \textbf{void}). Every file should include \bfilename{bacula.h}. It pulls in just about everything, with very few exceptions. If you have system dependent ifdefing, please do it in \bfilename{baconfig.h}. The version number and date are kept in \bfilename{version.h}. Each of the subdirectories (\bdirectoryname{console}, \bdirectoryname{cats}, \bdirectoryname{dird}, \bfilename{filed}, \bfilename{findlib}, \bfilename{lib}, \bfilename{stored}, \ldots{}) contains a single directory dependent include file generally the name of the directory, which should be included just after the include of \bfilename{bacula.h}. This file (for example, for the dird directory, it is \bfilename{dird.h}) contains either definitions of things generally needed in this directory, or it includes the appropriate header files. It always includes \bfilename{protos.h}. See below. Each subdirectory contains a header file named \bfilename{protos.h}, which contains the prototypes for subroutines exported by files in that directory. \bfilename{protos.h} is always included by the main directory dependent include file. \section{Programming Standards} \index{Standards!Programming} \index{Programming Standards} For the most part, all code should be written in \texttt{C} unless there is a burning reason to use \texttt{C++}, and then only the simplest \texttt{C++} constructs will be used. Note, \bacula{} is slowly evolving to use more and more \texttt{C++}. Code should have some documentation -- not a lot, but enough so that I can understand it. Look at the current code, and you will see that I document more than most, but am definitely not a fanatic. We prefer simple linear code where possible. Gotos are strongly discouraged except for handling an error to either bail out or to retry some code, and such use of gotos can vastly simplify the program. Remember this is a \texttt{C} program that is migrating to a \textbf{tiny} subset of \texttt{C++}, so be conservative in your use of \texttt{C++} features. \section{Do Not Use} \index{Use!Do Not} \index{Do Not Use} \begin{bitemize} \item STL -- it is totally incomprehensible. \end{bitemize} \section{Avoid if Possible} \index{Possible!Avoid if} \index{Avoid if Possible} \begin{bitemize} \item Using \textbf{void *} because this generally means that one must using casting, and in \texttt{C++} casting is rather ugly. It is OK to use \texttt{void *} to pass structure address where the structure is not known to the routines accepting the packet (typically callback routines). However, declaring \texttt{void *buf} is a bad idea. Please use the correct types whenever possible. \item Using undefined storage specifications such as (\texttt{short}, \texttt{int}, \texttt{long}, \texttt{long long}, \texttt{size\_t \ldots{}}). The problem with all these is that the number of bytes they allocate depends on the compiler and the system. Instead use \bacula{}'s types (\texttt{int8\_t}, \texttt{uint8\_t}, \texttt{int32\_t}, \texttt{uint32\_t}, \texttt{int64\_t}, and \texttt{uint64\_t}). This guarantees that the variables are given exactly the size you want. Please try at all possible to avoid using \texttt{size\_t ssize\_t} and the such. They are very system dependent. However, some system routines may need them, so their use is often unavoidable. \item Returning a malloc'ed buffer from a subroutine -- someone will forget to release it. \item Heap allocation (malloc) unless needed -- it is expensive. Use \texttt{POOL\_MEM} instead. \item Templates -- they can create portability problems. \item Fancy or tricky \texttt{C} or \texttt{C++} code, unless you give a good explanation of why you used it. \item Too much inheritance -- it can complicate the code, and make reading it difficult (unless you are in love with colons) \end{bitemize} \section{Do Use Whenever Possible} \index{Possible!Do Use Whenever} \index{Do Use Whenever Possible} \begin{bitemize} \item Locking and unlocking within a single subroutine. \item A single point of exit from all subroutines. A goto is perfectly OK to use to get out early, but only to a label named \texttt{bail\_out}, and possibly an \texttt{ok\_out}. See current code examples. \item \texttt{malloc} and \texttt{free} within a single subroutine. \item Comments and global explanations on what your code or algorithm does. \item When committing a fix for a bug, make the comment of the following form: \begin{bVerbatim} Reason for bug fix or other message. Fixes bug #1234 \end{bVerbatim} It is important to write the \textbf{ bug \#1234} like that because our program that automatically pulls messages from the \btool{git} repository to make the ChangeLog looks for that pattern. Obviously the \textbf{1234} should be replaced with the number of the bug you actually fixed. Providing the commit comment line has one of the following keywords (or phrases), it will be ignored: \begin{bVerbatim} tweak typo cleanup bweb: regress: again .gitignore fix compilation technotes update version update technotes update kernstodo update projects update releasenotes update version update home update release update todo update notes update changelog \end{bVerbatim} \item Use the following keywords at the beginning of a \btool{git} commit message \end{bitemize} \section{Indenting Standards} \index{Standards!Indenting} \index{Indenting Standards} We find it very hard to read code indented 8 columns at a time. Even 4 at a time uses a lot of space, so we have adopted indenting 3 spaces at every level. Note, indention is the visual appearance of the source on the page, while tabbing is replacing a series of up to 8 spaces from a tab character. The closest set of parameters for the Linux \textbf{indent} program that will produce reasonably indented code are: \begin{bVerbatim} -nbad -bap -bbo -nbc -br -brs -c36 -cd36 -ncdb -ce -ci3 -cli0 -cp36 -d0 -di1 -ndj -nfc1 -nfca -hnl -i3 -ip0 -l85 -lp -npcs -nprs -npsl -saf -sai -saw -nsob -nss -nbc -ncs -nbfda \end{bVerbatim} You can put the above in your \bfilename{.indent.pro} file, and then just invoke indent on your file. However, be warned. This does not produce perfect indenting, and it will mess up \texttt{C++} class statements pretty badly. Braces are required in all if statements (missing in some very old code). To avoid generating too many lines, the first brace appears on the first line (e.g. of an \texttt{if}), and the closing brace is on a line by itself. E.g. \begin{bVerbatim} if (abc) { some_code; } \end{bVerbatim} Just follow the convention in the code. For example we I prefer non-indented cases. \begin{bVerbatim} switch (code) { case 'A': do something break; case 'B': again(); break; default: break; } \end{bVerbatim} Avoid using \texttt{//} style comments except for temporary code or turning off debug code. Standard \texttt{C} comments are preferred (this also keeps the code closer to \texttt{C}). Attempt to keep all lines less than 85 characters long so that the whole line of code is readable at one time. This is not a rigid requirement. Always put a brief description at the top of any new file created describing what it does and including your name and the date it was first written. Please don't forget any Copyrights and acknowledgments if it isn't 100\% your code. Also, include the \bacula{} copyright notice that is in \bdirectoryname{src/c}. In general you should have two includes at the top of the an include for the particular directory the code is in, for includes are needed, but this should be rare. In general (except for self-contained packages), prototypes should all be put in \bfilename{protos.h} in each directory. Always put space around assignment and comparison operators. \begin{bVerbatim} a = 1; if (b >= 2) { cleanup(); } \end{bVerbatim} but your can compress things in a \textbf{for} statement: \begin{bVerbatim} for (i=0; i < del.num_ids; i++) { ... \end{bVerbatim} Don't overuse the inline if (\texttt{?:}). A full \texttt{if} is preferred, except in a print statement, e.g.: \begin{bVerbatim} if (ua->verbose && del.num_del != 0) { bsendmsg(ua, _("Pruned %d %s on Volume %s from catalog.\n"), del.num_del, del.num_del == 1 ? "Job" : "Jobs", mr->VolumeName); } \end{bVerbatim} Leave a certain amount of debug code (Dmsg) in code you submit, so that future problems can be identified. This is particularly true for complicated code likely to break. However, try to keep the debug code to a minimum to avoid bloating the program and above all to keep the code readable. Please keep the same style in all new code you develop. If you include code previously written, you have the option of leaving it with the old indenting or re-indenting it. If the old code is indented with 8 spaces, then please re-indent it to \bacula{} standards. If you are using \btool{vim}, simply set your tabstop to 8 and your shiftwidth to 3. \section{Naming Convention} A fairly big amount of functions and variable are using the ``snake\_case'' format, although you may find some classes that use the ``CamelFormat''. \smallskip We strongly recommend to stick to the ``snake\_case'' format to ease the reading of the code flow. Switching from one to the other is difficult and requires extra brain power. \smallskip Functions associated with a concept or a module and be prefixed with the name of the concept. For example, all Bacula Virtual FileSystem (bvfs) functions are prefixed with \texttt{bvfs\_}. \smallskip Function and variable names must be readable, in general it is a bad idea to use striped down version of a function or a variable. In general you will find the following variables, functions or label in the code: \begin{itemize} \item \texttt{ret} - int/bool return code for a function \item \texttt{buf} - buffer \item \texttt{bail\_out} - cleanup label for a function \item \texttt{ed1, ed2, ed3} - temporary buffer used to convert integers (50c long) \item \texttt{len} - length of a buffer \item \texttt{src} - source \item \texttt{dest} - destination \item \texttt{argc} - argument count \item \texttt{argk} - argument key (key=val) \item \texttt{argv} - argument value (key=val) \item \texttt{ua} - User Agent object \item \texttt{bvfs} - Bacula Virtual File System \item \texttt{lmgr} - Lock Manager \item \texttt{DCR} - Device Control Resource \item \texttt{ctx} - Context \end{itemize} \section{Locks and Threads} Bacula has a builtin lock manager called \texttt{lmgr}. This lock manager is a wrapper for common pthread or mutex operation. The lock manager will overwrite the different POSIX thread functions via \texttt{src/lockmgr.h}. \smallskip{} It can detect deadlock situation during the run time. \texttt{Found a deadlock !!!!} \smallskip{} The lock manager can prevent deadlock and help developers to design with the mutex list (\texttt{lib/mutex\_list.h}). If all mutex are acquired/released with some predefined order, deadlocks are not possible. \begin{bVerbatim} ERROR: V out of order lock=%p %s:%i dumping lock \end{bVerbatim} The lock manager will also dump all the mutex map during a backtrace. It can be analyzed easily to find the incorrect lock path. In the following example, we can see that one thread (0x7f67abe5f700) has requested to lock the same lock (0x6a8b40) two times from two different location. \begin{bVerbatim} threadid=0x7f67abe5f700 max=4 current=1 lock=0x6a8b40 state=Granted priority=0 jobq.c:476 lock=0x6a8b40 state=Wanted priority=0 jobq.c:324 \end{bVerbatim} Using the thread id, we can locate in the backtrace what the thread was doing \begin{bVerbatim} Thread 982 (Thread 0x7f67abe5f700 (LWP 6211)): #0 0x00007f6d1d8684ed in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f6d1d863dcb in _L_lock_883 () from /lib64/libpthread.so.0 #2 0x00007f6d1d863c98 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007f6d1dcebdcb in lmgr_p (m=m@entry=0x6a8b40 ) at lockmgr.c:105 #4 0x00007f6d1dced2ed in bthread_mutex_lock_p ("jobq.c", 324) at lockmgr.c:1026 #5 0x0000000000430d82 in jobq_remove (jcr=jcr@entry=0x7f6cfcb542c8) at jobq.c:324 #6 0x000000000042ec9e in cancel_job (ua=ua@entry=0x7f6be807a7f8, jcr=jcr@entry=0x7f6cfcb542c8, wait=wait@entry=60, cancel=cancel@entry=true) at job.c:746 #7 0x000000000042f10a in allow_duplicate_job (jcr=jcr@entry=0x7f6cfc361838) at job.c:1155 #8 0x00000000004317e5 in reschedule_job (je=0x7f6cfcac48d8, jq=0x6a8b40, jcr=0x7f6cfc361838) at jobq.c:675 #9 jobq_server (arg=arg@entry=0x6a8b40 ) at jobq.c:494 #10 0x00007f6d1dcece25 in lmgr_thread_launcher (x=0x7f6cfc369578) at lockmgr.c:1184 #11 0x00007f6d1d861dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f6d1c289ead in clone () from /lib64/libc.so.6 \end{bVerbatim} We can also have some information about the job 0x7f67abe5f700 looking the jcr dump list. \begin{bVerbatim} threadid=0x7f67abe5f700 JobId=936628 JobStatus=t jcr=0x7f6cfc361838 name=XXX-SQL.2019-08-17_21.30.00_07 use_count=1 killable=0 JobType=B JobLevel=F sched_time=18-Aug-2019 03:13 start_time=18-Aug-2019 00:18 end_time=18-Aug-2019 02:43 wait_time=01-Jan-1970 01:00 db=0x7f69b0921e28 db_batch=0x7f6bb0033d38 batch_started=0 wstore=0x7f6b246262e8 rstore=(nil) wjcr=(nil) client=0x7f6b240c5438 reschedule_count=1 SD_msg_chan_started=0 BDB=0x7f69b0921e28 db_name=bacula db_user=bacula connected=true cmd="SELECT MediaId,VolumeName,VolJobs,... RWLOCK=0x7f69b0921e40 w_active=0 w_wait=0 \end{bVerbatim} The lock manager can also handle threads, by default on Linux, it is not safe to call \texttt{pthread\_kill} on a non existing thread. On linux, pthread\_t is a pointer to a struct. As detached threads are released automatically, trying to kill an old thread will raise a segmentation fault. With the lockmanager, the replacement of the \texttt{pthread\_kill} will check if the thread is registered in the lock manager before to kill it. \smallskip{} The lock manager also implements a ring of events. This list can be displayed in the backtrace file. It can be used to analyze the life of a mutex for example. (the backtrace is a fixed picture at a given time). \section{Tabbing} \index{Tabbing} Tabbing (inserting the tab character in place of spaces) is as normal on all Unix systems -- a tab is converted space up to the next column multiple of 8. My editor converts strings of spaces to tabs automatically -- this results in significant compression of the files. Thus, you can remove tabs by replacing them with spaces if you wish. Please don't confuse tabbing (use of tab characters) with indenting (visual alignment of the code). \section{Don'ts} \index{Don'ts} Please don't use: \begin{bVerbatim} strcpy() strcat() strncpy() strncat(); sprintf() snprintf() \end{bVerbatim} They are system dependent and un-safe. These should be replaced by the \bacula{} safe equivalents: \begin{bVerbatim} char *bstrncpy(char *dest, char *source, int dest_size); char *bstrncat(char *dest, char *source, int dest_size); int bsnprintf(char *buf, int32_t buf_len, const char *fmt, ...); int bvsnprintf(char *str, int32_t size, const char *format, va_list ap); \end{bVerbatim} See \bfilename{src/lib/bsys.c} for more details on these routines. Don't use the \textbf{\%lld} or the \textbf{\%q} printf format editing types to edit 64 bit integers -- they are not portable. Instead, use \textbf{\%s} with \textbf{edit\_uint64()}. For example: \begin{bVerbatim} char buf[100]; uint64_t num = something; char ed1[50]; bsnprintf(buf, sizeof(buf), "Num=%s\n", edit_uint64(num, ed1)); \end{bVerbatim} \paragraph{Note}: \textbf{\%lld} is now permitted in \bacula{} code -- we have our own \texttt{printf} routines which handle it correctly. The \texttt{edit\_uint64()} subroutine can still be used if you wish, but over time, most of that old style will be removed. The edit buffer \textbf{ed1} must be at least 27 bytes long to avoid overflow. See \bfilename{src/lib/edit.c} for more details. If you look at the code, don't start screaming that I use \textbf{lld}. I actually use subtle trick taught to me by John Walker. The \textbf{lld} that appears in the editing routine is actually \textbf{\#define} to a what is needed on your OS (usually ``\texttt{lld}'' or ``\texttt{q}'') and is defined in \bfilename{autoconf/configure.in} for each OS. \texttt{C} string concatenation causes the appropriate string to be concatenated to the ``\%''. Also please don't use the STL or Templates or any complicated \texttt{C++} code. \section{Message Classes} \index{Classes!Message} \index{Message Classes} Currently, there are five classes of messages: Debug, Error, Job, Memory, and Queued. \section{Debug Messages} \index{Messages!Debug} \index{Debug Messages} Debug messages are designed to be turned on at a specified debug level and are always sent to \texttt{STDOUT}. There are designed to only be used in the development debug process. They are coded as: \begin{bVerbatim} DmsgN(level, message, arg1, ...) \end{bVerbatim} where the N is a number indicating how many arguments are to be substituted into the message (i.e. it is a count of the number arguments you have in your message -- generally the number of percent signs (\%)). \textbf{level} is the debug level at which you wish the message to be printed. message is the debug message to be printed, and arg1, \ldots{} are the arguments to be substituted. Since not all compilers support \texttt{\#defines} with varargs, you must explicitly specify how many arguments you have. When the debug message is printed, it will automatically be prefixed by the name of the daemon which is running, the filename where the Dmsg is, and the line number within the file. Some actual examples are: \begin{bVerbatim} Dmsg2(20, "MD5len=\%d MD5=\%s\n", strlen(buf), buf); \end{bVerbatim} \begin{bVerbatim} Dmsg1(9, "Created client \%s record\n", client->hdr.name); \end{bVerbatim} \subsection{Debug Tags} It is possible to use tags to organize debug messages. When using a tag, it is possible to hide/show an entire class of messages. The message tags are defined in \texttt{lib/messages.c/h} \begin{bVerbatim} ... #define DT_SCHEDULER (1<<23) /* scheduler */ #define DT_PROTOCOL (1<<22) /* protocol */ #define DT_DEDUP (1<<21) /* BEEF deduplication */ #define DT_DDE (1<<20) /* BEEF dedup engine */ #define DT_SNAPSHOT (1<<19) /* Snapshot */ #define DT_RECORD (1<<18) /* Record/block */ #define DT_CLOUD (1<<17) /* cloud */ #define DT_ALL (0x7FFF0000) /* all (up to debug_level 65635, 15 flags available) */ /* setdebug tag=all,-plugin */ static struct debugtags debug_tags[] = { ... { NT_("scheduler"), DT_SCHEDULER,_("Debug scheduler information")}, { NT_("protocol"), DT_PROTOCOL, _("Debug protocol information")}, { NT_("snapshot"), DT_SNAPSHOT, _("Debug snapshots")}, { NT_("record"), DT_RECORD, _("Debug records")}, { NT_("all"), DT_ALL, _("Debug all information")}, { NULL, 0, NULL} }; \end{bVerbatim} In the code, the tag can be used like in the following example: \begin{bVerbatim} Dmsg0(DT_CLOUD|50, "This is a message about cloud\n"); \end{bVerbatim} The message will be displayed if the \texttt{cloud} tag is set AND if the debug level is at least 50. \begin{bVerbatim} setdebug level=50 tags=cloud storage \end{bVerbatim} \section{Error Messages} \index{Messages!Error} \index{Error Messages} Error messages are messages that are related to the daemon as a whole rather than a particular job. For example, an out of memory condition my generate an error message. They should be very rarely needed. In general, you should be using Job and Job Queued messages (Jmsg and Qmsg). They are coded as: EmsgN(error-code, level, message, arg1, \ldots{}) As with debug messages, you must explicitly code the of arguments to be substituted in the message. Error-code indicates the severity or class of error, and it may be one of the following (See table \vref{blb:tabdev:errorcodes}: \LTXtable{\linewidth}{t-dev-errorcodes} There are other error message classes, but they are in a state of being redesigned or deprecated, so please do not use them. Some actual examples are: \begin{bVerbatim} Emsg1(M_ABORT, 0, "Cannot create message thread: \%s\n",strerror(status)); \end{bVerbatim} \begin{bVerbatim} Emsg3(M_WARNING, 0, "Connect to File daemon %s at %s:%d failed. Retrying...\n", client->hdr.name, client->address,client->port); \end{bVerbatim} \begin{bVerbatim} Emsg3(M_FATAL, 0, "bdird