\chapter{New Features in 15.0.0} \subsection{Security Enhancements} The password information that might be present in some specific plugin FileSets is now hidden from the \texttt{status client} console output. \smallskip New security events are generated after incorrect network connections. \smallskip The restricted console have now a better control over various bconsole commands such last \textbf{list}, \textbf{restore}, ... \subsection{Storage Daemon Encryption} The Bacula Storage Daemon can now encrypt the data at the volume level to enhance security of data at rest. The volumes cannot be read by a system that doesn't have the correct encryption keys. More information can be found in the Security chapter of the documentation \vref{volumeencryption}. \subsection{Malware Detection} Bacula allows you to configure your jobs to detect known Malware. The detection can be done at the end of the Backup job and/or with a Verify job. The Malware database can be downloaded from different providers, the default is set to abuse.ch. If a Backup job detects a malware in the backup content, an error is reported and the Job status is adapted. \begin{bVerbatim} 20-Sep 12:26 zog8-dir JobId 9: Start Backup JobId 9, Job=backup.2022-09-20_12.26.30_13 ... 20-Sep 12:26 zog8-dir JobId 9: [DI0002] Checking file metadata for Malwares 20-Sep 12:26 zog8-dir JobId 9: Error: [DE0007] Found Malware(s) on JobIds 9 Build OS: x86_64-pc-linux-gnu archlinux JobId: 9 Job: backup.2022-09-20_12.26.30_13 Backup Level: Full ... Last Volume Bytes: 659,912,644 (659.9 MB) Non-fatal FD errors: 1 SD Errors: 0 FD termination status: OK SD termination status: OK Termination: Backup OK -- with warnings \end{bVerbatim} The list of the Malware detected in a given Job can be displayed with the \texttt{list files type=malware} command. \begin{bVerbatim} *list files type=malware jobid=1 +-------+-----------------------------+---------------+----------+ | jobid | filename | description | source | +-------+-----------------------------+---------------+----------+ | 1 | /tmp/regress/build/po/fr.po | Malware found | abuse.ch | +-------+-----------------------------+---------------+----------+ \end{bVerbatim} See the \vref{MalwareDetection} section of this manual for more information. \subsection{TOTP Console Authentication Plugin} The TOTP (Time based One Time Password) Authentication Plugin is compatible with the RFC 6238. Many smartphone apps are available to store the keys and compute the TOTP code. The standard password and encryption mechanisms will still be used to accept an incoming console connection. Once accepted, the Console will prompt for a second level of authentication with a TOTP secret key generated from a shared token. To enable this feature, you needed to install the \texttt{bacula-totp-dir-plugin} package on your Director system, then to set the \texttt{PluginDirectory} directive of the \texttt{Director} resource and configure the \texttt{AuthenticationPlugin} directive of a given restricted Console in the Director configuration file. \begin{bVerbatim} # in bacula-dir.conf Director { Name = myname-dir ... Plugin Directory = /opt/bacula/plugins } Console { Name = "totpconsole" Password = "xxx" Authentication Plugin = "totp" CommandACL = *all* JobACL = *all* ClientAcl = *all* PoolACL = *all* DirectoryACL = *all* WhereACL = *all* FileSetACL = *all* StorageACL = *all* CatalogACL = *all* ScheduleACL = *all* } \end{bVerbatim} The matching Console configuration in \texttt{bconsole.conf} has no extra settings compared to a standard restricted Console. \begin{bVerbatim} # in bconsole.conf Console { Name = totpconsole Password = "xxx" # Same as in bacula-dir.conf/Console } Director { Name = mydir-dir Address = localhost Password = notused } \end{bVerbatim} At the first console connection, if the TLS link is correctly setup (using the shared secret key), the plugin will generate a specific random key for the console and display a QR code in the console output. The user must then scan the QR code with his smartphone using an app such as Aegis (Opensource) or Google Authenticator. The plugin can also be configured to send the QR code via an extranal program. More information can be found in the main manual on \vref{TOTPPlugin}. \subsection{Storage Group Enhancements} The StorageGroup feature is now compatible with Copy and Migration jobs. \subsubsection{FreeSpace Storage Policy} This policy queries each Storage Daemon in the list for its FreeSpace (as a sum of devices specified in the Storage Daemon config) and sort the list by FreeSpace returned. \subsubsection{LastBackedUpTo Storage Policy} This policy ensures that job is backed up to the storage where the same job (of the same level, i.e. Full or Incremental) has been backed up the longest time ago. The goal is to split the jobs to improve redundancy. \subsubsection{LastBackupedToStore Storage Policy} This policy ensures that job is backed up on the storage where the same job (with same level i.e. Full or Incremental) has been backed up to the longest time ago. The goal is to spread the jobs to improve redundancy. \subsubsection{FreeSpaceLeastUsed Storage Policy} This policy ensures that a job is backed up to the storage with the most free space and least running jobs. Within the candidates storages, the least used one will be selected. Candidate storages are determined by the \texttt{StorageGroupPolicyThreshold} Job directive. If MaxFreeSpace is the largest amount of free space for all storages in the group, a storage will be a candidate if its free space is above $$MaxFreeSpace - StorageGroupPolicyThreshold$$. For example: \begin{bVerbatim} with StorageGroupPolicyThreshold=100MB and storages free space being: Storage1 = 500GB free Storage2 = 200GB free Storage3 = 400GB free storage4 = 500GB free In this case MaxFreeSpace=500GB. Storage 1, 4 and 3 are candidates. If 5 jobs are running on Storage1, 2 on Storage4, and 3 on Storage3 then Storage4 will be the selected storage. \end{bVerbatim} Storage Groups can be used as follows (as part of Job and Pool configuration): \begin{bVerbatim} Job { ... Storage = File1, File2, File3 ... } Pool { ... Storage = File4, File5, File6 StorageGroupPolicy = FreeSpaceLeastUsed StorageGroupPolicyThreshold = 200 MB ... } \end{bVerbatim} \subsection{FileDaemon Security Enhancements} \subsubsection{Allowed Backup and Restore Directories} New FileDaemon directives lets the deamon control which client directories are allowed for backup on a per-director basis. Directives can be specified as comma-separated list of directories. The simplest version of the \texttt{AllowedBackupDirectories} and \texttt{ExcludedBackupDirectories} directives may look as follows: \begin{bVerbatim} # in bacula-fd.conf Director { Name = myname-dir ... AllowedBackupDirectories = "/path/to/allowed/directory" } Director { Name = my-other-dir ... ExcludedBackupDirectories = "/path/to/excluded/directory" } \end{bVerbatim} This directive works on the FD side, and is fully independent of the include/exclude part of the Fileset defined in the Director's config file. Nothing is backed up if none of the files defined in the Fileset is inside FD's allowed directory. \subsubsection{Allowed Restore Directories} This new directive lets the deamon control which directories are allowed to be used as a restore destination on a per-director basis. Directive can be specified as a list of directories. The simplest version of the AllowedRestoreDirectories directive may look as follows: \begin{bVerbatim} # in bacula-fd.conf Director { Name = myname-dir ... AllowedRestoreDirectories = "/path/to/directory" } \end{bVerbatim} \subsubsection{Allowed Script Directories} New FileDaemon directive lets the deamon control directories which from the Director can execute client's scripts and programs (e.g. using the Runscript feature or with the Fileset's 'File=' directive). Directive can be specified as a list of directories. The simplest version of the AllowedScriptDirectories directive may look as follows: \begin{bVerbatim} # in bacula-fd.conf Director { Name = myname-dir ... AllowedScriptDirectories = "/path/to/directory" } \end{bVerbatim} When this directive is set, Bacula is also checking programs to be run against set of not-allowed characters. When following resource: \begin{bVerbatim} FileSet { Name = "Fileset_1" Include { File = "\\|/path/to/binary &" } } \end{bVerbatim} is defined inside the Director's config file, Bacula won't backup any file for such Fileset. It's because of the '\&' character, which is not allowed when the 'Allowed Script Directories' is used on the Client's side. Full list of not-allowed characters: \begin{bVerbatim} $ ! ; \ & < > ` ( ) \end{bVerbatim} % $ To disable all command sent by the Director, it is possible to use the following configuration in your FileDaemon: \begin{bVerbatim} AllowedScriptDirectories = none \end{bVerbatim} \subsection{Antivirus Plugin} The FileDaemon Antivirus plugin provides integration between the ClamAV Antivirus daemon and Bacula Verify Jobs, allowing post-backup virus detection within Bacula Enterprise. More information can be found in the Antivirus Plugin user's guide. \subsection{Volume Protection} \paragraph{Warning}: This feature is only for file-based devices. This feature can only be used if Bacula is run as a \btool{systemd} service because only then, with proper capabilities set for the daemon, it's allowed to manage Volume Files' attributes. The `show storage` bconsole command informs if Bacula has needed capabilities. For File-based volumes Bacula will set the \bvalue{Append Only} attribute during the first backup job that uses the new volume. Note that the flag is set when Bacula actually uses volume for the first time, not when the volume is labelled (either automatically or using the `label` command). This will prevent Volumes to lost data by being ovewritten. The \bvalue{Append Only} file attribute is cleared when the volume is being relabeled. Bacula is now also able to set the \bvalue{Immutable} file attribute on a file volume which is marked as Full. Note that as of now, Immutable flag is set only for Full volumes, setting it for other statuses (e.g. 'Used' volumes) may come in the future releases. When a volume is Full and has the \bvalue{Immutable} flag set, it cannot be relabeled and reused until the expiration period elapses. This helps to protect volumes from being reused too early, according to the protection period set. If Volume's filesystem does not support the \bvalue{Append only} or \bvalue{Immutable} flags, a proper warning message is printed in the job log and Bacula proceeds with the usual backup workflow. There are three new directives available to set on a per-device basis to control the the Volume Protection behavior: \begin{bdescription} \item [SetVolumeAppendOnly]: Determines if Bacula should set the \bvalue{Append\_Only} attribute when writing on the volume for the rist time. \item [SetVolumeImmutable]: Determines if Bacula should set the \bvalue{Immutable} attribute when marking volume as 'Full'. \item [Minimum Volume Protection Time]: Specifies how much time has to elapse before Bacula is able to clear the attribute \end{bdescription} Note: Both Append Only and Immutable flags set for volumes cannot be modified using catalog-related commands, e.g. purging a volume won't clear the Immutable flag, that can only be done when MinimumVolumeProtection time expires. If administrator wants to manually remove file attributes, \btool{chattr} must be used: \begin{itemize} \item [chattr -i path-to-vol/VolumeName] - removes the Immutable flag \item [chattr -a path-to-vol/VolumeName] - removes the Append Only flag \end{itemize} This should not be needed, and it should be used only in a state of emergency. chattr tool must be used with user with proper capabilities to set/clear file attributes (e.g. root) In some cases, for example when the status of the Volume is changed by the Director via the \texttt{update volume} command, the Storage Daemon will not be able to change the permission on the Volume. Some Volumes may have the Full/Used status without the proper protection. The command \texttt{update volumeprotect} is designed to determine the list of the volumes that are not protected and connect the Storage Daemon to update the permissions. It can be executed in an Admin job once a day. \begin{bVerbatim} *update Update choice: 1: Volume parameters 2: Pool from resource 3: Slots from autochanger 4: Long term statistics 5: Snapshot parameters 6: Volume protection attributes on Storage Daemon Choose catalog item to update (1-6): 6 Found 1 volumes with status Used/Full that must be protected Connected to Storage "File2" at zog8:8103 with TLS 3000 Marking volume "Vol-0009" as read-only. \end{bVerbatim} or via \texttt{update volumeprotect} \begin{bVerbatim} *update volumeprotect Found 1 volumes with status Used/Full that must be protected Connected to Storage "File2" at zog8:8103 with TLS 3000 Marking volume "Vol-0009" as read-only. \end{bVerbatim} The command can be scheduled in an Admin job \begin{bVerbatim} Job { Name = adm-update-protected Type = Admin Runscript { Console = "update volumeprotect" RunsOnClient = no RunsWhen = Before } JobDefs = DefaultJob } \end{bVerbatim} \subsection{ZStandard FileSet Compression Option} The ZSTD compression algorithm is now available in the FileSet option directive \texttt{Compression}. It is possible to configure ZSTD level 1 \texttt{zstd1}, level 10 \texttt{zstd10} and level 19 \texttt{zstd19}. The default \texttt{zstd} compression is 10. \subsection{Kubernetes Plugin} The Bacula Kubernetes plugin will save all the important Kubernetes resources which make up the application or service. The plugin can now use the CSI Snapshot method to backup persistent volumes. \subsection{Amazon Cloud Driver} A new Amazon Cloud Driver is available for beta testing. In the long term, it will enhance and replace the existing S3 cloud driver. The \texttt{aws} tool provided by Amazon is needed to use this cloud driver. The Amazon cloud driver is available within the bacula-cloud-storage-s3 package and can be enabled by using ``Driver = Amazon'' in the Cloud Storage Daemon resource. \subsection{Misc} \subsubsection{New Storage Daemon Disk Volume Format} Bacula version 15.0 uses a new volume version named \texttt{BB03}. The new format adds the support for Volume Encryption, and the previous 32bits CRC32 checksum was replaced by the faster 64bits \texttt{XXH64}. Volumes written with the BB03 format can only be read by Bacula version 15 or later. Old BB02 volumes can still be restored, and Volumes may start with BB02 blocks, and continue with BB03 blocks. It is not possible to use the \texttt{Volume Encryption=yes} directive on a volume that was labeled using the BB02 format. In that case, the volume will be automatically marked as Used. \subsubsection{Progress Status for Copy/Migration Jobs} The \textbf{status director} command can now report the progress of Copy and Migration Jobs. \subsubsection{Scheduled Job List} The \textbf{status director} bconsole command has been updated to limit the number of scheduled jobs listed by default to 50. The keyword \textbf{limit} can be used to choose how many lines will be printed. \begin{bVerbatim} * status director limit=5 127.0.0.1-dir Version: 14.1.5 (26 October 2022) x86_64-pc-linux-gnu archlinux Daemon started 31-Oct-22 17:27, conf reloaded 31-Oct-2022 17:27:55 Jobs: run=2, running=0 max=4 mode=1,2010 Crypto: fips=no crypto=OpenSSL 1.0.2u 20 Dec 2019 Heap: heap=675,840 smbytes=774,259 max_bytes=2,058,994 bufs=2,470 max_bufs=4,024 Res: njobs=24 nclients=1 nstores=3 npools=1 ncats=1 nfsets=25 nscheds=3 Plugin: ldap totp Scheduled Jobs (5/1440): Level Type Pri Scheduled Job Name Volume =================================================================================== Full Backup 10 31-Oct-22 17:29 NightlySave0 TestVolume001 Full Backup 10 31-Oct-22 17:30 NightlySave1 TestVolume001 Full Backup 10 31-Oct-22 17:31 NightlySave2 TestVolume001 Full Backup 10 31-Oct-22 17:32 NightlySave3 TestVolume001 Full Backup 10 31-Oct-22 17:33 NightlySave4 TestVolume001 5 scheduled Jobs over 1440 are displayed. Use the limit parameter to display more Jobs. ==== Running Jobs: Console connected using TLS at 31-Oct-22 17:27 No Jobs running. ==== Terminated Jobs: JobId Level Files Bytes Status Finished Name ==================================================================== 1 Full 35 5.070 M OK 31-Oct-22 15:50 NightlySave 2 Full 35 5.070 M OK 31-Oct-22 17:27 NightlySave 3 Full 35 5.070 M OK 31-Oct-22 17:28 NightlySave ==== \end{bVerbatim} The APIv2 json output support has been added for the \textbf{status dir scheduled} command. \subsubsection{Catalog Enhancements} \subsubsection*{FileSet Content Overview} Now, the Catalog stores an overview of the FileSet definition in the **Content** field. If the FileSet handles files and directories, the Content field will be set to **files**. If any plugins are used, each plugin will be inserted into the Content field. \begin{bVerbatim} *sql SELECT Content FROM FileSet; +------------------------------+ | content | +------------------------------+ | bpipe | +------------------------------+ \end{bVerbatim} \subsubsection*{New Job Attributes} New SQL attributes have been added to the Job table such as isVirtualFull, Encrypted, LastReadStorageId, WriteStorageId, Rate, CompressRatio, StatusInfo, and so on. \subsubsection*{New Media Attributes} New SQL attributes have been added to the Media table such as **UseProtect**, **Protected**, **VolEncrypted** \subsubsection{Search Function} Bconsole has a new \textbf{.search} command to search into the catalog accross client, job and volumes. \begin{bVerbatim} * .search text=sometext volume= client=sometext-fd job= \end{bVerbatim} The text to be searched must have 4 characters at the minimum. \subsubsection{Advanced Job Queue Control} The Job RunScript feature has been enhanced to control the start of a Job inside the Run Queue. When a Job is starting, the Director controls that resources are available for the Job to start properly, if these resources are not available, the Job will stay in the queue, waiting to acquire them. It is now possible to execute a script to control any kind of external and custom resources and decide when a Job should start. For example, a script might control the load average of a server before to start a Job to find the best execution time. More information can be found in the Director documentation \vref{Director:Job:Runscript}. \subsubsection{Director's PID File timestamp} The Director PID file timestamp is now updated after a successful \textbf{reload} command. External tools can use this information to perform certain actions such as to clear the cache. \subsubsection{RunScript Enhancements} \index[general]{RunScript Enhancements} A new Director RunScript {\bf RunsWhen} keyword of {\bf AtJobCompletion} has been implemented, which runs the command after at the end of job and can update the job status if the command fails. \begin{bVerbatim} Job { ... Runscript { RunsOnClient = no RunsWhen = AtJobCompletion Command = "mail command" AbortJobOnError = yes } } \end{bVerbatim} It has been added because the \textbf{RunsWhen} keyword \textbf{After} was not designed to update the job status if the command fails. \subsubsection{JSON Output} The console has been improved to support a JSON output to list catalog objects and various daemon output. The new ``.jlist'' command is a shortcut of the standard ``list'' command and will display the results in a JSON table. All options and filters of the ``list'' command can be used in the ``.jlist'' command. Only catalog objects are listed with the ``list'' or ``.jlist'' commands. Resources such as Schedule, FileSets, etc... are not handled by the ``list'' command. See the ``help list'' bconsole output for more information about the ``list'' command. The Bacula configuration can be displayed in JSON format with the standard ``bdirjson'', ``bsdjson'', ``bfdjson'' and ``bbconsjson'' tools. \begin{bVerbatim} *.jlist jobs {"type": "jobs", "data":[{"jobid": 1,"job": "CopyJobSave.2021-10-04_18.35.55_03",... \end{bVerbatim} \begin{bVerbatim} *.api 2 api_opts=j *.status dir header {"header":{"name":"127.0.0.1-dir","version":"12.8.2 (09 September 2021)"... \end{bVerbatim} \chapter{New Features in 13.0.0} \subsection{Kubernetes Plugin} The Bacula Kubernetes plugin will save all the important Kubernetes resources which make up the application or service. The plugin has the following features: \begin{bitemize} \item Kubernetes cluster resources configuration backup \item Ability to restore single Kubernetes configuration resource \item Ability to restore Kubernetes resource configuration to local directory \item Kubernetes Persistent Volumes data backup and restore \item Ability to restore Kubernetes Persistent Volumes data to local directory \item Ability to use new Kubernetes CSI driver snapshot features to perform Persistent Volume data backup using the clone method \item Ability to execute user defined commands on required Pod containers to prepare and clean data backup \item Configure Kubernetes workload backup requirements directly with Pod annotations \end{bitemize} More information about the Kubernetes plugin can be found on \vref{kubernetesplugin}. \subsection{Storage Group} It is now possible to configure more than one Storage resource for each Job/Pool. Storage can be specified as a comma separated list of Storage resources to use. Along with specifying a Storage list it is now possible to specify a Storage Group Policy which will be used for determining which Storage will be used for the Job. If no policy is specified, Bacula tries to use the first available Storage from the list. If the first few storage daemons are unavailable due to network problems; broken or unreachable for some other reason, Bacula will use the first one from the list (sorted according to the policy used) which is network reachable and healthy. Currently supported policies are: \begin{itemize} \item [ListedOrder] - This is the default policy. It uses the first available storage from the list provided \item [LeastUsed] - This policy scans all storage daemons from the list and chooses the one with the least number of Jobs currently running \end{itemize} Storage Groups may be used as follows (as a part of Job and Pool resources): \begin{bVerbatim} Job { ... Storage = File1, File2, File3 ... } Pool { ... Storage = File4, File5, File6 StorageGroupPolicy = LeastUsed ... } \end{bVerbatim} When a Job or Pool with a Storage Group is used, the user will observe some messages related to the choice of Storage such as: \begin{bVerbatim} messages 31-maj 19:23 VBox-dir JobId 1: Start Backup JobId 1, Job=StoreGroupJob.2021-05-31_19.23.36_03 31-maj 19:23 VBox-dir JobId 1: Possible storage choices: "File1, File2" 31-maj 19:23 VBox-dir JobId 1: Storage daemon "File1" didn't accept Device "FileChgr1-Dev1" because: 3924 Device "FileChgr1-Dev1" not in SD Device resources or no matching Media Type or is disabled. 31-maj 19:23 VBox-dir JobId 1: Selected storage: File2, device: FileChgr2-Dev1, StorageGroupPolicy: "ListedOrder" \end{bVerbatim} \subsection{New Accurate Option to Save Only File's Metadata} The new 'o' Accurate directive option for a Fileset has been added to save only the metadata of a file when possible. \smallskip{} The new 'o' option should be used in conjunction with one of the signature checking options (1, 2, 3, or 5). When the 'o' option is specified, the signature is computed only for files that have one of the other accurate options specified triggering a backup of the file (for example an inode change, a permission change, etc...). \smallskip{} In cases where only the file's metadata has changed (ie: the signature is identical), only the file's attributes will be backed up. If the file's data has been changed (hence a different singature), the file will be backed up in the usual way (attributes as well as the file's contents will be saved on the volume). \smallskip{} For example: \begin{bVerbatim} Job { Name = JobTest JobDefs = DefaultJob FileSet = TestFS Accurate = yes } FileSet { Name = TestFS Options { Signature = MD5 Accurate = pino5 } File = / } \end{bVerbatim} The backup job will compare permission bits, inodes, number of links and, if any of it changes, it will also compute file's signature to verify if only the metadata must be backed up or if the full file must be saved. \subsection{External LDAP Console Authentication} The new Bacula Plugable Authentication Module (BPAM) API framework introduced in Bacula 13.0 comes with the first plugin which handles user authentication against any LDAP Directory Server (including OpenLDAP and Active Directory). To enable the build of the LDAP Console Authentication plugin, add the \texttt{ldap} options to your \texttt{./configure} command line as shown below: \begin{bVerbatim} ./configure --with-ldap --enable-ldap-bpam [... other options] \end{bVerbatim} \begin{bVerbatim} # bconsole *status director ... Plugin: ldap-dir.so ... \end{bVerbatim} When the LDAP plugin is loaded you can configure a named console resource to use LDAP to authenticate users. BConsole will prompt for a User and Password and it will be verified by the Director. TLS PSK (activated by default) is recommended. To use this plugin, you have to specify the \texttt{PluginDirectory} Director resource directive, and add a new console resource directive \texttt{Authentication Plugin} as shown below: \begin{bVerbatim} Director { ... Plugin Directory = /opt/bacula/plugins } Console { Name = "ldapconsole" Password = "xxx" # New directive Authentication Plugin = "ldap:" ... } \end{bVerbatim} where \texttt{parameters} are the space separated list of one or more plugin parameters: \begin{itemize} \item [url] - LDAP Directory service location, i.e. "url=ldap://10.0.0.1/" \item [binddn] - DN used to connect to LDAP Directory service to perform required \texttt{query} \item [bindpass] - DN password used to connect to LDAP Directory service to perform required \texttt{query} \item [query] - A query performed on the LDAP Directory serice to find user for authentication The \texttt{query} string is composed as \texttt{/}. Where `\texttt{} is a DN search starting point and \texttt{} is a standard LDAP search object filter which support dynamic string substitution: \texttt{\%u} will be replaced by credential's username and \texttt{\%p} by credential's password, i.e. \texttt{query=dc=bacula,dc=com/(cn=\%u)}. \item [starttls] - Instruct the BPAM LDAP Plugin to use the \texttt{StartTLS} extension if the LDAP Directory service supports it and fallback to no TLS if this extension is not available. \item [starttlsforce] - Does the same as the `starttls` setting does but reports error on fallback. \end{itemize} Working configuration examples: \smallskip bacula-dir.conf - Console resource configuration for BPAM LDAP Plugin with OpenLDAP authentication example. \begin{bVerbatim} Console { Name = "bacula_ldap_console" Password = "xxx" # New directive (on a single line) Authentication Plugin = "ldap: url=ldap://ldapsrv/ binddn=cn=root,dc=bacula,dc=com bindpass=secret query=dc=bacula,dc=com/(cn=%u) starttls" ... } \end{bVerbatim} bacula-dir.conf - Console resource configuration for BPAM LDAP Plugin with Active Directory authentication example. \begin{bVerbatim} Console { Name = "bacula_ad_console" Password = "xxx" # New directive (on a single line) Authentication Plugin = "ldap: url=ldaps://ldapsrv/ binddn=cn=bacula,ou=Users,dc=bacula,dc=com bindpass=secret query=dc=bacula,dc=com/(&(objectCategory=person)(objectClass=user)(sAMAccountName=%u))" ... } \end{bVerbatim} \subsection{New Baculum features} The following features were introduced in version 11.0.6 \subsubsection{New Advanced Schedule Settings} The schedule function has been reworked to support all possible schedule settings. Now it can be configured in five different ways: hourly, daily, weekly, monthly, and also custom for various uncommon settings. \subsubsection{New Copy and Migrate Job Wizards} Using the new wizards it is possible to easily create new copy and migrate jobs. The Wizard's steps are prepared and optimized to make the process as easy as possible. \subsubsection{New Director Page} This new page provides two major functions: The Director status and the Director configuration. The first one is for displaying in a graphical and textual way the Director status. The second one is for configuring all Director resources (Jobs, Clients, Pools, Storage ...etc.). From one place users can configure various Director resources. \subsubsection{Bulk Delete Volumes Action} This feature is available on the media page. Apart from prune and purge bulk actions, now users are also able to bulk delete volumes. It is possible by selecting volumes in the volume table and using this new delete volumes action. \subsubsection{Loading Pages Optimization} Pages have been optimized to faster loading. The entire Baculum Web interface works faster. \subsubsection{Directive Documentation} In all Bacula component configurations, for almost all directives, official Bacula documentation has been added. It is available directly within the web interface. \subsection{Tag Support} It is now possible to assign custom Tags to various catalog records in Bacula such as: \begin{itemize} \item Volume \item Client \item Job \end{itemize} \begin{bVerbatim} *tag Automatically selected Catalog: MyCatalog Using Catalog "MyCatalog" Available Tag operations: 1: Add 2: Delete 3: List Select Tag operation (1-3): 1 Available Tag target: 1: Client 2: Job 3: Volume Select Tag target (1-3): 1 The defined Client resources are: 1: 127.0.0.1-fd 2: test1-fd 3: test2-fd 4: test-rst-fd 5: test-bkp-fd Select Client (File daemon) resource (1-5): 1 Enter the Tag value: test1 1000 Tag added *tag add client=127.0.0.1-fd name=#important" 1000 Tag added *tag list client +--------------+----------+--------------+ | tag | clientid | client | +--------------+----------+--------------+ | #tagviamenu3 | 1 | 127.0.0.1-fd | | test1 | 1 | 127.0.0.1-fd | | #tagviamenu2 | 1 | 127.0.0.1-fd | | #tagviamenu1 | 1 | 127.0.0.1-fd | | #important | 1 | 127.0.0.1-fd | +--------------+----------+--------------+ *tag list client name=#important +----------+--------------+ | clientid | client | +----------+--------------+ | 1 | 127.0.0.1-fd | +----------+--------------+ \end{bVerbatim} It is possible to assign Tags to a Job record with the new 'Tag' directive in a Job resource. \begin{bVerbatim} Job { Name = backup ... Tag = "#important", "#production" } \end{bVerbatim} \begin{bVerbatim} *tag list job +--------------+-------+-------------+ | tag | jobid | job | +--------------+-------+-------------+ | #important | 2 | backup | | #production | 2 | backup | +--------------+-------+-------------+ \end{bVerbatim} The Tags are also accessible from various BWeb Management Suite screens. (See \vref{tagbweb}). \subsection{Support for SHA256 and SHA512 in FileSet} The support for strong signature algorithms SHA256 and SHA512 has been added to Verify Jobs. It is now possible to check if data generated by a Job that uses an SHA256 or SHA512 signature is valid. \begin{bVerbatim} FileSet { Name = Linux-Etc Options { Signature = SHA512 Verify = pins3 } File = /etc } \end{bVerbatim} In the \texttt{FileSet} \bdirectivename{Verify} option directive, the following code has been added: \begin{itemize} \item [3] for SHA512 \item [2] for SHA256 \end{itemize} \subsection{Windows Installer Silent Mode Enhancement} The following command line options may be used to control the regular Bacula installer values in silent mode: \begin{itemize} \item -ConfigClientName \item -ConfigClientPort \item -ConfigClientPassword \item -ConfigClientMaxJobs \item -ConfigClientInstallService \item -ConfigClientStartService \item -ConfigStorageName \item -ConfigStoragePort \item -ConfigStorageMaxJobs \item -ConfigStoragePassword \item -ConfigStorageInstallService \item -ConfigStorageStartService \item -ConfigDirectorName \item -ConfigDirectorPort \item -ConfigDirectorMaxJobs \item -ConfigDirectorPassword \item -ConfigDirectorDB \item -ConfigDirectorInstallService \item -ConfigDirectorStartService \item -ConfigMonitorName \item -ConfigMonitorPassword \end{itemize} The following options control the components that will be installed: \begin{itemize} \item -ComponentFile \item -ComponentStorage \item -ComponentTextConsole \item -ComponentBatConsole \item -ComponentTrayMonitor \item -ComponentAllDrivesPlugin \item -ComponentWinBMRPlugin \item -ComponentCDPPlugin \end{itemize} \subsubsection*{Example} \begin{bVerbatim} bacula-win64-13.0.0.exe /S -ComponentFile \ -ConfigClientName foo -ConfigClientPassword bar \end{bVerbatim} Will install only file deamon with bacula-fd.conf configured. \begin{bVerbatim} bacula-win64-13.0.0.exe /S -ComponentStorage \ -ComponentFile-ConfigClientName foo \ -ConfigClientPassword bar -ConfigStorageName foo2 \ -ConfigStoragePassword bar2 \end{bVerbatim} Will install the Storage Deamon plus File Deamon with bacula-sd.conf and bacula-fd.conf configured. \subsection{New Message Identification Format} We are starting to add unique message indentifiers to each message (other than debug and the Job report) that Bacula prints. At the current time only two files in the Storage Daemon have these message identifiers and over time with subsequent releases we will modify all messages. \smallskip The message identifier will be kept unique for each message and once assigned to a message it will not change even if the text of the message changes. This means that the message identifier will be the same no matter what language the text is displayed in, and more importantly, it will allow us to make listing of the messages with in some cases, an additional explanation or instructions on how to correct the problem. All this will take several years since it is a lot of work and requires some new programs that are not yet written to manage these message identifiers. \smallskip The format of the message identifier is: \begin{bVerbatim} [AAnnnn] \end{bVerbatim} where A is an upper case character and nnnn is a four digit number, where the first character indicates the software component (daemon); the second letter indicates the severity, and the number is unique for a given component and severity. \smallskip For example: \begin{bVerbatim} [SF0001] \end{bVerbatim} The first character representing the component at the current time one of the following: \begin{bVerbatim} S Storage daemon D Director F File daemon \end{bVerbatim} \smallskip The second character representing the severity or level can be: \begin{bVerbatim} A Abort F Fatal E Error W Warning S Security I Info D Debug O OK (i.e. operation completed normally) \end{bVerbatim} So in the example above [SF0001] indicates it is a message id, because of the brackets and because it is at the beginning of the message, and that it was generated by the Storage Daemon as a fatal error. \smallskip As mentioned above it will take some time to implement these message ids everywhere, and over time we may add more component letters and more severity levels as needed. \subsection{Misc} \subsubsection{Plugin Object Status Support} The Object table has now a ObjectStatus field that can be used by plugins to report more precise information about the backup of an Object (generated by plugins). \subsubsection{Network Buffer Management} The new \texttt{SDPacketCheck} FileDaemon directive can be used to control the network flow in some specific use cases. See \vref{FileDaemon:FileDaemon:SDPacketCheck} for more information. \subsubsection{IBM Lintape Driver (BETA)} The new \texttt{Use Lintape} Storage Daemon directive has been added to support the Lintape Kernel driver. See \vref{Storage:Device:UseLintape} for more information. \chapter{New Features in 11.0.0} \section{Catalog Performance Improvements} There is a new \bacula{} database format (schema) in this version of \bacula{} that eliminates the FileName table by placing the Filename into the File record of the File table. This substantiallly improves performance, particularly for large databases. The \btool{update\_xxx\_catalog} script will automatically update the \bacula{} database format, but you should realize that for very large databases (greater than 50GB), it may take some time and it will double the size of the database on disk during the migration. \smallskip This database format change can provide very significant improvements in the speed of metadata insertion into the database, and in some cases (backup of large email servers) can significantly reduce the size of the database. \section{Automatic TLS Encryption} Starting with Bacula 11.0, all daemons and consoles are now using TLS automatically for all network communications. It is no longer required to setup TLS keys in advance. It is possible to turn off automatic TLS PSK encryption using the \bdirectivename{TLS PSK Enable} directive. \section{Client Behind NAT Support with the \bdirectivename{Connect To Director} Directive} A Client can now initiate a connection to the Director (permanently or scheduled) to allow the Director to communicate to the Client when a new Job is started or a bconsole command such as \bcommandname{status client} or \bcommandname{estimate} is issued. \smallskip{} This new network configuration option is particularly useful for Clients that are not directly reachable by the Director. \bimageH%[0.9\linewidth] {client-behind-nat}{Client Behind NAT Example}{fig:clientbehindnat} \begin{bVerbatim} # cat /opt/bacula/etc/bacula-fd.conf Director { Name = bac-dir Password = aigh3wu7oothieb4geeph3noo # Password used to connect # New directives Address = bac-dir.mycompany.com # Director address to connect Connect To Director = yes # FD will call the Director } # cat /opt/bacula/etc/bacula-dir.conf Client { Name = bac-fd Password = aigh3wu7oothieb4geeph3noo # New directive Allow FD Connections = yes } \end{bVerbatim} \smallskip{} It is possible to schedule the Client connenction at certain periods of the day: \begin{bVerbatim} # cat /opt/bacula/etc/bacula-fd.conf Director { Name = bac-dir Password = aigh3wu7oothieb4geeph3noo # Password used to connect # New directives Address = bac-dir.mycompany.com # Director address to connect Connect To Director = yes # FD will call the Director Schedule = WorkingHours } Schedule { Name = WorkingHours # Connect the Director between 12:00 and 14:00 Connect = MaxConnectTime=2h on mon-fri at 12:00 } \end{bVerbatim} Note that in the current version, if the File Daemon is started after 12:00, the next connection to the Director will occur at 12:00 the next day. A Job can be scheduled in the Director around 12:00, and if the Client is connected, the Job will be executed as if the Client was reachable from the Director. \section{Continuous Data Protection Plugin} Continuous Data Protection (CDP), also called continuous backup or real-time backup, refers to backup of Client data by automatically saving a copy of every change made to that data, essentially capturing every version of the data that the user saves. It allows the user or administrator to restore data to any point in time. \bimageH%[0.9\linewidth] {cdp}{CDP Example}{fig:cdp} The Bacula CDP feature is composed of two components: An application (\texttt{cdp-client} or \texttt{tray-monitor}) that will monitor a set of directories configured by the user, and a \bacula{} FileDaemon plugin responsible to secure the data using \bacula{} infrastructure. The user application (\texttt{cdp-client} or \texttt{tray-monitor}) is responsible for monitoring files and directories. When a modification is detected, the new data is copied into a \textsl{spool} directory. At a regular interval, a \bacula{} backup job will contact the FileDaemon and will save all the files archived by the \texttt{cdp-client}. The locally copied data can be restored at any time without a network connection to the Director. See the CDP (Continious Data Protection) chapter \vref{blb:cdp} for more information. \subsubsection{Global \bdirectivename{Autoprune} Control Directive} The \bDirector{} \bdirectivename{Autoprune} directive can now globally control the Autoprune feature. This directive will take precedence over \bresourcename{Pool} or \bresourcename{Client} \bdirectivename{Autoprune} directives. \begin{bVerbatim} Director { Name = mydir-dir ... AutoPrune = no # switch off Autoprune globally } \end{bVerbatim} \section{Event and Auditing} The Director daemon can now record events such as: \begin{bitemize} \item Console connection/disconnection \item Daemon startup/shutdown \item Command execution \item ... \end{bitemize} The events may be stored in a new catalog table, to disk, or sent via syslog. \begin{bVerbatim} Messages { Name = Standard catalog = all, events append = /opt/bacula/working/bacula.log = all, !skipped append = /opt/bacula/working/audit.log = events, !events.bweb } Messages { Name = Daemon catalog = all, events append = /opt/bacula/working/bacula.log = all, !skipped append = /opt/bacula/working/audit.log = events, !events.bweb append = /opt/bacula/working/bweb.log = events.bweb } \end{bVerbatim} The new message category ``events'' is not included in the default configuration files by default. It is possible to filter out some events using "!events." form. It is possible to specify 10 custom events per Messages resource. All event types are recorded by default. \smallskip When stored in the catalog, the events can be listed with the ``list events'' command. \begin{bVerbatim} * list events [type= | limit= | order= | days= | start= | end=] +---------------------+------------+-----------+--------------------------------+ | time | type | source | event | +---------------------+------------+-----------+--------------------------------+ | 2020-04-24 17:04:07 | daemon | *Daemon* | Director startup | | 2020-04-24 17:04:12 | connection | *Console* | Connection from 127.0.0.1:8101 | | 2020-04-24 17:04:20 | command | *Console* | purge jobid=1 | +---------------------+------------+-----------+--------------------------------+ \end{bVerbatim} The \texttt{.events} command can be used to record an external event. The source recorded will be recorded as ``**source**''. The events type can have a custom name. \begin{bVerbatim} * .events type=baculum source=joe text="User login" \end{bVerbatim} \section{New Prune Command Option} The \bcommandname{prune jobs all} command will query the catalog to find all combinations of Client/Pool, and will run the pruning algorithm on each of them. At the end, all files and jobs not needed for restore that have passed the relevant retention times will be pruned. \smallskip The \bcommandname{prune} command \bcommandname{prune jobs all yes} can be scheduled in a RunScript to prune the catalog once per day for example. All Clients and Pools will be analyzed automatically. \begin{bVerbatim} Job { ... RunScript { Console = "prune jobs all yes" RunsWhen = Before failjobonerror = no runsonclient = no } } \end{bVerbatim} \subsubsection{Dynamic Client Address Directive} It is now possible to use a script to determine the address of a Client when dynamic DNS option is not a viable solution: \begin{bVerbatim} Client { Name = my-fd ... Address = "|/opt/bacula/bin/compute-ip my-fd" } \end{bVerbatim} The command used to generate the address should return one single line with a valid address and end with the exit code~0. An example would be \begin{bVerbatim} Address = "|echo 127.0.0.1" \end{bVerbatim} This option might be useful in some complex cluster environments. \section{Volume Retention Enhancements} The Pool/Volume parameter \texttt{Volume Retention} can now be disabled to never prune a volume based on the \texttt{Volume Retention} time. When \texttt{Volume Retention} is disabled, only the \texttt{Job Retention} time will be used to prune jobs. \begin{bVerbatim} Pool { Volume Retention = 0 ... } \end{bVerbatim} \section{Windows Enhancements} \begin{bitemize} \item Support for Windows files with non-UTF16 names. \item Snapshot management has been improved, and a backup job now relies exclusively on the snapshot tree structure. \item Support for the \texttt{system.cifs\_acl} extended attribute backup with Linux \acs{CIFS} has been added. It can be used to backup Windows security attributes from a CIFS share mounted on a Linux system. Note that only recent Linux kernels can handle the \texttt{system.cifs\_acl} feature correctly. The FileSet must use the \texttt{XATTR Support=yes} option, and the CIFS share must be mounted with the \texttt{cifsacl} options. See mount.cifs(8) for more information. \end{bitemize} \section{GPFS ACL Support} The new Bacula FileDaemon supports the GPFS filesystem specific ACL. The GPFS libraries must be installed in the standard location. To know if the GPFS support is available on your system, the following commands can be used. \begin{bVerbatim} *setdebug level=1 client=stretch-amd64-fd Connecting to Client stretch-amd64-fd at stretch-amd64:9102 2000 OK setdebug=1 trace=0 hangup=0 blowup=0 options= tags= *st client=stretch-amd64-fd Connecting to Client stretch-amd64-fd at stretch-amd64:9102 stretch-amd64-fd Version: 11.0.0 (01 Dec 2020) x86_64-pc-linux-gnu-bacula debian 9.11 Daemon started 21-Jul-20 14:42. Jobs: run=0 running=0. Ulimits: nofile=1024 memlock=65536 status=ok Heap: heap=135,168 smbytes=199,993 max_bytes=200,010 bufs=104 max_bufs=105 Sizes: boffset_t=8 size_t=8 debug=1 trace=0 mode=0,2010 bwlimit=0kB/s Crypto: fips=no crypto=OpenSSL 1.0.2u 20 Dec 2019 APIs: GPFS Plugin: bpipe-fd.so(2) \end{bVerbatim} The \texttt{APIs} line will indicate if the \texttt{/usr/lpp/mmfs/libgpfs.so} was loaded at the start of the Bacula FD service or not. The standard \texttt{ACL Support} (cf \vref{blb:ACLSupport}) directive can be used to enable automatically the support for the GPFS ACL backup. \section{New Baculum Features} \subsection{Multi-user interface improvements} There have been added new functions and improvements to the multi-user interface and restricted access. The Security page has new tabs: \begin{bitemize} \item Console ACLs \item OAuth2 clients \item API hosts \end{bitemize} These new tabs help to configure OAuth2 accounts, create restricted Bacula Console for users and create API hosts. They ease the process of creating users with a restricted Bacula resources access. \subsection{Add searching jobs by filename in the restore wizard} In the restore wizard now is possible to select job to restore by filename of file stored in backups. There is also possible to limit results to specific path. \subsection{Show more detailed job file list} The job file list now displays file details like: file attributes, UID, GID, size, mtime and information if the file record for saved or deleted file. \subsection{Add graphs to job view page} On the job view page, new pie and bar graphs for selected job are available. \subsection{Implement graphical status storage} On the storage status page are available two new types of the status (raw and graphical). The graphical status page is modern and refreshed asynchronously. \subsection{Add Russian translations} \subsection{Global messages log window} There has been added new window to browse Bacula logs in a friendly way. \subsection{Job status weather} Add the job status weather on job list page to express current job condition. \subsection{Restore wizard improvements} In the restore wizard has been added listing and browsing names encoded in non-UTF encoding. \subsection{New API endpoints} \begin{bitemize} \item /oauth2/clients \item /oauth2/clients/{client\_id} \item /jobs/files \end{bitemize} \subsection{New parameters in API endpoints} \begin{bitemize} \item /jobs/{jobid}/files - 'details' parameter \item /storages/show - 'output' parameter \item /storages/{storageid}/show - 'output' parameter \end{bitemize} \chapter{New Features in 9.6.0} \subsection{Building 9.6.4 and later} Version 9.6.4 is a major security and bug fix release. We suggest everyone to upgrade as soon as possible.

One significant improvement in this version is for the AWS S3 cloud driver. First the code base has been brought much closer to the Enterprise version (still a long ways to go). Second major change is that the community code now uses the latest version of libs3 as maintained by Bacula Systems. The libs3 code is available as a tar file for Bacula version 9.6.4 at:

\elink{http://www.bacula.org/downloads/libs3-20200523.tar.gz}{http://www.bacula.org/downloads/libs3-20200523.tar.gz}

Note: Version 9.6.4 must be compiled with the above libs3 version or later. To build libs3: \begin{bVerbatim} - Remove any libs3 package loaded by your OS - Download above link - tar xvfz libs3-20200523.tar.gz - cd libs3-20200523 - make # should have no errors - sudo make install \end{bVerbatim}

Then when you do your Bacula ./configure it should automatically detect and use the libs3. The output from the ./configure will show whether or not libs3 was found during the configuration. E.g.

\begin{bVerbatim} S3 support: yes \end{bVerbatim}

in the output from ./configure. \subsection{Docker Plugin} Containers is a relatively new system level virtualization concept that has less overhead than traditional virtualation. This is true because Container use the underlying operating system to provide all the needed services thus eliminating the need for multiple operating systems. Docker containers rely on sophisticated file system level data abstraction with a number of read-only images to create templates used for container initialization. With its Docker Plugin, the Bacula will save the full container image including all read-only and writable layers into a single image archive. It is not necessary to install a Bacula File daemon in each container, so each container can be backed up from a common image repository. The Bacula Docker Plugin will contact the Docker service to read and save the contents of any system image or container image using snapshots (default behavior) and dump them using the Docker~API. The Docker Plugin whitepaper provides more detailed information. \subsection{Real-Time Statistics Monitoring} \label{CollectCommand} All Bacula daemons can now collect internal performance statistics periodically and provide mechanisms to store the values to a CSV file or to send the values to a Graphite daemon via the network. Graphite is an enterprise-ready monitoring tool (\url{https://graphiteapp.org}). \begin{figure}[htbp] \centering \includegraphics[width=0.47\linewidth]{graphite1} \includegraphics[width=0.47\linewidth]{graphite2} \caption{Graphite Examples} \label{fig:graphite} \end{figure} \smallskip To activate the statistic collector feature, simply define a \textbf{Statistics} resource in the daemon of your choice: \begin{bVerbatim} Statistics { Name = "Graphite" Type = Graphite # Graphite host information Host = "localhost" Port = 2003 } \end{bVerbatim} It is possible to change the interval that is used to collect the statistics with the \textbf{Interval} directive (\textbf{5 mins} by default), and use the \textbf{Metrics} directive to select the data to collect (\textbf{all} by default). \smallskip If the Graphite daemon cannot be reached, the statistics data are spooled on disk and are sent automatically when the Graphite daemon is available again. \smallskip The bconsole \textbf{statistics} command can be used to display the current statistics in various formats (text or json for now). \begin{bVerbatim} *statistics Statistics available for: 1: Director 2: Storage 3: Client Select daemon type for statistics (1-3): 1 bacula.dir.config.clients=1 bacula.dir.config.jobs=3 bacula.dir.config.filesets=2 bacula.dir.config.pools=3 bacula.dir.config.schedules=2 ... *statistics storage ... bacula.storage.bac-sd.device.File1.readbytes=214 bacula.storage.bac-sd.device.File1.readtime=12 bacula.storage.bac-sd.device.File1.readspeed=0.000000 bacula.storage.bac-sd.device.File1.writespeed=0.000000 bacula.storage.bac-sd.device.File1.status=1 bacula.storage.bac-sd.device.File1.writebytes=83013529 bacula.storage.bac-sd.device.File1.writetime=20356 ... \end{bVerbatim} The \textbf{statistics} bconsole command can accept parameters to be scripted, for example it is possible to export the data in JSON, or to select which metrics to display. \begin{bVerbatim} *statistics bacula.dir.config.clients bacula.dir.config.jobs json [ { "name": "bacula.dir.config.clients", "value": 1, "type": "Integer", "unit": "Clients", "description": "The number of defined clients in the Director." }, { "name": "bacula.dir.config.jobs", "value": 3, "type": "Integer", "unit": "Jobs", "description": "The number of defined jobs in the Director." } ] \end{bVerbatim} The \textbf{.status statistics} command can be used to query the status of the Statistic collector thread. \begin{bVerbatim} *.status dir statistics Statistics backend: Graphite is running type=2 lasttimestamp=12-Sep-18 09:45 interval=300 secs spooling=in progress lasterror=Could not connect to localhost:2003 Err=Connection refused Update Statistics: running interval=300 secs lastupdate=12-Sep-18 09:45 * \end{bVerbatim} \chapter{New Features in 9.4.0} \subsubsection{Cloud Backup} A major problem of Cloud backup is that data transmission to and from the Cloud is very slow compared to traditional backup to disk or tape. The Bacula Cloud drivers provide a means to quickly finish the backups and then to transfer the data from the local cache to the Cloud in the background. This is done by first splitting the data Volumes into small parts that are cached locally then uploading those parts to the Cloud storage service in the background, either while the job continues to run or after the backup Job has terminated. Once the parts are written to the Cloud, they may either be left in the local cache for quick restores or they can be removed (truncate cache). \subsubsection{Cloud Volume Architecture} \begin{figure}[htbp] \centering \includegraphics[width=0.8\linewidth]{nativeCloudStorage-diagram} \caption{Bacula Cloud Architecture} \label{fig:cloud0} \end{figure} The picture shown above shows two Volumes (Volume0001 and Volume0002) with their parts in the cache. Below the cache, one can see that Volume0002 has been uploaded or synchronized with the Cloud. \smallskip Note: Regular Bacula disk Volumes are implemented as standard files that reside in the user defined \textbf{Archive Directory}. On the other hand, Bacula Cloud Volumes are directories that reside in the user defined \textbf{Archive Directory}. Each Cloud Volume's directory contains the cloud Volume parts which are implemented as numbered files (part.1, part.2, ...). \subsubsection{Cloud Restore} During a restore, if the needed parts are in the local cache, they will be immediately used, otherwise, they will be downloaded from the Cloud as needed. The restore starts with parts already in the local cache but will wait in turn for any part that needs to be downloaded. The Cloud part downloads proceed while the restore is running. \smallskip With most Cloud providers, uploads are usually free of charge, but downloads of data from the Cloud are billed. By using local cache and multiple small parts, you can configure Bacula to substantially reduce download costs. \smallskip The \texttt{MaximumFileSize} Device directive is still valid within the Storage Daemon and defines the granularity of a restore chunk. In order to limit volume parts to download during restore (specially when restoring single files), it might be useful to set the \texttt{MaximumFileSize} to a value smaller than or equal to the \texttt{MaximumPartSize}. \subsubsection{Compatibility} Since a Cloud Volume contains the same data as an ordinary Bacula Volume, all existing types of Bacula data may be stored in the cloud -- that is client encrypted, compressed data, plugin data, etc. All existing Bacula functionality, with the exception of deduplication, is available with the Bacula Cloud drivers. \subsubsection{Deduplication and the Cloud} At the current time, Bacula Global Endpoint Backup does not support writing to the cloud because the cloud would be too slow to support large hashed and indexed containers of deduplication data. \subsubsection{Virtual Autochangers and Disk Autochangers} If you use a Bacula Virtual Autochanger you will find it compatible with the new Bacula Cloud drivers. However, if you use a third party disk autochanger script such as \textbf{Vchanger}, unless or until it is modified to handle Volume directories, it may not be compatible with Bacula Cloud drivers. \subsubsection{Security} All data that is sent to and received from the cloud by default uses the HTTPS protocol, so your data is encrypted while being transmitted and received. However, data that resides in the Cloud is not encrypted by default. If you wish extra security of your data while it resides in the cloud, you should consider using Bacula's PKI data encryption feature during the backup. \subsubsection{Cache and Pruning} The Cache is treated much like a normal Disk based backup, so that in configuring Cloud the administrator should take care to set "Archive Device" in the Device resource to a directory where he/she would normally start data backed up to disk. Obviously, unless he/she uses the truncate/prune cache commands, the Archive Device will continue to fill. \smallskip The cache retention can be controlled per Volume with the ``CacheRetention'' attribute. The default value is 0, meaning that the pruning of the cache is disabled. \smallskip The ``CacheRetention'' value for a volume can be modified with the ``update'' command or via the Pool directive ``CacheRetention'' for newly created volumes. \subsection{New Commands, Resource, and Directives for Cloud} To support Cloud, there are new \textbf{bconsole} commands, new Storage Daemon directives and a new Cloud resource that is specified in the Storage Daemon's Device resource. \subsubsection{New Cloud Bconsole Commands} \begin{itemize} \item \textbf{Cloud} The new cloud \textbf{bconsole} command allows you to do a number of things with cloud volumes. The options are the following: \begin{itemize} \item None. If you specify no arguments to the command, bconsole will prompt with: \begin{bVerbatim} Cloud choice: 1: List Cloud Volumes in the Cloud 2: Upload a Volume to the Cloud 3: Prune the Cloud Cache 4: Truncate a Volume Cache 5: Done Select action to perform on Cloud (1-5): \end{bVerbatim} The different choices should be rather obvious. \item \textbf{Truncate} This command will attempt to truncate the local cache for the specified Volume. Bacula will prompt you for the information needed to determine the Volume name or names. To avoid the prompts, the following additional command line options may be specified: \begin{itemize} \item \textbf{Storage=xxx} \item \textbf{Volume=xxx} \item \textbf{AllPools} \item \textbf{AllFromPool} \item \textbf{Pool=xxx} \item \textbf{MediaType=xxx} \item \textbf{Drive=xxx} \item \textbf{Slots=nnn} \end{itemize} \item \textbf{Prune} This command will attempt to prune the local cache for the specified Volume. Bacula will respect the CacheRetention volume attribute to determine if the cache can be truncated or not. Only parts that are uploaded to the cloud will be deleted from the cache. Bacula will prompt you for the information needed to determine the Volume name or names. To avoid the prompts, the following additional command line options may be specified: \begin{itemize} \item \textbf{Storage=xxx} \item \textbf{Volume=xxx} \item \textbf{AllPools} \item \textbf{AllFromPool} \item \textbf{Pool=xxx} \item \textbf{MediaType=xxx} \item \textbf{Drive=xxx} \item \textbf{Slots=nnn} \end{itemize} \item \textbf{Upload} This command will attempt to upload the specified Volumes. It will prompt you for the information needed to determine the Volume name or names. To avoid the prompts, you may specify any of the following additional command line options: \begin{itemize} \item \textbf{Storage=xxx} \item \textbf{Volume=xxx} \item \textbf{AllPools} \item \textbf{AllFromPool} \item \textbf{Pool=xxx} \item \textbf{MediaType=xxx} \item \textbf{Drive=xxx} \item \textbf{Slots=nnn} \end{itemize} \item \textbf{List} This command will list volumes stored in the Cloud. If a volume name is specified, the command will list all parts for the given volume. To avoid the prompts, you may specify any of the following additional command line options: \begin{itemize} \item \textbf{Storage=xxx} \item \textbf{Volume=xxx} \item \textbf{Storage=xxx} \end{itemize} \end{itemize} \end{itemize} \subsubsection{Cloud Additions to the DIR Pool Resource} Within the \textbf{bacula-dir.conf} file each \textbf{Pool} resource there is an additional keyword \textbf{CacheRetention} that can be specified. \subsubsection{Cloud Additions to the SD Device Resource} Within the \textbf{bacula-sd.conf} file each \textbf{Device} resource there is an additional keyword \textbf{Cloud} that must be specified on the \textbf{Device Type} directive, and two new directives \textbf{Maximum Part Size} and \textbf{Cloud}. \subsubsection{New Cloud SD Device Directives} \begin{itemize} \item \textbf{Device Type} The Device Type has been extended to include the new keyword \textbf{Cloud} to specify that the device supports cloud Volumes. Example: \begin{bVerbatim} Device Type = Cloud \end{bVerbatim} \item \textbf{Cloud} The new Cloud directive permits specification of a new Cloud Resource. As with other Bacula resource specifications, one specifies the name of the Cloud resource. Example: \begin{bVerbatim} Cloud = S3Cloud \end{bVerbatim} \item \textbf{Maximum Part Size} This directive allows one to specify the maximum size for each part. Smaller part sizes will reduce restore costs, but may require a small additional overhead to handle multiple parts. The maximum number of parts permitted in a Cloud Volume is 524,288. The maximum size of any given part is approximately 17.5TB. \end{itemize} \subsubsection{Example Cloud Device Specification} An example of a Cloud Device Resource might be: \begin{bVerbatim} Device { Name = CloudStorage Device Type = Cloud Cloud = S3Cloud Archive Device = /opt/bacula/backups Maximum Part Size = 10000000 Media Type = CloudType LabelMedia = yes Random Access = Yes; AutomaticMount = yes RemovableMedia = no AlwaysOpen = no } \end{bVerbatim} As you can see from the above, the \textbf{Cloud} directive in the \textbf{Device} resource contains the name (\textbf{S3Cloud}) of the \textbf{Cloud} resource that is shown below. \smallskip Note also the \textbf{Archive Device} is specified in the same manner as one would use for a File device. However, in place of containing files with Volume names, the archive device for the Cloud drivers will contain the local cache, which consists of directories with the Volume name; and these directories contain the parts associated with the particular Volume. So with the above Device resource, and the two cache Volumes shown in figure \vref{fig:cloud0ay2} above would have the following layout on disk: \begin{bVerbatim} /opt/bacula/backups /opt/bacula/backups/Volume0001 /opt/bacula/backups/Volume0001/part.1 /opt/bacula/backups/Volume0001/part.2 /opt/bacula/backups/Volume0001/part.3 /opt/bacula/backups/Volume0001/part.4 /opt/bacula/backups/Volume0002 /opt/bacula/backups/Volume0002/part.1 /opt/bacula/backups/Volume0002/part.2 /opt/bacula/backups/Volume0002/part.3 \end{bVerbatim} % A good part of this section is also in storedconf.tex and the whitepaper % Please, keep everything synchronized \subsubsection{The Cloud Resource} The \textbf{Cloud} resource has a number of directives that may be specified as exemplified in the following example: default east USA location: \begin{bVerbatim} Cloud { Name = S3Cloud Driver = "S3" HostName = "s3.amazonaws.com" BucketName = "BaculaVolumes" AccessKey = "BZIXAIS39DP9YNER5DFZ" SecretKey = "beesheeg7iTe0Gaexee7aedie4aWohfuewohGaa0" Protocol = HTTPS UriStyle = VirtualHost Truncate Cache = No Upload = EachPart Region = "us-east-1" MaximumUploadBandwidth = 5MB/s } \end{bVerbatim} central europe location: \begin{bVerbatim} Cloud { Name = S3Cloud Driver = "S3" HostName = "s3-eu-central-1.amazonaws.com" BucketName = "BaculaVolumes" AccessKey = "BZIXAIS39DP9YNER5DFZ" SecretKey = "beesheeg7iTe0Gaexee7aedie4aWohfuewohGaa0" Protocol = HTTPS UriStyle = VirtualHost Truncate Cache = No Upload = EachPart Region = "eu-central-1" MaximumUploadBandwidth = 4MB/s }\end{bVerbatim} For Amazon Cloud, refer to \url{http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region} to get a complete list of regions and corresponding endpoints and use them respectively as Region and HostName directive. For CEPH S3 interface: \begin{bVerbatim} Cloud { Name = CEPH_S3 Driver = "S3" HostName = ceph.mydomain.lan BucketName = "CEPHBucket" AccessKey = "xxxXXXxxxx" SecretKey = "xxheeg7iTe0Gaexee7aedie4aWohfuewohxx0" Protocol = HTTPS Upload = EachPart UriStyle = Path # Must be set for CEPH } \end{bVerbatim} The directives of the above Cloud resource for the S3 driver are defined as follows: \begin{description} \label{Storage:Cloud:Name} \item [Name = \lt{}Device-Name\gt{}] \index[sd]{Name} \index[sd]{Directive!Name} The name of the Cloud resource. This is the logical Cloud name, and may be any string up to 127 characters in length. Shown as \textbf{S3Cloud} above. \label{Storage:Cloud:Description} \item [Description = \lt{}Text\gt{}] The description is used for display purposes as is the case with all resource. \label{Storage:Cloud:Driver} \item [Driver = \lt{}DriverName\gt{}] \index[sd]{Driver} This defines which driver to use. It can be \textbf{S3}. There is also a \textbf{File} driver, which is used mostly for testing. \label{Storage:Cloud:HostName} \item [Host Name = \lt{}Name\gt{}] \index[sd]{HostName} This directive specifies the hostname to be used in the URL. Each Cloud service provider has a different and unique hostname. The maximum size is 255 characters and may contain a tcp port specification. \label{Storage:Cloud:BucketName} \item [Bucket Name = \lt{}Name\gt{}] \index[sd]{BucketName} This directive specifies the bucket name that you wish to use on the Cloud service. This name is normally a unique name name that identifies where you want to place your Cloud Volume parts. With Amazon S3, the bucket must be created previously on the Cloud service. The maximum bucket name size is 255 characters. \label{Storage:Cloud:AccessKey} \item [Access Key = \lt{}String\gt{}] \index[sd]{AccessKey} The access key is your unique user identifier given to you by your cloud service provider. \label{Storage:Cloud:SecretKey} \item [Secret Key = \lt{}String\gt{}] \index[sd]{SecretKey} The secret key is the security key that was given to you by your cloud service provider. It is equivalent to a password. \label{Storage:Cloud:Protocol} \item [Protocol = \lt{}HTTP | HTTPS\gt{}] \index[sd]{Protocol} The protocol defines the communications protocol to use with the cloud service provider. The two protocols currently supported are: \textbf{HTTPS} and \textbf{HTTP}. The default is \textbf{HTTPS}. \label{Storage:Cloud:UriStyle} \item [Uri Style = \lt{}VirtualHost | Path\gt{}] \index[sd]{UriStyle} This directive specifies the URI style to use to communicate with the cloud service provider. The two Uri Styles currently supported are: \textbf{VirtualHost} and \textbf{Path}. The default is \textbf{VirtualHost}. \label{Storage:Cloud:TruncateCache} \item [Truncate Cache = \lt{}Truncate-kw\gt{}] \index[sd]{TruncateCache} This directive specifies when Bacula should automatically remove (truncate) the local cache parts. Local cache parts can only be removed if they have been uploaded to the cloud. The currently implemented values are: \begin{itemize} \item \textbf{No} Do not remove cache. With this option you must manually delete the cache parts with a \textbf{bconsole} \textbf{Truncate Cache} command, or do so with an \textbf{Admin} Job that runs an \textbf{Truncate Cache} command. This is the default. \item \textbf{AfterUpload} Each part will be removed just after it is uploaded. Note, if this option is specified, all restores will require a download from the Cloud. {\textbf Note: Not yet implemented.} \item \textbf{AtEndOfJob} With this option, at the end of the Job, every part that has been uploaded to the Cloud will be removed (truncated). {\textbf Note: Not yet implemented.} \end{itemize} \label{Storage:Cloud:Upload} \item [Upload = \lt{}Upload-kw\gt{}] This directive specifies when local cache parts will be uploaded to the Cloud. The options are: \begin{itemize} \item \textbf{No} Do not upload cache parts. With this option you must manually upload the cache parts with a \textbf{bconsole} \textbf{Upload} command, or do so with an \textbf{Admin} Job that runs an \textbf{Upload} command. This is the default. \item \textbf{EachPart} With this option, each part will be uploaded when it is complete i.e. when the next part is created or at the end of the Job. \item \textbf{AtEndOfJob} With this option all parts that have not been previously uploaded will be uploaded at the end of the Job. {\textbf Note: Not yet implemented.} \end{itemize} \label{Storage:Cloud:MaximumUploadBandwidth} \item [Maximum Upload Bandwidth = \lt{}speed\gt{}] The default is unlimited, but by using this directive, you may limit the upload bandwidth used globally by all devices referencing this Cloud resource. \label{Storage:Cloud:MaximumDownloadBandwidth} \item [Maximum Download Bandwidth = \lt{}speed\gt{}] The default is unlimited, but by using this directive, you may limit the download bandwidth used globally by all devices referencing this Cloud resource. \label{Storage:Cloud:Region} \item [Region = \lt{}String\lt{}] The Cloud resource can be configured to use a specific endpoint within a region. This directive is required for AWS-V4 regions. ex: \texttt{Region="eu-central-1"} \end{description} \subsubsection{File Driver for the Cloud} As mentioned above, one may specify the keyword \textbf{File} on the \textbf{Driver} directive of the Cloud resource. Instead of writing to the Cloud, Bacula will instead create a Cloud Volume but write it to disk. The rest of this section applies to the Cloud resource directives when the File driver is specified. \smallskip The following Cloud directives are ignored: Bucket Name, Access Key, Secret Key, Protocol, Uri Style. The directives {\bf Truncate Cache} and {\bf Upload} work on the local cache in the same manner as they do for the S3 driver. \smallskip The main difference to note is that the Host Name, specifies the destination directory for the Cloud Volume files, and this Host Name must be different from the Archive Device name, or there will be a conflict between the local cache (in the Archive Device directory) and the destination Cloud Volumes (in the Host Name directory). \smallskip As noted above, the File driver is mostly used for testing purposes, and we do not particularly recommend using it. However, if you have a particularly slow backup device you might want to stage your backup data into an SSD or disk using the local cache feature of the Cloud device, and have your Volumes transferred in the background to a slow File device. \subsection{WORM Tape Support} Automatic WORM (Write Once Read Multiple) tapes detection has been added in 10.2. \smallskip{} When a WORM tape is detected, the catalog volume entry is changed automatically to set \texttt{Recycle=no}. It will prevent the volume from being automatically recycled by Bacula. \smallskip{} There is no change in how the Job and File records are pruned from the catalog as that is a separate issue that is currently adequately implemented in Bacula. \smallskip{} When a WORM tape is detected, the SD will show WORM on the device state output (must have debug greater or equal to 6) otherwise the status shows as \texttt{!WORM} \begin{bVerbatim} Device state: OPENED !TAPE LABEL APPEND !READ !EOT !WEOT !EOF WORM !SHORT !MOUNTED ... \end{bVerbatim} \smallskip{} The output of the used volume status has been modified to include the worm state. It shows \texttt{worm=1} for a worm cassette and \texttt{worm=0} otherwise. Example: \begin{bVerbatim} Used Volume status: Reserved volume: TestVolume001 on Tape device "nst0" (/dev/nst0) Reader=0 writers=0 reserves=0 volinuse=0 worm=1 \end{bVerbatim} \smallskip{} The following programs are needed for the WORM tape detection: \begin{itemize} \item sdparm \item tapeinfo \end{itemize} \smallskip{} The new Storage Device directive \texttt{Worm Command} must be configured as well as the \texttt{Control Device} directive (used with the Tape Alert feature). \begin{bVerbatim} Device { Name = "LTO-0" Archive Device = "/dev/nst0" Control Device = "/dev/sg0" # from lsscsi -g Worm Command = "/opt/bacula/scripts/isworm %l" ... } \end{bVerbatim} \chapter{New Features in 9.2.0} This chapter describes new features that have been added to the current version of Bacula in version 9.2.0 \smallskip In general, this is a fairly substantial release because it contains a very large number of bug fixes backported from the Bacula Enterprise version. There are also a few new features backported from Bacula Enterprise. \subsection{Enhanced Autochanger Support} Note: this feature was actually backported into version 9.0.0, but the documentation was added much after the 9.0.0 release. To call your attention to this new feature, we have also included the documentation here. To make Bacula function properly with multiple Autochanger definitions, in the Director's configuration, you must adapt your {\bf bacula-dir.conf} {\bf Storage} directives. \smallskip Each autochanger that you have defined in an {\bf Autochanger} resource in the Storage daemon's {\bf bacula-sd.conf} file, must have a corresponding {\bf Autochanger} resource defined in the Director's {\bf bacula-dir.conf} file. Normally you will already have a {\bf Storage} resource that points to the Storage daemon's {\bf Autochanger} resource. Thus you need only to change the name of the {\bf Storage} resource to {\bf Autochanger}. In addition the {\bf Autochanger = yes} directive is not needed in the Director's {\bf Autochanger} resource, since the resource name is {\bf Autochanger}, the Director already knows that it represents an autochanger. \smallskip In addition to the above change ({\bf Storage} to {\bf Autochanger}), you must modify any additional {\bf Storage} resources that correspond to devices that are part of the {\bf Autochanger} device. Instead of the previous {\bf Autochanger = yes} directive they should be modified to be {\bf Autochanger = xxx} where you replace the {\bf xxx} with the name of the Autochanger. \smallskip For example, in the bacula-dir.conf file: \begin{bVerbatim} Autochanger { # New resource Name = Changer-1 Address = cibou.company.com SDPort = 9103 Password = "xxxxxxxxxx" Device = LTO-Changer-1 Media Type = LTO-4 Maximum Concurrent Jobs = 50 } Storage { Name = Changer-1-Drive0 Address = cibou.company.com SDPort = 9103 Password = "xxxxxxxxxx" Device = LTO4_1_Drive0 Media Type = LTO-4 Maximum Concurrent Jobs = 5 Autochanger = Changer-1 # New directive } Storage { Name = Changer-1-Drive1 Address = cibou.company.com SDPort = 9103 Password = "xxxxxxxxxx" Device = LTO4_1_Drive1 Media Type = LTO-4 Maximum Concurrent Jobs = 5 Autochanger = Changer-1 # New directive } ... \end{bVerbatim} Note that Storage resources {\bf Changer-1-Drive0} and {\bf Changer-1-Drive1} are not required since they make up part of an autochanger, and normally, Jobs refer only to the Autochanger resource. However, by referring to those Storage definitions in a Job, you will use only the indicated drive. This is not normally what you want to do, but it is very useful and often used for reserving a drive for restores. See the Storage daemon example .conf below and the use of {\bf AutoSelect = no}. So, in summary, the changes are: \begin{itemize} \item Change {\bf Storage} to {\bf Autochanger} in the LTO4 resource. \item Remove the {\bf Autochanger = yes} from the {\bf Autochanger} LTO4 resource. \item Change the {\bf Autochanger = yes} in each of the {\bf Storage} device that belong to the {\bf Autochanger} to point to the {\bf Autochanger} resource with for the example above the directive {\bf Autochanger = LTO4}. \smallskip Please note that if you define two different autochangers, you must give a unique Media Type to the Volumes in each autochanger. More specifically, you may have multiple Media Types, but you cannot have Volumes with the same Media Type in two different autochangers. If you attempt to do so, Bacula will most likely reference the wrong autochanger (Storage) and not find the correct Volume. \end{itemize} \subsection{New Prune Command Option} The bconsole ``prune'' command can now run the pruning algorithm on all volumes from a Pool or on all Pools. \begin{bVerbatim} * prune allfrompool pool=Default yes * prune allfrompool allpools yes \end{bVerbatim} \subsection{BConsole Features} \subsubsection{Delete a Client} The \texttt{delete client} bconsole command delete the database record of a client that is no longer defined in the configuration file. It also removes all other records (Jobs, Files, ...) associated with the client that is deleted. \subsubsection{Status Schedule Enhancements} The \texttt{status schedule} command can now accept multiple \texttt{client} or \texttt{job} keywords on the command line. The \texttt{limit} parameter is disabled when the \texttt{days} parameter is used. The output is now ordered by day. \subsubsection{Restore option ``noautoparent''} During a \texttt{bconsole} restore session, parent directories are automatically selected to avoid issues with permissions. It is possible to disable this feature with the \texttt{noautoparent} command line parameter. \subsection{Tray Monitor Restore Screen} It is now possible to restore files from the Tray Monitor GUI program. \begin{figure}[htbp] \centering \includegraphics[width=0.8\linewidth]{tray-monitor-restore} \caption{Tray Monitor Restore Wizard} \label{fig:tray-monitor-restore} \end{figure} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%5 \chapter{New Features in 9.0.0} This chapter describes new features that have been added to the current version of Bacula in version 9.0.0 \subsection{Enhanced Autochanger Support} To make Bacula function properly with multiple Autochanger definitions, in the Director's configuration, you must adapt your {\bf bacula-dir.conf} {\bf Storage} directives. \smallskip Each autochanger that you have defined in an {\bf Autochanger} resource in the Storage daemon's {\bf bacula-sd.conf} file, must have a corresponding {\bf Autochanger} resource defined in the Director's {\bf bacula-dir.conf} file. Normally you will already have a {\bf Storage} resource that points to the Storage daemon's {\bf Autochanger} resource. Thus you need only to change the name of the {\bf Storage} resource to {\bf Autochanger}. In addition the {\bf Autochanger = yes} directive is not needed in the Director's {\bf Autochanger} resource, since the resource name is {\bf Autochanger}, the Director already knows that it represents an autochanger. \smallskip In addition to the above change ({\bf Storage} to {\bf Autochanger}), you must modify any additional {\bf Storage} resources that correspond to devices that are part of the {\bf Autochanger} device. Instead of the previous {\bf Autochanger = yes} directive they should be modified to be {\bf Autochanger = xxx} where you replace the {\bf xxx} with the name of the Autochanger. \smallskip For example, in the bacula-dir.conf file: \begin{bVerbatim} Autochanger { # New resource Name = Changer-1 Address = cibou.company.com SDPort = 9103 Password = "xxxxxxxxxx" Device = LTO-Changer-1 Media Type = LTO-4 Maximum Concurrent Jobs = 50 } Storage { Name = Changer-1-Drive0 Address = cibou.company.com SDPort = 9103 Password = "xxxxxxxxxx" Device = LTO4_1_Drive0 Media Type = LTO-4 Maximum Concurrent Jobs = 5 Autochanger = Changer-1 # New directive } Storage { Name = Changer-1-Drive1 Address = cibou.company.com SDPort = 9103 Password = "xxxxxxxxxx" Device = LTO4_1_Drive1 Media Type = LTO-4 Maximum Concurrent Jobs = 5 Autochanger = Changer-1 # New directive } ... \end{bVerbatim} Note that Storage resources {\bf Changer-1-Drive0} and {\bf Changer-1-Drive1} are not required since they make up part of an autochanger, and normally, Jobs refer only to the Autochanger resource. However, by referring to those Storage definitions in a Job, you will use only the indicated drive. This is not normally what you want to do, but it is very useful and often used for reserving a drive for restores. See the Storage daemon example .conf below and the use of {\bf AutoSelect = no}. So, in summary, the changes are: \begin{itemize} \item Change {\bf Storage} to {\bf Autochanger} in the LTO4 resource. \item Remove the {\bf Autochanger = yes} from the {\bf Autochanger} LTO4 resource. \item Change the {\bf Autochanger = yes} in each of the {\bf Storage} device that belong to the {\bf Autochanger} to point to the {\bf Autochanger} resource with for the example above the directive {\bf Autochanger = LTO4}. \end{itemize} \subsection{Source Code for Windows} With this version of Bacula, we have included the old source code for Windows and also updated it to contain the code from the latest Bacula Enterprise version. The project is also directly distributing binaries for Windows rather than relying on Bacula Systems to supply them. \subsection{Maximum Virtual Full Interval Option} Two new director directives have been added: \begin{bVerbatim} Max Virtual Full Interval and Virtual Full Backup Pool \end{bVerbatim} The {\bf Max Virtual Full Interval} directive should behave similar to the {\bf Max Full Interval}, but for Virtual Full jobs. If Bacula sees that there has not been a Full backup in Max Virtual Full Interval time then it will upgrade the job to Virtual Full. If you have both {\bf Max Full Interval} and {\bf Max Virtual Full Interval} set then Max Full Interval should take precedence. The {\bf Virtual Full Backup Pool} directive allows one to change the pool as well. You probably want to use these two directives in conjunction with each other but that may depend on the specifics of one's setup. If you set the {\bf Max Full Interval} without setting {\bf Max Virtual Full Interval} then Bacula will use whatever the "default" pool is set to which is the same behavior as with the Max Full Interval. \subsection{Progressive Virtual Full} In Bacula version 9.0.0, we have added a new Directive named {\bf Backups To Keep} that permits you to implement Progressive Virtual Fulls within Bacula. Sometimes this feature is known as Incremental Forever with Consolidation. \smallskip \begin{figure}[htbp] \centering \includegraphics[width=.8\linewidth]{pvf-slidingbackups} \caption{Backup Sequence Slides Forward One Day, Each Day} \label{fig:slidingbackups} \end{figure} To implement the Progressive Virtual Full feature, simply add the {\bf Backups To Keep} directive to your Virtual Full backup Job resource. The value specified on the directive indicates the number of backup jobs that should not be merged into the Virtual Full (i.e. the number of backup jobs that should remain after the Virtual Full has completed. The default is zero, which reverts to a standard Virtual Full than consolidates all the backup jobs that it finds. \subsubsection{Backups To Keep Directive} The new {\bf BackupsToKeep} directive is specified in the Job Resource and has the form: \begin{bVerbatim} Backups To Keep = 30 \end{bVerbatim} where the value (30 in the above figure and example) is the number of backups to retain. When this directive is present during a Virtual Full (it is ignored for other Job types), it will look for the most recent Full backup that has more subsequent backups than the value specified. In the above example the Job will simply terminate unless there is a Full back followed by at least 31 backups of either level Differential or Incremental. \smallskip Assuming that the last Full backup is followed by 32 Incremental backups, a Virtual Full will be run that consolidates the Full with the first two Incrementals that were run after the Full. The result is that you will end up with a Full followed by 30 Incremental backups. The Job Resource in {\bf bacula-dir.conf} to accomplish this would be: \begin{bVerbatim} Job { Name = "VFull" Type = Backup Level = VirtualFull Client = "my-fd" File Set = "FullSet" Accurate = Yes Backups To Keep = 10 } \end{bVerbatim} \subsubsection{Delete Consolidated Jobs} The new directive {\bf Delete Consolidated Jobs} expects a {\bf yes} or {\bf no} value that if set to {\bf yes} will cause any old Job that is consolidated during a Virtual Full to be deleted. In the example above we saw that a Full plus one other job (either an Incremental or Differential) were consolidated into a new Full backup. The original Full plus the other Job consolidated will be deleted. The default value is {\bf no}. \subsubsection{Virtual Full Compatibility} Virtual Full as well as Progressive Virtual Full works with any standard backup Job. \smallskip However, it should be noted that Virtual Full jobs are not compatible with any plugins that you may be using. \subsection{TapeAlert Enhancements} There are some significant enhancements to the TapeAlert feature of Bacula. Several directives are used slightly differently, which unfortunately causes a compatibility problem with the old TapeAlert implementation. Consequently, if you are already using TapeAlert, you must modify your {\bf bacula-sd.conf} in order for Tape Alerts to work. See below for the details ... \subsubsection{What is New} First, you must define a \textbf{Alert Command} directive in the Device resource that calls the new \textbf{tapealert} script that is installed in the scripts directory (normally: /opt/bacula/scripts). It is defined as follows: \begin{bVerbatim} Device { Name = ... Archive Device = /dev/nst0 Alert Command = "/opt/bacula/scripts/tapealert %l" Control Device = /dev/sg1 # must be SCSI ctl for /dev/nst0 ... } \end{bVerbatim} In addition the \textbf{Control Device} directive in the Storage Daemon's conf file must be specified in each Device resource to permit Bacula to detect tape alerts on a specific devices (normally only tape devices). Once the above mentioned two directives (Alert Command and Control Device) are in place in each of your Device resources, Bacula will check for tape alerts at two points: \begin{itemize} \item After the Drive is used and it becomes idle. \item After each read or write error on the drive. \end{itemize} At each of the above times, Bacula will call the new \textbf{tapealert} script, which uses the \textbf{tapeinfo} program. The tapeinfo utility is part of the apt sg3-utils and rpm sg3\_utils packages that must be installed on your systems. Then after each alert that Bacula finds for that drive, Bacula will emit a Job message that is either INFO, WARNING, or FATAL depending on the designation in the Tape Alert published by the T10 Technical Committee on SCSI Storage Interfaces (www.t10.org). For the specification, please see: www.t10.org/ftp/t10/document.02/02-142r0.pdf \smallskip As a somewhat extreme example, if tape alerts 3, 5, and 39 are set, you will get the following output in your backup job. {\small \begin{bVerbatim} 17-Nov 13:37 rufus-sd JobId 1: Error: block.c:287 Write error at 0:17 on device "tape" (/home/kern/bacula/k/regress/working/ach/drive0) Vol=TestVolume001. ERR=Input/output error. 17-Nov 13:37 rufus-sd JobId 1: Fatal error: Alert: Volume="TestVolume001" alert=3: ERR=The operation has stopped because an error has occurred while reading or writing data which the drive cannot correct. The drive had a hard read or write error 17-Nov 13:37 rufus-sd JobId 1: Fatal error: Alert: Volume="TestVolume001" alert=5: ERR=The tape is damaged or the drive is faulty. Call the tape drive supplier helpline. The drive can no longer read data from the tape 17-Nov 13:37 rufus-sd JobId 1: Warning: Disabled Device "tape" (/home/kern/bacula/k/regress/working/ach/drive0) due to tape alert=39. 17-Nov 13:37 rufus-sd JobId 1: Warning: Alert: Volume="TestVolume001" alert=39: ERR=The tape drive may have a fault. Check for availability of diagnostic information and run extended diagnostics if applicable. The drive may have had a failure which may be identified by stored diagnostic information or by running extended diagnostics (eg Send Diagnostic). Check the tape drive users manual for instructions on running extended diagnostic tests and retrieving diagnostic data. \end{bVerbatim} } Without the tape alert feature enabled, you would only get the first error message above, which is the error return Bacula received when it gets the error. Notice also, that in the above output the alert number 5 is a critical error, which causes two things to happen. First the tape drive is disabled, and second the Job is failed. \smallskip If you attempt to run another Job using the Device that has been disabled, you will get a message similar to the following: \begin{bVerbatim} 17-Nov 15:08 rufus-sd JobId 2: Warning: Device "tape" requested by DIR is disabled. \end{bVerbatim} and the Job may be failed if no other drive can be found. \smallskip Once the problem with the tape drive has been corrected, you can clear the tape alerts and re-enable the device with the Bacula bconsole command such as the following: \begin{bVerbatim} enable Storage=Tape \end{bVerbatim} Note, when you enable the device, the list of prior tape alerts for that drive will be discarded. \smallskip Since is is possible to miss tape alerts, Bacula maintains a temporary list of the last 8 alerts, and each time Bacula calls the \textbf{tapealert} script, it will keep up to 10 alert status codes. Normally there will only be one or two alert errors for each call to the tapealert script. \smallskip Once a drive has one or more tape alerts, you can see them by using the bconsole status command as follows: \begin{bVerbatim} status storage=Tape \end{bVerbatim} which produces the following output: \begin{bVerbatim} Device Vtape is "tape" (/home/kern/bacula/k/regress/working/ach/drive0) mounted with: Volume: TestVolume001 Pool: Default Media type: tape Device is disabled. User command. Total Bytes Read=0 Blocks Read=1 Bytes/block=0 Positioned at File=1 Block=0 Critical Alert: at 17-Nov-2016 15:08:01 Volume="TestVolume001" alert=Hard Error Critical Alert: at 17-Nov-2016 15:08:01 Volume="TestVolume001" alert=Read Failure Warning Alert: at 17-Nov-2016 15:08:01 Volume="TestVolume001" alert=Diagnostics Required \end{bVerbatim} if you want to see the long message associated with each of the alerts, simply set the debug level to 10 or more and re-issue the status command: \begin{bVerbatim} setdebug storage=Tape level=10 status storage=Tape \end{bVerbatim} \begin{bVerbatim} ... Critical Alert: at 17-Nov-2016 15:08:01 Volume="TestVolume001" flags=0x0 alert=The operation has stopped because an error has occurred while reading or writing data which the drive cannot correct. The drive had a hard read or write error Critical Alert: at 17-Nov-2016 15:08:01 Volume="TestVolume001" flags=0x0 alert=The tape is damaged or the drive is faulty. Call the tape drive supplier helpline. The drive can no longer read data from the tape Warning Alert: at 17-Nov-2016 15:08:01 Volume="TestVolume001" flags=0x1 alert=The tape drive may have a fault. Check for availability of diagnostic information and run extended diagnostics if applicable. The drive may have had a failure which may be identified by stored diagnostic information or by running extended diagnostics (eg Send Diagnostic). Check the tape drive users manual for instructions on running extended diagnostic tests and retrieving diagnostic data. ... \end{bVerbatim} The next time you \textbf{enable} the Device by either using \textbf{bconsole} or you restart the Storage Daemon, all the saved alert messages will be discarded. \subsubsection{Handling of Alerts} Tape Alerts numbered 7,8,13,14,20,22,52,53, and 54 will cause Bacula to disable the current Volume. \smallskip Tape Alerts numbered 14,20,29,30,31,38, and 39 will cause Bacula to disable the drive. \smallskip Please note certain tape alerts such as 14 have multiple effects (disable the Volume and disable the drive). \subsection{New Console ACL Directives} By default, if a Console ACL directive is not set, Bacula will assume that the ACL list is empty. If the current Bacula Director configuration uses restricted Consoles and allows restore jobs, it is mandatory to configure the new directives. \subsubsection{DirectoryACL} \index[dir]{Directive!DirectoryACL} This directive is used to specify a list of directories that can be accessed by a restore session. Without this directive, a restricted console cannot restore any file. Multiple directories names may be specified by separating them with commas, and/or by specifying multiple DirectoryACL directives. For example, the directive may be specified as: \footnotesize \begin{bVerbatim} DirectoryACL = /home/bacula/, "/etc/", "/home/test/*" \end{bVerbatim} \normalsize With the above specification, the console can access the following directories: \begin{itemize} \item \texttt{/etc/password} \item \texttt{/etc/group} \item \texttt{/home/bacula/.bashrc} \item \texttt{/home/test/.ssh/config} \item \texttt{/home/test/Desktop/Images/something.png} \end{itemize} But not to the following files or directories: \begin{itemize} \item \texttt{/etc/security/limits.conf} \item \texttt{/home/bacula/.ssh/id\_dsa.pub} \item \texttt{/home/guest/something} \item \texttt{/usr/bin/make} \end{itemize} If a directory starts with a Windows pattern (ex: c:/), Bacula will automatically ignore the case when checking directory names. \subsection{New Bconsole ``list'' Command Behavior} The bconsole \texttt{list} commands can now be used safely from a restricted bconsole session. The information displayed will respect the ACL configured for the Console session. For example, if a restricted Console has access to JobA, JobB and JobC, information about JobD will not appear in the \texttt{list jobs} command. \subsection{New Console ACL Directives} \index[dir]{Directive!BackupClientACL} It is now possible to configure a restricted Console to distinguish Backup and Restore job permissions. The \texttt{BackupClientACL} can restrict backup jobs on a specific set of clients, while the \texttt{RestoreClientACL} can restrict restore jobs. {\small \begin{bVerbatim} # cat /opt/bacula/etc/bacula-dir.conf ... Console { Name = fd-cons # Name of the FD Console Password = yyy ... ClientACL = localhost-fd # everything allowed RestoreClientACL = test-fd # restore only BackupClientACL = production-fd # backup only } \end{bVerbatim} } The \texttt{ClientACL} directive takes precedence over the \texttt{RestoreClientACL} and the \texttt{BackupClientACL}. In the Console resource resource above, it means that the bconsole linked to the Console{} named "fd-cons" will be able to run: \begin{itemize} \item backup and restore for ``localhost-fd'' \item backup for ``production-fd'' \item restore for ``test-fd'' \end{itemize} At the restore time, jobs for client ``localhost-fd'', ``test-fd'' and ``production-fd'' will be available. If \texttt{*all*} is set for \texttt{ClientACL}, backup and restore will be allowed for all clients, despite the use of \texttt{RestoreClientACL} or \texttt{"BackupClientACL}. \subsection{Client Initiated Backup} \label{sec:featurecib} A console program such as the new \texttt{tray-monitor} or \texttt{bconsole} can now be configured to connect a File Daemon. There are many new features available (see the New Tray Monitor section below), but probably the most important is the ability for the user to initiate a backup of his own machine. The connection established by the FD to the Director for the backup will be used by the Director for the backup, thus not only can clients (users) initiate backups, but a File Daemon that is NATed (cannot be reached by the Director) can now be backed up without using advanced tunneling techniques providing that the File Daemon can connect to the Director. \smallskip The flow of information is shown in the picture below: \bimageH{nat}{Client Initiated Backup Network Flow}{fig:nat3} \newpage \subsection{Configuring Client Initiated Backup} \smallskip In order to ensure security, there are a number of new directives that must be enabled in the new \texttt{tray-monitor}, the File Daemon and in the Director. A typical configuration might look like the following: {\small \begin{bVerbatim} # cat /opt/bacula/etc/bacula-dir.conf ... Console { Name = fd-cons # Name of the FD Console Password = yyy # These commands are used by the tray-monitor, it is possible to restrict CommandACL = run, restore, wait, .status, .jobs, .clients CommandACL = .storages, .pools, .filesets, .defaults, .estimate # Adapt for your needs jobacl = *all* poolacl = *all* clientacl = *all* storageacl = *all* catalogacl = *all* filesetacl = *all* } \end{bVerbatim} } {\small \begin{bVerbatim} # cat /opt/bacula/etc/bacula-fd.conf ... Console { # Console to connect the Director Name = fd-cons DIRPort = 9101 address = localhost Password = "yyy" } Director { Name = remote-cons # Name of the tray monitor/bconsole Password = "xxx" # Password of the tray monitor/bconsole Remote = yes # Allow to use send commands to the Console defined } \end{bVerbatim} } {\small \begin{bVerbatim} cat /opt/bacula/etc/bconsole-remote.conf .... Director { Name = localhost-fd address = localhost # Specify the FD address DIRport = 9102 # Specify the FD Port Password = "notused" } Console { Name = remote-cons # Name used in the auth process Password = "xxx" } \end{bVerbatim} } {\small \begin{bVerbatim} cat ~/.bacula-tray-monitor.conf Monitor { Name = remote-cons } Client { Name = localhost-fd address = localhost # Specify the FD address Port = 9102 # Specify the FD Port Password = "xxx" Remote = yes } \end{bVerbatim} } \bimageH{conf-nat}{Relation Between Resources (bconsole)}{fig:nat} \bimageH{conf-nat2}{Relation Between Resources (tray-monitor)}{fig:nat2} \medskip A more detailed description with complete examples is available in chapter~\ref{TrayMonitorChapter}. \subsection{New Tray Monitor} A new tray monitor has been added to the 9.0 release, the tray monitor offers the following features: \begin{itemize} \item Director, File and Storage Daemon status page \item Support for the Client Initiated Backup protocol (See \vref{sec:featurecib}). To use the Client Initiated Backup option from the tray monitor, the Client option ``Remote'' should be checked in the configuration (Fig \vref{fig:tray2}). \item Wizard to run new job (Fig \vref{fig:tray4}) \item Display an estimation of the number of files and the size of the next backup job (Fig \vref{fig:tray4}) \item Ability to configure the tray monitor configuration file directly from the GUI (Fig \vref{fig:tray2}) \item Ability to monitor a component and adapt the tray monitor task bar icon if a jobs are running. \item TLS Support \item Better network connection handling \item Default configuration file is stored under \texttt{\$HOME/.bacula-tray-monitor.conf} \item Ability to ``schedule'' jobs \item Available on Linux and Windows platforms \end{itemize} % \medskip % Please see chapter \ref{TrayMonitorChapter} for more details about this new % functionality. \begin{figure}[htbp] \centering \includegraphics[width=0.8\linewidth]{tray-monitor-status} \caption{Tray Monitor Status} \label{fig:tray0} \end{figure} \begin{figure}[htbp] \centering \includegraphics[width=0.9\linewidth]{tray-monitor-conf-fd} \caption{Tray Monitor Client Configuration} \label{fig:tray2} \end{figure} \begin{figure}[htbp] \centering \includegraphics[width=0.8\linewidth]{tray-monitor-run1} \smallskip \includegraphics[width=0.8\linewidth]{tray-monitor-run2} \caption{Tray Monitor Run a Job} \label{fig:tray4} \end{figure} \subsection{Schedule Jobs via the Tray Monitor} The Tray Monitor can scan periodically a specific directory ``Command Directory'' and process ``*.bcmd'' files to find jobs to run. The format of the ``file.bcmd'' command file is the following: \begin{bVerbatim} : : ... = string = string (bconsole command line) \end{bVerbatim} For example: \begin{bVerbatim} localhost-fd: run job=backup-localhost-fd level=full localhost-dir: run job=BackupCatalog \end{bVerbatim} The command file should contain at least one command. The component specified in the first part of the command line should be defined in the tray monitor. Once the command file is detected by the tray monitor, a popup is displayed to the user and it is possible for the user to cancel the job directly. \smallskip{} The file can be created with tools such as ``cron'' or the ``task scheduler'' on Windows. It is possible to verify the network connection at that time to avoid network errors. \begin{bVerbatim} #!/bin/sh if ping -c 1 director &> /dev/null then echo "my-dir: run job=backup" > /path/to/commands/backup.bcmd fi \end{bVerbatim} %\bimageH{tray-monitor-status}{Tray Monitor Status}{fig:tray0} %\bimageH{tray-monitor1}{Tray Monitor Configuration}{fig:tray1} %\bimageH{tray-monitor-conf-fd}{Tray Monitor Client Configuration}{fig:tray2} %\bimageH{tray-monitor-conf-dir}{Tray Monitor Director Configuration}{fig:tray3} %\bimageH{tray-monitor-run1}{Tray Monitor Run new Job}{fig:tray4} % find a way to group them together %\bimageH{tray-monitor-run2}{Tray Monitor Setup new Job}{fig:tray5} \subsection{Accurate Option for Verify ``Volume Data'' Job} Since Bacula version 8.4.1, it has been possible to have a Verify Job configured with \texttt{level=Data} that will reread all records from a job and optionally check the size and the checksum of all files. Starting with \smallskip Bacula version 9.0, it is now possible to use the \texttt{accurate} option to check catalog records at the same time. When using a Verify job with \texttt{level=Data} and \texttt{accurate=yes} can replace the \texttt{level=VolumeToCatalog} option. For more information on how to setup a Verify Data job, see \vref{blb:verifyvolumedata}. To run a Verify Job with the \texttt{accurate} option, it is possible to set the option in the Job definition or set use the \texttt{accurate=yes} on the command line. \begin{bVerbatim} * run job=VerifyData jobid=10 accurate=yes \end{bVerbatim} \subsection{FileDaemon Saved Messages Resource Destination} It is now possible to send the list of all saved files to a Messages resource with the \texttt{saved} message type. It is not recommended to send this flow of information to the director and/or the catalog when the client FileSet is pretty large. To avoid side effects, the \texttt{all} keyword doesn't include the \texttt{saved} message type. The \texttt{saved} message type should be explicitly set. \begin{bVerbatim} # cat /opt/bacula/etc/bacula-fd.conf ... Messages { Name = Standard director = mydirector-dir = all, !terminate, !restored, !saved append = /opt/bacula/working/bacula-fd.log = all, saved, restored } \end{bVerbatim} \subsection{Minor Enhancements} \subsubsection{New Bconsole ".estimate" Command} The new \texttt{.estimate} command can be used to get statistics about a job to run. The command uses the database to approximate the size and the number of files of the next job. On a PostgreSQL database, the command uses regression slope to compute values. On MySQL, where these statistical functions are not available, the command uses a simple ``average'' estimation. The correlation number is given for each value. {\small \begin{bVerbatim} *.estimate job=backup level=I nbjob=0 corrbytes=0 jobbytes=0 corrfiles=0 jobfiles=0 duration=0 job=backup *.estimate job=backup level=F level=F nbjob=1 corrbytes=0 jobbytes=210937774 corrfiles=0 jobfiles=2545 duration=0 job=backup \end{bVerbatim} } \subsubsection{Traceback and Lockdump} After the reception of a signal, \texttt{traceback} and \texttt{lockdump} information are now stored in the same file. \subsection{Bconsole ``list jobs'' command options} The \texttt{list jobs} bconsole command now accepts new command line options: \begin{itemize} \item \textbf{joberrors} Display jobs with JobErrors \item \textbf{jobstatus=T} Display jobs with the specified status code \item \textbf{client=cli} Display jobs for a specified client \item \textbf{order=asc/desc} Change the output format of the job list. The jobs are sorted by start time and JobId, the sort can use ascendant (asc) or descendant (desc) (default) value. \end{itemize} \subsection{Minor Enhancements} \subsubsection{New Bconsole "Tee All" Command} The ``@tall'' command allows logging all input/output from a console session. \begin{bVerbatim} *@tall /tmp/log *st dir ... \end{bVerbatim} \subsection{Bconsole ``list jobs'' command options} The \texttt{list jobs} bconsole command now accepts new command line options: \begin{itemize} \item \textbf{joberrors} Display jobs with JobErrors \item \textbf{jobstatus=T} Display jobs with the specified status code \item \textbf{client=cli} Display jobs for a specified client \item \textbf{order=asc/desc} Change the output format of the job list. The jobs are sorted by start time and JobId, the sort can use ascendant (asc) or descendant (desc) (default) value. \end{itemize} \subsection{New Bconsole "Tee All" Command} The ``@tall'' command allows logging all input/output from a console session. \begin{bVerbatim} *@tall /tmp/log *st dir ... \end{bVerbatim} \subsection{New Job Edit Codes \%I} In various places such as RunScripts, you have now access to \%I to get the JobId of the copy or migration job started by a migrate job. \begin{bVerbatim} Job { Name = Migrate-Job Type = Migrate ... RunAfter = "echo New JobId is %I" } \end{bVerbatim} \subsection*{.api version 2} In Bacula version 9.0 and later, we introduced a new .api version to help external tools to parse various Bacula bconsole output. % waa - 20150317 - this section needs just a little more to explain what the "43" in "s43" mean. Perhaps % if it is not a good place to list the possibilities here, then list where a reference % is. Also, I think .api 2 ... Means "use API version 2" but that should be stated too The \texttt{api\_opts} option can use the following arguments: \begin{itemize} \item [C] Clear current options \item [tn] Use a specific time format (1 ISO format, 2 Unix Timestamp, 3 Default Bacula time format) \item [sn] Use a specific separator between items (new line by default). \item [Sn] Use a specific separator between objects (new line by default). \item [o] Convert all keywords to lowercase and convert all non \textsl{isalpha} characters to \_ \end{itemize} % waa - 20150317 - I think there should either be more output listed here to give a better feeling % or, perhaps another output listing for different .status commands \begin{bVerbatim} .api 2 api_opts=t1s43S35 .status dir running ================================== jobid=10 job=AJob ... \end{bVerbatim} \subsection*{New Debug Options} In Bacula version 9.0 and later, we introduced a new \texttt{options} parameter for the \texttt{setdebug} bconsole command. \smallskip{} The following arguments to the new \texttt{option} parameter are available to control debug functions. \begin{itemize} \item [0] Clear debug flags \item [i] Turn off, ignore bwrite() errors on restore on File Daemon \item [d] Turn off decomp of BackupRead() streams on File Daemon \item [t] Turn on timestamps in traces \item [T] Turn off timestamps in traces % waa - 20150306 - does this "c" item mean to say "Truncate trace file if one exists, otherwise append to it" ??? \item [c] Truncate trace file if trace file is activated \item [l] Turn on recording events on P() and V() \item [p] Turn on the display of the event ring when doing a backtrace \end{itemize} \smallskip{} The following command will enable debugging for the File Daemon, truncate an existing trace file, and turn on timestamps when writing to the trace file. \begin{bVerbatim} * setdebug level=10 trace=1 options=ct fd \end{bVerbatim} \smallskip{} It is now possible to use a \textsl{class} of debug messages called \texttt{tags} to control the debug output of Bacula daemons. \begin{itemize} \item [all] Display all debug messages \item [bvfs] Display BVFS debug messages \item [sql] Display SQL related debug messages \item [memory] Display memory and poolmem allocation messages \item [scheduler] Display scheduler related debug messages \end{itemize} \begin{bVerbatim} * setdebug level=10 tags=bvfs,sql,memory * setdebug level=10 tags=!bvfs # bacula-dir -t -d 200,bvfs,sql \end{bVerbatim} The \texttt{tags} option is composed of a list of tags. Tags are separated by ``,'' or ``+'' or ``-'' or ``!''. To disable a specific tag, use ``-'' or ``!'' in front of the tag. Note that more tags are planned for future versions. %\LTXtable{\linewidth}{table_debugtags} \subsection{Communication Line Compression} Bacula version 9.0.0 and later now includes communication line compression. It is turned on by default, and if the two Bacula components (Dir, FD, SD, bconsole) are both version 6.6.0 or greater, communication line compression) will be enabled, by default. If for some reason, you do not want communication line compression, you may disable it with the following directive: \begin{bVerbatim} Comm Compression = no \end{bVerbatim} This directive can appear in the following resources: \begin{bVerbatim} bacula-dir.conf: Director resource bacula-fd.conf Client (or FileDaemon) resource bacula-sd.conf: Storage resource bconsole.conf: Console resource bat.conf: Console resource \end{bVerbatim} \smallskip In many cases, the volume of data transmitted across the communications line can be reduced by a factor of three when this directive is enabled (default) In the case that the compression is not effective, Bacula turns it off on a. record by record basis. \smallskip If you are backing up data that is already compressed the comm line compression will not be effective, and you are likely to end up with an average compression ratio that is very small. In this case, Bacula reports {\bf None} in the Job report. \subsection{Deduplication Optimized Volumes} This version of Bacula includes a new alternative (or additional) volume format that optimizes the placement of files so that an underlying deduplicating filesystem such as ZFS can optimally deduplicate the backup data that is written by Bacula. These are called Deduplication Optimized Volumes or Aligned Volumes for short. The details of how to use this feature and its considerations are in the Deduplication Optimized Volumes whitepaper. \smallskip This feature is available if you have Bacula Community produced binaries and the Aligned Volumes plugin. \subsection{baculabackupreport} I have added a new script called {\bf baculabackupreport} to the scripts directory. This script was written by Bill Arlofski. It prints a backup summary of the backups that occurred in the prior number of hours specified on the command line. You need to edit the first few lines of the file to ensure that your email address is correct and the database type you are using is correct (default is PostgreSQL). Once you do that, you can manually implement it with: \begin{bVerbatim} /opt/bacula/scripts/baculabackupreport 24 \end{bVerbatim} I have put the above line in my {\bf scripts/delete\_catalog\_backup} script so that it will be mailed to me nightly. \subsection{New Message Identification Format} We are starting to add unique message identifiers to each message (other than debug and the Job report) that Bacula prints. At the current time only two files in the Storage Daemon have these message identifiers and over time with subsequent releases we will modify all messages. \smallskip The message identifier will be kept unique for each message and once assigned to a message it will not change even if the text of the message changes. This means that the message identifier will be the same no matter what language the text is displayed in, and more importantly, it will allow us to make listing of the messages with in some cases, additional explanation or instructions on how to correct the problem. All this will take several years since it is a lot of work and requires some new programs that are not yet written to manage these message identifiers. \smallskip The format of the message identifier is: \begin{bVerbatim} [AAnnnn] \end{bVerbatim} where A is an upper case character and nnnn is a four digit number, where the first character indicates the software component (daemon); the second letter indicates the severity, and the number is unique for a given component and severity. \smallskip For example: \begin{bVerbatim} [SF0001] \end{bVerbatim} The first character representing the component at the current time one of the following: \begin{bVerbatim} S Storage daemon D Director F File daemon \end{bVerbatim} \smallskip The second character representing the severity or level can be: \begin{bVerbatim} A Abort F Fatal E Error W Warning S Security I Info D Debug O OK (i.e. operation completed normally) \end{bVerbatim} So in the example above [SF0001] indicates it is a message id, because of the brackets and because it is at the beginning of the message, and that it was generated by the Storage daemon as a fatal error. \smallskip As mentioned above it will take some time to implement these message ids everywhere, and over time we may add more component letters and more severity levels as needed. \chapter{New Features in 7.4.0} This chapter presents the new features that have been added to the various versions of Bacula. \section{New Features in 7.4.3} \subsection{RunScripts} There are two new RunScript short cut directives implemented in the Director. They are: \begin{bVerbatim} Job { ... ConsoleRunBeforeJob = "console-command" ... } \end{bVerbatim} \begin{bVerbatim} Job { ... ConsoleRunAfterJob = "console-command" ... } \end{bVerbatim} As with other RunScript commands, you may have multiple copies of either the {\bf ConsoleRunBeforeJob} or the {\bf ConsoleRunAfterJob} in the same Job resource definition. \smallskip Please note that not all console commands are permitted, and that if you run a console command that requires a response, the results are not determined (i.e. it will probably fail). \section{New Features in 7.4.0} \subsection{Verify Volume Data} It is now possible to have a Verify Job configured with \texttt{level=Data} to reread all records from a job and optionally check the size and the checksum of all files. \begin{bVerbatim} # Verify Job definition Job { Name = VerifyData Level = Data Client = 127.0.0.1-fd # Use local file daemon FileSet = Dummy # Will be adapted during the job Storage = File # Should be the right one Messages = Standard Pool = Default } # Backup Job definition Job { Name = MyBackupJob Type = Backup Client = windows1 FileSet = MyFileSet Pool = 1Month Storage = File } FileSet { Name = MyFileSet Include { Options { Verify = s5 Signature = MD5 } File = / } \end{bVerbatim} To run the Verify job, it is possible to use the ``jobid'' parameter of the ``run'' command. \begin{bVerbatim} *run job=VerifyData jobid=10 Run Verify Job JobName: VerifyData Level: Data Client: 127.0.0.1-fd FileSet: Dummy Pool: Default (From Job resource) Storage: File (From Job resource) Verify Job: MyBackupJob.2015-11-11_09.41.55_03 Verify List: /opt/bacula/working/working/VerifyVol.bsr When: 2015-11-11 09:47:38 Priority: 10 OK to run? (yes/mod/no): yes Job queued. JobId=14 ... 11-Nov 09:46 my-dir JobId 13: Bacula 7.4.0 (13Nov15): Build OS: x86_64-unknown-linux-gnu archlinux JobId: 14 Job: VerifyData.2015-11-11_09.46.29_03 FileSet: MyFileSet Verify Level: Data Client: 127.0.0.1-fd Verify JobId: 10 Verify Job:q Start time: 11-Nov-2015 09:46:31 End time: 11-Nov-2015 09:46:32 Files Expected: 1,116 Files Examined: 1,116 Non-fatal FD errors: 0 SD Errors: 0 FD termination status: Verify differences SD termination status: OK Termination: Verify Differences \end{bVerbatim} The current Verify Data implementation requires specifying the correct Storage resource in the Verify job. The Storage resource can be changed with the bconsole command line and with the menu. \subsection{Bconsole ``list jobs'' command options} The \texttt{list jobs} bconsole command now accepts new command line options: \begin{itemize} \item \textbf{joberrors} Display jobs with JobErrors \item \textbf{jobstatus=T} Display jobs with the specified status code \item \textbf{client=cli} Display jobs for a specified client \item \textbf{order=asc/desc} Change the output format of the job list. The jobs are sorted by start time and JobId, the sort can use ascendant (asc) or descendant (desc) (default) value. \end{itemize} \subsection{Minor Enhancements} \subsubsection{New Bconsole "Tee All" Command} The ``@tall'' command allows logging all input/output from a console session. \begin{bVerbatim} *@tall /tmp/log *st dir ... \end{bVerbatim} \subsection{Windows Encrypted File System (EFS) Support} The Bacula Enterprise Windows File Daemon for the community version 7.4.0 now automatically supports files and directories that are encrypted on Windows filesystem. \subsection{SSL Connections to MySQL} There are five new Directives for the Catalog resource in the {\bf bacula-dir.conf} file that you can use to encrypt the communications between Bacula and MySQL for additional security. \begin{description} \item [dbsslkey] takes a string variable that specifies the filename of an SSL key file. \item [dbsslcert] takes a string variable that specifies the filename of an SSL certificate file. \item [dbsslca] takes a string variable that specifies the filename of a SSL CA (certificate authority) certificate. \item [dbsslcipher] takes a string variable that specifies the cipher to be used. \end{description} \subsection{Max Virtual Full Interval} This is a new Job resource directive that specifies the time in seconds that is a maximum time between Virtual Full jobs. It is much like the Max Full Interval directive but applies to Virtual Full jobs rather that Full jobs. \subsection{New List Volumes Output} The {\bf list} and {\bf llist} commands have been modified so that when listing Volumes a new pseudo field {\bf expiresin} will be printed. This field is the number of seconds in which the retention period will expire. If the retention period has already expired the value will be zero. Any non-zero value means that the retention period is still in effect. An example with many columns shorted for display purpose is: \begin{bVerbatim} *list volumes Pool: Default *list volumes Pool: Default +----+---------------+-----------+---------+-------------+-----------+ | id | volumename | volstatus | enabled | volbytes | expiresin | +----+---------------+-----------+---------+-------------+-----------+ | 1 | TestVolume001 | Full | 1 | 249,940,696 | 0 | | 2 | TestVolume002 | Full | 1 | 249,961,704 | 1 | | 3 | TestVolume003 | Full | 1 | 249,961,704 | 2 | | 4 | TestVolume004 | Append | 1 | 127,367,896 | 3 | +----+---------------+-----------+---------+-------------+-----------+ \end{bVerbatim} %% %% \chapter{New Features in 7.2.0} This chapter presents the new features that have been added to the various versions of Bacula. \section{New Features in 7.2.0} \subsection{New Job Edit Codes \%E \%R} In various places such as RunScripts, you have now access to \%E to get the number of non-fatal errors for the current Job and \%R to get the number of bytes read from disk or from the network during a job. \subsection{Enable/Disable commands} The \textbf{bconsole} \textbf{enable} and \textbf{disable} commands have been extended from enabling/disabling Jobs to include Clients, Schedule, and Storage devices. Examples: \begin{bVerbatim} disable Job=NightlyBackup Client=Windows-fd \end{bVerbatim} will disable the Job named \textbf{NightlyBackup} as well as the client named \textbf{Windows-fd}. \begin{bVerbatim} disable Storage=LTO-changer Drive=1 \end{bVerbatim} will disable the first drive in the autochanger named \textbf{LTO-changer}. Please note that doing a \textbf{reload} command will set any values changed by the enable/disable commands back to the values in the bacula-dir.conf file. The Client and Schedule resources in the bacula-dir.conf file now permit the directive Enable = yes or Enable = no. \section{Bacula 7.2} \subsection{Snapshot Management} Bacula 7.2 is now able to handle Snapshots on Linux/Unix systems. Snapshots can be automatically created and used to backup files. It is also possible to manage Snapshots from Bacula's \texttt{bconsole} tool through a unique interface. \subsubsection{Snapshot Backends} The following Snapshot backends are supported: \begin{itemize} \item BTRFS \item ZFS \item LVM\footnote{Some restrictions described in \vref{LVMBackend} applies to the LVM backend} \end{itemize} By default, Snapshots are mounted (or directly available) under \textbf{.snapshots} directory on the root filesystem. (On ZFS, the default is \textbf{.zfs/snapshots}). \smallskip{} The Snapshot backend program is called \textbf{bsnapshot} and is available in the \textbf{bacula-enterprise-snapshot} package. In order to use the Snapshot Management feature, the package must be installed on the Client. \smallskip{} \label{bsnapshotconf} The \textbf{bsnapshot} program can be configured using \texttt{/opt/bacula/etc/bsnapshot.conf} file. The following parameters can be adjusted in the configuration file: \begin{itemize} \item \texttt{trace=} Specify a trace file \item \texttt{debug=} Specify a debug level \item \texttt{sudo=} Use sudo to run commands \item \texttt{disabled=} Disable snapshot support \item \texttt{retry=} Configure the number of retries for some operations \item \texttt{snapshot\_dir=} Use a custom name for the Snapshot directory. (\textbf{.SNAPSHOT}, \textbf{.snapdir}, etc...) \item \texttt{lvm\_snapshot\_size=} Specify a custom snapshot size for a given LVM volume \end{itemize} \begin{bVerbatim} # cat /opt/bacula/etc/bsnapshot.conf trace=/tmp/snap.log debug=10 lvm_snapshot_size=/dev/ubuntu-vg/root:5% \end{bVerbatim} \subsubsection{Application Quiescing} When using Snapshots, it is very important to quiesce applications that are running on the system. The simplest way to quiesce an application is to stop it. Usually, taking the Snapshot is very fast, and the downtime is only about a couple of seconds. If downtime is not possible and/or the application provides a way to quiesce, a more advanced script can be used. An example is described on \vref{SnapRunScriptExample}. \subsubsection{New Director Directives} The use of the Snapshot Engine on the FileDaemon is determined by the new \textbf{Enable Snapshot} FileSet directive. The default is \textbf{no}. \begin{bVerbatim} FileSet { Name = LinuxHome Enable Snapshot = yes Include { Options = { Compression = LZO } File = /home } } \end{bVerbatim} By default, Snapshots are deleted from the Client at the end of the backup. To keep Snapshots on the Client and record them in the Catalog for a determined period, it is possible to use the \textbf{Snapshot Retention} directive in the Client or in the Job resource. The default value is 0 secconds. If, for a given Job, both Client and Job \textbf{Snapshot Retention} directives are set, the Job directive will be used. \begin{bVerbatim} Client { Name = linux1 ... Snapshot Retention = 5 days } \end{bVerbatim} To automatically prune Snapshots, it is possible to use the following RunScript command: \begin{bVerbatim} Job { ... Client = linux1 ... RunScript { RunsOnClient = no Console = "prune snapshot client=%c yes" RunsAfter = yes } } \end{bVerbatim} \smallskip{} In RunScripts, the \texttt{AfterSnapshot} keyword for the \texttt{RunsWhen} directive will allow a command to be run just after the Snapshot creation. \texttt{AfterSnapshot} is a synonym for the \texttt{AfterVSS} keyword. \label{SnapRunScriptExample} \begin{bVerbatim} Job { ... RunScript { Command = "/etc/init.d/mysql start" RunsWhen = AfterSnapshot RunsOnClient = yes } RunScript { Command = "/etc/init.d/mysql stop" RunsWhen = Before RunsOnClient = yes } } \end{bVerbatim} \subsubsection{Job Output Information} Information about Snapshots are displayed in the Job output. The list of all devices used by the Snapshot Engine is displayed, and the Job summary indicates if Snapshots were available. \begin{bVerbatim} JobId 3: Create Snapshot of /home/build JobId 3: Create Snapshot of /home/build/subvol JobId 3: Delete snapshot of /home/build JobId 3: Delete snapshot of /home/build/subvol ... JobId 3: Bacula 127.0.0.1-dir 7.2.0 (23Jul15): Build OS: x86_64-unknown-linux-gnu archlinux JobId: 3 Job: Incremental.2015-02-24_11.20.27_08 Backup Level: Full ... Snapshot/VSS: yes ... Termination: Backup OK \end{bVerbatim} \subsubsection{New ``snapshot'' Bconsole Commands} The new \textbf{snapshot} command will display by default the following menu: \begin{bVerbatim} *snapshot Snapshot choice: 1: List snapshots in Catalog 2: List snapshots on Client 3: Prune snapshots 4: Delete snapshot 5: Update snapshot parameters 6: Update catalog with Client snapshots 7: Done Select action to perform on Snapshot Engine (1-7): \end{bVerbatim} The \textbf{snapshot} command can also have the following parameters: \begin{bVerbatim} [client= | job= | jobid=] [delete | list | listclient | prune | sync | update] \end{bVerbatim} It is also possible to use traditional \texttt{list}, \texttt{llist}, \texttt{update}, \texttt{prune} or \texttt{delete} commands on Snapshots. \begin{bVerbatim} *llist snapshot jobid=5 snapshotid: 1 name: NightlySave.2015-02-24_12.01.00_04 createdate: 2015-02-24 12:01:03 client: 127.0.0.1-fd fileset: Full Set jobid: 5 volume: /home/.snapshots/NightlySave.2015-02-24_12.01.00_04 device: /home/btrfs type: btrfs retention: 30 comment: \end{bVerbatim} \begin{bVerbatim} * snapshot listclient Automatically selected Client: 127.0.0.1-fd Connecting to Client 127.0.0.1-fd at 127.0.0.1:8102 Snapshot NightlySave.2015-02-24_12.01.00_04: Volume: /home/.snapshots/NightlySave.2015-02-24_12.01.00_04 Device: /home CreateDate: 2015-02-24 12:01:03 Type: btrfs Status: OK Error: \end{bVerbatim} \smallskip{} With the \textsl{Update catalog with Client snapshots} option (or \textbf{snapshot sync}), the Director contacts the FileDaemon, lists snapshots of the system and creates catalog records of the Snapshots. \begin{bVerbatim} *snapshot sync Automatically selected Client: 127.0.0.1-fd Connecting to Client 127.0.0.1-fd at 127.0.0.1:8102 Snapshot NightlySave.2015-02-24_12.35.47_06: Volume: /home/.snapshots/NightlySave.2015-02-24_12.35.47_06 Device: /home CreateDate: 2015-02-24 12:35:47 Type: btrfs Status: OK Error: Snapshot added in Catalog *llist snapshot snapshotid: 13 name: NightlySave.2015-02-24_12.35.47_06 createdate: 2015-02-24 12:35:47 client: 127.0.0.1-fd fileset: jobid: 0 volume: /home/.snapshots/NightlySave.2015-02-24_12.35.47_06 device: /home type: btrfs retention: 0 comment: \end{bVerbatim} % list % llist % prune % delete % update snapshot % sync \subsubsection{LVM Backend Restrictions} \label{LVMBackend} LVM Snapshots are quite primitive compared to ZFS, BTRFS, NetApp and other systems. For example, it is not possible to use Snapshots if the Volume Group (VG) is full. The administrator must keep some free space in the VG to create Snapshots. The amount of free space required depends on the activity of the Logical Volume (LV). \textbf{bsnapshot} uses 10\% of the LV by default. This number can be configured per LV in the \textbf{bsnapshot.conf} file. \begin{bVerbatim} [root@system1]# vgdisplay --- Volume group --- VG Name vg_ssd System ID Format lvm2 ... VG Size 29,81 GiB PE Size 4,00 MiB Total PE 7632 Alloc PE / Size 125 / 500,00 MiB Free PE / Size 7507 / 29,32 GiB ... \end{bVerbatim} It is also not advisable to leave snapshots on the LVM backend. Having multiple snapshots of the same LV on LVM will slow down the system. \subsubsection{Debug Options} To get low level information about the Snapshot Engine, the debug tag ``snapshot'' should be used in the \textbf{setdebug} command. \begin{bVerbatim} * setdebug level=10 tags=snapshot client * setdebug level=10 tags=snapshot dir \end{bVerbatim} \subsection{Minor Enhancements} \subsubsection{Storage Daemon Reports Disk Usage} The \texttt{status storage} command now reports the space available on disk devices: \begin{bVerbatim} ... Device status: Device file: "FileStorage" (/bacula/arch1) is not open. Available Space=5.762 GB == Device file: "FileStorage1" (/bacula/arch2) is not open. Available Space=5.862 GB \end{bVerbatim} \subsection{Data Encryption Cipher Configuration} Bacula Enterprise version 8.0 and later now allows configuration of the data encryption cipher and the digest algorithm. Previously, the cipher was forced to AES 128, but it is now possible to choose between the following ciphers: \begin{itemize} \item AES128 (default) \item AES192 \item AES256 \item blowfish \end{itemize} The digest algorithm was set to SHA1 or SHA256 depending on the local OpenSSL options. We advise you to not modify the PkiDigest default setting. Please, refer to the OpenSSL documentation to understand the pros and cons regarding these options. \begin{bVerbatim} FileDaemon { ... PkiCipher = AES256 } \end{bVerbatim} \subsubsection*{New Option Letter ``M'' for Accurate Directive in FileSet} % waa - 20150317 - is 8.0.5 correct here? Added in version 8.0.5, the new ``M'' option letter for the Accurate directive in the FileSet Options block, which allows comparing the modification time and/or creation time against the last backup timestamp. This is in contrast to the existing options letters ``m'' and/or ``c'', mtime and ctime, which are checked against the stored catalog values, which can vary accross different machines when using the BaseJob feature. The advantage of the new ``M'' option letter for Jobs that refer to BaseJobs is that it will instruct Bacula to backup files based on the last backup time, which is more useful because the mtime/ctime timestamps may differ on various Clients, causing files to be needlessly backed up. \smallskip{} \begin{bVerbatim} Job { Name = USR Level = Base FileSet = BaseFS ... } Job { Name = Full FileSet = FullFS Base = USR ... } FileSet { Name = BaseFS Include { Options { Signature = MD5 } File = /usr } } FileSet { Name = FullFS Include { Options { Accurate = Ms # check for mtime/ctime of last backup timestamp and Size Signature = MD5 } File = /home File = /usr } } \end{bVerbatim} \subsection{Read Only Storage Devices} This version of Bacula allows you to define a Storage deamon device to be read-only. If the {\bf Read Only} directive is specified and enabled, the drive can only be used for read operations. The {\bf Read Only} directive can be defined in any bacula-sd.conf Device resource, and is most useful for reserving one or more drives for restores. An example is: \begin{bVerbatim} Read Only = yes \end{bVerbatim} \subsection{New Resume Command} The new \texttt{resume} command does exactly the same thing as a {\bf restart} command, but for some users the name may be more logical because in general the {\bf restart} command is used to resume running a Job that was incomplete. \subsection{New Prune ``Expired'' Volume Command} In Bacula Enterprise 6.4, it is now possible to prune all volumes (from a pool, or globally) that are ``expired''. This option can be scheduled after or before the backup of the catalog and can be combined with the \texttt{Truncate On Purge} option. The \texttt{prune expired volme} command may be used instead of the \texttt{manual\_prune.pl} script. \begin{bVerbatim} * prune expired volume * prune expired volume pool=FullPool \end{bVerbatim} To schedule this option automatically, it can be added to the Catalog backup job definition. \begin{bVerbatim} Job { Name = CatalogBackup ... RunScript { Console = "prune expired volume yes" RunsWhen = Before } } \end{bVerbatim} \subsection{New Job Edit Codes \%P \%C} In various places such as RunScripts, you have now access to \%P to get the current Bacula process ID (PID) and \%C to know if the current job is a cloned job. \subsection{Enhanced Status and Error Messages} We have enhanced the Storage daemon status output to be more readable. This is important when there are a large number of devices. In addition to formatting changes, it also includes more details on which devices are reading and writing. A number of error messages have been enhanced to have more specific data on what went wrong. If a file changes size while being backed up the old and new size are reported. \subsection{Miscellaneous New Features} \begin{itemize} \item Allow unlimited line lengths in .conf files (previously limited to 2000 characters). \item Allow /dev/null in ChangerCommand to indicated a Virtual Autochanger. \item Add a --fileprune option to the manual\_prune.pl script. \item Add a -m option to make\_catalog\_backup.pl to do maintenance on the catalog. \item Safer code that cleans up the working directory when starting the daemons. It limits what files can be deleted, hence enhances security. \item Added a new .ls command in bconsole to permit browsing a client's filesystem. \item Fixed a number of bugs, includes some obscure seg faults, and a race condition that occurred infrequently when running Copy, Migration, or Virtual Full backups. \item Upgraded to a newer version of Qt4 for bat. All indications are that this will improve bat's stability on Windows machines. \item The Windows installers now detect and refuse to install on an OS that does not match the 32/64 bit value of the installer. \end{itemize} \subsection{FD Storage Address} When the Director is behind a NAT, in a WAN area, to connect to % the FileDaemon or the StorageDaemon, the Director uses an ``external'' ip address, and the FileDaemon should use an ``internal'' IP address to contact the StorageDaemon. The normal way to handle this situation is to use a canonical name such as ``storage-server'' that will be resolved on the Director side as the WAN address and on the Client side as the LAN address. This is now possible to configure this parameter using the new directive \texttt{FDStorageAddress} in the Storage or Client resource. %%\bimageH{BackupOverWan1}{Backup Over WAN}{figbs6:fdstorageaddress} % \label{fig:fdstorageaddress} \begin{bVerbatim} Storage { Name = storage1 Address = 65.1.1.1 FD Storage Address = 10.0.0.1 SD Port = 9103 ... } \end{bVerbatim} % # or in the Client resouce % \begin{bVerbatim} Client { Name = client1 Address = 65.1.1.2 FD Storage Address = 10.0.0.1 FD Port = 9102 ... } \end{bVerbatim} Note that using the Client \texttt{FDStorageAddress} directive will not allow to use multiple Storage Daemon, all Backup or Restore requests will be sent to the specified \texttt{FDStorageAddress}. \subsection{Maximum Concurrent Read Jobs} This is a new directive that can be used in the {\bf bacula-dir.conf} file in the Storage resource. The main purpose is to limit the number of concurrent Copy, Migration, and VirtualFull jobs so that they don't monopolize all the Storage drives causing a deadlock situation where all the drives are allocated for reading but none remain for writing. This deadlock situation can occur when running multiple simultaneous Copy, Migration, and VirtualFull jobs. \smallskip The default value is set to 0 (zero), which means there is no limit on the number of read jobs. Note, limiting the read jobs does not apply to Restore jobs, which are normally started by hand. A reasonable value for this directive is one half the number of drives that the Storage resource has rounded down. Doing so, will leave the same number of drives for writing and will generally avoid over committing drives and a deadlock. \subsection{Incomplete Jobs} During a backup, if the Storage daemon experiences disconnection with the File daemon during backup (normally a comm line problem or possibly an FD failure), under conditions that the SD determines to be safe it will make the failed job as Incomplete rather than failed. This is done only if there is sufficient valid backup data that was written to the Volume. The advantage of an Incomplete job is that it can be restarted by the new bconsole {\bf restart} command from the point where it left off rather than from the beginning of the jobs as is the case with a cancel. \subsection{The Stop Command} Bacula has been enhanced to provide a {\bf stop} command, very similar to the {\bf cancel} command with the main difference that the Job that is stopped is marked as Incomplete so that it can be restarted later by the {\bf restart} command where it left off (see below). The {\bf stop} command with no arguments, will like the cancel command, prompt you with the list of running jobs allowing you to select one, which might look like the following: \begin{bVerbatim} *stop Select Job: 1: JobId=3 Job=Incremental.2012-03-26_12.04.26_07 2: JobId=4 Job=Incremental.2012-03-26_12.04.30_08 3: JobId=5 Job=Incremental.2012-03-26_12.04.36_09 Choose Job to stop (1-3): 2 2001 Job "Incremental.2012-03-26_12.04.30_08" marked to be stopped. 3000 JobId=4 Job="Incremental.2012-03-26_12.04.30_08" marked to be stopped. \end{bVerbatim} \subsection{The Restart Command} The new {\bf Restart command} allows console users to restart a canceled, failed, or incomplete Job. For canceled and failed Jobs, the Job will restart from the beginning. For incomplete Jobs the Job will restart at the point that it was stopped either by a stop command or by some recoverable failure. \smallskip If you enter the {\bf restart} command in bconsole, you will get the following prompts: \begin{bVerbatim} *restart You have the following choices: 1: Incomplete 2: Canceled 3: Failed 4: All Select termination code: (1-4): \end{bVerbatim} If you select the {\bf All} option, you may see something like: \begin{bVerbatim} Select termination code: (1-4): 4 +-------+-------------+---------------------+------+-------+----------+-----------+-----------+ | jobid | name | starttime | type | level | jobfiles | jobbytes | jobstatus | +-------+-------------+---------------------+------+-------+----------+-----------+-----------+ | 1 | Incremental | 2012-03-26 12:15:21 | B | F | 0 | 0 | A | | 2 | Incremental | 2012-03-26 12:18:14 | B | F | 350 | 4,013,397 | I | | 3 | Incremental | 2012-03-26 12:18:30 | B | F | 0 | 0 | A | | 4 | Incremental | 2012-03-26 12:18:38 | B | F | 331 | 3,548,058 | I | +-------+-------------+---------------------+------+-------+----------+-----------+-----------+ Enter the JobId list to select: \end{bVerbatim} Then you may enter one or more JobIds to be restarted, which may take the form of a list of JobIds separated by commas, and/or JobId ranges such as {\bf 1-4}, which indicates you want to restart JobIds 1 through 4, inclusive. \subsection{Job Bandwidth Limitation} The new {\bf Job Bandwidth Limitation} directive may be added to the File daemon's and/or Director's configuration to limit the bandwidth used by a Job on a Client. It can be set in the File daemon's conf file for all Jobs run in that File daemon, or it can be set for each Job in the Director's conf file. The speed is always specified in bytes per second. For example: \begin{bVerbatim} FileDaemon { Name = localhost-fd Working Directory = /some/path Pid Directory = /some/path ... Maximum Bandwidth Per Job = 5Mb/s } \end{bVerbatim} The above example would cause any jobs running with the FileDaemon to not exceed 5 megabytes per second of throughput when sending data to the Storage Daemon. Note, the speed is always specified in bytes per second (not in bits per second), and the case (upper/lower) of the specification characters is ignored (i.e. 1MB/s = 1Mb/s). You may specify the following speed parameter modifiers: k/s (1,000 bytes per second), kb/s (1,024 bytes per second), m/s (1,000,000 bytes per second), or mb/s (1,048,576 bytes per second). For example: \begin{bVerbatim} Job { Name = locahost-data FileSet = FS_localhost Accurate = yes ... Maximum Bandwidth = 5Mb/s ... } \end{bVerbatim} The above example would cause Job \texttt{localhost-data} to not exceed 5MB/s of throughput when sending data from the File daemon to the Storage daemon. A new console command \texttt{setbandwidth} permits to set dynamically the maximum throughput of a running Job or for future jobs of a Client. \begin{bVerbatim} * setbandwidth limit=1000 jobid=10 \end{bVerbatim} Please note that the value specified for the \texttt{limit} command line parameter is always in units of 1024 bytes (i.e. the number is multiplied by 1024 to give the number of bytes per second). As a consequence, the above limit of 1000 will be interpreted as a limit of 1000 * 1024 = 1,024,000 bytes per second. \subsection{Always Backup a File} When the Accurate mode is turned on, you can decide to always backup a file by using then new {\bf A} Accurate option in your FileSet. For example: \begin{bVerbatim} Job { Name = ... FileSet = FS_Example Accurate = yes ... } FileSet { Name = FS_Example Include { Options { Accurate = A } File = /file File = /file2 } ... } \end{bVerbatim} This project was funded by Bacula Systems based on an idea of James Harper and is available with the Bacula Enterprise Edition. \subsection{Setting Accurate Mode at Runtime} You are now able to specify the Accurate mode on the \texttt{run} command and in the Schedule resource. \begin{bVerbatim} * run accurate=yes job=Test \end{bVerbatim} \begin{bVerbatim} Schedule { Name = WeeklyCycle Run = Full 1st sun at 23:05 Run = Differential accurate=yes 2nd-5th sun at 23:05 Run = Incremental accurate=no mon-sat at 23:05 } \end{bVerbatim} It can allow you to save memory and and CPU resources on the catalog server in some cases. \medskip These advanced tuning options are available with the Bacula Enterprise Edition. % Common with community \subsection{Additions to RunScript variables} You can have access to JobBytes, JobFiles and Director name using \%b, \%F and \%D in your runscript command. The Client address is now available through \%h. \begin{bVerbatim} RunAfterJob = "/bin/echo Job=%j JobBytes=%b JobFiles=%F ClientAddress=%h Dir=%D" \end{bVerbatim} \subsection{LZO Compression} LZO compression was added in the Unix File Daemon. From the user point of view, it works like the GZIP compression (just replace {\bf compression=GZIP} with {\bf compression=LZO}). For example: \begin{bVerbatim} Include { Options { compression=LZO } File = /home File = /data } \end{bVerbatim} LZO provides much faster compression and decompression speed but lower compression ratio than GZIP. It is a good option when you backup to disk. For tape, the built-in compression may be a better option. LZO is a good alternative for GZIP1 when you don't want to slow down your backup. On a modern CPU it should be able to run almost as fast as: \begin{itemize} \item your client can read data from disk. Unless you have very fast disks like SSD or large/fast RAID array. \item the data transfers between the file daemon and the storage daemon even on a 1Gb/s link. \end{itemize} Note that bacula only use one compression level LZO1X-1. \medskip The code for this feature was contributed by Laurent Papier. \subsection{Purge Migration Job} The new {\bf Purge Migration Job} directive may be added to the Migration Job definition in the Director's configuration file. When it is enabled the Job that was migrated during a migration will be purged at the end of the migration job. For example: \begin{bVerbatim} Job { Name = "migrate-job" Type = Migrate Level = Full Client = localhost-fd FileSet = "Full Set" Messages = Standard Storage = DiskChanger Pool = Default Selection Type = Job Selection Pattern = ".*Save" ... Purge Migration Job = yes } \end{bVerbatim} \medskip This project was submitted by Dunlap Blake; testing and documentation was funded by Bacula Systems. \subsection{Changes in the Pruning Algorithm} We rewrote the job pruning algorithm in this version. Previously, in some users reported that the pruning process at the end of jobs was very long. It should not be longer the case. Now, Bacula won't prune automatically a Job if this particular Job is needed to restore data. Example: \begin{bVerbatim} JobId: 1 Level: Full JobId: 2 Level: Incremental JobId: 3 Level: Incremental JobId: 4 Level: Differential .. Other incrementals up to now \end{bVerbatim} In this example, if the Job Retention defined in the Pool or in the Client resource causes that Jobs with Jobid in 1,2,3,4 can be pruned, Bacula will detect that JobId 1 and 4 are essential to restore data at the current state and will prune only JobId 2 and 3. \texttt{Important}, this change affect only the automatic pruning step after a Job and the \texttt{prune jobs} Bconsole command. If a volume expires after the \texttt{VolumeRetention} period, important jobs can be pruned. \subsection{Ability to Verify any specified Job} You now have the ability to tell Bacula which Job should verify instead of automatically verify just the last one. This feature can be used with VolumeToCatalog, DiskToCatalog and Catalog level. To verify a given job, just specify the Job jobid in argument when starting the job. \begin{bVerbatim} *run job=VerifyVolume jobid=1 level=VolumeToCatalog Run Verify job JobName: VerifyVolume Level: VolumeToCatalog Client: 127.0.0.1-fd FileSet: Full Set Pool: Default (From Job resource) Storage: File (From Job resource) Verify Job: VerifyVol.2010-09-08_14.17.17_03 Verify List: /tmp/regress/working/VerifyVol.bsr When: 2010-09-08 14:17:31 Priority: 10 OK to run? (yes/mod/no): \end{bVerbatim} \chapter{New Features in 7.0.0} This chapter presents the new features that have been added to the various versions of Bacula. \section{New Features in 7.0.0} \subsection{Storage daemon to Storage daemon} Bacula version 7.0 permits SD to SD transfer of Copy and Migration Jobs. This permits what is commonly referred to as replication or off-site transfer of Bacula backups. It occurs automatically, if the source SD and destination SD of a Copy or Migration job are different. The following picture shows how this works. \includegraphics[width=0.8\linewidth]{sd-to-sd} \subsection{SD Calls Client} If the {\bf SD Calls Client} directive is set to true in a Client resource any Backup, Restore, Verify, Copy, or Migration Job where the client is involved, the client will wait for the Storage daemon to contact it. By default this directive is set to false, and the Client will call the Storage daemon. This directive can be useful if your Storage daemon is behind a firewall that permits outgoing connections but not incoming one. The following picture shows the communications connection paths in both cases. \includegraphics[width=0.8\linewidth]{sd-calls-client} \subsection{Next Pool} In previous versions of Bacula the Next Pool directive could be specified in the Pool resource for use with Migration and Copy Jobs. The Next Pool concept has been extended in Bacula version 7.0.0 to allow you to specify the Next Pool directive in the Job resource as well. If specified in the Job resource, it will override any value specified in the Pool resource. In addition to being permitted in the Job resource, the {\bf nextpool=xxx} specification can be specified as a run override in the {\bf run} directive of a Schedule resource. Any {\bf nextpool} specification in a {\bf run} directive will override any other specification in either the Job or the Pool. In general, more information is displayed in the Job log on exactly which Next Pool specification is ultimately used. \subsection{status storage} The bconsole {\bf status storage} has been modified to attempt to eliminate duplicate storage resources and only show one that references any given storage daemon. This might be confusing at first, but tends to make a much more compact list of storage resource from which to select if there are multiple storage devices in the same storage daemon. If you want the old behavior (always display all storage resources) simply add the keyword {\bf select} to the command -- i.e. use {\bf status select storage}. \subsection{status schedule} A new status command option called {\bf scheduled} has been implemented in bconsole. By default it will display 20 lines of the next scheduled jobs. For example, with the default bacula-dir.conf configuration file, a bconsole command {\bf status scheduled} produces: \begin{bVerbatim} Scheduled Jobs: Level Type Pri Scheduled Job Name Schedule ====================================================================== Differential Backup 10 Sun 30-Mar 23:05 BackupClient1 WeeklyCycle Incremental Backup 10 Mon 24-Mar 23:05 BackupClient1 WeeklyCycle Incremental Backup 10 Tue 25-Mar 23:05 BackupClient1 WeeklyCycle ... Full Backup 11 Mon 24-Mar 23:10 BackupCatalog WeeklyCycleAfterBackup Full Backup 11 Wed 26-Mar 23:10 BackupCatalog WeeklyCycleAfterBackup ... ==== \end{bVerbatim} Note, the output is listed by the Jobs found, and is not sorted chronologically. \smallskip This command has a number of options, most of which act as filters: \begin{itemize} \item {\bf days=nn} This specifies the number of days to list. The default is 10 but can be set from 0 to 500. \item {\bf limit=nn} This specifies the limit to the number of lines to print. The default is 100 but can be any number in the range 0 to 2000. \item {\bf time="YYYY-MM-DD HH:MM:SS"} Sets the start time for listing the scheduled jobs. The default is to use the current time. Note, the time value must be specified inside double quotes and must be in the exact form shown above. \item {\bf schedule=schedule-name} This option restricts the output to the named schedule. \item {\bf job=job-name} This option restricts the output to the specified Job name. \end{itemize} \subsection{Data Encryption Cipher Configuration} Bacula version 7.0 and later now allows to configure the data encryption cipher and the digest algorithm. The cipher was forced to AES 128, and it is now possible to choose between the following ciphers: \begin{itemize} \item AES128 (default) \item AES192 \item AES256 \item blowfish \end{itemize} The digest algorithm was set to SHA1 or SHA256 depending on the local OpenSSL options. We advise you to not modify the PkiDigest default setting. Please, refer to OpenSSL documentation to know about pro and cons on these options. \begin{bVerbatim} FileDaemon { ... PkiCipher = AES256 } \end{bVerbatim} \subsection{New Truncate Command} We have added a new truncate command to bconsole, which will truncate a Volume if the Volume is purged and if the Volume is also marked {\bf Action On Purge = Truncate}. This feature was originally added in Bacula version 5.0.1, but the mechanism for actually doing the truncate required the user to enter a command such as: \begin{bVerbatim} purge volume action=truncate storage=File pool=Default \end{bVerbatim} The above command is now simplified to be: \begin{bVerbatim} truncate storage=File pool=Default \end{bVerbatim} \subsection{Migration/Copy/VirtualFull Performance Enhancements} The Bacula Storage daemon now permits multiple jobs to simultaneously read the same disk Volume, which gives substantial performance enhancements when running Migration, Copy, or VirtualFull jobs that read disk Volumes. Our testing shows that when running multiple simultaneous jobs, the jobs can finish up to ten times faster with this version of Bacula. This is built-in to the Storage daemon, so it happens automatically and transparently. \subsection{VirtualFull Backup Consolidation Enhancements} By default Bacula selects jobs automatically for a VirtualFull, however, you may want to create the Virtual backup based on a particular backup (point in time) that exists. For example, if you have the following backup Jobs in your catalog: \begin{bVerbatim} +-------+---------+-------+----------+----------+-----------+ | JobId | Name | Level | JobFiles | JobBytes | JobStatus | +-------+---------+-------+----------+----------+-----------+ | 1 | Vbackup | F | 1754 | 50118554 | T | | 2 | Vbackup | I | 1 | 4 | T | | 3 | Vbackup | I | 1 | 4 | T | | 4 | Vbackup | D | 2 | 8 | T | | 5 | Vbackup | I | 1 | 6 | T | | 6 | Vbackup | I | 10 | 60 | T | | 7 | Vbackup | I | 11 | 65 | T | | 8 | Save | F | 1758 | 50118564 | T | +-------+---------+-------+----------+----------+-----------+ \end{bVerbatim} and you want to consolidate only the first 3 jobs and create a virtual backup equivalent to Job 1 + Job 2 + Job 3, you will use \texttt{jobid=3} in the \texttt{run} command, then Bacula will select the previous Full backup, the previous Differential (if any) and all subsequent Incremental jobs. \begin{bVerbatim} run job=Vbackup jobid=3 level=VirtualFull \end{bVerbatim} If you want to consolidate a specific job list, you must specify the exact list of jobs to merge in the run command line. For example, to consolidate the last Differential and all subsequent Incremental, you will use \texttt{jobid=4,5,6,7} or \texttt{jobid=4-7} on the run command line. As one of the Job in the list is a Differential backup, Bacula will set the new job level to Differential. If the list is composed only with Incremental jobs, the new job will have a level set to Incremental. \begin{bVerbatim} run job=Vbackup jobid=4-7 level=VirtualFull \end{bVerbatim} When using this feature, Bacula will automatically discard jobs that are not related to the current Job. For example, specifying \texttt{jobid=7,8}, Bacula will discard JobId 8 because it is not part of the same backup Job. We do not recommend it, but really want to consolidate jobs that have different names (so probably different clients, filesets, etc...), you must use \texttt{alljobid=} keyword instead of \texttt{jobid=}. \begin{bVerbatim} run job=Vbackup alljobid=1-3,6-8 level=VirtualFull \end{bVerbatim} \subsection{FD Storage Address} When the Director is behind a NAT, in a WAN area, to connect to % the FileDaemon or the StorageDaemon, the Director uses an ``external'' ip address, and the FileDaemon should use an ``internal'' IP address to contact the StorageDaemon. The normal way to handle this situation is to use a canonical name such as ``storage-server'' that will be resolved on the Director side as the WAN address and on the Client side as the LAN address. This is now possible to configure this parameter using the new directive \texttt{FDStorageAddress} in the Storage or Client resource. \includegraphics[width=0.8\linewidth]{BackupOverWan1} \label{fig:fdstorageaddress} \begin{bVerbatim} Storage { Name = storage1 Address = 65.1.1.1 FD Storage Address = 10.0.0.1 SD Port = 9103 ... } \end{bVerbatim} % # or in the Client resouce % \begin{bVerbatim} Client { Name = client1 Address = 65.1.1.2 FD Storage Address = 10.0.0.1 FD Port = 9102 ... } \end{bVerbatim} Note that using the Client \texttt{FDStorageAddress} directive will not allow to use multiple Storage Daemon, all Backup or Restore requests will be sent to the specified \texttt{FDStorageAddress}. \subsection{Job Bandwidth Limitation} The new {\bf Job Bandwidth Limitation} directive may be added to the File daemon's and/or Director's configuration to limit the bandwidth used by a Job on a Client. It can be set in the File daemon's conf file for all Jobs run in that File daemon, or it can be set for each Job in the Director's conf file. The speed is always specified in bytes per second. For example: \begin{bVerbatim} FileDaemon { Name = localhost-fd Working Directory = /some/path Pid Directory = /some/path ... Maximum Bandwidth Per Job = 5Mb/s } \end{bVerbatim} The above example would cause any jobs running with the FileDaemon to not exceed 5 megabytes per second of throughput when sending data to the Storage Daemon. Note, the speed is always specified in bytes per second (not in bits per second), and the case (upper/lower) of the specification characters is ignored (i.e. 1MB/s = 1Mb/s). You may specify the following speed parameter modifiers: k/s (1,000 bytes per second), kb/s (1,024 bytes per second), m/s (1,000,000 bytes per second), or mb/s (1,048,576 bytes per second). For example: \begin{bVerbatim} Job { Name = locahost-data FileSet = FS_localhost Accurate = yes ... Maximum Bandwidth = 5Mb/s ... } \end{bVerbatim} The above example would cause Job \texttt{localhost-data} to not exceed 5MB/s of throughput when sending data from the File daemon to the Storage daemon. A new console command \texttt{setbandwidth} permits to set dynamically the maximum throughput of a running Job or for future jobs of a Client. \begin{bVerbatim} * setbandwidth limit=1000 jobid=10 \end{bVerbatim} Please note that the value specified for the \texttt{limit} command line parameter is always in units of 1024 bytes (i.e. the number is multiplied by 1024 to give the number of bytes per second). As a consequence, the above limit of 1000 will be interpreted as a limit of 1000 * 1024 = 1,024,000 bytes per second. \medskip This project was funded by Bacula Systems. \subsection{Maximum Concurrent Read Jobs} This is a new directive that can be used in the {\bf bacula-dir.conf} file in the Storage resource. The main purpose is to limit the number of concurrent Copy, Migration, and VirtualFull jobs so that they don't monopolize all the Storage drives causing a deadlock situation where all the drives are allocated for reading but none remain for writing. This deadlock situation can occur when running multiple simultaneous Copy, Migration, and VirtualFull jobs. \smallskip The default value is set to 0 (zero), which means there is no limit on the number of read jobs. Note, limiting the read jobs does not apply to Restore jobs, which are normally started by hand. A reasonable value for this directive is one half the number of drives that the Storage resource has rounded down. Doing so, will leave the same number of drives for writing and will generally avoid over committing drives and a deadlock. \subsection{Director job Codes in Message Resource Commands} Before submitting the specified mail command to the operating system, Bacula performs character substitution like in Runscript commands. Bacula will now perform also specific Director character substitution. \smallskip{} The code for this feature was contributed by Bastian Friedrich. \subsection{Additions to RunScript variables} The following variables are now available in runscripts: \begin{itemize} \item current PID using \%P \item if the job is a clone job using \%C \end{itemize} \begin{bVerbatim} RunAfterJob = "/bin/echo Pid=%P isCloned=%C" \end{bVerbatim} \subsection{Read Only Storage Devices} This version of Bacula permits defining a Storage daemon device to be read-only. That is if the {\bf ReadOnly} directive is specified and enabled, the drive can only be used for read operations. The the {\bf ReadOnly} directive can be defined in any bacula-sd.conf Device resource, and is most useful to reserve one or more drives for restores. An example is: \begin{bVerbatim} Read Only = yes \end{bVerbatim} \subsection{New Prune ``Expired'' Volume Command} It is now possible to prune all volumes (from a pool, or globally) that are ``expired''. This option can be scheduled after or before the backup of the Catalog and can be combined with the Truncate On Purge option. The Expired Prune option can be used instead of the \texttt{manual\_prune.pl} script. \begin{bVerbatim} * prune expired volumes * prune expired volumes pool=FullPool \end{bVerbatim} To schedule this option automatically, it can be added to the BackupCatalog job definition. \begin{bVerbatim} Job { Name = CatalogBackup ... RunScript { Console = "prune expired volume yes" RunsWhen = Before } } \end{bVerbatim} \subsection{Hardlink Performance Enhancements} If you use a program such as Cyrus IMAP that creates very large numbers of hardlinks, the time to build the interactive restore tree can be excessively long. This version of Bacula has a new feature that automatically keeps the hardlinks associated with the restore tree in memory, which consumes a bit more memory but vastly speeds up building the tree. If the memory usage is too big for your system, you can reduce the amount of memory used during the restore command by adding the option {\bf optimizespeed=false} on the bconsole run command line. This feature was developed by Josip Almasi, and enhanced to be runtime dynamic by Kern Sibbald. \subsection{DisableCommand Directive} There is a new Directive named {\bf Disable Command} that can be put in the File daemon Client or Director resource. If it is in the Client, it applies globally, otherwise the directive applies only to the Director in which it is found. The Disable Command adds security to your File daemon by disabling certain commands. The commands that can be disabled are: \begin{bVerbatim} backup cancel setdebug= setbandwidth= estimate fileset JobId= level = restore endrestore session status .status storage verify RunBeforeNow RunBeforeJob RunAfterJob Run accurate \end{bVerbatim} On or more of these command keywords can be placed in quotes and separated by spaces on the Disable Command directive line. Note: the commands must be written exactly as they appear above. \subsection{Multiple Console Directors} Support for multiple bconsole and bat Directors in the bconsole.conf and bat.conf files has been implemented and/or improved. \subsection{Restricted Consoles} Better support for Restricted consoles has been implement for bconsole and bat. \subsection{Configuration Files} In previous versions of Bacula the configuration files for each component were limited to a maximum of 499 bytes per configuration file line. This version of Bacula permits unlimited input line lengths. This can be especially useful for specifying more complicated Migration/Copy SQL statements and in creating long restricted console ACL lists. \subsection{Maximum Spawned Jobs} The Job resource now permits specifying a number of {\bf Maximum Spawned Jobs}. The default is 600. This directive can be useful if you have big hardware and you do a lot of Migration/Copy jobs which start at the same time. In prior versions of Bacula, Migration/Copy was limited to spawning a maximum of 100 jobs at a time. \subsection{Progress Meter} The new File daemon has been enhanced to send its progress (files processed and bytes written) to the Director every 30 seconds. These figures can then be displayed with a bconsole {\bf status dir} command. \subsection{Scheduling a 6th Week} Prior version of Bacula permits specifying 1st through 5th week of a month (first through fifth) as a keyword on the {\bf run} directive of a Schedule resource. This version of Bacula also permits specifying the 6th week of a month with the keyword {\bf sixth} or {\bf 6th}. \subsection{Scheduling the Last Day of a Month} This version of Bacula now permits specifying the {\bf lastday} keyword in the {\bf run} directive of a Schedule resource. If {\bf lastday} is specified, it will apply only to those months specified on the {\bf run} directive. Note: by default all months are specified. \subsection{Improvements to Cancel and Restart bconsole Commands} The Restart bconsole command now allow selection of either canceled or failed jobs to be restarted. In addition both the {\bf cancel} and {\bf restart} bconsole commands permit entering a number of JobIds separated by commas or a range of JobIds indicated by a dash between the begin and end range (e.g. 3-10). Finally the two commands also allow one to enter the special keyword {\bf all} to select all the appropriate Jobs. \subsection{bconsole Performance Improvements} In previous versions of Bacula certain bconsole commands could wait a long time due to catalog lock contention. This was especially noticeable when a large number of jobs were running and putting their attributes into the catalog. This version uses a separate catalog connection that should significantly enhance performance. \subsection{New .bvfs\_decode\_lstat Command} There is a new bconsole command, which is {\bf .bvfs\_decode\_lstat} it requires one argument, which is {\bf lstat="lstat value to decode"}. An example command in bconsole and the output might be: \small \begin{bVerbatim} .bvfs_decode_lstat lstat="A A EHt B A A A JP BAA B BTL/A7 BTL/A7 BTL/A7 A A C" st_nlink=1 st_mode=16877 st_uid=0 st_gid=0 st_size=591 st_blocks=1 st_ino=0 st_ctime=1395650619 st_mtime=1395650619 st_mtime=1395650619 st_dev=0 LinkFI=0 \end{bVerbatim} \normalsize \subsection*{New Debug Options} In Bacula Enterprise version 8.0 and later, we introduced new options to the \texttt{setdebug} command. \smallskip{} If the \texttt{options} parameter is set, the following arguments can be used to control debug functions. \begin{itemize} \item [0] clear debug flags \item [i] Turn off, ignore bwrite() errors on restore on File Daemon \item [d] Turn off decomp of BackupRead() streams on File Daemon \item [t] Turn on timestamp in traces \item [T] Turn off timestamp in traces \item [c] Truncate trace file if trace file is activated \item [l] Turn on recoding events on P() and V() \item [p] Turn on the display of the event ring when doing a bactrace \end{itemize} \smallskip{} The following command will truncate the trace file and will turn on timestamps in the trace file. \begin{bVerbatim} * setdebug level=10 trace=1 options=ct fd \end{bVerbatim} \smallskip{} It is now possible to use \textsl{class} of debug messages called \texttt{tags} to control the debug output of Bacula daemons. \begin{itemize} \item [all] Display all debug messages \item [bvfs] Display BVFS debug messages \item [sql] Display SQL related debug messages \item [memory] Display memory and poolmem allocation messages \item [scheduler] Display scheduler related debug messages \end{itemize} \begin{bVerbatim} * setdebug level=10 tags=bvfs,sql,memory * setdebug level=10 tags=!bvfs # bacula-dir -t -d 200,bvfs,sql \end{bVerbatim} The \texttt{tags} option is composed of a list of tags, tags are separated by ``,'' or ``+'' or ``-'' or ``!''. To disable a specific tag, use ``-'' or ``!'' in front of the tag. Note that more tags will come in future versions. %\LTXtable{\linewidth}{table_debugtags} \chapter{New Features in 5.2.13} This chapter presents the new features that have been added to the current Community version of Bacula that is now released. \subsection{Additions to RunScript variables} You can have access to Director name using \%D in your runscript command. \begin{bVerbatim} RunAfterJob = "/bin/echo Director=%D \end{bVerbatim} \section{New Features in 5.2.1} This chapter presents the new features were added in the Community release version 5.2.1. There are additional features (plugins) available in the Enterprise version that are described in another chapter. A subscription to Bacula Systems is required for the Enterprise version. \subsection{LZO Compression} LZO compression has been to the File daemon. From the user's point of view, it works like the GZIP compression (just replace {\bf compression=GZIP} with {\bf compression=LZO}). For example: \begin{bVerbatim} Include { Options {compression=LZO } File = /home File = /data } \end{bVerbatim} LZO provides a much faster compression and decompression speed but lower compression ratio than GZIP. It is a good option when you backup to disk. For tape, the hardware compression is almost always a better option. LZO is a good alternative for GZIP1 when you don't want to slow down your backup. With a modern CPU it should be able to run almost as fast as: \begin{itemize} \item your client can read data from disk. Unless you have very fast disks like SSD or large/fast RAID array. \item the data transfers between the file daemon and the storage daemon even on a 1Gb/s link. \end{itemize} Note, Bacula uses compression level LZO1X-1. \medskip The code for this feature was contributed by Laurent Papier. \subsection{New Tray Monitor} Since the old integrated Windows tray monitor doesn't work with recent Windows versions, we have written a new Qt Tray Monitor that is available for both Linux and Windows. In addition to all the previous features, this new version allows you to run Backups from the tray monitor menu. \begin{figure}[htbp] \centering \includegraphics[width=0.8\linewidth]{tray-monitor} \label{fig:traymonitor} \caption{New tray monitor} \end{figure} \begin{figure}[htbp] \centering \includegraphics[width=0.8\linewidth]{tray-monitor1} \label{fig:traymonitor1} \caption{Run a Job through the new tray monitor} \end{figure} To be able to run a job from the tray monitor, you need to allow specific commands in the Director monitor console: \begin{bVerbatim} Console { Name = win2003-mon Password = "xxx" CommandACL = status, .clients, .jobs, .pools, .storage, .filesets, .messages, run ClientACL = *all* # you can restrict to a specific host CatalogACL = *all* JobACL = *all* StorageACL = *all* ScheduleACL = *all* PoolACL = *all* FileSetACL = *all* WhereACL = *all* } \end{bVerbatim} \medskip This project was funded by Bacula Systems and is available with Bacula the Enterprise Edition and the Community Edition. \subsection{Purge Migration Job} The new {\bf Purge Migration Job} directive may be added to the Migration Job definition in the Director's configuration file. When it is enabled the Job that was migrated during a migration will be purged at the end of the migration job. For example: \begin{bVerbatim} Job { Name = "migrate-job" Type = Migrate Level = Full Client = localhost-fd FileSet = "Full Set" Messages = Standard Storage = DiskChanger Pool = Default Selection Type = Job Selection Pattern = ".*Save" ... Purge Migration Job = yes } \end{bVerbatim} \medskip This project was submitted by Dunlap Blake; testing and documentation was funded by Bacula Systems. \subsection{Changes in Bvfs (Bacula Virtual FileSystem)} Bat has now a bRestore panel that uses Bvfs to display files and directories. \begin{figure}[htbp] \centering \includegraphics[width=0.8\linewidth]{bat-brestore} \label{fig:batbrestore} \caption{Bat Brestore Panel} \end{figure} the Bvfs module works correctly with BaseJobs, Copy and Migration jobs. \medskip This project was funded by Bacula Systems. \subsubsection*{General notes} \begin{itemize} \item All fields are separated by a tab \item You can specify \texttt{limit=} and \texttt{offset=} to list smoothly records in very big directories \item All operations (except cache creation) are designed to run instantly \item At this time, Bvfs works faster on PostgreSQL than MySQL catalog. If you can contribute new faster SQL queries we will be happy, else don't complain about speed. \item The cache creation is dependent of the number of directories. As Bvfs shares information across jobs, the first creation can be slow \item All fields are separated by a tab \item Due to potential encoding problem, it's advised to always use pathid in queries. \end{itemize} \subsubsection*{Get dependent jobs from a given JobId} Bvfs allows you to query the catalog against any combination of jobs. You can combine all Jobs and all FileSet for a Client in a single session. To get all JobId needed to restore a particular job, you can use the \texttt{.bvfs\_get\_jobids} command. \begin{bVerbatim} .bvfs_get_jobids jobid=num [all] \end{bVerbatim} \begin{bVerbatim} .bvfs_get_jobids jobid=10 1,2,5,10 .bvfs_get_jobids jobid=10 all 1,2,3,5,10 \end{bVerbatim} In this example, a normal restore will need to use JobIds 1,2,5,10 to compute a complete restore of the system. With the \texttt{all} option, the Director will use all defined FileSet for this client. \subsubsection*{Generating Bvfs cache} The \texttt{.bvfs\_update} command computes the directory cache for jobs specified in argument, or for all jobs if unspecified. \begin{bVerbatim} .bvfs_update [jobid=numlist] \end{bVerbatim} Example: \begin{bVerbatim} .bvfs_update jobid=1,2,3 \end{bVerbatim} You can run the cache update process in a RunScript after the catalog backup. \subsubsection*{Get all versions of a specific file} Bvfs allows you to find all versions of a specific file for a given Client with the \texttt{.bvfs\_version} command. To avoid problems with encoding, this function uses only PathId and FilenameId. The jobid argument is mandatory but unused. \begin{bVerbatim} .bvfs_versions client=filedaemon pathid=num filenameid=num jobid=1 PathId FilenameId FileId JobId LStat Md5 VolName Inchanger PathId FilenameId FileId JobId LStat Md5 VolName Inchanger ... \end{bVerbatim} Example: \begin{bVerbatim} .bvfs_versions client=localhost-fd pathid=1 fnid=47 jobid=1 1 47 52 12 gD HRid IGk D Po Po A P BAA I A /uPgWaxMgKZlnMti7LChyA Vol1 1 \end{bVerbatim} \subsubsection*{List directories} Bvfs allows you to list directories in a specific path. \begin{bVerbatim} .bvfs_lsdirs pathid=num path=/apath jobid=numlist limit=num offset=num PathId FilenameId FileId JobId LStat Path PathId FilenameId FileId JobId LStat Path PathId FilenameId FileId JobId LStat Path ... \end{bVerbatim} You need to \texttt{pathid} or \texttt{path}. Using \texttt{path=""} will list ``/'' on Unix and all drives on Windows. If FilenameId is 0, the record listed is a directory. \begin{bVerbatim} .bvfs_lsdirs pathid=4 jobid=1,11,12 4 0 0 0 A A A A A A A A A A A A A A . 5 0 0 0 A A A A A A A A A A A A A A .. 3 0 0 0 A A A A A A A A A A A A A A regress/ \end{bVerbatim} In this example, to list directories present in \texttt{regress/}, you can use \begin{bVerbatim} .bvfs_lsdirs pathid=3 jobid=1,11,12 3 0 0 0 A A A A A A A A A A A A A A . 4 0 0 0 A A A A A A A A A A A A A A .. 2 0 0 0 A A A A A A A A A A A A A A tmp/ \end{bVerbatim} \subsubsection*{List files} Bvfs allows you to list files in a specific path. \begin{bVerbatim} .bvfs_lsfiles pathid=num path=/apath jobid=numlist limit=num offset=num PathId FilenameId FileId JobId LStat Path PathId FilenameId FileId JobId LStat Path PathId FilenameId FileId JobId LStat Path ... \end{bVerbatim} You need to \texttt{pathid} or \texttt{path}. Using \texttt{path=""} will list ``/'' on Unix and all drives on Windows. If FilenameId is 0, the record listed is a directory. \begin{bVerbatim} .bvfs_lsfiles pathid=4 jobid=1,11,12 4 0 0 0 A A A A A A A A A A A A A A . 5 0 0 0 A A A A A A A A A A A A A A .. 1 0 0 0 A A A A A A A A A A A A A A regress/ \end{bVerbatim} In this example, to list files present in \texttt{regress/}, you can use \begin{bVerbatim} .bvfs_lsfiles pathid=1 jobid=1,11,12 1 47 52 12 gD HRid IGk BAA I BMqcPH BMqcPE BMqe+t A titi 1 49 53 12 gD HRid IGk BAA I BMqe/K BMqcPE BMqe+t B toto 1 48 54 12 gD HRie IGk BAA I BMqcPH BMqcPE BMqe+3 A tutu 1 45 55 12 gD HRid IGk BAA I BMqe/K BMqcPE BMqe+t B ficheriro1.txt 1 46 56 12 gD HRie IGk BAA I BMqe/K BMqcPE BMqe+3 D ficheriro2.txt \end{bVerbatim} \subsubsection*{Restore set of files} Bvfs allows you to create a SQL table that contains files that you want to restore. This table can be provided to a restore command with the file option. \begin{bVerbatim} .bvfs_restore fileid=numlist dirid=numlist hardlink=numlist path=b2num OK restore file=?b2num ... \end{bVerbatim} To include a directory (with \texttt{dirid}), Bvfs needs to run a query to select all files. This query could be time consuming. \texttt{hardlink} list is always composed of a series of two numbers (jobid, fileindex). This information can be found in the LinkFI field of the LStat packet. The \texttt{path} argument represents the name of the table that Bvfs will store results. The format of this table is \texttt{b2[0-9]+}. (Should start by b2 and followed by digits). Example: \begin{bVerbatim} .bvfs_restore fileid=1,2,3,4 hardlink=10,15,10,20 jobid=10 path=b20001 OK \end{bVerbatim} \subsubsection*{Cleanup after Restore} To drop the table used by the restore command, you can use the \texttt{.bvfs\_cleanup} command. \begin{bVerbatim} .bvfs_cleanup path=b20001 \end{bVerbatim} \subsubsection*{Clearing the BVFS Cache} To clear the BVFS cache, you can use the \texttt{.bvfs\_clear\_cache} command. \begin{bVerbatim} .bvfs_clear_cache yes OK \end{bVerbatim} \subsection{Changes in the Pruning Algorithm} We rewrote the job pruning algorithm in this version. Previously, in some users reported that the pruning process at the end of jobs was very long. It should not be longer the case. Now, Bacula won't prune automatically a Job if this particular Job is needed to restore data. Example: \begin{bVerbatim} JobId: 1 Level: Full JobId: 2 Level: Incremental JobId: 3 Level: Incremental JobId: 4 Level: Differential .. Other incrementals up to now \end{bVerbatim} In this example, if the Job Retention defined in the Pool or in the Client resource causes that Jobs with Jobid in 1,2,3,4 can be pruned, Bacula will detect that JobId 1 and 4 are essential to restore data at the current state and will prune only JobId 2 and 3. \texttt{Important}, this change affect only the automatic pruning step after a Job and the \texttt{prune jobs} Bconsole command. If a volume expires after the \texttt{VolumeRetention} period, important jobs can be pruned. \subsection{Ability to Verify any specified Job} You now have the ability to tell Bacula which Job should verify instead of automatically verify just the last one. This feature can be used with VolumeToCatalog, DiskToCatalog and Catalog level. To verify a given job, just specify the Job jobid in argument when starting the job. \begin{bVerbatim} *run job=VerifyVolume jobid=1 level=VolumeToCatalog Run Verify job JobName: VerifyVolume Level: VolumeToCatalog Client: 127.0.0.1-fd FileSet: Full Set Pool: Default (From Job resource) Storage: File (From Job resource) Verify Job: VerifyVol.2010-09-08_14.17.17_03 Verify List: /tmp/regress/working/VerifyVol.bsr When: 2010-09-08 14:17:31 Priority: 10 OK to run? (yes/mod/no): \end{bVerbatim} \medskip This project was funded by Bacula Systems and is available with Bacula Enterprise Edition and Community Edition. \subsection{Additions to RunScript variables} You can have access to JobBytes and JobFiles using \%b and \%F in your runscript command. The Client address is now available through \%h. \begin{bVerbatim} RunAfterJob = "/bin/echo Job=%j JobBytes=%b JobFiles=%F ClientAddress=%h" \end{bVerbatim} %\subsection{Changes in drivetype.exe} % %Now the \texttt{drivetype.exe} program allows you to list all local hard %drives. It can help to build dynamic FileSet on Windows. % %\begin{bVerbatim} %File = "\\|\"c:/program files/bacula/bin32/drivetype\" -l -a" %\end{bVerbatim} % \subsection{Additions to the Plugin API} The bfuncs structure has been extended to include a number of new entrypoints. \subsubsection{bfuncs} The bFuncs structure defines the callback entry points within Bacula that the plugin can use register events, get Bacula values, set Bacula values, and send messages to the Job output or debug output. The exact definition as of this writing is: \begin{bVerbatim} typedef struct s_baculaFuncs { uint32_t size; uint32_t version; bRC (*registerBaculaEvents)(bpContext *ctx, ...); bRC (*getBaculaValue)(bpContext *ctx, bVariable var, void *value); bRC (*setBaculaValue)(bpContext *ctx, bVariable var, void *value); bRC (*JobMessage)(bpContext *ctx, const char *file, int line, int type, utime_t mtime, const char *fmt, ...); bRC (*DebugMessage)(bpContext *ctx, const char *file, int line, int level, const char *fmt, ...); void *(*baculaMalloc)(bpContext *ctx, const char *file, int line, size_t size); void (*baculaFree)(bpContext *ctx, const char *file, int line, void *mem); /* New functions follow */ bRC (*AddExclude)(bpContext *ctx, const char *file); bRC (*AddInclude)(bpContext *ctx, const char *file); bRC (*AddIncludeOptions)(bpContext *ctx, const char *opts); bRC (*AddRegex)(bpContext *ctx, const char *item, int type); bRC (*AddWild)(bpContext *ctx, const char *item, int type); bRC (*checkChanges)(bpContext *ctx, struct save_pkt *sp); } bFuncs; \end{bVerbatim} \begin{description} \item [AddExclude] can be called to exclude a file. The file string passed may include wildcards that will be interpreted by the {\bf fnmatch} subroutine. This function can be called multiple times, and each time the file specified will be added to the list of files to be excluded. Note, this function only permits adding excludes of specific file or directory names, or files matched by the rather simple fnmatch mechanism. See below for information on doing wild-card and regex excludes. \item [NewPreInclude] can be called to create a new Include block. This block will be added after the current defined Include block. This function can be called multiple times, but each time, it will create a new Include section (not normally needed). This function should be called only if you want to add an entirely new Include block. \item [NewInclude] can be called to create a new Include block. This block will be added before any user defined Include blocks. This function can be called multiple times, but each time, it will create a new Include section (not normally needed). This function should be called only if you want to add an entirely new Include block. \item [AddInclude] can be called to add new files/directories to be included. They are added to the current Include block. If NewInclude has not been included, the current Include block is the last one that the user created. This function should be used only if you want to add totally new files/directories to be included in the backup. \item [NewOptions] adds a new Options block to the current Include in front of any other Options blocks. This permits the plugin to add exclude directives (wild-cards and regexes) in front of the user Options, and thus prevent certain files from being backed up. This can be useful if the plugin backs up files, and they should not be also backed up by the main Bacula code. This function may be called multiple times, and each time, it creates a new prepended Options block. Note: normally you want to call this entry point prior to calling AddOptions, AddRegex, or AddWild. \item [AddOptions] allows the plugin it set options in the current Options block, which is normally created with the NewOptions call just prior to adding Include Options. The permitted options are passed as a character string, where each character has a specific meaning as defined below: \begin{description} \item [a] always replace files (default). \item [e] exclude rather than include. \item [h] no recursion into subdirectories. \item [H] do not handle hard links. \item [i] ignore case in wildcard and regex matches. \item [M] compute an MD5 sum. \item [p] use a portable data format on Windows (not recommended). \item [R] backup resource forks and Findr Info. \item [r] read from a fifo \item [S1] compute an SHA1 sum. \item [S2] compute an SHA256 sum. \item [S3] comput an SHA512 sum. \item [s] handle sparse files. \item [m] use st\_mtime only for file differences. \item [k] restore the st\_atime after accessing a file. \item [A] enable ACL backup. \item [Vxxx:] specify verify options. Must terminate with : \item [Cxxx:] specify accurate options. Must terminate with : \item [Jxxx:] specify base job Options. Must terminate with : \item [Pnnn:] specify integer nnn paths to strip. Must terminate with : \item [w] if newer \item [Zn] specify gzip compression level n. \item [K] do not use st\_atime in backup decision. \item [c] check if file changed during backup. \item [N] honor no dump flag. \item [X] enable backup of extended attributes. \end{description} \item [AddRegex] adds a regex expression to the current Options block. The following options are permitted: \begin{description} \item [ ] (a blank) regex applies to whole path and filename. \item [F] regex applies only to the filename (directory or path stripped). \item [D] regex applies only to the directory (path) part of the name. \end{description} \item [AddWild] adds a wildcard expression to the current Options block. The following options are permitted: \begin{description} \item [ ] (a blank) regex applies to whole path and filename. \item [F] regex applies only to the filename (directory or path stripped). \item [D] regex applies only to the directory (path) part of the name. \end{description} \item [checkChanges] call the \texttt{check\_changes()} function in Bacula code that can use Accurate code to compare the file information in argument with the previous file information. The \texttt{delta\_seq} attribute of the \texttt{save\_pkt} will be updated, and the call will return \texttt{bRC\_Seen} if the core code wouldn't decide to backup it. \end{description} \subsubsection{Bacula events} The list of events has been extended to include: \begin{bVerbatim} typedef enum { bEventJobStart = 1, bEventJobEnd = 2, bEventStartBackupJob = 3, bEventEndBackupJob = 4, bEventStartRestoreJob = 5, bEventEndRestoreJob = 6, bEventStartVerifyJob = 7, bEventEndVerifyJob = 8, bEventBackupCommand = 9, bEventRestoreCommand = 10, bEventLevel = 11, bEventSince = 12, /* New events */ bEventCancelCommand = 13, bEventVssBackupAddComponents = 14, bEventVssRestoreLoadComponentMetadata = 15, bEventVssRestoreSetComponentsSelected = 16, bEventRestoreObject = 17, bEventEndFileSet = 18, bEventPluginCommand = 19, bEventVssBeforeCloseRestore = 20, bEventVssPrepareSnapshot = 21 } bEventType; \end{bVerbatim} \begin{description} \item [bEventCancelCommand] is called whenever the currently running Job is canceled */ \item [bEventVssBackupAddComponents] \item [bEventVssPrepareSnapshot] is called before creating VSS snapshots, it provides a char[27] table where the plugin can add Windows drives that will be used during the Job. You need to add them without duplicates, and you can use in \texttt{fd\_common.h} \texttt{add\_drive()} and \texttt{copy\_drives()} for this purpose. \end{description} \subsection{ACL enhancements} The following enhancements are made to the Bacula Filed with regards to Access Control Lists (ACLs) \begin{itemize} \item Added support for AIX 5.3 and later new aclx\_get interface which supports POSIX and NFSv4 ACLs. \item Added support for new acl types on FreeBSD 8.1 and later which supports POSIX and NFSv4 ACLs. \item Some generic cleanups for internal ACL handling. \item Fix for acl storage on OSX \item Cleanup of configure checks for ACL detection, now configure only tests for a certain interface type based on the operating system this should give less false positives on detection. Also when ACLs are detected no other acl checks are performed anymore. \end{itemize} \medskip This project was funded by Planets Communications B.V. and ELM Consultancy B.V. and is available with Bacula Enterprise Edition and Community Edition. \subsection{XATTR enhancements} The following enhancements are made to the Bacula Filed with regards to Extended Attributes (XATTRs) \begin{itemize} \item Added support for IRIX extended attributes using the attr\_get interface. \item Added support for Tru64 (OSF1) extended attributes using the getproplist interface. \item Added support for AIX extended attributes available in AIX 6.x and higher using the listea/getea/setea interface. \item Added some debugging to generic xattr code so it easier to debug. \item Cleanup of configure checks for XATTR detection, now configure only tests for a certain interface type based on the operating system this should give less false positives on detection. Also when xattrs are detected no other xattr checks are performed anymore. \end{itemize} \medskip This project was funded by Planets Communications B.V. and ELM Consultancy B.V. and is available with Bacula Enterprise Edition and Community Edition. \subsection{Class Based Database Backend Drivers} The main Bacula Director code is independent of the SQL backend in version 5.2.0 and greater. This means that the Bacula Director can be packaged by itself, then each of the different SQL backends supported can be packaged separately. It is possible to build all the DB backends at the same time by including multiple database options at the same time. ./configure can be run with multiple database configure options. \begin{bVerbatim} --with-mysql --with-postgresql \end{bVerbatim} Order of testing for databases is: \begin{itemize} \item postgresql \item mysql \end{itemize} Each configured backend generates a file named: \verb+libbaccats--.so+ A dummy catalog library is created named libbaccats-version.so At configure time the first detected backend is used as the so called default backend and at install time the dummy \verb+libbaccats-.so+ is replaced with the default backend type. If you configure all three backends you get three backend libraries and the postgresql gets installed as the default. When you want to switch to another database, first save any old catalog you may have then you can copy one of the three backend libraries over the \verb+libbaccats-.so+ e.g. An actual command, depending on your Bacula version might be: \begin{bVerbatim} cp libbaccats-postgresql-5.2.2.so libbaccats-5.2.2.so \end{bVerbatim} where the \verb+5.2.2+ must be replaced by the Bacula release version number. Then you must update the default backend in the following files: \begin{bVerbatim} create_bacula_database drop_bacula_database drop_bacula_tables grant_bacula_privileges make_bacula_tables make_catalog_backup update_bacula_tables \end{bVerbatim} And re-run all the above scripts. Please note, this means you will have a new empty database and if you had a previous one it will be lost. All current database backend drivers for catalog information are rewritten to use a set of multi inherited C++ classes which abstract the specific database specific internals and make sure we have a more stable generic interface with the rest of SQL code. From now on there is a strict boundary between the SQL code and the low-level database functions. This new interface should also make it easier to add a new backend for a currently unsupported database. An extra bonus of the new code is that you can configure multiple backends in the configure and build all backends in one compile session and select the correct database backend at install time. This should make it a lot easier for packages maintainers. \medskip We also added cursor support for PostgreSQL backend, this improves memory usage for large installation. \medskip This project was implemented by Planets Communications B.V. and ELM Consultancy B.V. and Bacula Systems and is available with both the Bacula Enterprise Edition and the Community Edition. \subsection{Hash List Enhancements} The htable hash table class has been extended with extra hash functions for handling next to char pointer hashes also 32 bits and 64 bits hash keys. Also the hash table initialization routines have been enhanced with support for passing a hint as to the number of initial pages to use for the size of the hash table. Until now the hash table always used a fixed value of 10 Mb. The private hash functions of the mountpoint entry cache have been rewritten to use the new htable class with a small memory footprint. \medskip This project was funded by Planets Communications B.V. and ELM Consultancy B.V. and Bacula Systems and is available with Bacula Enterprise Edition and Community Edition. %% %% %%% ===================================================================== %% %% \section{Release Version 5.0.3} There are no new features in version 5.0.2. This version simply fixes a number of bugs found in version 5.0.1 during the ongoing development process. \section{Release Version 5.0.2} There are no new features in version 5.0.2. This version simply fixes a number of bugs found in version 5.0.1 during the ongoing development process. %% %% \section{New Features in 5.0.1} This chapter presents the new features that are in the released Bacula version 5.0.1. This version mainly fixes a number of bugs found in version 5.0.0 during the ongoing development process. \subsection{Truncate Volume after Purge} \label{sec:actiononpurge} The Pool directive \textbf{ActionOnPurge=Truncate} instructs Bacula to truncate the volume when it is purged with the new command \texttt{purge volume action}. It is useful to prevent disk based volumes from consuming too much space. \begin{bVerbatim} Pool { Name = Default Action On Purge = Truncate ... } \end{bVerbatim} As usual you can also set this property with the \texttt{update volume} command \begin{bVerbatim} *update volume=xxx ActionOnPurge=Truncate *update volume=xxx actiononpurge=None \end{bVerbatim} To ask Bacula to truncate your \texttt{Purged} volumes, you need to use the following command in interactive mode or in a RunScript as shown after: \begin{bVerbatim} *purge volume action=truncate storage=File allpools # or by default, action=all *purge volume action storage=File pool=Default \end{bVerbatim} This is possible to specify the volume name, the media type, the pool, the storage, etc\dots (see \texttt{help purge}) Be sure that your storage device is idle when you decide to run this command. \begin{bVerbatim} Job { Name = CatalogBackup ... RunScript { RunsWhen=After RunsOnClient=No Console = "purge volume action=all allpools storage=File" } } \end{bVerbatim} \textbf{Important note}: This feature doesn't work as expected in version 5.0.0. Please do not use it before version 5.0.1. \subsection{Allow Higher Duplicates} This directive did not work correctly and has been depreciated (disabled) in version 5.0.1. Please remove it from your bacula-dir.conf file as it will be removed in a future release. \subsection{Cancel Lower Level Duplicates} This directive was added in Bacula version 5.0.1. It compares the level of a new backup job to old jobs of the same name, if any, and will kill the job which has a lower level than the other one. If the levels are the same (i.e. both are Full backups), then nothing is done and the other Cancel XXX Duplicate directives will be examined. \section{New Features in 5.0.0} \subsection{Maximum Concurrent Jobs for Devices} \label{sec:maximumconcurrentjobdevice} {\bf Maximum Concurrent Jobs} is a new Device directive in the Storage Daemon configuration permits setting the maximum number of Jobs that can run concurrently on a specified Device. Using this directive, it is possible to have different Jobs using multiple drives, because when the Maximum Concurrent Jobs limit is reached, the Storage Daemon will start new Jobs on any other available compatible drive. This facilitates writing to multiple drives with multiple Jobs that all use the same Pool. This project was funded by Bacula Systems. \subsection{Restore from Multiple Storage Daemons} \index[general]{Restore} Previously, you were able to restore from multiple devices in a single Storage Daemon. Now, Bacula is able to restore from multiple Storage Daemons. For example, if your full backup runs on a Storage Daemon with an autochanger, and your incremental jobs use another Storage Daemon with lots of disks, Bacula will switch automatically from one Storage Daemon to an other within the same Restore job. You must upgrade your File Daemon to version 3.1.3 or greater to use this feature. This project was funded by Bacula Systems with the help of Equiinet. \subsection{File Deduplication using Base Jobs} A base job is sort of like a Full save except that you will want the FileSet to contain only files that are unlikely to change in the future (i.e. a snapshot of most of your system after installing it). After the base job has been run, when you are doing a Full save, you specify one or more Base jobs to be used. All files that have been backed up in the Base job/jobs but not modified will then be excluded from the backup. During a restore, the Base jobs will be automatically pulled in where necessary. This is something none of the competition does, as far as we know (except perhaps BackupPC, which is a Perl program that saves to disk only). It is big win for the user, it makes Bacula stand out as offering a unique optimization that immediately saves time and money. Basically, imagine that you have 100 nearly identical Windows or Linux machine containing the OS and user files. Now for the OS part, a Base job will be backed up once, and rather than making 100 copies of the OS, there will be only one. If one or more of the systems have some files updated, no problem, they will be automatically restored. See the \bilink{Base Job Chapter}{blb:basejobs} for more information. This project was funded by Bacula Systems. \subsection{AllowCompression = \lt{}yes\vb{}no\gt{}} \index[dir]{AllowCompression} This new directive may be added to Storage resource within the Director's configuration to allow users to selectively disable the client compression for any job which writes to this storage resource. For example: \begin{bVerbatim} Storage { Name = UltriumTape Address = ultrium-tape Password = storage_password # Password for Storage Daemon Device = Ultrium Media Type = LTO 3 AllowCompression = No # Tape drive has hardware compression } \end{bVerbatim} The above example would cause any jobs running with the UltriumTape storage resource to run without compression from the client file daemons. This effectively overrides any compression settings defined at the FileSet level. This feature is probably most useful if you have a tape drive which supports hardware compression. By setting the \texttt{AllowCompression = No} directive for your tape drive storage resource, you can avoid additional load on the file daemon and possibly speed up tape backups. This project was funded by Collaborative Fusion, Inc. \subsection{Accurate Fileset Options} \label{sec:accuratefileset} In previous versions, the accurate code used the file creation and modification times to determine if a file was modified or not. Now you can specify which attributes to use (time, size, checksum, permission, owner, group, \dots), similar to the Verify options. \begin{bVerbatim} FileSet { Name = Full Include = { Options { Accurate = mcs Verify = pin5 } File = / } } \end{bVerbatim} \begin{description} \item {\bf i} compare the inodes \item {\bf p} compare the permission bits \item {\bf n} compare the number of links \item {\bf u} compare the user id \item {\bf g} compare the group id \item {\bf s} compare the size \item {\bf a} compare the access time \item {\bf m} compare the modification time (st\_mtime) \item {\bf c} compare the change time (st\_ctime) \item {\bf d} report file size decreases \item {\bf 5} compare the MD5 signature \item {\bf 1} compare the SHA1 signature \end{description} \textbf{Important note:} If you decide to use checksum in Accurate jobs, the File Daemon will have to read all files even if they normally would not be saved. This increases the I/O load, but also the accuracy of the deduplication. By default, Bacula will check modification/creation time and size. This project was funded by Bacula Systems. \subsection{Tab-completion for Bconsole} \label{sec:tabcompletion} If you build \texttt{bconsole} with readline support, you will be able to use the new auto-completion mode. This mode supports all commands, gives help inside command, and lists resources when required. It works also in the restore mode. To use this feature, you should have readline development package loaded on your system, and use the following option in configure. \begin{bVerbatim} ./configure --with-readline=/usr/include/readline --disable-conio ... \end{bVerbatim} The new bconsole won't be able to tab-complete with older directors. This project was funded by Bacula Systems. \subsection{Pool File and Job Retention} \label{sec:poolfilejobretention} We added two new Pool directives, \texttt{FileRetention} and \texttt{JobRetention}, that take precedence over Client directives of the same name. It allows you to control the Catalog pruning algorithm Pool by Pool. For example, you can decide to increase Retention times for Archive or OffSite Pool. It seems obvious to us, but apparently not to some users, that given the definition above that the Pool File and Job Retention periods is a global override for the normal Client based pruning, which means that when the Job is pruned, the pruning will apply globally to that particular Job. Currently, there is a bug in the implementation that causes any Pool retention periods specified to apply to {\bf all} Pools for that particular Client. Thus we suggest that you avoid using these two directives until this implementation problem is corrected. \subsection{Read-only File Daemon using capabilities} \label{sec:fdreadonly} This feature implements support of keeping \textbf{ReadAll} capabilities after UID/GID switch, this allows FD to keep root read but drop write permission. It introduces new \texttt{bacula-fd} option (\texttt{-k}) specifying that \textbf{ReadAll} capabilities should be kept after UID/GID switch. \begin{bVerbatim} root@localhost:~# bacula-fd -k -u nobody -g nobody \end{bVerbatim} The code for this feature was contributed by our friends at AltLinux. \subsection{Bvfs API} \label{sec:bvfs} To help developers of restore GUI interfaces, we have added new \textsl{dot commands} that permit browsing the catalog in a very simple way. \begin{itemize} \item \texttt{.bvfs\_update [jobid=x,y,z]} This command is required to update the Bvfs cache in the catalog. You need to run it before any access to the Bvfs layer. \item \texttt{.bvfs\_lsdirs jobid=x,y,z path=/path | pathid=101} This command will list all directories in the specified \texttt{path} or \texttt{pathid}. Using \texttt{pathid} avoids problems with character encoding of path/filenames. \item \texttt{.bvfs\_lsfiles jobid=x,y,z path=/path | pathid=101} This command will list all files in the specified \texttt{path} or \texttt{pathid}. Using \texttt{pathid} avoids problems with character encoding. \end{itemize} You can use \texttt{limit=xxx} and \texttt{offset=yyy} to limit the amount of data that will be displayed. \begin{bVerbatim} * .bvfs_update jobid=1,2 * .bvfs_update * .bvfs_lsdir path=/ jobid=1,2 \end{bVerbatim} This project was funded by Bacula Systems. \subsection{Testing your Tape Drive} \label{sec:btapespeed} To determine the best configuration of your tape drive, you can run the new \texttt{speed} command available in the \texttt{btape} program. This command can have the following arguments: \begin{itemize} \item[\texttt{file\_size=n}] Specify the Maximum File Size for this test (between 1 and 5GB). This counter is in GB. \item[\texttt{nb\_file=n}] Specify the number of file to be written. The amount of data should be greater than your memory ($file\_size*nb\_file$). \item[\texttt{skip\_zero}] This flag permits to skip tests with constant data. \item[\texttt{skip\_random}] This flag permits to skip tests with random data. \item[\texttt{skip\_raw}] This flag permits to skip tests with raw access. \item[\texttt{skip\_block}] This flag permits to skip tests with Bacula block access. \end{itemize} \begin{bVerbatim} *speed file_size=3 skip_raw btape.c:1078 Test with zero data and bacula block structure. btape.c:956 Begin writing 3 files of 3.221 GB with blocks of 129024 bytes. ++++++++++++++++++++++++++++++++++++++++++ btape.c:604 Wrote 1 EOF to "Drive-0" (/dev/nst0) btape.c:406 Volume bytes=3.221 GB. Write rate = 44.128 MB/s ... btape.c:383 Total Volume bytes=9.664 GB. Total Write rate = 43.531 MB/s btape.c:1090 Test with random data, should give the minimum throughput. btape.c:956 Begin writing 3 files of 3.221 GB with blocks of 129024 bytes. +++++++++++++++++++++++++++++++++++++++++++ btape.c:604 Wrote 1 EOF to "Drive-0" (/dev/nst0) btape.c:406 Volume bytes=3.221 GB. Write rate = 7.271 MB/s +++++++++++++++++++++++++++++++++++++++++++ ... btape.c:383 Total Volume bytes=9.664 GB. Total Write rate = 7.365 MB/s \end{bVerbatim} When using compression, the random test will give your the minimum throughput of your drive . The test using constant string will give you the maximum speed of your hardware chain. (CPU, memory, SCSI card, cable, drive, tape). You can change the block size in the Storage Daemon configuration file. \subsection{New {\bf Block Checksum} Device Directive} You may now turn off the Block Checksum (CRC32) code that Bacula uses when writing blocks to a Volume. This is done by adding: \begin{bVerbatim} Block Checksum = no \end{bVerbatim} doing so can reduce the Storage daemon CPU usage slightly. It will also permit Bacula to read a Volume that has corrupted data. The default is {\bf yes} -- i.e. the checksum is computed on write and checked on read. We do not recommend to turn this off particularly on older tape drives or for disk Volumes where doing so may allow corrupted data to go undetected. \subsection{New Bat Features} Those new features were funded by Bacula Systems. \subsubsection{Media List View} By clicking on ``Media'', you can see the list of all your volumes. You will be able to filter by Pool, Media Type, Location,\dots And sort the result directly in the table. The old ``Media'' view is now known as ``Pool''. \begin{figure}[htbp] \centering \includegraphics[width=0.8\linewidth]{bat-mediaview} \label{fig:mediaview} \end{figure} \subsubsection{Media Information View} By double-clicking on a volume (on the Media list, in the Autochanger content or in the Job information panel), you can access a detailed overview of your Volume. (cf figure \vref{fig:mediainfo}.) \begin{figure}[htbp] \centering \includegraphics[width=0.8\linewidth]{bat11} \caption{Media information} \label{fig:mediainfo} \end{figure} \subsubsection{Job Information View} By double-clicking on a Job record (on the Job run list or in the Media information panel), you can access a detailed overview of your Job. (cf figure \vref{fig:jobinfo}.) \begin{figure}[htbp] \centering \includegraphics[width=0.8\linewidth]{bat12} \caption{Job information} \label{fig:jobinfo} \end{figure} \subsubsection{Autochanger Content View} By double-clicking on a Storage record (on the Storage list panel), you can access a detailed overview of your Autochanger. (cf figure \vref{fig:jobinfo}.) \begin{figure}[htbp] \centering \includegraphics[width=0.8\linewidth]{bat13} \caption{Autochanger content} \label{fig:achcontent} \end{figure} To use this feature, you need to use the latest mtx-changer script version. (With new \texttt{listall} and \texttt{transfer} commands) \subsection{Bat on Windows} We have ported {\bf bat} to Windows and it is now installed by default when the installer is run. It works quite well on Win32, but has not had a lot of testing there, so your feedback would be welcome. Unfortunately, even though it is installed by default, it does not yet work on 64 bit Windows operating systems. \subsection{New Win32 Installer} The Win32 installer has been modified in several very important ways. \begin{itemize} \item You must deinstall any current version of the Win32 File daemon before upgrading to the new one. If you forget to do so, the new installation will fail. To correct this failure, you must manually shutdown and deinstall the old File daemon. \item All files (other than menu links) are installed in {\bf c:/Program Files/Bacula}. \item The installer no longer sets this file to require administrator privileges by default. If you want to do so, please do it manually using the {\bf cacls} program. For example: \begin{bVerbatim} cacls "C:\Program Files\Bacula" /T /G SYSTEM:F Administrators:F \end{bVerbatim} \item The server daemons (Director and Storage daemon) are no longer included in the Windows installer. If you want the Windows servers, you will either need to build them yourself (note they have not been ported to 64 bits), or you can contact Bacula Systems about this. \end{itemize} \subsection{Win64 Installer} We have corrected a number of problems that required manual editing of the conf files. In most cases, it should now install and work. {\bf bat} is by default installed in {\bf c:/Program Files/Bacula/bin32} rather than {\bf c:/Program Files/Bacula} as is the case with the 32 bit Windows installer. \subsection{Linux Bare Metal Recovery USB Key} We have made a number of significant improvements in the Bare Metal Recovery USB key. Please see the README files it the {\bf rescue} release for more details. We are working on an equivalent USB key for Windows bare metal recovery, but it will take some time to develop it (best estimate 3Q2010 or 4Q2010) \subsection{bconsole Timeout Option} You can now use the -u option of {\bf bconsole} to set a timeout in seconds for commands. This is useful with GUI programs that use {\bf bconsole} to interface to the Director. \subsection{Important Changes} \label{sec:importantchanges} \begin{itemize} \item You are now allowed to Migrate, Copy, and Virtual Full to read and write to the same Pool. The Storage daemon ensures that you do not read and write to the same Volume. \item The \texttt{Device Poll Interval} is now 5 minutes. (previously did not poll by default). \item Virtually all the features of {\bf mtx-changer} have now been parametrized, which allows you to configure mtx-changer without changing it. There is a new configuration file {\bf mtx-changer.conf} that contains variables that you can set to configure mtx-changer. This configuration file will not be overwritten during upgrades. We encourage you to submit any changes that are made to mtx-changer and to parametrize it all in mtx-changer.conf so that all configuration will be done by changing only mtx-changer.conf. \item The new \texttt{mtx-changer} script has two new options, \texttt{listall} and \texttt{transfer}. Please configure them as appropriate in mtx-changer.conf. \item To enhance security of the \texttt{BackupCatalog} job, we provide a new script (\texttt{make\_catalog\_backup.pl}) that does not expose your catalog password. If you want to use the new script, you will need to manually change the \texttt{BackupCatalog} Job definition. \item The \texttt{bconsole} \texttt{help} command now accepts an argument, which if provided produces information on that command (ex: \texttt{help run}). \end{itemize} \subsubsection*{Truncate volume after purge} Note that the Truncate Volume after purge feature doesn't work as expected in 5.0.0 version. Please, don't use it before version 5.0.1. \subsubsection{Custom Catalog queries} If you wish to add specialized commands that list the contents of the catalog, you can do so by adding them to the \texttt{query.sql} file. This \texttt{query.sql} file is now empty by default. The file \texttt{examples/sample-query.sql} has an a number of sample commands you might find useful. \subsubsection{Deprecated parts} The following items have been \textbf{deprecated} for a long time, and are now removed from the code. \begin{itemize} \item Gnome console \item Support for SQLite. Note the source code still exists, but the project does not maintain it. Please do not use it. \end{itemize} \subsection{Misc Changes} \label{sec:miscchanges} \begin{itemize} \item Updated Nagios check\_bacula \item Updated man files \item Added OSX package generation script in platforms/darwin \item Added Spanish and Ukrainian Bacula translations \item Enable/disable command shows only Jobs that can change \item Added \texttt{show disabled} command to show disabled Jobs \item Many ACL improvements \item Added Level to FD status Job output \item Begin Ingres DB driver (not yet working) \item Split RedHat spec files into bacula, bat, mtx, and docs \item Reorganized the manuals (fewer separate manuals) \item Added lock/unlock order protection in lock manager \item Allow 64 bit sizes for a number of variables \item Fixed several deadlocks or potential race conditions in the SD \end{itemize} \chapter{Released Version 3.0.3 and 3.0.3a} There are no new features in version 3.0.3. This version simply fixes a number of bugs found in version 3.0.2 during the ongoing development process. \section{New Features in Released Version 3.0.2} This chapter presents the new features added to the Released Bacula Version 3.0.2. \subsection{Full Restore from a Given JobId} \index[general]{Restore menu} This feature allows selecting a single JobId and having Bacula automatically select all the other jobs that comprise a full backup up to and including the selected date (through JobId). Assume we start with the following jobs: \begin{bVerbatim} +-------+--------------+---------------------+-------+----------+------------+ | jobid | client | starttime | level | jobfiles | jobbytes | +-------+--------------+---------------------+-------+----------+------------ | 6 | localhost-fd | 2009-07-15 11:45:49 | I | 2 | 0 | | 5 | localhost-fd | 2009-07-15 11:45:45 | I | 15 | 44143 | | 3 | localhost-fd | 2009-07-15 11:45:38 | I | 1 | 10 | | 1 | localhost-fd | 2009-07-15 11:45:30 | F | 1527 | 44143073 | +-------+--------------+---------------------+-------+----------+------------+ \end{bVerbatim} Below is an example of this new feature (which is number 12 in the menu). \begin{bVerbatim} * restore To select the JobIds, you have the following choices: 1: List last 20 Jobs run 2: List Jobs where a given File is saved ... 12: Select full restore to a specified Job date 13: Cancel Select item: (1-13): 12 Enter JobId to get the state to restore: 5 Selecting jobs to build the Full state at 2009-07-15 11:45:45 You have selected the following JobIds: 1,3,5 Building directory tree for JobId(s) 1,3,5 ... +++++++++++++++++++ 1,444 files inserted into the tree. \end{bVerbatim} This project was funded by Bacula Systems. \subsection{Source Address} \index[general]{Source Address} A feature has been added which allows the administrator to specify the address from which the Director and File daemons will establish connections. This may be used to simplify system configuration overhead when working in complex networks utilizing multi-homing and policy-routing. To accomplish this, two new configuration directives have been implemented: \begin{bVerbatim} FileDaemon { FDSourceAddress=10.0.1.20 # Always initiate connections from this address } Director { DirSourceAddress=10.0.1.10 # Always initiate connections from this address } \end{bVerbatim} Simply adding specific host routes on the OS would have an undesirable side-effect: any application trying to contact the destination host would be forced to use the more specific route possibly diverting management traffic onto a backup VLAN. Instead of adding host routes for each client connected to a multi-homed backup server (for example where there are management and backup VLANs), one can use the new directives to specify a specific source address at the application level. Additionally, this allows the simplification and abstraction of firewall rules when dealing with a Hot-Standby director or storage daemon configuration. The Hot-standby pair may share a CARP address, which connections must be sourced from, while system services listen and act from the unique interface addresses. This project was funded by Collaborative Fusion, Inc. \subsection{Show volume availability when doing restore} When doing a restore the selection dialog ends by displaying this screen: \begin{bVerbatim} The job will require the following Volume(s) Storage(s) SD Device(s) =========================================================================== *000741L3 LTO-4 LTO3 *000866L3 LTO-4 LTO3 *000765L3 LTO-4 LTO3 *000764L3 LTO-4 LTO3 *000756L3 LTO-4 LTO3 *001759L3 LTO-4 LTO3 *001763L3 LTO-4 LTO3 001762L3 LTO-4 LTO3 001767L3 LTO-4 LTO3 Volumes marked with ``*'' are online (in the autochanger). \end{bVerbatim} This should help speed up large restores by minimizing the time spent waiting for the operator to discover that he must change tapes in the library. This project was funded by Bacula Systems. \subsection{Accurate estimate command} The \texttt{estimate} command can now use the accurate code to detect changes and give a better estimation. You can set the accurate behavior on the command line by using \texttt{accurate=yes\vb{}no} or use the Job setting as default value. \begin{bVerbatim} * estimate listing accurate=yes level=incremental job=BackupJob \end{bVerbatim} This project was funded by Bacula Systems. \section{New Features in 3.0.0} \label{NewFeaturesChapter} \index[general]{New Features} This chapter presents the new features added to the development 2.5.x versions to be released as Bacula version 3.0.0 sometime in April 2009. \subsection{Accurate Backup} \index[general]{Accurate Backup} As with most other backup programs, by default Bacula decides what files to backup for Incremental and Differential backup by comparing the change (st\_ctime) and modification (st\_mtime) times of the file to the time the last backup completed. If one of those two times is later than the last backup time, then the file will be backed up. This does not, however, permit tracking what files have been deleted and will miss any file with an old time that may have been restored to or moved onto the client filesystem. \subsubsection{Accurate = \lt{}yes\vb{}no\gt{}} If the {\bf Accurate = \lt{}yes\vb{}no\gt{}} directive is enabled (default no) in the Job resource, the job will be run as an Accurate Job. For a {\bf Full} backup, there is no difference, but for {\bf Differential} and {\bf Incremental} backups, the Director will send a list of all previous files backed up, and the File daemon will use that list to determine if any new files have been added or or moved and if any files have been deleted. This allows Bacula to make an accurate backup of your system to that point in time so that if you do a restore, it will restore your system exactly. One note of caution about using Accurate backup is that it requires more resources (CPU and memory) on both the Director and the Client machines to create the list of previous files backed up, to send that list to the File daemon, for the File daemon to keep the list (possibly very big) in memory, and for the File daemon to do comparisons between every file in the FileSet and the list. In particular, if your client has lots of files (more than a few million), you will need lots of memory on the client machine. Accurate must not be enabled when backing up with a plugin that is not specially designed to work with Accurate. If you enable it, your restores will probably not work correctly. This project was funded by Bacula Systems. \subsection{Copy Jobs} \index[general]{Copy Jobs} A new {\bf Copy} job type 'C' has been implemented. It is similar to the existing Migration feature with the exception that the Job that is copied is left unchanged. This essentially creates two identical copies of the same backup. However, the copy is treated as a copy rather than a backup job, and hence is not directly available for restore. The {\bf restore} command lists copy jobs and allows selection of copies by using \texttt{jobid=} option. If the keyword {\bf copies} is present on the command line, Bacula will display the list of all copies for selected jobs. \begin{bVerbatim} * restore copies [...] These JobIds have copies as follows: +-------+------------------------------------+-----------+------------------+ | JobId | Job | CopyJobId | MediaType | +-------+------------------------------------+-----------+------------------+ | 2 | CopyJobSave.2009-02-17_16.31.00.11 | 7 | DiskChangerMedia | +-------+------------------------------------+-----------+------------------+ +-------+-------+----------+----------+---------------------+------------------+ | JobId | Level | JobFiles | JobBytes | StartTime | VolumeName | +-------+-------+----------+----------+---------------------+------------------+ | 19 | F | 6274 | 76565018 | 2009-02-17 16:30:45 | ChangerVolume002 | | 2 | I | 1 | 5 | 2009-02-17 16:30:51 | FileVolume001 | +-------+-------+----------+----------+---------------------+------------------+ You have selected the following JobIds: 19,2 Building directory tree for JobId(s) 19,2 ... ++++++++++++++++++++++++++++++++++++++++++++ 5,611 files inserted into the tree. ... \end{bVerbatim} The Copy Job runs without using the File daemon by copying the data from the old backup Volume to a different Volume in a different Pool. See the Migration documentation for additional details. For copy Jobs there is a new selection directive named {\bf PoolUncopiedJobs} which selects all Jobs that were not already copied to another Pool. As with Migration, the Client, Volume, Job, or SQL query, are other possible ways of selecting the Jobs to be copied. Selection types like SmallestVolume, OldestVolume, PoolOccupancy and PoolTime also work, but are probably more suited for Migration Jobs. If Bacula finds a Copy of a job record that is purged (deleted) from the catalog, it will promote the Copy to a \textsl{real} backup job and will make it available for automatic restore. If more than one Copy is available, it will promote the copy with the smallest JobId. A nice solution which can be built with the new Copy feature is often called disk-to-disk-to-tape backup (DTDTT). A sample config could look something like the one below: \begin{bVerbatim} Pool { Name = FullBackupsVirtualPool Pool Type = Backup Purge Oldest Volume = Yes Storage = vtl NextPool = FullBackupsTapePool } Pool { Name = FullBackupsTapePool Pool Type = Backup Recycle = Yes AutoPrune = Yes Volume Retention = 365 days Storage = superloader } # # Fake fileset for copy jobs # Fileset { Name = None Include { Options { signature = MD5 } } } # # Fake client for copy jobs # Client { Name = None Address = localhost Password = "NoNe" Catalog = MyCatalog } # # Default template for a CopyDiskToTape Job # JobDefs { Name = CopyDiskToTape Type = Copy Messages = StandardCopy Client = None FileSet = None Selection Type = PoolUncopiedJobs Maximum Concurrent Jobs = 10 SpoolData = No Allow Duplicate Jobs = Yes Cancel Queued Duplicates = No Cancel Running Duplicates = No Priority = 13 } Schedule { Name = DaySchedule7:00 Run = Level=Full daily at 7:00 } Job { Name = CopyDiskToTapeFullBackups Enabled = Yes Schedule = DaySchedule7:00 Pool = FullBackupsVirtualPool JobDefs = CopyDiskToTape } \end{bVerbatim} The example above had 2 pool which are copied using the PoolUncopiedJobs selection criteria. Normal Full backups go to the Virtual pool and are copied to the Tape pool the next morning. The command \texttt{list copies [jobid=x,y,z]} lists copies for a given \textbf{jobid}. \begin{bVerbatim} *list copies +-------+------------------------------------+-----------+------------------+ | JobId | Job | CopyJobId | MediaType | +-------+------------------------------------+-----------+------------------+ | 9 | CopyJobSave.2008-12-20_22.26.49.05 | 11 | DiskChangerMedia | +-------+------------------------------------+-----------+------------------+ \end{bVerbatim} \subsection{ACL Updates} \index[general]{ACL Updates} The whole ACL code had been overhauled and in this version each platforms has different streams for each type of acl available on such an platform. As ACLs between platforms tend to be not that portable (most implement POSIX acls but some use an other draft or a completely different format) we currently only allow certain platform specific ACL streams to be decoded and restored on the same platform that they were created on. The old code allowed to restore ACL cross platform but the comments already mention that not being to wise. For backward compatibility the new code will accept the two old ACL streams and handle those with the platform specific handler. But for all new backups it will save the ACLs using the new streams. Currently the following platforms support ACLs: \begin{itemize} \item {\bf AIX} \item {\bf Darwin/OSX} \item {\bf FreeBSD} \item {\bf HPUX} \item {\bf IRIX} \item {\bf Linux} \item {\bf Tru64} \item {\bf Solaris} \end{itemize} Currently we support the following ACL types (these ACL streams use a reserved part of the stream numbers): \begin{itemize} \item {\bf STREAM\_ACL\_AIX\_TEXT} 1000 AIX specific string representation from acl\_get \item {\bf STREAM\_ACL\_DARWIN\_ACCESS\_ACL} 1001 Darwin (OSX) specific acl\_t string representation from acl\_to\_text (POSIX acl) \item {\bf STREAM\_ACL\_FREEBSD\_DEFAULT\_ACL} 1002 FreeBSD specific acl\_t string representation from acl\_to\_text (POSIX acl) for default acls. \item {\bf STREAM\_ACL\_FREEBSD\_ACCESS\_ACL} 1003 FreeBSD specific acl\_t string representation from acl\_to\_text (POSIX acl) for access acls. \item {\bf STREAM\_ACL\_HPUX\_ACL\_ENTRY} 1004 HPUX specific acl\_entry string representation from acltostr (POSIX acl) \item {\bf STREAM\_ACL\_IRIX\_DEFAULT\_ACL} 1005 IRIX specific acl\_t string representation from acl\_to\_text (POSIX acl) for default acls. \item {\bf STREAM\_ACL\_IRIX\_ACCESS\_ACL} 1006 IRIX specific acl\_t string representation from acl\_to\_text (POSIX acl) for access acls. \item {\bf STREAM\_ACL\_LINUX\_DEFAULT\_ACL} 1007 Linux specific acl\_t string representation from acl\_to\_text (POSIX acl) for default acls. \item {\bf STREAM\_ACL\_LINUX\_ACCESS\_ACL} 1008 Linux specific acl\_t string representation from acl\_to\_text (POSIX acl) for access acls. \item {\bf STREAM\_ACL\_TRU64\_DEFAULT\_ACL} 1009 Tru64 specific acl\_t string representation from acl\_to\_text (POSIX acl) for default acls. \item {\bf STREAM\_ACL\_TRU64\_DEFAULT\_DIR\_ACL} 1010 Tru64 specific acl\_t string representation from acl\_to\_text (POSIX acl) for default acls. \item {\bf STREAM\_ACL\_TRU64\_ACCESS\_ACL} 1011 Tru64 specific acl\_t string representation from acl\_to\_text (POSIX acl) for access acls. \item {\bf STREAM\_ACL\_SOLARIS\_ACLENT} 1012 Solaris specific aclent\_t string representation from acltotext or acl\_totext (POSIX acl) \item {\bf STREAM\_ACL\_SOLARIS\_ACE} 1013 Solaris specific ace\_t string representation from from acl\_totext (NFSv4 or ZFS acl) \end{itemize} In future versions we might support conversion functions from one type of acl into an other for types that are either the same or easily convertible. For now the streams are separate and restoring them on a platform that doesn't recognize them will give you a warning. \subsection{Extended Attributes} \index[general]{Extended Attributes} Something that was on the project list for some time is now implemented for platforms that support a similar kind of interface. Its the support for backup and restore of so called extended attributes. As extended attributes are so platform specific these attributes are saved in separate streams for each platform. Restores of the extended attributes can only be performed on the same platform the backup was done. There is support for all types of extended attributes, but restoring from one type of filesystem onto an other type of filesystem on the same platform may lead to surprises. As extended attributes can contain any type of data they are stored as a series of so called value-pairs. This data must be seen as mostly binary and is stored as such. As security labels from selinux are also extended attributes this option also stores those labels and no specific code is enabled for handling selinux security labels. Currently the following platforms support extended attributes: \begin{itemize} \item {\bf Darwin/OSX} \item {\bf FreeBSD} \item {\bf Linux} \item {\bf NetBSD} \end{itemize} On Linux acls are also extended attributes, as such when you enable ACLs on a Linux platform it will NOT save the same data twice e.g. it will save the ACLs and not the same extended attribute. To enable the backup of extended attributes please add the following to your fileset definition. \begin{bVerbatim} FileSet { Name = "MyFileSet" Include { Options { signature = MD5 xattrsupport = yes } File = ... } } \end{bVerbatim} \subsection{Shared objects} \index[general]{Shared objects} A default build of Bacula will now create the libraries as shared objects (.so) rather than static libraries as was previously the case. The shared libraries are built using {\bf libtool} so it should be quite portable. An important advantage of using shared objects is that on a machine with the Directory, File daemon, the Storage daemon, and a console, you will have only one copy of the code in memory rather than four copies. Also the total size of the binary release is smaller since the library code appears only once rather than once for every program that uses it; this results in significant reduction in the size of the binaries particularly for the utility tools. In order for the system loader to find the shared objects when loading the Bacula binaries, the Bacula shared objects must either be in a shared object directory known to the loader (typically /usr/lib) or they must be in the directory that may be specified on the {\bf ./configure} line using the {\bf {-}{-}libdir} option as: \begin{bVerbatim} ./configure --libdir=/full-path/dir \end{bVerbatim} the default is /usr/lib. If {-}{-}libdir is specified, there should be no need to modify your loader configuration provided that the shared objects are installed in that directory (Bacula does this with the make install command). The shared objects that Bacula references are: \begin{bVerbatim} libbaccfg.so libbacfind.so libbacpy.so libbac.so \end{bVerbatim} These files are symbolically linked to the real shared object file, which has a version number to permit running multiple versions of the libraries if desired (not normally the case). If you have problems with libtool or you wish to use the old way of building static libraries, or you want to build a static version of Bacula you may disable libtool on the configure command line with: \begin{bVerbatim} ./configure --disable-libtool \end{bVerbatim} \subsection{Building Static versions of Bacula} \index[general]{Static linking} In order to build static versions of Bacula, in addition to configuration options that were needed you now must also add --disable-libtool. Example \begin{bVerbatim} ./configure --enable-static-client-only --disable-libtool \end{bVerbatim} \subsection{Virtual Backup (Vbackup)} \index[general]{Virtual Backup} \index[general]{Vbackup} Bacula's virtual backup feature is often called Synthetic Backup or Consolidation in other backup products. It permits you to consolidate the previous Full backup plus the most recent Differential backup and any subsequent Incremental backups into a new Full backup. This new Full backup will then be considered as the most recent Full for any future Incremental or Differential backups. The VirtualFull backup is accomplished without contacting the client by reading the previous backup data and writing it to a volume in a different pool. In some respects the Vbackup feature works similar to a Migration job, in that Bacula normally reads the data from the pool specified in the Job resource, and writes it to the {\bf Next Pool} specified in the Job resource. Note, this means that usually the output from the Virtual Backup is written into a different pool from where your prior backups are saved. Doing it this way guarantees that you will not get a deadlock situation attempting to read and write to the same volume in the Storage daemon. If you then want to do subsequent backups, you may need to move the Virtual Full Volume back to your normal backup pool. Alternatively, you can set your {\bf Next Pool} to point to the current pool. This will cause Bacula to read and write to Volumes in the current pool. In general, this will work, because Bacula will not allow reading and writing on the same Volume. In any case, once a VirtualFull has been created, and a restore is done involving the most current Full, it will read the Volume or Volumes by the VirtualFull regardless of in which Pool the Volume is found. The Vbackup is enabled on a Job by Job in the Job resource by specifying a level of {\bf VirtualFull}. A typical Job resource definition might look like the following: \begin{bVerbatim} Job { Name = "MyBackup" Type = Backup Client=localhost-fd FileSet = "Full Set" Storage = File Messages = Standard Pool = Default SpoolData = yes } # Default pool definition Pool { Name = Default Pool Type = Backup Recycle = yes # Automatically recycle Volumes AutoPrune = yes # Prune expired volumes Volume Retention = 365d # one year NextPool = Full Storage = File } Pool { Name = Full Pool Type = Backup Recycle = yes # Automatically recycle Volumes AutoPrune = yes # Prune expired volumes Volume Retention = 365d # one year Storage = DiskChanger } # Definition of file storage device Storage { Name = File Address = localhost Password = "xxx" Device = FileStorage Media Type = File Maximum Concurrent Jobs = 5 } # Definition of DDS Virtual tape disk storage device Storage { Name = DiskChanger Address = localhost # N.B. Use a fully qualified name here Password = "yyy" Device = DiskChanger Media Type = DiskChangerMedia Maximum Concurrent Jobs = 4 Autochanger = yes } \end{bVerbatim} Then in bconsole or via a Run schedule, you would run the job as: \begin{bVerbatim} run job=MyBackup level=Full run job=MyBackup level=Incremental run job=MyBackup level=Differential run job=MyBackup level=Incremental run job=MyBackup level=Incremental \end{bVerbatim} So providing there were changes between each of those jobs, you would end up with a Full backup, a Differential, which includes the first Incremental backup, then two Incremental backups. All the above jobs would be written to the {\bf Default} pool. To consolidate those backups into a new Full backup, you would run the following: \begin{bVerbatim} run job=MyBackup level=VirtualFull \end{bVerbatim} And it would produce a new Full backup without using the client, and the output would be written to the {\bf Full} Pool which uses the Diskchanger Storage. If the Virtual Full is run, and there are no prior Jobs, the Virtual Full will fail with an error. Note, the Start and End time of the Virtual Full backup is set to the values for the last job included in the Virtual Full (in the above example, it is an Increment). This is so that if another incremental is done, which will be based on the Virtual Full, it will backup all files from the last Job included in the Virtual Full rather than from the time the Virtual Full was actually run. \subsection{Catalog Format} \index[general]{Catalog Format} Bacula 3.0 comes with some changes to the catalog format. The upgrade operation will convert the FileId field of the File table from 32 bits (max 4 billion table entries) to 64 bits (very large number of items). The conversion process can take a bit of time and will likely DOUBLE THE SIZE of your catalog during the conversion. Also you won't be able to run jobs during this conversion period. For example, a 3 million file catalog will take 2 minutes to upgrade on a normal machine. Please don't forget to make a valid backup of your database before executing the upgrade script. See the ReleaseNotes for additional details. \subsection{64 bit Windows Client} \index[general]{Win64 Client} Unfortunately, Microsoft's implementation of Volume Shadown Copy (VSS) on their 64 bit OS versions is not compatible with a 32 bit Bacula Client. As a consequence, we are also releasing a 64 bit version of the Bacula Windows Client (win64bacula-3.0.0.exe) that does work with VSS. These binaries should only be installed on 64 bit Windows operating systems. What is important is not your hardware but whether or not you have a 64 bit version of the Windows OS. Compared to the Win32 Bacula Client, the 64 bit release contains a few differences: \begin{enumerate} \item Before installing the Win64 Bacula Client, you must totally deinstall any prior 2.4.x Client installation using the Bacula deinstallation (see the menu item). You may want to save your .conf files first. \item Only the Client (File daemon) is ported to Win64, the Director and the Storage daemon are not in the 64 bit Windows installer. \item bwx-console is not yet ported. \item bconsole is ported but it has not been tested. \item The documentation is not included in the installer. \item Due to Vista security restrictions imposed on a default installation of Vista, before upgrading the Client, you must manually stop any prior version of Bacula from running, otherwise the install will fail. \item Due to Vista security restrictions imposed on a default installation of Vista, attempting to edit the conf files via the menu items will fail. You must directly edit the files with appropriate permissions. Generally double clicking on the appropriate .conf file will work providing you have sufficient permissions. \item All Bacula files are now installed in {\bf C:/Program Files/Bacula} except the main menu items, which are installed as before. This vastly simplifies the installation. \item If you are running on a foreign language version of Windows, most likely {\bf C:/Program Files} does not exist, so you should use the Custom installation and enter an appropriate location to install the files. \item The 3.0.0 Win32 Client continues to install files in the locations used by prior versions. For the next version we will convert it to use the same installation conventions as the Win64 version. \end{enumerate} This project was funded by Bacula Systems. \subsection{Duplicate Job Control} \index[general]{Duplicate Jobs} The new version of Bacula provides four new directives that give additional control over what Bacula does if duplicate jobs are started. A duplicate job in the sense we use it here means a second or subsequent job with the same name starts. This happens most frequently when the first job runs longer than expected because no tapes are available. The four directives each take as an argument a {\bf yes} or {\bf no} value and are specified in the Job resource. They are: \subsubsection{Allow Duplicate Jobs = \lt{}yes\vb{}no\gt{}} \index[general]{Allow Duplicate Jobs} If this directive is set to {\bf yes}, duplicate jobs will be run. If the directive is set to {\bf no} (default) then only one job of a given name may run at one time, and the action that Bacula takes to ensure only one job runs is determined by the other directives (see below). If {\bf Allow Duplicate Jobs} is set to {\bf no} and two jobs are present and none of the three directives given below permit Canceling a job, then the current job (the second one started) will be canceled. \subsubsection{Allow Higher Duplicates = \lt{}yes\vb{}no\gt{}} \index[general]{Allow Higher Duplicates} This directive was in version 5.0.0, but does not work as expected. If used, it should always be set to no. In later versions of Bacula the directive is disabled (disregarded). \subsubsection{Cancel Running Duplicates = \lt{}yes\vb{}no\gt{}} \index[general]{Cancel Running Duplicates} If {\bf Allow Duplicate Jobs} is set to {\bf no} and if this directive is set to {\bf yes} any job that is already running will be canceled. The default is {\bf no}. \subsubsection{Cancel Queued Duplicates = \lt{}yes\vb{}no\gt{}} \index[general]{Cancel Queued Duplicates} If {\bf Allow Duplicate Jobs} is set to {\bf no} and if this directive is set to {\bf yes} any job that is already queued to run but not yet running will be canceled. The default is {\bf no}. \subsection{TLS Authentication} \index[general]{TLS Authentication} In Bacula version 2.5.x and later, in addition to the normal Bacula CRAM-MD5 authentication that is used to authenticate each Bacula connection, you can specify that you want TLS Authentication as well, which will provide more secure authentication. This new feature uses Bacula's existing TLS code (normally used for communications encryption) to do authentication. To use it, you must specify all the TLS directives normally used to enable communications encryption (TLS Enable, TLS Verify Peer, TLS Certificate, ...) and a new directive: \subsubsection{TLS Authenticate = yes} \begin{bVerbatim} TLS Authenticate = yes \end{bVerbatim} in the main daemon configuration resource (Director for the Director, Client for the File daemon, and Storage for the Storage daemon). When {\bf TLS Authenticate} is enabled, after doing the CRAM-MD5 authentication, Bacula will also do TLS authentication, then TLS encryption will be turned off, and the rest of the communication between the two Bacula daemons will be done without encryption. If you want to encrypt communications data, use the normal TLS directives but do not turn on {\bf TLS Authenticate}. \subsection{bextract non-portable Win32 data} \index[general]{bextract handles Win32 non-portable data} {\bf bextract} has been enhanced to be able to restore non-portable Win32 data to any OS. Previous versions were unable to restore non-portable Win32 data to machines that did not have the Win32 BackupRead and BackupWrite API calls. \subsection{State File updated at Job Termination} \index[general]{State File} In previous versions of Bacula, the state file, which provides a summary of previous jobs run in the {\bf status} command output was updated only when Bacula terminated, thus if the daemon crashed, the state file might not contain all the run data. This version of the Bacula daemons updates the state file on each job termination. \subsection{MaxFullInterval = \lt{}time-interval\gt{}} \index[general]{MaxFullInterval} The new Job resource directive {\bf Max Full Interval = \lt{}time-interval\gt{}} can be used to specify the maximum time interval between {\bf Full} backup jobs. When a job starts, if the time since the last Full backup is greater than the specified interval, and the job would normally be an {\bf Incremental} or {\bf Differential}, it will be automatically upgraded to a {\bf Full} backup. \subsection{MaxDiffInterval = \lt{}time-interval\gt{}} \index[general]{MaxDiffInterval} The new Job resource directive {\bf Max Diff Interval = \lt{}time-interval\gt{}} can be used to specify the maximum time interval between {\bf Differential} backup jobs. When a job starts, if the time since the last Differential backup is greater than the specified interval, and the job would normally be an {\bf Incremental}, it will be automatically upgraded to a {\bf Differential} backup. \subsection{Honor No Dump Flag = \lt{}yes\vb{}no\gt{}} \index[general]{MaxDiffInterval} On FreeBSD systems, each file has a {\bf no dump flag} that can be set by the user, and when it is set it is an indication to backup programs to not backup that particular file. This version of Bacula contains a new Options directive within a FileSet resource, which instructs Bacula to obey this flag. The new directive is: \begin{bVerbatim} Honor No Dump Flag = yes\vb{}no \end{bVerbatim} The default value is {\bf no}. \subsection{Exclude Dir Containing = \lt{}filename-string\gt{}} \index[general]{IgnoreDir} The {\bf ExcludeDirContaining = \lt{}filename\gt{}} is a new directive that can be added to the Include section of the FileSet resource. If the specified filename ({\bf filename-string}) is found on the Client in any directory to be backed up, the whole directory will be ignored (not backed up). For example: \begin{bVerbatim} # List of files to be backed up FileSet { Name = "MyFileSet" Include { Options { signature = MD5 } File = /home Exclude Dir Containing = .excludeme } } \end{bVerbatim} But in /home, there may be hundreds of directories of users and some people want to indicate that they don't want to have certain directories backed up. For example, with the above FileSet, if the user or sysadmin creates a file named {\bf .excludeme} in specific directories, such as \begin{bVerbatim} /home/user/www/cache/.excludeme /home/user/temp/.excludeme \end{bVerbatim} then Bacula will not backup the two directories named: \begin{bVerbatim} /home/user/www/cache /home/user/temp \end{bVerbatim} NOTE: subdirectories will not be backed up. That is, the directive applies to the two directories in question and any children (be they files, directories, etc). \subsection{Bacula Plugins} \index[general]{Plugin} Support for shared object plugins has been implemented in the Linux, Unix and Win32 File daemons. The API will be documented separately in the Developer's Guide or in a new document. For the moment, there is a single plugin named {\bf bpipe} that allows an external program to get control to backup and restore a file. Plugins are also planned (partially implemented) in the Director and the Storage daemon. \subsubsection{Plugin Directory} \index[general]{Plugin Directory} Each daemon (DIR, FD, SD) has a new {\bf Plugin Directory} directive that may be added to the daemon definition resource. The directory takes a quoted string argument, which is the name of the directory in which the daemon can find the Bacula plugins. If this directive is not specified, Bacula will not load any plugins. Since each plugin has a distinctive name, all the daemons can share the same plugin directory. \subsubsection{Plugin Options} \index[general]{Plugin Options} The {\bf Plugin Options} directive takes a quoted string argument (after the equal sign) and may be specified in the Job resource. The options specified will be passed to all plugins when they are run. This each plugin must know what it is looking for. The value defined in the Job resource can be modified by the user when he runs a Job via the {\bf bconsole} command line prompts. Note: this directive may be specified, and there is code to modify the string in the run command, but the plugin options are not yet passed to the plugin (i.e. not fully implemented). \subsubsection{Plugin Options ACL} \index[general]{Plugin Options ACL} The {\bf Plugin Options ACL} directive may be specified in the Director's Console resource. It functions as all the other ACL commands do by permitting users running restricted consoles to specify a {\bf Plugin Options} that overrides the one specified in the Job definition. Without this directive restricted consoles may not modify the Plugin Options. \subsubsection{Plugin = \lt{}plugin-command-string\gt{}} \index[general]{Plugin} The {\bf Plugin} directive is specified in the Include section of a FileSet resource where you put your {\bf File = xxx} directives. For example: \begin{bVerbatim} FileSet { Name = "MyFileSet" Include { Options { signature = MD5 } File = /home Plugin = "bpipe:..." } } \end{bVerbatim} In the above example, when the File daemon is processing the directives in the Include section, it will first backup all the files in {\bf /home} then it will load the plugin named {\bf bpipe} (actually bpipe-dir.so) from the Plugin Directory. The syntax and semantics of the Plugin directive require the first part of the string up to the colon (:) to be the name of the plugin. Everything after the first colon is ignored by the File daemon but is passed to the plugin. Thus the plugin writer may define the meaning of the rest of the string as he wishes. Please see the next section for information about the {\bf bpipe} Bacula plugin. \subsection{The bpipe Plugin} \index[general]{The bpipe Plugin} The {\bf bpipe} plugin is provided in the directory src/plugins/fd/bpipe-fd.c of the Bacula source distribution. When the plugin is compiled and linking into the resulting dynamic shared object (DSO), it will have the name {\bf bpipe-fd.so}. Please note that this is a very simple plugin that was written for demonstration and test purposes. It is and can be used in production, but that was never really intended. The purpose of the plugin is to provide an interface to any system program for backup and restore. As specified above the {\bf bpipe} plugin is specified in the Include section of your Job's FileSet resource. The full syntax of the plugin directive as interpreted by the {\bf bpipe} plugin (each plugin is free to specify the sytax as it wishes) is: \begin{bVerbatim} Plugin = ":::" \end{bVerbatim} where \begin{description} \item {\bf field1} is the name of the plugin with the trailing {\bf -fd.so} stripped off, so in this case, we would put {\bf bpipe} in this field. \item {\bf field2} specifies the namespace, which for {\bf bpipe} is the pseudo path and filename under which the backup will be saved. This pseudo path and filename will be seen by the user in the restore file tree. For example, if the value is {\bf /MYSQL/regress.sql}, the data backed up by the plugin will be put under that "pseudo" path and filename. You must be careful to choose a naming convention that is unique to avoid a conflict with a path and filename that actually exists on your system. \item {\bf field3} for the {\bf bpipe} plugin specifies the "reader" program that is called by the plugin during backup to read the data. {\bf bpipe} will call this program by doing a {\bf popen} on it. \item {\bf field4} for the {\bf bpipe} plugin specifies the "writer" program that is called by the plugin during restore to write the data back to the filesystem. \end{description} Please note that for two items above describing the "reader" and "writer" fields, these programs are "executed" by Bacula, which means there is no shell interpretation of any command line arguments you might use. If you want to use shell characters (redirection of input or output, ...), then we recommend that you put your command or commands in a shell script and execute the script. In addition if you backup a file with the reader program, when running the writer program during the restore, Bacula will not automatically create the path to the file. Either the path must exist, or you must explicitly do so with your command or in a shell script. Putting it all together, the full plugin directive line might look like the following: \begin{bVerbatim} Plugin = "bpipe:/MYSQL/regress.sql:mysqldump -f --opt --databases bacula:mysql" \end{bVerbatim} The directive has been split into two lines, but within the {\bf bacula-dir.conf} file would be written on a single line. This causes the File daemon to call the {\bf bpipe} plugin, which will write its data into the "pseudo" file {\bf /MYSQL/regress.sql} by calling the program {\bf mysqldump -f --opt --database bacula} to read the data during backup. The mysqldump command outputs all the data for the database named {\bf bacula}, which will be read by the plugin and stored in the backup. During restore, the data that was backed up will be sent to the program specified in the last field, which in this case is {\bf mysql}. When {\bf mysql} is called, it will read the data sent to it by the plugn then write it back to the same database from which it came ({\bf bacula} in this case). The {\bf bpipe} plugin is a generic pipe program, that simply transmits the data from a specified program to Bacula for backup, and then from Bacula to a specified program for restore. By using different command lines to {\bf bpipe}, you can backup any kind of data (ASCII or binary) depending on the program called. \subsection{Microsoft Exchange Server 2003/2007 Plugin} \index[general]{Microsoft Exchange Server 2003/2007 Plugin} \subsubsection{Background} The Exchange plugin was made possible by a funded development project between Equiinet Ltd -- www.equiinet.com (many thanks) and Bacula Systems. The code for the plugin was written by James Harper, and the Bacula core code by Kern Sibbald. All the code for this funded development has become part of the Bacula project. Thanks to everyone who made it happen. \subsubsection{Concepts} Although it is possible to backup Exchange using Bacula VSS the Exchange plugin adds a good deal of functionality, because while Bacula VSS completes a full backup (snapshot) of Exchange, it does not support Incremental or Differential backups, restoring is more complicated, and a single database restore is not possible. Microsoft Exchange organises its storage into Storage Groups with Databases inside them. A default installation of Exchange will have a single Storage Group called 'First Storage Group', with two Databases inside it, "Mailbox Store (SERVER NAME)" and "Public Folder Store (SERVER NAME)", which hold user email and public folders respectively. In the default configuration, Exchange logs everything that happens to log files, such that if you have a backup, and all the log files since, you can restore to the present time. Each Storage Group has its own set of log files and operates independently of any other Storage Groups. At the Storage Group level, the logging can be turned off by enabling a function called "Enable circular logging". At this time the Exchange plugin will not function if this option is enabled. The plugin allows backing up of entire storage groups, and the restoring of entire storage groups or individual databases. Backing up and restoring at the individual mailbox or email item is not supported but can be simulated by use of the "Recovery" Storage Group (see below). \subsubsection{Installing} The Exchange plugin requires a DLL that is shipped with Microsoft Exchanger Server called {\bf esebcli2.dll}. Assuming Exchange is installed correctly the Exchange plugin should find this automatically and run without any additional installation. If the DLL can not be found automatically it will need to be copied into the Bacula installation directory (eg C:\verb+\+Program Files\verb+\+Bacula\verb+\+bin). The Exchange API DLL is named esebcli2.dll and is found in C:\verb+\+Program Files\verb+\+Exchsrvr\verb+\+bin on a default Exchange installation. \subsubsection{Backing Up} To back up an Exchange server the Fileset definition must contain at least {\bf Plugin = "exchange:/@EXCHANGE/Microsoft Information Store"} for the backup to work correctly. The 'exchange:' bit tells Bacula to look for the exchange plugin, the '@EXCHANGE' bit makes sure all the backed up files are prefixed with something that isn't going to share a name with something outside the plugin, and the 'Microsoft Information Store' bit is required also. It is also possible to add the name of a storage group to the "Plugin =" line, eg \\ {\bf Plugin = "exchange:/@EXCHANGE/Microsoft Information Store/First Storage Group"} \\ if you want only a single storage group backed up. Additionally, you can suffix the 'Plugin =' directive with ":notrunconfull" which will tell the plugin not to truncate the Exchange database at the end of a full backup. An Incremental or Differential backup will backup only the database logs for each Storage Group by inspecting the "modified date" on each physical log file. Because of the way the Exchange API works, the last logfile backed up on each backup will always be backed up by the next Incremental or Differential backup too. This adds 5MB to each Incremental or Differential backup size but otherwise does not cause any problems. By default, a normal VSS fileset containing all the drive letters will also back up the Exchange databases using VSS. This will interfere with the plugin and Exchange's shared ideas of when the last full backup was done, and may also truncate log files incorrectly. It is important, therefore, that the Exchange database files be excluded from the backup, although the folders the files are in should be included, or they will have to be recreated manually if a bare metal restore is done. \begin{bVerbatim} FileSet { Include { File = C:/Program Files/Exchsrvr/mdbdata Plugin = "exchange:..." } Exclude { File = C:/Program Files/Exchsrvr/mdbdata/E00.chk File = C:/Program Files/Exchsrvr/mdbdata/E00.log File = C:/Program Files/Exchsrvr/mdbdata/E000000F.log File = C:/Program Files/Exchsrvr/mdbdata/E0000010.log File = C:/Program Files/Exchsrvr/mdbdata/E0000011.log File = C:/Program Files/Exchsrvr/mdbdata/E00tmp.log File = C:/Program Files/Exchsrvr/mdbdata/priv1.edb } } \end{bVerbatim} The advantage of excluding the above files is that you can significantly reduce the size of your backup since all the important Exchange files will be properly saved by the Plugin. \subsubsection{Restoring} The restore operation is much the same as a normal Bacula restore, with the following provisos: \begin{itemize} \item The {\bf Where} restore option must not be specified \item Each Database directory must be marked as a whole. You cannot just select (say) the .edb file and not the others. \item If a Storage Group is restored, the directory of the Storage Group must be marked too. \item It is possible to restore only a subset of the available log files, but they {\bf must} be contiguous. Exchange will fail to restore correctly if a log file is missing from the sequence of log files \item Each database to be restored must be dismounted and marked as "Can be overwritten by restore" \item If an entire Storage Group is to be restored (eg all databases and logs in the Storage Group), then it is best to manually delete the database files from the server (eg C:\verb+\+Program Files\verb+\+Exchsrvr\verb+\+mdbdata\verb+\+*) as Exchange can get confused by stray log files lying around. \end{itemize} \subsubsection{Restoring to the Recovery Storage Group} The concept of the Recovery Storage Group is \bulink{well documented by Microsoft}{http://support.microsoft.com/kb/824126}, but to briefly summarize\ldots{} Microsoft Exchange allows the creation of an additional Storage Group called the Recovery Storage Group, which is used to restore an older copy of a database (e.g. before a mailbox was deleted) into without messing with the current live data. This is required as the Standard and Small Business Server versions of Exchange can not ordinarily have more than one Storage Group. To create the Recovery Storage Group, drill down to the Server in Exchange System Manager, right click, and select {\bf "New -> Recovery Storage Group..."}. Accept or change the file locations and click OK. On the Recovery Storage Group, right click and select {\bf "Add Database to Recover..."} and select the database you will be restoring. Restore only the single database nominated as the database in the Recovery Storage Group. Exchange will redirect the restore to the Recovery Storage Group automatically. Then run the restore. \subsubsection{Restoring on Microsoft Server 2007} Apparently the {\bf Exmerge} program no longer exists in Microsoft Server 2007, and hence you use a new procedure for recovering a single mail box. \bulink{This procedure is documented by Microsoft}{http://technet.microsoft.com/en-us/library/aa997694.aspx}, and involves using the {\bf Restore-Mailbox} and {\bf Get-Mailbox Statistics} shell commands. \subsubsection{Caveats} This plugin is still being developed, so you should consider it currently in BETA test, and thus use in a production environment should be done only after very careful testing. When doing a full backup, the Exchange database logs are truncated by Exchange as soon as the plugin has completed the backup. If the data never makes it to the backup medium (eg because of spooling) then the logs will still be truncated, but they will also not have been backed up. A solution to this is being worked on. You will have to schedule a new Full backup to ensure that your next backups will be usable. The "Enable Circular Logging" option cannot be enabled or the plugin will fail. Exchange insists that a successful Full backup must have taken place if an Incremental or Differential backup is desired, and the plugin will fail if this is not the case. If a restore is done, Exchange will require that a Full backup be done before an Incremental or Differential backup is done. The plugin will most likely not work well if another backup application (eg NTBACKUP) is backing up the Exchange database, especially if the other backup application is truncating the log files. The Exchange plugin has not been tested with the {\bf Accurate} option, so we recommend either carefully testing or that you avoid this option for the current time. The Exchange plugin is not called during processing the bconsole {\bf estimate} command, and so anything that would be backed up by the plugin will not be added to the estimate total that is displayed. \subsection{libdbi Framework} \index[general]{libdbi Framework} As a general guideline, Bacula has support for a few catalog database drivers (MySQL and PostgreSQL) coded natively by the Bacula team. With the libdbi implementation, which is a Bacula driver that uses libdbi to access the catalog, we have an open field to use many different kinds database engines following the needs of users. The according to libdbi (http://libdbi.sourceforge.net/) project: libdbi implements a database-independent abstraction layer in C, similar to the DBI/DBD layer in Perl. Writing one generic set of code, programmers can leverage the power of multiple databases and multiple simultaneous database connections by using this framework. Currently the libdbi driver in Bacula project only supports the same drivers natively coded in Bacula. However the libdbi project has support for many others database engines. You can view the list at http://libdbi-drivers.sourceforge.net/. In the future all those drivers can be supported by Bacula, however, they must be tested properly by the Bacula team. Some of benefits of using libdbi are: \begin{itemize} \item The possibility to use proprietary databases engines in which your proprietary licenses prevent the Bacula team from developing the driver. \item The possibility to use the drivers written for the libdbi project. \item The possibility to use other database engines without recompiling Bacula to use them. Just change one line in bacula-dir.conf \item Abstract Database access, this is, unique point to code and profiling catalog database access. \end{itemize} The following drivers have been tested: \begin{itemize} \item PostgreSQL, with and without batch insert \item Mysql, with and without batch insert \end{itemize} In the future, we will test and approve to use others databases engines (proprietary or not) like DB2, Oracle, Microsoft SQL. To compile Bacula to support libdbi we need to configure the code with the --with-dbi and --with-dbi-driver=[database] ./configure options, where [database] is the database engine to be used with Bacula (of course we can change the driver in file bacula-dir.conf, see below). We must configure the access port of the database engine with the option --with-db-port, because the libdbi framework doesn't know the default access port of each database. The next phase is checking (or configuring) the bacula-dir.conf, example: \begin{bVerbatim} Catalog { Name = MyCatalog dbdriver = dbi:mysql; dbaddress = 127.0.0.1; dbport = 3306 dbname = regress; user = regress; password = "" } \end{bVerbatim} The parameter {\bf dbdriver} indicates that we will use the driver dbi with a mysql database. Currently the drivers supported by Bacula are: postgresql, amd mysql; these are the names that may be added to string "dbi:". The following limitations apply when Bacula is set to use the libdbi framework: - Not tested on the Win32 platform - A little performance is lost if comparing with native database driver. The reason is bound with the database driver provided by libdbi and the simple fact that one more layer of code was added. It is important to remember, when compiling Bacula with libdbi, the following packages are needed: \begin{itemize} \item libdbi version 1.0.0, http://libdbi.sourceforge.net/ \item libdbi-drivers 1.0.0, http://libdbi-drivers.sourceforge.net/ \end{itemize} You can download them and compile them on your system or install the packages from your OS distribution. \subsection{Console Command Additions and Enhancements} \index[general]{Console Additions} \subsubsection{Display Autochanger Content} \index[general]{StatusSlots} The {\bf status slots storage=\lt{}storage-name\gt{}} command displays autochanger content. \footnotesize \begin{bVerbatim} Slot | Volume Name | Status | Media Type | Pool | ------+---------------+----------+-------------------+------------| 1 | 00001 | Append | DiskChangerMedia | Default | 2 | 00002 | Append | DiskChangerMedia | Default | 3*| 00003 | Append | DiskChangerMedia | Scratch | 4 | | | | | \end{bVerbatim} \normalsize If you an asterisk ({\bf *}) appears after the slot number, you must run an {\bf update slots} command to synchronize autochanger content with your catalog. \subsubsection{list joblog job=xxx or jobid=nnn} \index[general]{list joblog} A new list command has been added that allows you to list the contents of the Job Log stored in the catalog for either a Job Name (fully qualified) or for a particular JobId. The {\bf llist} command will include a line with the time and date of the entry. Note for the catalog to have Job Log entries, you must have a directive such as: \begin{bVerbatim} catalog = all \end{bVerbatim} In your Director's {\bf Messages} resource. \subsubsection{Use separator for multiple commands} \index[general]{Command Separator} When using bconsole with readline, you can set the command separator with \textbf{@separator} command to one of those characters to write commands who require multiple input in one line. \begin{bVerbatim} !$%&'()*+,-/:;<>?[]^`{|}~ \end{bVerbatim} \subsubsection{Deleting Volumes} The delete volume bconsole command has been modified to require an asterisk (*) in front of a MediaId otherwise the value you enter is a taken to be a Volume name. This is so that users may delete numeric Volume names. The previous Bacula versions assumed that all input that started with a number was a MediaId. This new behavior is indicated in the prompt if you read it carefully. \subsection{Bare Metal Recovery} The old bare metal recovery project is essentially dead. One of the main features of it was that it would build a recovery CD based on the kernel on your system. The problem was that every distribution has a different boot procedure and different scripts, and worse yet, the boot procedures and scripts change from one distribution to another. This meant that maintaining (keeping up with the changes) the rescue CD was too much work. To replace it, a new bare metal recovery USB boot stick has been developed by Bacula Systems. This technology involves remastering a Ubuntu LiveCD to boot from a USB key. Advantages: \begin{enumerate} \item Recovery can be done from within graphical environment. \item Recovery can be done in a shell. \item Ubuntu boots on a large number of Linux systems. \item The process of updating the system and adding new packages is not too difficult. \item The USB key can easily be upgraded to newer Ubuntu versions. \item The USB key has writable partitions for modifications to the OS and for modification to your home directory. \item You can add new files/directories to the USB key very easily. \item You can save the environment from multiple machines on one USB key. \item Bacula Systems is funding its ongoing development. \end{enumerate} The disadvantages are: \begin{enumerate} \item The USB key is usable but currently under development. \item Not everyone may be familiar with Ubuntu (no worse than using Knoppix) \item Some older OSes cannot be booted from USB. This can be resolved by first booting a Ubuntu LiveCD then plugging in the USB key. \item Currently the documentation is sketchy and not yet added to the main manual. See below ... \end{enumerate} The documentation and the code can be found in the {\bf rescue} package in the directory {\bf linux/usb}. \subsection{Miscellaneous} \index[general]{Misc New Features} \subsubsection{Allow Mixed Priority = \lt{}yes\vb{}no\gt{}} \index[general]{Allow Mixed Priority} This directive is only implemented in version 2.5 and later. When set to {\bf yes} (default {\bf no}), this job may run even if lower priority jobs are already running. This means a high priority job will not have to wait for other jobs to finish before starting. The scheduler will only mix priorities when all running jobs have this set to true. Note that only higher priority jobs will start early. Suppose the director will allow two concurrent jobs, and that two jobs with priority 10 are running, with two more in the queue. If a job with priority 5 is added to the queue, it will be run as soon as one of the running jobs finishes. However, new priority 10 jobs will not be run until the priority 5 job has finished. \subsubsection{Bootstrap File Directive -- FileRegex} \index[general]{Bootstrap File Directive} {\bf FileRegex} is a new command that can be added to the bootstrap (.bsr) file. The value is a regular expression. When specified, only matching filenames will be restored. During a restore, if all File records are pruned from the catalog for a Job, normally Bacula can restore only all files saved. That is there is no way using the catalog to select individual files. With this new feature, Bacula will ask if you want to specify a Regex expression for extracting only a part of the full backup. \begin{bVerbatim} Building directory tree for JobId(s) 1,3 ... There were no files inserted into the tree, so file selection is not possible.Most likely your retention policy pruned the files Do you want to restore all the files? (yes\vb{}no): no Regexp matching files to restore? (empty to abort): /tmp/regress/(bin|tests)/ Bootstrap records written to /tmp/regress/working/zog4-dir.restore.1.bsr \end{bVerbatim} \subsubsection{Bootstrap File Optimization Changes} In order to permit proper seeking on disk files, we have extended the bootstrap file format to include a {\bf VolStartAddr} and {\bf VolEndAddr} records. Each takes a 64 bit unsigned integer range (i.e. nnn-mmm) which defines the start address range and end address range respectively. These two directives replace the {\bf VolStartFile}, {\bf VolEndFile}, {\bf VolStartBlock} and {\bf VolEndBlock} directives. Bootstrap files containing the old directives will still work, but will not properly take advantage of proper disk seeking, and may read completely to the end of a disk volume during a restore. With the new format (automatically generated by the new Director), restores will seek properly and stop reading the volume when all the files have been restored. \subsubsection{Solaris ZFS/NFSv4 ACLs} This is an upgrade of the previous Solaris ACL backup code to the new library format, which will backup both the old POSIX(UFS) ACLs as well as the ZFS ACLs. The new code can also restore POSIX(UFS) ACLs to a ZFS filesystem (it will translate the POSIX(UFS)) ACL into a ZFS/NFSv4 one) it can also be used to transfer from UFS to ZFS filesystems. \subsubsection{Virtual Tape Emulation} \index[general]{Virtual Tape Emulation} We now have a Virtual Tape emulator that allows us to run though 99.9\% of the tape code but actually reading and writing to a disk file. Used with the \textbf{disk-changer} script, you can now emulate an autochanger with 10 drives and 700 slots. This feature is most useful in testing. It is enabled by using {\bf Device Type = vtape} in the Storage daemon's Device directive. This feature is only implemented on Linux machines and should not be used for production. \subsubsection{Bat Enhancements} \index[general]{Bat Enhancements} Bat (the Bacula Administration Tool) GUI program has been significantly enhanced and stabilized. In particular, there are new table based status commands; it can now be easily localized using Qt4 Linguist. The Bat communications protocol has been significantly enhanced to improve GUI handling. Note, you {\bf must} use a the bat that is distributed with the Director you are using otherwise the communications protocol will not work. \subsubsection{RunScript Enhancements} \index[general]{RunScript Enhancements} The {\bf RunScript} resource has been enhanced to permit multiple commands per RunScript. Simply specify multiple {\bf Command} directives in your RunScript. \begin{bVerbatim} Job { Name = aJob RunScript { Command = "/bin/echo test" Command = "/bin/echo an other test" Command = "/bin/echo 3 commands in the same runscript" RunsWhen = Before } ... } \end{bVerbatim} A new Client RunScript {\bf RunsWhen} keyword of {\bf AfterVSS} has been implemented, which runs the command after the Volume Shadow Copy has been made. Console commands can be specified within a RunScript by using: {\bf Console = \lt{}command\gt{}}, however, this command has not been carefully tested and debugged and is known to easily crash the Director. We would appreciate feedback. Due to the recursive nature of this command, we may remove it before the final release. \subsubsection{Status Enhancements} \index[general]{Status Enhancements} The bconsole {\bf status dir} output has been enhanced to indicate Storage daemon job spooling and despooling activity. \subsubsection{Connect Timeout} \index[general]{Connect Timeout} The default connect timeout to the File daemon has been set to 3 minutes. Previously it was 30 minutes. \subsubsection{ftruncate for NFS Volumes} \index[general]{ftruncate for NFS Volumes} If you write to a Volume mounted by NFS (say on a local file server), in previous Bacula versions, when the Volume was recycled, it was not properly truncated because NFS does not implement ftruncate (file truncate). This is now corrected in the new version because we have written code (actually a kind user) that deletes and recreates the Volume, thus accomplishing the same thing as a truncate. \subsubsection{Support for Ubuntu} The new version of Bacula now recognizes the Ubuntu (and Kubuntu) version of Linux, and thus now provides correct autostart routines. Since Ubuntu officially supports Bacula, you can also obtain any recent release of Bacula from the Ubuntu repositories. \subsubsection{Recycle Pool = \lt{}pool-name\gt{}} \index[general]{Recycle Pool} The new \textbf{RecyclePool} directive defines to which pool the Volume will be placed (moved) when it is recycled. Without this directive, a Volume will remain in the same pool when it is recycled. With this directive, it can be moved automatically to any existing pool during a recycle. This directive is probably most useful when defined in the Scratch pool, so that volumes will be recycled back into the Scratch pool. \subsubsection{FD Version} \index[general]{FD Version} The File daemon to Director protocol now includes a version number, which although there is no visible change for users, will help us in future versions automatically determine if a File daemon is not compatible. \subsubsection{Max Run Sched Time = \lt{}time-period-in-seconds\gt{}} \index[general]{Max Run Sched Time} The time specifies the maximum allowed time that a job may run, counted from when the job was scheduled. This can be useful to prevent jobs from running during working hours. We can see it like \texttt{Max Start Delay + Max Run Time}. \subsubsection{Max Wait Time = \lt{}time-period-in-seconds\gt{}} \index[general]{Max Wait Time} Previous \textbf{MaxWaitTime} directives aren't working as expected, instead of checking the maximum allowed time that a job may block for a resource, those directives worked like \textbf{MaxRunTime}. Some users are reporting to use \textbf{Incr/Diff/Full Max Wait Time} to control the maximum run time of their job depending on the level. Now, they have to use \textbf{Incr/Diff/Full Max Run Time}. \textbf{Incr/Diff/Full Max Wait Time} directives are now deprecated. \subsubsection{Incremental|Differential Max Wait Time = \lt{}time-period-in-seconds\gt{}} \index[general]{Incremental Max Wait Time} \index[general]{Differential Max Wait Time} These directives have been deprecated in favor of \texttt{Incremental|Differential Max Run Time}. \subsubsection{Max Run Time directives} \index[general]{Max Run Time directives} Using \textbf{Full/Diff/Incr Max Run Time}, it's now possible to specify the maximum allowed time that a job can run depending on the level. \addcontentsline{lof}{figure}{Job time control directives} \begin{center} \includegraphics[width=\linewidth]{different_time} \end{center} \subsubsection{Statistics Enhancements} \index[general]{Statistics Enhancements} If you (or probably your boss) want to have statistics on your backups to provide some \textit{Service Level Agreement} indicators, you could use a few SQL queries on the Job table to report how many: \begin{itemize} \item jobs have run \item jobs have been successful \item files have been backed up \item ... \end{itemize} However, these statistics are accurate only if your job retention is greater than your statistics period. Ie, if jobs are purged from the catalog, you won't be able to use them. Now, you can use the \textbf{update stats [days=num]} console command to fill the JobHistory table with new Job records. If you want to be sure to take in account only \textbf{good jobs}, ie if one of your important job has failed but you have fixed the problem and restarted it on time, you probably want to delete the first \textit{bad} job record and keep only the successful one. For that simply let your staff do the job, and update JobHistory table after two or three days depending on your organization using the \textbf{[days=num]} option. These statistics records aren't used for restoring, but mainly for capacity planning, billings, etc. The Bweb interface provides a statistics module that can use this feature. You can also use tools like Talend or extract information by yourself. The \textbf{Statistics Retention = \lt{}time\gt{}} director directive defines the length of time that Bacula will keep statistics job records in the Catalog database after the Job End time. (In \texttt{JobHistory} table) When this time period expires, and if user runs \texttt{prune stats} command, Bacula will prune (remove) Job records that are older than the specified period. You can use the following Job resource in your nightly \textbf{BackupCatalog} job to maintain statistics. \begin{bVerbatim} Job { Name = BackupCatalog ... RunScript { Console = "update stats days=3" Console = "prune stats yes" RunsWhen = After RunsOnClient = no } } \end{bVerbatim} \subsubsection{ScratchPool = \lt{}pool-resource-name\gt{}} \index[general]{ScratchPool} This directive permits to specify a specific \textsl{Scratch} pool for the current pool. This is useful when using multiple storage sharing the same mediatype or when you want to dedicate volumes to a particular set of pool. \subsubsection{Enhanced Attribute Despooling} \index[general]{Attribute Despooling} If the storage daemon and the Director are on the same machine, the spool file that contains attributes is read directly by the Director instead of being transmitted across the network. That should reduce load and speedup insertion. \subsubsection{SpoolSize = \lt{}size-specification-in-bytes\gt{}} \index[general]{SpoolSize} A new Job directive permits to specify the spool size per job. This is used in advanced job tunning. {\bf SpoolSize={\it bytes}} \subsubsection{MaximumConsoleConnections = \lt{}number\gt{}} \index[general]{MaximumConsoleConnections} A new director directive permits to specify the maximum number of Console Connections that could run concurrently. The default is set to 20, but you may set it to a larger number. \subsubsection{VerId = \lt{}string\gt{}} \index[general]{VerId} A new director directive permits to specify a personnal identifier that will be displayed in the \texttt{version} command. \subsubsection{dbcheck enhancements} \index[general]{dbcheck enhancements} If you are using Mysql, dbcheck will now ask you if you want to create temporary indexes to speed up orphaned Path and Filename elimination. A new \texttt{-B} option allows you to print catalog information in a simple text based format. This is useful to backup it in a secure way. \begin{bVerbatim} $ dbcheck -B catalog=MyCatalog db_type=MySQL db_name=regress db_driver= db_user=regress db_password= db_address= db_port=0 db_socket= \end{bVerbatim} You can now specify the database connection port in the command line. \subsubsection{{-}{-}docdir configure option} \index[general]{{-}{-}docdir configure option} You can use {-}{-}docdir= on the ./configure command to specify the directory where you want Bacula to install the LICENSE, ReleaseNotes, ChangeLog, ... files. The default is {\bf /usr/share/doc/bacula}. \subsubsection{{-}{-}htmldir configure option} \index[general]{{-}{-}htmldir configure option} You can use {-}{-}htmldir= on the ./configure command to specify the directory where you want Bacula to install the bat html help files. The default is {\bf /usr/share/doc/bacula/html} \subsubsection{{-}{-}with-plugindir configure option} \index[general]{{-}{-}plugindir configure option} You can use {-}{-}plugindir= on the ./configure command to specify the directory where you want Bacula to install the plugins (currently only bpipe-fd). The default is /usr/lib.