\section{The FileSet Resource} \label{blb:FileSetResource} \index[general]{Resource!FileSet} \index[general]{FileSet Resource} The FileSet resource defines what files are to be included or excluded in a backup job. A \textbf{FileSet} resource is required for each backup Job. It consists of a list of files or directories to be included, a list of files or directories to be excluded and the various backup options such as compression, encryption, and signatures that are to be applied to each file. Any change to the list of the included files will cause \bacula{} to automatically create a new FileSet (defined by the name and an MD5 checksum of the Include/Exclude contents). Each time a new FileSet is created, \bacula{} will ensure that the next backup is always a Full save. \bacula{} is designed to handle most character sets of the world, US \acs{ASCII}, German, French, Chinese, \ldots{} However, it does this by encoding everything in UTF-8, and it expects all configuration files (including those read on Win32 machines) to be in UTF-8 format. UTF-8 is typically the default on Linux machines, but not on all Unix machines, nor on Windows, so you must take some care to ensure that your locale is set properly before starting \bacula{}. On most modern Win32 machines, you can edit the conf files with \textbf{notepad} and choose output encoding UTF-8. To ensure that \bacula{} configuration files can be correctly read including foreign characters the \textbf{LANG} environment variable must end in \textbf{.UTF-8}. A full example is \textbf{en\_US.UTF-8}. The exact syntax may vary a bit from OS to OS, and exactly how you define it will also vary. \bacula{} assumes that all filenames are in UTF-8 format on Linux and Unix machines. On Win32 they are in Unicode (UTF-16), and will be automatically converted to UTF-8 format. \begin{bdescription} \item [FileSet] \index[dir]{FileSet} \index[dir]{Directive!FileSet} Start of the FileSet resource. One \textbf{FileSet} resource must be defined for each Backup job. \bwebdoc{Director:Fileset:Name} \item [Name = \bbracket{name}] \index[dir]{Name} \index[dir]{Directive!Name} The name of the FileSet resource. This directive is required. \bwebdoc{Director:Fileset:IgnoreFilesetChanges} \item [Ignore FileSet Changes = \byesorno{}] \index[dir]{Ignore FileSet Changes} \index[dir]{Directive!Ignore FileSet Changes} Normally, if you modify the FileSet Include or Exclude lists, the next backup will be forced to a Full so that \bacula{} can guarantee that any additions or deletions are properly saved. We strongly recommend against setting this directive to yes, since doing so may cause you to have an incomplete set of backups. If this directive is set to \bvalue{yes}, any changes you make to the FileSet Include or Exclude lists, will not force a Full during subsequent backups. Note that any changes to Options resources in the FileSet are not considered by this directive. You can use the Accurate mode for this to be treated correctly, or schedule a new Full backup manually. The default is \bdefaultvalue{no}, in which case, if you change the Include or Exclude lists, \bacula{} will force a Full backup to ensure that everything is properly backed up. \bwebdoc{Director:Fileset:EnableVss} \item [Enable VSS = \byesorno{}] \index[dir]{Enable VSS} \index[dir]{Directive!Enable VSS} If this directive is set to \bvalue{yes} the File daemon will be notified that the user wants to use a \ac{VSS} %Volume Shadow Copy Service (VSS) backup for this job. The default is \bdefaultvalue{yes}. This directive is effective only for \acs{VSS} enabled Win32 File daemons. It permits a consistent copy of open files to be made for cooperating writer applications, and for applications that are not \acs{VSS} aware, \bacula{} can at least copy open files. The \acl{VSS}%Volume Shadow Copy will only be done on Windows drives where the drive (e.g. C\string:, D\string:, \ldots{}) is explicitly mentioned in a \bdirectivename{File} directive. For more information, please see the \bilink{Windows}{blb:VSS} chapter of this manual. \bwebdoc{Director:Fileset:EnableSnapshot} \item [Enable Snapshot = \byesorno{}] \index[dir]{Enable Snapshot} \index[dir]{Directive!Enable Snapshot} If this directive is set to \textbf{yes} the File daemon will be notified that the user wants to use the Snapshot Engine for this job. The default is \bdefaultvalue{no}. This directive is effective only for Snapshot enabled Unix File daemons. It permits a consistent copy of open files to be made for cooperating applications. The \texttt{bsnapshot} tool should be installed on the Client. \bwebdoc{Director:Fileset:Include} \item [Include \{Options \{\bbracket{file-options}\} \ldots{}; \bbracket{file-list} \} ] \index[dir]{Include \{[ Options \{\bbracket{file-options}\} \ldots{}] \bbracket{file-list} \} } \index[dir]{Directive!Include} %\item [Options \{\bbracket{file-options} \} ] %\index[dir]{Options \{\bbracket{file-options} \} } \bwebdoc{Director:Fileset:Exclude} \item [Exclude \{\bbracket{file-list} \}] \index[dir]{Exclude \{\bbracket{file-list} \} } \index[dir]{Directive!Exclude} \bwebdoc{Director:Fileset:End} \end{bdescription} The Include resource must contain a list of directories and/or files to be processed in the backup job. Normally, all files found in all subdirectories of any directory in the Include File list will be backed up. Note, see below for the definition of \bbracket{file-list}. The Include resource may also contain one or more Options resources that specify options such as compression to be applied to all or any subset of the files found when processing the file-list for backup. Please see below for more details concerning Options resources. There can be any number of \textbf{Include} resources within the FileSet, each having its own list of directories or files to be backed up and the backup options defined by one or more Options resources. The \textbf{file-list} consists of one file or directory name per line. Directory names should be specified without a trailing slash with Unix path notation. Windows users, please take note to specify directories (even \bdirectoryname{c:/\ldots{}}) in Unix path notation. If you use Windows conventions, you will most likely not be able to restore your files due to the fact that the Windows path separator was defined as an escape character long before Windows existed, and \bacula{} adheres to that convention (i.e. \\ means the next character appears as itself). You should always specify a full path for every directory and file that you list in the FileSet. In addition, on Windows machines, you should \textbf{always} prefix the directory or filename with the drive specification (e.g. \bdirectoryname{c:/xxx}) using Unix directory name separators (forward slash). The drive letter itself can be upper or lower case (e.g. \bdirectoryname{c:/xxx} or \bdirectoryname{C:/xxx}). \bacula{}'s default for processing directories is to recursively descend in the directory saving all files and subdirectories. \bacula{} will not by default cross filesystems (or mount points in Unix parlance). This means that if you specify the root partition (e.g. \bdirectoryname{/}), \bacula{} will save only the root partition and not any of the other mounted filesystems. Similarly on Windows systems, you must explicitly specify each of the drives you want saved (e.g. \bdirectoryname{c:/} and \bdirectoryname{d:/} \ldots{}). In addition, at least for Windows systems, you will most likely want to enclose each specification within double quotes particularly if the directory (or file) name contains spaces. The \btool{df} command on Unix systems will show you which mount points you must specify to save everything. See below for an example. Take special care not to include a directory twice or \bacula{} will backup the same files two times wasting a lot of space on your archive device. Including a directory twice is very easy to do. For example: \begin{bVerbatim} Include { Options {compression=GZIP } File = / File = /usr } \end{bVerbatim} on a Unix system where \bdirectoryname{/usr} is a subdirectory (rather than a mounted filesystem) will cause \bdirectoryname{/usr} to be backed up twice. Please take note of the following items in the FileSet syntax: \begin{benumerate} \item There is no equal sign (=) after the Include and before the opening brace (\{). The same is true for the Exclude. \item Each directory (or filename) to be included or excluded is preceded by a \textbf{File =}. Previously they were simply listed on separate lines. \item The options that previously appeared on the Include line now must be specified within their own Options resource. \item The Exclude resource does not accept Options. \item When using wild-cards or regular expressions, directory names are always terminated with a slash (/) and filenames have no trailing slash. \end{benumerate} The Options resource is optional, but when specified, it will contain a list of \textbf{keyword=value} options to be applied to the file-list. See below for the definition of file-list. Multiple Options resources may be specified one after another. As the files are found in the specified directories, the Options will applied to the filenames to determine if and how the file should be backed up. The wildcard and regular expression pattern matching parts of the Options resources are checked in the order they are specified in the FileSet until the first one that matches. Once one matches, the compression and other flags within the Options specification will apply to the pattern matched. A key point is that in the absence of an Option or no other Option is matched, every file is accepted for backing up. This means that if you want to exclude something, you must explicitly specify an Option with an \textbf{exclude = yes} and some pattern matching. Once \bacula{} determines that the Options resource matches the file under consideration, that file will be saved without looking at any other Options resources that may be present. This means that any wild cards must appear before an Options resource without wild cards. If for some reason, \bacula{} checks all the Options resources to a file under consideration for backup, but there are no matches (generally because of wild cards that don't match), \bacula{} as a default will then backup the file. This is quite logical if you consider the case of no Options clause is specified, where you want everything to be backed up, and it is important to keep in mind when excluding as mentioned above. However, one additional point is that in the case that no match was found, \bacula{} will use the options found in the last Options resource. As a consequence, if you want a particular set of ``default'' options, you should put them in an Options resource after any other Options. It is a good idea to put all your wild-card and regex expressions inside double quotes to prevent conf file scanning problems. This is perhaps a bit overwhelming, so there are a number of examples included below to illustrate how this works. You find yourself using a lot of Regex statements, which will cost quite a lot of CPU time, we recommend you simplify them if you can, or better yet convert them to Wild statements which are much more efficient. The directives within an Options resource may be one of the following: \begin{bdescription} \item [compression=\acs{GZIP}] \index[dir]{compression} \index[dir]{Directive!compression} All files saved will be software compressed using the GNU ZIP compression format. The compression is done on a file by file basis by the File daemon. If there is a problem reading the tape in a single record of a file, it will at most affect that file and none of the other files on the tape. Normally this option is \textbf{not} needed if you have a modern tape drive as the drive will do its own compression. In fact, if you specify software compression at the same time you have hardware compression turned on, your files may actually take more space on the volume. Software compression is very important if you are writing your Volumes to a file, and it can also be helpful if you have a fast computer but a slow network, otherwise it is generally better to rely your tape drive's hardware compression. As noted above, it is not generally a good idea to do both software and hardware compression. Specifying \textbf{\acs{GZIP}} uses the default compression level \bdefaultvalue{6} (i.e. \textbf{\acs{GZIP}} is identical to \textbf{\acs{GZIP}6}). If you want a different compression level (1 through 9), you can specify it by appending the level number with no intervening spaces to \textbf{\acs{GZIP}}. Thus \textbf{compression=\acs{GZIP}1} would give minimum compression but the fastest algorithm, and \textbf{ compression=\acs{GZIP}9} would give the highest level of compression, but requires more computation. According to the \acs{GZIP} documentation, compression levels greater than six generally give very little extra compression and are rather CPU intensive. You can overwrite this option per Storage resource with \bilink{AllowCompression}{blb:AllowCompression} option. \item [compression=\acs{LZO}] \index[dir]{compression} \index[dir]{Directive!compression} All files saved will be software compressed using the \acs{LZO} compression format. The compression is done on a file by file basis by the File daemon. Everything else about \acs{GZIP} is true for \acs{LZO}. \acs{LZO} provides much faster compression and decompression speed but lower compression ratio than \acs{GZIP}. If your CPU is fast enough you should be able to compress your data without making the backup duration longer. Note that bacula only use one compression level \acs{LZO}1X-1 specified by \acs{LZO}. You can overwrite this option per Storage resource with \bilink{AllowCompression}{blb:AllowCompression} option. \item [compression=\acs{ZSTD}] \index[dir]{compression} \index[dir]{Directive!compression} All files saved will be software compressed using the \acs{ZSTD} compression format. The compression is done on a file by file basis by the File daemon. Everything else about \acs{GZIP} is true for \acs{ZSTD}. \acs{ZSTD} provides much faster compression and decompression speed but lower compression ratio than \acs{GZIP}. If your CPU is fast enough you should be able to compress your data without making the backup duration longer. You can overwrite this option per Storage resource with \bilink{AllowCompression}{blb:AllowCompression} option. \item [signature=SHA1] \index[dir]{signature} \index[dir]{SHA1} \index[dir]{Directive!signature} An SHA1 signature will be computed for all The SHA1 algorithm is purported to be some what slower than the MD5 algorithm, but at the same time is significantly better from a cryptographic point of view (i.e. much fewer collisions, much lower probability of being hacked.) It adds four more bytes than the MD5 signature. We strongly recommend that either this option or MD5 be specified as a default for all files. Note, only one of the two options MD5 or SHA1 can be computed for any file. \item [signature=SHA256] \index[dir]{signature} \index[dir]{SHA256} \index[dir]{Directive!signature} A SHA256 signature will be computed for all files saved. Adding this option generates about 5\% extra overhead for each file saved. In addition to the additional CPU time, the SHA256 signature adds 44 more bytes per file to your catalog. \item [signature=SHA512] \index[dir]{signature} \index[dir]{SHA256} \index[dir]{Directive!signature} A SHA256 signature will be computed for all files saved. Adding this option generates about 5\% extra overhead for each file saved. In addition to the additional CPU time, the SHA512 signature adds 87 more bytes per file to your catalog. \item [signature=MD5] \index[dir]{signature} \index[dir]{MD5} \index[dir]{Directive!signature} An MD5 signature will be computed for all files saved. Adding this option generates about 5\% extra overhead for each file saved. In addition to the additional CPU time, the MD5 signature adds 16 more bytes per file to your catalog. We strongly recommend that this option or the SHA1 option be specified as a default for all files. \item [signature=XXHASH] \index[dir]{signature} \index[dir]{XXHASH} \index[dir]{Directive!signature} An XXHASH signature will be computed for all files saved. \item [dedup=none|storage|bothsides] \index[dir]{dedup} The \textbf{Dedup} Fileset option can use the following values: \begin{bdescription} \item[*] \textbf{storage} All the deduplication work is done on Storage Daemon side if the device type is \bdefaultvalue{dedup}. (Default value) \item \textbf{none} Force File Daemon and Storage Daemon to not use deduplication even if the device type is \textbf{dedup}. \item \textbf{bothsides} The deduplication work is done on the FD and the SD if the device type is \textbf{dedup}. \end{bdescription} \item[basejob=\bbracket{options}] \index[dir]{basejob} \index[dir]{Directive!basejob} The options letters specified are used when running a \textbf{Backup Level=Full} with BaseJobs. The options letters are the same as in the \textbf{accurate=} option below. \item[accurate=\bbracket{options}] \index[dir]{accurate} \index[dir]{Directive!accurate} The options letters specified are used when running a \textbf{Backup Level=Incremental/Differential} in Accurate mode. The options letters are the same as in the \textbf{verify=} option below with the addition of the following options: \begin{bdescription} \item [A] Always backup files \item [M] Look mtime/ctime like normal incremental backup \item [o] Save only the metadata of a file when possible\footnote{Available from version 13.0.0}. The 'o' option should be used in conjunction with one of the signature checking options (1, 2, 3, or 5). When the option is specified, the signature is computed only for files that have one of the other accurate options specified triggering a backup of the file (for example an inode change, a permission change, etc...). In cases where only the file's metadata has changed (ie: the signature is identical), only the file's attributes will be backed up. If the file's data has been changed (hence a different singature), the file will be backed up in the usual way \end{bdescription} \item [verify=\bbracket{options}] \index[dir]{verify} \index[dir]{Directive!verify} The options letters specified are used when running a \textbf{Verify Level=Catalog} as well as the \textbf{DiskToCatalog} level job. The options letters may be any combination of the following: \begin{bdescription} \item [i] compare the inodes \item [p] compare the permission bits \item [n] compare the number of links \item [u] compare the user id \item [g] compare the group id \item [s] compare the size \item [a] compare the access time \item [m] compare the modification time (st\_mtime) % Not implemented % \item [M] % compare the file modification time and/or creation time (depending % on the \texttt{mtimeonly} FileSet option) with the last backup % timestamp. \item [c] compare the change time (st\_ctime) \item [d] report file size decreases \item [5] compare the MD5 signature \item [1] compare the SHA1 signature \item [2] compare the SHA256 signature \item [3] compare the SHA512 signature \end{bdescription} A useful set of general options on the \textbf{Level=Catalog} or \textbf{Level=DiskToCatalog} verify is \textbf{pins5} i.e. compare permission bits, then inodes, number of links, size, and finally MD5 changes. To save some space when backing up files which have only metadata part changed (e.g. changed permissions) the 'pino5' options could be used (or some other combination with the 'o' flag). It will then compare permision bits, inotes, number of links and if any of it changes it will also compute file's singature to verify if only metadata need to be backed up or is it needed to update file's contents. \item [onefs=\byesorno{}] \index[dir]{onefs} \index[dir]{Directive!onefs} If set to \bdefaultvalue{yes} (the default), \textbf{\bacula{}} will remain on a single file system. That is it will not backup file systems that are mounted on a subdirectory. If you are using a *nix system, you may not even be aware that there are several different filesystems as they are often automatically mounted by the OS (e.g. \bdirectoryname{/dev}, \bdirectoryname{/net}, \bdirectoryname{/sys}, \bdirectoryname{/proc}, \ldots{}). With \bacula{} 1.38.0 or later, it will inform you when it decides not to traverse into another filesystem. This can be very useful if you forgot to backup a particular partition. An example of the informational message in the job report is: \begin{bVerbatim} rufus-fd: /misc is a different filesystem. Will not descend from / into /misc rufus-fd: /net is a different filesystem. Will not descend from / into /net rufus-fd: /var/lib/nfs/rpc_pipefs is a different filesystem. Will not descend from /var/lib/nfs into /var/lib/nfs/rpc_pipefs rufus-fd: /selinux is a different filesystem. Will not descend from / into /selinux rufus-fd: /sys is a different filesystem. Will not descend from / into /sys rufus-fd: /dev is a different filesystem. Will not descend from / into /dev rufus-fd: /home is a different filesystem. Will not descend from / into /home \end{bVerbatim} Note: in older versions of \bacula{}, the above message was of the form: \begin{bVerbatim} Filesystem change prohibited. Will not descend into /misc \end{bVerbatim} If you wish to backup multiple filesystems, you can explicitly list each filesystem you want saved. Otherwise, if you set the onefs option to \textbf{no}, \bacula{} will backup all mounted file systems (i.e. traverse mount points) that are found within the \textbf{FileSet}. Thus if you have NFS or Samba file systems mounted on a directory listed in your FileSet, they will also be backed up. Normally, it is preferable to set \textbf{onefs=yes} and to explicitly name each filesystem you want backed up. Explicitly naming the filesystems you want backed up avoids the possibility of getting into a infinite loop recursing filesystems. Another possibility is to use \textbf{onefs=no} and to set \textbf{fstype=ext2, \ldots{}}. See the example below for more details. If you think that \bacula{} should be backing up a particular directory and it is not, and you have \textbf{onefs=no} set, before you complain, please do: \begin{bVerbatim} stat / stat \end{bVerbatim} where you replace \textbf{filesystem} with the one in question. If the \textbf{Device:} number is different for \bdirectoryname{/} and for your filesystem, then they are on different filesystems. E.g. \begin{bVerbatim} stat / File: `/' Size: 4096 Blocks: 16 IO Block: 4096 directory Device: 302h/770d Inode: 2 Links: 26 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2005-11-10 12:28:01.000000000 +0100 Modify: 2005-09-27 17:52:32.000000000 +0200 Change: 2005-09-27 17:52:32.000000000 +0200 stat /net File: `/home' Size: 4096 Blocks: 16 IO Block: 4096 directory Device: 308h/776d Inode: 2 Links: 7 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2005-11-10 12:28:02.000000000 +0100 Modify: 2005-11-06 12:36:48.000000000 +0100 Change: 2005-11-06 12:36:48.000000000 +0100 \end{bVerbatim} Also be aware that even if you include \bdirectoryname{/home} in your list of files to backup, as you most likely should, you will get the informational message that ``/home is a different filesystem'' when \bacula{} is processing the \bdirectoryname{/} directory. This message does not indicate an error. This message means that while examining the \textbf{File =} referred to in the second part of the message, \bacula{} will not descend into the directory mentioned in the first part of the message. However, it is possible that the separate filesystem will be backed up despite the message. For example, consider the following FileSet: \begin{bVerbatim} File = / File = /var \end{bVerbatim} where \bdirectoryname{/var} is a separate filesystem. In this example, you will get a message saying that \bacula{} will not decend from \bdirectoryname{/} into \bdirectoryname{/var}. But it is important to realise that \bacula{} will descend into \bdirectoryname{/var} from the second File directive shown above. In effect, the warning is bogus, but it is supplied to alert you to possible omissions from your FileSet. In this example, \bdirectoryname{/var} will be backed up. If you changed the FileSet such that it did not specify \bdirectoryname{/var}, then \bdirectoryname{/var} will not be backed up. \item [honor nodump flag=\byesorno{}] \index[dir]{honornodumpflag} \index[dir]{Directive!honornodumpflag} If your file system supports the \textbf{nodump} flag (e. g. most BSD-derived systems) \bacula{} will honor the setting of the flag when this option is set to \textbf{yes}. Files having this flag set will not be included in the backup and will not show up in the catalog. For directories with the \textbf{nodump} flag set recursion is turned off and the directory will be listed in the catalog. If the \textbf{honor nodump flag} option is not defined or set to \textbf{no} every file and directory will be eligible for backup. \label{blb:portable} \item [portable=\byesorno{}] \index[dir]{portable} \index[dir]{Directive!portable} If set to \textbf{yes} (default is \bdefaultvalue{no}), the \bacula{} File daemon will backup Win32 files in a portable format, but not all Win32 file attributes will be saved and restored. By default, this option is set to \bdefaultvalue{no}, which means that on Win32 systems, the data will be backed up using Windows BackupRead API calls and all the security and ownership attributes will be properly backed up (and restored). However this format is not portable to other systems -- e.g. Unix, Win95/98/Me. When backing up Unix systems, this option is ignored, and unless you have a specific need to have portable backups, we recommend accept the default ({\bf no}) so that the maximum information concerning your Windows files is saved. \item [recurse=\byesorno{}] \index[dir]{recurse} \index[dir]{Directive!recurse} If set to \bdefaultvalue{yes} (the default), \bacula{} will recurse (or descend) into all subdirectories found unless the directory is explicitly excluded using an \textbf{exclude} definition. If you set \textbf{recurse=no}, \bacula{} will save the subdirectory entries, but not descend into the subdirectories, and thus will not save the files or directories contained in the subdirectories. Normally, you will want the default (\bdefaultvalue{yes}). \item [sparse=\byesorno{}] \index[dir]{sparse} \index[dir]{Directive!sparse} Enable special code that checks for sparse files such as created by ndbm. The default is \bdefaultvalue{no}, so no checks are made for sparse files. You may specify \textbf{sparse=yes} even on files that are not sparse file. No harm will be done, but there will be a small additional overhead to check for buffers of all zero, and if there is a 32K block of all zeros (see below), that block will become a hole in the file, which may not be desirable if the original file was not a sparse file. \textbf{Restrictions:} \bacula{} reads files in 64K buffers. If the whole buffer is zero, it will be treated as a sparse block and not written to tape. However, if any part of the buffer is non-zero, the whole buffer will be written to tape, possibly including some disk sectors (generally 4098 bytes) that are all zero. As a consequence, \bacula{}'s detection of sparse blocks is in 64K increments rather than the system block size. If anyone considers this to be a real problem, please send in a request for change with the reason. If you are not familiar with sparse files, an example is say a file where you wrote 512 bytes at address zero, then 512 bytes at address 1 million. The operating system will allocate only two blocks, and the empty space or hole will have nothing allocated. However, when you read the sparse file and read the addresses where nothing was written, the OS will return all zeros as if the space were allocated, and if you backup such a file, a lot of space will be used to write zeros to the volume. Worse yet, when you restore the file, all the previously empty space will now be allocated using much more disk space. By turning on the \textbf{sparse} option, \bacula{} will specifically look for empty space in the file, and any empty space will not be written to the Volume, nor will it be restored. The price to pay for this is that \bacula{} must search each block it reads before writing it. On a slow system, this may be important. If you suspect you have sparse files, you should benchmark the difference or set sparse for only those files that are really sparse. You probably should not use this option on files or raw disk devices that are not really sparse files (i.e. have holes in them). \label{blb:readfifo} \item [readfifo=\byesorno{}] \index[dir]{readfifo} \index[dir]{Directive!readfifo} If enabled, tells the Client to read the data on a backup and write the data on a restore to any FIFO (pipe) that is explicitly mentioned in the FileSet. In this case, you must have a program already running that writes into the FIFO for a backup or reads from the FIFO on a restore. This can be accomplished with the \textbf{RunBeforeJob} directive. If this is not the case, \bacula{} will hang indefinitely on reading/writing the FIFO. When this is not enabled (\bdefaultvalue{no}, the default), the Client simply saves the directory entry for the FIFO. Unfortunately, when \bacula{} runs a RunBeforeJob, it waits until that script terminates, and if the script accesses the FIFO to write into the it, the \bacula{} job will block and everything will stall. However, Vladimir Stavrinov as supplied tip that allows this feature to work correctly. He simply adds the following to the beginning of the RunBeforeJob script: \begin{bVerbatim} exec > /dev/null \end{bVerbatim} \item [noatime=\byesorno{}] \index[dir]{noatime} \index[dir]{Directive!noatime} If enabled, and if your Operating System supports the O\_NOATIME file open flag, \bacula{} will open all files to be backed up with this option. It makes it possible to read a file without updating the inode atime (and also without the inode ctime update which happens if you try to set the atime back to its previous value). It also prevents a race condition when two programs are reading the same file, but only one does not want to change the atime. It's most useful for backup programs and file integrity checkers (and bacula can fit on both categories). This option is particularly useful for sites where users are sensitive to their MailBox file access time. It replaces both the \textbf{keepatime} option without the inconveniences of that option (see below). If your Operating System does not support this option, it will be silently ignored by \bacula{}. \item [mtimeonly=\byesorno{}] \index[dir]{mtimeonly} \index[dir]{Directive!mtimeonly} If enabled, tells the Client that the selection of files during Incremental and Differential backups should based only on the st\_mtime value in the stat() packet. The default is \bdefaultvalue{no} which means that the selection of files to be backed up will be based on both the st\_mtime and the st\_ctime values. In general, it is not recommended to use this option. \item [keepatime=\byesorno{}] \index[dir]{keepatime} \index[dir]{Directive!keepatime} The default is \bdefaultvalue{no}. When enabled, \bacula{} will reset the st\_atime (access time) field of files that it backs up to their value prior to the backup. This option is not generally recommended as there are very few programs that use st\_atime, and the backup overhead is increased because of the additional system call necessary to reset the times. However, for some files, such as mailboxes, when \bacula{} backs up the file, the user will notice that someone (\bacula{}) has accessed the file. In this, case keepatime can be useful. (I'm not sure this works on Win32). Note, if you use this feature, when \bacula{} resets the access time, the change time (st\_ctime) will automatically be modified by the system, so on the next incremental job, the file will be backed up even if it has not changed. As a consequence, you will probably also want to use \textbf{mtimeonly = yes} as well as keepatime (thanks to Rudolf Cejka for this tip). \item [checkfilechanges=\byesorno{}] \index[dir]{checkfilechanges} \index[dir]{Directive!checkfilechanges} If enabled, the Client will check size, age of each file after their backup to see if they have changed during backup. If time or size mismatch, an error will raise. The default value is \bvalue{yes}. \begin{bVerbatim} zog-fd: Client1.2007-03-31_09.46.21 Error: /tmp/test mtime changed during backup. \end{bVerbatim} In general, it is recommended to use this option to backup regular files. It is not compatible with fifo or plugin streams. \item [hardlinks=\byesorno{}] \index[dir]{hardlinks} \index[dir]{Directive!hardlinks} When enabled (\bdefaultvalue{yes}, the default), this directive will cause hard links to be backed up. However, the File daemon keeps track of hard linked files and will backup the data only once. The process of keeping track of the hard links can be quite expensive if you have lots of them (tens of thousands or more). This doesn't occur on normal Unix systems, but if you use a program like BackupPC, it can create hundreds of thousands, or even millions of hard links. Backups become very long and the File daemon will consume a lot of CPU power checking hard links. In such a case, set \textbf{hardlinks=no} and hard links will not be backed up. Note, using this option will most likely backup more data and on a restore the file system will not be restored identically to the original. \item [wild=\bbracket{string}] \index[dir]{wild} \index[dir]{Directive!wild} Specifies a wild-card string to be applied to the filenames and directory names. Note, if \textbf{Exclude} is not enabled, the wild-card will select which files are to be included. If \textbf{Exclude=yes} is specified, the wild-card will select which files are to be excluded. Multiple wild-card directives may be specified, and they will be applied in turn until the first one that matches. Note, if you exclude a directory, no files or directories below it will be matched. You may want to test your expressions prior to running your backup by using the bwild program. Please see the \bxlink{Utilities}{blb:bwild}{utility}{chapter} of the \butilityman{} for more information. You can also test your full FileSet definition by using the \bxlink{\bcommandname{estimate}}{blb:estimate}{console}{command} in the \bconsoleman{}. % the estimate command in the \bconsoleman{}. It is recommended to enclose the string in double quotes. \item [enhancedwild=\byesorno{}] \index[dir]{enhancedwild} \index[dir]{Directive!enhancedwild} The \texttt{Enhanced Wild} directive controls how path name matching with wildcards is done. The default is \texttt{no}, which makes wildcards not match match path separators. If set to \texttt{yes}, an asterisk, question mark, or a bracketed slash will also be matched by a slash. In other words, wildcards will then span path hierarchy. \item [wilddir=\bbracket{string}] \index[dir]{wilddir} \index[dir]{Directive!wilddir} Specifies a wild-card string to be applied to directory names only. No filenames will be matched by this directive. Note, if \textbf{Exclude} is not enabled, the wild-card will select directories to be included. If \textbf{Exclude=yes} is specified, the wild-card will select which directories are to be excluded. Multiple wild-card directives may be specified, and they will be applied in turn until the first one that matches. Note, if you exclude a directory, no files or directories below it will be matched. It is recommended to enclose the string in double quotes. You may want to test your expressions prior to running your backup by using the bwild program. Please see the \bxlink{Utilities}{blb:bwild}{utility}{chapter} of the \butilityman{} for more information. You can also test your full FileSet definition by using the \bxlink{\bcommandname{estimate}}{blb:estimate}{console}{command} in the \bconsoleman{}. % the estimate command in the \bconsoleman{}. An example of excluding with the WildDir option on Win32 machines is presented below. \item [wildfile=\bbracket{string}] \index[dir]{wildfile} \index[dir]{Directive!wildfile} Specifies a wild-card string to be applied to non-directories. That is no directory entries will be matched by this directive. However, note that the match is done against the full path and filename, so your wild-card string must take into account that filenames are preceded by the full path. If \textbf{Exclude} is not enabled, the wild-card will select which files are to be included. If \textbf{Exclude=yes} is specified, the wild-card will select which files are to be excluded. Multiple wild-card directives may be specified, and they will be applied in turn until the first one that matches. It is recommended to enclose the string in double quotes. You may want to test your expressions prior to running your backup by using the bwild program. Please see the \bxlink{Utilities}{blb:bwild}{utility}{chapter} of the \butilityman{} for more information. You can also test your full FileSet definition by using the \bxlink{\bcommandname{estimate}}{blb:estimate}{console}{command} in the \bconsoleman{}. % the estimate command in the \bconsoleman{}. An example of excluding with the WildFile option on Win32 machines is presented below. \item [regex=\bbracket{string}] \index[dir]{regex} \index[dir]{Directive!regex} Specifies a POSIX extended regular expression to be applied to the filenames and directory names, which include the full path. If \textbf{Exclude} is not enabled, the regex will select which files are to be included. If \textbf{Exclude=yes} is specified, the regex will select which files are to be excluded. Multiple regex directives may be specified within an Options resource, and they will be applied in turn until the first one that matches. Note, if you exclude a directory, no files or directories below it will be matched. It is recommended to enclose the string in double quotes. The regex libraries differ from one operating system to another, and in addition, regular expressions are complicated, so you may want to test your expressions prior to running your backup by using the bregex program. Please see the \bxlink{Utilities}{blb:bwild}{utility}{chapter} of the \butilityman{} for more information. You can also test your full FileSet definition by using the \bxlink{\bcommandname{estimate}}{blb:estimate}{console}{command} in the \bconsoleman{}. %% the estimate command in the \bconsoleman{}. You find yourself using a lot of Regex statements, which will cost quite a lot of CPU time, we recommend you simplify them if you can, or better yet convert them to Wild statements which are much more efficient. \item [regexfile=\bbracket{string}] \index[dir]{regexfile} \index[dir]{Directive!regexfile} Specifies a \acs{POSIX} extended regular expression to be applied to non-directories. No directories will be matched by this directive. However, note that the match is done against the full path and filename, so your regex string must take into account that filenames are preceded by the full path. If \textbf{Exclude} is not enabled, the regex will select which files are to be included. If \textbf{Exclude=yes} is specified, the regex will select which files are to be excluded. Multiple regex directives may be specified, and they will be applied in turn until the first one that matches. It is recommended to enclose the string in double quotes. The regex libraries differ from one operating system to another, and in addition, regular expressions are complicated, so you may want to test your expressions prior to running your backup by using the bregex program. Please see the \bxlink{bregex}{blb:bregex}{utility}{command} of the \butilityman{} more. \item [regexdir=\bbracket{string}] \index[dir]{regexdir} \index[dir]{Directive!regexdir} Specifies a POSIX extended regular expression to be applied to directory names only. No filenames will be matched by this directive. Note, if \textbf{Exclude} is not enabled, the regex will select directories files are to be included. If \textbf{Exclude=yes} is specified, the regex will select which files are to be excluded. Multiple regex directives may be specified, and they will be applied in turn until the first one that matches. Note, if you exclude a directory, no files or directories below it will be matched. It is recommended to enclose the string in double quotes. The regex libraries differ from one operating system to another, and in addition, regular expressions are complicated, so you may want to test your expressions prior to running your backup by using the bregex program. Please see the \bxlink{bregex}{blb:bregex}{utility}{command} of the \butilityman{} more. \item [exclude=\byesorno{}] \index[dir]{exclude} \index[dir]{Directive!exclude} The default is \bdefaultvalue{no}. When enabled, any files matched within the Options will be excluded from the backup. \label{blb:ACLSupport} \item [aclsupport=\byesorno{}] \index[dir]{aclsupport} \index[dir]{Directive!aclsupport} The default is \bdefaultvalue{no}. If this option is set to yes, and you have the \acs{POSIX} \textbf{libacl} installed on your Linux system, \bacula{} will backup the file and directory Unix \acp{ACL} as defined in IEEE Std 1003.1e draft 17 and ``POSIX.1e'' (abandoned). This feature is available on Unix systems only and requires the Linux ACL library. \bacula{} is automatically compiled with ACL support if the \textbf{libacl} library is installed on your Linux system (shown in config.out). While restoring the files \bacula{} will try to restore the ACLs, if there is no \ac{ACL} support available on the system, \bacula{} restores the files and directories but not the ACL information. Please note, if you backup an EXT3 or XFS filesystem with \acsp{ACL}, then you restore them to a different filesystem (perhaps reiserfs) that does not have ACLs, the ACLs will be ignored. For other operating systems there is support for either POSIX ACLs or the more extensible NFSv4 ACLs. The ACL stream format between Operation Systems is \textbf{not} compatible so for example an ACL saved on Linux cannot be restored on Solaris. The following Operating Systems are currently supported: \begin{benumerate} \item AIX (pre-5.3 (POSIX) and post 5.3 (POSIX and NFSv4) ACLs) \item Darwin \item FreeBSD (POSIX and NFSv4/ZFS ACLs) \item HPUX \item IRIX \item Linux (POSIX and GPFS) \item Solaris (POSIX and NFSv4/ZFS ACLs) \item Tru64 \end{benumerate} \label{blb:XattrSupport} \item [xattrsupport=\byesorno{}] \index[dir]{xattrsupport} \index[dir]{Directive!xattrsupport} The default is \bdefaultvalue{no}. If this option is set to yes, and your operating system support either so called Extended Attributes or Extensible Attributes \bacula{} will backup the file and directory XATTR data. This feature is available on UNIX only and depends on support of some specific library calls in libc. The XATTR stream format between Operating Systems is \textbf{not} compatible so an XATTR saved on Linux cannot for example be restored on Solaris. On some operating systems ACLs are also stored as Extended Attributes (Linux, Darwin, FreeBSD) \bacula{} checks if you have the aclsupport option enabled and if so will not save the same info when saving extended attribute information. Thus ACLs are only saved once. The following Operating Systems are currently supported: \begin{benumerate} \item AIX (Extended Attributes) \item Darwin (Extended Attributes) \item FreeBSD (Extended Attributes) \item IRIX (Extended Attributes) \item Linux (Extended Attributes) \item NetBSD (Extended Attributes) \item Solaris (Extended Attributes and Extensible Attributes) \item Tru64 (Extended Attributes) \end{benumerate} \item [ignore case=\byesorno{}] \index[dir]{ignore case} \index[dir]{Directive!ignore case} The default is \bdefaultvalue{no}. On Windows systems, you will almost surely want to set this to \textbf{yes}. When this directive is set to \textbf{yes} all the case of character will be ignored in wild-card and regex comparisons. That is an uppercase A will match a lowercase a. \item [fstype=filesystem-type-list] \index[dir]{fstype} \index[dir]{Directive!fstype} This option ensures that the FD, while processing files with this option block effective, will only descend into filesystems of types mentioned in the \texttt{Filesystem-type-list}. This is most useful in combination with \texttt{One~FS~= no} set. This directive may appear multiple times, and all list elements will be added, i.e. a specification of \begin{bVerbatim} FS Type = ext2, xfs FS Type = msdos \end{bVerbatim} will result in all three mentioned file system types being accepted. On Windows, this functionality is not available, but see ``Drive Type'' above (\vref{Director:Fileset:DriveType}). \label{Director:Fileset:DriveType} \item [DriveType=Windows-drive-type] \index[dir]{DriveType} \index[dir]{Directive!DriveType} This option is effective only on Windows machines and is somewhat similar to the Unix/Linux \textbf{fstype} described above, except that it allows you to select what Windows drive types you want to allow. By default all drive types are accepted. The permitted drivetype names are: removable, fixed, remote, cdrom, ramdisk You may have multiple Driveype directives, and thus permit matching of multiple drive types within a single Options resource. If the type specified on the drivetype directive does not match the filesystem for a particular directive, that directory will not be backed up. This directive can be used to prevent backing up non-local filesystems. Normally, when you use this directive, you would also set \textbf{onefs=no} so that \bacula{} will traverse filesystems. This option is not implemented in Unix/Linux systems. \item [hfsplussupport=\byesorno{}] \index[dir]{hfsplussupport} \index[dir]{Directive!hfsplussupport} This option allows you to turn on support for Mac OSX HFS plus finder information. \item [strippath=\bbracket{integer}] \index[dir]{strippath} \index[dir]{Directive!strippath} This option will cause \textbf{integer} paths to be stripped from the front of the full path/filename being backed up. This can be useful if you are migrating data from another vendor or if you have taken a snapshot into some subdirectory. This directive can cause your filenames to be overlayed with regular backup data, so should be used only by experts and with great care. \end{bdescription} \textbf{\bbracket{file-list}} is a list of directory and/or filename names specified with a \textbf{File =} directive. To include names containing spaces, enclose the name between double-quotes. Wild-cards are not interpreted in file-lists. They can only be specified in Options resources. There are a number of special cases when specifying directories and files in a \textbf{file-list}. They are: \begin{bitemize} \item Any name preceded by an at-sign (@) is assumed to be the name of a file, which contains a list of files each preceded by a ``File =''. The named file is read once when the configuration file is parsed during the Director startup. Note, that the file is read on the Director's machine and not on the Client's. In fact, the @filename can appear anywhere within the conf file where a token would be read, and the contents of the named file will be logically inserted in the place of the @filename. What must be in the file depends on the location the @filename is specified in the conf file. For example: \begin{bVerbatim} Include { Options {compression=GZIP } @/home/files/my-files } \end{bVerbatim} \item Any name beginning with a vertical bar (|) is assumed to be the name of a program. This program will be executed on the Director's machine at the time the Job starts (not when the Director reads the configuration file), and any output from that program will be assumed to be a list of files or directories, one per line, to be included. Before submitting the specified command bacula will performe \bilink{character substitution}{blb:character substitution}. This allows you to have a job that, for example, includes all the local partitions even if you change the partitioning by adding a disk. The examples below show you how to do this. However, please note two things: \begin{benumerate} \item if you want the local filesystems, you probably should be using the new \textbf{fstype} directive, which was added in version 1.36.3 and set \textbf{onefs=no}. \item the exact syntax of the command needed in the examples below is very system dependent. For example, on recent Linux systems, you may need to add the -P option, on FreeBSD systems, the options will be different as well. \end{benumerate} In general, you will need to prefix your command or commands with a \btool{sh -c} so that they are invoked by a shell. This will not be the case if you are invoking a script as in the second example below. Also, you must take care to escape (precede with a \textbackslash{}) wild-cards, shell character, and to ensure that any spaces in your command are escaped as well. If you use a single quotes (') within a double quote ("), \bacula{} will treat everything between the single quotes as one field so it will not be necessary to escape the spaces. In general, getting all the quotes and escapes correct is a real pain as you can see by the next example. As a consequence, it is often easier to put everything in a file and simply use the file name within \bacula{}. In that case the \btool{sh -c} will not be necessary providing the first line of the file is \textbf{\#!/bin/sh}. As an example: \begin{bVerbatim} Include { Options {signature = SHA1 } File = "|sh -c 'df -l | grep \"^/dev/hd[ab]\" | grep -v \".*/tmp\" \ | awk \"{print \\$6}\"'" } \end{bVerbatim} will produce a list of all the local partitions on a Red Hat Linux system. Note, the above line was split, but should normally be written on one line. Quoting is a real problem because you must quote for \bacula{} which consists of preceding every \textbackslash{} and every " with a \textbackslash{}, and you must also quote for the shell command. In the end, it is probably easier just to execute a small file with: \begin{bVerbatim} Include { Options { signature=MD5 } File = "|my_partitions" } \end{bVerbatim} where my\_partitions has: \begin{bVerbatim} #!/bin/sh df -l | grep "^/dev/hd[ab]" | grep -v ".*/tmp" \ | awk "{print \$6}" \end{bVerbatim} If the vertical bar (\vb{}) in front of my\_partitions is preceded by a backslash as in \verb:\|:, the program will be executed on the Client's machine instead of on the Director's machine. Please note that if the filename is given within quotes, you will need to use two slashes. An example, provided by John Donagher, that backs up all the local UFS partitions on a remote system is: \begin{bVerbatim} FileSet { Name = "All local partitions" Include { Options {signature=SHA1; onefs=yes; } File = "\\|bash -c \"df -klF ufs | tail +2 | awk '{print \$6}'\"" } } \end{bVerbatim} The above requires two backslash characters after the double quote (one preserves the next one). If you are a Linux user, just change the \textbf{ufs} to \textbf{ext3} (or your preferred filesystem type), and you will be in business. If you know what filesystems you have mounted on your system, e.g. for Red Hat Linux normally only ext2 and ext3, you can backup all local filesystems using something like: \begin{bVerbatim} Include { Options {signature = SHA1; onfs=no; fstype=ext2 } File = / } \end{bVerbatim} On Windows, the command is executed with ``cmd /c'' prefix, and it is recommended to keep the path of the scripts as simple as possible (without spaces for example). The exact escaping sequence of the string can be difficult to determine. \item Any file-list item preceded by a less-than sign (\lt{}) will be taken to be a file. This file will be read on the Director's machine (see below for doing it on the Client machine) at the time the Job starts, and the data will be assumed to be a list of directories or files, one per line, to be included. The names should start in column 1 and should not be quoted even if they contain spaces. This feature allows you to modify the external file and change what will be saved without stopping and restarting \bacula{} as would be necessary if using the @ modifier noted above. For example: \begin{bVerbatim} Include { Options {signature = SHA1 } File = " listing client= fileset=Test \end{bVerbatim} to give you a listing of all files that match. In the above example, it should be only files with names ending in \bfilename{.c}.