Debian Codegraph @

manuals/en/main/oldbsfeatures.tex

raw
1 \chapter{New Features in Older Bacula Enterprise Versions}
2 \label{blb:chapter:newfeaturesolderbacula}
3 This chapter presents some of the new features that have been added to the
4 older Enterprise versions of \bacula{}. These features are available only
5 with a \baculasystems{} subscription.
6
7 \section{\baculaenterprise{} 8.8}
8 %% \subsection{Native S3 Cloud Support}
9 %% Cloud storage has been around for a number of years, but until
10 %% recently, it was not very interesting for backup because of the lack of
11 %% bandwidth for uploading and downloading the large volumes of data that
12 %% today's backups use. With the advent of fibre optics and high-speed copper
13 %% Internet connections, the Cloud has become a viable and cost effective
14 %% solution for data backup.
15 %% \smallskip
16
17 %% \baculasystems{} has implemented in \baculaenterprise{} 8.8.0 drivers for
18 %% backup and restore to and from any S3 compatible cloud whether public
19 %% or private. The architecture of the \baculaenterprise{} cloud backup is new,
20 %% and it will provide the user with an array of features to keep
21 %% the cloud costs to a minimum and the performance to a maximum.
22 %% \smallskip
23
24 %% To provide this new Cloud functionality we have made significant changes
25 %% to the Storage daemon backend device architecture. The new \bacula{} ersion
26 %% is more modular and permits the devices to select from multiple
27 %% drivers to handle different Cloud technologies. For the first release we
28 %% have included an S3 driver that is compatible with a large number of S3
29 %% cloud services. There is also a file driver that is useful for testing
30 %% the Cloud architecture, and could possibly be useful for a disk media
31 %% device that is very slow.
32
33 \subsubsection{Cloud Backup}
34 A major problem of Cloud backup is that data transmission to and from the
35 Cloud is very slow compared to traditional backup to disk or tape.
36 The \bacula{} Cloud drivers provide a means to quickly finish the backups
37 and then to transfer the data from the local cache to the Cloud in the
38 background. This is done by first splitting the data Volumes into small
39 parts that are cached locally, and then uploading those parts to the Cloud
40 storage service in the background, either while the job continues to run or
41 after the backup job has terminated. Once the parts are written to the
42 Cloud, they may either be left in the local cache for quick restores or
43 they can be removed (truncate cache).
44
45 \subsubsection{Cloud Volume Architecture}
46
47 \bimageH[0.8\linewidth]{nativeCloudStorage-diagram}{\bacula{} Cloud Architecture}{fig:cloud0}
48
49
50 The picture \vref{fig:cloud0} shows two Volumes (Volume0001 and Volume0002) with
51 their parts in the cache. Below the cache, one can see that Volume0002 has be uploaded or synchronized with the Cloud.
52
53 \paragraph{Note}: Normal \bacula{} disk Volumes are implemented as standard files that reside in the user-defined \bdirectivename{Archive Directory}. On the other hand, \bacula{} Cloud Volumes are directories that reside in the user-defined \bdirectivename{Archive Directory}. The directory contains the cloud Volume parts, implemented as numbered files (\bfilename{part.1}, \bfilename{part.2}, \ldots{}).
54
55 \subsubsection{Cloud Restore}
56 During a restore, if the needed parts are available in the local cache, they will
57 immediately be used. Otherwise, they will be downloaded from cloud storage as
58 needed. The restore starts with parts already in the local cache but
59 will wait as needed for any part that must be downloaded. The download
60 proceeds while the restore is running.
61 \smallskip
62
63 With most cloud providers uploads are free of charge, but
64 downloads of data from the cloud are billed. By using the local cache and
65 multiple small parts, Bacula can be configured to substantially reduce
66 download costs.
67 \smallskip
68
69 The \bdirectivename{Maximum File Size} Device directive is valid within the Storage
70 Daemon's cloud device configuration and defines the granularity of a restore chunk. In order to minimize the number of volume
71 parts to download during a restore (in particular when restoring single files), it
72 is useful to set the \bdirectivename{Maximum File Size} to a value smaller than or
73 equal to the configured \bdirectivename{Maximum Part Size}.
74
75 \subsubsection{Compatibility}
76 Since a Cloud Volume contains the same data an ordinary \bacula{} Volume does, all existing types of \bacula{} data may be stored in the cloud -- that is client data encryption, client-side compreion, plugin usage are all available. In fact, all existing \bacula{} functionality, with the exception of deduplication, is compatible with the \bacula{} Cloud drivers.
77
78 \subsubsection{Deduplication and the Cloud}
79 At the current time, \bacula{} Global Endpoint Backup does not support
80 writing to the cloud because cloud storage would be too slow to support large
81 hashed and indexed containers of deduplication data.
82
83
84 \subsubsection{Virtual Autochangers and Disk Autochangers}
85 \bacula{} Virtual Autochangers are compatible with
86 the \bacula{} Cloud drivers. However, if you use a third party disk
87 autochanger script such as \textbf{Vchanger}, unless or until it is
88 modified to handle Volume directories, it may not be compatible with \bacula{}
89 Cloud drivers.
90
91 \subsubsection{Security}
92 All data that is sent to and received from the cloud by default uses
93 the \acs{HTTPS} protocol, so data is encrypted while being transmitted
94 and received. However, data that resides in the cloud is not encrypted by
95 default. If that extra security of backed up data is required,
96 \bacula{}'s \acs{PKI} data encryption feature should be used
97 during the backup.
98
99 \subsection{New Commands, Resource, and Directives for Cloud}
100 To support Cloud storage devices, some new \bconsolename{} commands,
101 new Storage Daemon directives, and a new Cloud resource that is
102 referenced in the Storage Daemon's Device resource are available as of
103 \baculaenterprise{} 8.8
104
105 \subsubsection{Cache and Pruning}
106
107 The Cache is treated much like a normal Disk based backup, so that in
108 configuring Cloud the administrator should take care to set \bdirectivename{Archive Device} in
109 the \bDevice{} resource to a dirctory that would also be suitable for storing backup data.
110 Obviously, unless the \bcommandname{truncate}/\bcommandname{prune} cache
111 commands are used, the \bdirectivename{Archive Device} will continue to fill.
112 \smallskip
113
114 The cache retention can be controlled per Volume with the \bdirectivename{Cache Retention}
115 attribute. The default value is \bdefaultvalue{0}, meaning that pruning of the cache is
116 disabled.
117 \smallskip
118
119 The \bdirectivename{Cache Retention} value for a volume can be modified with the \bcommandname{update}
120 command, or configured via the \bPool{} directive \bdirectivename{Cache Retention} for newly created volumes.
121
122
123 \subsubsection{New Cloud \bconsole{} Commands}
124 \begin{bitemize}
125 \item \bcommandname{truncate cache}
126 \item \bcommandname{upload}
127 \item \bcommandname{cloud} The new cloud \bconsole{} command allows inspecting and manipulating cloud volumes in different ways. The options are the following:
128 \begin{bitemize}
129 \item None. If you specify no arguments to the command, \bconsole{} will prompt with:
130 \begin{bVerbatim}
131 Cloud choice:
132 1: List Cloud Volumes in the Cloud
133 2: Upload a Volume to the Cloud
134 3: Prune the Cloud Cache
135 4: Truncate a Volume Cache
136 5: Done
137 Select action to perform on Cloud (1-5):
138 \end{bVerbatim}
139 The different choices should be rather obvious.
140
141 \item \textbf{truncate} This command will attempt to truncate the local cache for the specified Volume. \bacula{} will prompt you for the information needed to determine the Volume name or names. To avoid the prompts, the following additional command line options may be specified:
142 \begin{bitemize}
143 \item {Storage=xxx}
144 \item {Volume=xxx}
145 \item {AllPools}
146 \item {AllFromPool}
147 \item {Pool=xxx}
148 \item {Storage=xxx}
149 \item {MediaType=xxx}
150 \item {Drive=xxx}
151 \item {Slots=nnn}
152 \end{bitemize}
153 \item \bcommandname{prune} This command will attempt to prune the local cache for
154 the specified Volume. \bacula{} will respect the \bdirectivename{Cache Retention} volume
155 attribute to determine if the cache can be truncated or not. Only parts
156 that are uploaded to the cloud will be deleted from the cache. \bacula{}
157 will prompt you for the information needed to determine the Volume name or
158 names. To avoid the prompts, the following additional command line
159 options may be specified:
160 \begin{bitemize}
161 \item {Storage=xxx}
162 \item {Volume=xxx}
163 \item {AllPools}
164 \item {AllFromPool}
165 \item {Pool=xxx}
166 \item {Storage=xxx}
167 \item {MediaType=xxx}
168 \item {Drive=xxx}
169 \item {Slots=nnn}
170 \end{bitemize}
171 \item \bcommandname{upload} This command will attempt to upload the specified Volumes. It will prompt for the information needed to determine the Volume name or names. To avoid the prompts, any of the following additional command line options can be specified:
172 \begin{bitemize}
173 \item {Storage=xxx}
174 \item {Volume=xxx}
175 \item {AllPools}
176 \item \textbf{AllFromPool}
177 \item {Storage=xxx}
178 \item {Pool=xxx}
179 \item {MediaType=xxx}
180 \item {Drive=xxx}
181 \item {Slots=nnn}
182 \end{bitemize}
183 \item \bcommandname{list} This command will list volumes stored in the Cloud. If a
184 volume name is specified, the command will list all parts for the given volume.
185 To avoid the prompts, the operator may specify any of the following additional command line options:
186 \begin{bitemize}
187 \item {Storage=xxx}
188 \item {Volume=xxx}
189 \item {Storage=xxx}
190 \end{bitemize}
191 \end{bitemize}
192 \end{bitemize}
193
194 \subsubsection{Cloud Additions to the DIR \bPool{} Resource}
195 In \bfilename{bacula-dir.conf} \bresourcename{Pool} resources,
196 the directive \bkeyword{Cache Retention} can be specified. It is only effective for cloud storage backed volumes, and ignored when used with volumes stored on any different storage device.
197
198
199 \subsubsection{Cloud Additions to the SD Device Resource}
200 \bDevice{} resource configured in the \bfilename{bacula-sd.conf} file can use
201 the \bkeyword{Cloud} keyword in the \bdirectivename{Device Type} directive,
202 and the two directives \bdirectivename{Maximum Part Size} and \bdirectivename{Cloud}.
203 %%% duplicated information here, but not really bad... I leave it as is, for now. ARL
204
205 \subsubsection{New Cloud SD Device Directives}
206 \begin{bdescription}
207 \item [Device Type] The Device Type has been extended to include the new keyword \bkeyword{Cloud} to specify that the device supports cloud Volumes. Example:
208 \begin{bVerbatim}
209 Device Type = Cloud
210 \end{bVerbatim}
211 \item [Cloud] The new Cloud directive references a \bresourcename{Cloud} Resource. As with other \bacula{} resource references, the name of the \bresourcename{Cloud} resource is used as the value. Example:
212 \begin{bVerbatim}
213 Cloud = S3Cloud
214 \end{bVerbatim}
215 \item [Maximum Part Size] This directive allows specification of the maximum size for each part of any volume written by the current device. Smaller part sizes will reduce restore costs, but will cause additional but small overhead to handle multiple parts.
216 The maximum number of parts permitted per Cloud Volume is~524,288. The maximum size of any given part is approximately~17.5\,TB.
217 \end{bdescription}
218
219 \subsubsection{Example Cloud Device Specification}
220 An example of a Cloud Device Resource might be:
221
222 \begin{bVerbatim}
223 Device {
224 Name = CloudStorage
225 Device Type = Cloud
226 Cloud = S3Cloud
227 Archive Device = /opt/bacula/backups
228 Maximum Part Size = 10000000
229 Media Type = File
230 LabelMedia = yes
231 Random Access = Yes;
232 AutomaticMount = yes
233 RemovableMedia = no
234 AlwaysOpen = no
235 }
236 \end{bVerbatim}
237
238 As can be seen above, the \bdirectivename{Cloud} directive in the \bresourcename{Device} resource contains the name (\textbf{S3Cloud}), which references the \bresourcename{Cloud} resource that is shown below.
239 \smallskip
240
241 Note also that the \bdirectivename{Archive Device} is specified in the
242 same manner as used for a File device, i.\,e. by indicating a
243 directory name. However, in place of containing regular files as
244 Volumes, the archive device for the Cloud drivers will contain the
245 local cache, which consists a directory per Volume, and
246 these directories contain the parts associated with the particular
247 Volume. So with the above \bresourcename{Device} resource, and the
248 two cached Volumes shown in figure \vref{fig:cloud0} above,
249 the following layout on disk would result:
250
251 \begin{bVerbatim}
252 /opt/bacula/backups
253 /opt/bacula/backups/Volume0001
254 /opt/bacula/backups/Volume0001/part.1
255 /opt/bacula/backups/Volume0001/part.2
256 /opt/bacula/backups/Volume0001/part.3
257 /opt/bacula/backups/Volume0001/part.4
258 /opt/bacula/backups/Volume0002
259 /opt/bacula/backups/Volume0002/part.1
260 /opt/bacula/backups/Volume0002/part.2
261 /opt/bacula/backups/Volume0002/part.3
262 \end{bVerbatim}
263
264 % A good part of this section is also in storedconf.tex and the whitepaper
265 % Please, keep everything synchronized
266 % Probably a good idea to ensure we have only one location of the
267 % relevant documentation... for now, I just don't touch the below
268 % part. ARL
269 \subsubsection{The Cloud Resource}
270 The \bresourcename{Cloud} resource has a number of directives that may be specified as exemplified in the following example:
271
272 Default east USA location:
273 \begin{bVerbatim}
274 Cloud {
275 Name = S3Cloud
276 Driver = "S3"
277 HostName = "s3.amazonaws.com"
278 BucketName = "BaculaVolumes"
279 AccessKey = "BZIXAIS39DP9YNER5DFZ"
280 SecretKey = "beesheeg7iTe0Gaexee7aedie4aWohfuewohGaa0"
281 Protocol = HTTPS
282 URIStyle = VirtualHost
283 Truncate Cache = No
284 Upload = EachPart
285 Region = ``us-east-1"
286 Maximum Upload Bandwidth = 5MB/s
287 }
288 \end{bVerbatim}
289
290 For central europe location:
291 \begin{bVerbatim}
292 Cloud {
293 Name = S3Cloud
294 Driver = "S3"
295 HostName = "s3-eu-central-1.amazonaws.com"
296 BucketName = "BaculaVolumes"
297 AccessKey = "BZIXAIS39DP9YNER5DFZ"
298 SecretKey = "beesheeg7iTe0Gaexee7aedie4aWohfuewohGaa0"
299 Protocol = HTTPS
300 UriStyle = VirtualHost
301 Truncate Cache = No
302 Upload = EachPart
303 Region = "eu-central-1"
304 Maximum Upload Bandwidth = 4MB/s
305 }
306 \end{bVerbatim}
307
308 For Amazon Cloud, refer to \url{http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region}
309 to get a complete list of regions and corresponding endpoints and use them respectively as
310 \bdirectivename{Region} and \bdirectivename{HostName} directives.
311
312 or in the following example for CEPH S3 interface:
313 \begin{bVerbatim}
314 Cloud {
315 Name = CEPH_S3
316 Driver = "S3"
317 HostName = ceph.mydomain.lan
318 BucketName = "CEPHBucket"
319 AccessKey = "xxxXXXxxxx"
320 SecretKey = "xxheeg7iTe0Gaexee7aedie4aWohfuewohxx0"
321 Protocol = HTTPS
322 Upload = EachPart
323 UriStyle = Path # Must be set for CEPH
324 }
325 \end{bVerbatim}
326
327 For Azure:
328 \begin{bVerbatim}
329 Cloud {
330 Name = MyCloud
331 Driver = "Azure"
332 HostName = "MyCloud" #not used but needs to be specified
333 BucketName = "baculaAzureContainerName"
334 AccessKey = "baculaaccess"
335 SecretKey = "/Csw1SECRETUmZkfQ=="
336 Protocol = HTTPS
337 UriStyle = Path
338 }
339 \end{bVerbatim}
340
341 The directives of the above \bresourcename{Cloud} resource for the S3 driver are defined as follows:
342
343 \begin{bdescription}
344
345 \bwebdoc{Storage:Cloud:Name}
346 \item [Name = \bbracket{Device-Name}]
347 \index[sd]{Name}
348 \index[sd]{Directive!Name}
349 The name of the Cloud resource. This is the logical Cloud name, and may be
350 any string up to 127 characters in length. Shown as \textbf{S3Cloud} above.
351
352 \bwebdoc{Storage:Cloud:Description}
353 \item [Description = \bbracket{Text}]
354 The description is used for display purposes as is the case with all resources.
355
356 \bwebdoc{Storage:Cloud:Driver}
357 \item [Driver = \bbracket{Driver-Name}]
358 \index[sd]{Driver}
359
360 This defines which driver to use. At the moment, the only Cloud driver that
361 is implemented is \bvalue{S3}. There is also a \bvalue{File} driver, which is
362 used mostly for testing.
363
364
365 \bwebdoc{Storage:Cloud:HostName}
366 \item [Host Name = \bbracket{Name}]
367 \index[sd]{HostName}
368 This directive specifies the hostname to be used in the \acs{URL}. Each Cloud
369 service provider has a different and unique hostname. The maximum size is
370 \bvalue{255} characters and may contain a \acs{TCP} port specification.
371
372 \bwebdoc{Storage:Cloud:BucketName}
373 \item [Bucket Name = \bbracket{Name}]
374 \index[sd]{BucketName}
375
376 This directive specifies the bucket name that you wish to use on the Cloud
377 service. This name is normally a unique name that identifies where
378 you want to place your Cloud Volume parts.
379 With Amazon S3, the bucket must be created previously on the Cloud service.
380 With Azure Storage, it is generaly refered as Container and it can be created
381 automatically by Bacula when it does not exist.
382 The maximum bucket name size is \bvalue{255} characters.
383
384 \bwebdoc{Storage:Cloud:AccessKey}
385 \item [Access Key = \bbracket{String}]
386 \index[sd]{AccessKey}
387
388 The access key is your unique user identifier given to you by your cloud
389 service provider.
390
391 \bwebdoc{Storage:Cloud:SecretKey}
392 \item [Secret Key = \bbracket{String}]
393 \index[sd]{SecretKey}
394 The secret key is the security key that was given to you by your cloud
395 service provider. It is equivalent to a password.
396
397 \bwebdoc{Storage:Cloud:Protocol}
398 \item [Protocol = \bbracket{HTTP | HTTPS}]
399 \index[sd]{Protocol}
400
401 The protocol defines the communications protocol to use with the cloud
402 service provider. The two protocols currently supported are: \acs{HTTPS}
403 and \acs{HTTP}. The default is \acs{HTTPS}.
404
405 \bwebdoc{Storage:Cloud:UriStyle}
406 \item [Uri Style = \bbracket{VirtualHost | Path}]
407 \index[sd]{UriStyle}
408
409 This directive specifies the \acs{URI} style to use to communicate with the cloud
410 service provider. The two \bdirectivename{Uri Styles} currently supported are:
411 \bvalue{VirtualHost} and \bvalue{Path}. The default is \bdefaultvalue{VirtualHost}.
412
413 \bwebdoc{Storage:Cloud:TruncateCache}
414 \item [Truncate Cache = \bbracket{truncate-kw}]
415 \index[sd]{TruncateCache}
416
417 This directive specifies when \bacula{} should automatically remove (truncate)
418 the local cache parts. Local cache parts can only be removed if they have
419 been uploaded to the cloud. The currently implemented values are:
420 \begin{bdescription}
421 \item [ No ] Do not remove cache. With this option you must manually delete the cache parts with a \bconsolename{} \bcommandname{truncate cache}, or do so with an \textbf{Admin} Job that runs an \bcommandname{truncate cache} command. This is the default.
422 \item [ AfterUpload ] Each part will be removed just after it is uploaded. Note, if this option is specified, all restores will require a download from the Cloud\footnote{Not yet implemented}.
423 \item [ AtEndOfJob ] With this option, at the end of the Job, every part that has been uploaded to the Cloud will be removed\footnote{Not yet implemented} (truncated).
424 \end{bdescription}
425
426 \bwebdoc{Storage:Cloud:Upload}
427 \item [Upload = \bbracket{upload-kw}]
428
429 This directive specifies when local cache parts
430 will be uploaded to the Cloud. The options are:
431
432 \begin{bdescription}
433 \item [No] Do not upload cache parts. With this option you must manually upload the cache parts with a \bconsole{} \bcommandname{upload} command, or do so with an \textbf{Admin} Job that runs an \bcommandname{upload} command. This is the default.
434 \item [EachPart] With this option, each part will be uploaded when it is complete i.e. when the next part is created or at the end of the Job.
435 \item [AtEndOfJob] With this option all parts that have not been previously uploaded will be uploaded at the end of the Job.\footnote{Not yet implemented}
436 \end{bdescription}
437 \bwebdoc{Storage:Cloud:MaximumConcurrentUploads}
438 \item [Maximum Concurrent Uploads = \bbracket{number}]
439 The default is \bdefaultvalue{3}, but by using this directive, you may set it to any value you want.
440
441 \bwebdoc{Storage:Cloud:MaximumConcurrentDownloads}
442 \item [Maximum Concurrent Downloads = \bbracket{number}]
443 The default is \bdefaultvalue{3}, but by using this directive, you may set it to any value
444 you want.
445
446 \bwebdoc{Storage:Cloud:MaximumUploadBandwidth}
447 \item [Maximum Upload Bandwidth = \bbracket{speed}]
448
449 The default is \bdefaultvalue{unlimited}, but by using this directive, you may limit the
450 upload bandwidth used globally by all devices referencing this \bresourcename{Cloud}
451 resource.
452
453 \bwebdoc{Storage:Cloud:MaximumDownloadBandwidth}
454 \item [Maximum Download Bandwidth = \bbracket{speed}]
455
456 The default is \bdefaultvalue{unlimited}, but by using this directive, you may limit the
457 download bandwidth used globally by all devices referencing this Cloud
458 resource.
459
460 \bwebdoc{Storage:Cloud:Region}
461 \item [Region = \bbracket{String}]
462 The Cloud resource can be configured to use a specific endpoint within a
463 region. This directive is required for AWS-V4 regions. ex:
464 \bvalueddirective{Region}{"eu-central-1"}
465
466 \bwebdoc{Storage:Cloud:BlobEndpoint}
467 \item [BlobEndpoint = \bbracket{String}]
468 This resource can be used to specify a custom \acs{URL}
469 for Azure Blob (see \url{https://docs.microsoft.com/en-us/azure/storage/blobs/storage-custom-domain-name}).
470
471 \bwebdoc{Storage:Cloud:EndpointSuffix}
472 \item [EndpointSuffix = \bbracket{String}]
473 Use this resource to specify a custom \acs{URL} postfix for Azure. ex:
474 \bvalueddirective{EnbpointSuffix}{"core.chinacloudapi.cn"}
475 \end{bdescription}
476
477 \subsubsection{File Driver for the Cloud}
478 As mentioned above, one may specify the keyword \bkeyword{File} on the \bdirectivename{Driver} directive of the Cloud resource. Instead of writing to the Cloud, \bacula{} will instead create a Cloud Volume but write it to disk. The rest of this section applies to the \bresourcename{Cloud} resource directives when the File driver is specified.
479 \smallskip
480
481 The following Cloud directives are ignored: \bdirectivename{Bucket Name}, \bdirectivename{Access Key}, \bdirectivename{Secret Key}, \bdirectivename{Protocol}, \bdirectivename{URI Style}.
482 The directives \bdirectivename{Truncate Cache} and \bdirectivename{Upload} work on the local cache in the same manner as they do for the S3 driver.
483 \smallskip
484
485 The main difference to note is that the \bdirectivename{Host Name}, specifies the destination directory for the Cloud Volume files, and this \bdirectivename{Host Name} must be different from the \bdirectivename{Archive Device} name, or there will be a conflict between the local cache (in the Archive Device directory) and the destination Cloud Volumes (in the Host Name directory).
486 \smallskip
487
488 As noted above, the File driver is mostly used for testing purposes, and we do not particularly recommend using it. However, if you have a particularly slow backup device you might want to stage your backup data into an \acs{SSD} or disk using the local cache feature of the Cloud device, and have your Volumes transferred in the background to a slow File device.
489
490
491 %%% ARL: cloud parts above not touched--for now :-)
492
493 \subsection{Progressive Virtual Full}
494 Instead of the implementation of Perpetual Virtual Full backups with a
495 Perl script which needs to be run regularly, with \baculaenterprise{}
496 version~8.8.0, a new job directive named \bdirectivename{Backups To Keep}
497 has been added. This permits implementation of Progressive
498 Virtual Fulls fully within \bacula{} itself.
499
500
501 \bimageH[.8\linewidth]{pvf-slidingbackups}{Backup Sequence Slides Forward One Day, Each Day}{fig:slidingbackups}
502 \smallskip
503
504 To use the Progressive Virtual Full feature, the
505 \bdirectivename{Backups To Keep} directive is added to a \bJob{}
506 resource. The value specified for the directive indicates the number of backup jobs that should not be merged into the Virtual Full. The default is zero and behaves the same way the prior script \btool{pvf} worked.
507
508 \subsection{Backups To Keep Directive}
509 The new \bdirectivename{BackupsToKeep} directive is specified in the \bJob{} Resource and has the form:
510
511 \begin{bVerbatim}
512 Backups To Keep = 30
513 \end{bVerbatim}
514
515 where the value (\bvalue{30} in the figure \vref{fig:slidingbackups} )
516 is the number of backups to retain. When this directive is present
517 during a Virtual Full (it is ignored for any other Job types),
518 \bacula{} will check if the latest Full backup has more subsequent
519 backups than the value specified. In the above example, the Job would
520 simply terminate unless there is a Full back followed by at least 31
521 backups of either Differential or Incremental level.
522 \smallskip
523
524 Assuming that the latest Full backup is followed by 32 Incremental
525 backups, a Virtual Full will be run that consolidates the Full with
526 the first two Incrementals that were run after the Full backup. The
527 result is a Full backup followed by 30 Incremental ones. The \bJob{}
528 Resource in \bfilename{bacula-dir.conf} to accomplish this would be:
529
530 \begin{bVerbatim}
531 Job {
532 Name = "VFull"
533 Type = Backup
534 Level = VirtualFull
535 Client = "my-fd"
536 File Set = "FullSet"
537 Accurate = Yes
538 Backups To Keep = 10
539 }
540 \end{bVerbatim}
541
542 \subsection{Delete Consolidated Jobs}
543 The additional directive \bdirectivename{Delete Consolidated Jobs}
544 expects a \byesorno{} value that, if set to \bvalue{yes}, will cause
545 any old Job that is consolidated during a Virtual Full to be
546 deleted. In the example above we saw that a Full plus one other job
547 (either an Incremental or Differential) were consolidated into a new
548 Full backup. The original Full and the other Job consolidated would be
549 deleted if this directive were set to \bvalue{yes}. The default value
550 is \bdefaultvalue{no}.
551
552
553 \subsection{Virtual Full Compatibility}
554 Virtual Full as well as Progressive Virtual Full backups work with any standard backup Job including Jobs that use the Global Endpoint Deduplication.
555 \smallskip
556
557 However, it should be noted that Virtual Full jobs are \textbf{not compatible} with
558 Windows backups using \acs{VSS} writers (mostly plugins),
559 nor are they compatible with a number of non-Windows \baculasystems{} plugins.
560 Please contact \baculasystems{} Support team for more details Virtual Full
561 compatibility.
562
563 \subsection{TapeAlert Enhancements}
564 There are some significant enhancements to the TapeAlert feature of \bacula{}. Several directives are used slightly differently, and there is a minor compatibility problem with the old TapeAlert implementation.
565
566 \subsection{What is New}
567 First, the \bdirectivename{Alert Command} directive needs to be added
568 in the Device resource that calls the new \btool{tapealert} script
569 that is installed in the scripts directory (normally:
570 \bdirectoryname{/opt/bacula/scripts}):
571
572 \begin{bVerbatim}
573 Device {
574 Name = ...
575 Archive Device = /dev/nst0
576 Alert Command = "/opt/bacula/scripts/tapealert %l"
577 Control Device = /dev/sg1 # must be SCSI ctl for Archive Device
578 ...
579 }
580 \end{bVerbatim}
581
582 The \bdirectivename{Control Device} directive in the Storage Daemon's configuration was previously used only for the \acs{SAN} Shared Storage feature. With \bacula{} version~8.8, it is also used for the TapeAlert command to permit \bacula{} to detect tape alerts on a specific device (normally only tape devices).
583
584 Once the above mentioned two directives (\bdirectivename{Alert Command} and \bdirectivename{Control Device}) are in place in all \bDevice{} resources, \bacula{} will check for tape alerts at two points:
585 \begin{bitemize}
586 \item After the Drive is used and it becomes idle.
587 \item After each read or write error on the drive.
588 \end{bitemize}
589
590 At each of the above times, \bacula{} will call the new \btool{tapealert} script, which uses the \btool{tapeinfo} program. The \btool{tapeinfo} utility is part of the \btool{apt} \bfilename{sg3-utils} and \btool{rpm} \bfilename{sg3\_utils} packages. Then for each tape alert that \bacula{} finds for that drive, it will emit a Job message that is either INFO, WARNING, or FATAL depending on the designation in the Tape Alert published by the \bfootref{T10 Technical Committee on \acs{SCSI} Storage Interfaces}{https://www.t10.org}. For the specification, please see: \url{http://www.t10.org/ftp/t10/document.02/02-142r0.pdf}
591 \smallskip
592
593 As a somewhat extreme example, if tape alerts~3, 5, and~39 are set, you will get the following output in your backup job:
594
595 \begin{bVerbatim}
596 17-Nov 13:37 rufus-sd JobId 1: Error: block.c:287
597 Write error at 0:17 on device "tape"
598 (/home/kern/bacula/k/regress/working/ach/drive0)
599 Vol=TestVolume001. ERR=Input/output error.
600
601 17-Nov 13:37 rufus-sd JobId 1: Fatal error: Alert:
602 Volume="TestVolume001" alert=3: ERR=The operation has stopped because
603 an error has occurred while reading or writing data which the drive
604 cannot correct. The drive had a hard read or write error
605
606 17-Nov 13:37 rufus-sd JobId 1: Fatal error: Alert:
607 Volume="TestVolume001" alert=5: ERR=The tape is damaged or the drive
608 is faulty. Call the tape drive supplier helpline. The drive can no
609 longer read data from the tape
610
611 17-Nov 13:37 rufus-sd JobId 1: Warning: Disabled Device "tape"
612 (/home/kern/bacula/k/regress/working/ach/drive0) due to tape alert=39.
613
614 17-Nov 13:37 rufus-sd JobId 1: Warning: Alert: Volume="TestVolume001"
615 alert=39: ERR=The tape drive may have a fault. Check for availability
616 of diagnostic information and run extended diagnostics if applicable.
617 The drive may have had a failure which may be identified by stored
618 diagnostic information or by running extended diagnostics (eg Send
619 Diagnostic). Check the tape drive users manual for instructions on
620 running extended diagnostic tests and retrieving diagnostic data.
621 \end{bVerbatim}
622
623 Without the tape alert feature enabled, you would only get the first error message above, which is the error \bacula{} received. Notice also, in this case the alert number 5 is a critical error, which causes two things to happen: First, the tape drive is disabled, and second, the Job is failed.
624 \smallskip
625
626 If you attempt to run another Job using the Device that has been disabled, you will get a message similar to the following:
627 \begin{bVerbatim}
628 17-Nov 15:08 rufus-sd JobId 2: Warning:
629 Device "tape" requested by DIR is disabled.
630 \end{bVerbatim}
631
632 and the Job may be failed if no other usable drive can be found.
633 \smallskip
634
635 Once the problem with the tape drive has been corrected, you can clear the tape alerts and re-enable the device with the \bacula{} \bconsole{} command such as the following:
636
637 \begin{bVerbatim}
638 enable Storage=Tape
639 \end{bVerbatim}
640
641 Note, when you enable the device, the list of prior tape alerts for that drive will be discarded.
642 \smallskip
643
644 Since is is possible to miss tape alerts, \bacula{} maintains a temporary list of the last 8~alerts, and each time \bacula{} calls the \btool{tapealert} script, it will keep up to 10~alert status codes. Normally there will only be one or two alert errors for each call to the \btool{tapealert} script.
645 \smallskip
646
647 Once a drive has one or more tape alerts, they can be inspected by using the \bconsole{} status command as follows:
648 \begin{bVerbatim}
649 status storage=Tape
650 \end{bVerbatim}
651 which produces the following output:
652 \begin{bVerbatim}
653 Device Vtape is "tape" (/home/kern/bacula/k/regress/working/ach/drive0)
654 mounted with:
655 Volume: TestVolume001
656 Pool: Default
657 Media type: tape
658 Device is disabled. User command.
659 Total Bytes Read=0 Blocks Read=1 Bytes/block=0
660 Positioned at File=1 Block=0
661 Critical Alert: at 17-Nov-2016 15:08:01 Volume="TestVolume001"
662 alert=Hard Error
663 Critical Alert: at 17-Nov-2016 15:08:01 Volume="TestVolume001"
664 alert=Read Failure
665 Warning Alert: at 17-Nov-2016 15:08:01 Volume="TestVolume001"
666 alert=Diagnostics Required
667 \end{bVerbatim}
668 If you want to see the long message associated with each of the alerts, simply set the debug level to 10 or more and re-issue the status command:
669 \begin{bVerbatim}
670 setdebug storage=Tape level=10
671 status storage=Tape
672 \end{bVerbatim}
673 \begin{bVerbatim}
674 ...
675 Critical Alert: at 17-Nov-2016 15:08:01 Volume="TestVolume001"
676 flags=0x0 alert=The operation has stopped because an error has occurred
677 while reading or writing data which the drive cannot correct. The drive had
678 a hard read or write error
679 Critical Alert: at 17-Nov-2016 15:08:01 Volume="TestVolume001"
680 flags=0x0 alert=The tape is damaged or the drive is faulty. Call the tape
681 drive supplier helpline. The drive can no longer read data from the tape
682 Warning Alert: at 17-Nov-2016 15:08:01 Volume="TestVolume001" flags=0x1
683 alert=The tape drive may have a fault. Check for availability of diagnostic
684 information and run extended diagnostics if applicable. The drive may
685 have had a failure which may be identified by stored diagnostic information
686 or by running extended diagnostics (eg Send Diagnostic). Check the tape
687 drive users manual for instructions on running extended diagnostic tests
688 and retrieving diagnostic data.
689 ...
690 \end{bVerbatim}
691 The next time you \textbf{enable} the Device by either using \bconsole{} or you restart the Storage Daemon, all the saved alert messages will be discarded.
692
693 \subsection{Handling of Alerts}
694 Tape Alerts numbered 7, 8, 13, 14, 20, 22, 52, 53, and~54 will cause \bacula{} to disable the current Volume.
695 \smallskip
696
697 Tape Alerts numbered 14, 20, 29, 30, 31, 38, and~39 will cause \bacula{} to disable the drive.
698 \smallskip
699
700 Please note certain tape alerts such as 14 have multiple effects (disable the Volume and disable the drive).
701
702
703 \subsection{Multi-Tenancy Enhancements}
704 %%% patch 29
705
706 \subsection{New BWeb Management Suite Self User Restore}
707 The BWeb Management Suite can be configured to allow authorized users
708 to restore their own files on their own Unix or Linux system through BWeb. More information can be found in the BWeb Management Suite user's guide.
709
710 \subsection{New Console ACL Directives}
711 By default, if a \bdirectivename{Console ACL} directive is not set, \bacula{} will assume that the
712 \acs{ACL} list is empty. If the current \bacula{} \bdirectorname{} configuration uses restricted
713 Consoles and allows restore jobs, it is mandatory to configure the new
714 directives.
715
716 \subsubsection{Directory ACL}
717 \index[dir]{Directive!Directory ACL}
718
719 This directive is used to specify a list of directories that can be accessed by
720 a restore session. Without this directive, the console cannot restore any
721 file. Multiple directories names may be specified by separating them with
722 commas, and/or by specifying multiple \bdirectivename{DirectoryACL} directives. For example,
723 the directive may be specified as:
724
725 \begin{bVerbatim}
726 DirectoryACL = /home/bacula/, "/etc/", "/home/test/*"
727 \end{bVerbatim}
728
729 With the above specification, the console can access the following files:
730 \begin{bitemize}
731 \item \bfilename{/etc/password}
732 \item \bfilename{/etc/group}
733 \item \bfilename{/home/bacula/.bashrc}
734 \item \bfilename{/home/test/.ssh/config}
735 \item \bfilename{/home/test/Desktop/Images/something.png}
736 \end{bitemize}
737
738 But not the following files or directories:
739 \begin{bitemize}
740 \item \bfilename{/etc/security/limits.conf}
741 \item \bfilename{/home/bacula/.ssh/id\_dsa.pub}
742 \item \bdirectoryname{/home/guest/something}
743 \item \bfilename{/usr/bin/make}
744 \end{bitemize}
745
746 If a directory starts with a Windows pattern (ex: \bdirectoryname{c:/}), \bacula{} will
747 automatically ignore the case when checking directories.
748
749 \subsubsection{UserId ACL}
750 \index[dir]{Directive!UserId ACL}
751
752 This directive is used to specify a list of UID/GID that can be accessed from a
753 restore session. Without this directive, the console cannot restore any
754 file. During the restore session, the Director will compute the restore list
755 and will exclude files and directories that cannot be accessed. \bacula{} uses the
756 LStat database field to retrieve \texttt{st\_mode}, \texttt{st\_uid} and
757 \texttt{st\_gid} information for each file and compare them with the
758 \bdirectivename{UserId ACL}
759 elements. If a parent directory doesn't have a proper catalog entry, access
760 to this directory will be automatically granted.
761
762 UID/GID names are resolved with \texttt{getpwnam()} function within the Director. The
763 UID/GID mapping might be different from one system to an other.
764
765 Windows systems are \textbf{not compatible} with the \bdirectivename{UserId ACL} feature. The use of
766 \bdirectivename{UserId ACL = *all*} is required to restore Windows systems from a
767 restricted Console.
768
769 Multiple UID/GID names may be specified by separating them with
770 commas, and/or by specifying multiple \bdirectivename{UserId ACL} directives. For example,
771 the directive may be specified as:
772
773 \begin{bVerbatim}
774 UserIdACL = "bacula", "100", "100:100", ":100", "bacula:bacula"
775 \end{bVerbatim}
776
777 \begin{bVerbatim}
778 # ls -l /home
779 total 28
780 drwx------ 45 bacula bacula 12288 Oct 24 17:05 bacula
781 drwx------ 45 test test 12288 Oct 24 17:05 test
782 drwx--x--x 45 test2 test2 12288 Oct 24 17:05 test2
783 drwx------ 2 root root 16384 Aug 30 14:57 backup
784 -rwxr--r-- 1 root root 1024 Aug 30 14:57 afile
785 \end{bVerbatim}
786
787 In the example above, if the uid of the user test is 100, the following files
788 will be accessible:
789 \begin{bitemize}
790 \item \bfilename{bacula/*}
791 \item \bfilename{test/*}
792 \item \bfilename{test2/*}
793 %ARL: I don't think so... \item \bfilename{afile}
794 \end{bitemize}
795
796 The directory \bdirectoryname{backup} will not be accessible.
797
798 \subsection{Restore Job Security Enhancement}
799
800 The \bconsole{} \bcommandname{restore} command can now accept the new \texttt{jobuser=}
801 and \texttt{jobgroup=} parameters to restrict the restore process to a given
802 user account. Files and directories created during the restore session will
803 be restricted.
804
805 \begin{bVerbatim}
806 * restore jobuser=joe jobgroup=users
807 \end{bVerbatim}
808
809 The Restore Job restriction can be used on Linux and on FreeBSD. If the restore
810 Client OS doesn't support the needed thread-level user impersonation, the restore job will be
811 aborted.
812
813 \subsection{New Bconsole ``list'' Command Behavior}
814
815 The \bconsole{} \bcommandname{list} commands can now be used safely from a restricted bconsole
816 session. The information displayed will respect the \acs{ACL} configured for the Console session.
817 For example, if a Console has access to JobA, JobB and JobC, information about JobD will
818 not appear in the \bcommandname{list jobs} command.
819 %% --- patch 27
820
821 \section{\baculaenterprise{} 8.6.3}
822
823 \subsection{New Console \acs{ACL} Directives}
824 \index[dir]{Directive!BackupClientACL}
825 It is now possible to configure a restricted Console to distinguish Backup and
826 Restore jobs permissions. The \bdirectivename{Backup Client ACL} can restrict backup jobs
827 on a specific set of clients, while the \bdirectivename{Restore Client
828 ACL} can restrict
829 restore jobs.
830
831 \begin{bVerbatim}
832 # cat /opt/bacula/etc/bacula-dir.conf
833 ...
834
835 Console {
836 Name = fd-cons # Name of the FD Console
837 Password = yyy
838 ...
839 ClientACL = localhost-fd # everything allowed
840 RestoreClientACL = test-fd # restore only
841 BackupClientACL = production-fd # backup only
842 }
843 \end{bVerbatim}
844
845 The \bdirectivename{Client ACL} directive takes precedence over the
846 \bdirectivename{Restore Client ACL} and the \bdirectivename{Backup
847 Client ACL} settings. In the \bresourcename{Console}
848 resource above, this means that the bconsole linked to the \bresourcename{Console}
849 named ``fd-cons'' will be able to run:
850
851 \begin{bitemize}
852 \item backup and restore for ``localhost-fd''
853 \item backup for ``production-fd''
854 \item restore for ``test-fd''
855 \end{bitemize}
856
857 At restore time, jobs for client ``localhost-fd'', ``test-fd'' and
858 ``production-fd'' will be available.
859
860 If \bvalue{*all*} is set for \bdirectivename{Client ACL}, backup and restore will be
861 allowed for all clients, despite the use of \bdirectivename{Restore
862 Client ACL} or
863 \bdirectivename{Backup Client ACL}.
864
865
866 \section{Bacula Enterprise 8.6.0}
867
868 \subsection{Client Initiated Backup}
869 \label{blb:sec:featurecib}
870
871 A console program such as the new \bcommandname{tray-monitor} or \bconsolename{}
872 can now be configured to connect a \bresourcename{File Daemon}. There
873 are many new features available (see the New Tray Monitor \vref{blb:sec:tray-monitor}),
874 but probably the most important one is the ability for the user to initiate a
875 backup of her own machine. The connection established by the FD to the
876 \bDirector{} for the backup can be used by the \bDirector{} for the backup, thus
877 not only can clients (users) initiate backups, but a \bresourcename{File Daemon} that is
878 \acs{NAT}ed (cannot be reached by the Director) can now be backed up without
879 using advanced tunneling techniques.
880 \smallskip
881
882 The flow of information is shown in the picture \vref{fig:nat3}
883
884 \bimageH{nat}{Client Initiated Backup Network Flow}{fig:nat3}
885
886 \newpage
887 \subsection{Configuring Client Initiated Backup}
888 In order to ensure security, there are a number of new directives
889 that must be enabled in the new \texttt{tray-monitor}, the File Daemon and in the Director.
890 A typical configuration might look like the following:
891
892 \begin{bVerbatim}
893 # cat /opt/bacula/etc/bacula-dir.conf
894 ...
895
896 Console {
897 Name = fd-cons # Name of the FD Console
898 Password = yyy
899
900 # These commands are used by the tray-monitor, it is possible to restrict
901 CommandACL = run, restore, wait, .status, .jobs, .clients
902 CommandACL = .storages, .pools, .filesets, .defaults, .info
903
904 # Adapt for your needs
905 jobacl = *all*
906 poolacl = *all*
907 clientacl = *all*
908 storageacl = *all*
909 catalogacl = *all*
910 filesetacl = *all*
911 }
912 \end{bVerbatim}
913
914 \begin{bVerbatim}
915 # cat /opt/bacula/etc/bacula-fd.conf
916 ...
917
918 Console { # Console to connect the Director
919 Name = fd-cons
920 DIRPort = 9101
921 address = localhost
922 Password = "yyy"
923 }
924
925 Director {
926 Name = remote-cons # Name of the tray monitor/bconsole
927 Password = "xxx" # Password of the tray monitor/bconsole
928 Remote = yes # Allow to use send commands to the Console defined
929 }
930 \end{bVerbatim}
931
932 \begin{bVerbatim}
933 cat /opt/bacula/etc/bconsole-remote.conf
934 ....
935
936 Director {
937 Name = localhost-fd
938 address = localhost # Specify the FD address
939 DIRport = 9102 # Specify the FD Port
940 Password = "notused"
941 }
942
943 Console {
944 Name = remote-cons # Name used in the auth process
945 Password = "xxx"
946 }
947 \end{bVerbatim}
948
949 \begin{bVerbatim}
950 cat ~/.bacula-tray-monitor.conf
951
952 Monitor {
953 Name = remote-cons
954 }
955
956 Client {
957 Name = localhost-fd
958 address = localhost # Specify the FD address
959 Port = 9102 # Specify the FD Port
960 Password = "xxx"
961 Remote = yes
962 }
963 \end{bVerbatim}
964
965 \bimageH{conf-nat}{Relation Between Resources (bconsole)}{fig:nat}
966
967 \bimageH{conf-nat2}{Relation Between Resources (tray-monitor)}{fig:nat2}
968
969 A more detailed description with complete examples is available in the \bilink{Tray monitor chapter}{blb:TrayMonitorChapter} of this manual.
970
971 \subsection{New Tray Monitor}
972 \label{blb:sec:tray-monitor}
973
974 A new tray monitor has been added to the 8.6~release, which offers
975 the following features:
976
977 \begin{bitemize}
978 \item Director, File and Storage Daemon status page
979 \item Support for the Client Initiated Backup protocol (See \vref{blb:sec:featurecib}). To use the Client Initiated Backup option from the
980 tray monitor, the Client option ``Remote'' should be checked in the
981 configuration (Fig.~\vref{fig:tray2}).
982 \item Wizard to run new job (Fig.~\vref{fig:tray4})
983 \item Display an estimation of the number of files and the size of the next backup job (Fig.~\vref{fig:tray4})
984 \item Ability to configure the tray monitor configuration file directly from the \acs{GUI} (Fig.~\vref{fig:tray2})
985 \item Ability to monitor a component and adapt the tray monitor task bar icon if a jobs are running.
986 \item \acs{TLS} Support
987 \item Better network connection handling
988 \item Default configuration file is stored under \bfilename{\$HOME/.bacula-tray-monitor.conf}
989 \item Ability to ``schedule'' jobs
990 \item Available for Linux and Windows platforms
991 \end{bitemize}
992
993 \bimageH[0.8\linewidth]{tray-monitor-status}{Tray Monitor Status}{fig:tray0}
994
995 \bimageH[0.9\linewidth]{tray-monitor-conf-fd}{Tray Client Configuration}{fig:tray2}
996
997 \bimageH[0.8\linewidth]{tray-monitor-run1}{Tray Monitor Run a Job (1)}{fig:tray3}
998 \bimageH[0.8\linewidth]{tray-monitor-run2}{Tray Monitor Run a Job (2)}{fig:tray4}
999
1000 \subsection{Scheduling Jobs via the Tray Monitor}
1001
1002 The Tray Monitor can periodically scan a specific directory configured
1003 as \bdirectivename{Command Directory}
1004 and process ``\bfilename{*.bcmd}'' files to find jobs to run.
1005
1006 The format of the ``\bfilename{file.bcmd}'' command file is the following:
1007 \begin{bVerbatim}
1008 <component name>:<run command>
1009 <component name>:<run command>
1010 ...
1011
1012 <component name> = string
1013 <run command> = string (bconsole command line)
1014 \end{bVerbatim}
1015
1016 For example:
1017 \begin{bVerbatim}
1018 localhost-fd: run job=backup-localhost-fd level=full
1019 localhost-dir: run job=BackupCatalog
1020 \end{bVerbatim}
1021
1022 A command file should contain at least one command. The component specified
1023 in the first part of the command line should be defined in the tray
1024 monitor. Once the command file is detected by the tray monitor, a popup is
1025 displayed to the user and it is possible for the user to cancel the job.
1026 \smallskip{}
1027
1028 Command files can be created with tools such as \btool{cron} or the \btool{task scheduler}
1029 on Windows. It is possible and recommended to verify network connectivity at that time to
1030 avoid network errors:
1031
1032 \begin{bVerbatim}
1033 #!/bin/sh
1034 if ping -c 1 director &> /dev/null
1035 then
1036 echo "my-dir: run job=backup" > /path/to/commands/backup.bcmd
1037 fi
1038 \end{bVerbatim}
1039
1040
1041 \subsection{Concurrent VSS Snapshot Support}
1042 It is now possible to run multiple concurrent jobs that use \acs{VSS} snapshots on
1043 the File Daemon for Microsoft Windows.
1044
1045 \subsection{Accurate Option for Verify ``Volume Data'' Job}
1046
1047 As of \bacula{} version~8.4.1, it has been possible to have a Verify Job
1048 configured with \bvalueddirective{level}{Data} that will reread all records from a job
1049 and optionally check size and checksum of all files.
1050 \smallskip
1051
1052 Starting with~8.6, it is now possible to use the \bdirectivename{accurate} option to check
1053 catalog records at the same time. Using a Verify job with \bvalueddirective{level}{Data}
1054 and \bvalueddirective{accurate}{yes} can replace the \bvalueddirective{level}{VolumeToCatalog} option.
1055
1056 For more information on how to setup a Verify Data job, see \vref{blb:verifyvolumedata}.
1057
1058 To run a Verify Job with the \bdirectivename{accurate} option, it is possible to set
1059 the option in the Job definition or set use the \bvalueddirective{accurate}{yes} on the
1060 command line.
1061
1062 \begin{bVerbatim}
1063 * run job=VerifyData jobid=10 accurate=yes
1064 \end{bVerbatim}
1065
1066 \subsection{Single Item Restore Optimisation}
1067
1068 \bacula{} version~8.6.0 can generate indexes stored in the catalog to speed up
1069 file access during a Single Item Restore session for VMWare or for
1070 Exchange. The index can be displayed in bconsole with the
1071 \bcommandname{list filemedia} command.
1072
1073 \begin{bVerbatim}
1074 * list filemedia jobid=1
1075 \end{bVerbatim}
1076
1077 \subsection{FileDaemon Saved Messages Resource Destination}
1078
1079 It is now possible to send the list of all saved files to a \bresourcename{Messages} resource
1080 with the \bkeyword{saved} message type. It is not recommended to send this flow
1081 of information to the \bDirector{} and/or the \bCatalog{} when the client \bFileSet{} is
1082 large. To avoid side effects, the \bkeyword{all} keyword doesn't include
1083 the \bkeyword{saved} message type. The \bkeyword{saved} message type should be
1084 explicitly set.
1085
1086 \begin{bVerbatim}
1087 # cat /opt/bacula/etc/bacula-fd.conf
1088 ...
1089 Messages {
1090 Name = Standard
1091 director = mydirector-dir = all, !terminate, !restored, !saved
1092 append = /opt/bacula/working/bacula-fd.log = all, saved, restored
1093 }
1094 \end{bVerbatim}
1095
1096 \subsection{BWeb New Features}
1097
1098 The 8.6~release adds some new BWeb features, such as:
1099 \begin{bitemize}
1100 \item Two sets of wizards to help users to configure Copy/Migration
1101 jobs (Figures \vref{fig:cjcw} and~\vref{fig:mjcw})
1102 \item A wizard to run jobs (Fig.~\vref{fig:rjw})
1103 \item \acs{SSH} integration in BWeb Security Center to restart components remotely (Fig.~\vref{fig:src})
1104 \item Global Endpoint Deduplication Overview screen (Fig.~\vref{fig:dow})
1105 \end{bitemize}
1106
1107 \bimageH{dedup_usage}{BWeb Global Endpoint Deduplication Overview}{fig:dow}
1108
1109 \bimageH{copy_job_wizard}{Copy Job Creation Wizard}{fig:cjcw}
1110 \bimageH{migrate_job_wizard}{Migrate Job Creation Wizard}{fig:mjcw}
1111 \bimageH{run_job_wizard}{Run Job Wizard}{fig:rjw}
1112 \bimageH{ssh_remote_commands}{BWeb SSH Remote Control}{fig:src}
1113
1114 \subsection{Minor Enhancements}
1115
1116 \subsubsection{New Bconsole \bcommandname{.estimate} Command}
1117
1118 The new \bcommandname{.estimate} command can be used to get statistics about a job to
1119 run. The command uses the database to estimate the size and the number of
1120 files of the next job. On a \postgresql{} database, the command uses regression
1121 slope to compute values. On \sqlite{} or \mysql{}, where these statistical functions
1122 are not available, the command uses a simple ``average'' estimation. The
1123 correlation number is given for each value.
1124
1125 \begin{bVerbatim}
1126 *.estimate job=backup
1127 level=I
1128 nbjob=0
1129 corrbytes=0
1130 jobbytes=0
1131 corrfiles=0
1132 jobfiles=0
1133 duration=0
1134 job=backup
1135
1136 *.estimate job=backup level=F
1137 level=F
1138 nbjob=1
1139 corrbytes=0
1140 jobbytes=210937774
1141 corrfiles=0
1142 jobfiles=2545
1143 duration=0
1144 job=backup
1145 \end{bVerbatim}
1146
1147 \subsubsection{Traceback and Lockdump}
1148 %%% TODO: daemons, any binary, or a particular subset? -- ARL
1149 After the reception of a signal by any of the \bacula{} daemon binaries, \texttt{traceback} and \texttt{lockdump}
1150 information are now stored in the same file.
1151
1152 \section{\baculaenterprise{} 8.4.10}
1153
1154 \subsection{Plugin for Microsoft \acs{SQL} Server}
1155
1156 A plugin for Microsoft \acs{SQL} Server (\mssql{}) is now available. The plugin uses
1157 \mssql{} advanced backup and restore features (like \acl{PITR}, Log
1158 backup, Differential backup, \ldots{}).
1159
1160 \begin{bVerbatim}
1161 Job {
1162 Name = MSSQLJob
1163 Type = Backup
1164 Client = windows1
1165 FileSet = MSSQL
1166 Pool = 1Month
1167 Storage = File
1168 Level = Incremental
1169 }
1170
1171 FileSet {
1172 Name = MSSQL
1173 Enable VSS = no
1174 Include {
1175 Options {
1176 Signature = MD5
1177 }
1178 Plugin = "mssql"
1179 }
1180 }
1181
1182 FileSet {
1183 Name = MSSQL2
1184 Enable VSS = no
1185 Include {
1186 Options {
1187 Signature = MD5
1188 }
1189 Plugin = "mssql: database=production"
1190 }
1191 }
1192 \end{bVerbatim}
1193
1194 \section{\baculaenterprise{} 8.4.1}
1195
1196 \subsection{Verify Volume Data}
1197 \label{blb:verifyvolumedata}
1198 It is now possible to have a Verify Job configured with \texttt{level=Data} to
1199 reread all records from a job and optionally check the size and the checksum
1200 of all files.
1201
1202 \begin{bVerbatim}
1203 # Verify Job definition
1204 Job {
1205 Name = VerifyData
1206 Type = Verify
1207 Level = Data
1208 Client = 127.0.0.1-fd # Use local file daemon
1209 FileSet = Dummy # Will be adapted during the job
1210 Storage = File # Should be the right one
1211 Messages = Standard
1212 Pool = Default
1213 }
1214
1215 # Backup Job definition
1216 Job {
1217 Name = MyBackupJob
1218 Type = Backup
1219 Client = windows1
1220 FileSet = MyFileSet
1221 Pool = 1Month
1222 Storage = File
1223 }
1224
1225 FileSet {
1226 Name = MyFileSet
1227 Include {
1228 Options {
1229 Verify = s5
1230 Signature = MD5
1231 }
1232 File = /
1233 }
1234 \end{bVerbatim}
1235
1236 To run the Verify job, it is possible to use the ``jobid'' parameter of the \bcommandname{run} command.
1237
1238 \begin{bVerbatim}
1239 *run job=VerifyData jobid=10
1240 Run Verify Job
1241 JobName: VerifyData
1242 Level: Data
1243 Client: 127.0.0.1-fd
1244 FileSet: Dummy
1245 Pool: Default (From Job resource)
1246 Storage: File (From Job resource)
1247 Verify Job: MyBackupJob.2015-11-11_09.41.55_03
1248 Verify List: /opt/bacula/working/working/VerifyVol.bsr
1249 When: 2015-11-11 09:47:38
1250 Priority: 10
1251 OK to run? (yes/mod/no): yes
1252 Job queued. JobId=14
1253
1254 ...
1255
1256 11-Nov 09:46 my-dir JobId 13: Bacula Enterprise 8.4.1 (13Nov15):
1257 Build OS: x86_64-unknown-linux-gnu archlinux
1258 JobId: 14
1259 Job: VerifyData.2015-11-11_09.46.29_03
1260 FileSet: MyFileSet
1261 Verify Level: Data
1262 Client: 127.0.0.1-fd
1263 Verify JobId: 10
1264 Verify Job:q
1265 Start time: 11-Nov-2015 09:46:31
1266 End time: 11-Nov-2015 09:46:32
1267 Files Expected: 1,116
1268 Files Examined: 1,116
1269 Non-fatal FD errors: 0
1270 SD Errors: 0
1271 FD termination status: Verify differences
1272 SD termination status: OK
1273 Termination: Verify Differences
1274 \end{bVerbatim}
1275
1276 The current Verify Data implementation requires specifying the correct Storage
1277 resource in the Verify job. The Storage resource can be changed with the \btool{bconsole}
1278 command line and with the menu.
1279
1280 \subsection{Bconsole \bcommandname{list jobs} Command Options}
1281
1282 The \bcommandname{list jobs} bconsole command now accepts new command line options:
1283
1284 \begin{bitemize}
1285 \item \textbf{joberrors} Display jobs with JobErrors
1286 \item \textbf{jobstatus=T} Display jobs with the specified status code
1287 \item \textbf{client=client-name} Display jobs for a specified client
1288 \item \textbf{order=asc/desc} Change the output format of the job list. The
1289 jobs are sorted by start time and JobId, the sort can use ascending (asc) or
1290 descending (desc) order, the latter being the default.
1291 \end{bitemize}
1292
1293 \subsection{Minor Enhancements}
1294
1295 \subsubsection{New Bconsole \bcommandname{tee all} Command}
1296
1297 The ``@tall'' command allows logging all input and output of a console session.
1298
1299 \begin{bVerbatim}
1300 *@tall /tmp/log
1301 *st dir
1302 ...
1303 @tall
1304 \end{bVerbatim}
1305
1306 \subsubsection{\mysql{} Plugin Restore Options}
1307
1308 It is now possible to specify the database name during a restore in the Plugin
1309 Option menu. It is still possible to use the ``Where'' parameter to
1310 specify the target database name.
1311
1312 \subsubsection{\postgresql{} Plugin}
1313
1314 We added a ``timeout'' option to the \postgresql{} plugin command line that is set
1315 to \bdefaultvalue{60s} by default. Users may want to change this value when the \postgresql{} cluster
1316 is slow to complete \acs{SQL} queries used during the backup.
1317
1318 %%% ARL: read and modified up to here
1319
1320 \section{\baculaenterprise{} 8.4}
1321
1322 \subsection{VMWare Single File Restore}
1323
1324 It is now possible to explore VMWare virtual machines backup jobs (Full,
1325 Incremental and Differential) made with the \baculaenterprise{} vSphere plugin to
1326 restore individual files and directories. The \texttt{Single Item Restore}
1327 feature comes with both a console interface and a \bweb{} Management Suite specific
1328 interface. See the VMWare Single File Restore whitepaper for more information.
1329
1330 \bimageH{bweb-vmware-sir}{\bweb{} VMWare Single File Restore}{fig:bwebvmwaresir}
1331
1332
1333 \subsection{Microsoft Exchange Single MailBox Restore}
1334
1335 It is now possible to explore Microsoft Exchange databases backups made with
1336 the \baculaenterprise{} \acs{VSS} plugin to restore individual mailboxes. The
1337 \texttt{Single Item Restore} feature comes with both a console interface and a
1338 web interface. See the Exchange Single Mailbox Restore whitepaper for more
1339 information.
1340
1341 \section{\baculaenterprise{} 8.2.8}
1342
1343 \subsection{New Job Edit Codes \%I}
1344 In various places such as RunScripts, you have now access to \%I to get the
1345 JobId of the copy or migration job started by a migrate job.
1346
1347 \begin{bVerbatim}
1348 Job {
1349 Name = Migrate-Job
1350 Type = Migrate
1351 ...
1352 RunAfter = "echo New JobId is %I"
1353 }
1354 \end{bVerbatim}
1355
1356 \section{\baculaenterprise{} 8.2.2}
1357
1358 \subsection{New Job Edit Codes \%E \%R}
1359 In various places such as RunScripts, you have now access to \%E to get the
1360 number of non-fatal errors for the current Job and \%R to get the number of
1361 bytes read from disk or from the network during a job.
1362
1363 \subsection{Enable/Disable commands}
1364 The \btool{bconsole} \bcommandname{enable} and \bcommandname{disable} commands have
1365 been extended from enabling/disabling Jobs to include Clients, Schedule,
1366 and Storage devices. Examples:
1367
1368 \begin{bVerbatim}
1369 disable Job=NightlyBackup Client=Windows-fd
1370 \end{bVerbatim}
1371
1372 will disable the Job named \textbf{NightlyBackup} as well as the
1373 client named \textbf{Windows-fd}.
1374
1375 \begin{bVerbatim}
1376 disable Storage=LTO-changer Drive=1
1377 \end{bVerbatim}
1378
1379 will disable the first drive in the autochanger named \textbf{LTO-changer}.
1380
1381 Please note that doing a \bcommandname{reload} command will set any values
1382 changed by the enable/disable commands back to the values in the
1383 \bfilename{bacula-dir.conf} file.
1384
1385 The \bresourcename{Client} and \bresourcename{Schedule} resources in the \bfilename{bacula-dir.conf} file now permit
1386 the directive \bdirectivename{Enabled} = \bvalue{yes} or \bdirectivename{Enabled} = \bvalue{no}.
1387
1388
1389 \section{\baculaenterprise{} 8.2}
1390
1391 \subsection{Snapshot Management}
1392
1393 \baculaenterprise{} 8.2 is now able to handle Snapshots on Linux/Unix
1394 systems. Snapshots can be automatically created and used to backup files. It is
1395 also possible to manage Snapshots from \bacula{}'s \btool{bconsole} tool through a
1396 unique interface.
1397
1398 \subsubsection{Snapshot Backends}
1399
1400 The following Snapshot backends are supported with \baculaenterprise{} 8.2:
1401
1402 \begin{bitemize}
1403 \item \acs{BTRFS}
1404 \item \acs{ZFS}
1405 \item \acs{LVM}\footnote{Some restrictions described in \vref{blb:LVMBackend} applies to
1406 the \acs{LVM} backend}
1407 \end{bitemize}
1408
1409 By default, Snapshots are mounted (or directly available) under
1410 \bdefaultvalue{.snapshots} directory on the root filesystem. (On \acs{ZFS}, the default
1411 is \bdefaultvalue{.zfs/snapshots}).
1412 \smallskip{}
1413
1414 The Snapshot backend program is called \btool{bsnapshot} and is available in
1415 the \textbf{bacula-enterprise-snapshot} package. In order to use the Snapshot
1416 Management feature, the package must be installed on the Client.
1417 \smallskip{}
1418
1419 \label{blb:bsnapshotconf}
1420 The \btool{bsnapshot} program can be configured using
1421 \bfilename{/opt/bacula/etc/bsnapshot.conf} file. The following parameters can
1422 be adjusted in the configuration file:
1423
1424 \begin{bitemize}
1425 \item \texttt{trace=\bbracket{file}} Specify a trace file
1426 \item \texttt{debug=\bbracket{num}} Specify a debug level
1427 \item \texttt{sudo=\byesorno{}} Use sudo to run commands
1428 \item \texttt{disabled=\byesorno{}} Disable snapshot support
1429 \item \texttt{retry=\bbracket{num}} Configure the number of retries for some operations
1430 \item \texttt{snapshot\_dir=\bbracket{dirname}} Use a custom name for the Snapshot directory. (\bdirectoryname{.SNAPSHOT}, \bdirectoryname{.snapdir}, etc.)
1431 \item \texttt{lvm\_snapshot\_size=\bbracket{lvpath:size}} Specify a custom snapshot size for a given \acs{LVM} volume
1432 \item \texttt{mountopts=\bbracket{devpath:options}} Specify a custom mount option for a give device (avaiable in 10.0.4)
1433 \end{bitemize}
1434
1435 \begin{bVerbatim}
1436 # cat /opt/bacula/etc/bsnapshot.conf
1437 trace=/tmp/snap.log
1438 debug=10
1439 lvm_snapshot_size=/dev/ubuntu-vg/root:5%
1440 mountopts=nouuid
1441 mountopts=/dev/ubuntu-vg/root:nouuid,nosuid
1442 \end{bVerbatim}
1443
1444 \subsubsection{Application Quiescing}
1445
1446 When using Snapshots, it is very important to quiesce applications that are
1447 running on the system. The simplest way to quiesce an application is to stop
1448 it. Usually, taking the Snapshot is very fast, and the downtime is only about a
1449 couple of seconds. If downtime is not possible and/or the application provides
1450 a way to quiesce, a more advanced script can be used. An example is
1451 described on \vref{blb:SnapRunScriptExample}.
1452
1453 \subsubsection{New Director Directives}
1454
1455 The use of the Snapshot Engine on the FileDaemon is determined by the
1456 new \textbf{Enable Snapshot} FileSet directive. The default is \bdefaultvalue{no}.
1457
1458 \begin{bVerbatim}
1459 FileSet {
1460 Name = LinuxHome
1461
1462 Enable Snapshot = yes
1463
1464 Include {
1465 Options = { Compression = LZO }
1466 File = /home
1467 }
1468 }
1469 \end{bVerbatim}
1470
1471 By default, Snapshots are deleted from the Client at the end of the backup. To
1472 keep Snapshots on the Client and record them in the Catalog for a determined
1473 period, it is possible to use the \textbf{Snapshot Retention} directive in the
1474 Client or in the Job resource. The default value is \bdefaultvalue{0 secconds}. If, for a given Job,
1475 both Client and Job \textbf{Snapshot Retention} directives are set, the Job
1476 directive will be used.
1477
1478 \begin{bVerbatim}
1479 Client {
1480 Name = linux1
1481 ...
1482
1483 Snapshot Retention = 5 days
1484 }
1485 \end{bVerbatim}
1486
1487 To automatically prune Snapshots, it is possible to use the following RunScript
1488 command:
1489
1490 \begin{bVerbatim}
1491 Job {
1492 ...
1493 Client = linux1
1494 ...
1495 RunScript {
1496 RunsOnClient = no
1497 Console = "prune snapshot client=%c yes"
1498 RunsAfter = yes
1499 }
1500 }
1501 \end{bVerbatim}
1502 \smallskip{}
1503
1504 In RunScripts, the \texttt{AfterSnapshot} keyword for the \texttt{RunsWhen} directive will
1505 allow a command to be run just after the Snapshot creation.
1506
1507 \texttt{AfterSnapshot} is a synonym for the \texttt{AfterVSS} keyword.
1508
1509 \label{blb:SnapRunScriptExample}
1510 \begin{bVerbatim}
1511 Job {
1512 ...
1513 RunScript {
1514 Command = "/etc/init.d/mysql start"
1515 RunsWhen = AfterSnapshot
1516 RunsOnClient = yes
1517 }
1518 RunScript {
1519 Command = "/etc/init.d/mysql stop"
1520 RunsWhen = Before
1521 RunsOnClient = yes
1522 }
1523 }
1524 \end{bVerbatim}
1525
1526 \subsubsection{Job Output Information}
1527
1528 Information about Snapshots are displayed in the Job output. The list of all
1529 devices used by the Snapshot Engine is displayed, and the Job summary
1530 indicates if Snapshots were available.
1531
1532 \begin{bVerbatim}
1533 JobId 3: Create Snapshot of /home/build
1534 JobId 3: Create Snapshot of /home/build/subvol
1535 JobId 3: Delete snapshot of /home/build
1536 JobId 3: Delete snapshot of /home/build/subvol
1537 ...
1538 JobId 3: Bacula 127.0.0.1-dir 8.2.0 (23Feb15):
1539 Build OS: x86_64-unknown-linux-gnu archlinux
1540 JobId: 3
1541 Job: Incremental.2015-02-24_11.20.27_08
1542 Backup Level: Full
1543 ...
1544 Snapshot/VSS: yes
1545 ...
1546 Termination: Backup OK
1547 \end{bVerbatim}
1548
1549
1550 \subsubsection{New \bcommandname{snapshot} Bconsole Commands}
1551
1552 The new \bcommandname{snapshot} command will display by default the following menu:
1553 \begin{bVerbatim}
1554 *snapshot
1555 Snapshot choice:
1556 1: List snapshots in Catalog
1557 2: List snapshots on Client
1558 3: Prune snapshots
1559 4: Delete snapshot
1560 5: Update snapshot parameters
1561 6: Update catalog with Client snapshots
1562 7: Done
1563 Select action to perform on Snapshot Engine (1-7):
1564 \end{bVerbatim}
1565
1566 The \bcommandname{snapshot} command can also have the following parameters:
1567 \begin{bVerbatim}
1568 [client=<client-name> | job=<job-name> | jobid=<jobid>]
1569 [delete | list | listclient | prune | sync | update]
1570 \end{bVerbatim}
1571
1572 It is also possible to use traditional \bcommandname{list}, \bcommandname{llist},
1573 \bcommandname{update}, \bcommandname{prune} or \bcommandname{delete} commands on Snapshots.
1574
1575 \begin{bVerbatim}
1576 *llist snapshot jobid=5
1577 snapshotid: 1
1578 name: NightlySave.2015-02-24_12.01.00_04
1579 createdate: 2015-02-24 12:01:03
1580 client: 127.0.0.1-fd
1581 fileset: Full Set
1582 jobid: 5
1583 volume: /home/.snapshots/NightlySave.2015-02-24_12.01.00_04
1584 device: /home/btrfs
1585 type: btrfs
1586 retention: 30
1587 comment:
1588 \end{bVerbatim}
1589
1590 \begin{bVerbatim}
1591 * snapshot listclient
1592 Automatically selected Client: 127.0.0.1-fd
1593 Connecting to Client 127.0.0.1-fd at 127.0.0.1:9102
1594 Snapshot NightlySave.2015-02-24_12.01.00_04:
1595 Volume: /home/.snapshots/NightlySave.2015-02-24_12.01.00_04
1596 Device: /home
1597 CreateDate: 2015-02-24 12:01:03
1598 Type: btrfs
1599 Status: OK
1600 Error:
1601 \end{bVerbatim}
1602 \smallskip{}
1603
1604 With the \emph{Update catalog with Client snapshots} option (or
1605 \bcommandname{snapshot sync}), the Director contacts the FileDaemon, lists snapshots
1606 of the system and creates catalog records of the Snapshots.
1607
1608 \begin{bVerbatim}
1609 *snapshot sync
1610 Automatically selected Client: 127.0.0.1-fd
1611 Connecting to Client 127.0.0.1-fd at 127.0.0.1:9102
1612 Snapshot NightlySave.2015-02-24_12.35.47_06:
1613 Volume: /home/.snapshots/NightlySave.2015-02-24_12.35.47_06
1614 Device: /home
1615 CreateDate: 2015-02-24 12:35:47
1616 Type: btrfs
1617 Status: OK
1618 Error:
1619 Snapshot added in Catalog
1620
1621 *llist snapshot
1622 snapshotid: 13
1623 name: NightlySave.2015-02-24_12.35.47_06
1624 createdate: 2015-02-24 12:35:47
1625 client: 127.0.0.1-fd
1626 fileset:
1627 jobid: 0
1628 volume: /home/.snapshots/NightlySave.2015-02-24_12.35.47_06
1629 device: /home
1630 type: btrfs
1631 retention: 0
1632 comment:
1633 \end{bVerbatim}
1634
1635 % list
1636 % llist
1637 % prune
1638 % delete
1639 % update snapshot
1640 % sync
1641
1642 \subsubsection{\acs{LVM} Backend Restrictions}
1643 \label{blb:LVMBackend}
1644
1645 \acs{LVM} Snapshots are quite primitive compared to \acs{ZFS}, \acs{BTRFS}, NetApp and other
1646 systems. For example, it is not possible to use Snapshots if the \ac{VG} is full.
1647 The administrator must keep some free space in the \ac{VG} to create
1648 Snapshots. The amount of free space required depends on the activity of the
1649 \ac{LV}. \btool{bsnapshot} uses \bdefaultvalue{10\%} of the \ac{LV} by default. This
1650 number can be configured per \ac{LV} in the \bfilename{bsnapshot.conf} file (See
1651 \vref{blb:bsnapshotconf}).
1652
1653 % waa - 20150316 - Not exactly sure why this vgdisplay information is here with no explanation.
1654 \begin{bVerbatim}
1655 [root@system1]# vgdisplay
1656 --- Volume group ---
1657 VG Name vg_ssd
1658 ...
1659 VG Size 29,81 GiB
1660 ...
1661 Alloc PE / Size 125 / 500,00 MiB
1662 Free PE / Size 7507 / 29,32 GiB <---- Free Space
1663 ...
1664 \end{bVerbatim}
1665
1666 It is also not advisable to leave snapshots on the \acs{LVM} backend. Having multiple
1667 snapshots of the same \acs{LV} on \acs{LVM} will slow down the system.
1668 \smallskip{}
1669
1670 Only Ext4, XFS and EXT3 \footnote{XFS and EXT3 are available in 8.2.7 and
1671 later} filesystems are supported with the Snapshot \acs{LVM} backend.
1672
1673 \subsubsection{Debug Options}
1674
1675 To get low level information about the Snapshot Engine, the debug tag ``snapshot''
1676 should be used in the \bcommandname{setdebug} command.
1677
1678 \begin{bVerbatim}
1679 * setdebug level=10 tags=snapshot client
1680 * setdebug level=10 tags=snapshot dir
1681 \end{bVerbatim}
1682
1683 \subsection{Global Endpoint Deduplication(TM)}
1684 \subsubsection{Storage to Storage Copy/Migration}
1685
1686 Copy and Migration Jobs now use the Global Endpoint Deduplication protocol if
1687 the destination Device Type is \texttt{dedup}.
1688
1689 \subsubsection{Performance Enhancements}
1690
1691 A new automatic Deduplication index optimization has been added to the Vacuum
1692 procedure.
1693 \smallskip{}
1694
1695 Part of the Deduplication index can be locked into memory to improve
1696 performance.
1697 \smallskip{}
1698
1699 Users can now configure parameters related to the size of the Deduplication
1700 index and the amount of memory that can be used to cache the index.
1701
1702 \subsection{Hypervisor Plugins}
1703 \subsubsection{Hyper-V VSS Plugin}
1704
1705 Backing up and restoring Hyper-V virtual machines is supported with Full level
1706 backups using the \acs{VSS} \acs{API}. Use of the Global Endpoint Deduplication plugin and the
1707 \texttt{bothsides} FileSet option minimizes the amount of data transfered and the
1708 amount of storage used.
1709
1710 \subsubsection{KVM Plugin}
1711
1712 The \acs{KVM} plugin provides the following main features:
1713
1714 \begin{bitemize}
1715 \item File level backup
1716 \item Automatic virtual machine discovery
1717 \item Full, Differential, Incremental backup level support
1718 \item The ability to handle inclusion/exclusion of files
1719 \end{bitemize}
1720
1721 The \acs{KVM} plugin is designed to be used when the hypervisor uses local storage for virtual
1722 machine disks and \btool{libvirtd} for virtual machine management.
1723
1724 \subsection{Windows Encrypted File System (EFS) Support}
1725
1726 The \baculaenterprise{} Windows File Daemon now automatically supports files and
1727 directories that are encrypted on Windows filesystem.
1728
1729 \subsection{\bweb{} Management Suite}
1730
1731 \bimageH{bweb-sd-status}{\bweb{} Storage Daemon Status/Overview}{fig:bwebsdoverview}
1732 \bimageH{bweb-tls-security-center}{\bweb{} TLS Security Center}{fig:bwebtls}
1733 \bimageH{bweb-fileset-wizard-browser}{\bweb{} FileSet Wizard}{fig:bwebfswiz}
1734 \bimageH{bweb-configurable-table}{\bweb{} Job/Media Table Configuration}{fig:bwebconftable}
1735
1736
1737 \subsection{Minor Enhancements}
1738 \subsubsection{Copy/Migration/VirtualFull Performance Enhancements}
1739
1740 The Copy, Migration and VirtualFull performance on large jobs with millions of
1741 files has been greatly enhanced.
1742
1743 \subsubsection{Storage Daemon Reports Disk Usage}
1744
1745 The \bcommandname{status storage} command now reports the space available on disk devices:
1746 \begin{bVerbatim}
1747 ...
1748 Device status:
1749
1750 Device file: "FileStorage" (/bacula/arch1) is not open.
1751 Available Space=5.762 GB
1752 ==
1753
1754 Device file: "FileStorage1" (/bacula/arch2) is not open.
1755 Available Space=5.862 GB
1756 \end{bVerbatim}
1757
1758 \section{\baculaenterprise{} 8.0}
1759
1760 \subsection{Global Endpoint Deduplication\texttrademark{}}
1761
1762 The \emph{Global Endpoint Deduplication} solution minimizes network
1763 transfers and \bacula{} Volume size using deduplication technology.
1764 \smallskip
1765
1766 The new Global Endpoint Deduplication Storage daemon directives are:
1767
1768 \begin{bdescription}
1769 \item [Device Type = Dedup] sets the Storage device for
1770 deduplication. Deduplication is performed only on disk volumes.
1771 \item [Dedup Directory =] this directive specifies where the
1772 deduplicated blocks will be stored. Blocks that are deduplicated
1773 will be placed in this directory rather than in the \bacula{} Volume,
1774 which will only contain a reference pointer to the deduplicated
1775 blocks.
1776 \item [Dedup Index Directory] in addition to the deduplicated
1777 blocks, when deduplication is enabled, the Storage daemon keeps an
1778 index of the deduplicated block locations. This index will be
1779 frequently consulted during the deduplication backup process,
1780 so it should be placed on the fastest device possible (e.g.
1781 an \acs{SSD}).
1782 \end{bdescription}
1783 \smallskip
1784
1785 See below for a FileSet example using the new \texttt{dedup} directive.
1786
1787 \subsubsection{Configuration Example}
1788 In the Storage Daemon configuration file, you must define a Device with the
1789 \textbf{DeviceType = Dedup}. It is also possible to configure where the
1790 Storage Daemon will store blocks and indexes. Blocks will be stored in the
1791 \textbf{Dedup Directory}, the directory is common for all \textbf{Dedup}
1792 devices and should have a large amount of free space. Indexes will be
1793 stored in the \textbf{Dedup Index Directory}, indexes will have a lot of random
1794 update access, and can benefit from \acs{SSD} drives.
1795
1796 \begin{bVerbatim}
1797 # from bacula-sd.conf
1798 Storage {
1799 Name = my-sd
1800 Working Directory = /opt/bacula/working
1801 Pid Directory = /opt/bacula/working
1802
1803 Plugin Directory = /opt/bacula/plugins
1804 Dedup Directory = /opt/bacula/dedup
1805 Dedup Index Directory = /opt/bacula/ssd # default for Dedup Directory
1806 }
1807
1808 Device {
1809 Name = DedupDisk
1810 Archive Device = /opt/bacula/storage
1811 Media Type = DedupVolume
1812 Label Media = yes
1813 Random Access = yes
1814 Automatic Mount = yes
1815 Removable Media = no
1816 Always Open = no
1817
1818 Device Type = Dedup # Required
1819 }
1820 \end{bVerbatim}
1821
1822 The Global Endpoint Deduplication Client cache system can speed up restore
1823 jobs by getting blocks from the local client disk instead of requesting
1824 them over the network. Note that if blocks are not available locally, the
1825 FileDaemon will get blocks from the Storage Daemon. This feature can be
1826 enabled with the \textbf{Dedup Index Directory} directive in the FileDaemon
1827 resource. When using this option, the File Daemon will have to maintain the
1828 cache during Backup jobs.
1829
1830 \begin{bVerbatim}
1831 # from bacula-fd.conf
1832 FileDaemon {
1833 Name = my-fd
1834 Working Directory = /opt/bacula/working
1835 Pid Directory = /opt/bacula/working
1836
1837 # Optional, Keep indexes on the client for faster restores
1838 Dedup Index Directory = /opt/bacula/dedupindex
1839 }
1840 \end{bVerbatim}
1841
1842 It is possible to configure the Global Endpoint
1843 Deduplication system in the Director with a FileSet directive called
1844 \textbf{Dedup}. Each FileSet Include section can specify a different
1845 deduplication behavior depending on your needs.
1846
1847 \begin{bVerbatim}
1848 FileSet {
1849 Name = FS_BASE
1850
1851 # Send everything to the Storage Daemon as usual
1852 # and let the Storage Daemon do the deduplication
1853 Include {
1854 Options {
1855 Dedup = storage
1856 }
1857 File = /opt/bacula/etc
1858 }
1859
1860 # Send only references and new blocks to the Storage Daemon
1861 Include {
1862 Options {
1863 Dedup = bothsides
1864 }
1865 File = /VirtualBox
1866 }
1867
1868 # Do not try to dedup my encrypted directory
1869 Include {
1870 Options {
1871 Dedup = none
1872 }
1873 File = /encrypted
1874 }
1875 }
1876 \end{bVerbatim}
1877
1878 The FileSet \textbf{Dedup} directive accepts the following values:
1879
1880 \begin{bitemize}
1881 %%% Philippe: Why is the label defined as a start just for this item?
1882 \item[*] \textbf{storage} All the deduplication work is done on the SD
1883 side if the device type is \bdefaultvalue{dedup} (default value). This
1884 option is useful if you want to avoid the extra client-side disk space overhead
1885 that will occur with the \textbf{bothsides} option.
1886 \item \textbf{none} Force FD and SD to not use deduplication
1887 \item \textbf{bothsides} The deduplication work is done on both the FD and the
1888 SD. Only references and new blocks will be transfered over the network.
1889 \end{bitemize}
1890
1891
1892 \subsection{Storage Daemon to Storage Daemon}
1893 \baculaenterprise{} version 8.0 now permits SD to SD transfer of Copy and
1894 Migration Jobs. This permits what is commonly referred to as replication
1895 or off-site transfer of \bacula{} backups. It occurs automatically if the
1896 source SD and destination SD of a Copy or Migration job are different.
1897 That is, the SD to SD transfers need no additional configuration
1898 directives. The following picture shows how this works.
1899
1900 \bimageH{sd-to-sd}{SD to SD}{fig:sdtosd}
1901 %\includegraphics[width=0.8\linewidth]{sd-to-sd}
1902
1903 \subsection{Windows Mountpoint Support}
1904 \baculaenterprise{} version 8.0 is now able to detect Windows mountpoints and include
1905 volumes automatically in the \acs{VSS} snapshot set. To backup all local disks
1906 on a Windows server, the following FileSet is now accepted. It depreciates
1907 the alldrives plugin.
1908
1909 \begin{bVerbatim}
1910 FileSet {
1911 Name = "All Drives"
1912 Include {
1913 Options {
1914 Signature = MD5
1915 }
1916
1917 File = /
1918 }
1919 }
1920 \end{bVerbatim}
1921
1922 If you have mountpoints, the \texttt{onefs=no} option should be used as it is with
1923 Unix systems.
1924
1925 \begin{bVerbatim}
1926 FileSet {
1927 Name = "All Drives with mountpoints"
1928 Include {
1929 Options {
1930 Signature = MD5
1931 OneFS = no
1932 }
1933 File = C:/ # will include mountpoint C:/mounted/...
1934 }
1935 }
1936 \end{bVerbatim}
1937
1938 To exclude a mountpoint from a backup when \texttt{OneFS = no}, use the Exclude block as usual:
1939
1940 \begin{bVerbatim}
1941 FileSet {
1942 Name = "All Drives with mountpoints"
1943 Include {
1944 Options {
1945 Signature = MD5
1946 OneFS = no
1947 }
1948 File = C:/ # will include all mounted mountpoints under C:/
1949 # including C:/mounted (see Exclude below)
1950 }
1951
1952 Exclude {
1953 File = C:/mounted # will not include C:/mounted
1954 }
1955 }
1956 \end{bVerbatim}
1957
1958 % waa - 20150317- Pretty sure Migration jobs do not involve the Client, so that should be removed
1959 % from this paragraph.
1960 \subsection{SD Calls Client}
1961 If the \bdirectivename{SD Calls Client} directive is set to \bvalue{true} in a \bresourcename{Client} resource
1962 any Backup, Restore, Verify Job where the client
1963 is involved, the client will wait for the Storage daemon to contact it.
1964 By default this directive is set to \bdefaultvalue{false}, and the Client will call
1965 the Storage daemon as it always has. This directive can be useful if your Storage daemon
1966 is behind a firewall that permits outgoing connections but not incoming
1967 connections. The picture \vref{fig:sdcallsclient} shows the communications connection paths in
1968 both cases.
1969
1970 \bimageH[0.8\linewidth]{sd-calls-client}{SD Calls Client}{fig:sdcallsclient}
1971
1972
1973
1974 \subsection{Data Encryption Cipher Configuration}
1975 \baculaenterprise{} version 8.0 and later now allows configuration of the data
1976 encryption cipher and the digest algorithm. Previously, the cipher was forced to AES 128,
1977 but it is now possible to choose between the following ciphers:
1978
1979 \begin{bitemize}
1980 \item \bdefaultvalue{AES128} (default)
1981 \item AES192
1982 \item AES256
1983 \item blowfish
1984 \end{bitemize}
1985
1986 The digest algorithm was set to SHA1 or SHA256 depending on the local OpenSSL
1987 options. We advise you to not modify the PkiDigest default setting. Please,
1988 refer to the OpenSSL documentation to understand the pros and cons regarding these options.
1989
1990 \begin{bVerbatim}
1991 FileDaemon {
1992 ...
1993 PkiCipher = AES256
1994 }
1995 \end{bVerbatim}
1996
1997 \subsection{Minor Enhancements}
1998
1999 \subsubsection*{New Option Letter ``M'' for Accurate Directive in FileSet}
2000
2001 % waa - 20150317 - is 8.0.5 correct here?
2002 Added in version 8.0.5, the new ``M'' option letter for the Accurate directive
2003 in the FileSet Options block, which allows comparing the modification time and/or
2004 creation time against the last backup timestamp. This is in contrast to the
2005 existing options letters ``m'' and/or ``c'', mtime and ctime, which are checked
2006 against the stored catalog values, which can vary accross different machines
2007 when using the BaseJob feature.
2008
2009 The advantage of the new ``M'' option letter for Jobs that refer to BaseJobs is
2010 that it will instruct \bacula{} to backup files based on the last backup time, which
2011 is more useful because the mtime/ctime timestamps may differ on various Clients,
2012 causing files to be needlessly backed up.
2013 \smallskip{}
2014
2015 \begin{bVerbatim}
2016 Job {
2017 Name = USR
2018 Level = Base
2019 FileSet = BaseFS
2020 ...
2021 }
2022
2023 Job {
2024 Name = Full
2025 FileSet = FullFS
2026 Base = USR
2027 ...
2028 }
2029
2030 FileSet {
2031 Name = BaseFS
2032 Include {
2033 Options {
2034 Signature = MD5
2035 }
2036 File = /usr
2037 }
2038 }
2039
2040 FileSet {
2041 Name = FullFS
2042 Include {
2043 Options {
2044 Accurate = Ms # check for mtime/ctime of last backup timestamp and Size
2045 Signature = MD5
2046 }
2047 File = /home
2048 File = /usr
2049 }
2050 }
2051 \end{bVerbatim}
2052
2053 \subsubsection*{.api version 2}
2054
2055 In \baculaenterprise{} version 8.0 and later, we introduced a new .api version
2056 to help external tools to parse various \bacula{} bconsole output.
2057
2058 % waa - 20150317 - this section needs just a little more to explain what the "43" in "s43" mean. Perhaps
2059 % if it is not a good place to list the possibilities here, then list where a reference
2060 % is. Also, I think .api 2 ... Means "use API version 2" but that should be stated too
2061
2062 The \texttt{api\_opts} option can use the following arguments:
2063 \begin{bdescription}
2064 \item [C] Clear current options
2065 \item [tn] Use a specific time format (1 ISO format, 2 Unix Timestamp, 3 Default \bacula{} time format)
2066 \item [sn] Use a specific separator between items (new line by default).
2067 \item [Sn] Use a specific separator between objects (new line by default).
2068 \item [o] Convert all keywords to lowercase and convert all non \emph{isalpha} characters to \_
2069 \end{bdescription}
2070
2071 % waa - 20150317 - I think there should either be more output listed here to give a better feeling
2072 % or, perhaps another output listing for different .status commands
2073
2074 \begin{bVerbatim}
2075 .api 2 api_opts=t1s43S35
2076 .status dir running
2077 ==================================
2078 jobid=10
2079 job=AJob
2080 ...
2081 \end{bVerbatim}
2082
2083 \subsubsection*{New Debug Options}
2084
2085 In \baculaenterprise{} version 8.0 and later, we introduced a new \texttt{options} parameter for
2086 the \bcommandname{setdebug} bconsole command.
2087 \smallskip{}
2088
2089 The following arguments to the new \texttt{option} parameter are available to control debug functions.
2090
2091 \begin{bdescription}
2092 \item [0] Clear debug flags
2093 \item [i] Turn off, ignore bwrite() errors on restore on File Daemon
2094 \item [d] Turn off decomp of BackupRead() streams on File Daemon
2095 \item [t] Turn on timestamps in traces
2096 \item [T] Turn off timestamps in traces
2097
2098 % waa - 20150306 - does this "c" item mean to say "Truncate trace file if one exists, otherwise append to it" ???
2099 \item [c] Truncate trace file if trace file is activated
2100
2101 \item [l] Turn on recoding events on P() and V()
2102 \item [p] Turn on the display of the event ring when doing a backtrace
2103 \end{bdescription}
2104 \smallskip{}
2105
2106 The following command will enable debugging for the File Daemon, truncate an existing trace file,
2107 and turn on timestamps when writing to the trace file.
2108
2109 \begin{bVerbatim}
2110 * setdebug level=10 trace=1 options=ct fd
2111 \end{bVerbatim}
2112 \smallskip{}
2113
2114 It is now possible to use a \emph{class} of debug messages called \texttt{tags}
2115 to control the debug output of \bacula{} daemons.
2116
2117 \begin{bdescription}
2118 \item [all] Display all debug messages
2119 \item [bvfs] Display BVFS debug messages
2120 \item [sql] Display \acs{SQL} related debug messages
2121 \item [memory] Display memory and poolmem allocation messages
2122 \item [scheduler] Display scheduler related debug messages
2123 \end{bdescription}
2124
2125 \begin{bVerbatim}
2126 * setdebug level=10 tags=bvfs,sql,memory
2127 * setdebug level=10 tags=!bvfs
2128
2129 # bacula-dir -t -d 200,bvfs,sql
2130 \end{bVerbatim}
2131
2132 The \texttt{tags} option is composed of a list of tags. Tags are separated by
2133 ``,'' or ``+'' or ``-'' or ``!''. To disable a specific tag, use ``-'' or ``!''
2134 in front of the tag. Note that more tags are planned for future versions.
2135
2136 \LTXtable{\linewidth}{t-main-debugtags}
2137
2138 \section{\baculaenterprise{} 6.6.0}
2139 \subsection{Communication Line Compression}
2140 \baculaenterprise{} version 6.6.0 and later now includes communication
2141 line compression. It is turned on by default, and if the
2142 two \bacula{} components (DIR, FD, SD, bconsole) are both
2143 version 6.6.0 or greater, communication line compression)
2144 will be enabled, by default. If for some reason, you do not want
2145 communication line compression, you may disable it with the
2146 following directive:
2147
2148 \begin{bVerbatim}
2149 Comm Compression = no
2150 \end{bVerbatim}
2151
2152 This directive can appear in the following resources:
2153 \begin{bitemize}
2154 \item \bfilename{bacula-dir.conf}: Director resource
2155 \item \bfilename{bacula-fd.conf}: Client (or FileDaemon) resource
2156 \item \bfilename{bacula-sd.conf}: Storage resource
2157 \item \bfilename{bconsole.conf}: Console resource
2158 \item \bfilename{bat.conf}: Console resource
2159 \end{bitemize}
2160 \smallskip
2161
2162 In many cases, the volume of data transmitted across the communications line can be reduced by a factor of three when this directive is \bdefaultvalue{enabled}. In the case that the compression is not effective, \bacula{} turns it off on a record by record basis.
2163 \smallskip
2164
2165 If you are backing up data that is already compressed the comm line compression will not be effective, and you are likely to end up with an average compression ratio that is very small.
2166 In this case, \bacula{} reports \bvalue{None} in the Job report.
2167
2168
2169 \subsection{Read Only Storage Devices}
2170 This version of \bacula{} allows you to define a Storage deamon device
2171 to be read-only. If the \textbf{Read Only} directive is specified and
2172 enabled, the drive can only be used for read operations.
2173 The \textbf{Read Only} directive can be defined in any \bfilename{bacula-sd.conf}
2174 Device resource, and is most useful for reserving one or more
2175 drives for restores. An example is:
2176
2177 \begin{bVerbatim}
2178 Read Only = yes
2179 \end{bVerbatim}
2180
2181 \subsection{Catalog Performance Improvements}
2182 There is a new \bacula{} database format (schema) in this version
2183 of \bacula{} that eliminates the FileName table by placing the
2184 Filename into the File record of the File table.
2185 This substantiallly improves performance,
2186 particularly for large (1GB or greater) databases.
2187
2188 % waa - 20150317 - Is 1GB _really_ considered to be a large database? Do we mean to say 100GB??
2189
2190 The \btool{update\_xxx\_catalog} script will automatically update the
2191 \bacula{} database format, but you should realize that for
2192 very large databases (greater than 1GB), it may take some
2193 time, and there are several different options for doing the
2194 update:
2195 \begin{benumerate}
2196 \item Shudown the database and update it
2197 \item Update the database while production jobs are running.
2198 \end{benumerate}
2199 See the \baculasystems{} White Paper ``Migration-to-6.6'' on this subject.
2200 \smallskip
2201
2202 This database format change can provide very significant improvements in
2203 the speed of metadata insertion into the database, and in some cases
2204 (backup of large email servers) can significantly reduce the size of the
2205 database.
2206
2207 \subsection{Plugin Restore Options}
2208 This version of \bacula{} permits user configuration of Plugins at restore
2209 time. For example, it is now possible to choose the \texttt{datastore}
2210 where your VMware image will be restored, or to choose
2211 \btool{pg\_restore} options directly. See specific Plugin whitepapers
2212 for more information about new restore options.
2213
2214 The restore options, if implemented in a plugin, will be
2215 presented to you during initiation of a restore either by
2216 command line or if available by a \acs{GUI} such as \bweb{}. For
2217 examples of the command line interface and
2218 the \acs{GUI} interface, please see below:
2219
2220 \begin{bVerbatim}
2221 *run restore jobid=11766
2222 Run Restore job
2223 JobName: RestoreFiles
2224 Bootstrap: /tmp/regress/working/my-dir.restore.1.bsr
2225 Where: /tmp/regress/tmp/bacula-restores
2226 ...
2227 Plugin Options: *None*
2228 OK to run? (yes/mod/no): mod
2229 Parameters to modify:
2230 1: Level
2231 ...
2232 13: Plugin Options
2233 Select parameter to modify (1-13): 13
2234 Automatically selected : vsphere: host=squeeze2
2235 Plugin Restore Options
2236 datastore: *None*
2237 restore_host: *None*
2238 new_hostname: *None*
2239 Use above plugin configuration? (yes/mod/no): mod
2240 You have the following choices:
2241 1: datastore (Datastore to use for restore)
2242 2: restore_host (ESX host to use for restore)
2243 3: new_hostname (Restore host to specified name)
2244 Select parameter to modify (1-3): 3
2245 Please enter a value for new_hostname: test
2246 Plugin Restore Options
2247 datastore: *None*
2248 restore_host: *None*
2249 new_hostname: test
2250 Use above plugin configuration? (yes/mod/no): yes
2251 \end{bVerbatim}
2252
2253 Or via the \bweb{} restore interface (see Fig \vref{fig:bms-restore})
2254
2255 \bimageH{vsphere-restore-gui}{Choose datastore, ESXi or hostname at restore time}{fig:bms-restore}
2256
2257 \subsection{Alldrives Plugin Improvements}
2258 The \texttt{alldrives} plugin simplifies the FileSet creation of
2259 Windows Clients by automatically generating a FileSet which includes all
2260 local drives.
2261 \smallskip
2262
2263 The \texttt{alldrives} plugin now accepts the \texttt{snapshot} option that
2264 generates snapshots for all local Windows drives, but without
2265 explicitly adding them to the FileSet. It may be combined with the
2266 \acs{VSS} plugin. For example:
2267
2268 \begin{bVerbatim}
2269 FileSet {
2270 ...
2271 Include {
2272 Plugin = "vss:/@MSSQL/"
2273 Plugin = "alldrives: snapshot" # should be placed after vss plugin
2274 }
2275 }
2276 \end{bVerbatim}
2277
2278 \subsection{New Truncate Command}
2279 We have added a new \bcommandname{truncate} command to bconsole which
2280 will truncate a volume if the volume is purged, and if
2281 the volume is also marked \textbf{Action On Purge = Truncate}.
2282 This feature was originally added in \bacula{} version 5.0.1,
2283 but the mechanism for actually doing the truncate required
2284 the user to enter a complicated command such as:
2285
2286 \begin{bVerbatim}
2287 purge volume action=truncate storage=File pool=Default
2288 \end{bVerbatim}
2289
2290 The above command is now simplified to be:
2291
2292 \begin{bVerbatim}
2293 truncate storage=File pool=Default
2294 \end{bVerbatim}
2295
2296 \section{\baculaenterprise{} 6.4.x}
2297
2298 The following features were added during the 6.4.x life cycle.
2299
2300 \subsection{SAP Plugin}
2301
2302 The \baculaenterprise{} SAP Plugin is designed to implement the official SAP
2303 Backint interface to simplify the backup and restore procedure through your
2304 traditional SAP database tools. See SAP-Backint whitepaper for more
2305 information.
2306
2307 \subsection{Oracle SBT Plugin}
2308
2309 By default, the Oracle backup Manager, \acs{RMAN}, sends all backups to an operating
2310 system specific directory on disk. You can also configure \acs{RMAN} to make backups
2311 to media such as tape using the SBT module. \bacula{} will act as Media Manager, and
2312 the data will be transfered directly from \acs{RMAN} to \bacula{}. See Oracle Plugin
2313 whitepaper for more information.
2314
2315 \subsection{\mysql{} Plugin}
2316
2317 The \mysql{} plugin is designed to simplify the backup and restore of your \mysql{}
2318 database, the backup administrator doesn't need to know about the internals of \mysql{}
2319 backup techniques or how to write complex scripts. This plugin will
2320 automatically backup essential information such as configurations and user
2321 definitions. The \mysql{} plugin supports both dump (with support for Incremental
2322 backup) and binary backup techniques. See the \mysql{} Plugin whitepaper for more
2323 information.
2324
2325 \section{\baculaenterprise{} 6.4.0}
2326 \subsection{Deduplication Optimized Volumes}
2327 This version of \bacula{} includes a new alternative (or additional)
2328 volume format that optimizes the placement of files so
2329 that an underlying deduplicating filesystem such as \acs{ZFS}
2330 can optimally deduplicate the backup data that is written
2331 by \bacula{}. These are called Deduplication Optimized Volumes
2332 or Aligned Volumes for short. The details of how to use this
2333 feature and its considerations are in the \baculasystems{}
2334 Deduplication Optimized Volumes whitepaper.
2335
2336 \subsection{Migration/Copy/VirtualFull Performance Enhancements}
2337 The \bacula{} Storage daemon now permits multiple jobs to simultaneously read
2338 from the same disk volume which gives substantial performance enhancements when
2339 running Migration, Copy, or VirtualFull jobs that read disk volumes. Our
2340 testing shows that when running multiple simultaneous jobs, the jobs can
2341 finish up to ten times faster with this version of \bacula{}. This is
2342 built-in to the Storage daemon, so it happens automatically and
2343 transparently.
2344
2345 \subsection{VirtualFull Backup Consolidation Enhancements}
2346 By default \bacula{} selects jobs automatically for a VirtualFull backup.
2347 However, you may want to create the virtual backup based on a
2348 particular backup (point in time) that exists.
2349
2350 For example, if you have the following backup Jobs in your catalog:
2351 \begin{bVerbatim}
2352 +-------+---------+-------+----------+----------+-----------+
2353 | JobId | Name | Level | JobFiles | JobBytes | JobStatus |
2354 +-------+---------+-------+----------+----------+-----------+
2355 | 1 | Vbackup | F | 1754 | 50118554 | T |
2356 | 2 | Vbackup | I | 1 | 4 | T |
2357 | 3 | Vbackup | I | 1 | 4 | T |
2358 | 4 | Vbackup | D | 2 | 8 | T |
2359 | 5 | Vbackup | I | 1 | 6 | T |
2360 | 6 | Vbackup | I | 10 | 60 | T |
2361 | 7 | Vbackup | I | 11 | 65 | T |
2362 | 8 | Save | F | 1758 | 50118564 | T |
2363 +-------+---------+-------+----------+----------+-----------+
2364 \end{bVerbatim}
2365
2366 and you want to consolidate only the first 3 jobs and create a
2367 virtual backup equivalent to Job 1 + Job 2 + Job 3, you will use
2368 \texttt{jobid=3} in the \bcommandname{run} command, then \bacula{} will select the
2369 previous Full backup, the previous Differential (if any) and all subsequent
2370 Incremental jobs.
2371
2372 \begin{bVerbatim}
2373 run job=Vbackup jobid=3 level=VirtualFull
2374 \end{bVerbatim}
2375
2376 If you want to consolidate a specific job list, you must specify the exact
2377 list of jobs to merge in the \bcommandname{run} command line. For example, to consolidate
2378 the last Differential and all subsequent Incrementals, you will use
2379 \texttt{jobid=4,5,6,7} or \texttt{jobid=4-7} on the \bcommandname{run} command line. Because one
2380 of the Jobs in the list is a Differential backup, \bacula{} will set the new job
2381 level to Differential. If the list is composed of only Incremental jobs,
2382 the new job will have its level set to Incremental.
2383
2384 \begin{bVerbatim}
2385 run job=Vbackup jobid=4-7 level=VirtualFull
2386 \end{bVerbatim}
2387
2388 When using this feature, \bacula{} will automatically discard jobs that are
2389 not related to the current Job. For example, specifying
2390 \texttt{jobid=7,8}, \bacula{} will discard JobId 8 because it is not
2391 part of the same backup Job.
2392
2393 We do not recommend it, but if you really want to consolidate jobs that have
2394 different names (so probably different clients, filesets, etc...), you must
2395 use \texttt{alljobid=} keyword instead of \texttt{jobid=}.
2396
2397 \begin{bVerbatim}
2398 run job=Vbackup alljobid=1-3,6-8 level=VirtualFull
2399 \end{bVerbatim}
2400
2401 \subsection{New Prune ``Expired'' Volume Command}
2402 In \baculaenterprise{} 6.4, it is now possible to prune all volumes
2403 (from a pool, or globally) that are ``expired''. This option can be
2404 scheduled after or before the backup of the catalog and can be
2405 combined with the \texttt{Truncate On Purge} option. The \bcommandname{prune expired volme} command may
2406 be used instead of the \btool{manual\_prune.pl} script.
2407
2408 \begin{bVerbatim}
2409 * prune expired volume
2410
2411 * prune expired volume pool=FullPool
2412 \end{bVerbatim}
2413
2414 To schedule this option automatically, it can be added to the Catalog backup job
2415 definition.
2416
2417 \begin{bVerbatim}
2418 Job {
2419 Name = CatalogBackup
2420 ...
2421 RunScript {
2422 Console = "prune expired volume yes"
2423 RunsWhen = Before
2424 }
2425 }
2426 \end{bVerbatim}
2427
2428 \section{\baculaenterprise{} 6.2.3}
2429
2430 \subsection{New Job Edit Codes \%P \%C}
2431 In various places such as RunScripts, you have now access to \%P to get the
2432 current \bacula{} process ID (PID) and \%C to know if the current job is a
2433 cloned job.
2434
2435 \section{\baculaenterprise{} 6.2.0}
2436
2437 \subsection{\bweb{} \bacula{} Configuration GUI}
2438 In \baculaenterprise{} version 6.2, the \bweb{} Management Suite integrates a
2439 \bacula{} configuration GUI module which is designed to help you create and
2440 modify the \bacula{} configuration files such as \bfilename{bacula-dir.conf},
2441 \bfilename{bacula-sd.conf}, \bfilename{bacula-fd.conf} and \bfilename{bconsole.conf}.
2442
2443 The \bweb{} Management Suite offers a number of Wizards which support the
2444 Administrator in his daily work. The wizards provide a step by step set of
2445 required actions that graphically guide the Administrator to perform quick
2446 and easy creation and modification of configuration files.
2447
2448 \bweb{} also provides diagnostic tools that enable the Administrator to check
2449 that the Catalog Database is well configured, and that \bweb{} is installed
2450 properly.
2451
2452 The new Online help mode displays automatic help text suggestions when the
2453 user searches data types.
2454
2455 \bimageH{bweb-config-screen}{Configuration with \bweb{} Management Suite}{fig:BwebBconfigScreen}
2456 \smallskip
2457
2458 This project was funded by \baculasystems{} and is available with the \bee{}.
2459
2460 \subsection{Performance Improvements}
2461 \baculaenterprise{} 6.2 has a number of new performance improvements:
2462
2463 \begin{bitemize}
2464 \item An improved way of storing \bacula{} Resources (as defined in
2465 the .conf files). This new handling permits much faster loading or
2466 reloading of the conf files, and permits larger numbers of resources.
2467
2468 \item Improved performance when inserting large numbers of files in
2469 the DB catalog by breaking the insertion into smaller chunks, thus
2470 allowing better sharing when running multiple simultaneous jobs.
2471
2472 \item Performance enhancements in \acs{BVFS} concerning eliminating
2473 duplicate path records.
2474
2475 \item Performance improvement when getting Pool records.
2476
2477 \item Pruning performance enhancements.
2478 \end{bitemize}
2479
2480 \subsection{Enhanced Status and Error Messages}
2481 We have enhanced the Storage daemon status output to be more
2482 readable. This is important when there are a large number of
2483 devices. In addition to formatting changes, it also includes more
2484 details on which devices are reading and writing.
2485
2486 A number of error messages have been enhanced to have more specific
2487 data on what went wrong.
2488
2489 If a file changes size while being backed up the old and new size
2490 are reported.
2491
2492 \subsection{WinBMR 3}
2493 The Windows \ac{BMR} plugin enables you to do safe, reliable
2494 Disaster Recovery with \bee{} on Windows and allows you to
2495 get critical systems up and running again quickly. The \enterpriseedition{}
2496 Windows \acs{BMR} is a toolkit that allows the Administrator to perform the restore
2497 of a complete operating system to the same or similar hardware without actually
2498 going through the operating system's installation procedure.
2499 \smallskip
2500
2501 The WinBMR 3 version is a major rewrite of the product that support all x86
2502 Windows versions and technologies. Especially \acs{UEFI} and secure boot systems. The
2503 WinBMR 3 File Daemon plugin is now part of the plugins included with the \bacula{}
2504 File Daemon package. The rescue CD or USB key is available separately.
2505
2506 \subsection{Miscellaneous New Features}
2507 \begin{bitemize}
2508 \item Allow unlimited line lengths in .conf files (previously limited
2509 to 2000 characters).
2510
2511 \item Allow \bfilename{/dev/null} in ChangerCommand to indicated a Virtual Autochanger.
2512
2513 \item Add a \texttt{{-}{-}fileprune} option to the \btool{manual\_prune.pl} script.
2514
2515 \item Add a \texttt{-m} option to \btool{make\_catalog\_backup.pl} to do maintenance
2516 on the catalog.
2517
2518 \item Safer code that cleans up the working directory when starting
2519 the daemons. It limits what files can be deleted, hence enhances
2520 security.
2521
2522 \item Added a new \bcommandname{.ls} command in bconsole to permit browsing a client's
2523 filesystem.
2524
2525 \item Fixed a number of bugs, includes some obscure seg faults, and a
2526 race condition that occurred infrequently when running Copy, Migration,
2527 or Virtual Full backups.
2528
2529 \item Included a new vSphere library version, which will hopefully
2530 fix some of the more obscure bugs.
2531
2532 \item Upgraded to a newer version of Qt4 for \bat{}. All indications
2533 are that this will improve \bat{}'s stability on Windows machines.
2534
2535 \item The Windows installers now detect and refuse to install on
2536 an \acs{OS} that does not match the 32/64 bit value of the installer.
2537 \end{bitemize}
2538
2539
2540 \section{\baculaenterprise{} 6.0.6}
2541
2542 \subsection{Incremental Accelerator Plugin for NetApp}
2543
2544 The Incremental Accelerator for NetApp Plugin is designed to simplify the
2545 backup and restore procedure of your NetApp \acs{NAS} hosting a huge number of files.
2546 \smallskip{}
2547
2548 When using the NetApp \acs{HFC} Plugin, \baculaenterprise{} will query the
2549 NetApp device to get the list of all files modified since the last backup
2550 instead of having to walk through the entire filesystem. Once \bacula{} have the
2551 list of all files to back's up, it will use a standard network share (such as
2552 \acs{NFS} or \acs{CIFS}) to access files.
2553 \smallskip
2554
2555 This project was funded by \baculasystems{} and is available with the \bee{}.
2556
2557 \subsection{\postgresql{} Plugin}
2558
2559 The \postgresql{} plugin is designed to simplify the backup and restore procedure
2560 of your \postgresql{} cluster, the backup administrator doesn't need to learn about
2561 internals of \postgresql{} backup techniques or write complex scripts. The plugin
2562 will automatically take care for you to backup essential information such as
2563 configuration, users definition or tablespaces. The \postgresql{} plugin supports
2564 both dump and \ac{PITR} backup techniques.
2565 \smallskip
2566
2567 This project was funded by \baculasystems{} and is available with the \bee{}.
2568
2569 \subsection{Maximum Reload Requests}
2570
2571 The new Director directive \texttt{Maximum Reload Requests} permits to
2572 configure the number of reload requests that can be done while jobs are
2573 running.
2574
2575 \begin{bVerbatim}
2576 Director {
2577 Name = localhost-dir
2578 Maximum Reload Requests = 64
2579 ...
2580 }
2581 \end{bVerbatim}
2582
2583 \subsection{FD Storage Address}
2584
2585 When the Director is behind a \acs{NAT}, in a \acs{WAN} area, to connect to
2586 % the FileDaemon or
2587 the StorageDaemon, the Director uses an ``external'' ip address,
2588 and the FileDaemon should use an ``internal'' IP address to contact the
2589 StorageDaemon.
2590
2591 The normal way to handle this situation is to use a canonical name such as
2592 ``storage-server'' that will be resolved on the Director side as the \acs{WAN}
2593 address and on the Client side as the \acs{LAN} address. This is now possible to
2594 configure this parameter using the new directive \bdirectivename{FDStorageAddress} in
2595 the Storage or Client resource.
2596
2597
2598 \bimageH{BackupOverWan1}{Backup Over WAN}{figbs6:fdstorageaddress}
2599 % \label{blb:fig:fdstorageaddress}
2600
2601 \begin{bVerbatim}
2602 Storage {
2603 Name = storage1
2604 Address = 65.1.1.1
2605 FD Storage Address = 10.0.0.1
2606 SD Port = 9103
2607 ...
2608 }
2609 \end{bVerbatim}
2610
2611 % # or in the Client resouce
2612 %
2613
2614 \begin{bVerbatim}
2615 Client {
2616 Name = client1
2617 Address = 65.1.1.2
2618 FD Storage Address = 10.0.0.1
2619 FD Port = 9102
2620 ...
2621 }
2622 \end{bVerbatim}
2623
2624 Note that using the Client \bdirectivename{FDStorageAddress} directive will not allow
2625 to use multiple Storage Daemon, all Backup or Restore requests will be sent to
2626 the specified \bdirectivename{FDStorageAddress}.
2627
2628 \subsection{Maximum Concurrent Read Jobs}
2629 This is a new directive that can be used in the \bfilename{bacula-dir.conf} file
2630 in the Storage resource. The main purpose is to limit the number
2631 of concurrent Copy, Migration, and VirtualFull jobs so that
2632 they don't monopolize all the Storage drives causing a deadlock situation
2633 where all the drives are allocated for reading but none remain for
2634 writing. This deadlock situation can occur when running multiple
2635 simultaneous Copy, Migration, and VirtualFull jobs.
2636 \smallskip
2637
2638 The default value is set to \bdefaultvalue{0} (zero), which means there is no
2639 limit on the number of read jobs. Note, limiting the read jobs
2640 does not apply to Restore jobs, which are normally started by
2641 hand. A reasonable value for this directive is one half the number
2642 of drives that the Storage resource has rounded down. Doing so,
2643 will leave the same number of drives for writing and will generally
2644 avoid over committing drives and a deadlock.
2645
2646
2647 \section{\baculaenterprise{} 6.0.4}
2648
2649 \subsection{VMWare vSphere VADP Plugin}
2650
2651 The \baculaenterprise{} vSphere plugin provides virtual
2652 machine bare metal recovery, while the backup at the guest level simplify data
2653 protection of critical applications.
2654
2655 The plugin integrates the VMware's \ac{CBT} technology to
2656 ensure only blocks that have changed since the initial Full, and/or the last
2657 Incremental or Differential Backup are sent to the current Incremental or
2658 Differential backup stream to give you more efficient backups and reduced
2659 network load.
2660
2661 \subsection{Oracle \acs{RMAN} Plugin}
2662
2663 The \baculaenterprise{} Oracle Plugin is designed to simplify the backup and
2664 restore procedure of your Oracle Database instance, the backup administrator
2665 don't need to learn about internals of Oracle backup techniques or write
2666 complex scripts. The \baculaenterprise{} Oracle plugin supports both dump and
2667 \ac{PITR} with \acs{RMAN} backup techniques.
2668
2669
2670 \section{\baculaenterprise{} 6.0.2}
2671
2672 To make \bacula{} function properly with multiple Autochanger definitions, in
2673 the Director's configuration, you must adapt your \bfilename{bacula-dir.conf}
2674 \textbf{Storage} directives.
2675 \smallskip
2676
2677 Each autochanger that you have defined in an \textbf{Autochanger}
2678 resource in the Storage daemon's \bfilename{bacula-sd.conf} file,
2679 must have a corresponding \textbf{Autochanger} resource defined
2680 in the Director's \bfilename{bacula-dir.conf} file. Normally you will
2681 already have a \textbf{Storage} resource that points to the
2682 Storage daemon's \textbf{Autochanger} resource. Thus you need
2683 only to change the name of the \textbf{Storage} resource to
2684 \textbf{Autochanger}. In addition the \textbf{Autochanger = yes}
2685 directive is not needed in the Director's \textbf{Autochanger}
2686 resource, since the resource name is \textbf{Autochanger}, the
2687 Director already knows that it represents an autochanger.
2688 \smallskip
2689
2690 In addition to the above change (\textbf{Storage} to \textbf{Autochanger}),
2691 you must modify any additional \textbf{Storage} resources that correspond
2692 to devices that are part of the \textbf{Autochanger} device.
2693 Instead of the previous \textbf{Autochanger = yes} directive they
2694 should be modified to be \textbf{Autochanger = xxx} where you
2695 replace the \textbf{xxx} with the name of the Autochanger.
2696 \smallskip
2697
2698 For example, in the \bfilename{bacula-dir.conf} file:
2699
2700 \begin{bVerbatim}
2701 Autochanger { # New resource
2702 Name = Changer-1
2703 Address = cibou.company.com
2704 SDPort = 9103
2705 Password = "xxxxxxxxxx"
2706 Device = LTO-Changer-1
2707 Media Type = LTO-4
2708 Maximum Concurrent Jobs = 50
2709 }
2710
2711 Storage {
2712 Name = Changer-1-Drive0
2713 Address = cibou.company.com
2714 SDPort = 9103
2715 Password = "xxxxxxxxxx"
2716 Device = LTO4_1_Drive0
2717 Media Type = LTO-4
2718 Maximum Concurrent Jobs = 5
2719 Autochanger = Changer-1 # New directive
2720 }
2721
2722 Storage {
2723 Name = Changer-1-Drive1
2724 Address = cibou.company.com
2725 SDPort = 9103
2726 Password = "xxxxxxxxxx"
2727 Device = LTO4_1_Drive1
2728 Media Type = LTO-4
2729 Maximum Concurrent Jobs = 5
2730 Autochanger = Changer-1 # New directive
2731 }
2732
2733 ...
2734 \end{bVerbatim}
2735
2736 Note that Storage resources \textbf{Changer-1-Drive0} and
2737 \textbf{Changer-1-Drive1} are not required since they make
2738 up part of an autochanger, and normally, Jobs refer only
2739 to the Autochanger resource.
2740 However, by referring to those
2741 Storage definitions in a Job, you will use only
2742 the indicated drive. This is not normally what
2743 you want to do, but it is very useful and often used
2744 for reserving a drive for restores. See the Storage daemon
2745 example .conf below and the use of \textbf{AutoSelect = no}.
2746
2747 So, in summary, the changes are:
2748 \begin{bitemize}
2749 \item Change \textbf{Storage} to \textbf{Autochanger} in the LTO4 resource.
2750 \item Remove the \textbf{Autochanger = yes} from the \textbf{Autochanger}
2751 LTO4 resource.
2752 \item Change the \textbf{Autochanger = yes} in each of the \textbf{Storage}
2753 device that belong to the \textbf{Autochanger} to point to the
2754 \textbf{Autochanger} resource with for the example above the
2755 directive \textbf{Autochanger = LTO4}.
2756 \end{bitemize}
2757
2758 \section{\baculaenterprise{} 6.0.0}
2759
2760 \subsection{Incomplete Jobs}
2761 During a backup, if the Storage daemon experiences disconnection
2762 with the File daemon during backup (normally a comm line problem
2763 or possibly an FD failure), under conditions that the SD determines
2764 to be safe it will make the failed job as Incomplete rather than
2765 failed. This is done only if there is sufficient valid backup
2766 data that was written to the Volume. The advantage of an Incomplete
2767 job is that it can be restarted by the new bconsole \bcommandname{restart}
2768 command from the point where it left off rather than from the
2769 beginning of the jobs as is the case with a cancel.
2770
2771 \subsection{The \bcommandname{stop} Command}
2772 \bacula{} has been enhanced to provide a \bcommandname{stop} command,
2773 very similar to the \bcommandname{cancel} command with the main difference
2774 that the Job that is stopped is marked as Incomplete so that
2775 it can be restarted later by the \bcommandname{restart} command where
2776 it left off (see below). The \bcommandname{stop} command with no
2777 arguments, will like the cancel command, prompt you with the
2778 list of running jobs allowing you to select one, which might
2779 look like the following:
2780
2781 \begin{bVerbatim}
2782 *stop
2783 Select Job:
2784 1: JobId=3 Job=Incremental.2012-03-26_12.04.26_07
2785 2: JobId=4 Job=Incremental.2012-03-26_12.04.30_08
2786 3: JobId=5 Job=Incremental.2012-03-26_12.04.36_09
2787 Choose Job to stop (1-3): 2
2788 2001 Job "Incremental.2012-03-26_12.04.30_08" marked to be stopped.
2789 3000 JobId=4 Job="Incremental.2012-03-26_12.04.30_08" marked to be stopped.
2790 \end{bVerbatim}
2791
2792 \subsection{The \bcommandname{restart} Command}
2793 The new \bcommandname{restart} command allows console users to restart
2794 a canceled, failed, or incomplete Job. For canceled and failed
2795 Jobs, the Job will restart from the beginning. For incomplete
2796 Jobs the Job will restart at the point that it was stopped either
2797 by a stop command or by some recoverable failure.
2798 \smallskip
2799
2800 If you enter the \bcommandname{restart} command in bconsole, you will get the
2801 following prompts:
2802
2803 \begin{bVerbatim}
2804 *restart
2805 You have the following choices:
2806 1: Incomplete
2807 2: Canceled
2808 3: Failed
2809 4: All
2810 Select termination code: (1-4):
2811 \end{bVerbatim}
2812
2813 If you select the \textbf{All} option, you may see something like:
2814
2815 \begin{bVerbatim}
2816 Select termination code: (1-4): 4
2817 +-------+-------------+---------------------+------+-------+----------+-----------+-----------+
2818 | jobid | name | starttime | type | level | jobfiles | jobbytes | jobstatus |
2819 +-------+-------------+---------------------+------+-------+----------+-----------+-----------+
2820 | 1 | Incremental | 2012-03-26 12:15:21 | B | F | 0 | 0 | A |
2821 | 2 | Incremental | 2012-03-26 12:18:14 | B | F | 350 | 4,013,397 | I |
2822 | 3 | Incremental | 2012-03-26 12:18:30 | B | F | 0 | 0 | A |
2823 | 4 | Incremental | 2012-03-26 12:18:38 | B | F | 331 | 3,548,058 | I |
2824 +-------+-------------+---------------------+------+-------+----------+-----------+-----------+
2825 Enter the JobId list to select:
2826 \end{bVerbatim}
2827
2828 Then you may enter one or more JobIds to be restarted, which may
2829 take the form of a list of JobIds separated by commas, and/or JobId
2830 ranges such as \textbf{1-4}, which indicates you want to restart JobIds
2831 1 through 4, inclusive.
2832
2833 \subsection{Support for Exchange Incremental Backups}
2834 The \baculaenterprise{} version 6.0 \acs{VSS} plugin now supports
2835 Full and Incremental backups for Exchange. We strongly
2836 recommend that you do not attempt to run Differential jobs with
2837 Exchange as it is likely to produce a situation where restores
2838 will no longer select the correct jobs, and thus the
2839 Windows Exchange \acs{VSS} writer will fail when applying log files.
2840 There is a \baculasystems{} Enterprise white paper that provides
2841 the details of backup and restore of Exchange 2010 with the
2842 \bacula{} \acs{VSS} plugin.
2843 \smallskip
2844
2845 Restores can be done while Exchange is running, but you
2846 must first unmount (dismount in Microsoft terms) any database
2847 you wish to restore and explicitly mark them to permit a
2848 restore operation (see the white paper for details).
2849 \smallskip
2850
2851 This project was funded by \baculasystems{} and is available with the \bee{}.
2852
2853 \subsection{Support for \mssql{} Block Level Backups}
2854 With the addition of block level backup support to the
2855 \baculaenterprise{} \acs{VSS} \mssql{} component, you can now do
2856 Differential backups in addition to Full backups.
2857 Differential backups use Microsoft's partial block backup
2858 (a block differencing or deduplication that we call Delta).
2859 This partial block backup permits backing up only those
2860 blocks that have changed. Database restores can be made while
2861 the \mssql{} server is running, but any databases selected for
2862 restore will be automatically taken offline by the \mssql{}
2863 server during the restore process.
2864
2865 Incremental backups for \mssql{} are not support by
2866 Microsoft. We strongly recommend that you not perform Incremental
2867 backups with \mssql{} as they will probably produce a situation
2868 where restore will no longer work correctly.
2869 \smallskip
2870
2871 We are currently working on producing a white paper that will give more
2872 details of backup and restore with \mssql{}. One point to note is that during
2873 a restore, you will normally not want to restore the \textbf{master} database.
2874 You must exclude it from the backup selections that you have made or the
2875 restore will fail.
2876 \smallskip
2877
2878 It is possible to restore the \textbf{master} database, but you must
2879 first shutdown the \mssql{} server, then you must perform special
2880 recovery commands. Please see Microsoft documentation on how
2881 to restore the master database.
2882 \smallskip
2883
2884 This project was funded by \baculasystems{} and is available with the \bee{}.
2885
2886
2887 \subsection{Job Bandwidth Limitation}
2888
2889 The new \textbf{Job Bandwidth Limitation} directive may be added to the File
2890 daemon's and/or Director's configuration to limit the bandwidth used by a
2891 Job on a Client. It can be set in the File daemon's conf file for all Jobs
2892 run in that File daemon, or it can be set for each Job in the Director's
2893 conf file. The speed is always specified in bytes per second.
2894
2895 For example:
2896 \begin{bVerbatim}
2897 FileDaemon {
2898 Name = localhost-fd
2899 Working Directory = /some/path
2900 Pid Directory = /some/path
2901 ...
2902 Maximum Bandwidth Per Job = 5Mb/s
2903 }
2904 \end{bVerbatim}
2905
2906 The above example would cause any jobs running with the FileDaemon to not
2907 exceed 5 megabytes per second of throughput when sending data to the
2908 Storage Daemon. Note, the speed is always specified in \bhighlight{bytes per second}
2909 (not in bits per second), and the case (upper/lower) of the specification
2910 characters is ignored (i.e. 1MB/s = 1Mb/s).
2911
2912 You may specify the following speed parameter modifiers:
2913 k/s (1,024 bytes per second), kb/s (1,000 bytes per second),
2914 m/s (1,048,576 bytes per second), or mb/s (1,000,000 bytes per second).
2915
2916 For example:
2917 \begin{bVerbatim}
2918 Job {
2919 Name = locahost-data
2920 FileSet = FS_localhost
2921 Accurate = yes
2922 ...
2923 Maximum Bandwidth = 5Mb/s
2924 ...
2925 }
2926 \end{bVerbatim}
2927
2928 The above example would cause Job \texttt{localhost-data} to not exceed 5MB/s
2929 of throughput when sending data from the File daemon to the Storage daemon.
2930
2931 A new console command \bcommandname{setbandwidth} permits to set dynamically the
2932 maximum throughput of a running Job or for future jobs of a Client.
2933
2934 \begin{bVerbatim}
2935 * setbandwidth limit=1000 jobid=10
2936 \end{bVerbatim}
2937
2938 Number of bytes can be expressed using modifiers mentioned above (k/s, kb/s, m/s or mb/s).
2939 \medskip
2940
2941 This project was funded by \baculasystems{} and is available in \bee{}.
2942
2943 \subsection{Incremental/Differential Block Level Difference Backup}
2944
2945 The new \texttt{delta} Plugin is able to compute and apply signature-based file
2946 differences. It can be used to backup only changes in a big binary file like
2947 Outlook PST, VirtualBox/VMware images or database files.
2948
2949 It supports both Incremental and Differential backups and stores signatures
2950 database in the File Daemon working directory. This plugin is available on all
2951 platform including Windows 32 and 64bit.
2952
2953 Accurate option should be turned on in the Job resource.
2954 \begin{bVerbatim}
2955 Job {
2956 Accurate = yes
2957 FileSet = DeltaFS
2958 ...
2959 }
2960
2961 FileSet {
2962 Name = DeltaFS
2963 ...
2964 Include {
2965 # Specify one file
2966 Plugin = "delta:/home/eric/.VirtualBox/HardDisks/lenny-i386.vdi"
2967 }
2968 }
2969
2970 FileSet {
2971 Name = DeltaFS-Include
2972 ...
2973 Include {
2974 Options {
2975 Compression = GZIP1
2976 Signature = MD5
2977 Plugin = delta
2978 }
2979 # Use the Options{} filtering and options
2980 File = /home/user/.VirtualBox
2981 }
2982 }
2983 \end{bVerbatim}
2984
2985 Please contact \baculasystems{} support to get Delta Plugin specific
2986 documentation.
2987 \medskip
2988
2989 This project was funded by \baculasystems{} and is available with the \bee{}.
2990
2991 \subsection{SAN Shared Tape Storage Plugin}
2992
2993 The problem with backing up multiple servers at the same time to the
2994 same tape library (or autoloader) is that if both servers access the
2995 same tape drive same time, you will very likely get data corruption.
2996 This is where the \baculasystems{} shared tape storage plugin comes into play. The
2997 plugin ensures that only one server at a time can connect to each device
2998 (tape drive) by using the SPC-3 SCSI reservation protocol. Please contact
2999 \baculasystems{} support to get SAN Shared Storage Plugin specific
3000 documentation.
3001 \medskip
3002
3003 This project was funded by \baculasystems{} and is available with \bee{}.
3004
3005 \subsection{Advanced Autochanger Usage}
3006
3007 The new \texttt{Shared Storage} Director's directive is a \baculaenterprise{}
3008 feature that allows you to share volumes between different Storage
3009 resources. This directive should be used \textbf{only} if all
3010 \texttt{Media Type} are correctly set across all Devices.
3011
3012 The \texttt{Shared Storage} directive should be used when using the SAN
3013 Shared Storage plugin or when accessing from the Director Storage resources
3014 directly to Devices of an Autochanger.
3015
3016 When sharing volumes between different Storage resources, you will
3017 need also to use the \btool{reset-storageid} script before using the
3018 \bcommandname{update slots} command. This script can be scheduled once a day in
3019 an Admin job.
3020
3021 \begin{bVerbatim}
3022 $ /opt/bacula/scripts/reset-storageid MediaType StorageName
3023 $ bconsole
3024 * update slots storage=StorageName drive=0
3025 \end{bVerbatim}
3026
3027 Please contact \baculasystems{} support to get help on this advanced
3028 configuration.
3029 \medskip
3030
3031 This project was funded by \baculasystems{} and is available with \bee{}.
3032 \medskip
3033
3034 \textbf{The reset-storageid procedure is no longer required when using
3035 the appropriate Autochanger configuration in the Director configuration
3036 side.}
3037
3038 \subsection{Enhancement of the \acs{NDMP} Plugin}
3039
3040 The previous \acs{NDMP} Plugin 4.0 was fully supporting only the NetApp hardware, the
3041 new \acs{NDMP} Plugin should now be able to support all \acs{NAS} vendors with the
3042 \texttt{volume\_format} plugin command option.
3043
3044 On some \acs{NDMP} devices such as Celera or Blueray, the administrator can use arbitrary
3045 volume structure name, ex:
3046
3047 \begin{bVerbatim}
3048 /dev/volume_home
3049 /rootvolume/volume_tmp
3050 /VG/volume_var
3051 \end{bVerbatim}
3052
3053 The \acs{NDMP} plugin should be aware of the structure organization in order to
3054 detect if the administrator wants to restore in a new volume
3055 (\texttt{where=/dev/vol\_tmp}) or inside a subdirectory of the targeted volume
3056 (\texttt{where=/tmp}).
3057
3058 \begin{bVerbatim}
3059 FileSet {
3060 Name = NDMPFS
3061 ...
3062 Include {
3063 Plugin = "ndmp:host=nasbox user=root pass=root file=/dev/vol1 volume_format=/dev/"
3064 }
3065 }
3066 \end{bVerbatim}
3067
3068 Please contact \baculasystems{} support to get \acs{NDMP} Plugin specific
3069 documentation.
3070 \medskip
3071
3072 This project was funded by \baculasystems{} and is available with the \bee{}
3073
3074 \subsection{Always Backup a File}
3075
3076 When the Accurate mode is turned on, you can decide to always backup a file
3077 by using then new \textbf{A} Accurate option in your FileSet. For example:
3078
3079 \begin{bVerbatim}
3080 Job {
3081 Name = ...
3082 FileSet = FS_Example
3083 Accurate = yes
3084 ...
3085 }
3086
3087 FileSet {
3088 Name = FS_Example
3089 Include {
3090 Options {
3091 Accurate = A
3092 }
3093 File = /file
3094 File = /file2
3095 }
3096 ...
3097 }
3098 \end{bVerbatim}
3099
3100 This project was funded by \baculasystems{} based on an idea of James Harper and
3101 is available with the \bee{}.
3102
3103 \subsection{Setting Accurate Mode at Runtime}
3104
3105 You are now able to specify the Accurate mode on the \bcommandname{run} command and
3106 in the Schedule resource.
3107
3108 \begin{bVerbatim}
3109 * run accurate=yes job=Test
3110 \end{bVerbatim}
3111
3112 \begin{bVerbatim}
3113 Schedule {
3114 Name = WeeklyCycle
3115 Run = Full 1st sun at 23:05
3116 Run = Differential accurate=yes 2nd-5th sun at 23:05
3117 Run = Incremental accurate=no mon-sat at 23:05
3118 }
3119 \end{bVerbatim}
3120
3121 It can allow you to save memory and and CPU resources on the catalog server in
3122 some cases.
3123 \medskip
3124
3125 These advanced tuning options are available with the \bee{}.
3126
3127 % Common with community
3128 \subsection{Additions to RunScript variables}
3129 You can have access to JobBytes, JobFiles and Director name using \%b, \%F and \%D
3130 in your runscript command. The Client address is now available through \%h.
3131
3132 \begin{bVerbatim}
3133 RunAfterJob = "/bin/echo Job=%j JobBytes=%b JobFiles=%F ClientAddress=%h Dir=%D"
3134 \end{bVerbatim}
3135
3136 \subsection{\acs{LZO} Compression}
3137
3138 \acs{LZO} compression was added in the Unix File Daemon. From the user point of view,
3139 it works like the \acs{GZIP} compression (just replace \textbf{compression=GZIP} with
3140 \textbf{compression=LZO}).
3141
3142 For example:
3143 \begin{bVerbatim}
3144 Include {
3145 Options { compression=LZO }
3146 File = /home
3147 File = /data
3148 }
3149 \end{bVerbatim}
3150
3151 \acs{LZO} provides much faster compression and decompression speed but lower
3152 compression ratio than \acs{GZIP}. It is a good option when you backup to disk. For
3153 tape, the built-in compression may be a better option.
3154
3155 \acs{LZO} is a good alternative for \acs{GZIP}1 when you don't want to slow down your
3156 backup. On a modern CPU it should be able to run almost as fast as:
3157
3158 \begin{bitemize}
3159 \item your client can read data from disk. Unless you have very fast disks like
3160 \acs{SSD} or large/fast \acs{RAID} array.
3161 \item the data transfers between the file daemon and the storage daemon even on
3162 a 1Gb/s link.
3163 \end{bitemize}
3164
3165 Note that bacula only use one compression level \acs{LZO}1X-1.
3166 \medskip
3167
3168 The code for this feature was contributed by Laurent Papier.
3169
3170 \subsection{New Tray Monitor}
3171
3172 Since the old integrated Windows tray monitor doesn't work with
3173 recent Windows versions, we have written a new Qt Tray Monitor that is available
3174 for both Linux and Windows. In addition to all the previous features,
3175 this new version allows you to run Backups from
3176 the tray monitor menu.
3177
3178 \bimageH{tray-monitor}{New tray monitor}{figbs6:traymonitor}
3179
3180 \bimageH{tray-monitor1}{Run a Job through the new tray monitor}{figbs6:traymonitor1}
3181
3182
3183
3184 To be able to run a job from the tray monitor, you need to
3185 allow specific commands in the Director monitor console:
3186 \begin{bVerbatim}
3187 Console {
3188 Name = win2003-mon
3189 Password = "xxx"
3190 CommandACL = status, .clients, .jobs, .pools, .storage, .filesets, .messages, run
3191 ClientACL = *all* # you can restrict to a specific host
3192 CatalogACL = *all*
3193 JobACL = *all*
3194 StorageACL = *all*
3195 ScheduleACL = *all*
3196 PoolACL = *all*
3197 FileSetACL = *all*
3198 WhereACL = *all*
3199 }
3200 \end{bVerbatim}
3201 \medskip
3202
3203 This project was funded by \baculasystems{} and is available with \bee{} and \bce{}.
3204
3205 \subsection{Purge Migration Job}
3206
3207 The new \bdirectivename{Purge Migration Job} directive may be added to the Migration
3208 Job definition in the \bdirectorname{}'s configuration file. When it is enabled
3209 the Job that was migrated during a migration will be purged at
3210 the end of the migration job.
3211
3212 For example:
3213 \begin{bVerbatim}
3214 Job {
3215 Name = "migrate-job"
3216 Type = Migrate
3217 Level = Full
3218 Client = localhost-fd
3219 FileSet = "Full Set"
3220 Messages = Standard
3221 Storage = DiskChanger
3222 Pool = Default
3223 Selection Type = Job
3224 Selection Pattern = ".*Save"
3225 ...
3226 Purge Migration Job = yes
3227 }
3228 \end{bVerbatim}
3229 \medskip
3230
3231 This project was submitted by Dunlap Blake; testing and documentation was funded
3232 by \baculasystems{}.
3233
3234 \subsection{Changes in the Pruning Algorithm}
3235
3236 We rewrote the job pruning algorithm in this version. Previously, in some
3237 users reported that the pruning process at the end of jobs was very long.
3238 It should not be longer the case. Now, \bacula{} won't prune automatically a
3239 Job if this particular Job is needed to restore data. Example:
3240
3241 \begin{bVerbatim}
3242 JobId: 1 Level: Full
3243 JobId: 2 Level: Incremental
3244 JobId: 3 Level: Incremental
3245 JobId: 4 Level: Differential
3246 .. Other incrementals up to now
3247 \end{bVerbatim}
3248
3249 In this example, if the Job Retention defined in the Pool or in the Client
3250 resource causes that Jobs with Jobid in 1,2,3,4 can be pruned, \bacula{} will
3251 detect that JobId 1 and 4 are essential to restore data at the current state
3252 and will prune only JobId 2 and 3.
3253
3254 \texttt{Important}, this change affect only the automatic pruning step
3255 after a Job and the \bcommandname{prune jobs} \bconsole{} command. If a volume
3256 expires after the \bdirectivename{VolumeRetention} period, important jobs can be
3257 pruned.
3258
3259 \subsection{Ability to Verify any specified Job}
3260 You now have the ability to tell \bacula{} which Job should verify instead of
3261 automatically verify just the last one.
3262
3263 This feature can be used with VolumeToCatalog, DiskToCatalog and Catalog level.
3264
3265 To verify a given job, just specify the Job jobid in argument when starting the
3266 job.
3267 \begin{bVerbatim}
3268 *run job=VerifyVolume jobid=1 level=VolumeToCatalog
3269 Run Verify job
3270 JobName: VerifyVolume
3271 Level: VolumeToCatalog
3272 Client: 127.0.0.1-fd
3273 FileSet: Full Set
3274 Pool: Default (From Job resource)
3275 Storage: File (From Job resource)
3276 Verify Job: VerifyVol.2010-09-08_14.17.17_03
3277 Verify List: /tmp/regress/working/VerifyVol.bsr
3278 When: 2010-09-08 14:17:31
3279 Priority: 10
3280 OK to run? (yes/mod/no):
3281 \end{bVerbatim}
3282 \medskip
3283
3284 This project was funded by \baculasystems{} and is available with \bacula{} \enterpriseedition{} and \communityedition{}.