Debian Codegraph @

docs/manual/extensions.rst

raw
1 .. _cedar-extensions:
2
3 Official Extensions
4 ===================
5
6 .. _cedar-extensions-sysinfo:
7
8 System Information Extension
9 ----------------------------
10
11 The System Information Extension is a simple Cedar Backup extension used
12 to save off important system recovery information that might be useful
13 when reconstructing a “broken” system. It is intended to be run either
14 immediately before or immediately after the standard collect action.
15
16 This extension saves off the following information to the configured
17 Cedar Backup collect directory. Saved off data is always compressed
18 using ``bzip2``.
19
20 - Currently-installed Debian packages via ``dpkg --get-selections``
21
22 - Disk partition information via ``fdisk -l``
23
24 - System-wide mounted filesystem contents, via ``ls -laR``
25
26 The Debian-specific information is only collected on systems where
27 ``/usr/bin/dpkg`` exists.
28
29 To enable this extension, add the following section to the Cedar Backup
30 configuration file:
31
32 ::
33
34 <extensions>
35 <action>
36 <name>sysinfo</name>
37 <module>CedarBackup3.extend.sysinfo</module>
38 <function>executeAction</function>
39 <index>99</index>
40 </action>
41 </extensions>
42
43
44 This extension relies on the options and collect configuration sections
45 in the standard Cedar Backup configuration file, but requires no new
46 configuration of its own.
47
48 .. _cedar-extensions-amazons3:
49
50 Amazon S3 Extension
51 -------------------
52
53 The Amazon S3 extension writes data to Amazon S3 cloud storage rather
54 than to physical media. It is intended to replace the store action, but
55 you can also use it alongside the store action if you'd prefer to backup
56 your data in more than one place. This extension must be run after the
57 stage action.
58
59 The underlying functionality relies on the `AWS
60 CLI <http://aws.amazon.com/documentation/cli/>`__ toolset. Before you
61 use this extension, you need to set up your Amazon S3 account and
62 configure AWS CLI as detailed in Amazon's `setup
63 guide <http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html>`__.
64 The extension assumes that the backup is being executed as root, and
65 switches over to the configured backup user to run the ``aws`` program.
66 So, make sure you configure the AWS CLI tools as the backup user and not
67 root. (This is different than the ``cback-amazons3-sync`` tool, which
68 executes AWS CLI command as the same user that is running the tool.)
69
70 You can use whichever Amazon-supported authentication mechanism you
71 would like when setting up connectivity for the AWS CLI. It's best to
72 set up a separate user in the `IAM
73 Console <https://console.aws.amazon.com/iam/home>`__ rather than using
74 your main administrative user.
75
76 You probably want to lock down this user so that it can only take backup
77 related actions in the AWS infrastructure. One option is to apply the
78 ``AmazonS3FullAccess`` policy, which grants full access to the S3
79 infrastructure. If you would like to lock down the user even further,
80 this appears to be the minimum set of permissions required for Cedar
81 Backup, written as a JSON policy statement:
82
83 ::
84
85 {
86 "Version": "2012-10-17",
87 "Statement": [
88 {
89 "Effect": "Allow",
90 "Action": [
91 "s3:ListBucket",
92 "s3:ListObjects",
93 "s3:PutObject",
94 "s3:PutObjectAcl",
95 "s3:DeleteObject"
96 ],
97 "Resource": [
98 "arn:aws:s3:::my-bucket",
99 "arn:aws:s3:::my-bucket/*"
100 ]
101 }
102 ]
103 }
104
105
106 In the ``Resource`` section, be sure to list the name of your S3 bucket
107 instead of ``my-bucket``.
108
109 When using physical media via the standard store action, there is an
110 implicit limit to the size of a backup, since a backup must fit on a
111 single disc. Since there is no physical media, no such limit exists for
112 Amazon S3 backups. This leaves open the possibility that Cedar Backup
113 might construct an unexpectedly-large backup that the administrator is
114 not aware of. Over time, this might become expensive, either in terms of
115 network bandwidth or in terms of Amazon S3 storage and I/O charges. To
116 mitigate this risk, set a reasonable maximum size using the
117 configuration elements shown below. If the backup fails, you have a
118 chance to review what made the backup larger than you expected, and you
119 can either correct the problem (i.e. remove a large temporary directory
120 that got inadvertently included in the backup) or change configuration
121 to take into account the new "normal" maximum size.
122
123 You can optionally configure Cedar Backup to encrypt data before sending
124 it to S3. To do that, provide a complete command line using the
125 ``${input}`` and ``${output}`` variables to represent the original input
126 file and the encrypted output file. This command will be executed as the
127 backup user.
128
129 For instance, you can use something like this with GPG:
130
131 ::
132
133 /usr/bin/gpg -c --no-use-agent --batch --yes --passphrase-file /home/backup/.passphrase -o ${output} ${input}
134
135
136 The GPG mechanism depends on a strong passphrase for security. One way
137 to generate a strong passphrase is using your system random number
138 generator, i.e.:
139
140 ::
141
142 dd if=/dev/urandom count=20 bs=1 | xxd -ps
143
144
145 (See
146 `StackExchange <http://security.stackexchange.com/questions/14867/gpg-encryption-security>`__
147 for more details about that advice.) If you decide to use encryption,
148 make sure you save off the passphrase in a safe place, so you can get at
149 your backup data later if you need to. And obviously, make sure to set
150 permissions on the passphrase file so it can only be read by the backup
151 user.
152
153 To enable this extension, add the following section to the Cedar Backup
154 configuration file:
155
156 ::
157
158 <extensions>
159 <action>
160 <name>amazons3</name>
161 <module>CedarBackup3.extend.amazons3</module>
162 <function>executeAction</function>
163 <index>201</index> <!-- just after stage -->
164 </action>
165 </extensions>
166
167
168 This extension relies on the options and staging configuration sections
169 in the standard Cedar Backup configuration file, and then also requires
170 its own ``amazons3`` configuration section. This is an example
171 configuration section with encryption disabled:
172
173 ::
174
175 <amazons3>
176 <s3_bucket>example.com-backup/staging</s3_bucket>
177 </amazons3>
178
179
180 The following elements are part of the Amazon S3 configuration section:
181
182 ``warn_midnite``
183 Whether to generate warnings for crossing midnite.
184
185 This field indicates whether warnings should be generated if the
186 Amazon S3 operation has to cross a midnite boundary in order to find
187 data to write to the cloud. For instance, a warning would be
188 generated if valid data was only found in the day before or day after
189 the current day.
190
191 Configuration for some users is such that the amazons3 operation will
192 always cross a midnite boundary, so they will not care about this
193 warning. Other users will expect to never cross a boundary, and want
194 to be notified that something “strange” might have happened.
195
196 This field is optional. If it doesn't exist, then ``N`` will be
197 assumed.
198
199 *Restrictions:* Must be a boolean (``Y`` or ``N``).
200
201 ``s3_bucket``
202 The name of the Amazon S3 bucket that data will be written to.
203
204 This field configures the S3 bucket that your data will be written
205 to. In S3, buckets are named globally. For uniqueness, you would
206 typically use the name of your domain followed by some suffix, such
207 as ``example.com-backup``. If you want, you can specify a
208 subdirectory within the bucket, such as
209 ``example.com-backup/staging``.
210
211 *Restrictions:* Must be non-empty.
212
213 ``encrypt``
214 Command used to encrypt backup data before upload to S3
215
216 If this field is provided, then data will be encrypted before it is
217 uploaded to Amazon S3. You must provide the entire command used to
218 encrypt a file, including the ``${input}`` and ``${output}``
219 variables. An example GPG command is shown above, but you can use any
220 mechanism you choose. The command will be run as the configured
221 backup user.
222
223 *Restrictions:* If provided, must be non-empty.
224
225 ``full_size_limit``
226 Maximum size of a full backup
227
228 If this field is provided, then a size limit will be applied to full
229 backups. If the total size of the selected staging directory is
230 greater than the limit, then the backup will fail.
231
232 You can enter this value in two different forms. It can either be a
233 simple number, in which case the value is assumed to be in bytes; or
234 it can be a number followed by a unit (KB, MB, GB).
235
236 Valid examples are “10240”, “250 MB” or “1.1 GB”.
237
238 *Restrictions:* Must be a value as described above, greater than
239 zero.
240
241 ``incr_size_limit``
242 Maximum size of an incremental backup
243
244 If this field is provided, then a size limit will be applied to
245 incremental backups. If the total size of the selected staging
246 directory is greater than the limit, then the backup will fail.
247
248 You can enter this value in two different forms. It can either be a
249 simple number, in which case the value is assumed to be in bytes; or
250 it can be a number followed by a unit (KB, MB, GB).
251
252 Valid examples are “10240”, “250 MB” or “1.1 GB”.
253
254 *Restrictions:* Must be a value as described above, greater than
255 zero.
256
257 .. _cedar-extensions-subversion:
258
259 Subversion Extension
260 --------------------
261
262 The Subversion Extension is a Cedar Backup extension used to back up
263 `Subversion <http://subversion.org>`__ version control repositories via the Cedar Backup
264 command line. It is intended to be run either immediately before or
265 immediately after the standard collect action.
266
267 Each configured Subversion repository can be backed using the same
268 collect modes allowed for filesystems in the standard Cedar Backup
269 collect action (weekly, daily, incremental) and the output can be
270 compressed using either ``gzip`` or ``bzip2``.
271
272 There are two different kinds of Subversion repositories at this
273 writing: BDB (Berkeley Database) and FSFS (a "filesystem within a
274 filesystem"). This extension backs up both kinds of repositories in the
275 same way, using ``svnadmin dump`` in an incremental mode.
276
277 It turns out that FSFS repositories can also be backed up just like any
278 other filesystem directory. If you would rather do the backup that way,
279 then use the normal collect action rather than this extension. If you
280 decide to do that, be sure to consult the Subversion documentation and
281 make sure you understand the limitations of this kind of backup.
282
283 To enable this extension, add the following section to the Cedar Backup
284 configuration file:
285
286 ::
287
288 <extensions>
289 <action>
290 <name>subversion</name>
291 <module>CedarBackup3.extend.subversion</module>
292 <function>executeAction</function>
293 <index>99</index>
294 </action>
295 </extensions>
296
297
298 This extension relies on the options and collect configuration sections
299 in the standard Cedar Backup configuration file, and then also requires
300 its own ``subversion`` configuration section. This is an example
301 Subversion configuration section:
302
303 ::
304
305 <subversion>
306 <collect_mode>incr</collect_mode>
307 <compress_mode>bzip2</compress_mode>
308 <repository>
309 <abs_path>/opt/public/svn/docs</abs_path>
310 </repository>
311 <repository>
312 <abs_path>/opt/public/svn/web</abs_path>
313 <compress_mode>gzip</compress_mode>
314 </repository>
315 <repository_dir>
316 <abs_path>/opt/private/svn</abs_path>
317 <collect_mode>daily</collect_mode>
318 </repository_dir>
319 </subversion>
320
321
322 The following elements are part of the Subversion configuration section:
323
324 ``collect_mode``
325 Default collect mode.
326
327 The collect mode describes how frequently a Subversion repository is
328 backed up. The Subversion extension recognizes the same collect modes
329 as the standard Cedar Backup collect action (see
330 :doc:`basic`).
331
332 This value is the collect mode that will be used by default during
333 the backup process. Individual repositories (below) may override this
334 value. If *all* individual repositories provide their own value, then
335 this default value may be omitted from configuration.
336
337 *Note:* if your backup device does not suppport multisession discs,
338 then you should probably use the ``daily`` collect mode to avoid
339 losing data.
340
341 *Restrictions:* Must be one of ``daily``, ``weekly`` or ``incr``.
342
343 ``compress_mode``
344 Default compress mode.
345
346 Subversion repositories backups are just specially-formatted text
347 files, and often compress quite well using ``gzip`` or ``bzip2``. The
348 compress mode describes how the backed-up data will be compressed, if
349 at all.
350
351 This value is the compress mode that will be used by default during
352 the backup process. Individual repositories (below) may override this
353 value. If *all* individual repositories provide their own value, then
354 this default value may be omitted from configuration.
355
356 *Restrictions:* Must be one of ``none``, ``gzip`` or ``bzip2``.
357
358 ``repository``
359 A Subversion repository be collected.
360
361 This is a subsection which contains information about a specific
362 Subversion repository to be backed up.
363
364 This section can be repeated as many times as is necessary. At least
365 one repository or repository directory must be configured.
366
367 The ``repository`` subsection contains the following fields:
368
369 ``collect_mode``
370 Collect mode for this repository.
371
372 This field is optional. If it doesn't exist, the backup will use
373 the default collect mode.
374
375 *Restrictions:* Must be one of ``daily``, ``weekly`` or ``incr``.
376
377 ``compress_mode``
378 Compress mode for this repository.
379
380 This field is optional. If it doesn't exist, the backup will use
381 the default compress mode.
382
383 *Restrictions:* Must be one of ``none``, ``gzip`` or ``bzip2``.
384
385 ``abs_path``
386 Absolute path of the Subversion repository to back up.
387
388 *Restrictions:* Must be an absolute path.
389
390 ``repository_dir``
391 A Subversion parent repository directory be collected.
392
393 This is a subsection which contains information about a Subversion
394 parent repository directory to be backed up. Any subdirectory
395 immediately within this directory is assumed to be a Subversion
396 repository, and will be backed up.
397
398 This section can be repeated as many times as is necessary. At least
399 one repository or repository directory must be configured.
400
401 The ``repository_dir`` subsection contains the following fields:
402
403 ``collect_mode``
404 Collect mode for this repository.
405
406 This field is optional. If it doesn't exist, the backup will use
407 the default collect mode.
408
409 *Restrictions:* Must be one of ``daily``, ``weekly`` or ``incr``.
410
411 ``compress_mode``
412 Compress mode for this repository.
413
414 This field is optional. If it doesn't exist, the backup will use
415 the default compress mode.
416
417 *Restrictions:* Must be one of ``none``, ``gzip`` or ``bzip2``.
418
419 ``abs_path``
420 Absolute path of the Subversion repository to back up.
421
422 *Restrictions:* Must be an absolute path.
423
424 ``exclude``
425 List of paths or patterns to exclude from the backup.
426
427 This is a subsection which contains a set of paths and patterns to
428 be excluded within this subversion parent directory.
429
430 This section is entirely optional, and if it exists can also be
431 empty.
432
433 The exclude subsection can contain one or more of each of the
434 following fields:
435
436 ``rel_path``
437 A relative path to be excluded from the backup.
438
439 The path is assumed to be relative to the subversion parent
440 directory itself. For instance, if the configured subversion
441 parent directory is ``/opt/svn`` a configured relative path of
442 ``software`` would exclude the path ``/opt/svn/software``.
443
444 This field can be repeated as many times as is necessary.
445
446 *Restrictions:* Must be non-empty.
447
448 ``pattern``
449 A pattern to be excluded from the backup.
450
451 The pattern must be a Python regular expression. It is assumed
452 to be bounded at front and back by the beginning and end of the
453 string (i.e. it is treated as if it begins with ``^`` and ends
454 with ``$``).
455
456 This field can be repeated as many times as is necessary.
457
458 *Restrictions:* Must be non-empty
459
460 .. _cedar-extensions-mysql:
461
462 MySQL Extension
463 ---------------
464
465 The MySQL Extension is a Cedar Backup extension used to back up
466 `MySQL <http://www.mysql.com>`__ databases via the Cedar Backup command line. It
467 is intended to be run either immediately before or immediately after the
468 standard collect action.
469
470 |note|
471
472 This extension always produces a full backup. There is currently no
473 facility for making incremental backups. If/when someone has a need
474 for this and can describe how to do it, I will update this extension
475 or provide another.
476
477 The backup is done via the ``mysqldump`` command included with the MySQL
478 product. Output can be compressed using ``gzip`` or ``bzip2``.
479 Administrators can configure the extension either to back up all
480 databases or to back up only specific databases.
481
482 The extension assumes that all configured databases can be backed up by
483 a single user. Often, the “root” database user will be used. An
484 alternative is to create a separate MySQL “backup” user and grant that
485 user rights to read (but not write) various databases as needed. This
486 second option is probably your best choice.
487
488 |warning|
489
490 The extension accepts a username and password in configuration.
491 However, you probably do not want to list those values in Cedar
492 Backup configuration. This is because Cedar Backup will provide these
493 values to ``mysqldump`` via the command-line ``--user`` and
494 ``--password`` switches, which will be visible to other users in the
495 process listing.
496
497 Instead, you should configure the username and password in one of
498 MySQL's configuration files. Typically, that would be done by putting
499 a stanza like this in ``/root/.my.cnf``:
500
501 ::
502
503 [mysqldump]
504 user = root
505 password = <secret>
506
507
508 Of course, if you are executing the backup as a user other than root,
509 then you would create the file in that user's home directory instead.
510
511 As a side note, it is also possible to configure ``.my.cnf`` such
512 that Cedar Backup can back up a remote database server:
513
514 ::
515
516 [mysqldump]
517 host = remote.host
518
519
520 For this to work, you will also need to grant privileges properly for
521 the user which is executing the backup. See your MySQL documentation
522 for more information about how this can be done.
523
524 Regardless of whether you are using ``~/.my.cnf`` or
525 ``/etc/cback3.conf`` to store database login and password
526 information, you should be careful about who is allowed to view that
527 information. Typically, this means locking down permissions so that
528 only the file owner can read the file contents (i.e. use mode
529 ``0600``).
530
531 To enable this extension, add the following section to the Cedar Backup
532 configuration file:
533
534 ::
535
536 <extensions>
537 <action>
538 <name>mysql</name>
539 <module>CedarBackup3.extend.mysql</module>
540 <function>executeAction</function>
541 <index>99</index>
542 </action>
543 </extensions>
544
545
546 This extension relies on the options and collect configuration sections
547 in the standard Cedar Backup configuration file, and then also requires
548 its own ``mysql`` configuration section. This is an example MySQL
549 configuration section:
550
551 ::
552
553 <mysql>
554 <compress_mode>bzip2</compress_mode>
555 <all>Y</all>
556 </mysql>
557
558
559 If you have decided to configure login information in Cedar Backup
560 rather than using MySQL configuration, then you would add the username
561 and password fields to configuration:
562
563 ::
564
565 <mysql>
566 <user>root</user>
567 <password>password</password>
568 <compress_mode>bzip2</compress_mode>
569 <all>Y</all>
570 </mysql>
571
572
573 The following elements are part of the MySQL configuration section:
574
575 ``user``
576 Database user.
577
578 The database user that the backup should be executed as. Even if you
579 list more than one database (below) all backups must be done as the
580 same user. Typically, this would be ``root`` (i.e. the database root
581 user, not the system root user).
582
583 This value is optional. You should probably configure the username
584 and password in MySQL configuration instead, as discussed above.
585
586 *Restrictions:* If provided, must be non-empty.
587
588 ``password``
589 Password associated with the database user.
590
591 This value is optional. You should probably configure the username
592 and password in MySQL configuration instead, as discussed above.
593
594 *Restrictions:* If provided, must be non-empty.
595
596 ``compress_mode``
597 Compress mode.
598
599 MySQL databases dumps are just specially-formatted text files, and
600 often compress quite well using ``gzip`` or ``bzip2``. The compress
601 mode describes how the backed-up data will be compressed, if at all.
602
603 *Restrictions:* Must be one of ``none``, ``gzip`` or ``bzip2``.
604
605 ``all``
606 Indicates whether to back up all databases.
607
608 If this value is ``Y``, then all MySQL databases will be backed up.
609 If this value is ``N``, then one or more specific databases must be
610 specified (see below).
611
612 If you choose this option, the entire database backup will go into
613 one big dump file.
614
615 *Restrictions:* Must be a boolean (``Y`` or ``N``).
616
617 ``database``
618 Named database to be backed up.
619
620 If you choose to specify individual databases rather than all
621 databases, then each database will be backed up into its own dump
622 file.
623
624 This field can be repeated as many times as is necessary. At least
625 one database must be configured if the all option (above) is set to
626 ``N``. You may not configure any individual databases if the all
627 option is set to ``Y``.
628
629 *Restrictions:* Must be non-empty.
630
631 .. _cedar-extensions-postgresql:
632
633 PostgreSQL Extension
634 --------------------
635
636 This is a community-contributed extension provided by Antoine Beaupre
637 ("The Anarcat"). I have added regression tests around the configuration
638 parsing code and I will maintain this section in the user manual based
639 on his source code documentation.
640
641 The PostgreSQL Extension is a Cedar Backup extension used to back up
642 `PostgreSQL <http://www.postgresql.org>`__ databases via the Cedar Backup
643 command line. It is intended to be run either immediately before or immediately
644 after the standard collect action.
645
646 The backup is done via the ``pg_dump`` or ``pg_dumpall`` commands
647 included with the PostgreSQL product. Output can be compressed using
648 ``gzip`` or ``bzip2``. Administrators can configure the extension either
649 to back up all databases or to back up only specific databases.
650
651 The extension assumes that the current user has passwordless access to
652 the database since there is no easy way to pass a password to the
653 ``pg_dump`` client. This can be accomplished using appropriate
654 configuration in the ``pg_hda.conf`` file.
655
656 This extension always produces a full backup. There is currently no
657 facility for making incremental backups.
658
659 |warning|
660
661 Once you place PostgreSQL configuration into the Cedar Backup
662 configuration file, you should be careful about who is allowed to see
663 that information. This is because PostgreSQL configuration will
664 contain information about available PostgreSQL databases and
665 usernames. Typically, you might want to lock down permissions so that
666 only the file owner can read the file contents (i.e. use mode
667 ``0600``).
668
669 To enable this extension, add the following section to the Cedar Backup
670 configuration file:
671
672 ::
673
674 <extensions>
675 <action>
676 <name>postgresql</name>
677 <module>CedarBackup3.extend.postgresql</module>
678 <function>executeAction</function>
679 <index>99</index>
680 </action>
681 </extensions>
682
683
684 This extension relies on the options and collect configuration sections
685 in the standard Cedar Backup configuration file, and then also requires
686 its own ``postgresql`` configuration section. This is an example
687 PostgreSQL configuration section:
688
689 ::
690
691 <postgresql>
692 <compress_mode>bzip2</compress_mode>
693 <user>username</user>
694 <all>Y</all>
695 </postgresql>
696
697
698 If you decide to back up specific databases, then you would list them
699 individually, like this:
700
701 ::
702
703 <postgresql>
704 <compress_mode>bzip2</compress_mode>
705 <user>username</user>
706 <all>N</all>
707 <database>db1</database>
708 <database>db2</database>
709 </postgresql>
710
711
712 The following elements are part of the PostgreSQL configuration section:
713
714 ``user``
715 Database user.
716
717 The database user that the backup should be executed as. Even if you
718 list more than one database (below) all backups must be done as the
719 same user.
720
721 This value is optional.
722
723 Consult your PostgreSQL documentation for information on how to
724 configure a default database user outside of Cedar Backup, and for
725 information on how to specify a database password when you configure
726 a user within Cedar Backup. You will probably want to modify
727 ``pg_hda.conf``.
728
729 *Restrictions:* If provided, must be non-empty.
730
731 ``compress_mode``
732 Compress mode.
733
734 PostgreSQL databases dumps are just specially-formatted text files,
735 and often compress quite well using ``gzip`` or ``bzip2``. The
736 compress mode describes how the backed-up data will be compressed, if
737 at all.
738
739 *Restrictions:* Must be one of ``none``, ``gzip`` or ``bzip2``.
740
741 ``all``
742 Indicates whether to back up all databases.
743
744 If this value is ``Y``, then all PostgreSQL databases will be backed
745 up. If this value is ``N``, then one or more specific databases must
746 be specified (see below).
747
748 If you choose this option, the entire database backup will go into
749 one big dump file.
750
751 *Restrictions:* Must be a boolean (``Y`` or ``N``).
752
753 ``database``
754 Named database to be backed up.
755
756 If you choose to specify individual databases rather than all
757 databases, then each database will be backed up into its own dump
758 file.
759
760 This field can be repeated as many times as is necessary. At least
761 one database must be configured if the all option (above) is set to
762 ``N``. You may not configure any individual databases if the all
763 option is set to ``Y``.
764
765 *Restrictions:* Must be non-empty.
766
767 .. _cedar-extensions-mbox:
768
769 Mbox Extension
770 --------------
771
772 The Mbox Extension is a Cedar Backup extension used to incrementally
773 back up UNIX-style “mbox” mail folders via the Cedar Backup command
774 line. It is intended to be run either immediately before or immediately
775 after the standard collect action.
776
777 Mbox mail folders are not well-suited to being backed up by the normal
778 Cedar Backup incremental backup process. This is because active folders
779 are typically appended to on a daily basis. This forces the incremental
780 backup process to back them up every day in order to avoid losing data.
781 This can result in quite a bit of wasted space when backing up large
782 mail folders.
783
784 The Mbox extension leverages the ``grepmail`` utility to back up only email
785 messages which have been received since the last incremental backup. This way,
786 even if a folder is added to every day, only the recently-added messages are
787 is backed up. This can potentially save a lot of space.
788
789 Each configured mbox file or directory can be backed using the same
790 collect modes allowed for filesystems in the standard Cedar Backup
791 collect action (weekly, daily, incremental) and the output can be
792 compressed using either ``gzip`` or ``bzip2``.
793
794 To enable this extension, add the following section to the Cedar Backup
795 configuration file:
796
797 ::
798
799 <extensions>
800 <action>
801 <name>mbox</name>
802 <module>CedarBackup3.extend.mbox</module>
803 <function>executeAction</function>
804 <index>99</index>
805 </action>
806 </extensions>
807
808
809 This extension relies on the options and collect configuration sections
810 in the standard Cedar Backup configuration file, and then also requires
811 its own ``mbox`` configuration section. This is an example mbox
812 configuration section:
813
814 ::
815
816 <mbox>
817 <collect_mode>incr</collect_mode>
818 <compress_mode>gzip</compress_mode>
819 <file>
820 <abs_path>/home/user1/mail/greylist</abs_path>
821 <collect_mode>daily</collect_mode>
822 </file>
823 <dir>
824 <abs_path>/home/user2/mail</abs_path>
825 </dir>
826 <dir>
827 <abs_path>/home/user3/mail</abs_path>
828 <exclude>
829 <rel_path>spam</rel_path>
830 <pattern>.*debian.*</pattern>
831 </exclude>
832 </dir>
833 </mbox>
834
835
836 Configuration is much like the standard collect action. Differences come
837 from the fact that mbox directories are *not* collected recursively.
838
839 Unlike collect configuration, exclusion information can only be
840 configured at the mbox directory level (there are no global exclusions).
841 Another difference is that no absolute exclusion paths are allowed ---
842 only relative path exclusions and patterns.
843
844 The following elements are part of the mbox configuration section:
845
846 ``collect_mode``
847 Default collect mode.
848
849 The collect mode describes how frequently an mbox file or directory
850 is backed up. The mbox extension recognizes the same collect modes as
851 the standard Cedar Backup collect action (see
852 :doc:`basic`).
853
854 This value is the collect mode that will be used by default during
855 the backup process. Individual files or directories (below) may
856 override this value. If *all* individual files or directories provide
857 their own value, then this default value may be omitted from
858 configuration.
859
860 *Note:* if your backup device does not suppport multisession discs,
861 then you should probably use the ``daily`` collect mode to avoid
862 losing data.
863
864 *Restrictions:* Must be one of ``daily``, ``weekly`` or ``incr``.
865
866 ``compress_mode``
867 Default compress mode.
868
869 Mbox file or directory backups are just text, and often compress
870 quite well using ``gzip`` or ``bzip2``. The compress mode describes
871 how the backed-up data will be compressed, if at all.
872
873 This value is the compress mode that will be used by default during
874 the backup process. Individual files or directories (below) may
875 override this value. If *all* individual files or directories provide
876 their own value, then this default value may be omitted from
877 configuration.
878
879 *Restrictions:* Must be one of ``none``, ``gzip`` or ``bzip2``.
880
881 ``file``
882 An individual mbox file to be collected.
883
884 This is a subsection which contains information about an individual
885 mbox file to be backed up.
886
887 This section can be repeated as many times as is necessary. At least
888 one mbox file or directory must be configured.
889
890 The file subsection contains the following fields:
891
892 ``collect_mode``
893 Collect mode for this file.
894
895 This field is optional. If it doesn't exist, the backup will use
896 the default collect mode.
897
898 *Restrictions:* Must be one of ``daily``, ``weekly`` or ``incr``.
899
900 ``compress_mode``
901 Compress mode for this file.
902
903 This field is optional. If it doesn't exist, the backup will use
904 the default compress mode.
905
906 *Restrictions:* Must be one of ``none``, ``gzip`` or ``bzip2``.
907
908 ``abs_path``
909 Absolute path of the mbox file to back up.
910
911 *Restrictions:* Must be an absolute path.
912
913 ``dir``
914 An mbox directory to be collected.
915
916 This is a subsection which contains information about an mbox
917 directory to be backed up. An mbox directory is a directory
918 containing mbox files. Every file in an mbox directory is assumed to
919 be an mbox file. Mbox directories are *not* collected recursively.
920 Only the files immediately within the configured directory will be
921 backed-up and any subdirectories will be ignored.
922
923 This section can be repeated as many times as is necessary. At least
924 one mbox file or directory must be configured.
925
926 The dir subsection contains the following fields:
927
928 ``collect_mode``
929 Collect mode for this file.
930
931 This field is optional. If it doesn't exist, the backup will use
932 the default collect mode.
933
934 *Restrictions:* Must be one of ``daily``, ``weekly`` or ``incr``.
935
936 ``compress_mode``
937 Compress mode for this file.
938
939 This field is optional. If it doesn't exist, the backup will use
940 the default compress mode.
941
942 *Restrictions:* Must be one of ``none``, ``gzip`` or ``bzip2``.
943
944 ``abs_path``
945 Absolute path of the mbox directory to back up.
946
947 *Restrictions:* Must be an absolute path.
948
949 ``exclude``
950 List of paths or patterns to exclude from the backup.
951
952 This is a subsection which contains a set of paths and patterns to
953 be excluded within this mbox directory.
954
955 This section is entirely optional, and if it exists can also be
956 empty.
957
958 The exclude subsection can contain one or more of each of the
959 following fields:
960
961 ``rel_path``
962 A relative path to be excluded from the backup.
963
964 The path is assumed to be relative to the mbox directory
965 itself. For instance, if the configured mbox directory is
966 ``/home/user2/mail`` a configured relative path of ``SPAM``
967 would exclude the path ``/home/user2/mail/SPAM``.
968
969 This field can be repeated as many times as is necessary.
970
971 *Restrictions:* Must be non-empty.
972
973 ``pattern``
974 A pattern to be excluded from the backup.
975
976 The pattern must be a Python regular expression. It is assumed
977 to be bounded at front and back by the beginning and end of the
978 string (i.e. it is treated as if it begins with ``^`` and ends
979 with ``$``).
980
981 This field can be repeated as many times as is necessary.
982
983 *Restrictions:* Must be non-empty
984
985 .. _cedar-extensions-encrypt:
986
987 Encrypt Extension
988 -----------------
989
990 The Encrypt Extension is a Cedar Backup extension used to encrypt
991 backups. It does this by encrypting the contents of a master's staging
992 directory each day after the stage action is run. This way, backed-up
993 data is encrypted both when sitting on the master and when written to
994 disc. This extension must be run before the standard store action,
995 otherwise unencrypted data will be written to disc.
996
997 There are several differents ways encryption could have been built in to
998 or layered on to Cedar Backup. I asked the mailing list for opinions on
999 the subject in January 2007 and did not get a lot of feedback, so I
1000 chose the option that was simplest to understand and simplest to
1001 implement. If other encryption use cases make themselves known in the
1002 future, this extension can be enhanced or replaced.
1003
1004 Currently, this extension supports only GPG. However, it would be
1005 straightforward to support other public-key encryption mechanisms, such
1006 as OpenSSL.
1007
1008 |warning|
1009
1010 If you decide to encrypt your backups, be *absolutely sure* that you
1011 have your GPG secret key saved off someplace safe --- someplace
1012 other than on your backup disc. If you lose your secret key, your
1013 backup will be useless.
1014
1015 I suggest that before you rely on this extension, you should execute
1016 a dry run and make sure you can successfully decrypt the backup that
1017 is written to disc.
1018
1019 Before configuring the Encrypt extension, you must configure GPG. Either
1020 create a new keypair or use an existing one. Determine which user will
1021 execute your backup (typically root) and have that user import *and
1022 lsign* the public half of the keypair. Then, save off the secret half of
1023 the keypair someplace safe, apart from your backup (i.e. on a floppy
1024 disk or USB drive). Make sure you know the recipient name associated
1025 with the public key because you'll need it to configure Cedar Backup.
1026 (If you can run ``gpg -e -r "Recipient Name" file.txt`` and it executes
1027 cleanly with no user interaction required, you should be OK.)
1028
1029 An encrypted backup has the same file structure as a normal backup, so
1030 all of the instructions in :doc:`recovering` apply. The only
1031 difference is that encrypted files will have an additional ``.gpg``
1032 extension (so for instance ``file.tar.gz`` becomes ``file.tar.gz.gpg``).
1033 To recover decrypted data, simply log on as a user which has access to
1034 the secret key and decrypt the ``.gpg`` file that you are interested in.
1035 Then, recover the data as usual.
1036
1037 *Note:* I am being intentionally vague about how to configure and use GPG,
1038 because I do not want to encourage neophytes to blindly use this
1039 extension. If you do not already understand GPG well enough to follow
1040 the two paragraphs above, *do not use this extension*. Instead, before
1041 encrypting your backups, check out the excellent GNU Privacy Handbook at
1042 `<http://www.gnupg.org/gph/en/manual.html>`__ and gain an understanding
1043 of how encryption can help you or hurt you.
1044
1045 To enable this extension, add the following section to the Cedar Backup
1046 configuration file:
1047
1048 ::
1049
1050 <extensions>
1051 <action>
1052 <name>encrypt</name>
1053 <module>CedarBackup3.extend.encrypt</module>
1054 <function>executeAction</function>
1055 <index>301</index>
1056 </action>
1057 </extensions>
1058
1059
1060 This extension relies on the options and staging configuration sections
1061 in the standard Cedar Backup configuration file, and then also requires
1062 its own ``encrypt`` configuration section. This is an example Encrypt
1063 configuration section:
1064
1065 ::
1066
1067 <encrypt>
1068 <encrypt_mode>gpg</encrypt_mode>
1069 <encrypt_target>Backup User</encrypt_target>
1070 </encrypt>
1071
1072
1073 The following elements are part of the Encrypt configuration section:
1074
1075 ``encrypt_mode``
1076 Encryption mode.
1077
1078 This value specifies which encryption mechanism will be used by the
1079 extension.
1080
1081 Currently, only the GPG public-key encryption mechanism is supported.
1082
1083 *Restrictions:* Must be ``gpg``.
1084
1085 ``encrypt_target``
1086 Encryption target.
1087
1088 The value in this field is dependent on the encryption mode. For the
1089 ``gpg`` mode, this is the name of the recipient whose public key will
1090 be used to encrypt the backup data, i.e. the value accepted by
1091 ``gpg -r``.
1092
1093 .. _cedar-extensions-split:
1094
1095 Split Extension
1096 ---------------
1097
1098 The Split Extension is a Cedar Backup extension used to split up large
1099 files within staging directories. It is probably only useful in
1100 combination with the ``cback3-span`` command, which requires individual
1101 files within staging directories to each be smaller than a single disc.
1102
1103 You would normally run this action immediately after the standard stage
1104 action, but you could also choose to run it by hand immediately before
1105 running ``cback3-span``.
1106
1107 The split extension uses the standard UNIX ``split`` tool to split the
1108 large files up. This tool simply splits the files on bite-size
1109 boundaries. It has no knowledge of file formats.
1110
1111 *Note: this means that in order to recover the data in your original
1112 large file, you must have every file that the original file was split
1113 into.* Think carefully about whether this is what you want. It doesn't
1114 sound like a huge limitation. However, ``cback3-span`` might put an
1115 indivdual file on *any* disc in a set --- the files split from one
1116 larger file will not necessarily be together. That means you will
1117 probably need every disc in your backup set in order to recover any data
1118 from the backup set.
1119
1120 To enable this extension, add the following section to the Cedar Backup
1121 configuration file:
1122
1123 ::
1124
1125 <extensions>
1126 <action>
1127 <name>split</name>
1128 <module>CedarBackup3.extend.split</module>
1129 <function>executeAction</function>
1130 <index>299</index>
1131 </action>
1132 </extensions>
1133
1134
1135 This extension relies on the options and staging configuration sections
1136 in the standard Cedar Backup configuration file, and then also requires
1137 its own ``split`` configuration section. This is an example Split
1138 configuration section:
1139
1140 ::
1141
1142 <split>
1143 <size_limit>250 MB</size_limit>
1144 <split_size>100 MB</split_size>
1145 </split>
1146
1147
1148 The following elements are part of the Split configuration section:
1149
1150 ``size_limit``
1151 Size limit.
1152
1153 Files with a size strictly larger than this limit will be split by
1154 the extension.
1155
1156 You can enter this value in two different forms. It can either be a
1157 simple number, in which case the value is assumed to be in bytes; or
1158 it can be a number followed by a unit (KB, MB, GB).
1159
1160 Valid examples are “10240”, “250 MB” or “1.1 GB”.
1161
1162 *Restrictions:* Must be a size as described above.
1163
1164 ``split_size``
1165 Split size.
1166
1167 This is the size of the chunks that a large file will be split into.
1168 The final chunk may be smaller if the split size doesn't divide
1169 evenly into the file size.
1170
1171 You can enter this value in two different forms. It can either be a
1172 simple number, in which case the value is assumed to be in bytes; or
1173 it can be a number followed by a unit (KB, MB, GB).
1174
1175 Valid examples are “10240”, “250 MB” or “1.1 GB”.
1176
1177 *Restrictions:* Must be a size as described above.
1178
1179 .. _cedar-extensions-capacity:
1180
1181 Capacity Extension
1182 ------------------
1183
1184 The capacity extension checks the current capacity of the media in the
1185 writer and prints a warning if the media exceeds an indicated capacity.
1186 The capacity is indicated either by a maximum percentage utilized or by
1187 a minimum number of bytes that must remain unused.
1188
1189 This action can be run at any time, but is probably best run as the last
1190 action on any given day, so you get as much notice as possible that your
1191 media is full and needs to be replaced.
1192
1193 To enable this extension, add the following section to the Cedar Backup
1194 configuration file:
1195
1196 ::
1197
1198 <extensions> <action>
1199 <name>capacity</name>
1200 <module>CedarBackup3.extend.capacity</module>
1201 <function>executeAction</function>
1202 <index>299</index>
1203 </action>
1204 </extensions>
1205
1206
1207 This extension relies on the options and store configuration sections in
1208 the standard Cedar Backup configuration file, and then also requires its
1209 own ``capacity`` configuration section. This is an example Capacity
1210 configuration section that configures the extension to warn if the media
1211 is more than 95.5% full:
1212
1213 ::
1214
1215 <capacity>
1216 <max_percentage>95.5</max_percentage>
1217 </capacity>
1218
1219
1220 This example configures the extension to warn if the media has fewer
1221 than 16 MB free:
1222
1223 ::
1224
1225 <capacity>
1226 <min_bytes>16 MB</min_bytes>
1227 </capacity>
1228
1229
1230 The following elements are part of the Capacity configuration section:
1231
1232 ``max_percentage``
1233 Maximum percentage of the media that may be utilized.
1234
1235 You must provide either this value *or* the ``min_bytes`` value.
1236
1237 *Restrictions:* Must be a floating point number between 0.0 and 100.0
1238
1239 ``min_bytes``
1240 Minimum number of free bytes that must be available.
1241
1242 You can enter this value in two different forms. It can either be a
1243 simple number, in which case the value is assumed to be in bytes; or
1244 it can be a number followed by a unit (KB, MB, GB).
1245
1246 Valid examples are “10240”, “250 MB” or “1.1 GB”.
1247
1248 You must provide either this value *or* the ``max_percentage`` value.
1249
1250 *Restrictions:* Must be a byte quantity as described above.
1251
1252 ----------
1253
1254 *Previous*: :doc:`config` • *Next*: :doc:`extenspec`
1255
1256 .. |note| image:: images/note.png
1257 .. |tip| image:: images/tip.png
1258 .. |warning| image:: images/warning.png