Debian Codegraph @

docs/manual/basic.rst

raw
1 .. _cedar-basic:
2
3 Basic Concepts
4 ==============
5
6 .. _cedar-basic-general:
7
8 General Architecture
9 --------------------
10
11 Cedar Backup is architected as a Python package (library) and a single
12 executable (a Python script). The Python package provides both
13 application-specific code and general utilities that can be used by
14 programs other than Cedar Backup. It also includes modules that can be
15 used by third parties to extend Cedar Backup or provide related
16 functionality.
17
18 The ``cback3`` script is designed to run as root, since otherwise it's
19 difficult to back up system directories or write to the CD/DVD device.
20 However, pains are taken to use the backup user's effective user id
21 (specified in configuration) when appropriate. *Note:* this does not mean
22 that ``cback3`` runs setuid [1]_ or setgid. However, all files on disk
23 will be owned by the backup user, and and all rsh-based network
24 connections will take place as the backup user.
25
26 The ``cback3`` script is configured via command-line options and an XML
27 configuration file on disk. The configuration file is normally stored in
28 ``/etc/cback3.conf``, but this path can be overridden at runtime. See
29 :doc:`config` for more information on how Cedar Backup is
30 configured.
31
32 |warning|
33
34 You should be aware that backups to CD/DVD media can probably be read
35 by any user which has permissions to mount the CD/DVD writer. If you
36 intend to leave the backup disc in the drive at all times, you may
37 want to consider this when setting up device permissions on your
38 machine. See also :doc:`extensions`.
39
40 .. _cedar-basic-datarecovery:
41
42 Data Recovery
43 -------------
44
45 Cedar Backup does not include any facility to restore backups. Instead,
46 it assumes that the administrator (using the procedures and references
47 in :doc:`recovering`) can handle the task of restoring their
48 own system, using the standard system tools at hand.
49
50 If I were to maintain recovery code in Cedar Backup, I would almost
51 certainly end up in one of two situations. Either Cedar Backup would
52 only support simple recovery tasks, and those via an interface a lot
53 like that of the underlying system tools; or Cedar Backup would have to
54 include a hugely complicated interface to support more specialized (and
55 hence useful) recovery tasks like restoring individual files as of a
56 certain point in time. In either case, I would end up trying to maintain
57 critical functionality that would be rarely used, and hence would also
58 be rarely tested by end-users. I am uncomfortable asking anyone to rely
59 on functionality that falls into this category.
60
61 My primary goal is to keep the Cedar Backup codebase as simple and
62 focused as possible. I hope you can understand how the choice of
63 providing documentation, but not code, seems to strike the best balance
64 between managing code complexity and providing the functionality that
65 end-users need.
66
67 .. _cedar-basic-pools:
68
69 Cedar Backup Pools
70 -------------------
71
72 There are two kinds of machines in a Cedar Backup pool. One machine (the
73 *master*) has a CD or DVD writer on it and writes the backup to disc. If you are
74 using Amazon S3 instead of physical media, then the master is the machine that
75 consolidates data and writes your backup to cloud storage. The other machines
76 (*clients*) collect data to be written to disc by the master. Collectively, the
77 master and client machines in a pool are called *peer machines*.
78
79 Cedar Backup has been designed primarily for situations where there is a single
80 master and a set of other clients that the master interacts with. However, it
81 will just as easily work for a single machine (a backup pool of one) and in
82 fact more users seem to use it like this than any other way.
83
84 .. _cedar-basic-process:
85
86 The Backup Process
87 ------------------
88
89 The Cedar Backup backup process is structured in terms of a set of
90 decoupled actions which execute independently (based on a schedule in
91 ``cron``) rather than through some highly coordinated flow of control.
92
93 This design decision has both positive and negative consequences. On the
94 one hand, the code is much simpler and can choose to simply abort or log
95 an error if its expectations are not met. On the other hand, the
96 administrator must coordinate the various actions during initial set-up.
97 See *Coordination between Master and Clients* (later in this chapter) for more
98 information on this subject.
99
100 A standard backup run consists of four steps (actions), some of which
101 execute on the master machine, and some of which execute on one or more
102 client machines. These actions are: collect, stage, store and purge.
103
104 In general, more than one action may be specified on the command-line. If more
105 than one action is specified, then actions will be taken in a sensible order
106 (generally *collect*, *stage*, *store*, *purge*). A special *all* action is
107 also allowed, which implies all of the standard actions in the same sensible
108 order.
109
110 The ``cback3`` command also supports several actions that are not part of the
111 standard backup run and cannot be executed along with any other actions. These
112 actions are *validate*, *initialize* and *rebuild*. All of the various actions
113 are discussed further below.
114
115 See :doc:`config` for more information on how a backup run is
116 configured.
117
118 Cedar Backup was designed to be flexible. It allows you to decide for
119 yourself which backup steps you care about executing (and when you
120 execute them), based on your own situation and your own priorities.
121
122 For example, no need to write to disc or to Amazon S3 at all. In fact, some
123 users prefer to use their master machine as a simple “consolidation point”.
124 They don't back up any data on the master, and don't write to disc at all. They
125 just use Cedar Backup to handle the mechanics of moving backed-up data to a
126 central location. This isn't quite what Cedar Backup was written to do, but it
127 is flexible enough to meet their needs.
128
129 .. _cedar-basic-process-collect:
130
131 The Collect Action
132 ~~~~~~~~~~~~~~~~~~
133
134 The collect action is the first action in a standard backup run. It
135 executes on both master and client nodes. Based on configuration, this
136 action traverses the peer's filesystem and gathers files to be backed
137 up. Each configured high-level directory is collected up into its own
138 ``tar`` file in the collect directory. The tarfiles can either be
139 uncompressed (``.tar``) or compressed with either ``gzip`` (``.tar.gz``)
140 or ``bzip2`` (``.tar.bz2``).
141
142 There are three supported collect modes: *daily*, *weekly* and *incremental*.
143 Directories configured for daily backups are backed up every day.
144 Directories configured for weekly backups are backed up on the first day
145 of the week. Directories configured for incremental backups are
146 traversed every day, but only the files which have changed (based on a
147 saved-off cryptographic hash) are actually backed up.
148
149 Collect configuration also allows for a variety of ways to filter files and
150 directories out of the backup. For instance, administrators can configure an
151 ignore indicator file or specify absolute paths or filename patterns to be
152 excluded. You can even configure a backup “link farm” rather than explicitly
153 listing files and directories in configuration.
154
155 This action is optional on the master. You only need to configure and
156 execute the collect action on the master if you have data to back up on
157 that machine. If you plan to use the master only as a “consolidation
158 point” to collect data from other machines, then there is no need to
159 execute the collect action there. If you run the collect action on the
160 master, it behaves the same there as anywhere else, and you have to
161 stage the master's collected data just like any other client (typically
162 by configuring a local peer in the stage action).
163
164 .. _cedar-basic-process-stage:
165
166 The Stage Action
167 ~~~~~~~~~~~~~~~~
168
169 The stage action is the second action in a standard backup run. It
170 executes on the master peer node. The master works down the list of
171 peers in its backup pool and stages (copies) the collected backup files
172 from each of them into a daily staging directory by peer name.
173
174 For the purposes of this action, the master node can be configured to
175 treat itself as a client node. If you intend to back up data on the
176 master, configure the master as a local peer. Otherwise, just configure
177 each of the clients as a remote peer.
178
179 Local and remote client peers are treated differently. Local peer
180 collect directories are assumed to be accessible via normal copy
181 commands (i.e. on a mounted filesystem) while remote peer collect
182 directories are accessed via an RSH-compatible command such as ``ssh``.
183
184 If a given peer is not ready to be staged, the stage process will log an
185 error, abort the backup for that peer, and then move on to its other
186 peers. This way, one broken peer cannot break a backup for other peers
187 which are up and running.
188
189 Keep in mind that Cedar Backup is flexible about what actions must be
190 executed as part of a backup. If you would prefer, you can stop the
191 backup process at this step, and skip the store step. In this case, the
192 staged directories will represent your backup rather than a disc or
193 an Amazon S3 bucket.
194
195 |note|
196
197 Directories “collected” by another process can alsoalso be staged by Cedar
198 Backup. If the file ``cback.collect`` exists in a collect directory
199 when the stage action is taken, then that directory will be staged.
200
201 .. _cedar-basic-process-store:
202
203 The Store Action
204 ~~~~~~~~~~~~~~~~
205
206 The store action is the third action in a standard backup run. It
207 executes on the master peer node. The master machine determines the
208 location of the current staging directory, and then writes the contents
209 of that staging directory to disc. After the contents of the directory
210 have been written to disc, an optional validation step ensures that the
211 write was successful.
212
213 If the backup is running on the first day of the week, if the drive does
214 not support multisession discs, or if the ``--full`` option is passed to
215 the ``cback3`` command, the disc will be rebuilt from scratch.
216 Otherwise, a new ISO session will be added to the disc each day the
217 backup runs.
218
219 This action is entirely optional. If you would prefer to just stage
220 backup data from a set of peers to a master machine, and have the staged
221 directories represent your backup rather than a disc, this is fine.
222
223 |warning|
224
225 The store action is not supported on the Mac OS X (darwin) platform.
226 On that platform, the “automount” function of the Finder interferes
227 significantly with Cedar Backup's ability to mount and unmount media
228 and write to the CD or DVD hardware. The Cedar Backup writer and
229 image functionality works on this platform, but the effort required
230 to fight the operating system about who owns the media and the device
231 makes it nearly impossible to execute the store action successfully.
232
233 The store action tries to be smart about finding the current staging
234 directory. It first checks the current day's staging directory. If that
235 directory exists, and it has not yet been written to disc (i.e. there is
236 no store indicator), then it will be used. Otherwise, the store action
237 will look for an unused staging directory for either the previous day or
238 the next day, in that order. A warning will be written to the log under
239 these circumstances (controlled by the ``<warn_midnite>`` configuration
240 value).
241
242 This behavior varies slightly when the ``--full`` option is in effect.
243 Under these circumstances, any existing store indicator will be ignored.
244 Also, the store action will always attempt to use the current day's
245 staging directory, ignoring any staging directories for the previous day
246 or the next day. This way, running a full store action more than once
247 concurrently will always produce the same results. (You might imagine a
248 use case where a person wants to make several copies of the same full
249 backup.)
250
251 .. _cedar-basic-process-purge:
252
253 The Purge Action
254 ~~~~~~~~~~~~~~~~
255
256 The purge action is the fourth and final action in a standard backup
257 run. It executes both on the master and client peer nodes. Configuration
258 specifies how long to retain files in certain directories, and older
259 files and empty directories are purged.
260
261 Typically, collect directories are purged daily, and stage directories
262 are purged weekly or slightly less often (if a disc gets corrupted,
263 older backups may still be available on the master). Some users also
264 choose to purge the configured working directory (which is used for
265 temporary files) to eliminate any leftover files which might have
266 resulted from changes to configuration.
267
268 .. _cedar-basic-process-all:
269
270 The All Action
271 ~~~~~~~~~~~~~~
272
273 The all action is a pseudo-action which causes all of the actions in a
274 standard backup run to be executed together in order. It cannot be
275 combined with any other actions on the command line.
276
277 Extensions *cannot* be executed as part of the all action. If you need
278 to execute an extended action, you must specify the other actions you
279 want to run individually on the command line. [2]_
280
281 The all action does not have its own configuration. Instead, it relies
282 on the individual configuration sections for all of the other actions.
283
284 .. _cedar-basic-process-validate:
285
286 The Validate Action
287 ~~~~~~~~~~~~~~~~~~~
288
289 The validate action is used to validate configuration on a particular
290 peer node, either master or client. It cannot be combined with any other
291 actions on the command line.
292
293 The validate action checks that the configuration file can be found,
294 that the configuration file is valid, and that certain portions of the
295 configuration file make sense (for instance, making sure that specified
296 users exist, directories are readable and writable as necessary, etc.).
297
298 .. _cedar-basic-process-initialize:
299
300 The Initialize Action
301 ~~~~~~~~~~~~~~~~~~~~~
302
303 The initialize action is used to initialize media for use with Cedar
304 Backup. This is an optional step. By default, Cedar Backup does not need
305 to use initialized media and will write to whatever media exists in the
306 writer device.
307
308 However, if the “check media” store configuration option is set to true,
309 Cedar Backup will check the media before writing to it and will error
310 out if the media has not been initialized.
311
312 Initializing the media consists of writing a mostly-empty image using a
313 known media label (the media label will begin with “CEDAR BACKUP”).
314
315 Note that only rewritable media (CD-RW, DVD+RW) can be initialized. It
316 doesn't make any sense to initialize media that cannot be rewritten
317 (CD-R, DVD+R), since Cedar Backup would then not be able to use that
318 media for a backup. You can still configure Cedar Backup to check
319 non-rewritable media; in this case, the check will also pass if the
320 media is apparently unused (i.e. has no media label).
321
322 .. _cedar-basic-process-rebuild:
323
324 The Rebuild Action
325 ~~~~~~~~~~~~~~~~~~
326
327 The rebuild action is an exception-handling action that is executed
328 independent of a standard backup run. It cannot be combined with any
329 other actions on the command line.
330
331 The rebuild action attempts to rebuild “this week's” disc from any
332 remaining unpurged staging directories. Typically, it is used to make a
333 copy of a backup, replace lost or damaged media, or to switch to new
334 media mid-week for some other reason.
335
336 To decide what data to write to disc again, the rebuild action looks
337 back and finds the first day of the current week. Then, it finds any
338 remaining staging directories between that date and the current date. If
339 any staging directories are found, they are all written to disc in one
340 big ISO session.
341
342 The rebuild action does not have its own configuration. It relies on
343 configuration for other other actions, especially the store action.
344
345 .. _cedar-basic-coordinate:
346
347 Coordination between Master and Clients
348 ---------------------------------------
349
350 Unless you are using Cedar Backup to manage a “pool of one”, you will
351 need to set up some coordination between your clients and master to make
352 everything work properly. This coordination isn't difficult --- it
353 mostly consists of making sure that operations happen in the right order
354 --- but some users are suprised that it is required and want to know
355 why Cedar Backup can't just “take care of it for me”.
356
357 Essentially, each client must finish collecting all of its data before
358 the master begins staging it, and the master must finish staging data
359 from a client before that client purges its collected data.
360 Administrators may need to experiment with the time between the collect
361 and purge entries so that the master has enough time to stage data
362 before it is purged.
363
364 .. _cedar-basic-managedbackups:
365
366 Managed Backups
367 ---------------
368
369 Cedar Backup also supports an optional feature called the “managed
370 backup”. This feature is intended for use with remote clients where cron
371 is not available.
372
373 When managed backups are enabled, managed clients must still be
374 configured as usual. However, rather than using a cron job on the client
375 to execute the collect and purge actions, the master executes these
376 actions on the client via a remote shell.
377
378 To make this happen, first set up one or more managed clients in Cedar
379 Backup configuration. Then, invoke Cedar Backup with the ``--managed``
380 command-line option. Whenever Cedar Backup invokes an action locally, it
381 will invoke the same action on each of the managed clients.
382
383 Technically, this feature works for any client, not just clients that
384 don't have cron available. Used this way, it can simplify the setup
385 process, because cron only has to be configured on the master. For some
386 users, that may be motivation enough to use this feature all of the
387 time.
388
389 However, please keep in mind that this feature depends on a stable
390 network. If your network connection drops, your backup will be
391 interrupted and will not be complete. It is even possible that some of
392 the Cedar Backup metadata (like incremental backup state) will be
393 corrupted. The risk is not high, but it is something you need to be
394 aware of if you choose to use this optional feature.
395
396 .. _cedar-basic-mediadevice:
397
398 Media and Device Types
399 ----------------------
400
401 Cedar Backup is focused around writing backups to CD or DVD media using
402 a standard SCSI or IDE writer. In Cedar Backup terms, the disc itself is
403 referred to as the media, and the CD/DVD drive is referred to as the
404 device or sometimes the backup device.
405
406 When using a new enough backup device, a new “multisession” ISO image [3]_
407 is written to the media on the first day of the week, and then additional
408 multisession images are added to the media each day that Cedar Backup runs.
409 This way, the media is complete and usable at the end of every backup run, but
410 a single disc can be used all week long. If your backup device does not support
411 multisession images --- which is really unusual today --- then a new ISO image
412 will be written to the media each time Cedar Backup runs (and you should
413 probably confine yourself to the “daily” backup mode to avoid losing data).
414
415 Cedar Backup currently supports four different kinds of CD media:
416
417 cdr-74
418 74-minute non-rewritable CD media
419
420 cdrw-74
421 74-minute rewritable CD media
422
423 cdr-80
424 80-minute non-rewritable CD media
425
426 cdrw-80
427 80-minute rewritable CD media
428
429 I have chosen to support just these four types of CD media because they
430 seem to be the most “standard” of the various types commonly sold in the
431 U.S. as of this writing (early 2005). If you regularly use an
432 unsupported media type and would like Cedar Backup to support it, send
433 me information about the capacity of the media in megabytes (MB) and
434 whether it is rewritable.
435
436 Cedar Backup also supports two kinds of DVD media:
437
438 dvd+r
439 Single-layer non-rewritable DVD+R media
440
441 dvd+rw
442 Single-layer rewritable DVD+RW media
443
444 The underlying ``growisofs`` utility does support other kinds of media
445 (including DVD-R, DVD-RW and BlueRay) which work somewhat differently
446 than standard DVD+R and DVD+RW media. I don't support these other kinds
447 of media because I haven't had any opportunity to work with them. The
448 same goes for dual-layer media of any type.
449
450 .. _cedar-basic-incremental:
451
452 Incremental Backups
453 -------------------
454
455 Cedar Backup supports three different kinds of backups for individual
456 collect directories. These are daily, weekly and incremental backups.
457 Directories using the daily mode are backed up every day. Directories
458 using the weekly mode are only backed up on the first day of the week,
459 or when the ``--full`` option is used. Directories using the incremental
460 mode are always backed up on the first day of the week (like a weekly
461 backup), but after that only the files which have changed are actually
462 backed up on a daily basis.
463
464 In Cedar Backup, incremental backups are not based on date, but are
465 instead based on saved checksums, one for each backed-up file. When a
466 full backup is run, Cedar Backup gathers a checksum value [4]_ for each
467 backed-up file. The next time an incremental backup is run, Cedar Backup
468 checks its list of file/checksum pairs for each file that might be
469 backed up. If the file's checksum value does not match the saved value,
470 or if the file does not appear in the list of file/checksum pairs, then
471 it will be backed up and a new checksum value will be placed into the
472 list. Otherwise, the file will be ignored and the checksum value will be
473 left unchanged.
474
475 Cedar Backup stores the file/checksum pairs in ``.sha`` files in its
476 working directory, one file per configured collect directory. The
477 mappings in these files are reset at the start of the week or when the
478 ``--full`` option is used. Because these files are used for an entire
479 week, you should never purge the working directory more frequently than
480 once per week.
481
482 .. _cedar-basic-extensions:
483
484 Extensions
485 ----------
486
487 Imagine that there is a third party developer who understands how to
488 back up a certain kind of database repository. This third party might
489 want to integrate his or her specialized backup into the Cedar Backup
490 process, perhaps thinking of the database backup as a sort of “collect”
491 step.
492
493 Prior to Cedar Backup version 2, any such integration would have been
494 completely independent of Cedar Backup itself. The “external” backup
495 functionality would have had to maintain its own configuration and would
496 not have had access to any Cedar Backup configuration.
497
498 Starting with version 2, Cedar Backup allows extensions to the backup
499 process. An extension is an action that isn't part of the standard
500 backup process (i.e. not collect, stage, store or purge), but can be
501 executed by Cedar Backup when properly configured.
502
503 Extension authors implement an “action process” function with a certain
504 interface, and are allowed to add their own sections to the Cedar Backup
505 configuration file, so that all backup configuration can be centralized.
506 Then, the action process function is associated with an action name
507 which can be executed from the ``cback3`` command line like any other
508 action.
509
510 Hopefully, as the Cedar Backup user community grows, users will
511 contribute their own extensions back to the community. Well-written
512 general-purpose extensions will be accepted into the official codebase.
513
514 |note|
515
516 See :doc:`config` for more information on how
517 extensions are configured, and :doc:`extensions` for
518 details on all of the officially-supported extensions.
519
520 Developers may be interested in :doc:`extenspec`.
521
522 ----------
523
524 *Previous*: :doc:`preface` • *Next*: :doc:`install`
525
526 ----------
527
528 .. [1]
529 See `<http://en.wikipedia.org/wiki/Setuid>`__
530
531 .. [2]
532 Some users find this surprising, because extensions are configured
533 with sequence numbers. I did it this way because I felt that running
534 extensions as part of the all action would sometimes result in
535 surprising behavior. I am not planning to change the way this works.
536
537 .. [3]
538 An ISO image is the standard way of creating a filesystem to be
539 copied to a CD or DVD. It is essentially a “filesystem-within-a-file”
540 and many UNIX operating systems can actually mount ISO image files
541 just like hard drives, floppy disks or actual CDs. See Wikipedia for
542 more information: `<http://en.wikipedia.org/wiki/ISO_image>`__.
543
544 .. [4]
545 The checksum is actually an SHA cryptographic hash. See Wikipedia for
546 more information: `<http://en.wikipedia.org/wiki/SHA-1>`__.
547
548 .. |note| image:: images/note.png
549 .. |tip| image:: images/tip.png
550 .. |warning| image:: images/warning.png