BackupWithRsync

Summary: How to backup a PmWiki installation with the program `rsync`.
Version: 0.1
Prerequisites: perl + dependencies
Status: abandoned
License: GPLv2
Maintainer: none (was: tamouse)
Users: (view? / edit)

How can I automate the backup a PmWiki installation with the program rsync.

The perl script wikibackups.zipΔ will automate the backup from a remote server to a local file system.

Updated: Tue Jul 10 22:33:06 CDT 2012 by tamouse

Description and Usage

Backups are a hugely important thing when you have a wiki full of information. PmWiki is not a difficult system to keep backed up, as everything is stored in flat files on the server. There have been a few solutions proposed. This method uses the a rolling backup method, using a hard-linking scheme to only pull over the changes in each backup iteration.

Obtaining wikibackups.pl

Download wikibackups.zipΔ to the system which will run the script and store the backups, and unzip into a safe place.

Alternately, you can make a clone from the git repository at gitorious and run make in the directory.

Dependencies

Wikibackups.pl uses the following perl modules:

Obtain the first two from cpan.org and install as usual.

Installation

  • copy wikibackups.pl to some place on your PATH, such as /usr/local/bin.
  • copy wikibackups.1 to a useful man path (/usr/local/share/man/man1 perhaps).
  • copy sample.wbu.rc to $HOME/.wbu.rc

Configuration

To configure wikibackups,

  • edit the configuration file in $HOME/.wbu.rc by putting in the wikiname, SOURCE, and TARGET parameters, and adjusting exclusions as you see fit. You can put information for multiple wikis in the same configuration file, just replicate the structure that is there and name each wiki appropriately.
    • note that $HOME/.wbu.rc is a YAML file. MAKE SURE YOU DON'T USE ANY TABS IN THIS FILE
  • ensure the TARGET directory exists; create it if it does not.

Running the script

SYNOPSIS:
    wikibackups.pl [-c config] [-n] [-v] [-d]

    wikibackups.pl [-h]

    wikibackups.pl [-V]
OPTIONS:
  • -c config - specify an alternate configuration file. The default is "$HOME/.wbu.rc".
  • -n - dry run only. If used with -v will show the list of files which rsync would have transfered.
  • -v - verbose mode. Will print out what is going on during the backup, including the output from the rsync command.
  • -d - debug mode. Will print out all sorts of internal structure information. Useful for checking your configuration file. If you want to check your configuration, make sure to set -n and -v as well.
  • -h - help. Prints a usage message.
  • -V - version. Prints the script version.

Testing

I strongly encourage thorough testing of the backup mechanism before automating it. In addition, make sure you can recover files you want from the backups generated.

Maintenance

This recipe will create backup versions, but it won't delete any. You may want to have something that will prune off old backup versions. The entire backup set should not take that much space unless your wiki experiences a great deal of entropy, since using the hard linking feature of *nix only takes extra space with files actually change, but it can be disconcerting to see many directories.

Further Info

See the man page, html page or txt page in the wikibackups installation for more information on configuring and running the backup.


Updated: tamouse October 23, 2011, at 02:26 PM

rsync is an extremely useful solution to backing up files to or from a remote location, provided you have ssh access to the remote machine. The rsync man page lists tons of options and examples for doing this well.

A simple example of using rsync to replicate your wiki:

    $ rsync -av user@server:path/to/wiki/ local/path/to/backup

This will bring over everything, including the pmwiki software and such, which you probably don't really need in a backup situation.

Inclusions and Exclusions: rsync filtering

Luckily, rsync let's you include and exclude things by using an --exclude-from parameter that you can drop file glob patterns into.

It's a bit arcane, and the man page is less than helpful, but I've come up with a set of filter rules that seem to work for backing up the important content of your wiki.

Create a file called, say, exclusions, and fill it with:

    # Exclude stuff from pmwiki except some
    + /cookbook/
    + /wiki.d/
    - /wiki.d/.flock
    - /wiki.d/.pageindex
    - /wiki.d/.lastmod
    - /wiki.d/*,del-*
    - /wiki.d/*/*,del-*
    + /pub/
    + /local/
    + /uploads/
    - /*
    - **~
    - **,v
    - **.bak
    - **.tgz
    - **.zip
    - **.gz
    - **.Z

This will allow the files matching include patters (prefix "+") and prevent the files matching exclude patterns (prefix "-") from being included in the rsync. Note well that the include patterns need to come before the exclude patterns. Here's what's going on above:

+ /cookbook/include the cookbook directory at the top level of the directory tree (i.e. if cookbook reappears in a subdirectory).
+ /wiki.d/include the wiki.d directory, again at the top level only
- /wiki.d/.flockexclude the .flock file under wiki.d
- /wiki.d/.pageindexexclude the .pageindex
- /wiki.d/.lastmodexclude .lastmod
- /wiki.d/*,del-*exclude deleted pages
- /wiki.d/*/*,del-*exclude deleted pages in a Group direcotry setting
+ /pub/include the pub directory
+ /local/include the local directory
+ /uploads/include the uploads directory
- /*exclude all other directories and files in root.

This last line here is really the key to the whole thing. If you don't have this line, you'll still get most everything in the installation. It basically says "exclude everything else from here".

What follows are more miscelaneous matching patterns for various kinds of files that may appear in the included directories up above.

- **~typical editor backupt file, such as emacs. the double star means match everything.
- **,vRCS vault file
- **.bakanother type of backup file, usually from filters
- **.tgztarballs, generally don't want to include those in your personal arvhive.
- **.zipzip files, similar to tarballs
- **.gzgzipped files, frequently tarballs are ended .tar.gz
- **.Zcompressed files

(Note: you may have good reason not to exclude some of the types of files I've shown above. Feel free to modify the above list; add and subtract entries as needed.)

Once you have your exclude file set up, then run the following command:

    $ rsync -av --exclude-from=exclusions user@server:path/to/wiki local/path/to/backup

Rsync has a nice parameter, --dry-run, which lets you see what it's going to do before you commit to the download. This is great for checking yout inclusions and exclusions to make sure nothing comes over that you don't want.

There are many more options to rsync that can make it the ideal backup solution, including making incremental and full backups, etc.

For a more robust backup setup using rsync, see http://www.sanitarium.net/golug/rsync_backups_2010.htm as it outlines a rather nice rolling backup. -- tamouse July 10, 2012, at 10:59 AM

Just keep in mind the 3 most important things:

BACKUPS, BACKUPS, BACKUPS

--tamouse October 23, 2011, at 09:40 AM. And updated tamouse October 23, 2011, at 02:26 PM.

Notes

Change log / Release notes

  • Version 0.1: 2012-07-10

If the recipe has multiple releases, then release notes can be placed here. Note that it's often easier for people to work with "release dates" instead of "version numbers".

See also

Contributors

Comments

See discussion at BackupWithRsync-Talk

User notes? : If you use, used or reviewed this recipe, you can add your name. These statistics appear in the Cookbook listings and will help newcomers browsing through the wiki.