RsyncNet22010-10-10 09:38:54noway.chem.columbia.eduadd link to BackupInfo and database backup info12010-10-10 09:21:54noway.chem.columbia.edudocumenting how our backups workThis page describes the backup strategy used to back up hcoop.net to rsync.net. This page is currently a work in progress, and is being used to document the current state of things as they were set up some years ago. BackupInfo explains things from a how-to perspective. What happensrsync.net ssh's automatically (every three days) to deleuze (user unknown, probably user=rsync and authenticates using pub key in ~rsync/ssh/authorized_keys ) the shell run by rsync.net is /afs/hcoop.net/common/etc/scripts/rsync-shell
. This shell wrapper allows two commands: backup
and rsync
with a verified command line. The shell wrapper logs to /var/log/rsync-shell.log
The first ssh time runs backup
which executes sudo /afs/hcoop.net/common/etc/scripts/hcoop-backup-wrapper
The backup wrapper logs to /var/log/backup-to-rsync-wrapper-log
Temporary: exits to abort the backup fs checkvol
k5start to user hcoop
runs /afs/hcoop.net/common/etc/scripts/hcoop-backup
, logging to /var/log/backup-to-rsync-log
delete and recreate /vicepa/hcoop-backups/files
and a subdirectory for today. queries dpkg to find a) all files on filesystem (allfiles) b) all files shipped with packages (debfiles) c) all registered config files in packages (conffiles) take all files in allfiles that aren't in debfiles exclude various paths which are NOT needed to be backed up (e.g. /var/cache
) Add in all conffiles Take backupfiles and exclude various paths which we know should be backed up. What is left is "annoying files" - things to mail the admins about to add to the "expected to be backed up" list or to an exclude list. This keeps a human in the loop in case extra files appear Back up everything in this list of files using tar
, pipe through ccrypt
to encrypt and $MOVE_OVER - SPAMD: Wait for a spamd lock to become free, tar and ccrypt /var/local/lib/spamd
VOLUMES: vos listvol deleuze
, exclude .backup
and .readonly
, volumes containing not-backed-up
, compress and add to the backup directory DATABASES: back up databases from /var/backups/databases/
(TODO: how do databases get here? Need to locate that script). Change permissions of staging dir and change owner to uid=rsync
Send mail to admins about unexpected files the second time ssh runs (immediately after the first) it runs rsync, with a verified command line, backing up everything placed in /vicepa/hcoop-backups/files/YYYY-MM-DD/
to rsync.net, using rsync. rsync.net maintains the last two dates of backup (so about a week's worth) non-deleuze systems don't seem to be backed up. Database backups: /etc/cron.daily/hcoop-all-db-backup-to-var
, which runs /afs/hcoop.net/common/etc/scripts/hcoop-all-db-backup
To figure out: how we manage files on the rsync.net end