22010-10-10 09:38:54noway.chem.columbia.eduadd link to BackupInfo and database backup info12010-10-10 09:21:54noway.chem.columbia.edudocumenting how our backups workThis page describes the backup strategy used to back up hcoop.net to rsync.net. This page is currently a work in progress, and is being used to document the current state of things as they were set up some years ago. BackupInfo explains things from a how-to perspective.

rsync.net ssh's automatically (every three days) to deleuze (user unknown, probably user=rsync and authenticates using pub key in ~rsync/ssh/authorized_keys ) the shell run by rsync.net is /afs/hcoop.net/common/etc/scripts/rsync-shell. This shell wrapper allows two commands: backup and rsync with a verified command line. The shell wrapper logs to /var/log/rsync-shell.log The first ssh time runs backup which executes sudo /afs/hcoop.net/common/etc/scripts/hcoop-backup-wrapper The backup wrapper logs to /var/log/backup-to-rsync-wrapper-log Temporary: exits to abort the backup fs checkvol k5start to user hcoop runs /afs/hcoop.net/common/etc/scripts/hcoop-backup, logging to /var/log/backup-to-rsync-log delete and recreate /vicepa/hcoop-backups/files and a subdirectory for today. queries dpkg to find a) all files on filesystem (allfiles) b) all files shipped with packages (debfiles) c) all registered config files in packages (conffiles) take all files in allfiles that aren't in debfiles exclude various paths which are NOT needed to be backed up (e.g. /var/cache) Add in all conffiles Take backupfiles and exclude various paths which we know should be backed up. What is left is "annoying files" - things to mail the admins about to add to the "expected to be backed up" list or to an exclude list. This keeps a human in the loop in case extra files appear Back up everything in this list of files using tar, pipe through ccrypt to encrypt and $MOVE_OVER - SPAMD: Wait for a spamd lock to become free, tar and ccrypt /var/local/lib/spamd VOLUMES: vos listvol deleuze, exclude .backup and .readonly, volumes containing not-backed-up, compress and add to the backup directory DATABASES: back up databases from /var/backups/databases/ (TODO: how do databases get here? Need to locate that script). Change permissions of staging dir and change owner to uid=rsync Send mail to admins about unexpected files the second time ssh runs (immediately after the first) it runs rsync, with a verified command line, backing up everything placed in /vicepa/hcoop-backups/files/YYYY-MM-DD/ to rsync.net, using rsync. rsync.net maintains the last two dates of backup (so about a week's worth) non-deleuze systems don't seem to be backed up. Database backups: /etc/cron.daily/hcoop-all-db-backup-to-var, which runs /afs/hcoop.net/common/etc/scripts/hcoop-all-db-backup To figure out: how we manage files on the rsync.net end