welcome: please sign in

The following 205 words could not be found in the dictionary of 7 words (including 7 LocalSpellingWords) and are highlighted below:
abort   about   add   Add   admins   afs   after   ago   all   allfiles   allows   an   and   annoying   appear   are   aren   as   authenticates   authorized   automatically   back   Back   backed   backing   Backup   backup   backupfiles   backups   be   become   being   by   cache   case   ccrypt   Change   change   checkvol   command   commands   common   compress   conffiles   config   containing   cron   current   currently   daily   Database   databases   dates   days   db   debfiles   delete   deleuze   describes   dir   directory   do   document   don   dpkg   encrypt   end   etc   every   everything   exclude   executes   exits   expected   explains   extra   figure   files   filesystem   find   first   for   free   from   fs   get   happens   hcoop   here   how   human   immediately   in   Info   is   it   k5start   keeps   key   keys   know   last   left   lib   line   list   listvol   local   locate   lock   log   logging   logs   loop   mail   maintains   manage   Need   needed   net   non   not   of   on   or   out   owner   packages   page   paths   permissions   perspective   pipe   placed   probably   progress   pub   queries   readonly   recreate   registered   rsync   run   runs   script   scripts   second   seem   Send   set   shell   shipped   should   so   some   spamd   ssh   staging   state   strategy   subdirectory   sudo   systems   Take   take   tar   Temporary   that   The   the   they   things   This   this   three   through   time   to   To   today   two   uid   unexpected   unknown   up   used   user   using   var   various   verified   vicepa   volumes   vos   Wait   we   week   were   What   which   with   work   worth   wrapper   years  

Clear message
Edit

RsyncNet

This page describes the backup strategy used to back up hcoop.net to rsync.net. This page is currently a work in progress, and is being used to document the current state of things as they were set up some years ago.

BackupInfo explains things from a how-to perspective.

1. What happens

  1. rsync.net ssh's automatically (every three days) to deleuze (user unknown, probably user=rsync and authenticates using pub key in ~rsync/ssh/authorized_keys )

  2. the shell run by rsync.net is /afs/hcoop.net/common/etc/scripts/rsync-shell. This shell wrapper allows two commands: backup and rsync with a verified command line. The shell wrapper logs to /var/log/rsync-shell.log

  3. The first ssh time runs backup which executes sudo /afs/hcoop.net/common/etc/scripts/hcoop-backup-wrapper

    1. The backup wrapper logs to /var/log/backup-to-rsync-wrapper-log

    2. Temporary: exits to abort the backup

    3. fs checkvol

    4. k5start to user hcoop

    5. runs /afs/hcoop.net/common/etc/scripts/hcoop-backup, logging to /var/log/backup-to-rsync-log

      1. delete and recreate /vicepa/hcoop-backups/files and a subdirectory for today.

      2. queries dpkg to find a) all files on filesystem (allfiles) b) all files shipped with packages (debfiles) c) all registered config files in packages (conffiles)
      3. take all files in allfiles that aren't in debfiles

      4. exclude various paths which are NOT needed to be backed up (e.g. /var/cache)

      5. Add in all conffiles
      6. Take backupfiles and exclude various paths which we know should be backed up. What is left is "annoying files" - things to mail the admins about to add to the "expected to be backed up" list or to an exclude list. This keeps a human in the loop in case extra files appear

      7. Back up everything in this list of files using tar, pipe through ccrypt to encrypt and $MOVE_OVER -

      8. SPAMD: Wait for a spamd lock to become free, tar and ccrypt /var/local/lib/spamd

      9. VOLUMES: vos listvol deleuze, exclude .backup and .readonly, volumes containing not-backed-up, compress and add to the backup directory

      10. DATABASES: back up databases from /var/backups/databases/ (TODO: how do databases get here? Need to locate that script).

      11. Change permissions of staging dir and change owner to uid=rsync

      12. Send mail to admins about unexpected files

  4. the second time ssh runs (immediately after the first) it runs rsync, with a verified command line, backing up everything placed in /vicepa/hcoop-backups/files/YYYY-MM-DD/ to rsync.net, using rsync.

  5. rsync.net maintains the last two dates of backup (so about a week's worth)
  6. non-deleuze systems don't seem to be backed up.

Database backups:

To figure out:

RsyncNet (last edited 2010-10-10 09:38:54 by noway)