Backup of my MyBook World II (blue ring)
I'm using my MyBook as the main storage for all of my documents. The most important documents are my pictures. I like to backup them once a day.
Everyday there are some new documents and some others got changed. All documents are stored in several directories. There are some directories and file types (e.g. temp files) which don't need to be backup'ed. Some files in other directories are changed every day, but I don't need a backup every day (e.g. emails, I have another copy on my providers server :-) ).
The whole backup has to be very easy to use and customizable to my needs. You will see that I preferred methods that can be done with standard tools on every operation system. There is no need to install software on the client. In fact you only need an editor and you must have the rights to create and modify files.
Requirements of my backup process
- backup runs automatically once a day
- it must be possible to exclude some specific file types; file types must be easily customizable
- it must be possible to exclude some specific directories by default; default exclude-directories must be easily customizable
- it must be possible to define that some specific directories are not backup'ed every day, but every week or month
- it must be possible to define that some specific directories are never backup'ed
- backups have to be differential backups, for saving backup disc space; the first backup must be a full backup
- number of saved backups must be customizable
- it must be possible to define the main source directory and the main target directory
- and last but not least - safety first
Safety is the most important thing for a backup. That includes that the backup can be read on any other (Linux) computer without need of some special software. Because of this requirement the usage of a backup software which uses container files (e.g. zip, tar) for collecting and compressing the files is not allowed. Disc space is very cheap today. So there is no need for me to compress the files. The only thing to save disc space is to make differential backups.
From my point of view it is not necessary to backup over a network. Backup is written to another disc (here an connected USB disc).
Maybe there are some solutions available somewhere in the web, maybe better ones. Nevertheless I decided to create my own script. This has two very important advantages:
- the script becomes very small and it can be adapted to the Mybook
- I really know what the script does (I should know it :-) )
I decided not to start from scratch, but to use some proved strategies. Many thanks to Mike Rubel who shows a really good scenario. My script is based on his work.
- backup is started once a day as a cron job
- rsync and hard links are used to create differential backups which look like complete backups
- rsync can handle exclude patterns in a exclude-file (works for files and directories)
For directories which are not backup'ed or not backup'ed every day the exclude pattern is dynamically added to the exclude-file
- the number of saved backups and all directory names are handled in the srcipt.
There are two different things which must be customized:
- static things like number of saved backups, source/target directories
- dynamical things like directories which should not be backup'ed or only backup'ed once a week or month
Static things are customized within the script itself. That is ok for me, because it is done only once. For dynamical things I decided to use a very easy way. I defined some keywords, or better keyfiles. That means in a directory which should never backup'ed a file named nobackup is created. The script is looking for the existence of that file and if it is found the directory and all sub-directories are never backup'ed. Similar for directories which should be backup'ed weekly (weeklybackup) or backup'ed monthly (monthlybackup).
The real advantage of this solution is: If you are copying some files to your MyBook with a file explorer then you can decide if these files should be backup'ed or not. If not then you create a file named nobackup. There is no need to write something to the file, it can stay empty. This is a very fast way. And you can see very quickly if a directory becomes backup'ed or not. Just search for the file.
The script MySnapshot
The backup script consists of the files:
- MySnapshot.sh: the script itself
- snapshot_rsync_exclude.txt: default exclude file; contains the default file names and default file patterns that are not included in the backup
The follwing additional files are created at run-time of the script, but deleted by the script itself (except one).
- snapshot_rsync.log: log
- snapshot_rsync_exclude.tmp: exclude file that is generated at runtime by the script. It contains everything from snapshot_rsync_exclude.txt and also the directories that should not be included into the backup (see nobackup, weeklybackup, monthlybackup).
- .TimeStamp: an empty file that serves as a temporary storage for a date / time of a directory, is not deleted, since it otherwise has to be constantly created and deleted.
#!/bin/bash # -------------------------------------------------------------- # Filename: MySnapshot.sh # JpgOrganizer's backup script, based on rsync # Thanks to Mike Rubel (http://www.mikerubel.org) # --------------------------------------------------------------
Default exclude file snapshot_rsync_exclude.txt
This file snapshot_rsync_exclude.txt contains one pattern per line for files and directories to exclude from backup.
*_old *.old *.bak *.svn *~* *.tmp *.sik *.log Thumbs.db nobackup weeklybackup monthlybackup .designerthumb
How to use
Installation and customizing
Only a few steps are necessary to install the backup script.
- connect your USB disc to your MyBook. Make sure it is formatted with ext2 or ext3 file system. NTFS, FAT, FAT32 will not work! If it is not ext2/3 then format it now -keep care - you will loose all your available data on your USB disc if you format it. And please be careful so that you do not format your MyBook disc's instead.
- copy Mysnapshot.sh and snapshot_rsync_exclude.txt to the directory you like on your MyBook. Both files are attached to this site.
- make sure the rights of Mysnapshot.sh are 755
- adjust the value of the variable SNAPSHOT_CONF in Mysnapshot.sh to that directory.
- adjust the value of the variables SNAPSHOT_MOUNT, DATA_SNAPSHOT_RW, DATA_BACKUP_DIRS_HOME to your situation
- have a look at all other configuration variables in the script and adjust them if you like
- make an entry in root's crontab to start the srcipt once a day.
All of your directories and files are now backup'ed every day. If you have a directory which should not be included into the backup, just create a file called nobackup (or what you have written in variable NOBACKUP. Similar with directories which you like to be backup'ed once a week or month (WEEKLYBACKUP, MONTHLYBACKUP).
There is one known bug:_
It is not possible to have files (!) in the directory given in DATA_BACKUP_DIRS_HOME. Only directories are allowed here. For me this is no restriction, maybe for others.
I hope this script can help somebody else to create his own backup solution. I'm sure that someone can improve my script. Feel free to do that. Some parts of the code are written in a way I found out - not the best one.