Backup and Restore
Differential Backup:
A differential backup will backup all files that have changed since the last full backup. In other words, if a full backup was done on Monday, Tuesday's differential will backup all changed files since Monday's full. Wednesday's differential will backup all changed files since Monday's full including the files that have changed on Tuesday.
The big advantage to this method comes when performing a complete restore since only the full and latest differential backups need to be restored. The downside is that the size of the differential backup will grow throughout the week and become progressively larger until the next full. This can adversely affect backup windows.
Incremental Backup
An incremental backup will backup all files that have changed since the last backup, regardless whether it was a full or incremental backup. In other words, if a full backup was done on Monday, Tuesday's incremental will backup all changed files since Monday's backup. However, Wednesday's incremental will only backup files that have changed since Tuesday's incremental backup.
The main advantage to this method is that a lot fewer files are backed up daily between full backups allowing for shorter backup windows. The disadvantage is that when performing a complete restore, the latest full and all subsequent incremental backups must be restored, which can take significantly longer.
The simplified technical explanation could be summarized by stating that both full and incremental backups reset the archive bit on a file (indicating it has been backed up) and a differential backup does not.
Backup and restore commands
cpio
To copy the current directory and all subdirectories onto tape
# find . -print | cpio -ov >/dev/rmt0
To copy a directory and all of its subdirectories, enter:
# find . -print | cpio -pdl /home/jim/newdir
To list the files that have been saved onto a tape with the cpio command, enter:
# cpio -itv </dev/rmt0
To copy the files previously saved with the cpio command from a diskette, enter:
# cpio -idmv </dev/rfd0
Using tar
tar
tar options:
-A append tar files to an archive -c create a new archive -d find differences between archive and file system -r append files to the end of an archive -t list the contents of an archive -u only append files that are newer than copy in archive -x extract files from an archive -l stay in local file system when creating an archive -M create/list/extract multi-volume archive -v verbose mode -j filter the archive through bzip2 -z filter the archive through gzip -Z filter the archive through compress -f <file_name> Name of the archive/Device
gnu tar options
--exclude=<pattern> # to exlcude directory or files matching the pattern.
# multiple --exclude option can be defined
--exclude-from=File # to exclude the list of pattern listed int he File. Use this option to read a list of
# patterns, one per line, from FILE; `tar' will ignore files matching those patterns.
To move all files in /tmp and it's sub directories to /tmp2
# cd /tmp # tar cf - ./* |(cd /tmp2; tar xvf -)
rsync
rsync examples:
Use below one to sync new and modified files via ssh custom port, but it will not clean the deleted files
#rsync -ave 'ssh -p 1661' -logptr <sorucedir> <remoteHost>:/<remote_dir>
Use below one to sync new and modified files, but it will not clean the deleted files and exclude the directories in the exclude file, It sync the data from myserver to currect directory
#rsync -ave 'ssh -p 1661' -logptr --exclude-from=/root/bin/exclude.list myserver:/mydata
Use below one to delete the files in destination also what ever deleted in source
#rsync -ave 'ssh' -logptr --delete --exclude-from=/root/bin/exclude.list myserver:/mydata/ /data/mydata
To compress the data and send it for saving bandwidth
# rsync -a -v -z -e "ssh -o Compression=no" <sorucedir> <remoteHost>:/<remote_dir>
You might want to customise how the remote shell behaves, for example this might be a better option for the backups:
# rsync -a -v -e "ssh -c arcfour -o Compression=no -x" user@3aims.com:/home/user web10
Here is what the options to -e mean:
ssh - use ssh instead of the default of rsh
-c arcfour - uses the weakest but fastest encryption that ssh supports
-o Compression=no - Turns off ssh's compression - rsync has its own if you want it which we'll discuss in a minute
-x - turns off ssh's X tunneling feature (if you actually have it on by default)
If bandwidth is a problem you might want to use the -z option to have rsync compress data it sends across the network. If you are using rsync compression it makes sense not to use ssh's compression in the way demonstrated above. Here's the command using rsync compression:
# rsync -a -v -z -e "ssh -c arcfour -o Compression=no -x" user@3aims.com:/home/user web10
If you want to test these how effective each of these commands are you will need to delete the web10 directory rsync creates otherwise rsync will only copy files which have changed. Whilst that's normally what you want, it isn't too useful for tests.
Finally, since rsync is very efficient it can saturate a network connection. If you still want to be able to use your network connection whilst rsync is running you can use the --bwlimit option which allows you to specify a maximum transfer rate in kilobytes per second. Due to the nature of rsync transfers, blocks of data are sent, then if rsync determines the transfer was too fast, it will wait before sending the next data block. The result is an average transfer rate equaling the specified limit. For example to limit rsync to using 100KB/sec you could do this:
# rsync -a -v -z --bwlimit=100 -e "ssh -c arcfour -o Compression=no -x" user@3aims.com:/home/user web10
You might also want to use --progress so that rsync prints out a %completion and transfer speed while transferring large files (but this isn't worth adding if you are running from a cron job). If you are performing a backup which you think you might want to restore at some point in the future you should use --numeric-ids. This tells rsync to not attempt to translate UID <> userid or GID <> groupid which is very important for avoiding permission problems when restoring. You might also want the -H option which forces rsync to maintain hardlinks on the server and the -x option which causes rsync to only copy files from one filesystem and not any other files which might be mounted as part of that directory structure. You can also use the --delete option which deletes files from the backup if they don't exist on the server. If you use --delete the files are deleted before the copying starts.
Putting this all together the command I use to backup one of my servers looks something like this:
# rsync -aHxvz --delete --progress --numeric-ids -e "ssh -c arcfour -o Compression=no -x" root@example.com:/ pauli/
Tape Drives and mt commands
Tape Processing in the Solaris OS
This Tech Tip offers advice on working with tapes and the Solaris OS, including tips for backing up data to tape.
A single file tape consists of one or more blocks of data followed by two EOF marks. The EOF marks are a tape drive hardware function.
A multi-file tape consists of each file separated by an EOF mark. The last file on a tape is marked by a sequence of two EOF marks. The two EOF marks (one after the other) show the logical end of tape, which is often not the physical end of tape.
Tape Terminology
The physical blocks on tape are the tape "records". These physical records are separated from each other by the inter-record gap, which is of fixed size, depending on the tape technology and also the tape manufacturer. To reduce the proportion of the tape that is made up of inter-record gaps, the block should be as large as possible, especially on high-density drives. This is accomplished by having the blocking factor as large as possible (this is the number of disk records per tape block). If you have small tape blocks, most of the tape is made up of inter-record gaps.
Often when you get a tape from a source other than the Solaris OS, the following could create problems:
- A header on the tape: For example, VERITAS NetBackup tapes have a tape header. Mainframe tapes often have a VOL1 header. The header is one tape file, so you need to skip over the first file by using the dd command.
- The character set could be different: Often tapes from the mainframe are in EBCDIC. These can be converted by the dd utility with the conv option.
- Big-endian/little-endian differences: On a 2-byte word, the left-most byte could be the high-order or low-order byte. This can be turned around using the swab option on dd.
Tape drives use the st (SCSI) tape driver. A lot of information is available from the manual pages of st.
When a new tape drive or any device is added, the Solaris OS will "see" the new device after a reboot. You can dynamically update the /devices tree by executing a devfsadm command.
The tape drives can be further configured by editing the st.conf file (/kernel/drv/st.conf). Normally this would be under the guidance of the tape drive vendor.
The most common tape device is the device with a name like /dev/rmt/0, which is accessing tape drive 0 using the raw interface. You can also have /dev/rmt/0c for tape compression (if the drive supports this feature). Also possible is /dev/rmt/0n, which will not rewind the tape after the current operation. (Tape drives will automatically rewind tapes at the end of the current operation unless the n option is used.) 0cn for non-rewindable compressed drive
Basic Tape Commands
To go through every tape file on the tape use:
# mt -f /dev/rmt/0n fsf 1
This moves the tape forward over one tape mark. Don't forget the n option -- Otherwise as soon as the tape is forwarded it will immediately rewind. This is useful if you want to skip over the VOL1 header.
To go backwards past an EOF marker use:
# mt -f /dev/rmt/0n bsf 1
To rewind the tape use:
# mt -f /dev/rmt/0n rew
To offline a tape use:
# mt -f /dev/rmt/0n offline
This typically opens the tape drive door and ejects the tape.
- Note that if you terminate the current tape operation it may take some time for the command prompt to come back as the tape is rewinding.
- There is no direct access to tape stacker (tape changer) functions, for example, mounting a tape on a tape drive from a tape library.
To find out the status of the tape drive use:
# mt -f /dev/rmt/0n status
You typically will get no reply while the tape drive is rewinding or the tape drive is busy.
If you need to read directly from the drive you can use the dd command:
# dd if=/dev/rmt/0 bs=<block size in bytes>
If you need to convert the character set, use this:
# dd if=/dev/rmt/0 bs=<block size in bytes> conv=ascii
This converts the data on the tape from EBCDIC to ASCII.
If you need to swap the bytes around (low endian to high endian), type:
#dd if=/dev/rmt/0 bs=<block size in bytes> conv=swab
Backing Up to Tape
Files can be backed up to tape using proprietary solutions such as VERITAS NetBackup or IBM Tivoli software. Files can also be backed up with the tar or cpio utility.
# cd /source # tar cvf b /dev/rmt/0cn 50 .
To make this run faster, you can get rid of the v (verbose) option.
Always make your tar backups without using absolute file names. An absolute tar backup (such as tar cvfb /dev/rmt/0cn 50 /source) would mean that you could only restore this archive to the /source directory.
To read back the tar file, use this:
# cd /restore # tar xfb /dev/rmt/0cn 50
To back up data with cpio, use this:
# cd /source # cpio -ocvB 50 . > /dev/rmt/0cn
Do the following to restore data using cpio:
# cd /restore # cpio -icvB 50 < /dev/rmt/0cn
http://www.sun.com/bigadmin/content/submitted/tape_processing.html