Tuesday, August 16. 2011
Backing Up Data Into an Encrypted External Disk
Today entry deals with a common and usually annoying task, backing up your laptop, so this time the post is going to be more useful for me than for anybody else. Last week my company gave me a 2TB USB 3.0 disk to make backups of my laptop data. But there is a requirement, the backup File System should be encrypted. As always there is an absolutely clear procedure to make all the task in Windows but not in Linux or Mac. Besides the procedure involves a closed source program (a McAfee application) to encrypt the external disk. In my opinion the requirements have never to mention particular applications and only guarantee the final goal. I can understand that a lot of custom solutions would be a mess for the company (to recover data when a employee leaves the company or whatever) but, please, at least think about all the OSes and provide a step by step procedure for all of them (Windows, Mac and Linux).
The mate that gave me the disk finally told me that, as I use Linux, I did not have to encrypt the disk. But searching for information over the Internet (I knew nothing about disk encryption before) I realized that this task would not be very difficult. I found two good pages: 7 Steps To An Encrypted Partition (local or removable disk) and Setting up an encrypted volume on an external hard drive on CentOS. I also found a video of how to install Debian on a encrypted root file system (this task has nothing to do with the backup but I think it will be the next requirement for my laptop). So I finally decided to encrypt the disk anyways. This entry summarizes the setup for the two tasks using Debian, encrypting the disk and performing the backup.
-
Step 1. The first step is configuring the partitions inside the disk. I will use two partitions, one encrypted (for the backup and confidential or customer data) and other not (for general use). The first time I plugged the disk my Debian box recognized it as /dev/sdb, so from here the /dev/sdb1 will be the encrypted and /dev/sdb2 the normal partition.
# fdisk /dev/sdb
I deleted the FAT32 partition the disk came with and created two new ones. First partition (/dev/sdb1) of 1TB:
Partition number (1-4, default 1): Using default value 1 First sector (2048-3907029167, default 2048): Using default value 2048 Last sector, +sectors or +size{K,M,G} (2048-3907029167, default 3907029167): +1024G
And the second one (/dev/sdb2) with the rest of the disk:
Partition number (1-4, default 2): Using default value 2 First sector (2147485696-3907029167, default 2147485696): Using default value 2147485696 Last sector, +sectors or +size{K,M,G} (2147485696-3907029167, default 3907029167): Using default value 3907029167
After saving the changes the disk presents the following two partitions:
# fdisk -l /dev/sdb Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xee7a7a06 Device Boot Start End Blocks Id System /dev/sdb1 2048 2147485695 1073741824 83 Linux /dev/sdb2 2147485696 3907029167 879771736 83 Linux
-
Step 2. It seems that for encrypted devices it is compulsory to write some random data in the partition (it is easier to decrypt a blank initialized disk). For this step there are various options (see the links I presented before) but I finally executed the badblocks command:
# badblocks -c 10240 -s -t random -v -w /dev/sdb1
Take care, in my 1TB partition this step took hours (my disk is giving around 26MB/s write rate).
-
Step 3. Now it is time to create the encrypted file system in the /dev/sdb1 disk partition. The method used in Linux to encrypt File Systems uses the dm-crypt kernel module and creates a virtual device below /dev/mapper (it is a kind of wrapper over the real /dev/sdb1 but encrypting the data before writing on disk).
First of all the package for the management command needs to be installed, the module has to be loaded in the kernel and added to the /etc/modules file (modules in this file are loaded at boot time).
# apt-get install cryptsetup # modprobe dm-crypt # echo "dm-crypt" >> /etc/modules
Once the module is in place the new encrypted mapper is created above /dev/sdb1. I have used default values and normal password setup (it is also possible to use a key file instead of a password, see this archlinux page for more information about this).
# cryptsetup luksFormat -v -y /dev/sdb1 WARNING! ======== This will overwrite data on /dev/sdb1 irrevocably. Are you sure? (Type uppercase yes): YES Enter LUKS passphrase: Verify passphrase: Command successful.
The default cypher is aes-cbc-essiv:sha256, sha1 for password hashing and 256 for key size (all this values can be changed using -c, -h and -s options of the command respectively). This is the summary for my device.
# cryptsetup luksDump /dev/sdb1 LUKS header information for /dev/sdb1 Version: 1 Cipher name: aes Cipher mode: cbc-essiv:sha256 Hash spec: sha1 Payload offset: 4096 MK bits: 256 UUID: ff088328-138a-4ca7-aa0a-5cbdd2064ce8 Key Slot 0: ENABLED Key Slot 1: DISABLED Key Slot 2: DISABLED Key Slot 3: DISABLED Key Slot 4: DISABLED Key Slot 5: DISABLED Key Slot 6: DISABLED Key Slot 7: DISABLED
Finally the new encrypted device is opened with backup name (the password is requested at this point).
# cryptsetup luksOpen /dev/sdb1 backup Enter passphrase for /dev/sdb1:
After the luksOpen a /dev/mapper/backup device is ready to be initialized and used.
-
Step 4. I have decided to use the new btrfs file system. The reason is that this next generation file system manages some nice features like on the fly compression and snapshots, encryption and some synchronization integration are also planned (see the project ideas). Obviously compression will save some space in the disk and snapshots let us create some backup copies with a minimal cost in terms of time and space (my idea is backing up data and maintaining some snapshots of the backups).
The btrfs module is currently marked as experimental in my 3.0.0 Debian kernel but its web page claims that since 2.6.31 only forward compatible disk format changes are planned. In order to create the FS the tools package has to be installed before.
# apt-get install btrfs-tools
Now the FS is made with BACKUP name.
# mkfs.btrfs -L BACKUP /dev/mapper/backup WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using fs created label BACKUP on /dev/mapper/backup nodesize 4096 leafsize 4096 sectorsize 4096 size 1024.00GB Btrfs Btrfs v0.19
The new FS is mounted for testing and its owner changed to my common user.
# mount /dev/mapper/backup /mnt # df -h /mnt Filesystem Size Used Avail Use% Mounted on /dev/mapper/backup 1008G 200M 957G 1% /mnt # chown ricky:ricky /mnt
At this point the FS can be unmounted and closed. I will never mount the partition again manually, the idea is that gnome mounts it automatically (password is requested as soon as LUKS FS is detected).
# umount /mnt # cryptsetup luksClose /dev/mapper/backup
-
Step 5. Now it is time to think about the backup. The easiest way to backup some directories in Linux is using the rsync utility. This application is a powerful, fast and versatile file copying tool which supports local and remote (ssh for example) copies. The idea for the backup is the following:
- When the external USB disk is plugged gnome mounts them (both partitions) automatically. For the encrypted file system the password is requested. Inside the root mount point a backup script is located, just clicking on it and running it the backup is started. Gnome automounter mounts the FS with default options. That is the reason I finally decided not to use the compression option of btrfs (UDEV is used for the automatic mount, and perhaps a new UDEV rule can be used to change the default behavior but I preferred not to complicate the solution).
- This script called backup-system-gksudo.sh is a simple script that just opens a new gnome-terminal executing gksudo (gnome graphical sudo) over the real backup script backup-system.sh (btrfs command needs root access).
- The backup-system.sh script reads a local and hidden configuration file called .backup-config. This file has some general variables like the directory to maintain in sync (usually it would be / directory but, in my case, /home is where all my data are placed), number of snapshots to maintain, directory for host backups, file for excluded dirs,...
- First the script creates a new btrfs sub-volume for the specified box (this way the same disk can hold backups for several boxes). The volume is placed in host-backup/$(hostname) directory (the first directory name is customizable via the configuration file). If the volume already exists this step is just skipped.
- After that the script continues executing the rsync command. The command is called with the following options:
rsync -a --delete -v --stats --progress \ --exclude-from="${dirName}/${EXCLUDED_DIRS_FILE}" \ "${SOURCE_BACKUP_DIR}" "${hostDirName}"
With this execution the SOURCE_BACKUP_DIR is in sync against the external and encrypted btrfs volume. The -a options is for archive mode, --delete option removes files deleted in the source directory, and -v, --stats and --progress options are for showing some information while copying. The --exclude-from option let us exclude some directories from the copy, the pattern file is defined inside the configuration (in my case this var points to a .exclude-dirs file that contains entries like videos or comics which are huge directories I do not like to backup). This step can take hours the first time but it is quite fast when only changes are sent. - When the rsync command finishes a btrfs snapshot is taken. This snapshot is located at the same level of the host sub-volume but labeled with the current date (the snapshot name is like this $(hostname)-YYYYMMDDHHMMSS).
- Finally old snapshots are deleted (in the configuration file there is a MAX_SNAPSHOTS variable, if the snapshots are more than this number the old ones are removed).
- At the end of the script a message is shown to the user. Zenity command is used to tell the user that the backup has been successfully done or the script has terminated with errors.
Here a video is presented to show the backup execution. It is at accelerated speed, the real backup took around two minutes. First the disk is inserted and gnome requests the password for the encrypted partition. When the device is automatically mounted the graphical script is executed (first I show that only one snapshot is in the hosts directory). This script launches a new gnome-terminal, after the root password is also requested the backup (rsync) and snapshot are performed. Once the script finishes a successful message is shown. Finally I enter in the host directory and show the just created snapshot directory.
In summary this entry comments how to implement a backup utility over an encrypted external disk in Linux (Debian). I have written it just to remember what I did (I had never worked with this utilities, cryptsetup, dm-crypt, btrfs or rsync). As you have seen my idea is keeping it as simple as possible. Only the chosen File System (btrfs) is a bit strange, but thinking about current and announced features it seems to be the proper one (maybe in the near future btrfs alone can do all the stuff explained here, sync and encryption included, avoiding the use of any other program). Please reuse and adapt it if you think it is useful for you.
Remember to backup your ideas too!
Comments