TL;DR After CrashPlan discontinued their home plan I found a good solution using open source software and an inexpensive yet reliable cloud storage provider. I've got automated replicated backups now using Borg, Rclone and Wasabi cloud storage - details below.

Several years ago I started using CrashPlan to back up one of my machines at home. Soon after, I signed up for the family plan and added more machines, including computers belonging to my siblings and my wife. The fact that it was essentially "always on" and doing frequent backups without ever having to think about it was fantastic. Additionally, the ability to do point-in-time restores came in handy on several occasions. Since I'm generally the IT guy for the family, I loved that the user interface was so easy to use. There were several times when someone in the family needed to recover all their data and they didn't need my help.

Recently CrashPlan announced that they were dropping the consumer customers and focusing on their enterprise customers. It makes sense I suppose, they were not making a lot of money off folks like me, and across our family plan we were using a whole lot of storage on their system.

In looking for a suitable replacement, there were several features any solution needed to have:

  • Cross-platform support for Linux and Mac
  • Automated, so there's no need to remember to click "backup"
  • Point-in-time recovery (or something close) so if you accidentally delete a file but don't notice until later, it's still recoverable
  • Inexpensive
  • Replicated data store for backup sets, so data exists in more than one place (i.e. not just backing up to a local USB drive)
  • Encrypted in case the backup files should fall into the wrong hands

As I looked for services similar to CrashPlan I googled around and asked my friends. One said he was using Arq Backup and was really happy with them, but no Linux support meant it was no good for me. Carbonite is similar to CrashPlan but would be expensive for my case since I've got multiple machines to backup. Backblaze offers unlimited backup at a good price ($5/month) but doesn't support Linux for their backup client, or at least they did not as of this writing. There were a few other options I looked at but none of them matched everything I was looking for. That meant I got to figure out a good approach to replicating what CrashPlan delivered for me and my family. I think what I worked out came out pretty close.

I knew there were lots of good options for backing up files on Linux systems. In fact I've been using rdiff-backup for at least ten years. This was usually for saving snapshots of remote filesystems locally. I had hopes of finding something that would do a better job of deduplicating backup data though, since I knew there were going to be some things (like music libraries and photos) that would potentially be duplicated on multiple computers.

BackupPC was also a really strong contender, but to be honest I had already gone a ways down the road of testing with my chosen solution before I was reminded of BackupPC.

backup diagram

Ultimately I landed on using Borg Backup and couldn't be happier with my decision. It fit all the criteria I listed above, and has a pretty healthy community of users and contributors. Borg offers de-duplication and compression, and works great on PC, Mac and Linux. Then I use Rclone to synchronize the backup repositories from the Borg host up to S3 compatible storage on Wasabi. Any S3-compatible storage will work for this, but I chose Wasabi because their price can't be beat and they out-perform Amazon's S3. With this setup I can restore files from the local Borg host or from Wasabi.

Installing Borg on my machine was as simple as "sudo apt install borgbackup". For the backup host I have one Linux machine that's always on, and a 1.5TB USB drive attached to it. This backup host could be something as lightweight as a Raspberry Pi if you don't already have a machine available for this. Just make sure all the client machines are able to reach this server over SSH and you are good to go.

On the backup host, you can initialize a new backup repository with:

$ borg init /mnt/backup/repo1

Depending on what you're backing up, you might choose to make multiple repositories per machine, or possibly one big repository for all your machines. Since Borg de-duplicates, if you have identical data on many computers it might make sense to send backups from all those machines to the same repository.

Installing Borg on the Linux client machines was very straightforward. On OSX I needed to install XCode and Homebrew first. I followed this how-to to install the command line tools, and then used "pip3 install borgbackup".

Each of the machines have a backup.sh script (contents below) that is kicked off by cron at regular intervals. It will only make one backup set per day, so it doesn't hurt to try a few times in the same day. The laptops are set to try every two hours because there's no guarantee they will be on at a certain time, but it's very likely they'll be on during ONE of those times. This could be improved by writing a daemon that's always running and triggers a backup attempt any time the laptop wakes up. For now though I'm happy with the way things are working.

I could skip the cron job and provide a relatively easy way for each user to trigger a backup using BorgWeb but I really don't want anyone to have to remember to back things up. In my personal experience I tend to forget to click that backup button until I'm in dire need of a restoration (at which point it's way too late!)

The backup script I'm using came from the Borg docs quickstart, plus I added a little check at the top to see if borg is already running, so the script will exit if the previous backup run is still in progress. This script makes a new backup set and labels it with the hostname and current date. It then prunes old backup sets with an easy to understand retention schedule.

backup.sh:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#!/bin/sh
REPOSITORY=borg@borgserver:/mnt/backup/repo1

#Bail if borg is already running, maybe previous run didn't finish
if pidof -x borg >/dev/null; then
    echo "Backup already running"
    exit
fi

# Setting this, so you won't be asked for your repository passphrase:
export BORG_PASSPHRASE='thisisnotreallymypassphrase'
# or this to ask an external program to supply the passphrase:
export BORG_PASSCOMMAND='pass show backup'

# Backup all of /home and /var/www except a few
# excluded directories
borg create -v --stats                          \
    $REPOSITORY::'{hostname}-{now:%Y-%m-%d}'    \
    /home/doc                                   \
    --exclude '/home/doc/.cache'                \
    --exclude '/home/doc/.minikube'             \
    --exclude '/home/doc/Downloads'             \
    --exclude '/home/doc/Videos'                \
    --exclude '/home/doc/Music'                 \

# Use the `prune` subcommand to maintain 7 daily, 4 weekly and 6 monthly
# archives of THIS machine. The '{hostname}-' prefix is very important to
# limit prune's operation to this machine's archives and not apply to
# other machine's archives also.
borg prune -v --list $REPOSITORY --prefix '{hostname}-' \
    --keep-daily=7 --keep-weekly=4 --keep-monthly=6

The output from a backup run looks like this:

------------------------------------------------------------------------------
Archive name: x250-2017-10-05
Archive fingerprint: xxxxxxxxxxxxxxxxxxx
Time (start): Thu, 2017-10-05 03:09:03
Time (end):   Thu, 2017-10-05 03:12:11
Duration: 3 minutes 8.12 seconds
Number of files: 171150
------------------------------------------------------------------------------
                       Original size      Compressed size Deduplicated size
This archive:               27.75 GB             27.76 GB 323.76 MB
All archives:                3.08 TB              3.08 TB 262.76 GB

                       Unique chunks         Total chunks
Chunk index:                 1682989             24007828
------------------------------------------------------------------------------
[...]
Keeping archive: x250-2017-09-17                      Sun, 2017-09-17 03:09:02
Pruning archive: x250-2017-09-28                      Thu, 2017-09-28 03:09:02

Once I had all the machines backing up to the host, I followed the instructions for installing a precompiled rclone binary, and set it up to access my wasabi account. You can find details on using rclone with Wasabi here.

This script runs each night to synchronize any changes to the backup sets:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#!/bin/bash
set -e

repos=( repo1 repo2 repo3 )

#Bail if rclone is already running, maybe previous run didn't finish
if pidof -x rclone >/dev/null; then
    echo "Process already running"
    exit
fi

for i in "${repos[@]}"
do
    #Lets see how much space is used by directory to back up
    #if directory is gone, or has gotten small, we will exit
    space=`du -s /mnt/backup/$i|awk '{print $1}'`

    if (( $space < 34500000 )); then
       echo "EXITING - not enough space used in $i"
       exit
    fi

    /usr/bin/rclone -v sync /mnt/backup/$i wasabi:$i >> /home/borg/wasabi-sync.log 2>&1
done

The first synchronization of the backup set up to Wasabi with rclone took several days, but that was because it was around 400gb of new data and my outbound connection is not super fast. Since then the daily delta is very small and completes in just a few minutes.

Restoring files is NOT as easy as it was with CrashPlan, but it is relatively straightforward. The fastest approach is to restore from the backup stored on the Borg backup server. Here are some example commands used to restore.

#List which backup sets are in the repo
$ borg list borg@borgserver:/mnt/backup/repo1
Remote: Authenticated with partial success.
Enter passphrase for key ssh://borg@borgserver/mnt/backup/repo1: 
x250-2017-09-17                      Sun, 2017-09-17 03:09:02
#List contents of a backup set
$ borg list borg@borgserver:/mnt/backup/repo1::x250-2017-09-17 | less
#Restore one file from the repo
$ borg extract borg@borgserver:/mnt/backup/repo1::x250-2017-09-17 home/doc/somefile.jpg
#Restore a whole directory
$ borg extract borg@borgserver:/mnt/backup/repo1::x250-2017-09-17 home/doc

If something happens to the local Borg server or the USB drive holding all the backup repositories, I can also easily restore directly from Wasabi. The machine needs to have rclone installed, and then using rclone mount I can mount the remote storage bucket as though it were a local filesystem.

#Mount the S3 store and run in the background
$ rclone mount wasabi:repo1 /mnt/repo1 &
#List archive contents
$ borg list /mnt/repo1
#Extract a file
$ borg extract /mnt/repo1::x250-2017-09-17 home/doc/somefile.jpg

Now that I've been using this backup approach for a few weeks I can say I'm really happy with it. Setting everything up and getting it running was a lot more complicated than just installing CrashPlan of course, but that's the difference between rolling your own solution and using a service. I will have to watch this more closely to be sure backups continue to run and the data gets properly synchronized up to Wasabi. But in general replacing CrashPlan with something offering comparable backup coverage at a really reasonable price turned out to be a little easier than I expected. If you see room for improvement please let me know!


Speaking, streaming and brewing

Fri September 08 2017 by Christopher Aedo

Science Hack Day PDX

Tomorrow and Sunday PSU is hosting Science Hack Day PDX. This free to attend avent is going to be a blast, I'm really looking forward to it. I will be there checking out what people are working on, and interviewing them on Twitch. We will likely end up editing together …

read more

Brewing Beer for IBM

Tue September 05 2017 by Christopher Aedo

TwitchTV logo

On Monday, September 11th at 9am PDT I'm going to brew beer live on twitchTV! I'm as surprised as anyone that I get to do this during working hours and as part of my job. Really looking forward to this :)

My employer (IBM) is trying to draw attention to some …

read more

Making Beer with Linux, Python and a Raspberry Pi

Thu July 13 2017 by Christopher Aedo

taps

I was lucky to have a talk about brewing beer with Linux, Python and a Respberry Pi chosen for OpenWest 2017. As part of that talk I wanted to put up a quick post with the slides from the talk and links to some of the resources I used. It's …

read more

Delivering a keynote at OSCON 2017

Sat May 27 2017 by Christopher Aedo

Working as a developer advocacy program director for IBM, improving how the world of developers perceive our company is my top priority. One of the ways I believe we can make a huge impact is by using our time in front of developers wisely. That means no product pitches at …

read more

Next Few Talks in 2017

Fri April 21 2017 by Christopher Aedo

image

I'm sitting here on a plane headed home, finally, after what has been a pretty long week of travel for work. All of it has been good stuff, and even included the opportunity to share the stage with IBMs Chief Digital Officer Bob Lord. During his quarterly "all hands" town …

read more

10 Bills You Should Read

Mon March 06 2017 by Christopher Aedo

I try not to post too much political stuff here unless I think it's something useful and interesting. In this case, I've seen this list of bills proposed in the US house of representatives several places. All the times I've seen it however, it's just text (and often on twitter …

read more

Live on Gitlab

Thu February 02 2017 by Christopher Aedo

Last year when gitlab announced they would offer free site hosting with custom domains (similar to github pages) I decided it was about time I looked into moving this blog. Until today, it had been hosted on a VM running on either Google Compute, OVH, Vexxhost or AWS. I think …

read more

A Typical Brewday

Fri December 30 2016 by Christopher Aedo

While writing about my electric brewing system I thought it would be helpful to outline what a typical brew day looks like with that setup. Since that post was already getting too long I decided to put this in it's own relatively small post.

The FIRST step is to figure …

read more

The Electric Brewery

Sun December 25 2016 by Christopher Aedo

One of my best friends (bones!) sent me an email asking if I'd ever heard of this thing called Hosehead (it's a control system for an electric brewery using open source software called Strangebrew Elsinore). I had never heard of it, but had been thinking on and off for some …

read more