Scandir errors with scripted backups on OSX

A few years ago I documented how I automatically back up my computer, plus my family members' machines and the process has been working really well. Recently however I noticed some directories were not getting backed up on OSX machines. Turns out since I updated to Catalina, the stricter security was preventing borg from accessing all the directories it was able to previously.

I was seeing errors like this:

/Users/myuser/Documents: scandir: [Errno 1] Operation not permitted: '/Users/myuser/Documents'

A few searches on duckduckgo showed me other borg users (along with lots of other users of things running from bash on OSX) were hitting the same problem. With OSX security you can grant full-disk access to binary applications, but you can't do it very easily or safely to a shell script. General consensus though was that if you wrap a call to your shell script in a simple compiled binary, OSX will gladly let you grant full disk access to that binary.

This page had a good example that I started from. With a little help from a coworker (I don't know golang at all!) I adjusted the example to ensure it sends output from the scripts to stdout.

package main

import (
    "log"
    "os"
    "os/exec"
    "path/filepath"
)

func main() {
    ex, err := os.Executable()
    if err != nil {
        log.Fatal(err)
    }
    dir := filepath.Dir(ex)
    script := filepath.Join(dir, "backup.sh")
    cmd := exec.Command("/bin/sh", script)
    output, err := cmd.CombinedOutput()
    if err != nil {
        log.Fatal(err)
    }
    log.Println(string(output))
}

After saving that to backup.go and compiling with "go build backup.go", I had a tiny go binary that called the backup script. I copied backup and backup.sh to /Applications, then updated ~/Library/LaunchAgents/net.aedo.backup.plist to point to /Applications/backup. Then all I had to do was grant full disk access to /Applications/backup!

Now things are back to normal, and once again I'm getting everything backed up to a local disk as well as an S3 bucket on Wasabi. Hopefully if you're running into a similar issue this helps!

NOTE: Good to set an event to wake the computer up every night when you want to run the backup. Use pmset to do that.