/ Today I learnt / notes

TIL: sync iPhone photos in Linux

December 22, 2019

I wrote a script that imports photos from iPhone into my Linux machine, converts HEIF/HEIC images into regular JPGs and MOV videos into VP9 videos in MKV.

While I was doing that, I learnt a bit about rsync, HandBrackeCLI, libheif and I'm sharing it here with the script itself.

Prerequisites

First of all, the tools that I needed to install. I'm using Arch Linux so all the package names and commands are Arch-specific.

Packages:

Requirements

I want a tool that does the following:

Implementation

Let's call our tool phsync for “photos synchronizer”.

First, several notes on directories structure. iPhone is trying to represent itself as a photo camera when you plug it into USB port so whenever you mount it with ifuse, photos are in DCIM directory there.

There can be multiple subdirectories in DCIM directory but file names in them never clash with one another so we can just dump all of the nested files into single target directory. I think the multiple subdirectories thing is done for performance reasons.

I'm going to need several directories:

And I'm going to need one file, I'll explain later why:

Now, I'll split process into multiple functions that the main script executes.

I'll be writing the script in most straight-forward way, no fancy bash features unless I really need them.

Get ready

First, lets ensure that we have all the environment ready:

validate_setup() {
    log "Validating the environment..."

    if which heif-convert; then
        log "Found 'heif-convert' installed"
    else
        error "'heif-convert' is required to convert photos, install libheif"
    fi

    if which HandBrakeCLI; then
        log "Found 'HandBrakeCLI' installed"
    else
        error "'HandBrakeCLI is require to convert videos"
    fi
}

OK, lets make sure that all the directories and files are present:

ensure_dirs() {
    log "Preparing directories..."

    mkdir -p "${MOUNT_DIR}"
    mkdir -p "${TARGET_DIR}"
    mkdir -p "${ORIGINALS_DIR}"
    touch "${RSYNC_DONE}"
}

And finally, mount the phone:

mount_phone() {
    log "Mounting phone to: $MOUNT_DIR"

    if ifuse "$MOUNT_DIR"; then
        log "Phone is mounted at: $MOUNT_DIR"
    else
        error "Failed to mount phone at: $MOUNT_DIR"
    fi
}

Now we're ready to download photos:

sync_photos() {
    local target="$TARGET_DIR"

    if [ $target == "" ]; then
        error "Target can't be empty"
    fi

    log "Synchronizing photos [target=$target]"

    if find "$SOURCE_DIR" -mindepth 1 -type d -name "*APPLE" -exec rsync \
        -azvhP \
        --size-only \
        --ignore-times \
        --no-perms \
        --no-owner \
        --no-group \
        --no-times \
        --exclude-from="$RSYNC_DONE" \
        {}/ \
        "$target" \;; then
        log "Photo sync completed"
    else
        umount_phone
        error "Failed to sync photos"
    fi
}

The if statement is a bit hard to read, so what here what it does in plain English:

I'm using find command to find all the subdirectories in DCIM directory

When the find finds something, I tell it to execute a command with that something: -exec.

Note that find requires -exec arguments to end with ; which in combination with shell syntax gives us \; so everything between -exec and \; is the command find will run for every found item. Find will replace {} with the found item.

In this case I'm running rsyc, lets go through the options I supply and what they do:

Last two arguments:

Now, the very important argument that is missing from the list above is this one:

--exclude-from="$RSYNC_DONE"

this will cause rsync to read the file $RSYNC_DONE and skip synchronization for every file mentioned in that file. One line - one file. This file does all the magic so we could delete photos and convert them and get rid of originals.

It requires a bit of house keeping on our part, so right after we've downloaded the photos, lets make a list of what we've got:

record_changes() {
    log "Marking received photos as 'synchronized'"

    find "${TARGET_DIR}" -type f -exec basename {} \; | xargs -I {} /bin/sh -c \
        'grep -qxF "{}" '"$RSYNC_DONE"' || echo "{}" >> '"$RSYNC_DONE"
}

again, it is a bit hard to read but what we do here is:

Now that we've got all the photos and we're keeping track of them, time to convert formats.

How do we do that?

libheif provides a number of command line utilities and one if them is heif-convert. It accepts HEIF photo as first argument and output file as second argument.

The photo format Apple uses called “HEIF” but extension in iPhone file system is “HEIC”. I decided to be on the safe side and write a script that would cover both “HEIF” and “HEIC” extensions.

process_photos_from_to() {
    local from="$1"
    local to="$2"

    log "Converting photos from: $from to: $to"

    if [ "$from" == "" ]; then
        error "'from' argument is not specified"
    fi

    if [ "$to" == "" ]; then
        error "'to' argument is not specified"
    fi

    find "${TARGET_DIR}" -type f -name "*.$from" -exec basename -s .$from {} \; | \
        xargs -I {} /bin/sh -c \
        'heif-convert '"${TARGET_DIR}"'/{}.'"$from"' '"${TARGET_DIR}"'/{}.'"$to"' && mv '"${TARGET_DIR}"'/{}.'"$from"' '"${ORIGINALS_DIR}/"
}

and use it for conversion:

process_photos() {
    log "Converting photos..."
    process_photos_from_to HEIC JPG
    process_photos_from_to HEIF JPG
    log "Done converting photos"
}

The conversion script above does the following:

Now that we've got rid of HEIC and have clean JPGs it is time to get rid of MOV and have VP9s.

NOTE: it may take a lot of time to convert videos. One of 4GB videos I had took about 24 hours to convert. Luckily you do it only once and you can actually disconnect the phone right after download is completed.

To convert videos we're going to use HandBrakeCLI.

HandBrakeCLI has a lot of presets for every occasion, I chose to use VP9 MKV 2160p60 It is fair bit slower than many other presets but I wanted to preserve the quality of my videos.

Here is how I convert the videos:

process_videos() {
    log "Converting videos..."
    find "${TARGET_DIR}" -type f -name "*.MOV" -exec basename -s .MOV {} \; | \
        xargs -I {} /bin/sh -c \
        'HandBrakeCLI -i '"${TARGET_DIR}"'/{}.MOV -o '"${TARGET_DIR}"'/{}.MKV --preset="VP9 MKV 2160p60" && mv '"${TARGET_DIR}"'/{}.MOV '"${ORIGINALS_DIR}/"
    log "Done converting videos"
}

Same as in images case it does:

Now we're ready to wrap it together:

sync() {
    validate_setup
    ensure_dirs
    mount_phone
    sync_photos
    record_changes
    process_photos
    process_videos
    umount_phone
}

and execute:

sync

Error handling

Something that I don't want is to have to manually unmount the phone if the script crashes. I just want to re-run it and expect it to do the right thing.

So I put this function at the very top of the file:

umount_phone() {
    log "Unmounting phone from: $MOUNT_DIR"

    if [ "$(mount | grep ifuse | wc -l)" -lt 1 ]; then
        error "Nothing to unmount"
    fi

    while true; do
        if fusermount -u "$MOUNT_DIR"; then
            log "Phone is unmounted"
            break
        else
            log "Failed to unmount phone, retrying in 1 sec"
            sleep 1s
        fi
    done
}

it does the following:

Also, at the very top of the file I set my script to fail on every error:

set -e

and I added a trap that runs umount_phone function before exiting from the script:

trap "umount_phone; exit" ERR EXIT

Conclusion

There you have it. The script downloads photos from an iPhone, converts photos from HEIF to JPG, converts videos from x264 MOV to VP9 MKV, keeps track of what is synchronized to not restore deleted files.

You can interrupt that script at any point, it will:

You can find full source of the script here