Told you it wouldn't be long until I posted again.
Spoiler Alert
We've finished up a bunch of internal testing over the past few weeks, and there's a beta of SuperDuper! for High Sierra and APFS linked at the bottom of this post. But it's so exciting, in a totally nerdy way, that it would be a mistake to not follow the whole story, with its twists and turns. So let's dive in.
Our Antagonist
As I mentioned in the last post, APFS is basically undocumented. That required us to make certain assumptions about its capabilities, and to draw inferences from what's on-disk in order to produce something that replicates the source.
APFS volumes are hosted in a container, which is basically a GUID partition on a drive. That container has a pool of storage which is shared between all the APFS volumes in the container. So, in a 1TB container, you could have a large number of volumes, each of which can draw from the 1TB of storage available in its container.
Only APFS volumes can be in the container: HFS+ volumes are not allowed.
In addition to the startup volume, and any volumes that you create yourself, a container that hosts bootable volumes also contains three hidden volumes with special "roles". One, called Preboot, contains all the information required to start the boot process. Another called Recovery, is the recovery volume. The third, VM, is where swap files are stored.
These volumes are minimally documented in the asr and diskutil man pages (the asr page actually discusses what makes up a valid system), so we're quite confident about what we've done in terms of the volumes and their roles. Unfortunately, their contents are not, and so here is where we've made assumptions. Also, since we can't create our own volume contents, we copy that data from the source's equivalent volumes.
Papers, Please
Both HFS+ and APFS volumes have their own unique ID (UUID). Each special volume mentioned above associates various information with the unique ID of the corresponding bootable, role-less APFS volume. Unfortunately, those unique IDs cannot be preserved across an erase: they're read-only. As such, there's significant housekeeping involved in maintaining these special volumes, and in positively identifying the volumes you've selected as the source and destination.
On top of that, the reboot and recovery data can't just be copied whole from the source, because multiple bootable APFS drives in a single container share the same special volumes... so, as you might expect, there's also new logic involved.
This all works its magic behind the scenes, and you shouldn't notice any changes...although you might notice some subtle improvements.
There and Back Again
As I mentioned in the previous post, we allow APFS to HFS+ copies (and back). However, copying from HFS+ requires the special volume contents to be stashed on the backup copy, since HFS+ volumes don't have all these special role volumes (and their partition schemes don't allow the same kind of 'guaranteed' partition creation). That's something we don't do in this first beta.
In addition, given the full extent of APFS's capabilities are not known, we're not fully confident in allowing this type of copying in our beta, and discourage the use of HFS+ to APFS copying. If you need to restore HFS+ to a bootable APFS volume, reinstall the OS to a clean volume and point to the backup when prompted during first boot.
And When You Smile for the Camera... (RIP Walter)
Ah, but these stories always have a twist! We think this is a good, Unbreakable twist, too.
One of the challenges of copying a live volume is that it's active and always changing. Even as we're stepping through the data, the user could delete files out from under us, change the data...almost anything can happen. In general, it doesn't. But it could. That's why the User's Guide suggests quitting applications before copying.
One awesome thing that APFS has is the ability to create "snapshots" that freeze the state of an APFS volume. Copies from the snapshot will exactly match the state of the source at the moment the snapshot is taken, so you don't have to worry about what applications are open, what might be written or deleted in the meantime, etc.
Snapshots have a lot of capabilities that were mentioned during the WWDC session, but most of those aren't yet available to developers. However, the ability to actually create and copy from them is both available and documented! So...
We're really happy to announce that SuperDuper! uses snapshots on APFS boot volumes when copying. Copies should be much more reliable and less error prone, and (as usual) the whole process happens completely transparently to you, the user.
To get this to work, there's literally nothing you have to do: no settings, no muss, no fuss. If your startup volume is an APFS volume, the details are all handled for you.
Continuity Errors
Apart from special-volume-stashing, we weren't able to finish implementing one thing in time for High Sierra's release: backing up to unmounted images. This also has to do with the special volumes mentioned earlier: when a volume isn't available, we don't know its format, nor do we know whether it requires volumes to be created.
We didn't want to mount the volume to find out, and there were so many cascading cases involved we decided to not release that part of what we've done. Instead, once you've created an APFS image backup, open it in Finder by double-clicking, then copy directly to the mounted image's volume. That'll create the special volumes so it can be properly restored.
You may also notice that we're copying a bit more than you'd expect: we see it too. We decided it was better to copy a little more than necessary than a little less, since it's just a matter of time taken and not overall fidelity. This will be improved in the final release.
The main application is now 64-bit, but the copy engine is still 32-bit. We didn't want to potentially destabilize the copy process by including our alpha-level 64-bit copy engine in a public beta, so we're 95% 64-bit, and 5% 32-bit.
Finally, there's a black window flash during animations that's related to the 64-bit changes. It's ugly. We know. But it's not a blocker, and there were more important things to get working (snapshots!) so, think of it as a brief moment to consider your place in the universe and...enjoy?
Dinner is Served
Since it's a beta, dinner is dog food. We've been eating it for a while now and it's delicious, but given FDA standards for dog-vs-people food, there may be a higher quantity of insect parts in your portion.
OK, that's kind of gross.
What I mean to say is that we've tested this internally, and we're pretty pleased with how it's working.
That said, we're talking about a brand new OS with a brand new file system and some brand new capabilities. This version is a beta, and while you can continue to use it as you normally would, you may encounter issues.
These things happen.
So, in conjunction, also use Time Machine as an extra backstop (always a good idea). I'm here for you at the support email; let me know if you have any problems.
We'll be working to move this from beta to final release as we finish up some of the remaining tasks, and the inspectors determine it's fit for human consumption. We might even get a chance to sleep a little. Probably not, though, so if any support replies are incoherent, please be understanding.
In the meantime, thanks for your patience and support: it's truly appreciated.
Download away!
