Told you it wouldn't be long until I posted again.
Spoiler Alert
We've finished up a bunch of internal testing over the past few weeks, and there's a beta of SuperDuper! for High Sierra and APFS linked at the bottom of this post. But it's so exciting, in a totally nerdy way, that it would be a mistake to not follow the whole story, with its twists and turns. So let's dive in.
Our Antagonist
As I mentioned in the last post, APFS is basically undocumented. That required us to make certain assumptions about its capabilities, and to draw inferences from what's on-disk in order to produce something that replicates the source.
APFS volumes are hosted in a container, which is basically a GUID partition on a drive. That container has a pool of storage which is shared between all the APFS volumes in the container. So, in a 1TB container, you could have a large number of volumes, each of which can draw from the 1TB of storage available in its container.
Only APFS volumes can be in the container: HFS+ volumes are not allowed.
In addition to the startup volume, and any volumes that you create yourself, a container that hosts bootable volumes also contains three hidden volumes with special "roles". One, called Preboot, contains all the information required to start the boot process. Another called Recovery, is the recovery volume. The third, VM, is where swap files are stored.
These volumes are minimally documented in the asr and diskutil man pages (the asr page actually discusses what makes up a valid system), so we're quite confident about what we've done in terms of the volumes and their roles. Unfortunately, their contents are not, and so here is where we've made assumptions. Also, since we can't create our own volume contents, we copy that data from the source's equivalent volumes.
Papers, Please
Both HFS+ and APFS volumes have their own unique ID (UUID). Each special volume mentioned above associates various information with the unique ID of the corresponding bootable, role-less APFS volume. Unfortunately, those unique IDs cannot be preserved across an erase: they're read-only. As such, there's significant housekeeping involved in maintaining these special volumes, and in positively identifying the volumes you've selected as the source and destination.
On top of that, the reboot and recovery data can't just be copied whole from the source, because multiple bootable APFS drives in a single container share the same special volumes... so, as you might expect, there's also new logic involved.
This all works its magic behind the scenes, and you shouldn't notice any changes...although you might notice some subtle improvements.
There and Back Again
As I mentioned in the previous post, we allow APFS to HFS+ copies (and back). However, copying from HFS+ requires the special volume contents to be stashed on the backup copy, since HFS+ volumes don't have all these special role volumes (and their partition schemes don't allow the same kind of 'guaranteed' partition creation). That's something we don't do in this first beta.
In addition, given the full extent of APFS's capabilities are not known, we're not fully confident in allowing this type of copying in our beta, and discourage the use of HFS+ to APFS copying. If you need to restore HFS+ to a bootable APFS volume, reinstall the OS to a clean volume and point to the backup when prompted during first boot.
And When You Smile for the Camera... (RIP Walter)
Ah, but these stories always have a twist! We think this is a good, Unbreakable twist, too.
One of the challenges of copying a live volume is that it's active and always changing. Even as we're stepping through the data, the user could delete files out from under us, change the data...almost anything can happen. In general, it doesn't. But it could. That's why the User's Guide suggests quitting applications before copying.
One awesome thing that APFS has is the ability to create "snapshots" that freeze the state of an APFS volume. Copies from the snapshot will exactly match the state of the source at the moment the snapshot is taken, so you don't have to worry about what applications are open, what might be written or deleted in the meantime, etc.
Snapshots have a lot of capabilities that were mentioned during the WWDC session, but most of those aren't yet available to developers. However, the ability to actually create and copy from them is both available and documented! So...
We're really happy to announce that SuperDuper! uses snapshots on APFS boot volumes when copying. Copies should be much more reliable and less error prone, and (as usual) the whole process happens completely transparently to you, the user.
To get this to work, there's literally nothing you have to do: no settings, no muss, no fuss. If your startup volume is an APFS volume, the details are all handled for you.
Continuity Errors
Apart from special-volume-stashing, we weren't able to finish implementing one thing in time for High Sierra's release: backing up to unmounted images. This also has to do with the special volumes mentioned earlier: when a volume isn't available, we don't know its format, nor do we know whether it requires volumes to be created.
We didn't want to mount the volume to find out, and there were so many cascading cases involved we decided to not release that part of what we've done. Instead, once you've created an APFS image backup, open it in Finder by double-clicking, then copy directly to the mounted image's volume. That'll create the special volumes so it can be properly restored.
You may also notice that we're copying a bit more than you'd expect: we see it too. We decided it was better to copy a little more than necessary than a little less, since it's just a matter of time taken and not overall fidelity. This will be improved in the final release.
The main application is now 64-bit, but the copy engine is still 32-bit. We didn't want to potentially destabilize the copy process by including our alpha-level 64-bit copy engine in a public beta, so we're 95% 64-bit, and 5% 32-bit.
Finally, there's a black window flash during animations that's related to the 64-bit changes. It's ugly. We know. But it's not a blocker, and there were more important things to get working (snapshots!) so, think of it as a brief moment to consider your place in the universe and...enjoy?
Dinner is Served
Since it's a beta, dinner is dog food. We've been eating it for a while now and it's delicious, but given FDA standards for dog-vs-people food, there may be a higher quantity of insect parts in your portion.
OK, that's kind of gross.
What I mean to say is that we've tested this internally, and we're pretty pleased with how it's working.
That said, we're talking about a brand new OS with a brand new file system and some brand new capabilities. This version is a beta, and while you can continue to use it as you normally would, you may encounter issues.
These things happen.
So, in conjunction, also use Time Machine as an extra backstop (always a good idea). I'm here for you at the support email; let me know if you have any problems.
We'll be working to move this from beta to final release as we finish up some of the remaining tasks, and the inspectors determine it's fit for human consumption. We might even get a chance to sleep a little. Probably not, though, so if any support replies are incoherent, please be understanding.
In the meantime, thanks for your patience and support: it's truly appreciated.
It's inevitable—another year, another macOS version (I am still not used to typing "macOS"), and another build up to a significant SuperDuper! release. Sorry I haven't been blogging the whole way through, but it's been busy, and I just haven't had the time.
As of this morning, we have a release date - September 25th. So, for those asking (and there are a lot of you), let's talk about High Sierra. Lucky 13! Woo!
APFS
First, we'll definitely be supporting APFS. That work has been in progress for some time, and continues as of this post. We already have copying to and from APFS volumes working "in the lab", as it were, and testing is ongoing.
The bad news is I'm not confident enough to say we're going to release our APFS support day-and-date.
I know this kind of hedging is disappointing. But it's important to note that Apple still hasn't released any documentation on the "proper" way to create a bootable APFS volume. An example of what they have in mind was released for the very first time when the High Sierra developer release came out a few months ago, but that's it. We basically have to make an educated guess about what they want.
We've designed and implemented that, and it's significantly different than HFS+'s boot setup, with various special partitions dedicated to specific purposes (even a separate VM volume!), and entire new volume management system, etc.
So, we have both copying and boot working, and we're testing it carefully to try to cover all of the various permutations that can occur. Because no guidance or documentation is available, and the process is all reverse-engineered from existing volumes, it's difficult to know what might happen in the future...and protecting against mistakes, and informing users what their potential actions mean, is really critical.
For example, what happens if you do an "Erase, then copy" from an HFS+ volume to an APFS volume? In our current version, we match the format of the source when we erase. But, HFS+ can't be in an APFS container. So, we'd have to convert the container to a regular GUID partition. And since there might be other APFS volumes in that container, you'd end up destroying them.
Not good. And that's just one case.
So, we're being cautious, knowing that users who have continually followed our advice (namely more backups are always better than fewer, and use more than one backup program, such as SuperDuper! and Time Machine rather than either on its own) will still be covered by Time Machine if they jump the High Sierra update as soon as it becomes available.
But what about HFS+?
Yes, making a backup from APFS to HFS+, and booting off that, can work as it always has (once our APFS support is released). And HFS+ to HFS+ works as expected as well, save for a few small issues in 2.9.1.
In particular, Apple has further tightened its System Integrity Protection process, and is completely denying access to some files on the startup volume, even when copying to a non-startup volume. We consider those errors ("Operation not permitted") to be fatal in 2.9.1, and so we stop. We have a simple, preference-based workaround for this that we've provided to people during the beta on request, and 2.9.2 has it built-in. (This is the kind of thing I'm talking about when I said "it's difficult to know what might happen"--we didn't know what could happen with SIP, and so we planned for possible changes.)
But since APFS is, again, basically undocumented... what could that backup be missing? How will changes like filename normalization affect future copies if a dot-update modifies behavior? How do you make a snapshot, since that is obviously intended to be a better source for a backup? And do androids actually dream of electric sheep?
These are the kinds of things that have been keeping us awake at night at Shirt Pocket HQ. I feel like I've aged as much as a two-term President.
Anyway, we're grinding through this all as quickly as we can. We don't want to finish until we're sure we have the shipping bits, because, unlike with earlier OS versions, we're dealing with an entire new file system. We'd rather be late than wrong.
We're probably not wrong, but we want to be sure.
Automatic conversion
Speaking of being sure, one of the scariest part of High Sierra is the fact that, if you have an SSD, you'll have no choice: you're going to be converted to APFS, whether you like it or not. Apple is obviously confident about this process, or they wouldn't do it, but even if the conversion is successful, it's important to note that while that means it worked from their point of view -- conversion accomplished! -- this changeover won't be without pain.
Your existing, valuable low-level tools, such as Disk Warrior, will instantly be rendered useless. In a pinch where Disk Utility can't fix it? Tough luck.
Want to host an HFS+ volume on your drive? You can't in an APFS container.
APFS doesn't seem to be faster than HFS+ (which is not to say it won't ever be, or that it won't be more stable...a low bar, I know). That cool demo where copying a whole directory is instantaneous? That's because the data isn't really being copied. It's being referenced, which is, of course, fast. Copying happens when you modify, so it's basically being done "lazily".
But think about it. That also means that if you made a copy of your data "just in case", and you developed a bad spot on the drive, all copies of the file are now damaged, because they share the same data. Disk damage can now cascade across far more files than before, because you can't tell what's shared. Even snapshots can't save you if they're referencing the same bucket of damaged bits.
So many other things.
There are going to be growing pains here, and a lot of it will be for long time developers who have been doing invaluable work for years, and unlike us, they have to start again from scratch. Need I say they deserve your support and encouragement?
64-bit
I'm also getting a lot of mail about our 64-bit plans, since there's been a pile of confusion coming out of the WWDC announcements, so let me try to set things straight.
High Sierra works fine with 32-bit apps. Come January, the App Store will stop accepting 32-bit store apps. And, at some point in the future, Apple will stop supporting 32-bit apps.
Now, for most apps, there's really no advantage to being 64-bit (save for some really minor system-wide memory benefits, which—frankly—were the other way around when 64-bit APIs were introduced, and no one complained about that). So for many developers this will be a lot of unnecessary work, with no end-user benefits.
All that said, while there's no rush to do so (they're not phasing out 32-bit for a while), our full High Sierra release will be 64-bit, so you can rest easy, knowing that we'll be ready for the actual phase-out well before it happens.
Should I Update to High Sierra?
I think I usually tell people to "take it slow" with an entirely new OS version. While there have been "huge win" updates in the past (10.6 is a great example), High Sierra isn't one of those.
Rather, while it may be a good (or even great) update over time, it also has the potential to destabilize things far more than anything that's come before. It's going to be hard to judge its impact until its wide release, and even then, it'll be a while.
If you're a normal user, I would strongly encourage you to not update to High Sierra right away. Let others take the risk. Wait until things calm down and the initial problems, which are inevitable, are fixed. Continue doing what you've been doing before High Sierra: you're not missing anything of significance.
Apple's going to be OK if their "adoption graph" doesn't go straight up in the air.
After all, why should you be a statistic when you have work to do?
But I Want to Update Right Away
Don't.
I Insist on Updating Immediately!
Sigh. OK. But don't say you haven't been warned: we don't all float down here.
Before you update, use SuperDuper to make a full backup or two. Check your backup by starting up from it. Then, unplug the backup drive from your Mac and put it somewhere safe.
As I've recommended elsewhere, you should be making multiple backups with other programs as well, including Time machine. So, after you update, continue to use Time Machine to back up. Given it's basically written by the same team as APFS, and Apple knows what is and isn't possible, even when they don't document it externally, this is your best bet to start with.
If you need to roll back to your previous OS, you cannot just use Smart Update. In fact, you also can't use Erase, then copy, because your internal drive has been converted to a storage scheme that won't work with anything earlier than High Sierra.
Surprise!
So, after starting up from your SuperDuper backup, use Disk Utility to erase the drive you want to restore to, by selecting the drive hardware above the volume, not the volume itself. That will roll you back to something that can properly host HFS+ and boot pre-High Sierra releases. You can then restore your SuperDuper! backup normally, using "Restore - all files" and "Erase, then copy" or "Smart Update".
But How Do I Back Up?
For the moment, as I said above, use Time Machine. Apple's teams know the details of APFS, and as such Time Machine isn't in the "limbo" we're in with regard to ensuring things are as they should be. Remember, disk space is cheap, and I know I say this too much, but it's true: no one was ever sad they had too many backups.
If you are running High Sierra with HFS+ (namely, if you have a spinning disk or Fusion drive), our just-released SuperDuper update will work fine. Use that in addition to Time Machine.
Our full, APFS-supporting version (with some other surprises) will be out as soon as we can get it finished. And we'll have a public beta when we're confident enough to put it in your hands.
This is Going to Cost Me a Ton, Isn't It.
Actually, no. Since 2004, we've never charged for SuperDuper updates, and our High Sierra update will be free of charge as well.
Clearly, we were dropped on the head as babies.
FIN
So there you go. I'll be back relatively soon with more.
