Shirt Pocket Watch

One of the weird parts of doing a public beta, as we did for 3.0, is that the release of the final version doesn't quite feel the same. So, even though it's only been two weeks since 3.0 came out...it feels like it was released months ago.

The 3.0 release has actually gone incredibly smoothly. The extended beta managed to attract a large number of diverse system configurations, and that helped us ensure that any post-release issues would be...obscure. And, indeed, they have been!

Pulling the Right Levers

A program like SuperDuper! needs to make choices about how to interface with the system. There are many different layers, and we try to choose appropriately, depending on what we're doing.

To ensure a maximum level of compatibility, it's incredibly important to work at as high a level as possible for any given activity. For example, when SuperDuper! needs to erase a drive, it would be a huge mistake to actually jump down to the driver level and start mucking with drive structures.

Instead, we do the sensible thing: we ask Disk Utility to do it for us.

Talk to the Command

Disk Utility, though, is what's called a command-line tool: you can run it yourself, from the Terminal application, if you want to: type diskutil to see what it does, and man diskutil for its documentation. Much of macOS is built upon these Unix tools, and (where appropriate), so is some of SuperDuper.

When you talk to a tool like this, though, you have to both check its "return value" (basically, a rough "I worked!" or "I failed!"), and its text output (which says why it worked or failed, or gives you an answer to a question you asked).

Some of these tools (diskutil, hdiutil, etc) will format their output in plist format: basically, a structured XML document that's far more predictable, easier to parse, and more reliable. So rather than returning "Drive formatted as HFS+", it might return (intentionally simplified)

<format>hfs+</format>

which is obviously less friendly to people, but a lot better for programs trying to figure out what's going on.

This plist output's advantage is that its structure is strictly defined. It has a prologue, a series of tags and values, and an end. This is all guaranteed.

Samsung!

So, when a few users started getting some crazy, inexplicable errors (typically, an error indicating that a volume that was there couldn't be accessed), we were shocked to find, after doing some extensive investigation, that a Samsung SMART driver (based on an open source project) was actually polluting diskutil's output with a bunch of debugging/progress junk.

Needless to say, this prevented parsing of the output, and resulted in a stream of invective that did not diminish for many minutes. So frustrating. But fixed.

Various Improvements

That was the big one. But we also fixed a number of other things:

• Finally fixed the update animation bug! So, if you install 3.0.1, the next release is going to show up properly in all cases. This time for sure!

  This took such a long time to fix because it was entirely timing related and didn't happen in the debugger. But, we figured it out. So embarrassing. Gonna miss you, you little rascal. (Please don't darken our UI door ever again.)

• Settings conversion related quit issues
• More reliable sleep prevention during a copy
• Supports encrypted destination volumes on 10.12
• Restored 10.9 support

I also took some time over the past week to turn on SSL for Shirt Pocket, so site access and downloads are all being done securely.

Happy Thanksgiving

So, with that, from our families to yours, have a Happy Thanksgiving, and enjoy SuperDuper! 3.0.1, which is available via the built-in updater, from the web site, or here:

SuperDuper! 3.0.1 Download

The short version

Blah-blah-blah what-the-heck-are-you-talking-about-Dave so many words oh, 3.0 is released? Got it.

The wordy-but-interesting version

Sometimes, on macOS, you run into behavior that really makes you go "huh".

An example: as you likely know, you can create a disk image with Disk Utility and format it as APFS. Which works fine.

And, if you've been reading the blog, you probably also know that APFS volumes reside in a container, which itself resides on a GUID partitioned drive. So, multiple layers: drive, partition, container, volume. You can see all those in Disk Utility.

But once you create the image, what you see in Finder is simpler: a single ejectable volume, with the name you gave it. So, you can open it, copy to it, eject it, and there you go.

Great, right?

Not so much

Ah, but programmatically, things are different.

If an application opens an image the old way, and then unmounts its volume using diskutil, umount or hdiutil, it doesn't do what it used to do.

You can easily see this in Disk Utility, too. After you create the image, try selecting the volume and clicking the eject button. When you do that, you'll see that while the volume ejected, the image is still sticking around in the sidebar. And if the image is hosted on an ejectable drive or network volume, and you then try to eject the host volume, you'll find the it can't be ejected.

Worse, if you force eject the host at this point (hey, folks—don't force eject!), you can damage the image.

Not good.

Huh.

Worlds within worlds

So, what's going on is that there are still two things "attached" from the image: the container and the partition. They don't get "released" after all the volumes are ejected for some inexplicable reason. And so, any application that mounted and ejected images can no longer rely on the previous behavior.

Mischief, managed

Fortunately, this is another situation that SuperDuper! handles for you. If you back up to an image file, you don't need to know what I've just explained: we deal with the details, and eject all the "parts" of an APFS image without any manual intervention.

As you've heard many times from me: it just works.

So while you may not want to know why, or how all this is accomplished...isn't it comforting to know that someone's figured it out and has your back?

#Finally

With that last bit of explanation, I'm happy to say that we've reached the end of this particular voyage. SuperDuper! 3.0 (release 100!) is done, and you'll find the download in the normal places, as well as in the built-in updater, for both Beta and Regular users.

SuperDuper! 3.0 has, literally, many hundreds of changes under the hood to support APFS, High Sierra and all versions of macOS from 10.9 to the present.

SuperDuper! 3.0 is the first bootable backup application to support snapshot copying on APFS, which provides an incredible extra level of safety, security and accuracy when backing up. It's super cool, entirely supported (after all, it's what Time Machine uses...and it was first overall), and totally transparent to the user.

It doesn't stop there, of course. We're already in progress on the next version, and we've got some really great things planned.

Thanks to all of you who participated in the beta, because the final release couldn't have happened without you.

And to those of you who didn't participate in the Beta: you are going to love the final result.

Without further ado, download SuperDuper! 3.0

Enjoy!

Gee, do you think the original plan was to release this update on Halloween? Spoooooky!

Poof!

When “Erase, then copy” is selected, SuperDuper! has always endeavored to match the format of the source. That is, if the source is Mac OS Extended [HFS+] (Journaled), and the destination is Mac OS Extended (Case Sensitive), on an erase, the destination would end up matching the source, adding Journaling and removing Case Sensitivity.

As I mentioned in one of my first High Sierra posts, we always knew this wouldn’t work when going HFS+ to APFS: you just can’t format an APFS container as HFS+ safely, since it can affect more than just the one volume.

We’ve discovered during our broader Beta rollout that, due to weird bugs in Disk Utility, formatting an HFS+ drive as APFS is unreliable too. Sometimes the drive just “vanishes” and doesn’t re-mount. Sometimes it fails for no reason. Sometimes it makes the one volume unreadable until it’s erased again.

It’s kind of scary crazy. But that seems to be par for the early version High Sierra course. It’s relatively solid in the main, but really weak at the margins.

...But I Don’t Do That

So, we’ve made a change in our “magic” behavior to deal with these issues. Basically, we continue to do what we’ve always done. But if the “major” part of the file system doesn’t match (that is, HFS+ and APFS), we just maintain the existing format of the drive.

So, if you copy APFS to an HFS+ volume with erase, we’ll erase the destination but leave it as HFS+.

Hopefully we’ll be able to change this as macOS stabilizes further.

Enter the Void

I regret to say that we’ve fixed the “black flash” bug, so you’ll no longer have a moment to stare into the blackness as it stares back at you.

Perhaps that’s for the best.

Spaced

We didn't notice (but should have) that our various numbers were now proportionally spaced rather than monospaced since we built against a newer SDK with different targeting when we made the UI 64-bit.

Of course, someone—and you know who you are—noticed right away. Embarrassing, and fixed.

It’s Log, It’s Log!

One of the things we’ve done is improve our diagnostic logging, so when people send in a bug report, we get a better feeling for the state of the drives attached to the system, and the problems that might be going on with the system, should there be any.

We changed this a bit to only collect the extended log information when copies fail.

In earlier OS versions we would include a bit of the system log, so we could see what hardware was doing at the time of the copy. That stopped working in recent macOS release, so we've changed it to use log show.

The problem is, log show -last 1m, including a kernel predicate so it only returns low-level kernel logging, can be slow. Not only that, but due to bugs in the logging subsystem, it can incorrectly return much more than one minute of logging. We've seen it return almost a gigabyte of log data!

Even though we also limit the data to a short snippet at the end of all that text, we have to wait for it to be collected, and when the system grabs too much, it can take a while to retrieve and then discard the extra stuff.

The end result of all this is that we only do it when copies fail, and we put up a modal sheet while it's going on.

And Also, Too

Previously, when users sent in a report, we only included the log information when they reported an "Error during copy". We were a bit too clever for our own good there, since people didn’t always choose the right issue type, and then we’d have to ask for the report to be sent in again. We now always attach the log when using “Send to Shirt Pocket”.

These are small things, I know, but they'll help me help you more quickly and efficiently, should you ever need support.

I Can't Quit You, Baby

There are still some issues with quit that we're aware of...we're working on them.

Various and Sundry

There has also been a bunch of polishing and fixing things at the margins, thanks to user reports from highly unusual configurations, but as before, things are working really well on the vast majority of systems, which we’re really happy about. It’s just kind of boring to write about how great things are working for most people!

So you get the point. And now you get the results. Download away!

Download SuperDuper! 3.0 B7