Turns out, a few of you have drives that don't have low-level media IDs...something we thought all drives had. And they did, at least, all the drives we and the external testers tried did.
Well, now we know...and, now we don't crash. Sorry about that.
Fix now available in v3.1.4 - download away!
Way back when, not so long ago, we transitioned the main SuperDuper! application to 64-bit. As I said then, we were 95% 64-bit, and 5% 32-bit.
Today, we're releasing SuperDuper! 3.13, which—after extensive testing—moves the copy engine to 64-bit as well. macOS 10.13.4 is going to start giving obnoxious warnings about 32-bit app usage, so we figured this was a good time to release the update, even without any significant user-facing feature changes.
I'd love to use this blog post to tell you that, because we're now 64-bit, we're 200% faster, but, well, no. Internal testing shows no significant gains or losses from being 64-bit. In fact, we have to do additional work packing and unpacking some file system structures, which are laid out (for legacy reasons) for 32-bit apps.
Of course, we fixed a bunch of other things, too. One specific change was to improve handling of source drives with bad recovery volumes.
We're not sure how people's drives get like this, but sometimes there will be a number of "Untitled" 650MB volumes on their drives. Disk Utility will indicate one of them is the Recovery volume, but when we try to mount and copy from it, we get an error, because the drive is bogus and can't be mounted.
Previously, we would stop and give an error. Now, we log the problem and generate a warning that explains how to resolve the problem. (Basically, you need to reinstall the OS, which refreshes the OS under your existing applications and data, and recreates/fixes Recovery.)
While we were at it, we fixed a few things in the last 10.9 compatible version (3.1.1), back ported some other fixes we made in 3.1.2, and updated it. Unfortunately, since it already had a version number, I just updated the binary. Bad form, I know, since there are now two 3.1.1s out there. But if you're using Mavericks, and can't create schedules with your existing 3.1.1, download 3.1.1 again and reinstall.
So yeah, we're older and we're losing our hair, but you really do still need us. Update inside the app, or download away!
Shipping software sucks.
It's not that we don't want to put the software in the hands of our users. And it's not that we're not proud of the work. We do, and we are.
What sucks is that you always ship with bugs. Always. Some of them you know about. Some of them you don't.
Anyone who does this for a living knows how this works. You keep track of everything you find in the software. Some things are reproducible. Some things aren't. You draw up general approaches for investigation for the latter, and propose fixes for the former.
You then prioritize things, hash out a schedule, and decide what's worth fixing and what you can, in the end, live with...for now.
During the run-up to SuperDuper v3.1.2 (available today, see below), we were dealing with this whole investigate-prioritize-allocate-test cycle, when someone brought a sheet to our attention that's been a known low-priority thing for years.
12 years, to be precise.
It's the basic "Stop the copy in progress" sheet that comes up when you want to stop a copy. It's something you can do by accident, and throw away a lot of progress, so we show a sheet that asks you whether you really want to stop.
Sensible, as far as it goes. But during the development of 2.0, there were a lot of potential cases that would have to be dealt with during a cancel, and while we were working through them, and how to present them to the user, a temporary cancel sheet was put in place.
A sheet that was immediately logged as "this has to be fixed" in our tracking database, because it was poorly worded, confusing and (please don't hurt us) had "Yes/No" buttons at the bottom. Enjoy this example of our overwhelming UI genius:
So many problems with that sheet. It's wrong, it uses crazy terminology, it doesn't match the rest of the app, it uses Yes/No buttons. I don't even...
There was a lot to do for v2.0, and development proceeded, other tasks took precedence, this sheet isn't used very much, and eventually we shipped 2.0 with the sheet in place, with the bug tagged as low-priority, known-terrible, embarrassingly bad, "we need to fix this post 2.0".
That cycle repeated over and over. It was an easy fix, but other things took precedence. This stuff happens. Shipping software sucks.
Well, it was terrible, and it's now 12 years post 2.0. So for the new year? Start celebrating, because it was fixed.
This version also fixes a bunch of other relatively minor things that should please one or more of you:
General cleanup, nothing too exciting, but a fix is a fix, and now they're available to you. Download away!
We've got something under the tree for all of you, and you get to open your present early.
SuperDuper! 3.1 is out, and now allows Time Machine-like restoration from bootable APFS copies.
Oh, decided to join us for the details? Welcome to an ocean of words: prepare to get your feet wet!
Well, I told you there would be more surprises in store, and, so, here we are. Since the release of 3.0, we've not only been copying from a snapshot when using an APFS source. In addition, as some of you have noticed, we've also been creating snapshots on the backup.
This is why.
In SuperDuper! v3.1, you may notice a small change to the Copy Now button: a little up-triangle glyph. If you've selected an APFS volume as your source, you can click that triangle, and up pops a list of the snapshots available to copy from.
This means you don't just have to copy from the drive as it is "now" (the default choice). You can select from any existing snapshot, and we'll copy the state of the files as they were at that time.
But why would you want to do that?
Well, in High Sierra, the system takes a snapshot of your drive before it installs a software update. Having a problem with an update and want to check if it was different before, but forgot to back up first? No problem: use SuperDuper! to copy from the snapshot taken just before the update was installed, start up from the copy, and check the behavior.
Even better, if you're backing up to an APFS volume as we encourage, this works on a bootable copy, too! Since v3.0, we've been taking a snapshot of the backup volume before every copy we make. That means there's not just one available backup on the drive—if you've been using Smart Update, there are many! Start up from your backup drive, click the triangle, and you'll be presented with a list of available snapshots. Pick one, "Copy Now", and you've restored a day ago's backup, or a week ago's.
In fact, not only can you use SuperDuper to copy from these snapshots, you can even open Time Machine, select your backup volume, and see older versions of files, deleted files - they're all being saved, automatically, every time you Smart Update. Even though you're not backing up your backup to Time Machine. Because that would be silly.
This new feature is not intended to replace or replicate what Time Machine does. Snapshots are managed by the system, and at present they have some lightly-to-not documented constraints. You need to have about 20% free in a container to create a snapshot, and the system consolidates and removes snapshots according to its own logic.
As I've said before, an effective backup strategy should have multiple parts: SuperDuper!, Time Machine and an online service. The more (and more varied) backups you have, the safer you are. This new feature gives SuperDuper!'s bootable backups some cool new capabilities and an additional measure of protection should you make a mistake.
Pretty awesome, right?
Needless to say, this is the kind of feature we love to introduce. It's completely transparent to the user, and is built on the extensive work we did to support APFS in v3.0. That support wasn't done at a surface level: we worked hard to research and make use of the more advanced features, like snapshots, and planned for the future in ways that truly make the backup experience better for everyone.
Including you. Update away!
One of the weird parts of doing a public beta, as we did for 3.0, is that the release of the final version doesn't quite feel the same. So, even though it's only been two weeks since 3.0 came out...it feels like it was released months ago.
The 3.0 release has actually gone incredibly smoothly. The extended beta managed to attract a large number of diverse system configurations, and that helped us ensure that any post-release issues would be...obscure. And, indeed, they have been!
A program like SuperDuper! needs to make choices about how to interface with the system. There are many different layers, and we try to choose appropriately, depending on what we're doing.
To ensure a maximum level of compatibility, it's incredibly important to work at as high a level as possible for any given activity. For example, when SuperDuper! needs to erase a drive, it would be a huge mistake to actually jump down to the driver level and start mucking with drive structures.
Instead, we do the sensible thing: we ask Disk Utility to do it for us.
Disk Utility, though, is what's called a command-line tool: you can run it yourself, from the Terminal application, if you want to: type diskutil to see what it does, and man diskutil for its documentation. Much of macOS is built upon these Unix tools, and (where appropriate), so is some of SuperDuper.
When you talk to a tool like this, though, you have to both check its "return value" (basically, a rough "I worked!" or "I failed!"), and its text output (which says why it worked or failed, or gives you an answer to a question you asked).
Some of these tools (diskutil, hdiutil, etc) will format their output in plist format: basically, a structured XML document that's far more predictable, easier to parse, and more reliable. So rather than returning "Drive formatted as HFS+", it might return (intentionally simplified)
<format>hfs+</format>
which is obviously less friendly to people, but a lot better for programs trying to figure out what's going on.
This plist output's advantage is that its structure is strictly defined. It has a prologue, a series of tags and values, and an end. This is all guaranteed.
So, when a few users started getting some crazy, inexplicable errors (typically, an error indicating that a volume that was there couldn't be accessed), we were shocked to find, after doing some extensive investigation, that a Samsung SMART driver (based on an open source project) was actually polluting diskutil's output with a bunch of debugging/progress junk.
Needless to say, this prevented parsing of the output, and resulted in a stream of invective that did not diminish for many minutes. So frustrating. But fixed.
That was the big one. But we also fixed a number of other things:
Finally fixed the update animation bug! So, if you install 3.0.1, the next release is going to show up properly in all cases. This time for sure!
This took such a long time to fix because it was entirely timing related and didn't happen in the debugger. But, we figured it out. So embarrassing. Gonna miss you, you little rascal. (Please don't darken our UI door ever again.)
I also took some time over the past week to turn on SSL for Shirt Pocket, so site access and downloads are all being done securely.
So, with that, from our families to yours, have a Happy Thanksgiving, and enjoy SuperDuper! 3.0.1, which is available via the built-in updater, from the web site, or here:
Blah-blah-blah what-the-heck-are-you-talking-about-Dave so many words oh, 3.0 is released? Got it.
Sometimes, on macOS, you run into behavior that really makes you go "huh".
An example: as you likely know, you can create a disk image with Disk Utility and format it as APFS. Which works fine.
And, if you've been reading the blog, you probably also know that APFS volumes reside in a container, which itself resides on a GUID partitioned drive. So, multiple layers: drive, partition, container, volume. You can see all those in Disk Utility.
But once you create the image, what you see in Finder is simpler: a single ejectable volume, with the name you gave it. So, you can open it, copy to it, eject it, and there you go.
Great, right?
Ah, but programmatically, things are different.
If an application opens an image the old way, and then unmounts its volume using diskutil, umount or hdiutil, it doesn't do what it used to do.
You can easily see this in Disk Utility, too. After you create the image, try selecting the volume and clicking the eject button. When you do that, you'll see that while the volume ejected, the image is still sticking around in the sidebar. And if the image is hosted on an ejectable drive or network volume, and you then try to eject the host volume, you'll find the it can't be ejected.
Worse, if you force eject the host at this point (hey, folks—don't force eject!), you can damage the image.
Not good.
Huh.
So, what's going on is that there are still two things "attached" from the image: the container and the partition. They don't get "released" after all the volumes are ejected for some inexplicable reason. And so, any application that mounted and ejected images can no longer rely on the previous behavior.
Fortunately, this is another situation that SuperDuper! handles for you. If you back up to an image file, you don't need to know what I've just explained: we deal with the details, and eject all the "parts" of an APFS image without any manual intervention.
As you've heard many times from me: it just works.
So while you may not want to know why, or how all this is accomplished...isn't it comforting to know that someone's figured it out and has your back?
With that last bit of explanation, I'm happy to say that we've reached the end of this particular voyage. SuperDuper! 3.0 (release 100!) is done, and you'll find the download in the normal places, as well as in the built-in updater, for both Beta and Regular users.
SuperDuper! 3.0 has, literally, many hundreds of changes under the hood to support APFS, High Sierra and all versions of macOS from 10.9 to the present.
SuperDuper! 3.0 is the first bootable backup application to support snapshot copying on APFS, which provides an incredible extra level of safety, security and accuracy when backing up. It's super cool, entirely supported (after all, it's what Time Machine uses...and it was first overall), and totally transparent to the user.
It doesn't stop there, of course. We're already in progress on the next version, and we've got some really great things planned.
Thanks to all of you who participated in the beta, because the final release couldn't have happened without you.
And to those of you who didn't participate in the Beta: you are going to love the final result.
Without further ado, download SuperDuper! 3.0
Enjoy!
Gee, do you think the original plan was to release this update on Halloween? Spoooooky!
When “Erase, then copy” is selected, SuperDuper! has always endeavored to match the format of the source. That is, if the source is Mac OS Extended [HFS+] (Journaled), and the destination is Mac OS Extended (Case Sensitive), on an erase, the destination would end up matching the source, adding Journaling and removing Case Sensitivity.
As I mentioned in one of my first High Sierra posts, we always knew this wouldn’t work when going HFS+ to APFS: you just can’t format an APFS container as HFS+ safely, since it can affect more than just the one volume.
We’ve discovered during our broader Beta rollout that, due to weird bugs in Disk Utility, formatting an HFS+ drive as APFS is unreliable too. Sometimes the drive just “vanishes” and doesn’t re-mount. Sometimes it fails for no reason. Sometimes it makes the one volume unreadable until it’s erased again.
It’s kind of scary crazy. But that seems to be par for the early version High Sierra course. It’s relatively solid in the main, but really weak at the margins.
So, we’ve made a change in our “magic” behavior to deal with these issues. Basically, we continue to do what we’ve always done. But if the “major” part of the file system doesn’t match (that is, HFS+ and APFS), we just maintain the existing format of the drive.
So, if you copy APFS to an HFS+ volume with erase, we’ll erase the destination but leave it as HFS+.
Hopefully we’ll be able to change this as macOS stabilizes further.
I regret to say that we’ve fixed the “black flash” bug, so you’ll no longer have a moment to stare into the blackness as it stares back at you.
Perhaps that’s for the best.
We didn't notice (but should have) that our various numbers were now proportionally spaced rather than monospaced since we built against a newer SDK with different targeting when we made the UI 64-bit.
Of course, someone—and you know who you are—noticed right away. Embarrassing, and fixed.
One of the things we’ve done is improve our diagnostic logging, so when people send in a bug report, we get a better feeling for the state of the drives attached to the system, and the problems that might be going on with the system, should there be any.
We changed this a bit to only collect the extended log information when copies fail.
In earlier OS versions we would include a bit of the system log, so we could see what hardware was doing at the time of the copy. That stopped working in recent macOS release, so we've changed it to use log show.
The problem is, log show -last 1m, including a kernel predicate so it only returns low-level kernel logging, can be slow. Not only that, but due to bugs in the logging subsystem, it can incorrectly return much more than one minute of logging. We've seen it return almost a gigabyte of log data!
Even though we also limit the data to a short snippet at the end of all that text, we have to wait for it to be collected, and when the system grabs too much, it can take a while to retrieve and then discard the extra stuff.
The end result of all this is that we only do it when copies fail, and we put up a modal sheet while it's going on.
Previously, when users sent in a report, we only included the log information when they reported an "Error during copy". We were a bit too clever for our own good there, since people didn’t always choose the right issue type, and then we’d have to ask for the report to be sent in again. We now always attach the log when using “Send to Shirt Pocket”.
These are small things, I know, but they'll help me help you more quickly and efficiently, should you ever need support.
There are still some issues with quit that we're aware of...we're working on them.
There has also been a bunch of polishing and fixing things at the margins, thanks to user reports from highly unusual configurations, but as before, things are working really well on the vast majority of systems, which we’re really happy about. It’s just kind of boring to write about how great things are working for most people!
So you get the point. And now you get the results. Download away!
Any program of sufficient complexity ends up having to work around behaviors that are undesirable. That might be a usability issue caused by default framework behavior, due to the interaction model selected by the designer. Or it might be an OS issue that only shows up in some situations (the recent iOS calculator bug is a good example of that).
Most, though, don't have to deal with the huge variation of hardware, software, OS, etc that a program like SuperDuper! has to handle.
The vast majority of the beta cycle has been focused on getting coverage of as many systems as possible. With a broad range of different setups and configurations, we can then find issues in the way macOS 10.13 is dealing with those configurations, and try come up with workarounds so, when when we release the GA version of SuperDuper! 3.0, the broader user community will have a smooth experience.
A lot of these changes have been focused on attached volumes, incorrect state information being returned by macOS calls or tools in some configurations and states, and coming up with ways to prod and probe until macOS returns the correct data.
It can be challenging to do, since you don't want to break "working" setups, and any workarounds should only come into effect in situations where they're needed. That way, when the OS problems are fixed, everything continues to operate.
All of that is a longwinded way of saying, we're happy to release Beta 6 of SuperDuper! 3.0, which has even more workarounds for macOS issues, covering a broad variety of situations we've seen in the past week or so. (It's workarounds all the way down!)
The new beta should work better on every system, and it fixes a problem with the beta autoupdate mechanism as well...so the next release's autoupdate should work better.
Download away, and thanks!
(I really wanted to embed the teaser trailer for Peter Greenaway's A Zed and Two Noughts above, but alas it's a tad NSFW. Really great movie, though.)
Content warning: some strong language is used in this post. Not without reason.
So, here's something that you might not expect me to say:
Because of the way APFS "file clones" work, no program operating at the file level, including SuperDuper!, can make an exact physical copy of every possible APFS volume.
That's right. There are cases where we can't make an exact copy of your APFS volume. And Time Machine can't either. Nothing can.
That doesn't mean the copy is bad! It just means it might not be as space efficient as the original.
Remember back in this post where I talked about the demo where Craig showed how fast it was to copy a gigantic amount of data?
I explained back then it was because the files aren't being copied. Rather, APFS creates new directory entries for the files, but references the same data blocks. So nothing is copied, which is fast. This is documented in Apple's APFS Guide.
From the user's perspective, these are different files. They're not like hard linked files, where changing one copy changes the others (not that most users know what hard links are). As far as users are concerned, they're totally separate, even if, at the file system level, they share the same data.
In APFS, if one of the cloned files is changed, even by a single byte, that changed data 'splits off' from the rest, and the files are now physically, and not just logically, separate—some of the data blocks now have two copies: the original ones, and the modified ones.
This process continues as the files diverge further.
The amount of logical drive space taken by the copies is twice the original, of course. However, the amount of actual space taken is, effectively zero...until the files are changed. At which point the space taken is the original plus the number of modified blocks.
This is all handled for you by APFS. You don't really have to think about it.
Until you do have to think about it.
Consider this case: you have a 1TB APFS drive, and three 333GB files, named A, B, and C. So the drive is nearly full.
You then create a folder, and copy the three files into that folder with Finder. Of course, you'd expect the copy to fail and the drive to fill...but it doesn't.
In fact, if you look at the volume's size with Get Info, you may be surprised to see it has the same amount of data on it as was there before you made the copies. But, if you look at Finder's size for the folder, you'll see you now have 2TB of data on a 1TB drive. It's like magic!
At least until you change one of the files.
But now, select those files and folders with Finder and try to copy them to another 1TB drive. What happens?
The drive fills.
You can't copy it to the same size drive! But why?
The reason is there's no (public) way to find out that two files are actually sharing the same data (they might even only be sharing some of the same data, as I explain above). So, when copied, the "clone" relationship is broken, as is the ten-pounds-of-shit-in-a-five-pound-bag magic. You now have a full ten pounds. It doesn't fit...so you end up covered in shit.
Yes, we know:
What if you kept track of checksums of every file on the drive, and then made "clones" for each file based on whether the files had the same data?
Leaving aside how ungodly slow that would be (think about trying to match ten million files to each other via checksums every time you copied), remember that cloning operates at a block level, where some blocks may be shared and some may not be. At a file level, it just won't work.
How about using hard links?
That won't work either: clones and hard links are not semantically equivalent at all, since changing one of the hard linked files would change all of them, by definition.
Just ask the file system!
While there are APIs to create clones, there's nothing there to find out whether two files are clones... and also, the shared data is at the block level, so still, no.
Time Machine does it!
Well, not really. Time Machine does seem to be able to determine if two files are clones (which I assume it's doing with private APIs, because I can't find any documented APIs to determine if two files are clones). When it backs up cloned files, it uses hard links to represent them (since HFS+ doesn't support clones, and Time Machine can only back up to HFS+ volumes), and when it restores, it checks to see if those files are clones (which it tracks in a special database), and restores them as clones to APFS...unless they're restored to an HFS+ volume, where all bets are off.
But even in the best case, restoring to APFS, when files get 'separated' when they're changed, again, only the part of the file that was changed is separate. The other blocks are still shared. So even though they've jumped through hoops to maintain the clone relationship, there are lots of cases where Time Machine's own copies will increase in size too, and it happens more and more as the files diverge further.
You guys are so smart, you figure it out! Why are you asking me?
Geez, don't get so defensive!
So, as you can see, given the low-level behavior, there's really no solution, even when you're Apple.
What does this mean for you? It means you can get in cases where data that fits on a source drive won't fit on a destination, even when the drives are exactly the same size.
To avoid problems, you need enough space to store the full logical size of the data (that is, with all the "clones" separated) when you copy, unless you're copying the entire container at a sector level.
Again, this doesn't mean your backup isn't good! It is! It has all your data!
What it does mean is that the data isn't stored as efficiently on the backup. So, it might not fit on your drive when you back it up. And it also might not fit when you restore, if the backup ends up larger than the capacity of the source.
That's easy to check, and the solution is also pretty easy: have plenty of free space on your drives, folks. It's always been good advice, and given all this hidden behavior that happens with cloned files, it's even more important with APFS.
OK, back to the code mines...
Good news: we've set things up on our update server so that the Beta version will now automatically tell you when there's a new beta ready, using our regular update mechanism.
Here's how it works:
Hopefully this will make it easier for everyone to keep up with the current Beta releases.
Don't forget, though: if you're running schedules, you won't notice the updates unless you launch SuperDuper! manually... so, if you want to be sure, check for updates manually.
Thanks to all the testers—we really do appreciate your help.
