Hard Problems are Hard

You need only look at Apple’s installer, sitting at “one minute left” for an hour, to understand that time estimates of I/O-based operations are very hard.

They seem like they’d be easy, though, don’t they?¹ You just take the size, figure out how fast things are going, and do some math, and bingo-bango-bongo, there you go! Perfect estimate!

Similarly, you take something like the Blackmagic Disk Speed Test and look at its read/write speed and think “geez, there’s the speed, what’s the problem”?

I mean, sure: if you were writing a single file (or some big files), to a blank destination, on a consistent connection. Bytes stream at a constant rate, and it really is just a matter of a few quick observations and a little math.

Real World Cases

But, no. It’s not that simple. I tried the best I could to get things as right as possible, and the estimates are still, sometimes, wrong.

Sometimes very wrong.

Let me walk you through the general (albeit highly simplified) approach:

• SuperDuper walks the drive and calculates about how much data has to be copied (while, at the same time, it’s copying)
• Once it knows, it looks as the rate copying is going
• It then shows you a time

Sounds familiar, right? Well, it goes further than that:

• It’s actually showing a moving average that’s taking speed variances into account
• It’s determining the approximate ratio of “up to date” vs “need to copy” files
• It’s figuring out how long it’s taking to copy small vs big files
• It’s taking previous runs against this content into account
• …and so much more…

Even with all that, sometimes it’ll be wrong.

I just don’t know, until I get there, whether something is going to need to be copied. If it doesn’t, and it’s large, the estimate will be too long. If it does, and the OS is slow to copy that particular file, or that size of file, it’ll be too short.

You get the idea. Hopefully.

“Your Estimates Suck, and So Do You”

For years, I resisted putting in anything like this. It’s just too hard to give an accurate number. But as I shaped the algorithms I’m using in SD4, I figured I could get close, most of the time, in most situations, and it would be worthwhile.

And I think it is. Usually. Sometimes it’s wrong…but it gets better the more you copy that particular content.

It’ll never be perfect, though (although I get very close for replication, which was very tricky, given Apple’s replicator gives virtually no information about what it’s doing, and certainly no speed or even size info), and I try to indicate that with lots of hedging and “weasel words” like approximately and about and estimate to tell you that, well, an estimate is an estimate. It’s an inexact science.

So, if the estimate I’m presenting annoys you, you may want to consider taking the Car Talk “engine check light” approach: put a piece of black electrical tape over it.

Or just close the window. The copy will continue in the background and finish when it finishes.

Problem solved!

A Treasure Trove of Crap?

Applications, especially browsers, make extensive use of caches: folders that are full of zillions¹ of files that make re-accessing data, especially network data, faster after the first time.

They’re typically considered disposable, temporary, and transient, since the content can be re-downloaded (or regenerated) later.

Sounds Perfect For a Skip!

Given the sheer quantity of this stuff, you’d think I’d configure SuperDuper’s copy rules to skip it. It’d certainly make some kinds of copies faster, and the copy itself smaller.

Case closed!

Not So Fast

But…I don’t, even though it would be easy. And there are a couple of reasons for that.

First: if you’ve got a bootable backup, it’s more likely to perform the way you want in the event you need to get to work ASAP if the cached information is retained. You don’t have to rebuild/redownload the info: once booted, things are available just like they were on the original drive.

Second: in the event of a disaster, I want to maximize the likelihood that you’ll be able to recover your data.

If you’ve had some sort of disaster, or hardware failure, or you’re doing data reconstruction, the data you need may be both where it was originally, and in a cache somewhere else on the drive.²

Copying the caches may take a little extra time and space, but if it proves useful in the event of a problem, it’s well worth it.

(And if you disagree, you can, of course, create rules that exclude anything you’d like.)³

Here be Monsters

This is a sort of technical post. If you’re not interested in the details of what’s happening during a backup, what files can and can’t be copied, when, and why they’re harder to do properly than you might think, feel free to skip.

Class is in Session

Recent versions of xOS (that is, macOS, iOS, iPadOS, etc) have Data Protection Classes from A to D that get assigned to files and folders.

Some files have Class A protection: where an application has specifically marked files that it thinks should not be accessible to anyone without authorization.

Class B protection means a file cannot be accessed by anything unless it’s open. That is, even if a device is locked, the file can be created and written, but once closed, Class B is treated like Class C.

Note: Since Class B files are created when Class A keys are not available, they’re encrypted on a per-file basis, in addition to normal FileVault encryption.¹

Class C are files that are inaccessible after reboot until you log in. Once you’ve logged in, they remain available, even if the Mac is locked.

Finally, there’s Class D: no protection. Curiously, this is not allowed under macOS. ¯\_(ツ)_/¯

The thing is, though, is that these files and folders are inaccessible unless the user is logged in, and the screen is unlocked. A few seconds after the screen locks (or you log out), the keys are purged, and they cannot be read.

And there’s no “easy” way to find out what files are what.²

That Seems…Good?

From a privacy/security standpoint, this is obviously good. Someone who gets into your computer cannot access those files unless the drive has been unlocked. On a phone, that means a locked phone has data that’s inaccessible, even via the USB port.

From a backup perspective, though, this is…troublesome.

Troublesome because people tend to schedule copies for when they’re not at their Mac. And if they’re not at the Mac, the screen is almost always locked.

That means the files any application considers to be worth protection cannot be accessed when a backup usually happens.

By anything. Even a backup program with “root” access.³

I Take it Back. That Seems…Bad?

Certainly if important files are marked as Class A, and those files are never backed up, that would be quite bad. And even Class C files are unavailable if you haven’t logged in yet…a broader class of files that could get missed until the next time you back up with the screen unlocked.

Which, given the way people schedule backups, may never happen.

Time for a Follow-Up

Fortunately, SuperDuper handles this with relative ease.

You may have noticed that if you had an overnight backup, when you unlock your screen, a copy starts to run. It’s typically quite brief, thanks to ⚡️Turbo Smart Update, but it might surprise you anyway.

If you look at SuperDuper when this is happening, you’ll see a little pill in the job that says it’s a “Follow-Up” (or, hopefully, ⚡️Turbo Follow-Up).

So what happened?

During your overnight copy, SuperDuper kept track of the files that it couldn’t access due to their protection class. And now that it can access them, it’s making sure they get backed up.

A Million Little Pieces

These are the kinds of details that are involved in putting together a backup program you can count on. There are a million little details, things that’ll catch you out if you’re not extremely careful.

It’s not so much that a file might be written with the wrong bytes, or that you’d have bit-rot…that’s incredibly rare, and the hardware is designed to deal with it to a great extent. But missing files?

That’s not something you should have to worry about. But I do…which is why SuperDuper handles it, without any fuss. You don’t even have to think about it.

And now you never have to think about it again.

Busy Box

SuperDuper v4.0.5 is now available. It fixes a failure to mount an intermediate volume when re-mounting an image.

Yes, that obscure.

Oops

I had thought I’d covered “every” image backup case (and there are a lot), including mounting the “intermediate” (host) volume, then mounting the image, then ejecting the ones that I mounted after the copy.

And I had. Yay.

Once. Boo!

Basically, the unmounting of the intermediate volume left things in a weird state that didn’t re-mount.

That’s it.

Velocity

Although I’d sent this to a number of external sites, and results were positive, I clearly didn’t have full coverage over every case.

Well, I do now. Hopefully the pace of releases will slow a bit soon, but I’m trying to address what people need (or are encountering) as soon as I can. In the meantime

Download SuperDuper v4.0.5 (or, easier, update inside the app)

and enjoy!

In Brief

SuperDuper! v4.0.4 is now available. It includes support for images and a bunch of little fixes and improvements. A reasonably full list is here.

tl;dr It’s good. You’ll like it. And if you don’t, well, I’m sure you’ll let me know…

Network Backups

In the Olden Days, SuperDuper made network backups to disk images, starting with sparse images, moving to sparse bundles, and then to ASIF images when they became available. We did that because NAS devices can’t effectively represent the source’s permissions, ownership, ACLs, etc: even with an Apple “sidecar”, it just wasn’t exact, and so, we went with a NAS-hosted “native” file system: a disk image.

This is, of course, what Time Machine also did, later: there just wasn’t a choice, if you wanted a faithful representation of the source.

Image Unreliability

There are, unfortunately, problems with this approach: network connection failures don’t just take out one file…they can destroy the entire image, or its directory structure, making the image unrecoverable.

You’ve seen this yourself, if you use Time Machine over a network connection: it’ll say “In order to improve reliability, we’ve started a new Time Machine backup”, when what it means is “your image has become damaged and we can’t recover any of its data”.

That’s bad. Very bad.

It was also indirect, so access to your files meant you had to open an image file, then look at its volume, etc. For the less experienced, this could be confusing.

Trying to Improve Things

So, when I set out to write SuperDuper 4, I decided that, instead of images, I’d support backups directly to the network device. That way, you can copy those important folders (say, your Documents or Photos) to the NAS, recognizing that all their metadata can’t be strictly maintained, and any failures would only take out one file, which could be re—copied by re-running.

Other computers on the network could see those photos, documents, files, and work with them directly rather than by opening a disk image.

Win-win. Or so I thought.

What Really Happened

Alas, what “really” happened was that people just tried to copy their entire startup drive up to their NAS device. While it was possible that this could work, a lot of NASen were super sad about having millions of files written to them over SMB.

Whereby “super sad” I mean “would fail randomly with bizarre errors that didn’t make a lot of sense”.

Not to mention copying a zillion tiny files is very slow.

Not the experience I wanted to offer.

Aargh.

Mea Culpa

So, I opened a door to failure, basically. Not my intention, but that was the result.

In SuperDuper 4.0.4, I’ve brought back image backups, but better.

If you select a NAS volume or folder as a destination, SuperDuper warns you that it can’t be full fidelity, might be slow, and offers a single button that will back up to an image instead. You can password protect it if you want.

And all this can be done with one click. You don’t have to set anything but the name…and I automatically provide a sensible default.

It’s reasonably foolproof (he said, keeping in mind that “you can’t make anything foolproof because fools are so ingenious”¹), and kind of nice!

Some Warnings

As I’ve written over the years (you should read that if you haven’t), backups to network destinations should never be your “primary” backup. They’re slower, inherently unreliable, and not as easy to recover from.

That doesn’t mean you shouldn’t use them! They should just be a second line of defense.

And now, with SuperDuper 4.0.4, you can mount that defense again.

Download away, or use the internal updater, thanks for your patience, and for using SuperDuper!

As far as I can tell from the auto-generated subtitles, the folks at Bits und so like the new version of SuperDuper enough that they might start backing up again!

(Which they definitely should do!)

Andy Ihnatko of Ihnatko.com, Material, and MacBreak Weekly recommends SuperDuper 4 as his Pick of the Week—seconded by Leo Laporte and Christina Warren:

Thanks, folks!

Generals and Majors

For those whose time is so valuable that my blathering would be too costly, SuperDuper v4.0.3 is now available for your enjoyment.

For the Smart (Update) Set

There are a bunch of changes in this update, covered, as usual, in the release notes, but the one that you might notice immediately is that ⚡️Turbo Smart Update should happen more often, and start more quickly than before.

Sorted

For those with a lot of volumes, there’s now a sort control for the source list: you can alphabetically by type, or purely alphabetically.

There’s now an option in the sort pop-up that adds a few sections to the source list. SOURCES is for, well, source volumes that have Jobs, as well as the startup drive, which is always at the top of the list.

Volumes that are used as the destinations of those Copy Jobs are under (you guessed it!) DESTINATIONS.

Volumes that aren’t either sources or destinations are in OTHER VOLUMES.

SuperDuper remembers whether or not the sections are expanded or collapsed across launches, so if you never want to see destinations in the list, you don’t have to. And any drives that aren’t involved at all are readily suppressed as well.

Really Big Folders

macOS is really bad at handling huge folders. When you ask for information about them, it can be very slow returning that info…and then you have to process what it gives back.

I can’t do much about the former, but I made some changes to the latter that should significantly improve performance.

And Speaking of Folders

After all files are copied, SuperDuper makes sure to correct all of the various folder modification times so they match the source. It was previously doing this silently, and now the step is called out, and accounted for in the time remaining.

VoiceOver Improvement

I’ve always tried to make SuperDuper accessible—one of the reasons for the text is so that the actions could be described and not inferred by folks using screen readers—but the initial releases of SuperDuper 4 weren’t as VoiceOver friendly as I’d hoped.

I’ve made some progress on that in this release, and I’m not done. Apologies to those frustrated by the issues…know I’m working on it.

Take this Job and Shove It (to another location)

Drag and drop job reordering was broken in the last update, and it’s fixed in this one. Drag away.

Jobs can also be reordered with Cmd+Option+Up/Down.

Availability

Now, of course, via the internal updater, or by downloading and installing manually.

Enjoy!

Before I crash after a long day working in the code mines (v4.0.3 is coming along nicely), I thought I’d put on my marketing/appreciation/thanks hat.

Michael Tsai of SpamSieve, DropDMG, and the MJTsai blog (all of which, if you’re not using/reading, well, you should) posted a review of SuperDuper 4 on his blog today.

Dan Moren of sixcolors…or should I say novelist Dan Moren (great books you should snag) posted a review on Six Colors as well.

Thanks, folks!

For the Executive Set

SuperDuper v4.0.2 is now available, both in-app and as a direct download.

It has “Bug fixes and performance improvements”.

For Those With a Bit More Patience

Doing this kind of work requires patience, focus, and care. And recognition that no matter how carefully you test, no matter how many configurations you put in your test suite, no matter how diligent your external testers, something always gets through.

Nothing bad, knock wood. But something.

And so, as the number of unusual configurations pop up, so do the unusual edge cases. The flaky drives, weird NAS devices, software environments that interfere with basic operations, etc.

This, for the most part, is a collection of fixes for those unusual things that you just don’t know about until you see them, and can’t really anticipate fully.

Exhaustive / Exhausting

Rather than restate the release notes here, I thought I’d focus on a couple of the more interesting cases.

Rules of the Game

Copy Rules look simple—a couple of operation types, some basic patterns. But interaction between the various elements is highly complex, and part of the challenge is ensuring that you follow the intentions of what the rule is doing.

As an example, consider a rule that is designed to tag some files or folders as “optional”—that is, files that could potentially generate errors during a copy, but since they’re inessential, those errors should be ignored.

Well, if you have that rule, and then later say “exclude all files”, should the rule be applied? What about a rule that preserves a specific folder in your Home folder, followed by a rule that excludes top level folders and then includes all of Users. Should the folder be preserved?

Anyway, stuff like that can drive you insane. v4.0.2 has various fixes in these areas that limit trips to the “it’s doing what?” asylum.

Ghosts

There’s an enormous amount of code that goes into maintaining volume identity, the sidebar, the job selections, etc.

As you might expect, it’s critical that this be right. You don’t want to end up accidentally targeting the wrong drive, or changing the user’s selection.

The balance between “safe re-selection” and “prompt for missing drives” is a delicate one, and it’s better to be a little too strict than not strict enough.

This is especially challenging when drives mutate in extensive ways, like when a single volume turns into a volume group and separate System and Data volumes, plus a series of support volumes. Put a bunch of those side-by-side in a container and, well, it’s a lot. I would not be exaggerating to say that just making crazy stuff going on behind the scenes look semi-normal “live” has taken months of tweaking.

Even with that, there were some situations like this that caused “ghost volumes” to appear in the sidebar. Those have been fixed.

Ye Olden Days

You’d think, given how long I’ve been doing this, that I would have covered Apple Partition Map. You’d be wrong. I fixed it, and have ashame.

NASty

Various NASen have unusual behaviors when dealing with file structures that aren’t necessarily native, and don’t all deal well with very rapid I/O. I had already implemented some heuristics that ramped our write speeds up and down depending on how well the network connection was handling it, and this version adjusts that behavior further to accommodate the specific performance characteristics and failures experienced during the copy session.

In addition to this, most AFP implementations out there are, well, quite defective. If you can connect with SMB, and plan to write to networks with SuperDuper, please do so.

And the Like

You get the idea. The margins have been aligned, some rough edges sanded down. Complete details are in the release notes.

You can update either in-app or by downloading directly.