iFixIt recently released a video showing how to upgrade to an SSD using SuperDuper.

Thanks, iFixIt!

This isn’t the first time Antivirus programs have caused problems for us, and it won’t be the last. But, if you’re using Sophos Antivirus and are seeing the error

| 05:21:41 PM | Info | ......COMMAND => Blessing OS X System Folder
| 05:21:41 PM | Error | umount(/private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/bless.5cmA): Resource busy -- try 'diskutil unmount'
| 05:21:41 PM | Error | /sbin/umount returned non-0 exit status

the problem is that Sophos’ “Anti-Ransomware” module is preventing macOS’s bless tool from working.

This is just another example of Antivirus programs trying to “help” but, in the end, making a system less reliable in an obscure, confusing way.

My guess is that it’s actively scanning the Preboot volume that bless has mounted in order to properly set up things for boot, and thus vetoes the ejection of that volume when bless tries to eject it.

One of the fundamental jobs of an antivirus tool is to not break basic system functionality while doing its thing. At least in its current release, Sophos’ Anti-Ransomware feature fails at that task. If you’re using it, and you’re getting errors, you’ll have to turn that feature off until they fix their bug.

Hey, folks.

As you may know, one of Mojave's focuses is stability and security. Part of that is denying applications, including those with 'root' privileges, the ability to access certain files on the drive.

This started in High Sierra's SIP implementation, where certain system files were inaccessible. But Mojave extends this protection to "sensitive" user files as well.

As you might expect, not being able to copy files throws a bit of a wrench into the whole "backing up" process, and bad things happened when we tried (crashes, etc).

Fortunately, as of Public Beta 2, you can now tell Mojave that SuperDuper is a "good actor", and you should allow us to access your data. Mojave doesn't prompt you to "OK" us, however, so you need to do this before you run.

Open System Preferences and switch to the Security & Privacy preference pane. Under Privacy > Application Data (Full Disk Access in later builds), add SuperDuper.

Once that's done, your backup should work.

Executive Summary

Changes in macOS 10.13.4's behavior caused some problems for scheduling, and for the final steps involved in preparing an earlier major OS version's copy for boot. The beta resolves these problems (among others), and adds the ability to multi-select schedules in the Scheduled Copies window and run them all with one click.

SuperDuper! 3.1.5 Beta 1 Download

Replacing Gaskets

Setting up a drive for boot involves running the appropriate system tools, from that OS version, so that things are configured as the older OS expects them to be.

Unfortunately, 10.13.4 broke compatibility with earlier major macOS versions of update_dyld_shared_cache, causing the tool to crash on launch.

While the copies are fine, the error was, at the very least, disconcerting. This beta resolves that issue, at the cost of slower boot...or, potentially, the need to attempt the boot more than once the first time it's started from after a copy.

COSC Timing

When 10.13.4 was released, we also started seeing an increase in user reports of either "missed" schedules or schedules that didn't run until the user logged in. (Typically, they'd see SuperDuper! launched, unexpanded, and not running; the run would then start a few seconds after logging in.)

We tried a lot of different things to resolve this problem over a number of weeks, and finally hit on the multi-factor solution we needed to get things working reliably again.

Basically, it all came down to the system driving back to sleep before we had a chance to hold an assertion to keep it awake.

Unfinished Base Movement

In previous versions, our launchd job responsible for running our schedules would run on a repeating interval, every 60 seconds, and check to see if there was a job to run. While it was guaranteed to run once a minute, the second wasn't guaranteed, and thus could potentially be as late as 59 seconds past the minute.

I did this because, at the time, I couldn't figure out how to get a job to run every minute on the minute.

This meant that the wake, typically a minute before, could take more than two minutes before the real copy started...and the system would go to sleep before we had a chance to tell it not to (since we didn't want to tell the system to not sleep when we're just up, but rather while we're running a copy).

Chamfered Edges

The first attempt at this was to use caffeinate in the launchd job to hold the system awake in the background while the scheduled copy had a chance to get going. But, even though this worked in some testing, inside launchd it just didn't work: the assertion wouldn't persist.

As a second cut, we changed SuperDuper! to hold a brief assertion when launched. That helped somewhat...but didn't solve the problem.

To improve things, I delved into launchd some more, and (finally) figured out how to get our launchd job to run every minute, on the minute (for those who want to do this, you counter-intuitively use an empty dictionary). This helped even more: we can now set the wake event for the same time as the schedule...but it didn't fully resolve the issue, especially when users had a bunch of schedules set to run overnight.

Finally, we thoroughly investigated the internal script client scheduler: the part of SuperDuper! that sequences multiple AppleScript clients so they don't interrupt each other or all try to run at once. In there, we found some obscure logic errors that caused individual schedules to sometimes not start immediately, leaving them waiting in the queue for a while, even though they should have run right away.

Since this potential delay was longer than the sleep assertion we added in the worst case, the "real" assertion, set up during the actual copy, would sometimes never get a chance to take hold...and once again, the system would sleep. This resolved that problem.

Blued Screws and Gold Chatons

At the same time, we fixed a longstanding problem with "progress spinners" sticking around in the Scheduled Copies window even though the schedule was complete, and with schedule sequencing when more than one schedule was pending: now, if you space the schedules one minute apart, they're guaranteed to run in the order they're scheduled, rather than in a semi-random order.

And as a final bonus just-in-case we retain a "don't sleep" assertion as long as there are any pending AppleScript clients, to ensure that the system doesn't go to sleep before they get a chance to start their copies.

The sum total of all these changes: problem resolved (with even better behavior than before), and groundwork laid for future improvements as well.

Glashütte Stripes

For a finishing touch, we added in something we've wanted for a while: you can now select multiple schedules in the Scheduled Copies window and run them all on demand by clicking Copy Now.

There are a bunch of other fixes in here, too. We worked around another(!!1one!) very weird system pipe bug that was causing communication between the copy engine and the UI to break. This also improved the retrieval of error information from the copy engine.

We fixed a problem with file copying when a file was deleted out from under us between the data and attribute parts of the copy - now, we continue past that point, since the problem is pretty clear.

We fixed another problem where an image file might get recreated, rather than updated, in some situations.

And we finally turned off the spell checker for the log view (yeah, I know, don't ask), and improved its operation, so it won't scroll out from under you if you open it during a copy...and people won't think that the spell checker underlines are errors.

There are things I'm forgetting, but since we're already nearly 1000 words in...

Final Adjustment in Six Positions

We're pretty confident that this version will work great for you, as it's been tested on a broad spectrum of user systems, macOS versions, etc. To be sure, though, before we update the world, we've decided to release this as a Beta today. As explained in the beta post, if you install this, and we need to release updates to the beta, it will update automatically and separately from the release update feed. And when the final version is released, it'll also automatically become a "release version" and transition to the regular update feed.

So...have at it, and let me know how it works for you.

SuperDuper! 3.1.5 Beta 1 Download

We've been getting a lot of questions lately about the APFS image bug (recently documented by Mike Bombich, author of CCC) where, when an image container can't grow, APFS doesn't reliably indicate an error, which can silently lose data. Worse still, double-checking the data seems to indicate it's OK—even a checksum succeeds—until you eject, at which point the data is lost.

This looks to be another manifestation of a problem we've known about for years (and documented in the User's Guide back in 2006): when an image can't grow, whether due to disk full errors, or because the host volume doesn't support large files (eg FAT32/EXT2), writes to that image can fail catastrophically.

Failure is Inevitable, Success Requires Planning

Whether the problem is a bug in the OS or a hardware error, failure is inevitable. But if you implement a successful backup plan, you can protect your data, even in the face of this sort of adversity.

So let's take a moment to talk about "best practices" for backup devices and methods.

Simpler is Better

This should be obvious, but is often ignored in the search for convenience (or expedience): the fewer layers there are between your computer and the storage device, the more reliable your backups are going to be.

What do I mean by that?

This: you should write directly to a local drive. That's going to be your most reliable, bootable solution. Don't write to an image stored on that drive. Don't format it as a foreign file system (like NTFS) and use a special driver to access it. Write. Directly. To. The. Drive.

Of course, things can still go wrong! But recovery will be easier and faster. Remember, a single bad sector isn't likely to take out your whole backup...but it could destroy an image.

You Can't Start Up From a Network Drive or Image

This is pretty basic, but important. You can only boot from a backup written to a properly partitioned and formatted, locally connected, non-network drive.

You can't boot from a disk image (it's not a "real" drive), whether it's stored on a local or network drive.

But I Want to Store More Than One Backup on a Drive!

If you need to store more than one backup on a physical device:

• If you're on 10.12 or earlier, partition the drive into the number of volumes you need to back up. So, three source volumes to back up? Three partitions on the backup drive.
• If you're running 10.13 or later, format the backup drive as APFS, and use APFS's very flexible "volumes" as your backup destinations, one per source volume.

We hear all the time that people are using images to "make more efficient use of space". That's the wrong thing to do—there's just no other way to put it. An image needs to be able to grow to its maximum size, unimpeded, no matter what. Images do not necessarily "hug" the data inside them, and may grow to full size even when their contents are smaller.

If the image cannot grow, it's quite likely you will corrupt that image and lose the data in it. So you need to have all the space available anyway!

High Sierra's native APFS volumes do this better. All the volumes share the same storage pool, and allocate that storage intelligently, managing that storage better than images ever did. Use them.

But I Want to Store Backups on the Same Volumes as Some Data!

Don't ever do this, except in a short term emergency situation.

What people seem to forget is that the data they're storing their backups alongside also needs to be backed up. And if you're storing your backup on that same volume, you're probably not backing up the data.

This is a mistake. A huge mistake that we see way too often.

Typically, we'll hear from people who work with large amounts of data, such as photographers, designers or musicians. They'll move their "older" work to an "archive" volume, and they'll want to store backups of their current work alongside that archive.

But they'll often forget they need to back up their archive too!

Don't be a goofus. Having an archive volume is fine. Keep it separate from your backup volume. And back it up.

But I Want to Back Up to a Network Volume!

OK, so we've covered the cases where you've got local drives. The best thing to do, to local drives, is to never use images unless you absolutely need to. Never.

But what about a network drive?

First, I truly believe you should never, ever use a network drive as your only backup. It's fine to have a network backup as a secondary backup. But by its very nature, it's going to be the least reliable one. It's not only subject to the potential flakiness of images, it's also subject to the flakiness of networking in general.

It's one thing when you try to access a web site and the page doesn't load. It's another thing entirely when the network "burps" when you are updating the structures of a backup volume.

You've probably already seen the result of this kind of failure. That message Time Machine displays that says "In order to improve reliability, we've started a new Time Machine backup"? That's because the image failed...and your backups have been completely lost. (Kind of a mild sounding message for a catastrophic failure, don't you think?)

So, here's my advice for networked backups.

If you're using a desktop, and you have a modern NAS device that supports iSCSI (like a Synology or QNAP), purchase an iSCSI initiator (like the Studio Network Solutions GlobalSAN initiator, or ATTO's Xtend SAN), create an iSCSI volume, format it natively for the Mac (as HFS+ or APFS) and back up to it directly.

Yes, iSCSI volumes are also containers, but those containers are managed on the far side of the connection, rather than the near side. As such, a near-side failure is much less likely to corrupt the container...and more closely resembles a typical local drive failure, which are almost always repairable by Disk Utility.

If you have a laptop, or can't use iSCSI, create a share that has more space than you need to back up, and back up to a sparse bundle. If your device doesn't support sparse bundles, use a sparse image.

Broad Spectrum Protection

Remember, you should never rely on a single backup device, or a single backup program. No matter what you're using for your backups...use something else too.

So, if you're using SuperDuper!, also use Time Machine, preferably to a separate device. Have multiple backups, with both SuperDuper! and Time Machine. And use something like Backblaze, so you get an offsite backup!

Backup Frequency

As I've indicated in the User's Guide, I suggest a daily, weekly and monthly backup with SuperDuper. It's easy to do - just make three schedules to different drives.

I set up my weekly and monthly backups "on connect" - that is, when I plug in the drive (which I keep separately), SuperDuper! launches, copies and then (since I've set an "On successful completion" action), ejects the backup and quits.

My daily backup is left unmounted but connected. SuperDuper! launches every day, mounts the drive, makes its copy, and then unmounts the backup.

I leave Time Machine on its default schedule, so it's doing a roughly hourly backup.

And Backblaze is backing up continuously.

Be Smart, and Don't Cheap Out

As I've said many times, no one was ever sad because they had too many backups.

Drives are inexpensive. It costs less then $100 (more like $60) to get a 1TB USB3 external drive. For $10 more you can get 2TB. 512GB External SSDs are less than $200. Spend the money. Your data's worth it.

Don't let your data loss story serve as a warning to others. Be the hero who planned ahead and saved your family's precious photographs. We're happy to be the invisible partner alongside you as you save the day.

Just remember the most important rule of all...

Turns out, a few of you have drives that don't have low-level media IDs...something we thought all drives had. And they did, at least, all the drives we and the external testers tried did.

Well, now we know...and, now we don't crash. Sorry about that.

Fix now available in v3.1.4 - download away!

SuperDuper! 3.1.4

Way back when, not so long ago, we transitioned the main SuperDuper! application to 64-bit. As I said then, we were 95% 64-bit, and 5% 32-bit.

Today, we're releasing SuperDuper! 3.13, which—after extensive testing—moves the copy engine to 64-bit as well. macOS 10.13.4 is going to start giving obnoxious warnings about 32-bit app usage, so we figured this was a good time to release the update, even without any significant user-facing feature changes.

I'd love to use this blog post to tell you that, because we're now 64-bit, we're 200% faster, but, well, no. Internal testing shows no significant gains or losses from being 64-bit. In fact, we have to do additional work packing and unpacking some file system structures, which are laid out (for legacy reasons) for 32-bit apps.

Of course, we fixed a bunch of other things, too. One specific change was to improve handling of source drives with bad recovery volumes.

We're not sure how people's drives get like this, but sometimes there will be a number of "Untitled" 650MB volumes on their drives. Disk Utility will indicate one of them is the Recovery volume, but when we try to mount and copy from it, we get an error, because the drive is bogus and can't be mounted.

Previously, we would stop and give an error. Now, we log the problem and generate a warning that explains how to resolve the problem. (Basically, you need to reinstall the OS, which refreshes the OS under your existing applications and data, and recreates/fixes Recovery.)

While we were at it, we fixed a few things in the last 10.9 compatible version (3.1.1), back ported some other fixes we made in 3.1.2, and updated it. Unfortunately, since it already had a version number, I just updated the binary. Bad form, I know, since there are now two 3.1.1s out there. But if you're using Mavericks, and can't create schedules with your existing 3.1.1, download 3.1.1 again and reinstall.

So yeah, we're older and we're losing our hair, but you really do still need us. Update inside the app, or download away!

SuperDuper! 3.1.3

Shipping software sucks.

It's not that we don't want to put the software in the hands of our users. And it's not that we're not proud of the work. We do, and we are.

What sucks is that you always ship with bugs. Always. Some of them you know about. Some of them you don't.

Anyone who does this for a living knows how this works. You keep track of everything you find in the software. Some things are reproducible. Some things aren't. You draw up general approaches for investigation for the latter, and propose fixes for the former.

You then prioritize things, hash out a schedule, and decide what's worth fixing and what you can, in the end, live with...for now.

"We need to fix this post 2.0"

During the run-up to SuperDuper v3.1.2 (available today, see below), we were dealing with this whole investigate-prioritize-allocate-test cycle, when someone brought a sheet to our attention that's been a known low-priority thing for years.

12 years, to be precise.

It's the basic "Stop the copy in progress" sheet that comes up when you want to stop a copy. It's something you can do by accident, and throw away a lot of progress, so we show a sheet that asks you whether you really want to stop.

Sensible, as far as it goes. But during the development of 2.0, there were a lot of potential cases that would have to be dealt with during a cancel, and while we were working through them, and how to present them to the user, a temporary cancel sheet was put in place.

A sheet that was immediately logged as "this has to be fixed" in our tracking database, because it was poorly worded, confusing and (please don't hurt us) had "Yes/No" buttons at the bottom. Enjoy this example of our overwhelming UI genius:

So many problems with that sheet. It's wrong, it uses crazy terminology, it doesn't match the rest of the app, it uses Yes/No buttons. I don't even...

There was a lot to do for v2.0, and development proceeded, other tasks took precedence, this sheet isn't used very much, and eventually we shipped 2.0 with the sheet in place, with the bug tagged as low-priority, known-terrible, embarrassingly bad, "we need to fix this post 2.0".

That cycle repeated over and over. It was an easy fix, but other things took precedence. This stuff happens. Shipping software sucks.

Well, it was terrible, and it's now 12 years post 2.0. So for the new year? Start celebrating, because it was fixed.

More Fixes!

This version also fixes a bunch of other relatively minor things that should please one or more of you:

• On some systems, the default settings didn't save if you had available snapshots and then went into Options, changed something, and exited.
• If your drive had a "#" in its name, an erase-then-copy backup would generate an error.
• Renaming a drive wasn't always reflected properly in the UI.
• Sometimes the Stop button would beep and not stop
• In some regions, snapshot date parsing could cause a crash at startup or drive selection

General cleanup, nothing too exciting, but a fix is a fix, and now they're available to you. Download away!

Download SuperDuper! v3.1.2

We've got something under the tree for all of you, and you get to open your present early.

Executive Summary

SuperDuper! 3.1 is out, and now allows Time Machine-like restoration from bootable APFS copies.

Wait, What?!?!

Oh, decided to join us for the details? Welcome to an ocean of words: prepare to get your feet wet!

Well, I told you there would be more surprises in store, and, so, here we are. Since the release of 3.0, we've not only been copying from a snapshot when using an APFS source. In addition, as some of you have noticed, we've also been creating snapshots on the backup.

This is why.

Turn Back the Clock

In SuperDuper! v3.1, you may notice a small change to the Copy Now button: a little up-triangle glyph. If you've selected an APFS volume as your source, you can click that triangle, and up pops a list of the snapshots available to copy from.

This means you don't just have to copy from the drive as it is "now" (the default choice). You can select from any existing snapshot, and we'll copy the state of the files as they were at that time.

But why would you want to do that?

Well, in High Sierra, the system takes a snapshot of your drive before it installs a software update. Having a problem with an update and want to check if it was different before, but forgot to back up first? No problem: use SuperDuper! to copy from the snapshot taken just before the update was installed, start up from the copy, and check the behavior.

Even better, if you're backing up to an APFS volume as we encourage, this works on a bootable copy, too! Since v3.0, we've been taking a snapshot of the backup volume before every copy we make. That means there's not just one available backup on the drive—if you've been using Smart Update, there are many! Start up from your backup drive, click the triangle, and you'll be presented with a list of available snapshots. Pick one, "Copy Now", and you've restored a day ago's backup, or a week ago's.

But Wait—There's More!

In fact, not only can you use SuperDuper to copy from these snapshots, you can even open Time Machine, select your backup volume, and see older versions of files, deleted files - they're all being saved, automatically, every time you Smart Update. Even though you're not backing up your backup to Time Machine. Because that would be silly.

This new feature is not intended to replace or replicate what Time Machine does. Snapshots are managed by the system, and at present they have some lightly-to-not documented constraints. You need to have about 20% free in a container to create a snapshot, and the system consolidates and removes snapshots according to its own logic.

As I've said before, an effective backup strategy should have multiple parts: SuperDuper!, Time Machine and an online service. The more (and more varied) backups you have, the safer you are. This new feature gives SuperDuper!'s bootable backups some cool new capabilities and an additional measure of protection should you make a mistake.

Now How Much Would You Pay?

Pretty awesome, right?

Needless to say, this is the kind of feature we love to introduce. It's completely transparent to the user, and is built on the extensive work we did to support APFS in v3.0. That support wasn't done at a surface level: we worked hard to research and make use of the more advanced features, like snapshots, and planned for the future in ways that truly make the backup experience better for everyone.

Including you. Update away!

Download SuperDuper! v3.1

One of the weird parts of doing a public beta, as we did for 3.0, is that the release of the final version doesn't quite feel the same. So, even though it's only been two weeks since 3.0 came out...it feels like it was released months ago.

The 3.0 release has actually gone incredibly smoothly. The extended beta managed to attract a large number of diverse system configurations, and that helped us ensure that any post-release issues would be...obscure. And, indeed, they have been!

Pulling the Right Levers

A program like SuperDuper! needs to make choices about how to interface with the system. There are many different layers, and we try to choose appropriately, depending on what we're doing.

To ensure a maximum level of compatibility, it's incredibly important to work at as high a level as possible for any given activity. For example, when SuperDuper! needs to erase a drive, it would be a huge mistake to actually jump down to the driver level and start mucking with drive structures.

Instead, we do the sensible thing: we ask Disk Utility to do it for us.

Talk to the Command

Disk Utility, though, is what's called a command-line tool: you can run it yourself, from the Terminal application, if you want to: type diskutil to see what it does, and man diskutil for its documentation. Much of macOS is built upon these Unix tools, and (where appropriate), so is some of SuperDuper.

When you talk to a tool like this, though, you have to both check its "return value" (basically, a rough "I worked!" or "I failed!"), and its text output (which says why it worked or failed, or gives you an answer to a question you asked).

Some of these tools (diskutil, hdiutil, etc) will format their output in plist format: basically, a structured XML document that's far more predictable, easier to parse, and more reliable. So rather than returning "Drive formatted as HFS+", it might return (intentionally simplified)

<format>hfs+</format>

which is obviously less friendly to people, but a lot better for programs trying to figure out what's going on.

This plist output's advantage is that its structure is strictly defined. It has a prologue, a series of tags and values, and an end. This is all guaranteed.

Samsung!

So, when a few users started getting some crazy, inexplicable errors (typically, an error indicating that a volume that was there couldn't be accessed), we were shocked to find, after doing some extensive investigation, that a Samsung SMART driver (based on an open source project) was actually polluting diskutil's output with a bunch of debugging/progress junk.

Needless to say, this prevented parsing of the output, and resulted in a stream of invective that did not diminish for many minutes. So frustrating. But fixed.

Various Improvements

That was the big one. But we also fixed a number of other things:

• Finally fixed the update animation bug! So, if you install 3.0.1, the next release is going to show up properly in all cases. This time for sure!

  This took such a long time to fix because it was entirely timing related and didn't happen in the debugger. But, we figured it out. So embarrassing. Gonna miss you, you little rascal. (Please don't darken our UI door ever again.)

• Settings conversion related quit issues
• More reliable sleep prevention during a copy
• Supports encrypted destination volumes on 10.12
• Restored 10.9 support

I also took some time over the past week to turn on SSL for Shirt Pocket, so site access and downloads are all being done securely.

Happy Thanksgiving

So, with that, from our families to yours, have a Happy Thanksgiving, and enjoy SuperDuper! 3.0.1, which is available via the built-in updater, from the web site, or here:

SuperDuper! 3.0.1 Download