Shirt Pocket Watch

Yeah, well, sometimes a change to the install process that shouldn't have any effect on anything else...does.

Beta 3 gave an error when using bless to bless the drive. The underlying issue was code we have that needs to ignore certain types of output on stderr that aren't actually errors. (For example, the EFI fix that I describe in this blog post produces an error that isn't actually an error–we know it's going to fail in a specific way).

The unexpected part is that just before the beta, we made a change to the installer to try to improve our workaround for systems that required rebooting post-install to make Full Disk Access work. After we made the change, we didn't re-run the full suite of tests because we (incorrectly) thought the change was isolated to the install process.

However, it was made in a runtime element that was shared with the way we executed bless. We didn't take that into account, and should have re-run all the tests.

The backups themselves are fine (and are even bootable, even though there's an error issued). Beta 4 fixes the false error indication: once installed, bless will not give a false failure.

Sorry about that. Beta. >sigh<

Download SuperDuper! 3.3 B4 (v119.5)

Third time isn't a charm; you'll note a bless error at the end of copies. We'll have a new beta Sunday that fixes this dumb goof.

(For the impatient - scroll down and you can download Beta 3 at the bottom of this post. Note that existing betas will, as always, automatically update.)

Well, we've reached the critical Beta 3 phase, where we think we've dealt with all the issues reported, save for one or two stragglers we're trying to figure out.

Mostly Beef, Some Filler

Overall, things are pretty solid. We tightened up some of the image and encryption handling, which affected some users (and provided interim betas to those who got in touch to confirm our fixes). There were some problems with volumes being ejected at the end of scheduled copies under Catalina that were fixed, as well as some related issues with auto mount (also only under Catalina).

Pretty minor stuff.

Mystery Meat

We've got a few users whose systems are in a bizarre state where the loader is outputting

dyld: warning, LC_RPATH @executable_path/Frameworks in /System/Library/PrivateFrameworks/TelephonyUtilities.framework/Versions/A/TelephonyUtilities being ignored in restricted program because of @executable_path (Codesign main executable with Library Validation to allow @ paths)

when we run certain system command-line tools. It's happening to a very small number of users, but for those users, neither rebuilding their dynamic libraries nor reinstalling their version of Catalina (current public release) fixes the problem.

We were able to reproduce the problem, once, with a non-admin user under the current public release, and it fixed itself after installation of the current developer preview, but given we can't reproduce it again, even with the public release, we're not entirely sure what weird system thing is going on here: we just know it's not good.

If this is happening to you, get in touch.

Certified Fresh

We also added a diagnostic that detects a rare situation where a user's system has broken scripting tools (like a bad Perl install), which can cause problems. Again, rare stuff.

Piping Hot Chinchilladas

But, apart from those relatively minor kinds of things, Beta 2 worked well. And Beta 3 should work even better.

Once again, thanks for your patience and support. Enjoy!

Download SuperDuper! 3.3 Beta 3 (v119.4)

As always, a download for the new beta is at the bottom of this post, and if you're uninterested in the changes, you should just scroll down a bit (or, if you have a previous Beta installed, launch SuperDuper and it will offer the update).

Beta 1 - Worked Well!

We had a lot of people download and run the new beta, and a gratifyingly small number of problems, which is always a good sign.

That doesn't mean nothing was wrong with it, of course. A developer's job is never done, and the only program with no bugs in it is, well, theoretical. And hey, it's a Beta—the whole point is to get broader testing so we can fix any problems we missed internally.

So thanks to everyone who wrote in with success and failure reports.

Small Issues

The first beta had some relatively rare problems:

1. If there was a space in the path to SuperDuper, we'd give a weird error in the log and fail.
2. The versions for some of our launch agents weren't set properly, so they'd get reloaded on every SuperDuper launch.
3. On some user systems, Full Disk Access doesn't take after install, and they have to restart after installing the new version. This is because our bundle ID has changed due to notarization and the OS doesn't handle it well. We did our best to try to get it work, and it does for most. One time issue for those it happens to.

Larger Issues

And a few larger issues:

1. Disk image backups would fail when copying source volumes with snapshots enabled.
2. Some valid scheduled copies would fail to run under Catalina.
3. In some circumstances, ownership wouldn't be properly enabled for the system volume of an external Catalina volume group, which made the backup not boot.

Sea of Memories

I could go into detail on the latter problem, but rather than bore you, I'll refer you instead to this old post from 2005 (14 years ago oh god I have been doing this for too long shut up inner monologue i am trying to type here no you shut up) where I discuss the general issue.

The new Beta resolves this issue. Note, though, that SuperDuper doesn't copy the System volume if it's not a different version. So, there are two options if your backup doesn't start up as expected:

• Navigate to /System/Library/CoreServices on the backup drive and remove SystemVersion.plist. The easiest way to do this is to use Finder's Go To Folder command, in the Go menu, and enter:

  /Volumes/your-backup-volume/System/Library/CoreServices

  Substituting your actual backup volume name for "your-backup-volume". Then, remove SystemVersion.plist. When you run the new version of SuperDuper, it'll fix all the files without recopying their data, and you should be all set.

• Alternatively, you can do an erase-then-copy backup rather than a Smart Update.

That wasn't too painful

See? When I'm tired I can't ramble on as much (yes you can you're rambling now shut up i said inner voice shut up shut up).

Note that if you don't have drives selected in the source and destination pop-ups, the internal updater won't work. Either select drives or download and install manually.

Thanks again to everyone who provided feedback, and an extra tip of the pocket to Ben, Sydney, Mathieu, Jeff, Gary and Norman for their reports and testing help.

Download SuperDuper! 3.3 Beta 2 (v119.3)

As promised, at the bottom of this post is a link to our Catalina beta.

A very dramatic announcement! (Also, see https://t.co/2mDlY6LUfu for details.) pic.twitter.com/AUFtTuesUh

— Dave Nanian (@dnanian) October 18, 2019

This post is mostly a just-the-facts-ma'am discussion of changes in Catalina that have affected us and may affect you. Details of our journey may come later, when I'm in a more narrative mood.

This version has been a long time coming, and while we hoped to have a beta during Catalina's development, we really didn't like the stability of Catalina during the process. Many things involving the drive setup were changing and being reworked as the summer wore on.

Rather than put out something that was at the mercy of the underlying system, whose release process we couldn't control or predict, we decided to hold off until the real release. Once that was out, we finalized our testing, and now we're putting the public beta into your waiting arms/hands/drives/computers...whatever metaphor here is appropriate.

Drives are Different

Apple's intention is to hide the details, so it's possible you didn't notice, but in Catalina APFS drives are now quite a bit different than they were before. Let's just do a quick refresher of how we got to where we are now.

Stage 1

macOS started out with pretty "standard" Unix-style protections. Permissions and ownership were used to establish what you could or couldn't do with a file. The "super user" could override those permissions, so while you were normally denied access to "system" files, a little "permission escalation" allowed you to modify anything you wanted.

And, since "you" could modify anything with a little work, so could something that had unkind intent.

Stage 2

In 10.4, Apple added ACLs, which are more fine-grained "Access Control Lists": super detailed permissions for files and folders that can be inherited. Most people aren't aware these exist, but they're super handy.

Stage 3

The next step was adding a special attribute ("com.apple.rootless") that caused the file system driver to "protect" certain files on the drive.

That caused some initial consternation that I documented on this blog, because Apple "overprotected" the system, even when it wasn't the startup drive. Fortunately, before shipping, they fixed that problem, and copying worked as expected.

Stage 4

Next came System Integrity Protection (SIP), which took that concept and extended it to some user files as well. Some areas of the drive couldn't even be viewed, let alone copied, but with certain types of authorization (like Full Disk Access), copying could be done.

Stage 5

And now we're at Stage 5, which is either denial or acceptance. I forget, but basically this is where the entire system has been made read-only, and placed on its own, separate volume.

It doesn't look that way in Finder: everything seems like it was. But behind the scenes, two separate volumes have been created. Your files, and dynamic system files, are on a new "Data" volume. The system is on its own "System" volume. And they're stitched together with "firmlinks", a new, system-only feature that makes two volumes look like one at the directory level.

Make this invisible

Again, Apple has tried to make this invisible, and for the most part, that's what they've done. And we've tried to do the same.

In SuperDuper, you won't notice any changes for the most part. But behind the scenes, we've made a lot of modifications:

• In order to replicate this new volume setup, system backups of APFS volumes must be to APFS formatted volumes. SuperDuper automatically converts any HFS+ destinations to APFS volumes for you (after prompting), so you won't have to do anything manually in most cases.
• Any existing APFS volumes are properly split into these new Data and System volumes. If your backup was previously called "Backup", your files are now on a volume called "Backup - Data". The read-only system volume, which contains Apple's files, is called "Backup". These volumes are now set up in a "volume group". Again, all this is done for you, and we hide the details...but if you see some new volumes, you now know why.
• And because of this, a single copy is really two copies: the system volume (when unmodified) and the data volume (where your stuff resides)
• Those two volumes are further linked together with "firmlinks", which tunnel folders from one volume to the other in a way that should be transparent to the user. But they can't be transparent to us, so we had to figure out how to recreate them on the copy, even though there's no documented API.
• Plus, we've needed to make sure we do the right thing when you copy from a regular volume to something that was previously a volume group, intuiting what you mean to do from your copy method.
• I really could go on and on and on here. It certainly felt endless during the process!

Yeah, no matter how much one might long for turtles, it's hacks all the way down.

Seriously, though, this is the kind of thing that requires an enormous amount of testing—not just tests of the program, but testing during development as well. One "little" change that totaled about 10 lines of code required over 1000 lines of unit tests to verify proper operation. And that's just one internal change.

Except when you can't.

Despite our efforts, the new setup means some adjustments must be made.

• You can't turn an already encrypted APFS volume into a volume group. As such, you'll have to decrypt any existing bootable volumes. Once you've backed up, you can boot to that backup, turn on FileVault, and boot back. Subsequent Smart Updates will maintain the encryption.

That doesn't mean you have to back up critical files to an unencrypted volume. If this is important to you (HIPAA or similar requirements are often quite strict about encryption), create a separate account called, say, Intermediate. Use a SuperDuper copy script to exclude your normal User folder (where the critical data is - for example, /Users/dnanian - see Help > User's Guide for a detailed discussion of Copy Scripts). Then, use that new script in "using" to back up to the unencrypted volume.

Start up from the unencrypted volume and log into Intermediate. Turn on FileVault.

Now, start back up from the regular volume and use "Backup - all files" with "Smart Update" to update the backup. This will add your regular account data, which will remain encrypted, as will subsequent Smart Updates.

• Unmounted images are a bit of a problem. We don't know what's "in" the image when it's a file, and so we don't have enough information to warn you before the copy starts.

  Because of the reasons listed in Make this Invisible, we may need to convert the image to APFS. Some cases might not work, and some we don't want to do without prompting you.

  So, we'll give you an error, after the image has been opened. You can then select your image's "open" volume (not the image file itself), and do the backup, once, like that. We'll warn you normally, do the conversions needed, and make the copy. Subsequent updates that don't require conversions will work normally.

• Because we've had to work all through the summer while everyone else was having fun, support may be crankier than normal. Sorry/not sorry.

Wrapping it up

So that's about it. Things should generally feel "normal" to you, although there will be some prompts for those who have encrypted or HFS+ destinations and a Catalina APFS source. Overall, though, the goal is for it to be relatively painless...and I'm sure you'll get in contact if you feel otherwise.

To get started, download the beta below. Note that you'll automatically get subsequent beta releases, and the final Catalina version when available. Once that final version is released, you'll only get release versions...until you manually install another beta.

Thanks for your patience, and we look forward to your (hopefully positive) feedback.

Download SuperDuper! 3.3 Beta 1 (v119.2)

Hey folks. Sorry for the relative silence here. We've been working during the summer on a Catalina update, and unfortunately we're just not quite ready for a general update. But we're close!

Catalina introduces some significant changes to the way a startup drive works, and while we've solved the various issues involved in making the backup process as transparent as possible, it's taken a lot of slow, careful work to get the testing done.

As you might expect, the "new way" Catalina splits your drive into two parts makes things more complicated. The details are hidden from you by Apple for the most part, but SuperDuper has to know, and handle, all the various tricky cases that arise from that split (not to mention the technical details of tying the volumes together, which we figured out early in the summer).

There are stories to tell--our initial intention was to take a different approach than the one we ended up taking--but those will have to wait for when I've got a bit more time.

That said, the GM version was just released on October 3rd, the final version was released today, we've got a beta almost ready to go, and I'll be posting it to the blog as soon as it's passed our internal testing.

That beta works great for most users, but will have some limitations around images: we're probably not going to work with APFS image destinations in the beta when selected as files, if the source is a volume group.

There are also some caveats regarding encrypted destinations: basically, we can't form a "volume group" from a destination that's already encrypted, so you'll have to unencrypt encrypted destinations, copy to them, then boot from them and turn on FileVault.

More soon; thanks for your patience, and thanks for using SuperDuper!