Executive Summary: Internally, SuperDuper! can now make bootable copies of "live" Catalina volumes in all three released Beta versions of macOS 10.15. While much work still needs to be done to make things ready for external use, no obstacles are blocking us from pursuing at least one "definitely works" path.
Right up front, allow me to understate something: booting macOS is definitely not getting simpler.
When things change, as they have in Catalina, I've got to delve into what's going on using a combination of on-disk evidence, boot-time kernel logging, and trial-and-error.
When things fail, there's no persistent evidence, so I also spend a lot of time taking pictures of a rapidly scrolling "verbose boot" screen, watching what the kernel is shouting before it fails the boot and shuts down.
A lot of it is kind of tedious work, and there aren't any CSI-style glass panels and black lights to make it look more exciting and cinematic. It's just a bunch of screens and drives and cups of coffee and notes. It looks a bit like a conspiracy theory evidence board. With a crazy-looking person doing the same thing, with minor modifications, over and over again, usually to failure.
Oh-No-Dependent
But, sometimes, to success! And we're now at the point where we have a definite path forward that isn't gated on things we can't control.
That latter issue is a big deal in this business.
When there are bugs or limitations in frameworks or tools or whatever that can't be worked around, it reflects poorly on the product that depends on them. And so we've endeavored to do as much ourselves as we can, to ensure we're not overly coupled to things that have proven to be problematic.
For example, in a previous post, I mentioned how our sparse file handling is much faster than what's built into the system (copyfile) and a typical naïve, cross-platform implementation (rsync). Had we been dependent on either, it would have been much harder to improve the speed as much as we did.
But since we wrote and use our own copy engine, we were able to extensively optimize it without waiting for system-level improvements.
Of course, that has its own potential downsides (copyfile, an Apple API, should handle any OS-level changes needed, but has proven over the years to be buggy, slow, etc), so a careful balance needs to be maintained between dependence and independence. Being more independent means you have to track changes carefully and update when necessary. Being more dependent means you may be broken by factors outside your control...forcing you to rewrite things to be more independent.
Tradeoffs. Whee.
Doing the Right Thing
Last post, I mentioned that we had considered putting in "quick-'n-dirty" Catalina support to try to have something, given the public beta was imminent.
That was going to basically "combine" the two volumes—System and Data—into one, recreating the "old" structure. It was bootable, and "worked", but the problem was with restoration: if you wanted to restore more than just a few files, you would have had to clean install Catalina and migrate from the backup.
That is, basically, what you have to do to restore from Time Machine (it's handled for you), so we decided it just wasn't offering enough of a benefit beyond what Apple was going to provide, especially since we just wouldn't have enough time to test against all the different scenarios involved.
So, we decided to take the "hit" of not having something available, rather than have something "not good enough".
I know that's frustrating to users out there using these early Catalina builds, but, frankly, if you're jumping on early betas you know what you're in for. We're working hard to have a real solution available as soon as we can.
3.2.5's Continued Success
The new public version of SuperDuper continues to perform really well. The biggest issue we've seen is that some people are running through the "old" purchase process on an old version of SuperDuper, rather than installing 3.2.5 and purchasing in the app or the web site. But I can fix that up as soon as they contact me at support.
So, if you have a problem, contact me at support...
Forecast: Brighter Days Ahead
We continue to work to produce a version of SuperDuper! that's usable by "regular users", with a quality level appropriate for a Beta version. Good progress is being made, the path is clear, and now it's just a matter of writing, debugging and testing the rest of the code.
That's not a small task, of course. But it could be a lot worse!
More as there's anything interesting to add...
Another June has come (and gone), and with it the promise of a new, improved version of macOS. In this case, macOS 10.15: Catalina. Sounds so refreshing, doesn't it? Cool breezes, beautiful sunsets, pristine beaches along an endless coast.
But no matter what the imagery looks like, it's a time when I sit at Shirt Pocket HQ, braced for the inevitable news that it's going to be a long, hot summer. In my office. Drinking a lot of coffee.
Thus, in preparation for a lot of upcoming changes, we released v3.2.5 of SuperDuper to wrap up the work we did over the past few months. That release went great, and so it's time to recap previous OS releases and look to the future.
High Sierra
Back in 2017, the announcement of APFS was a big one: it meant re-engineering a lot of SuperDuper! to support the new (and barely documented) file system, along with its new capabilities and requirements. It meant months of investigation and implementation.
A difficult trek for us, but in the end, it meant a pretty painless transition for users, and with it came new features like Snapshots.
But it took us quite a while before we had a version that could be used externally.
Mojave
macOS 10.14 brought its own challenges, and new restrictions on what data could and couldn't be accessed, how scripting could and couldn't work, etc. This required even more reengineering, and another busy summer, despite the fact that Mojave was intended as a Snow Leopard-like "cleanup" release.
But, again, with that work came new capabilities, including better scheduling, smoother operation, command-line support, Smart Wake and Smart Delete.
Unlike the Mojave version, though, we were able to release something that would work well enough pretty early in the Beta cycle.
Catalina
Before going into this, let me state what should be obvious: if you're not specifically writing software that requires you to install Catalina, you shouldn't install the Catalina beta. Really. Let those of us who have to do this take the arrows. Once everything looks great, then you can come rushing in, looking all smart and heroic, and win the day.
Right now, for "regular users", there's no winning. It's all just blood and pain.
Catalina presents more challenges. Not only is the execution environment tightened further, with new requirements and restrictions, but the whole way the startup drive works has been significantly changed.
And I mean very significantly.
In essence, the startup volume is now comprised of two different volumes. The first is the "System" volume, which is what you start up from. That volume is now entirely read-only. Nobody can write to it, except the system, and even then, only when doing OS installs or updates. Users can't write to it. Applications can't write to it.
Basically, think of it as a Catalina CD-ROM you boot from. But, like, faster. And it can be updated. And it's not shiny.
OK, so maybe that's a bad analogy, but you get the idea.
Accompanying that is a new "Data" volume. That's where "your" stuff is. It's read/write. But it's also not visible: the System volume and the Data volume are combined into a new low-level structure called a "Volume Group".
The System volume "points to" the Data volume in this group using another new feature: firmlinks. And as with writing to the System volume itself, only Apple can create firmlinks. (Well, they're "reserved to the system". Plus, additional "synthetic" firmlinks are coming for network resources, but the details of those aren't out yet.)
This sounds complicated (and it is), but it's all supposed to be completely invisible to the user. You might not even notice if you're not the kind of person who looks closely at Disk Utility. (Then again, you're reading this, so you'd probably notice.)
That said, it's not (and can't be) invisible to SuperDuper. This new arrangement presents those of us who are creating bootable backups with—and I'll employ my mildest language here; the forehead-shaped dents in my desk tell a different story—something of a challenge: we can't write to a system volume (again, it's read-only) and we can't create firmlinks.
So...how are we going to create backups? How are we going to restore them?
Yeah, right around here during the announcement is where I might have peed myself a little.
Fetch My Deerstalker Hat! And Some Dry Pants!
Rather than draw this out further, after that initial panic (which always happens during WWDC, so I make sure I've got a ready change of clothes), I've done quite a lot of investigative work, delving into the new, mostly undocumented capabilities, many new APFS volume roles, how they all interact, and I've developed an approach that should work. It does, as they say, in "the lab".
That approach will require, once again, a huge number of changes on our end. These changes will be as extensive as the ones we had to make when APFS was introduced, if not more so. We have to take a quite different approach to copying, make understandable errors appear when the underlying system APIs provide no details, and we have to depend on a bunch of new, unfinished, un-and-under-documented things to make any of this work at all.
It also definitely means you won't be able to back up APFS to an HFS+ volume in 10.15. It's APFS turtles all the way down from here on out, folks, so if you haven't moved your backup volumes to APFS yet, plan to do so once you install Catalina.
But What Does That Mean For MEEEEE?
It's always about you, isn't it...Stuart. (Yes, you, Stuart. Who did you think I was talking about?)
Our goal, of course, is to make all of this new stuff invisible, or as close to invisible as possible. So, when you upgrade to Catalina, and you want to back it up with SuperDuper, it'll basically work the way it always has.
During development, though, this means Catalina is going to be more like Mojave. It'll be a while until we have something to share that you can use. During 3.2.5's development, we tried to come up with something "quick and (not too) dirty" that would do the job well enough to give people with the early betas some coverage, and it just couldn't be done to a quality level we were happy with.
We don't want to release something that we aren't confident will work reliably, even if there are some limitations. That'd be bad for you, and bad for us. So for now, if you're on an early Catalina beta, use Time Machine (and cross your fingers/sacrifice a chocolate bunny/pray to the backup gods).
So far, while we've validated the general approach, we've run into a lot of problems around the edges. Catalina's file system and tools are rife with bugs. Every time we head down one path, we're confronted with unexpected behavior, undocumented tools, crashes and failures. While we're reporting things to Apple, we're two betas in now, and it's not getting better.
Yet. Which is important, as it's still early days. No doubt Apple's exhausted engineers are barely recovered from the push to get stuff ready for WWDC (working to hard dates is never fun; we've all been there), and typically Developer Beta 2 is just "the stuff that we couldn't get done for WWDC that we wanted in Developer Beta 1". And—hey!—Developer Beta 3 just dropped, so who knows!
Anyway, we're forging ahead, confident in our approach. When we have something we will, as always, post it to the blog...and I'll be sharing any interesting trials and tribulations along the way.
When I'm done banging my head against my desk, at least.
