Shirt Pocket Watch

During last year's Catalina introduction, Apple's APFS session introduced a new capability: the asr (Apple Software Restore) tool was now able to perform both APFS copies and, used appropriately, "delta" backups.

We were pretty excited about this new feature, until we actually tried to use it.

The new feature basically didn't work until Catalina's final beta. And even when it started working, while fast, it dealt with failures...poorly.

Sometimes it would leave unusually named volumes hanging around. Sometimes it would leave unmanaged snapshots on a volume. Sometimes the destination drive would be unreadable or unmounted.

And in all cases, it would fail...obscurely. With messages like "Error -49173". Or it would just crash.

Rocks and Hard Places

If you've seen that WWDC session, you know that Apple specifically mentions the new asr capability as being perfect for "Backup programs", and we sort of looked at that as code for you're going to be forced to use this eventually. So, during investigation, when we discovered how incredibly buggy and user-hostile it was, even when it started "working", we took the work we'd done to implement asr support and tabled it.

It's one thing to depend on system features to perform necessary functions: after all, to be a good application, you need to build yourself on top of the APIs that Apple provides.

Much of that work is deciding what "level" you're going to interact at. We're constantly choosing between "higher level" tools (like diskutil) or lower-level APIs (like the IORegistry) to accomplish our tasks.

When there's no significant disadvantage to using a higher level tool, we choose that, because it helps to insulate us from future OS changes: Apple tends to update their own tools to work with new OS releases, and so we have a higher chance of minimizing incompatibilities when a major OS is released.

But one of the "hidden" downside is: sometimes that particular box is way too black. When you put in a file and get out a failure, and the resulting errors are so obscure as to be useless, it's a pretty easy decision: don't use that tool.

Another is less obvious: Apple can move away from the "solution" they were pushing at WWDC, and replace it with "the new hotness", and now you've lost control over your product.

Ch-ch-ch-changes

Regardless of what you pick, sometimes things happen that you can't predict. When Apple makes a change and breaks something, as they have in 10.15.5, they break us...and then we need to react within the framework we've established...or replace what we've selected with something else, because that's the only solution available.

In this case, Apple has broken the ability to make new firmlinks. It's utterly unclear why they broke this capability, but they did. And that makes new and erased SuperDuper! backups unbootable.

Note, though (and this is important): Smart Updates of previous Catalina volumes continue to work properly.

We had hoped Apple would fix the problem before 10.15.5's release, but unfortunately, they didn't. And so we're working on an "automatic" workaround based on the asr work we'd done early in the Catalina beta cycle.

If you have need of a fix "now", please reach out to me at the support email and I'll be happy to assist.

One year after my Dad's death, I was flying back home, out of the first COVID-19 hot zone in the Northwestern US, listening to people cough, feeling the tension in the air.

It was the same flight I caught rushing home, after cutting our vacation short, after he said not to worry: there was plenty of time, and I should enjoy the snow. The same flight where, well before dawn, Mom said to hurry home, things had taken a turn for the worse. The same flight where, on landing in Denver, I found out there wasn't plenty of time.

We were too late. He died while we were in the air.

Today he would have been 92.

The last voicemail I have from him, he was trying to help Z with her ailing Mom, offering advice from a lifetime of medical experience, researching the best approaches to take.

And in the last few months, as we've all gone through this slow-moving wave of death, sickness and horror, he would have wanted to help. He would have found a way.

He was stubborn that way.

He was stupid that way.

He was selfless that way.

Happy Birthday, Dad. We all miss you.

Over the years, Shirt Pocket has been hosted on a number of local Unix-based systems. As I recall, it started out on a Cobalt Qube, then moved to a Power PC Mac mini, then the original "tall" Intel Mac mini, and finally to a "short" Intel mini.

These systems served us well, were easy to manage and, important for the time, meant we were fully "in control" of our site and mail.

There were downsides, though.

Apple has obviously moved away from the whole idea of "macOS Server", leaving a lot of the details of keeping things up to date and relevant to the "modern internet"...and eliminating the original "easy to use and manage" reasons for selecting it in the first place.

But even more than that, since the mini wasn't in a data center, extended power or network outages would take our site down completely—an obviously ridiculous situation. A few years ago, a huge snowstorm took cut power for two weeks, and while I was able to find alternate "hosting" for the site (thanks, Jon!), it wasn't ideal. Not to mention when it would happen when I was on vacation.

Fortunately, while macOS Server is no longer really relevant or useful, in its place are a million cloud-based providers, many from huge vendors like Microsoft, Google and Amazon.

A few years ago, I moved from our own Kerio mail server to one of those: Microsoft's Office 365 Exchange service. That went extremely well, and so, with the threat of winter storms looming, it was time to make the change.

As of Wednesday, all of Shirt Pocket is now "in the cloud", as it were. I'm using SSDNodes, a high quality provider with great support for many Unix variants, Docker, fast bandwidth, etc.

Bruce had used them, successfully, to bring up our Paddle-specific micro services, with no downtime during that transition. A good choice.

So, with that experience under his belt, Bruce guided me through the transition (doing much of the work while I looked over his shoulder, learning how to actually work with Docker, Traefic, etc) and the result?

So, we're now up entirely in the cloud. The Mac mini is off, after probably 10 years of always-on (well, mostly-on) service. One of its RAID drives had died and come back to life, and I had a SuperDuper! backed-up SSD ready to go, but now it won't be needed.

You'll get better service with hopefully no downtime, I don't have to worry about the power going off when I'm away, and thanks to Bruce, I now mostly understand how to use, configure and maintain a Docker-based set of services that, in aggregate, look just like the old web site.

While the dusty old graphics and layout still look like they were created in 2005 (yeah, I know - I have to allocate my limited time carefully), and you probably didn't notice anything had changed, the whole thing works...better!

Can't ask for more than that.

And now, after many years of faithful service, the mini gets a well deserved nap. Well done, little guy. Enjoy the rest.

Happy Thanksgiving, for those who celebrate.

Whenever there's a new, major release, there's an inevitable set of problems that weren't caught during testing (even the long public beta). We've got those mostly taken care of in the new 3.3.1 update, released today.

Most users having problems are running macOS 10.10 or 10.11. Here's the list:

• 10.10 users had an immediate failure with a dyld error due to a release build mess-up on our side. In addition, on some systems, there was a digital signature validation issue. We've corrected both in this update.
• Users with unusual characters in their drive names (eg "ø") were getting errors on all OS versions. This should be fixed.
• There was a problem with drive names with quote marks in them (eg "Fred's MacBook Pro 17" Backup"). Fixed.
• Read-only images would sometimes generate a symbol substitution error. Not any more.

There's one remaining issue for 10.10 and 10.11 users: Erase, then copy backups are failing due to some unexpected "volume transformation" events that are occurring. When we validate the result, we're being quite cautious, and we're not seeing what we expect, so we fail the copy.

That isn't yet fixed, because it's going to take some very careful investigation and, well, Thanksgiving. The workaround is to use Smart Update. If you need to erase, you can erase with Disk Utility, and then run Smart Update.

If you are unregistered, need to run a backup on macOS 10.10 or 10.11, and you're running into this issue, you can Download v3.2.5 here.

The update is available through the normal built-in updater, or you can...

Download v3.3.1

OK, back to celebrating with the family. Enjoy, everyone.

I'm happy to announce the release of v3.3 of SuperDuper, our fully Catalina-compatible version: happier, perhaps, then even you are in reading the news. It's available via the normal update mechanism, or by downloading it from the web site.

Everyone with a Beta version installed will get an update notice for this version as well.

Note: after SuperDuper! 3.3 is installed, if you keep getting prompted for Full Disk Access, you need to restart your Mac. Welcome to Windows 95!

Roads, Long: See Island Destinations

The process of getting to a final version took longer than I hoped it would, but for good reason: once again, Apple has made some pretty radical changes to the way things are stored on your drive, and we didn't want to release a final version—a version that would be installed by "everyone"—without getting extensive test coverage in "real world" situations...that is, setups other than our own test setups and real-world "developer" systems.

Testing has gone really well, and the broader coverage has provided us with some cases that we wouldn't have normally seen: broken systems, leftover volumes from bad OS updates, generally weird behavior with early Catalina releases.

So, thanks to everyone who participated in the public beta: your reports, feedback and encouragement helped create a better final release of v3.3.

What's Changed?

The whole idea of the new version is, if we did our job right (and I think we did), things should just work the way you expect them to. You'll notice a bunch of little things here and there that clarify things, or parts that work more smoothly than before. But overall, it should feel like SuperDuper always has.

But despite that, SuperDuper is doing a lot more things. Catalina has made some radical changes to the way your files are stored, and it's important to know what's going on "behind the scenes".

You may notice, during your first copy, that there's a new step in the Status View: Copy System Files, which will sometimes precede Copy Files. Why is that?

Catalina divides your drive into two volumes (which is what we've been working all spring, summer and fall to support properly). A read-only "system" volume, and a read/write "data" volume.

Things you are allowed to write to, in general, are on the Data volume. Things you can't (the OS, Apple's applications) are on the System volume.

Before Catalina, I often told users that they didn't "own" most of their drive: the vast majority of it was owned by Apple, or rather macOS. You only really "own" your Home folder, in /Users, and the applications you install. (Yes, yes, I know about /usr/local, etc, but work with me here.)

Catalina now formalizes that concept beyond just Unix permissions and SIP-protected locations. The stuff you "own" is now on the Data volume. The System volume is off-limits. For good.

Both macOS and SuperDuper "hide" the details of this from you by tying the two volumes together into a single "Volume Group". You actually start up from the System volume, and the Data volume is connected to it using a new file system feature called "firmlinks".

The details here are unimportant to the user, but the reason you'll occasionally see strange things (like, when you eject a backup drive in Finder, it may tell you there's more than one volume), or extra steps in our Status view, are because of this new setup.

So, your first copy will convert your destination into these two volumes and copy each source to its corresponding destination. We then do what's needed to "tie them together" using firmlinks so they work as one.

Note, though, there's one big benefit: subsequent Smart Updates will not copy the system volume if it's already up to date, which speeds things up quite a bit!

Forced March

Changes like those introduced in the last few OS versions have been frustrating, because they've meant we've had to spend most of our engineering time simply maintaining compatibility with the changes Apple is forcing on all of us.

Not only has the last three years introduced a new file system (APFS), it tightened "security" in a way that broke automation, and then, as I explained above, radically changed the way system and data files are stored on a drive, splitting them into multiple volumes...while hiding those facts from the user.

Until the curtain parts, and they suddenly don't understand what's going on.

Each one of these changes required very significant investigation into under-to-un-documented parts of the system, months-long checks of assumptions, followed by writing a ton of new code, and then extensive testing of all that stuff on both current and older OS versions to ensure we've maintained compatibility with the platforms we support.

And that takes time. Time we'd rather spend implementing new things.

Yes, it's our job. And as long as Apple keeps yanking the rug out from under applications like ours, we have to continue to move, as fast as we can, to wherever the new floor covering lands.

But doing that kind of thing every year is bad for everyone. It's bad for developers, like us, because it means we're constantly chasing the ball. It's bad for users, like us, because it breaks their software (this year more than most), and gives them a less stable system that doesn't let them get their work done, or requires them to replace working software with new software: software that they then have to learn all over again.

On top of that, there are background murmurs suggesting another platform shift, to ARM processors. Because it might give us slightly faster Macs, with slightly better battery life.

So more hoops. More time spent doing things other than the things we really want to do.

This isn't an argument against progress. We all benefit from forward momentum: real security and functionality improvements are valuable and welcomed.

But constantly changing a platform in ways that break compatibility, or confuse users, or make their data less safe and secure stops that progress. We all spend the year between the releases trying to catch up with the latest change, without actually being able to spend time reaping the purported benefits.

It can be frustrating. But it's the way it is, right now.

It doesn't have to be. We can make progress without jerking everyone around. Security can be improved without subjecting users to confusing prompts and bad UX. New APIs can be built without forcing users to throw away software and hardware they've been depending on for years.

Respecting users must also recognize the fact that they have a significant investment in their software and peripherals, let alone their Mac. Just because someone can't afford to purchase new hardware, or needs to maintain compatibility with an old scanner or printer, or really likes what they have...that shouldn't be reason enough to leave them behind. They shouldn't have to add their perfectly good gear to a landfill.

Climbing Off the Soapbox

With that off my chest, I hope you all enjoy the new version of SuperDuper. We've put a lot of love and care into it, and I think you'll find it works exactly as you'd hope it would.

Have at it, and thanks for being a SuperDuper user!

Not much to say about this build: we've fixed a bunch of things, basically. The vast majority of users had success with the last beta. But:

• Some had schedules that would fail right at the start, with a collapsed window. Fixed!
• Copies to HFS+ volumes would fail to copy recovery in some cases. Also fixed!
• Eject would sometimes not eject both volumes of a volume group. Can you believe it? Fixed!
• Some systems have a broken installation of Perl and xpath, which we detect. But that detection was broken. We fixed it!
• If you had just set up a bunch of scheduled copies to locked volumes, we were only preauthorizing against the keychain for the first one. I bet you can guess: fixed!
• Various other things. Fixed!
• Some people were impatient and didn't realize HFS+ to APFS conversion might take a while! We now tell them to get a tasty beverage! Which isn't so much "fixed!" as "indicated to them that patience is a virtue".

Yeah, like I said. Things are in good shape. We're busily making sure all the various panel gaps are even.

Keep reporting the things you find...until then, download away, install and enjoy.

Download SuperDuper! 3.3 B6 (v119.7)

I don't talk about it much on the blog, but I get an enormous amount of support email. The quantity can be overwhelming at times, and without some automation, it would be nearly impossible for me to do it alone.

Which I do. Every support reply, in all the years we've been here, has been written by me. (Yes, even the Dog's auto response. Sorry, he can't really type. But wouldn't it be sweet if he could?)

No Downtime

The basic problem with being a small "indie" shop is quite simple: you get no time off. I've literally worked every single day since starting Shirt Pocket, without fail, to ensure users get the help they request in their time of need. It's just part of the deal.

But, every so often you need a break, and to try to enforce the "less work" idea there, I try to bring something other than a Mac...since that means I can't do development, but can respond to users as needed.

Automation: It's Not Just for Print Bureaus

Many of the support requests are sent through the "Send to Shirt Pocket" button in the log window, especially when people want help determining what part of their hardware is failing. That submission includes a ZIP file of the settings involved in the backup, which contains the log, some supplementary diagnostic information, and any SuperDuper! crash logs that might have occurred.

One of the first things I did to automate my workflow, beyond some generally useful boilerplate, was to use Noodlesoft's Hazel to detect when I download the support ZIP from our tracking system.

When Hazel sees that happen, it automatically unzips the package, navigates through its content, pulls the most recent log and diagnostic information, and presents them to me so I can review them.

It's a pretty useful combination of Hazel's automation and a basic shell script, and I've used this setup for years. It's saved countless hours of tedium...something all automation should do.

Seriously if you have a repetitive task, take the time to automate it—you'll be happy you did.

Two Years Ago

So, a couple of winters ago, in order to fulfill the "try not to work a lot on vacation" pledge, I took a cellular connected iPad Pro along as my "travel computer". While it was plenty fast enough to do what I needed to do, the process of dealing with these support events was convoluted, at best.

I had to use a combination of applications to achieve my goal, and when that become tiresome (so much dragging and tapping and clicking), I couldn't figure out how to automate it with Workflow.

Now, I'm not inexperienced with this stuff: I've been writing software since something like 1975. But no matter what I tried, Workflow just couldn't accomplish what I wanted to do. Which made the iPad Pro impractical as my travel computer: I just couldn't work efficiently on it.

(I know a lot of people can accomplish a lot on an iPad. But, this was just not possible.)

One Year Ago

So, the next year, I decided to purchase a Surface Go with LTE. It's not a fast computer, but it's small and capable, and cheap: much cheaper than the iPad Pro was.

And, by using the Windows Subsystem for Linux, in combination with PowerShell, I was able to easily automate the same thing I was doing with Hazel on macOS.

I was rather surprised how quickly it came together, with execution flow passing trivially from Windows-native to Unix-native and back to Windows-native.

This made traveling with the Surface Go quite nice! Not only does the Surface Go have a good keyboard, I had no significant issues during the two vacations I took with that setup, plus it was small and light.

This Year

But I'm not always out-and-about with a laptop, and sometimes support requests come in when I've just got a phone.

With iOS, I was back to the same issues that iPadOS had: there was no good way to automate the workflow. Even with iOS/iPadOS 13, it could not be done.

In fact, iOS 13 made things worse: even the rudimentary process I'd used up until iOS 12 was made even more convoluted, with multiple steps going from a Download from the web page into Files, and then into Documents, and then unzipping, and then drilling down, and then scrolling, opening, etc.

On a iPhone, it's even worse.

Greenish Grass

Frustrated by this, a few weeks ago I purchased a Pixel 4, to see how things had progressed on the Android front.

I hadn't used an Android phone since the Galaxy S9, and Google continues to move the platform forward.

As I said in a "epic" review thread

So, I spent a few days exclusively using a Pixel 4 and had some thoughts.

There will be a lot of tweets (12).

First, and perhaps obviously, Android is getting better. And the Pixel 4 has to be considered as “pure Android” as you’re going to get. 1/12

— Dave Nanian (@dnanian) October 27, 2019

iOS and Android applications are kind of converging on a similar design and operational language. There are differences, but in general, it's pretty easy to switch back and forth, save for things that are intentionally hard (yes, Apple, you've built very tall walls around this lovely garden).

And while Android's security has, in general, improved, they haven't removed the ability to do some pretty cool things.

And one of those cool things was to actually bring up my automatic support workflow.

Mischief, Managed

Now, given you can get a small Linux terminal for Android, I probably could have done it the same way as with Windows, with a "monitoring" process that then called a shell script that did the other stuff just like before.

But, instead, I decided to try using Automate, a neat little semi-visual automation environment, to do it. And within about two hours, including the time needed to learn Automate, it was up and running.

I'm not saying the result isn't nerdy, but it was doable! And that made it entirely practical to respond to people when I'm using a phone, even when they send in a more complex case.

Will that be enough to encourage me to stay on Android? I don't know. But, combined with the other iOS 13 annoyances (apps that get killed when they shouldn't, constant location prompts even after you've said "Allow Always", general instability...so many things), it's been a comparatively pleasant experience...Android has come a long way, even in the last two years.

It's really nice to have alternatives. Maybe I'll just travel with a phone this year!

Beta updates are rolling along. We've incorporated a number of fixes for additional edge cases, polished some behavior, worked around some OS issues...the usual.

We've had a number of people ask whether it's "safe" to run these betas.

Now, barring the occasional, rapidly-fixed "goof" (eg the bless error we fixed in Beta 4), every beta version we release is basically "production ready". They're "betas" because we're dealing with a new OS that's changing rapidly, and we want to ensure we've covered all the cases we didn't think of before we do a general release.

Prepare for Boredom

For example, this version has an "automount" fix in it. As I've discussed previously, sdautomatedcopycontroller is used to run "automatic" copies. Those include scheduled copies, copies users can run from the command line or their own tools, etc.

sdautomatedcopycontroller (every time I type that I think why didn't I make this name shorter) handles the detail of automatically mounting source and destination volumes, and setting them up to eject when the copy is complete.

With volume groups, though, there are two potential volumes to mount...but keychain passwords might be under either the Data volume or the System volume, depending on what the user does. Previously, we were only looking under the System volume name...now we check both.

Not Boring Enough? Try This!

Another example: on some user systems, certain macOS command line tools that are written in Cocoa or Swift would output loader warnings as errors (on stderr), and so we'd think they failed. We now handle more of those cases, if the tool otherwise is successful.

Short and Sweet

So, yeah. These are little fixes that cover cases that come up in broader testing...and are examples of why these betas are "safe". These fixes just aren't things most people are going to notice.

Again, thanks for your feedback: your reports are making SuperDuper! 3.3 better for everyone!

Download SuperDuper! 3.3 Beta 5 (v119.6)

Yeah, well, sometimes a change to the install process that shouldn't have any effect on anything else...does.

Beta 3 gave an error when using bless to bless the drive. The underlying issue was code we have that needs to ignore certain types of output on stderr that aren't actually errors. (For example, the EFI fix that I describe in this blog post produces an error that isn't actually an error–we know it's going to fail in a specific way).

The unexpected part is that just before the beta, we made a change to the installer to try to improve our workaround for systems that required rebooting post-install to make Full Disk Access work. After we made the change, we didn't re-run the full suite of tests because we (incorrectly) thought the change was isolated to the install process.

However, it was made in a runtime element that was shared with the way we executed bless. We didn't take that into account, and should have re-run all the tests.

The backups themselves are fine (and are even bootable, even though there's an error issued). Beta 4 fixes the false error indication: once installed, bless will not give a false failure.

Sorry about that. Beta. >sigh<

Download SuperDuper! 3.3 B4 (v119.5)

Third time isn't a charm; you'll note a bless error at the end of copies. We'll have a new beta Sunday that fixes this dumb goof.

(For the impatient - scroll down and you can download Beta 3 at the bottom of this post. Note that existing betas will, as always, automatically update.)

Well, we've reached the critical Beta 3 phase, where we think we've dealt with all the issues reported, save for one or two stragglers we're trying to figure out.

Mostly Beef, Some Filler

Overall, things are pretty solid. We tightened up some of the image and encryption handling, which affected some users (and provided interim betas to those who got in touch to confirm our fixes). There were some problems with volumes being ejected at the end of scheduled copies under Catalina that were fixed, as well as some related issues with auto mount (also only under Catalina).

Pretty minor stuff.

Mystery Meat

We've got a few users whose systems are in a bizarre state where the loader is outputting

dyld: warning, LC_RPATH @executable_path/Frameworks in /System/Library/PrivateFrameworks/TelephonyUtilities.framework/Versions/A/TelephonyUtilities being ignored in restricted program because of @executable_path (Codesign main executable with Library Validation to allow @ paths)

when we run certain system command-line tools. It's happening to a very small number of users, but for those users, neither rebuilding their dynamic libraries nor reinstalling their version of Catalina (current public release) fixes the problem.

We were able to reproduce the problem, once, with a non-admin user under the current public release, and it fixed itself after installation of the current developer preview, but given we can't reproduce it again, even with the public release, we're not entirely sure what weird system thing is going on here: we just know it's not good.

If this is happening to you, get in touch.

Certified Fresh

We also added a diagnostic that detects a rare situation where a user's system has broken scripting tools (like a bad Perl install), which can cause problems. Again, rare stuff.

Piping Hot Chinchilladas

But, apart from those relatively minor kinds of things, Beta 2 worked well. And Beta 3 should work even better.

Once again, thanks for your patience and support. Enjoy!

Download SuperDuper! 3.3 Beta 3 (v119.4)