Bruce and I had spent a long time designing, writing and testing SuperDuper!—as you might expect, backup programs need to be extremely reliable, and we worked hard to make that happen. And it’s trickier than you might think—there’s a lot of obscure stuff in HFS+ and its metadata, and getting all of the information copied properly is quite a job.

We released SuperDuper! v1.0 in January 2004. This is obviously a really tense time in the development process. No matter how much testing you do, no matter how thoroughly you plan your coverage, no matter how many Beta testers you’ve got, no matter how good they are, there’s always something that shows up once you release the software into the wild. Always. (Just ask Apple!)

Fortunately, we’d done an excellent job: early feedback showed we had no reports of any significant nature—most problems were quite minor and easily handled.

But there’s always something. Always.

In our case, over the first few months, we’d get an occasional report of a user who couldn’t access their own files when they’d booted from (or restored) a backup. Sometimes, they’d try to log in and end up with an empty Home folder. Sometimes, certain applications wouldn’t run.

Investigation showed that, in these cases, the files on the cloned volume were owned by root, even though—when we looked at the original files on the system—they were owned by the user.

Here’s a good opportunity for a short digression.

Unlike OS9, OSX’s file system has a concept of ownership and permissions. While this can sometimes make your system feel like you don’t own it any more, it’s a fundamental part of the underlying security model: something that we all benefit from, even when the details can feel arcane, infuriating or random.

All files on the system are owned by someone, and usually your own files are owned by you. System files are usually owned by the system, also known as root. Owners are also members of groups with names like staff and wheel, and a file has both an owner and a group.

In addition to a file having a concept of an owning user and group, it also has a set of permissions. There are three separate permissions for each file: owner permissions, group permissions and other permissions. The owner of a file can give each of these permission to read or write the files (in addition to other things, but you get the idea).

By default, files you create are readable and writable by you and no one else. And the system stuff—things you shouldn’t be able to touch—are usually owned by root and can’t be modified by a regular user unless they “authorize”: which is why you need to enter your password when you run a system update… or SuperDuper! itself.

With that in mind: back to the main post!

Since the files were owned by root, it meant the user couldn’t write (or sometimes read) them—which caused all sorts of trouble.

It was easy to “fix”—we were able to give the user instructions on how to use Terminal to reset the ownership of the files back to their own user—but the real question was: how the heck was this happening?

This went on for months. We tested and tested and tested and were completely unable to duplicate the problem, yet it still showed up every so often. We were able to help the few users who were affected by the problem, but we couldn’t find anything wrong with SuperDuper! itself—it passed all our tests, and was working on the vast majority of systems out there.

Finally, it happened to a user who was willing to spend a relatively long time on the phone, trying various things. In the middle of the call, as I had the user try various things to try to diagnose the issue, I hit on something that didn’t make sense. When the user looked at their files, they were—indeed—owned by them. But, if we looked at those same files as a different user, the same files were owned by that other user!

Needless to say, this was more than a bit of a shock. How could a file be owned by different people, depending on who was looking at it?

We didn’t yet know what the deal was, but I coined a term for it: floating ownership. And, I had a theory about what was going on.

Because SuperDuper! needs to get access to every file on the system, it runs as root: the only user with that ability. And if these weird files had their owner floating, it meant that when SuperDuper! looked at them, they’d be owned by… root. So when we copied over the ownership, instead of copying the file as the user saw it, we’d copy it the way that we saw it: as root!

Eureka!

It took some time to come up with a solution that worked, since OSX tries really, really hard to hide this whole floating owner thing from programs: all programs. Even Apple’s own tools—like ditto—don’t handle floating owners properly when copying files! But, after some thinking and testing I came up with an idea that worked: something called on-the-fly verify.

Idea in hand, Bruce implemented it, and we tested and released it in relatively short order… and then held our breath.

It was clear soon after that we’d licked the problem, with an additional benefit: we started getting many more reports indicating that SuperDuper! could successfully create a bootable clone where all other tools failed—a great, and unexpected, benefit of on-the-fly verify.