One of the interesting things that happens when you do a “live” backup—that is, back up the volume that you’re booted from—is copy files that are currently open.
Normally, this isn’t a huge problem. But, on occasion, things go horribly w0rng. This is a story of one of those things that happens every so often with SuperDuper, and what we’ve done about it. It’s a mite technical, but hopefully interesting.
As you probably know, under OS X files have an owner, a group, and a “mode”. These three things work together to determine who can read, write, execute and do other things to the files on your system.
Less well known are the “flags” that are associated with the files as well. For example, the “immutable” flag—uchg—can be set to prevent a file from being modified, regardless of its “mode”. There are a number of flags that can be associated with files and SuperDuper needs to properly replicate them when copying.
So far so good.
Every so often, though, we get reports of users who have files on their destination volumes that can’t be removed during Smart Update. The error always looks like this:
| 06:02:18 PM | Info | Warning: error clearing immutable flags: 77600100 for: /Volumes/G5/private/tmp/sort9qlI6c
| 06:02:18 PM | Info | Error removing item: /Volumes/G5/private/tmp/sort9qlI6c of type: 100000, Operation not permitted
| 06:02:18 PM | Error | SDCopy: Error deleting /Volumes/G5/private/tmp/sort9qlI6c
| 06:02:18 PM | Error | : Operation not permitted
We found that we needed to remove a file on the destination. But when we tried to clear its immutable flag, we couldn’t. The only solution was to use Erase, then copy which—while it worked—was inconvenient.
We were recently finally able to reproduce the issue in house, and what we found was quite curious.
When a file is being written at the same time we’re backing it up, it’s possible for that file to look “weird” in the file system. When the above happens, it seems that the directory entry isn’t completely updated, and—in this unusual state—some flags are set that result in the above behavior. Here’s the above file’s directory entry, listed with “ls -lo” (which lists flags too):
---s-wS--- 1 nobody 3221220928 sappnd,arch,schg 524237 Dec 31 1969 sort9qlI6c
So, weird stuff. The group is complete garbage, and the sappnd and schg flags are set. And—worse—since the system returned that junk to us, we faithfully replicated it on the copy.
Why worse, you might ask? Because while you can set the sappnd and schg flags, you cannot reset them while booted normally. And, schg is the “system immutable” flag: it prevents the file from being deleted. Once you have this flag set, that file’s going nowhere.
The reason for this is that FreeBSD—the Unix that OS X is built on top of—has a concept of “security levels”. By default, our Macs boot up and run in “Secure level 1”, which prevents anyone from writing in certain folders like /dev/mem and /dev/kmem, even with escalated privileges (such as “root” access). And, in secure level 1, schg and sappnd flags cannot be removed.
And how can you get to securelevel 0 or -1, where they can? You have to boot into single user mode.
(You can learn some more about the various BSD security levels at this page.)
Yikes! What that means is once you’ve set these flags, you’ve basically got to go super-geek to unset them.
It’s clear we can’t ask our users to reboot into single user mode to fix this kind of thing, so we decided that the best thing is to detect the “garbage” case as best we could, and—in that situation—mask out the schg and sappnd flags when copying. The data we’re given by the OS is still wrong, but at least the copy won’t suffer for it, and the user won’t have to erase their backup next time around.
This fix, as well as some other stuff I’ll be talking about over the next few days (including native Intel support), will be coming in v2.1 of SuperDuper—keep an eye out for it!
