Shirt Pocket Watch

Wow, yesterday was a hectic day.

I’d had a new release of netTunes in process for some time, and had completed the final testing a few weeks ago. Since the main focus was Tiger compatibility, it seemed appropriate to release closer to Tiger’s release (and I didn’t want to leak any Tiger information by accident), and yesterday (Thursday) was the day I picked to put it out there.

Well, actually, I picked Wednesday, but Wednesday somehow got filled with other stuff, and it just didn’t happen. So, Thursday.

The release process is always kind of the same: I’ve already packaged up the software itself, but that’s only the engineering side. Then, the marketing side has to take over, and you have to:

• Write and send out a press release. Press releases are weird—they always have that amusing part where there’s a quote from someone in the company—typically the CEO—saying something like “I think this is the best work we’ve ever done.” Usually, Marketing just makes something up, runs it by the CEO (or whoever) and gets their OK along with a few tweaks. (Disillusioning, I know, but there it is.)

  Of course, in a small company like mine, I’m writing the press release. And, I’m quoting myself. In the 3rd person.

  Weird stuff. But this is the best netTunes release we’ve ever done.

  (Big apologies to all of you in the press who have to read my lousy press releases.)

• Get the Shirt Pocket web site updated and ready to go.
• Get the various VersionTracker and MacUpdate updates ready to post.
• Update Apple’s software site
• Various other things, email, etc.

Well, maybe that doesn’t sound like a lot. But it feels like it when it’s going on. And then you post the thing, and hope for the best!

Anyway, I’ve done this (and variations thereof) a lot of times over the past 22 years, and it’s always a bit nerve wracking: you just never get over the “release jitters”.

I was talking to Jonas Salling about this the other day, as he was releasing his own update (and, the three of you who haven’t gone out and bought the beautifully done Salling Clicker, please do so now), and he has the same nervousness—no matter how well prepared we are, it always seems like there’s a disaster waiting just around the corner.

Yesterday was especially nerve wracking for me because, immediately after posting the netTunes update, I had to do a whole bunch of errands out of the office. Which meant if something went wrong there was no way to fix it quickly—I nervously checked my email on my phone all day, waiting for the disaster to strike.

It never did. It went fine. It pretty much always does. So why are we always so nervous?

Maybe it’s because, as a small developer, you always feel like you’re one step away from the mistake that’ll kill your company. There’s not a lot of “wiggle room” for the small developer: we can’t absorb a Windows ME or Microsoft Bob or “iTunes Update that deletes your whole drive”.

Basically, we pretty much have to execute perfectly all the time. One mistake and we’ll look foolish and unprofessional: something the vast majority of us are not, but once that impression gets out there, the battle’s lost…

So, what should you expect from this blog?

My plan is to write about a relatively eclectic set of topics. I’ve sketched out my first 30 posts or so, and they seem to be falling into the following broad catagories:

• News about Shirt Pocket
• Software design and development
• Computing from a user’s perspective
• Occasional stories about small company development, marketing and support
• Consumer electronics and usability
• Dogs and Degenerative Myelopathy
• Gadgets I like
• Movies & Music
• The occasional random rant

No huge surprises there. I’m sure other things will creep in now and again, but for now, there you go.

Backing up has never been so easy:

SuperDuper is an incredibly useful piece of software. It doesn’t hurt that it’s virtually idiot-proof, with excellent documention, superb support and a visually attractive interface. I have been using it for several months and it didn’t take long for it to become one of my most valuable utilities. I’ve got a hunch you’ll feel the same way.

(Via The Unofficial Apple Weblog. Thanks, Laurie!)

To follow-up on yesterday’s post, the inevitable question is: how do files get like this in the first place?

As far as I can figure out, this floating owner “feature” is part of OSX’s OS9 compatibility, and also has to do with the “Ignore ownership on this volume” checkbox you’ll find in the Get Info window for non-boot volumes.

It’s important to remember that OS9 has no concept of ownership whatsoever. So, when a file is created by an OS9 application, running in OS9 itself, it has to choose a user that’s going to own the file.

But what user should it choose? As I said, there’s no real concept of ownership in OS9. And if you randomly just pick a user—say, the first one—there’s no guarantee that you only have one user on your machine… and no way to ensure that, once you’re in OSX, the correct user owns the file.

There’s a bit more complexity to this whole “user” thing. While a user has a name (the “short user name” you select when you first create the account), the system doesn’t really assign the “name” as the owner of the file. Instead, it uses a User ID, which is a number.

On OSX, the first user account created is given the User ID 501. The next account created is 502, etc. These numbers are what are associated with the files on your disk, not the short user name, which is more for your convenience than anything else.

This can create some confusing situations, though, when you bring a disk to another computer, and turn “Ownership” on. If it’s another user’s computer, and their account was the first created (most are), that means their User ID is the same as yours. So, your files will look like they’re owned by them!

Similarly, if you have more than one computer in your office (or family), and the accounts weren’t created in the same order, your account may be “501” on one machine, and “505” on another! (Here, to avoid this problem, I make a habit of creating all my accounts in the same order on all machines.)

No doubt this is why Apple allows you to turn ownership off when you attach a FireWire drive to your machine… this causes all the file owners to float!

So, Apple compromised. Since they couldn’t randomly pick a user, and they didn’t want to force permissions on OS9—which would cause serious compatibility problems—they did the next best thing. They made these OS9 files owned by “everyone”, all at once. In other words, they made the owner float.

When the owner’s floating, it looks like it’s owned by the user looking at it. So, if the file is saved in OS9 (or stored on a FireWire disk with “Ignore ownership on this volume” checked), it’ll be owned by whatever user looks at it.

Even if that user is a Backup program running as root, like SuperDuper!. Which brings us full circle.

Bruce and I had spent a long time designing, writing and testing SuperDuper!—as you might expect, backup programs need to be extremely reliable, and we worked hard to make that happen. And it’s trickier than you might think—there’s a lot of obscure stuff in HFS+ and its metadata, and getting all of the information copied properly is quite a job.

We released SuperDuper! v1.0 in January 2004. This is obviously a really tense time in the development process. No matter how much testing you do, no matter how thoroughly you plan your coverage, no matter how many Beta testers you’ve got, no matter how good they are, there’s always something that shows up once you release the software into the wild. Always. (Just ask Apple!)

Fortunately, we’d done an excellent job: early feedback showed we had no reports of any significant nature—most problems were quite minor and easily handled.

But there’s always something. Always.

In our case, over the first few months, we’d get an occasional report of a user who couldn’t access their own files when they’d booted from (or restored) a backup. Sometimes, they’d try to log in and end up with an empty Home folder. Sometimes, certain applications wouldn’t run.

Investigation showed that, in these cases, the files on the cloned volume were owned by root, even though—when we looked at the original files on the system—they were owned by the user.

Here’s a good opportunity for a short digression.

Unlike OS9, OSX’s file system has a concept of ownership and permissions. While this can sometimes make your system feel like you don’t own it any more, it’s a fundamental part of the underlying security model: something that we all benefit from, even when the details can feel arcane, infuriating or random.

All files on the system are owned by someone, and usually your own files are owned by you. System files are usually owned by the system, also known as root. Owners are also members of groups with names like staff and wheel, and a file has both an owner and a group.

In addition to a file having a concept of an owning user and group, it also has a set of permissions. There are three separate permissions for each file: owner permissions, group permissions and other permissions. The owner of a file can give each of these permission to read or write the files (in addition to other things, but you get the idea).

By default, files you create are readable and writable by you and no one else. And the system stuff—things you shouldn’t be able to touch—are usually owned by root and can’t be modified by a regular user unless they “authorize”: which is why you need to enter your password when you run a system update… or SuperDuper! itself.

With that in mind: back to the main post!

Since the files were owned by root, it meant the user couldn’t write (or sometimes read) them—which caused all sorts of trouble.

It was easy to “fix”—we were able to give the user instructions on how to use Terminal to reset the ownership of the files back to their own user—but the real question was: how the heck was this happening?

This went on for months. We tested and tested and tested and were completely unable to duplicate the problem, yet it still showed up every so often. We were able to help the few users who were affected by the problem, but we couldn’t find anything wrong with SuperDuper! itself—it passed all our tests, and was working on the vast majority of systems out there.

Finally, it happened to a user who was willing to spend a relatively long time on the phone, trying various things. In the middle of the call, as I had the user try various things to try to diagnose the issue, I hit on something that didn’t make sense. When the user looked at their files, they were—indeed—owned by them. But, if we looked at those same files as a different user, the same files were owned by that other user!

Needless to say, this was more than a bit of a shock. How could a file be owned by different people, depending on who was looking at it?

We didn’t yet know what the deal was, but I coined a term for it: floating ownership. And, I had a theory about what was going on.

Because SuperDuper! needs to get access to every file on the system, it runs as root: the only user with that ability. And if these weird files had their owner floating, it meant that when SuperDuper! looked at them, they’d be owned by… root. So when we copied over the ownership, instead of copying the file as the user saw it, we’d copy it the way that we saw it: as root!

Eureka!

It took some time to come up with a solution that worked, since OSX tries really, really hard to hide this whole floating owner thing from programs: all programs. Even Apple’s own tools—like ditto—don’t handle floating owners properly when copying files! But, after some thinking and testing I came up with an idea that worked: something called on-the-fly verify.

Idea in hand, Bruce implemented it, and we tested and released it in relatively short order… and then held our breath.

It was clear soon after that we’d licked the problem, with an additional benefit: we started getting many more reports indicating that SuperDuper! could successfully create a bootable clone where all other tools failed—a great, and unexpected, benefit of on-the-fly verify.

As you’d expect, people are really, really excited about Tiger and we’re getting a lot of questions about compatibility. So, here’s the deal:

• The current release of SuperDuper!, v1.5.5, is compatible with Tiger, so update if you haven’t already. There are some caveats if you’re trying to install Tiger on a safety clone, keeping Panther on the original volume: I’ve summarized the issues on the Shirt Pocket forums, so check it out.
• We’ll be releasing We released an update to netTunes next week Thursday that works around some new Tiger behavior that caused us to sometimes display a blank or strange looking window when the server was running on a Tiger machine. We’ve also improved performance and added a very cool feature that works side-by-side with iChat to automatically update your status with the currently playing track, even when it’s on your remote, netTunes server. Cool!
• launchTunes works great as-is, so there’s no need to update for Tiger.

There you go!

So, I’ve been avoiding blogging. More than that: I’ve been avoiding making eye contact with blogging.

But it keeps jumping up and down, waving its arms frantically, desperately trying to get my attention, insisting that it won’t be so bad.

I’m not so sure.

I mean, I’ve had some sort of online presence for a long time. We even announced our first product, BRIEF, on Usenet way back when. I was on FidoNet, Compuserve, BIX, Usenet, lots of forums—including my own at Shirt Pocket. So why is getting started here so difficult?

I think it’s the “What’s the sound of one blogger yapping?” issue. There’s something about this that feels less like a conversation, and more like a lecture. A post gets written. That post gets read. Sometimes, there’s a comment, trackback, link… but it feels more newspaper than café.

But—maybe that’s because of the way I’ve been using blogs. I’ve been reading, but not writing. I’m the lurker on the forum, the guy alone at the café sipping a coffee, reading the paper and listening—but never joining in.

Well, no longer.

This place is abuzz with ideas, and there seems to be lots of room at the table.

Mind if I join you?