Shirt Pocket Watch

I know it feels like that from the outside. But from the inside, we've been putting out quite a few external beta releases (11 at last count), incorporating feedback from our testers, and carefully observing what users with 2.1.4—who have tried to use it with Leopard—have tried to do.

Before I start, though, please recognize that I'm describing things we noticed back in October. So, this is not causing additional delay—these are some of the things that contributed to the delay in the first place.

Paying Attention to Users

We've always observed our users very carefully. And, after Leopard's release, we noticed one big thing: users with 2.1.4 and Leopard have been trying to store their SuperDuper! backup to the same volume they've been using for Time Machine. But there's a significant problem with that: namely, since the Time Machine backup isn't on the source volume, Smart Update is going to try to delete it.

Now, Time Machine "protects" its own backups through a number of complex mechanisms, so no damage is done—except the backup fails.

Choices

In some ways, this is quite similar to the "storing other files alongside a backup" case that has come up before (and has an entire section of the User's Guide dedicated to it)—basically, we could request that users partition the destination drive, put their Time Machine backups on one volume, and their SuperDuper! backup on another.

But, even with Leopard's new "live partitioning" capability, that's a pain, isn't obvious, requires a support round trip, doesn't meet user expectations, and isn't transparent.

It also wasn't something we took into consideration during our original compatibility design. We focused on the various new capabilities of the OS, ensuring that we worked well, but since we were used to partitions—and knew that Time Machine liked as much disk space as possible—we gave it a lot of space to work with and left it alone.

So, once we started seeing this—which was after Leopard's release, of course—we had to make a choice. Push responsibility to the user to do the right thing, or anticipate and handle it inside SuperDuper! so users wouldn't be immediately frustrated.

But I Don't Want to Use Time Machine!

We certainly understood—and understand—that some users don't want to use Time Machine. But the problem is, a large proportion of Leopard's users do, and we have to serve the needs of both groups. We can't come out with a "Leopard-compatible" release that doesn't work properly with Time Machine.

And we couldn't really release a "quick update" that just copied the basics first, and then an update that handled this case right. Had we done that, we would have still needed to take a full external cycle to test the copy engine changes we needed to make, and then run the same extensive cycle again when we made the low-level changes to handle this case.

So, we bit the bullet and made the necessary changes to handle the case properly, the way users would expect: it just works.

So, if you Smart Update a volume that you're using for Time Machine, it works just fine. Your Time Machine backups, and your bootable SuperDuper! backup, and stored on the same volume, side-by-side, without interfering with each other, with no need to partition or do anything special (other than use Smart Update, of course).

On top of that, you can still copy that volume—that is, a volume that has both Time Machine backups and a bootable SuperDuper! backup—to a 3rd volume to make a second backup, and everything there works great, too.

This is technically more complicated that you might expect, but all of that is under the covers. On the surface, there are no visible changes.

Side Benefits

So, this is the main reason why we were able to improve the handling of Spotlight and preserve icons as I discussed in the last post: we had to handle the Time Machine case, but it benefits all users by improving the way Spotlight (which everyone basically uses) and icon preservation (which a small, but vocal group use) works.

Why Focus on Time Machine?

Which, of course, is an extension of But I Don't Want to Use Time Machine!, above.

The fact is, Apple considers Time Machine to be the most important, most visible new feature in Leopard. They've put a lot of marketing muscle behind its rollout, and its tendrils are in everything, from the default desktop to Quick Look.

As I've discussed before, it's a big, good change, and it benefits all users, even those who use SuperDuper! We're truly two complementary technologies, and we need to ensure that SuperDuper! works perfectly with it to ensure that SuperDuper! itself remains a viable product.

And if the product remains viable, we'll be able to continue improving it for everyone—something we all want!

It's Not All Time Machine, Though—Really!

Another interesting thing we found as we got reports from new users, that we never encountered internally: a very old bug in the Migration Assistant (described in Slowly going insane while waiting for a fix back in 2005) is back, with a vengeance. But there's a new variation: rather than having a missing carriage return, there are now also blank lines in the file, which result in the same failure.

So, we've had to implement a second workaround to fix this problem, too.

How Much Longer?

Anyway, enough of that. At this point, we're basically locked down. We have a few UI tweaks to complete, and we should have a final release out to everyone within a few weeks.

You'll notice this is the first timeframe I've provided other than "soon"... and that means I'm quite confident it'll be in your hands shortly. So, once again thanks for your patience, and I really think you're going to be pleased with the update.

Another week, another Leopard Status Update. Please, hold your cheers and applause until the end.

Two parts

Although most people think about SuperDuper! as a single application, it's actually comprised of a number of parts. The two main ones are the UI application (which you interact with) and the Copy Engine, which actually performs the copy.

When changes are made to the UI component, testing takes significantly less time, because it's much easier to ensure that those changes are working properly, and improper operation is relatively low risk: if a button doesn't dim properly in some conditions, it's an easy bug to work around for an end user.

Changes to the copy engine, though, are extremely high risk. Errors made in copying files affect the backup itself, and often occur silently: everything looks like it's working, but, not so much.

Testing the tests

This is one of the reasons that this update is taking so long to release. Due to the nature of the changes in Leopard, we've had to make significant changes to the engine. And while we, of course, have extensive test suites internally, those tests are "necessary" but not "sufficient".

Final bits

As I've mentioned before, we did, of course, have betas of Leopard, and we were working with them. But a beta is just a beta: changes are made up until the shipping version, and that was the case with Leopard as well.

But, there's another consideration, too: we can't do our "big" external test cycles until the "public" has the real bits. Yes, we can roll to testers who have the same beta we have, but when you're making the kinds of changes we had to make this release, it's important to roll in new testers who have different file setups and different patterns of copying.

(Tester fatigue is an issue that all developers are constantly struggling with: it's not so much that a tester is "tired", but that they fall into a pattern of use, and rarely venture outside it.)

So, our internal tests couldn't be run until we received the final bits (we never release test versions to our testers that are risky, to ensure they can really use the test release as a production copy), and our big external test couldn't start until after that.

That testing -- of which there have been multiple rounds -- has been going very nicely indeed.

More progress

So, last week, we locked down the "cloner changes": given feedback from the testers and our external tests, and running against 10.5 and 10.5.1, we're pretty confident that the engine's working properly.

That's a good thing.

We've added some new capabilities to the engine, too. We can now preserve the contents of the Spotlight index on the destination (which also preserves its indexing status in a more reliable way) and preserve the "File System Event" database that Leopard uses. More on this in a later post.

While we were doing that, we decided to do one more thing.

Save the icons

We had always felt that the volume icon -- being a file on the source drive like any other -- should be copied to the backup drive, since you'd want to restore the original drive to the same condition it was originally in, and that included the icon.

Well, we had a lot of pushback on that choice (to put it mildly - I think the thread that discusses this issue and the workaround we came up with originally is the longest one on the discussion forums), and we decided that we were wrong. Since you rarely restore, and your backup drive is constantly visible, it was best to preserve the destination icon. Plus, if you restore with SuperDuper!, we'll preserve the icon of the destination for the restore, so it's all good.

The changes we made, above, made this possible. And so, the destination volume icon will now be preserved.

More coming

So, a lot more progress has been made. Testing's going really well. And I'll talk a bit more about some other new stuff later.