A few days ago, we were happy to get a workaround for the bug mentioned in a previous post, so we went about implementing and testing—and things were working quite well.

Unfortunately, we've found a case where the workaround doesn't, um, work. I think I have a way of dealing with it, and so we're going to implement Workaround Part IV: The Reworkening and run it through the wringer.

Again.

These tests are really time consuming, as you might guess: it can take eight hours to rebuild a complex-case volume (so I try to keep one building while another is testing), and then more hours to run the test. I can't begin to tell you how frustrating it is to have a set of simple tests work and then find that the thing we found rearing its ugly head again in another (more complex) scenario that can (and did) happen in "real life".

This is why we don't just implement a fix and release, of course: we want to ensure it works before we toss it out there, and so we smoke test, test internally and then—if internal tests are OK—test externally, every time. We always are hoping for green lights at every stage. Sometimes, 99% of the way there, you get red. Then, the failure needs to be investigated, understood, reproduced...

Anyway, we'll leave that between me and my ulcer. If my new idea works, this will hopefully only delay things a few days, which will be plenty of time to fill the comments with complaints of our laziness, incompetence, lack of communication and general suckitude. Have at it!

Note: I've had to turn comments off for a little while because the subscriptions to comments are overloading my outbound servers (every time someone posts something, it's sent to the 200 people above their post), causing delays for regular support mail, etc. You can still review existing comments by clicking the title of this post. I'll turn them back on after the backlog clears. Sorry for any inconvenience: it's not that I don't want to hear what you want to say...

Last post, I mentioned that we were bitten by a bug that showed up during late-in-the-game testing that didn't make a lot of sense, and was quite nasty in certain complex situations. This bug caused the "release process" to grind to a halt.

Well, I'm happy to say that, as of about two minutes ago, I've managed to figure out what's going on.

Basically, a folder can become "magic" in some situations, and even when the conditions that made it "magic" are reversed, the "magic" sticks around when it shouldn't. Unfortunately, this "magic" acted as a sort of "protective spell" on the folder, and was preventing us from doing anything.

Unfortunately, there's no 'external' visibility for when the "magic" sticks around, so we were seeing something that basically didn't make any sense. On top of that, it's new behavior in Leopard, which is why we've never seen it before. Fortunately, thanks to Amit Singh's recently-updated-for-Leopard hfsdebug (thanks, Amit -- love the book, too), I was able to drop down into the guts of HFS+ and determine what's happening.

Now that the problem's understood, we can implement an effective workaround. The workaround will mean that in some situations it'll re-copy a bit more than it should when Smart Updating. But, at least the result produced will be correct: and the workaround will break the spell, and remove the "magic".

Which is—let me tell you—a relief. (For those of you out there who have hit this kind of WTF-roadblock, where you have no idea what's wrong and thus can't even estimate how long it's going to take to figure it out and fix it, you know what I mean.) And for any Apple engineers reading, the (incorrect) behavior is described in rdar://5687977.

So, anyway, now that that data-integrity-related bug is getting wrapped up, we're back to putting together a final test build (should be in the next day or two), a bit of time to let our test group run their scenarios, etc.

So, barring another similar issue (please, no) showing up during testing, as I indicated in the comments of the previous post, it should be a week or so...

We look to still be on schedule, so hopefully you'll have the new version (which, by the way, I've decided will be 2.5, not 2.1.5) in a week or so.

We've been working through a problem with copying hard folder links in a complicated source volume recently that's got us a bit stymied—what we're seeing doesn't make sense—but hopefully we'll get it wrapped up shortly.

Sorry for the briefness of the entry: both busy and sick with some stomach flu... more when there's something new/interesting to say.

I know it feels like that from the outside. But from the inside, we've been putting out quite a few external beta releases (11 at last count), incorporating feedback from our testers, and carefully observing what users with 2.1.4—who have tried to use it with Leopard—have tried to do.

Before I start, though, please recognize that I'm describing things we noticed back in October. So, this is not causing additional delay—these are some of the things that contributed to the delay in the first place.

Paying Attention to Users

We've always observed our users very carefully. And, after Leopard's release, we noticed one big thing: users with 2.1.4 and Leopard have been trying to store their SuperDuper! backup to the same volume they've been using for Time Machine. But there's a significant problem with that: namely, since the Time Machine backup isn't on the source volume, Smart Update is going to try to delete it.

Now, Time Machine "protects" its own backups through a number of complex mechanisms, so no damage is done—except the backup fails.

Choices

In some ways, this is quite similar to the "storing other files alongside a backup" case that has come up before (and has an entire section of the User's Guide dedicated to it)—basically, we could request that users partition the destination drive, put their Time Machine backups on one volume, and their SuperDuper! backup on another.

But, even with Leopard's new "live partitioning" capability, that's a pain, isn't obvious, requires a support round trip, doesn't meet user expectations, and isn't transparent.

It also wasn't something we took into consideration during our original compatibility design. We focused on the various new capabilities of the OS, ensuring that we worked well, but since we were used to partitions—and knew that Time Machine liked as much disk space as possible—we gave it a lot of space to work with and left it alone.

So, once we started seeing this—which was after Leopard's release, of course—we had to make a choice. Push responsibility to the user to do the right thing, or anticipate and handle it inside SuperDuper! so users wouldn't be immediately frustrated.

But I Don't Want to Use Time Machine!

We certainly understood—and understand—that some users don't want to use Time Machine. But the problem is, a large proportion of Leopard's users do, and we have to serve the needs of both groups. We can't come out with a "Leopard-compatible" release that doesn't work properly with Time Machine.

And we couldn't really release a "quick update" that just copied the basics first, and then an update that handled this case right. Had we done that, we would have still needed to take a full external cycle to test the copy engine changes we needed to make, and then run the same extensive cycle again when we made the low-level changes to handle this case.

So, we bit the bullet and made the necessary changes to handle the case properly, the way users would expect: it just works.

So, if you Smart Update a volume that you're using for Time Machine, it works just fine. Your Time Machine backups, and your bootable SuperDuper! backup, and stored on the same volume, side-by-side, without interfering with each other, with no need to partition or do anything special (other than use Smart Update, of course).

On top of that, you can still copy that volume—that is, a volume that has both Time Machine backups and a bootable SuperDuper! backup—to a 3rd volume to make a second backup, and everything there works great, too.

This is technically more complicated that you might expect, but all of that is under the covers. On the surface, there are no visible changes.

Side Benefits

So, this is the main reason why we were able to improve the handling of Spotlight and preserve icons as I discussed in the last post: we had to handle the Time Machine case, but it benefits all users by improving the way Spotlight (which everyone basically uses) and icon preservation (which a small, but vocal group use) works.

Why Focus on Time Machine?

Which, of course, is an extension of But I Don't Want to Use Time Machine!, above.

The fact is, Apple considers Time Machine to be the most important, most visible new feature in Leopard. They've put a lot of marketing muscle behind its rollout, and its tendrils are in everything, from the default desktop to Quick Look.

As I've discussed before, it's a big, good change, and it benefits all users, even those who use SuperDuper! We're truly two complementary technologies, and we need to ensure that SuperDuper! works perfectly with it to ensure that SuperDuper! itself remains a viable product.

And if the product remains viable, we'll be able to continue improving it for everyone—something we all want!

It's Not All Time Machine, Though—Really!

Another interesting thing we found as we got reports from new users, that we never encountered internally: a very old bug in the Migration Assistant (described in Slowly going insane while waiting for a fix back in 2005) is back, with a vengeance. But there's a new variation: rather than having a missing carriage return, there are now also blank lines in the file, which result in the same failure.

So, we've had to implement a second workaround to fix this problem, too.

How Much Longer?

Anyway, enough of that. At this point, we're basically locked down. We have a few UI tweaks to complete, and we should have a final release out to everyone within a few weeks.

You'll notice this is the first timeframe I've provided other than "soon"... and that means I'm quite confident it'll be in your hands shortly. So, once again thanks for your patience, and I really think you're going to be pleased with the update.

Another week, another Leopard Status Update. Please, hold your cheers and applause until the end.

Two parts

Although most people think about SuperDuper! as a single application, it's actually comprised of a number of parts. The two main ones are the UI application (which you interact with) and the Copy Engine, which actually performs the copy.

When changes are made to the UI component, testing takes significantly less time, because it's much easier to ensure that those changes are working properly, and improper operation is relatively low risk: if a button doesn't dim properly in some conditions, it's an easy bug to work around for an end user.

Changes to the copy engine, though, are extremely high risk. Errors made in copying files affect the backup itself, and often occur silently: everything looks like it's working, but, not so much.

Testing the tests

This is one of the reasons that this update is taking so long to release. Due to the nature of the changes in Leopard, we've had to make significant changes to the engine. And while we, of course, have extensive test suites internally, those tests are "necessary" but not "sufficient".

Final bits

As I've mentioned before, we did, of course, have betas of Leopard, and we were working with them. But a beta is just a beta: changes are made up until the shipping version, and that was the case with Leopard as well.

But, there's another consideration, too: we can't do our "big" external test cycles until the "public" has the real bits. Yes, we can roll to testers who have the same beta we have, but when you're making the kinds of changes we had to make this release, it's important to roll in new testers who have different file setups and different patterns of copying.

(Tester fatigue is an issue that all developers are constantly struggling with: it's not so much that a tester is "tired", but that they fall into a pattern of use, and rarely venture outside it.)

So, our internal tests couldn't be run until we received the final bits (we never release test versions to our testers that are risky, to ensure they can really use the test release as a production copy), and our big external test couldn't start until after that.

That testing -- of which there have been multiple rounds -- has been going very nicely indeed.

More progress

So, last week, we locked down the "cloner changes": given feedback from the testers and our external tests, and running against 10.5 and 10.5.1, we're pretty confident that the engine's working properly.

That's a good thing.

We've added some new capabilities to the engine, too. We can now preserve the contents of the Spotlight index on the destination (which also preserves its indexing status in a more reliable way) and preserve the "File System Event" database that Leopard uses. More on this in a later post.

While we were doing that, we decided to do one more thing.

Save the icons

We had always felt that the volume icon -- being a file on the source drive like any other -- should be copied to the backup drive, since you'd want to restore the original drive to the same condition it was originally in, and that included the icon.

Well, we had a lot of pushback on that choice (to put it mildly - I think the thread that discusses this issue and the workaround we came up with originally is the longest one on the discussion forums), and we decided that we were wrong. Since you rarely restore, and your backup drive is constantly visible, it was best to preserve the destination icon. Plus, if you restore with SuperDuper!, we'll preserve the icon of the destination for the restore, so it's all good.

The changes we made, above, made this possible. And so, the destination volume icon will now be preserved.

More coming

So, a lot more progress has been made. Testing's going really well. And I'll talk a bit more about some other new stuff later.

So, another quick update on our progress on the Leopard version of SuperDuper! (which will work better than the 2.1.4 on Tiger, for those of you still there).

Link Hard

I'm happy to say that we've worked through the last few issues with hard links and the complexities of copying Time Machine backups. Yesterday, I was able to copy and restore an over 7 million file Time Machine drive successfully, and restore it. After some scanning, Time Machine picked up exactly where it left off, happily accepting the "new" drive as the right one.

This means that you can successfully copy your Time Machine backup to a different drive as a backup, or to a new, larger drive, without losing your history.

It also means, as I mentioned last post, that SD! is properly handling some of the new on-disk representations in Leopard.

Another test cycle

Now that we've successfully passed our internal tests for this, we're going to package it up (with some other new stuff I haven't talked about) and roll it to our external testers for them to bang on to make sure we haven't missed anything.

Good progress

So, we're making good progress. I'll write about some of the wackiness we've run into later on (example: did you know that journaling has to be on to be able to successfully copy a drive with hard links? Me neither.), but for now, back to the grind.

By popular demand (and demand and demand and demand), here's a quick update on the status of SuperDuper!'s Leopard update (likely to be called 2.1.5). Sorry it's been a bit quiet here. I've been getting a lot of email, and it's hard to find the time to write posts. I'll try to update the blog with status a bit more often in the future.

SuperDuper! is working well in our internal tests and for our external testers. Right now, we're ensuring that some of the (currently) lesser-used features of the Leopard file system are being handled properly, from both a copying and status standpoint.

Some technical details

The details of all these things are more than a bit obscure: the issues we deal with are things that end users should never need to think about.

For example, Leopard allows applications to "hard link" folders, something that's never been allowed before. Time Machine relies on this capability, but it's available to any application. Proper handling of these hard linked folders is important, and has its own challenges, including how you handle the counts and sizes of files on the drive for the status display, etc.

Not all parts of Leopard handle these new on-disk structures consistently, and as such we're working through the issues bit-by-bit, making sure we're doing things as right as we can, and in a way that won't break when (or if) -- things change in the future. Or when a new, Leopard-specific application you install starts to make more use of this stuff...

Another obscure example

One thing we found during testing was that users with Bluetooth keyboards were having trouble. The keyboard wasn't maintaining its pairing relationship with the system when starting up from the copy.

This was not actually due to any kind of file or metadata copying problem (as we initially feared): everything was being copied as it should, so when we ran our various comparison/verification tools, things checked out fine. Instead, it was due to a change in the way Leopard recreates (or, in this case, no longer recreates) a standard system folder (specifically, /private/var/run), which is certainly not the first place you'd look for this kind of thing, and running down the problem (and verifying the fix) took time.

Timeframe

So, we're working hard to get things right. We want to make sure that when you make a copy of an HFS+/HFSX Leopard volume with SuperDuper!—regardless of what new features were used—the copy will be correct.

Of course, we keep being asked—with various level of politeness—for a "timeframe" for release. Again, we're not targeting a drop-dead release date. It all depends on how testing continues to go after each beta release, whether any bizarre things are found that weren't covered in our test suite, and how long those things—if found—take to research and handle properly. It's not going to be in the next few days: we're hoping within a few weeks. I'll keep you informed about our status here.

Waiting sucks

I know. Believe me, I know. Waiting sucks. I understand your frustration: it's frustrating on this end, too. Once again, thanks for your patience as we try to maintain the level of quality you expect from us.

Now that the release of Leopard is imminent-ish, I thought I'd do an update of my Time's Arrow post from a year ago regarding SuperDuper! and Time Machine. Let's get into it!

Leopard Compatbitility

First, v2.1.4 of SuperDuper! is not fully compatible with Leopard.

Changes were made to Leopard since our last release in February. We've been following Leopard's development closely, and we have a working version of SuperDuper! nearly ready. Given the nature of SuperDuper!, though, we don't want to release it until we have the final "bits" of Leopard to test with. It should be available shortly after Leopard's release (earlier if we get the final build before the public does--which we won't).

To answer the inevitable question: the update will be free when released.

Note that netTunes is not Leopard compatible due to a Leopard bug I was unable to work around. I'm hoping for a fix in an upcoming Leopard release, since I do not have a workaround for the problem.

Leopard and Sandboxes

When the Leopard-compatible version is released, please realize that Leopard is a pretty different version of the OS. I recommend against using a Sandbox to share a Home folder between Tiger/Panther/Jaguar and Leopard: there have been a lot of changes to the way data is stored, and a Sandbox is not likely to provide sufficient isolation.

Time Machine

As far as Time Machine goes -- in general, I've always thought that some sort of backup functionality belongs in the OS. It's been a long time coming. The fact that it wasn't there left opportunities for 3rd parties, but that doesn't mean Apple shouldn't address the missing functionality.

And so, they have, with Time Machine. Really, I think that's a great thing. People need to back up more often, and I think Time Machine encourages them to do so, and gives them a relatively transparent way to do it.

Time Machine and SuperDuper!

That said, though, Time Machine isn't the be-all and end-all of backup programs. In fact, given how it works, I really do think that SuperDuper! remains both relevant and necessary -- a true complement to the functionality in Time Machine.

First, as is likely obvious, Time Machine is designed to provide automatic "temporal" backup (discussed in broad terms in the post The Ninety-Nine-Per-Cent Solution many months ago). Its primary usage scenario -- and the one that most demonstrations and documentation focuses on -- is to allow quick recovery of files and data that have gone missing, etc.

It does this in a way that's highly integrated with the OS, with a unique UI that's both cool and kinda cheesy (I'll tell you, the 'space' theme hasn't grown on me at all...)... and, as was the case with Spotlight, with a certain amount of application-level impact (something 3rd parties like Shirt Pocket could never mandate).

Fully, Directly Bootable Backups with SuperDuper!

What's important to note is that this isn't, and never was, what SuperDuper! was designed to do.

Our tagline, Heroic System Recovery for Mere Mortals, tries to sum up the whole idea: SuperDuper! is designed to provide excellent failover support for the all-too-common case where things fail in a pretty catastrophic way, such as when a drive fails, or your system becomes unbootable. We do this by quickly and efficiently creating a fully bootable copy of your source drive. Perhaps more importantly, recovery is near immediate, even if the original drive is completely unusable, because you can start up from your backup and continue working.

You can even take your backup to a totally different Macintosh, start up from it, and work while your failed Macintosh is in the shop... then, when it comes back all fresh and shiny, restore things and keep working. And even if the other Mac is a different CPU type, you can still open and edit the files on the backup.

You cannot do this with Time Machine: Time Machine copies are not bootable until they're restored.

In SuperDuper!, system recovery is done with a minimum of fuss and bother, and with respect for your time. Yes, Time Machine can restore a full system, but that's not its strength. Doing so requires you to actually start up from the Leopard DVD (which you'll need to have with you) and then take the time to restore the backup in full, which interrupts your workflow, requires a working, entirely separate destination device, and takes a lot of your time -- at the exact moment when you can least afford it.

So, when Leopard comes out, and Time Machine is released, be assured that SuperDuper!'s fully bootable backups continue to be relevant and necessary. We'll work alongside its rapid recovery of individual files, augmenting that with our rapid system recovery.

And, of course, we'll continue to improve every part of SuperDuper! to make backups faster and easier for all.

For those who haven't heard me blather at length about all things SuperDuper! and Shirt Pocket, there's a 45 minute interview of me up at MacVoices.

Remember: the game rules require you to drink every time you hear me say "Um", "Uh" or audibly gesticulate. Do not play while driving.

A little tip for those of you who might have installed Seagate SATA drives into your 3GB/s capable G5, MacPro, NAS device, or whatever.

Looks like recent Seagate drives ship with a jumper installed that limits the drive to 1.5GB/s speeds. While the jumper is documented in the User's Guide that ships with retail packs, it's specifically mentioned as something you might need to install if you have trouble with the drive. And OEM drives don't have any documentation at all.

To get 3GB/s, the jumper should not be present on the outer pins of the jumper block. So -- if you've got one of these drives, check it out: you might get that drive humming along twice as fast!