...we salute our testers, and the patience of our users.

It's going to be a SuperDuper! Tuesday.

A few days ago, we were happy to get a workaround for the bug mentioned in a previous post, so we went about implementing and testing—and things were working quite well.

Unfortunately, we've found a case where the workaround doesn't, um, work. I think I have a way of dealing with it, and so we're going to implement Workaround Part IV: The Reworkening and run it through the wringer.

Again.

These tests are really time consuming, as you might guess: it can take eight hours to rebuild a complex-case volume (so I try to keep one building while another is testing), and then more hours to run the test. I can't begin to tell you how frustrating it is to have a set of simple tests work and then find that the thing we found rearing its ugly head again in another (more complex) scenario that can (and did) happen in "real life".

This is why we don't just implement a fix and release, of course: we want to ensure it works before we toss it out there, and so we smoke test, test internally and then—if internal tests are OK—test externally, every time. We always are hoping for green lights at every stage. Sometimes, 99% of the way there, you get red. Then, the failure needs to be investigated, understood, reproduced...

Anyway, we'll leave that between me and my ulcer. If my new idea works, this will hopefully only delay things a few days, which will be plenty of time to fill the comments with complaints of our laziness, incompetence, lack of communication and general suckitude. Have at it!

Note: I've had to turn comments off for a little while because the subscriptions to comments are overloading my outbound servers (every time someone posts something, it's sent to the 200 people above their post), causing delays for regular support mail, etc. You can still review existing comments by clicking the title of this post. I'll turn them back on after the backlog clears. Sorry for any inconvenience: it's not that I don't want to hear what you want to say...

Last post, I mentioned that we were bitten by a bug that showed up during late-in-the-game testing that didn't make a lot of sense, and was quite nasty in certain complex situations. This bug caused the "release process" to grind to a halt.

Well, I'm happy to say that, as of about two minutes ago, I've managed to figure out what's going on.

Basically, a folder can become "magic" in some situations, and even when the conditions that made it "magic" are reversed, the "magic" sticks around when it shouldn't. Unfortunately, this "magic" acted as a sort of "protective spell" on the folder, and was preventing us from doing anything.

Unfortunately, there's no 'external' visibility for when the "magic" sticks around, so we were seeing something that basically didn't make any sense. On top of that, it's new behavior in Leopard, which is why we've never seen it before. Fortunately, thanks to Amit Singh's recently-updated-for-Leopard hfsdebug (thanks, Amit -- love the book, too), I was able to drop down into the guts of HFS+ and determine what's happening.

Now that the problem's understood, we can implement an effective workaround. The workaround will mean that in some situations it'll re-copy a bit more than it should when Smart Updating. But, at least the result produced will be correct: and the workaround will break the spell, and remove the "magic".

Which is—let me tell you—a relief. (For those of you out there who have hit this kind of WTF-roadblock, where you have no idea what's wrong and thus can't even estimate how long it's going to take to figure it out and fix it, you know what I mean.) And for any Apple engineers reading, the (incorrect) behavior is described in rdar://5687977.

So, anyway, now that that data-integrity-related bug is getting wrapped up, we're back to putting together a final test build (should be in the next day or two), a bit of time to let our test group run their scenarios, etc.

So, barring another similar issue (please, no) showing up during testing, as I indicated in the comments of the previous post, it should be a week or so...

We look to still be on schedule, so hopefully you'll have the new version (which, by the way, I've decided will be 2.5, not 2.1.5) in a week or so.

We've been working through a problem with copying hard folder links in a complicated source volume recently that's got us a bit stymied—what we're seeing doesn't make sense—but hopefully we'll get it wrapped up shortly.

Sorry for the briefness of the entry: both busy and sick with some stomach flu... more when there's something new/interesting to say.

I know it feels like that from the outside. But from the inside, we've been putting out quite a few external beta releases (11 at last count), incorporating feedback from our testers, and carefully observing what users with 2.1.4—who have tried to use it with Leopard—have tried to do.

Before I start, though, please recognize that I'm describing things we noticed back in October. So, this is not causing additional delay—these are some of the things that contributed to the delay in the first place.

Paying Attention to Users

We've always observed our users very carefully. And, after Leopard's release, we noticed one big thing: users with 2.1.4 and Leopard have been trying to store their SuperDuper! backup to the same volume they've been using for Time Machine. But there's a significant problem with that: namely, since the Time Machine backup isn't on the source volume, Smart Update is going to try to delete it.

Now, Time Machine "protects" its own backups through a number of complex mechanisms, so no damage is done—except the backup fails.

Choices

In some ways, this is quite similar to the "storing other files alongside a backup" case that has come up before (and has an entire section of the User's Guide dedicated to it)—basically, we could request that users partition the destination drive, put their Time Machine backups on one volume, and their SuperDuper! backup on another.

But, even with Leopard's new "live partitioning" capability, that's a pain, isn't obvious, requires a support round trip, doesn't meet user expectations, and isn't transparent.

It also wasn't something we took into consideration during our original compatibility design. We focused on the various new capabilities of the OS, ensuring that we worked well, but since we were used to partitions—and knew that Time Machine liked as much disk space as possible—we gave it a lot of space to work with and left it alone.

So, once we started seeing this—which was after Leopard's release, of course—we had to make a choice. Push responsibility to the user to do the right thing, or anticipate and handle it inside SuperDuper! so users wouldn't be immediately frustrated.

But I Don't Want to Use Time Machine!

We certainly understood—and understand—that some users don't want to use Time Machine. But the problem is, a large proportion of Leopard's users do, and we have to serve the needs of both groups. We can't come out with a "Leopard-compatible" release that doesn't work properly with Time Machine.

And we couldn't really release a "quick update" that just copied the basics first, and then an update that handled this case right. Had we done that, we would have still needed to take a full external cycle to test the copy engine changes we needed to make, and then run the same extensive cycle again when we made the low-level changes to handle this case.

So, we bit the bullet and made the necessary changes to handle the case properly, the way users would expect: it just works.

So, if you Smart Update a volume that you're using for Time Machine, it works just fine. Your Time Machine backups, and your bootable SuperDuper! backup, and stored on the same volume, side-by-side, without interfering with each other, with no need to partition or do anything special (other than use Smart Update, of course).

On top of that, you can still copy that volume—that is, a volume that has both Time Machine backups and a bootable SuperDuper! backup—to a 3rd volume to make a second backup, and everything there works great, too.

This is technically more complicated that you might expect, but all of that is under the covers. On the surface, there are no visible changes.

Side Benefits

So, this is the main reason why we were able to improve the handling of Spotlight and preserve icons as I discussed in the last post: we had to handle the Time Machine case, but it benefits all users by improving the way Spotlight (which everyone basically uses) and icon preservation (which a small, but vocal group use) works.

Why Focus on Time Machine?

Which, of course, is an extension of But I Don't Want to Use Time Machine!, above.

The fact is, Apple considers Time Machine to be the most important, most visible new feature in Leopard. They've put a lot of marketing muscle behind its rollout, and its tendrils are in everything, from the default desktop to Quick Look.

As I've discussed before, it's a big, good change, and it benefits all users, even those who use SuperDuper! We're truly two complementary technologies, and we need to ensure that SuperDuper! works perfectly with it to ensure that SuperDuper! itself remains a viable product.

And if the product remains viable, we'll be able to continue improving it for everyone—something we all want!

It's Not All Time Machine, Though—Really!

Another interesting thing we found as we got reports from new users, that we never encountered internally: a very old bug in the Migration Assistant (described in Slowly going insane while waiting for a fix back in 2005) is back, with a vengeance. But there's a new variation: rather than having a missing carriage return, there are now also blank lines in the file, which result in the same failure.

So, we've had to implement a second workaround to fix this problem, too.

How Much Longer?

Anyway, enough of that. At this point, we're basically locked down. We have a few UI tweaks to complete, and we should have a final release out to everyone within a few weeks.

You'll notice this is the first timeframe I've provided other than "soon"... and that means I'm quite confident it'll be in your hands shortly. So, once again thanks for your patience, and I really think you're going to be pleased with the update.

Another week, another Leopard Status Update. Please, hold your cheers and applause until the end.

Two parts

Although most people think about SuperDuper! as a single application, it's actually comprised of a number of parts. The two main ones are the UI application (which you interact with) and the Copy Engine, which actually performs the copy.

When changes are made to the UI component, testing takes significantly less time, because it's much easier to ensure that those changes are working properly, and improper operation is relatively low risk: if a button doesn't dim properly in some conditions, it's an easy bug to work around for an end user.

Changes to the copy engine, though, are extremely high risk. Errors made in copying files affect the backup itself, and often occur silently: everything looks like it's working, but, not so much.

Testing the tests

This is one of the reasons that this update is taking so long to release. Due to the nature of the changes in Leopard, we've had to make significant changes to the engine. And while we, of course, have extensive test suites internally, those tests are "necessary" but not "sufficient".

Final bits

As I've mentioned before, we did, of course, have betas of Leopard, and we were working with them. But a beta is just a beta: changes are made up until the shipping version, and that was the case with Leopard as well.

But, there's another consideration, too: we can't do our "big" external test cycles until the "public" has the real bits. Yes, we can roll to testers who have the same beta we have, but when you're making the kinds of changes we had to make this release, it's important to roll in new testers who have different file setups and different patterns of copying.

(Tester fatigue is an issue that all developers are constantly struggling with: it's not so much that a tester is "tired", but that they fall into a pattern of use, and rarely venture outside it.)

So, our internal tests couldn't be run until we received the final bits (we never release test versions to our testers that are risky, to ensure they can really use the test release as a production copy), and our big external test couldn't start until after that.

That testing -- of which there have been multiple rounds -- has been going very nicely indeed.

More progress

So, last week, we locked down the "cloner changes": given feedback from the testers and our external tests, and running against 10.5 and 10.5.1, we're pretty confident that the engine's working properly.

That's a good thing.

We've added some new capabilities to the engine, too. We can now preserve the contents of the Spotlight index on the destination (which also preserves its indexing status in a more reliable way) and preserve the "File System Event" database that Leopard uses. More on this in a later post.

While we were doing that, we decided to do one more thing.

Save the icons

We had always felt that the volume icon -- being a file on the source drive like any other -- should be copied to the backup drive, since you'd want to restore the original drive to the same condition it was originally in, and that included the icon.

Well, we had a lot of pushback on that choice (to put it mildly - I think the thread that discusses this issue and the workaround we came up with originally is the longest one on the discussion forums), and we decided that we were wrong. Since you rarely restore, and your backup drive is constantly visible, it was best to preserve the destination icon. Plus, if you restore with SuperDuper!, we'll preserve the icon of the destination for the restore, so it's all good.

The changes we made, above, made this possible. And so, the destination volume icon will now be preserved.

More coming

So, a lot more progress has been made. Testing's going really well. And I'll talk a bit more about some other new stuff later.

So, another quick update on our progress on the Leopard version of SuperDuper! (which will work better than the 2.1.4 on Tiger, for those of you still there).

Link Hard

I'm happy to say that we've worked through the last few issues with hard links and the complexities of copying Time Machine backups. Yesterday, I was able to copy and restore an over 7 million file Time Machine drive successfully, and restore it. After some scanning, Time Machine picked up exactly where it left off, happily accepting the "new" drive as the right one.

This means that you can successfully copy your Time Machine backup to a different drive as a backup, or to a new, larger drive, without losing your history.

It also means, as I mentioned last post, that SD! is properly handling some of the new on-disk representations in Leopard.

Another test cycle

Now that we've successfully passed our internal tests for this, we're going to package it up (with some other new stuff I haven't talked about) and roll it to our external testers for them to bang on to make sure we haven't missed anything.

Good progress

So, we're making good progress. I'll write about some of the wackiness we've run into later on (example: did you know that journaling has to be on to be able to successfully copy a drive with hard links? Me neither.), but for now, back to the grind.

By popular demand (and demand and demand and demand), here's a quick update on the status of SuperDuper!'s Leopard update (likely to be called 2.1.5). Sorry it's been a bit quiet here. I've been getting a lot of email, and it's hard to find the time to write posts. I'll try to update the blog with status a bit more often in the future.

SuperDuper! is working well in our internal tests and for our external testers. Right now, we're ensuring that some of the (currently) lesser-used features of the Leopard file system are being handled properly, from both a copying and status standpoint.

Some technical details

The details of all these things are more than a bit obscure: the issues we deal with are things that end users should never need to think about.

For example, Leopard allows applications to "hard link" folders, something that's never been allowed before. Time Machine relies on this capability, but it's available to any application. Proper handling of these hard linked folders is important, and has its own challenges, including how you handle the counts and sizes of files on the drive for the status display, etc.

Not all parts of Leopard handle these new on-disk structures consistently, and as such we're working through the issues bit-by-bit, making sure we're doing things as right as we can, and in a way that won't break when (or if) -- things change in the future. Or when a new, Leopard-specific application you install starts to make more use of this stuff...

Another obscure example

One thing we found during testing was that users with Bluetooth keyboards were having trouble. The keyboard wasn't maintaining its pairing relationship with the system when starting up from the copy.

This was not actually due to any kind of file or metadata copying problem (as we initially feared): everything was being copied as it should, so when we ran our various comparison/verification tools, things checked out fine. Instead, it was due to a change in the way Leopard recreates (or, in this case, no longer recreates) a standard system folder (specifically, /private/var/run), which is certainly not the first place you'd look for this kind of thing, and running down the problem (and verifying the fix) took time.

Timeframe

So, we're working hard to get things right. We want to make sure that when you make a copy of an HFS+/HFSX Leopard volume with SuperDuper!—regardless of what new features were used—the copy will be correct.

Of course, we keep being asked—with various level of politeness—for a "timeframe" for release. Again, we're not targeting a drop-dead release date. It all depends on how testing continues to go after each beta release, whether any bizarre things are found that weren't covered in our test suite, and how long those things—if found—take to research and handle properly. It's not going to be in the next few days: we're hoping within a few weeks. I'll keep you informed about our status here.

Waiting sucks

I know. Believe me, I know. Waiting sucks. I understand your frustration: it's frustrating on this end, too. Once again, thanks for your patience as we try to maintain the level of quality you expect from us.

Now that the release of Leopard is imminent-ish, I thought I'd do an update of my Time's Arrow post from a year ago regarding SuperDuper! and Time Machine. Let's get into it!

Leopard Compatbitility

First, v2.1.4 of SuperDuper! is not fully compatible with Leopard.

Changes were made to Leopard since our last release in February. We've been following Leopard's development closely, and we have a working version of SuperDuper! nearly ready. Given the nature of SuperDuper!, though, we don't want to release it until we have the final "bits" of Leopard to test with. It should be available shortly after Leopard's release (earlier if we get the final build before the public does--which we won't).

To answer the inevitable question: the update will be free when released.

Note that netTunes is not Leopard compatible due to a Leopard bug I was unable to work around. I'm hoping for a fix in an upcoming Leopard release, since I do not have a workaround for the problem.

Leopard and Sandboxes

When the Leopard-compatible version is released, please realize that Leopard is a pretty different version of the OS. I recommend against using a Sandbox to share a Home folder between Tiger/Panther/Jaguar and Leopard: there have been a lot of changes to the way data is stored, and a Sandbox is not likely to provide sufficient isolation.

Time Machine

As far as Time Machine goes -- in general, I've always thought that some sort of backup functionality belongs in the OS. It's been a long time coming. The fact that it wasn't there left opportunities for 3rd parties, but that doesn't mean Apple shouldn't address the missing functionality.

And so, they have, with Time Machine. Really, I think that's a great thing. People need to back up more often, and I think Time Machine encourages them to do so, and gives them a relatively transparent way to do it.

Time Machine and SuperDuper!

That said, though, Time Machine isn't the be-all and end-all of backup programs. In fact, given how it works, I really do think that SuperDuper! remains both relevant and necessary -- a true complement to the functionality in Time Machine.

First, as is likely obvious, Time Machine is designed to provide automatic "temporal" backup (discussed in broad terms in the post The Ninety-Nine-Per-Cent Solution many months ago). Its primary usage scenario -- and the one that most demonstrations and documentation focuses on -- is to allow quick recovery of files and data that have gone missing, etc.

It does this in a way that's highly integrated with the OS, with a unique UI that's both cool and kinda cheesy (I'll tell you, the 'space' theme hasn't grown on me at all...)... and, as was the case with Spotlight, with a certain amount of application-level impact (something 3rd parties like Shirt Pocket could never mandate).

Fully, Directly Bootable Backups with SuperDuper!

What's important to note is that this isn't, and never was, what SuperDuper! was designed to do.

Our tagline, Heroic System Recovery for Mere Mortals, tries to sum up the whole idea: SuperDuper! is designed to provide excellent failover support for the all-too-common case where things fail in a pretty catastrophic way, such as when a drive fails, or your system becomes unbootable. We do this by quickly and efficiently creating a fully bootable copy of your source drive. Perhaps more importantly, recovery is near immediate, even if the original drive is completely unusable, because you can start up from your backup and continue working.

You can even take your backup to a totally different Macintosh, start up from it, and work while your failed Macintosh is in the shop... then, when it comes back all fresh and shiny, restore things and keep working. And even if the other Mac is a different CPU type, you can still open and edit the files on the backup.

You cannot do this with Time Machine: Time Machine copies are not bootable until they're restored.

In SuperDuper!, system recovery is done with a minimum of fuss and bother, and with respect for your time. Yes, Time Machine can restore a full system, but that's not its strength. Doing so requires you to actually start up from the Leopard DVD (which you'll need to have with you) and then take the time to restore the backup in full, which interrupts your workflow, requires a working, entirely separate destination device, and takes a lot of your time -- at the exact moment when you can least afford it.

So, when Leopard comes out, and Time Machine is released, be assured that SuperDuper!'s fully bootable backups continue to be relevant and necessary. We'll work alongside its rapid recovery of individual files, augmenting that with our rapid system recovery.

And, of course, we'll continue to improve every part of SuperDuper! to make backups faster and easier for all.

It's a few weeks later, but Zabeth and I are back from our two week vacation, and so things around here are returning to normal: the Shirt Pocket Support "autoresponder" is turned off (since I'm able to respond more quickly now), and Z's begun her internship in Bridgewater.

The motivation for the vacation: Z just graduated from Vet School, and she had two weeks to recharge before her internship started. So, we decided to make a strong memory to carry her through the next year of work-without-sleep. And so -- we had a great time bopping around Germany and Italy, from Munich down to Matera -- actually, even further, down in Ostuni -- staying in Siena, Rome, Positano, Santa Agata sui due Golfi and Lake Garda along the way, and stopping at too many other places to list.

It was a "working vacation", which means a full work day, every day, but broken into little pieces all day long. It's a bit haphazard and nerve-wracking as the support cases build up and questions need to be answered, but it works well enough that I can still deliver good service to my valued users. Or try to!

Germany

Munich

One of the things we did while in Munich was tour the factory where BMWs are built, quite literally in the middle of the city. The plant -- which is huge and cannot grow beyond its current boundaries -- is on multiple levels, and uses a lot of techniques to reduce noise, pollution, and make efficient use of space. It's an extremely worthwhile three hours if you're a fan of sophisticated manufacturing (check out the videos at the plant site), and completely free.

It's amazing how much car manufacturing has changed -- one (award winning!) assembly station had something like 13 robots all working together to weld a unibody... incredibly complicated programming and layout. Very cool.

After the plant tour, we had a great time in Munich thanks to our friend Mathey and his girlfriend Sonja, who were kind enough to show us around their beautiful city. In which you can surf. Who knew: video evidence pending!

Anyway, we spent far too little time in Germany, something I plan to rectify next time. Hopefully we'll be back in less than ten years!

Italy

Siena

Our first stop in Italy was Siena, at a beautiful Bed and Breakfast (Frances' Lodge) outside the city. Our hosts -- Franco and Frances -- couldn't have been more gracious, nor their property more beautiful.

Internet access was extremely problematic, though, so I ended up spending a few hours of each day at an Internet Café in Siena itself -- so, if you saw a guy frantically typing at a MacBook Pro, in between the people browsing Facebook/My Space/Flickr, without stopping to breathe, that was me. (Note the laptop bag strap in this picture, just across from Siena's Duomo...)

Both Frances and Franco (Frances from Florence, and Franco Torre through-and-through) helped us find good places to visit, eat and helped fill us in on the details of Siena and the Palio di Siena, which is much more involved than you might expect given the length of the race, with tons of fascinating political intrigue and behind the scenes machinations.

We enjoyed taking various side-trips to towns and hikes through the Tuscan countryside, although I wish we had Taiko along -- he would have loved romping through the hills and fields during our long walks! And the gelato! (For the curious - cinnamon and black cherry.)

Rome

After (again) too few days, we headed down to Rome to meet our old friends Mike (my college roommate and old business partner at UnderWare) and Robin and their kids Ian and Leah. We hadn't been able to share vacation time in a long while, and it was great to spend some quality, relaxing time with them.

We were only able to spend two days in Rome (sound like a theme?), so we hardly scratched the surface of the place, but enjoyed seeing things we'd only read about, with a tour guide who supplied many helpful details as we walked -- quickly -- through history.

Positano via Pompeii

A few days after we arrived in Rome we left our friends and were off to the Amalfi coast -- specifically Positano. On the way, we stopped in Herculaneum and Pompeii to take in a bit of their amazing history (and it's well worth seeing both - the contrasts between the two towns are fascinating).

And from there down to the coast and Positano for a relaxing few days at the beach, and some invigorating hiking in the extremely steep hills above.

The Amalfi Drive and Matera

From there, we drove down the Amalfi Drive (which is a blast and would make a terrific video game)

to the Basilicata region and Matera to meet back up with our friends. Matera's old section, where we stayed, is quite ancient and built into the rock hills above a deep ravine. Squint a little and it's the 3rd century BC, and you're heading home after tending the flock all day.

Il Frantoio

One afternoon we drove from Matera to Ostuni to have a delightful homestyle meal at Masseria Il Frantoio, where our hosts Armando and Rosalba Balestrazzi stuffed us silly with all manner of delicious treats, all grown organically on their farm.

Don Alfonso 1890

And, finally, a long drive back from Matera to Sant'Agata sui due Golfi (near Naples), for a blowout last meal at Don Alfonso, celebrating Ian's 16th birthday and Zabeth's graduation and upcoming trial-by-fire Internship. Everybody's pretty comatose here, as we'd been eating for something like 5 hours...

A lovely location and spectacular dinner... which no doubt would have been followed by an amazing breakfast had we not been forced to check out so early due to inconvenient flights.

Lake Garda

Since Zabeth had to start her Internship on Monday, I dropped her off at the Naples Airport early, and started the long solo drive back to Munich, stopping at Lake Garda for a quick overnight, and continuing to Munich -- on the Autostrada/Autobahn through Austria, stopping every so often for a shot or two of Espresso -- much too early the next morning.

A long few flights, and I'm back!

Thanks

It'd been quite a while since we'd been in Europe, and it was really great to get away (even while still needing to work every day -- it's very cool that I have a job that I can do anywhere), to see things we'd never had a chance to see before, and to revisit where Zabeth and I got engaged ten years ago.

Big thanks to Mike -- our resident Italy/travel/food expert -- for doing much of the planning; Mike, Robin, Ian and Leah for sharing part of their vacation with us; Mathey and Sonja for their time in Munich; my parents and brother Paul for taking great care of Taiko; and to Jonas Salling of Salling Software for helping out in the forums.

Very special thanks to all of you for your patience during the trip, your understanding when support responses didn't get turned around in minutes, and the chance to recharge our batteries before her internship starts, which will be 14 months with less than a week's break.

Wish us luck!