You'd think, given I've written everything here, I'd learn to pay attention to what I've said in the past.

Let me explain. In v3.2.4, we started getting occasional reports of scheduled copies running more than once. Which seemed kind of crazy, because, well, our little time-based backup agent—sdbackupbytime—only runs once a minute, and I'd reviewed the code so many times. There was no chance of a logic error: it was only going to run these puppies one time.

But it was, sometimes, regardless. And one of the things I had to change to complete Smart Wake was to loop to stay active when there was a backup coming in the next minute. But the code was definitely right. Could that have caused this to happen?

(Spoiler alert after many code reviews, hand simulations and debugger step-throughs: NO.)

So if that code was right, why the heck were some users getting occasional multiple runs? And why wasn't it happening to us here at Shirt Pocket HQ?

Of course, anyone who reported any of these problems received an intermediate build that fixed them all. We didn't rush out a new update because the side effect was more copies rather than fewer.

Secondary Effects

Well, one thing we did in 3.2.4 was change what SuperDuper does when it starts.

Previously, in order to work around problems with Clean My Mac (which, for some reason, incorrectly disables our scheduling when a cleanup pass is run, much to my frustration), we changed SuperDuper to reload our LaunchAgents when it starts, in order to self-repair.

This worked fine, except when sdbackupbytime was waiting a minute to run another backup, or when sdbackuponmount was processing multiple drives. In that case, it could potentially kill the process waiting to complete its operation. So, in v3.2.4, we check to see if the version is different, and only perform the reload if the agent was updated.

The known problem with this is that it wouldn't fix Clean My Mac's disabling until the next update. But it also had a secondary effect: these occasional multiple runs. But why?

Well, believe it or not, it's because of ThrottleInterval. Again.

Readers may recall that launchd's ThrottleInterval doesn't really control how often a job might launch, "throttling" it to only once every n seconds. It actually forces a task to relaunch if it doesn't run for at least n seconds.

It does this even if a task succeeds, exits normally, and is set up to run, say, every minute.

sdbackupbytime is a pretty efficient little program, and takes but a fraction of a second to do what it needs to do. When it's done, it exits normally. But if it took less than 10 seconds to run, the system (sometimes) starts it again...and since it's in the same minute, it would run the same schedule a second time.

The previous behavior of SuperDuper masked this operation, because when it launched it killed the agent that had been re-launched: a secondary, unexpected effect. So the problem didn't show up until we stopped doing that.

And I didn't expect ThrottleInterval to apply to time-based agents, because you can set things to run down to every second, so why would it re-launch an agent that was going to have itself launched every minute? (It's not that I can't come up with reasons, it's just that those reasons are covered by other keys, like KeepAlive.)

Anyway, I changed sdbackupbytime to pointlessly sleep up to ThrottleInterval seconds if it was going to exit "too quickly", and the problem was solved...by doing something dumb.

Hey, you do what you have to do, you know? (And sometimes what you have to do is pay attention to your own damn posts.)

Queue'd

Another big thing we did was rework our AppleScript "queue" management to improve it. Which we did.

But we also broke something.

Basically, the queue maintains a semaphore that only allows one client at a time through to run copies using AppleScript. And we remove that client from the queue when its process exits.

The other thing we do is postpone quit when there are clients in the queue, to make sure it doesn't quit out from under someone who's waiting to take control.

In 3.2.4, we started getting reports that the "Shutdown on successful completion" command wasn't working, because SuperDuper wasn't quitting.

Basically, the process sending the Quit command was queued as a task trying to control us, and it never quit...so we deferred the quit until the queue drained, which never happened.

We fixed this and a few similar cases (people using keyboard managers or other things).

Leaving Things Out

As you might know, APFS supports "sparse files" which are, basically, files where "unwritten" data takes up no space on a drive. So, you might have a file that was preallocated with a size of 200GB, but if you only wrote to the first 1MB, it would only take up 1MB on the drive.

These types of files aren't used a lot, but they are used by Docker and VirtualBox, and we noticed that Docker and VirtualBox files were taking much longer to copy than we were comfortable with.

Our sparse file handling tried to copy a sparse file in a way that ensured it was taking a minimal amount of space. That meant we scanned every read block for zeros, and didn't write the sections of files that were 0 by seeking ahead from that point, and writing the non-zero part of the block.

The problem with this is that it takes a lot of time to scan every block. On top of that, there were some unusual situations where the OS would do...weird things...with certain types of seek operations, especially at the end of a file.

So, in 3.2.5, we've changed this so that rather than write a file "optimally", maximizing the number of holes, we write it "accurately". That is, we exactly replicate the sparse structure on the source. This speeds things up tremendously. For example, with a typical sparse docker image, the OS's low-level copyfile function takes 13 minutes to copy with full fidelity, rsync takes 3 minutes and doesn't provide full fidelity, whereas SuperDuper 3.2.5 takes 53 seconds and exactly replicates the source.

That's a win.

Don't Go Away Mad...

In Mojave 10.14.4 or so, we starting getting reports of an error unmounting the snapshot, after which the copy would fail.

I researched the situation, and in this unusual case, we'd ask the OS to eject the volume, it would say it wasn't able to, then we'd ask again (six times), and we'd get an error each time...because it was already unmounted.

So, it would fail to unmount something that it would then unmount. (Winning!)

That's been worked around in this update. (Actual winning!)

Improving the Terrible

As you've seen, Apple's process for granting Full Disk Access is awful. There's almost no guidance in the window—it's like they made it intentionally terrible (I think they did, to discourage people from doing it).

We'd hoped that they'd improve it, and while some small improvements have been made, it hasn't been enough. So, thanks to some work done and generously shared by Media Atelier, we now provide instructions and a draggable application proxy, overlaid on the original window. It's much, much better now.

Thanks again and a tip of the pocket to Pierre Bernard of Houdah Software and Stefan Fuerst of Media Atelier!

eSellerate Shutdown

Our long-time e-commerce provider, eSellerate, is shutting down as of 6/30. So, we had to move to a new "store".

After a long investigation, we've decided to move to Paddle, which—in our opinion—provides the best user experience of all the ones we tried.

Our new purchase process allows payment through more methods, including Apple Pay, and is simpler than before. And we've implemented a special URL scheme so your registration details can now be entered into SuperDuper from the receipt with a single click, which is cool, convenient, and helps to ensure they're correct.

Accomplishing this required quite a bit of additional engineering, including moving to a new serial number system, since eSellerate's was going away. We investigated a number of the existing solutions, and really didn't want to have the gigantically long keys they generated. So we developed our own.

We hope you like the new purchase process: thanks to the folks at Rogue Amoeba, Red Sweater Software, Bare Bones Software and Stand Alone—not to mention Thru-Hiker—for advice, examples and testing.

Note that this means versions of SuperDuper! older than 3.2.5 will not work with the new serial numbers (old serial numbers still work with the new SuperDuper). If you have a new serial number and you need to use it with an old version, please contact support.

(Note that this also means netTunes and launchTunes are no longer available for purchase. They'll be missed, by me at least.)

Various and Sundry

I also spent some time improving Smart Delete in this version; it now looks for files that have shrunk as candidates for pre-moval, and if it can't find any space, but we're writing to an APFS volume, I proactively thin any snapshots to free up space on the container.

All that means even fewer out of space errors. Hooray!

We also significantly improved our animations (which got weird during 10.13) by moving our custom animation code to Core Animation (wrapping ourselves in 2007's warm embrace) and fixed our most longstanding-but-surprisingly-hard-to-fix "stupid" bug: you can now tab between the fields in the Register... window. So, if you had long odds on that, contact your bookie: it's going to pay off big.

With a Bow

So there you go - the update is now available in-app, and of course has been released on the Shirt Pocket site. Download away!

Now that the release of Leopard is imminent-ish, I thought I'd do an update of my Time's Arrow post from a year ago regarding SuperDuper! and Time Machine. Let's get into it!

Leopard Compatbitility

First, v2.1.4 of SuperDuper! is not fully compatible with Leopard.

Changes were made to Leopard since our last release in February. We've been following Leopard's development closely, and we have a working version of SuperDuper! nearly ready. Given the nature of SuperDuper!, though, we don't want to release it until we have the final "bits" of Leopard to test with. It should be available shortly after Leopard's release (earlier if we get the final build before the public does--which we won't).

To answer the inevitable question: the update will be free when released.

Note that netTunes is not Leopard compatible due to a Leopard bug I was unable to work around. I'm hoping for a fix in an upcoming Leopard release, since I do not have a workaround for the problem.

Leopard and Sandboxes

When the Leopard-compatible version is released, please realize that Leopard is a pretty different version of the OS. I recommend against using a Sandbox to share a Home folder between Tiger/Panther/Jaguar and Leopard: there have been a lot of changes to the way data is stored, and a Sandbox is not likely to provide sufficient isolation.

Time Machine

As far as Time Machine goes -- in general, I've always thought that some sort of backup functionality belongs in the OS. It's been a long time coming. The fact that it wasn't there left opportunities for 3rd parties, but that doesn't mean Apple shouldn't address the missing functionality.

And so, they have, with Time Machine. Really, I think that's a great thing. People need to back up more often, and I think Time Machine encourages them to do so, and gives them a relatively transparent way to do it.

Time Machine and SuperDuper!

That said, though, Time Machine isn't the be-all and end-all of backup programs. In fact, given how it works, I really do think that SuperDuper! remains both relevant and necessary -- a true complement to the functionality in Time Machine.

First, as is likely obvious, Time Machine is designed to provide automatic "temporal" backup (discussed in broad terms in the post The Ninety-Nine-Per-Cent Solution many months ago). Its primary usage scenario -- and the one that most demonstrations and documentation focuses on -- is to allow quick recovery of files and data that have gone missing, etc.

It does this in a way that's highly integrated with the OS, with a unique UI that's both cool and kinda cheesy (I'll tell you, the 'space' theme hasn't grown on me at all...)... and, as was the case with Spotlight, with a certain amount of application-level impact (something 3rd parties like Shirt Pocket could never mandate).

Fully, Directly Bootable Backups with SuperDuper!

What's important to note is that this isn't, and never was, what SuperDuper! was designed to do.

Our tagline, Heroic System Recovery for Mere Mortals, tries to sum up the whole idea: SuperDuper! is designed to provide excellent failover support for the all-too-common case where things fail in a pretty catastrophic way, such as when a drive fails, or your system becomes unbootable. We do this by quickly and efficiently creating a fully bootable copy of your source drive. Perhaps more importantly, recovery is near immediate, even if the original drive is completely unusable, because you can start up from your backup and continue working.

You can even take your backup to a totally different Macintosh, start up from it, and work while your failed Macintosh is in the shop... then, when it comes back all fresh and shiny, restore things and keep working. And even if the other Mac is a different CPU type, you can still open and edit the files on the backup.

You cannot do this with Time Machine: Time Machine copies are not bootable until they're restored.

In SuperDuper!, system recovery is done with a minimum of fuss and bother, and with respect for your time. Yes, Time Machine can restore a full system, but that's not its strength. Doing so requires you to actually start up from the Leopard DVD (which you'll need to have with you) and then take the time to restore the backup in full, which interrupts your workflow, requires a working, entirely separate destination device, and takes a lot of your time -- at the exact moment when you can least afford it.

So, when Leopard comes out, and Time Machine is released, be assured that SuperDuper!'s fully bootable backups continue to be relevant and necessary. We'll work alongside its rapid recovery of individual files, augmenting that with our rapid system recovery.

And, of course, we'll continue to improve every part of SuperDuper! to make backups faster and easier for all.

For those who haven't heard me blather at length about all things SuperDuper! and Shirt Pocket, there's a 45 minute interview of me up at MacVoices.

Remember: the game rules require you to drink every time you hear me say "Um", "Uh" or audibly gesticulate. Do not play while driving.

tv is in the house -- two of them, actually -- and, well, it's good!

I'm not using it for music much -- my library is much too large for this device, and navigation of large collections, as has been said elsewhere, is quite lacking.

That said, for video material -- movies, tv shows, etc -- it works great. Playback starts quickly, even when streamed, and looks quite good. It's lacking in the audio department (it's really too bad that so much of this material, both movies and TV, encoded with Dolby Digital in full 5.1 or 7.1 surround, are reduced to ancient Dolby Surround playback, with no LFE channel, no split surrounds... much less impact), but visually things look quite reasonable.

It works great with netTunes, too, as you'd expect. As I've said many times, I run with a headless server, and it contains all my music and other content. With netTunes, it's trivial to connect to the server using a laptop while seated on the couch, and "pair" the tv with the server, change the synchronization information, purchase TV shows to be viewed -- all remotely.

I'm happy that the approach I took years ago -- truly remote controlling iTunes with its own interface -- continues to work with new versions of iTunes, and continues to prove it was the right way to go, moving forward with iTunes as iTunes itself changes.

Anyway, great stuff.

One expensive but potentially useful tip: you can use a scan converter to convert from Component input to regular Y/C (S-Video) or Composite, should you not have a component/HDMI capable TV. One example is the TV One AVT-3190 ($389). Expensive, but cheaper than replacing your TV...

You know, it’s pretty rare that I post up here about netTunes, mostly because… well… it just works, and there’s not much to say other than “Whoa! Cool!"… which gets a bit boring.

But, every so often I’ll get a question about how it’s possible to access commands that aren’t available in the main window—for example, Store > Check for Purchases...

So, from the department of obscure features, let’s get this done!

Normally, you’d access things like this with an onscreen control or keyboard shortcut. For example, to access iTunes’ Preferences, you can press Cmd+y (an alternate for Cmd+,) and it’ll come right up. And you can add a playlist with the + button in the main window. But Check for Purchases doesn’t have a keyboard shortcut or button, and so it’s not available in netTunes.

But there’s hope! In Tiger, you can add your own keyboard shortcuts using the Keyboard & Mouse preference pane. Open that up, switch to the Keyboard Shortcuts tab and click the + button. Set the Application pop-up to iTunes, then enter the full, properly capitalized text of the menu item (in this case, Check for Purchases..., including the ...). Choose a shortcut key (like Cmd+Opt+C).

Save, then quit and restart iTunes and poof! You can now Check for Purchases in netTunes on the client!

Wow.

It looks like SuperDuper! 2.1 has won another Macworld Eddy! Back-to-back awards for SuperDuper! in 2005 and 2006, and one for netTunes in 2004—w00t!

Thanks, Macworld—and thanks to all the users, too!

The software business is a strange one, because the products you make eventually just… vanish. The OS goes away, or the market moves on, and soon there’s nothing left to show when someone asks what you do, or what you did.

Getting these three Eddy awards—one for netTunes and two for SuperDuper!—has been really gratifying, both because it’s an acknowledgment of good work, and because it’s more of a permanent thing—a physical record of what I was doing for all those years.

Plus, the trophy doubles as a weapon in an emergency. Man, these things are heavy!

Well, the rollout of the new netTunes and launchTunes went well except for one thing—due to what looks to be some API change in the Accelerate framework, the server was sending a purple iTunes image when running on Intel.

Since I’d done the original engineering and testing of the server way back in April, and hadn’t changed the server code, I skipped one part of my final testing… namely, checking the server on Intel. I verified that it started and that the purchase process worked, but I didn’t check its server functionality… big mistake.

That kind of thing always comes back to bite you, and bite it did. Took me most of the morning to diagnose and fix, but 2.3.1 was released to take care of the problem a little while ago. Sorry about that.

Shirt Pocket releases netTunes 2.3 and launchTunes 1.1
The Macworld Eddy Award Winning “No Compromises” iTunes Remote - now Universal!

Weston, MA – August 23, 2006:  Shirt Pocket announces the immediate availability of netTunes 2.3, the latest update to the Macworld Eddy Award winning remote control for iTunes, and the perfect companion to Apple’s AirPort Express and launchTunes 1.1, the application that guarantees your shared iTunes libraries are available without all that pesky walking.

netTunes lets you control iTunes running on one Macintosh from another, using iTunes’ native interface. You get the same window, the same playlists, the same capabilities. You simply run netTunes and take complete control of the “remote” iTunes from any Macintosh in your house. It’s that easy — as easy as iTunes itself!

“It’s been a long time coming, but the new versions of netTunes and launchTunes are ready to go” says David Nanian, the founder of Shirt Pocket. “But we didn’t just recompile—we’ve taken advantage of both the Intel and Power PC platforms by improving performance across the board, and polished the user experience as well.”

netTunes and launchTunes are available for immediate download at the Shirt Pocket web site http://www.shirt-pocket.com. Users can evaluate all of the capabilities of netTunes for free for 30 minutes at a time; full licenses cost $19.95, and can be ordered at the Shirt Pocket web site, or directly from the application. launchTunes costs $7, and the Tune Suite—a bundle of netTunes and launchTunes—is offered at $23.95 - a $3 savings.

About Shirt Pocket
Shirt Pocket, based in Weston, Massachusetts, was formed in late 2000 as a Macintosh-only shareware creator and publisher. Shirt Pocket’s first product, the Eddy award winning netTunes, lets users control iTunes on one Mac from any other Mac on the network with iTunes own intuitive user interface. launchTunes, Shirt Pocket’s second product, made iTunes’ playlist sharing practical by automatically launching iTunes on remote servers when needed. And its third, the Eddy award winning SuperDuper!, is one of the most highly acclaimed backup/cloning programs available for the Mac. All are available from the Shirt Pocket web site at http://www.shirt-pocket.com.

Shirt Pocket was started by David Nanian, co-founder of UnderWare, Inc, and one of the original authors of the BRIEF programmer’s editor and Track Record bug tracking system.

A few quick pictures of Taiko as I prepare for the rollout of the new netTunes and launchTunes this week…

OK! The testing of netTunes and launchTunes, its purchasing process, Universal Support and integrated store is complete. Things are basically ready to go, except for one thing—I’m waiting for a new version of the eSellerate Integrated Web Store library to be released.

This was supposed to go sometime a week or two ago, but unfortunately got delayed. It’s important, though, because the store installation procedure leaves a file in /tmp that, for some reason, Tiger feels is necessary to move into “Recovered Items” in your trash.

They’ve been working on a fix for that, and I’ve been waiting for that fix to become available. Any day now, I’m told, and I’m passing that savings on to you!