You'd think, given I've written everything here, I'd learn to pay attention to what I've said in the past.
Let me explain. In v3.2.4, we started getting occasional reports of scheduled copies running more than once. Which seemed kind of crazy, because, well, our little time-based backup agent—sdbackupbytime—only runs once a minute, and I'd reviewed the code so many times. There was no chance of a logic error: it was only going to run these puppies one time.
But it was, sometimes, regardless. And one of the things I had to change to complete Smart Wake was to loop to stay active when there was a backup coming in the next minute. But the code was definitely right. Could that have caused this to happen?
(Spoiler alert after many code reviews, hand simulations and debugger step-throughs: NO.)
So if that code was right, why the heck were some users getting occasional multiple runs? And why wasn't it happening to us here at Shirt Pocket HQ?
Of course, anyone who reported any of these problems received an intermediate build that fixed them all. We didn't rush out a new update because the side effect was more copies rather than fewer.
Secondary Effects
Well, one thing we did in 3.2.4 was change what SuperDuper does when it starts.
Previously, in order to work around problems with Clean My Mac (which, for some reason, incorrectly disables our scheduling when a cleanup pass is run, much to my frustration), we changed SuperDuper to reload our LaunchAgents when it starts, in order to self-repair.
This worked fine, except when sdbackupbytime was waiting a minute to run another backup, or when sdbackuponmount was processing multiple drives. In that case, it could potentially kill the process waiting to complete its operation. So, in v3.2.4, we check to see if the version is different, and only perform the reload if the agent was updated.
The known problem with this is that it wouldn't fix Clean My Mac's disabling until the next update. But it also had a secondary effect: these occasional multiple runs. But why?
Well, believe it or not, it's because of ThrottleInterval. Again.
Readers may recall that launchd's ThrottleInterval doesn't really control how often a job might launch, "throttling" it to only once every n seconds. It actually forces a task to relaunch if it doesn't run for at least n seconds.
It does this even if a task succeeds, exits normally, and is set up to run, say, every minute.
sdbackupbytime is a pretty efficient little program, and takes but a fraction of a second to do what it needs to do. When it's done, it exits normally. But if it took less than 10 seconds to run, the system (sometimes) starts it again...and since it's in the same minute, it would run the same schedule a second time.
The previous behavior of SuperDuper masked this operation, because when it launched it killed the agent that had been re-launched: a secondary, unexpected effect. So the problem didn't show up until we stopped doing that.
And I didn't expect ThrottleInterval to apply to time-based agents, because you can set things to run down to every second, so why would it re-launch an agent that was going to have itself launched every minute? (It's not that I can't come up with reasons, it's just that those reasons are covered by other keys, like KeepAlive.)
Anyway, I changed sdbackupbytime to pointlessly sleep up to ThrottleInterval seconds if it was going to exit "too quickly", and the problem was solved...by doing something dumb.
Hey, you do what you have to do, you know? (And sometimes what you have to do is pay attention to your own damn posts.)
Queue'd
Another big thing we did was rework our AppleScript "queue" management to improve it. Which we did.
But we also broke something.
Basically, the queue maintains a semaphore that only allows one client at a time through to run copies using AppleScript. And we remove that client from the queue when its process exits.
The other thing we do is postpone quit when there are clients in the queue, to make sure it doesn't quit out from under someone who's waiting to take control.
In 3.2.4, we started getting reports that the "Shutdown on successful completion" command wasn't working, because SuperDuper wasn't quitting.
Basically, the process sending the Quit command was queued as a task trying to control us, and it never quit...so we deferred the quit until the queue drained, which never happened.
We fixed this and a few similar cases (people using keyboard managers or other things).
Leaving Things Out
As you might know, APFS supports "sparse files" which are, basically, files where "unwritten" data takes up no space on a drive. So, you might have a file that was preallocated with a size of 200GB, but if you only wrote to the first 1MB, it would only take up 1MB on the drive.
These types of files aren't used a lot, but they are used by Docker and VirtualBox, and we noticed that Docker and VirtualBox files were taking much longer to copy than we were comfortable with.
Our sparse file handling tried to copy a sparse file in a way that ensured it was taking a minimal amount of space. That meant we scanned every read block for zeros, and didn't write the sections of files that were 0 by seeking ahead from that point, and writing the non-zero part of the block.
The problem with this is that it takes a lot of time to scan every block. On top of that, there were some unusual situations where the OS would do...weird things...with certain types of seek operations, especially at the end of a file.
So, in 3.2.5, we've changed this so that rather than write a file "optimally", maximizing the number of holes, we write it "accurately". That is, we exactly replicate the sparse structure on the source. This speeds things up tremendously. For example, with a typical sparse docker image, the OS's low-level copyfile function takes 13 minutes to copy with full fidelity, rsync takes 3 minutes and doesn't provide full fidelity, whereas SuperDuper 3.2.5 takes 53 seconds and exactly replicates the source.
That's a win.
Don't Go Away Mad...
In Mojave 10.14.4 or so, we starting getting reports of an error unmounting the snapshot, after which the copy would fail.
I researched the situation, and in this unusual case, we'd ask the OS to eject the volume, it would say it wasn't able to, then we'd ask again (six times), and we'd get an error each time...because it was already unmounted.
So, it would fail to unmount something that it would then unmount. (Winning!)
That's been worked around in this update. (Actual winning!)
Improving the Terrible
As you've seen, Apple's process for granting Full Disk Access is awful. There's almost no guidance in the window—it's like they made it intentionally terrible (I think they did, to discourage people from doing it).
We'd hoped that they'd improve it, and while some small improvements have been made, it hasn't been enough. So, thanks to some work done and generously shared by Media Atelier, we now provide instructions and a draggable application proxy, overlaid on the original window. It's much, much better now.
Thanks again and a tip of the pocket to Pierre Bernard of Houdah Software and Stefan Fuerst of Media Atelier!
eSellerate Shutdown
Our long-time e-commerce provider, eSellerate, is shutting down as of 6/30. So, we had to move to a new "store".
After a long investigation, we've decided to move to Paddle, which—in our opinion—provides the best user experience of all the ones we tried.
Our new purchase process allows payment through more methods, including Apple Pay, and is simpler than before. And we've implemented a special URL scheme so your registration details can now be entered into SuperDuper from the receipt with a single click, which is cool, convenient, and helps to ensure they're correct.
Accomplishing this required quite a bit of additional engineering, including moving to a new serial number system, since eSellerate's was going away. We investigated a number of the existing solutions, and really didn't want to have the gigantically long keys they generated. So we developed our own.
We hope you like the new purchase process: thanks to the folks at Rogue Amoeba, Red Sweater Software, Bare Bones Software and Stand Alone—not to mention Thru-Hiker—for advice, examples and testing.
Note that this means versions of SuperDuper! older than 3.2.5 will not work with the new serial numbers (old serial numbers still work with the new SuperDuper). If you have a new serial number and you need to use it with an old version, please contact support.
(Note that this also means netTunes and launchTunes are no longer available for purchase. They'll be missed, by me at least.)
Various and Sundry
I also spent some time improving Smart Delete in this version; it now looks for files that have shrunk as candidates for pre-moval, and if it can't find any space, but we're writing to an APFS volume, I proactively thin any snapshots to free up space on the container.
All that means even fewer out of space errors. Hooray!
We also significantly improved our animations (which got weird during 10.13) by moving our custom animation code to Core Animation (wrapping ourselves in 2007's warm embrace) and fixed our most longstanding-but-surprisingly-hard-to-fix "stupid" bug: you can now tab between the fields in the Register... window. So, if you had long odds on that, contact your bookie: it's going to pay off big.
With a Bow
So there you go - the update is now available in-app, and of course has been released on the Shirt Pocket site. Download away!
